<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2405.16528] LoQT: Low Rank Adapters for Quantized Training</title><meta property="og:description" content="Training of large neural networks requires significant computational resources. Despite advances using low-rank adapters and quantization, pretraining of models such as LLMs on consumer hardware has not been possible w…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="LoQT: Low Rank Adapters for Quantized Training">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="LoQT: Low Rank Adapters for Quantized Training">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2405.16528">

<!--Generated on Wed Jun  5 16:24:33 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">LoQT: Low Rank Adapters for Quantized Training</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Sebastian Loeschcke 
<br class="ltx_break">University of Copenhagen 
<br class="ltx_break">IT University of Copenhagen
<br class="ltx_break"><a href="mailto:sebastianloeschcke@gmail.com" title="" class="ltx_ref ltx_href ltx_font_typewriter">sbl@di.ku.dk</a> 
<br class="ltx_break">&amp;Mads Toftrup<sup id="id3.2.id1" class="ltx_sup"><span id="id3.2.id1.1" class="ltx_text ltx_font_italic">∗</span></sup>
<br class="ltx_break">Aarhus University 
<br class="ltx_break"><a href="mailto:toftrup@cs.au.dk" title="" class="ltx_ref ltx_href ltx_font_typewriter">toftrup@cs.au.dk</a> 
<br class="ltx_break">Michael J. Kastoryano 
<br class="ltx_break">University of Copenhagen 
<br class="ltx_break"><a href="mailto:mika@di.ku.dk" title="" class="ltx_ref ltx_href ltx_font_typewriter">mika@di.ku.dk</a> 
<br class="ltx_break">&amp;Serge Belongie 
<br class="ltx_break">University of Copenhagen 
<br class="ltx_break"><a href="mailto:s.belongie@di.ku.dk" title="" class="ltx_ref ltx_href ltx_font_typewriter">s.belongie@di.ku.dk</a> 
<br class="ltx_break">&amp;Vésteinn Snæbjarnarson 
<br class="ltx_break">University of Copenhagen
<br class="ltx_break"><a href="mailto:vesteinn.snaebjarnarson@gmail.com" title="" class="ltx_ref ltx_href ltx_font_typewriter">vesn@di.ku.dk</a> 
<br class="ltx_break">
</span><span class="ltx_author_notes">Equal contribution.</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id4.id1" class="ltx_p">Training of large neural networks requires significant computational resources. Despite advances using low-rank adapters and quantization, pretraining of models such as LLMs on consumer hardware has not been possible without model sharding, offloading during training, or per-layer gradient updates. To address these limitations, we propose <span id="id4.id1.1" class="ltx_text ltx_font_smallcaps">LoQT</span>, a method for efficiently training quantized models. <span id="id4.id1.2" class="ltx_text ltx_font_smallcaps">LoQT</span> uses gradient-based tensor factorization to initialize low-rank trainable weight matrices that are periodically merged into quantized full-rank weight matrices. Our approach is suitable for both pretraining and fine-tuning models, which we demonstrate experimentally for language modeling and downstream task adaptation. We find that <span id="id4.id1.3" class="ltx_text ltx_font_smallcaps">LoQT</span> enables efficient training of models up to 7B parameters on a consumer-grade 24GB GPU. We also demonstrate the feasibility of training a 13B parameter model using per-layer gradient updates on the same hardware.</p>
<img src="/html/2405.16528/assets/src/Figures/github.png" id="id2.g1" class="ltx_graphics ltx_img_square" width="18" height="16" alt="[Uncaptioned image]">
<p id="id5.id2" class="ltx_p ltx_parbox ltx_align_middle" style="width:409.8pt;"><a target="_blank" href="https://github.com/sebulo/LoQT" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/sebulo/LoQT</a></p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<figure id="S1.F1" class="ltx_figure ltx_align_floatright"><img src="/html/2405.16528/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="197" height="138" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S1.F1.2.1.1" class="ltx_text" style="font-size:90%;">Figure 1</span>: </span><span id="S1.F1.3.2" class="ltx_text" style="font-size:90%;">Memory usage of Llama 13B, rank 1024. PL: per-layer gradient updates. A8bit: Adam 8-bit</span></figcaption>
</figure>
<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Training large neural networks requires substantial hardware and energy resources. Reducing these requirements is thus important for cost efficiency and environmental sustainability, while also lowering the entry barrier for researchers and practitioners. The main barriers in training large models are the compute operations required, as well as the memory needed to store those computations, in this paper we focus on the latter. Memory use during training comes primarily from the weights of the model itself as well as the optimizer states used to train the model.
To target the weights, variations on low-rank adaptation (LoRA) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>, <a href="#bib.bib12" title="" class="ltx_ref">12</a>, <a href="#bib.bib6" title="" class="ltx_ref">6</a>, <a href="#bib.bib22" title="" class="ltx_ref">22</a>, <a href="#bib.bib23" title="" class="ltx_ref">23</a>]</cite> have been suggested to decrease the number of trainable parameters, in combination with the use of low precision representations.To target the optimizer, low-rank approaches for projecting gradients to a lower rank have been used <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite>. Finally, various applications of quantization <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib10" title="" class="ltx_ref">10</a>, <a href="#bib.bib26" title="" class="ltx_ref">26</a>, <a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite> have been used to decrease memory requirements. In this work, we combine these approaches into a highly memory-efficient training configuration.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">In typical training setups, the optimizer states take up larger space than the model itself, as methods such as Adam <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib18" title="" class="ltx_ref">18</a>]</cite> need to keep track of two parameters for each weight of the model. GaLore <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite> significantly reduces the number of parameters needed for storing the optimizer states by only keeping track of the optimizer state in a low-rank projection which is then projected up to be applied to the model weights. Combining this method with quantization would further shrink the footprint of the model. However, updating the weights of a highly quantized model directly in low-precision space has not been shown to work. This is mainly due to the higher-precision gradient updates being too small to have an impact on the lower-precision quantized states. Lastly, while LoRA is memory efficient for parameter-efficient fine-tuning of pre-trained models, it does not work as a pretraining method by itself <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib22" title="" class="ltx_ref">22</a>]</cite>.</p>
</div>
<figure id="S1.F2" class="ltx_figure"><img src="/html/2405.16528/assets/x2.png" id="S1.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="368" height="139" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S1.F2.17.8.1" class="ltx_text" style="font-size:90%;">Figure 2</span>: </span><span id="S1.F2.14.7" class="ltx_text" style="font-size:90%;">Overview of <span id="S1.F2.14.7.1" class="ltx_text ltx_font_smallcaps">LoQT</span>. (1) Low-rank factors <math id="S1.F2.8.1.m1.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S1.F2.8.1.m1.1b"><mi mathcolor="#6C8EBF" id="S1.F2.8.1.m1.1.1" xref="S1.F2.8.1.m1.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S1.F2.8.1.m1.1c"><ci id="S1.F2.8.1.m1.1.1.cmml" xref="S1.F2.8.1.m1.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.F2.8.1.m1.1d">P</annotation></semantics></math> and <math id="S1.F2.9.2.m2.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S1.F2.9.2.m2.1b"><mi mathcolor="#D79B00" id="S1.F2.9.2.m2.1.1" xref="S1.F2.9.2.m2.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S1.F2.9.2.m2.1c"><ci id="S1.F2.9.2.m2.1.1.cmml" xref="S1.F2.9.2.m2.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.F2.9.2.m2.1d">B</annotation></semantics></math> are periodically initialized from the gradient of the dequantized model weights <math id="S1.F2.10.3.m3.1" class="ltx_Math" alttext="\nabla W" display="inline"><semantics id="S1.F2.10.3.m3.1b"><mrow id="S1.F2.10.3.m3.1.1" xref="S1.F2.10.3.m3.1.1.cmml"><mo mathcolor="#82B366" rspace="0.167em" id="S1.F2.10.3.m3.1.1.1" xref="S1.F2.10.3.m3.1.1.1.cmml">∇</mo><mi mathcolor="#82B366" id="S1.F2.10.3.m3.1.1.2" xref="S1.F2.10.3.m3.1.1.2.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S1.F2.10.3.m3.1c"><apply id="S1.F2.10.3.m3.1.1.cmml" xref="S1.F2.10.3.m3.1.1"><ci id="S1.F2.10.3.m3.1.1.1.cmml" xref="S1.F2.10.3.m3.1.1.1">∇</ci><ci id="S1.F2.10.3.m3.1.1.2.cmml" xref="S1.F2.10.3.m3.1.1.2">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F2.10.3.m3.1d">\nabla W</annotation></semantics></math>, (2) then only <math id="S1.F2.11.4.m4.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S1.F2.11.4.m4.1b"><mi mathcolor="#D79B00" id="S1.F2.11.4.m4.1.1" xref="S1.F2.11.4.m4.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S1.F2.11.4.m4.1c"><ci id="S1.F2.11.4.m4.1.1.cmml" xref="S1.F2.11.4.m4.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.F2.11.4.m4.1d">B</annotation></semantics></math> is trained while <math id="S1.F2.12.5.m5.1" class="ltx_Math" alttext="P_{q}" display="inline"><semantics id="S1.F2.12.5.m5.1b"><msub id="S1.F2.12.5.m5.1.1" xref="S1.F2.12.5.m5.1.1.cmml"><mi mathcolor="#6C8EBF" id="S1.F2.12.5.m5.1.1.2" xref="S1.F2.12.5.m5.1.1.2.cmml">P</mi><mi mathcolor="#6C8EBF" id="S1.F2.12.5.m5.1.1.3" xref="S1.F2.12.5.m5.1.1.3.cmml">q</mi></msub><annotation-xml encoding="MathML-Content" id="S1.F2.12.5.m5.1c"><apply id="S1.F2.12.5.m5.1.1.cmml" xref="S1.F2.12.5.m5.1.1"><csymbol cd="ambiguous" id="S1.F2.12.5.m5.1.1.1.cmml" xref="S1.F2.12.5.m5.1.1">subscript</csymbol><ci id="S1.F2.12.5.m5.1.1.2.cmml" xref="S1.F2.12.5.m5.1.1.2">𝑃</ci><ci id="S1.F2.12.5.m5.1.1.3.cmml" xref="S1.F2.12.5.m5.1.1.3">𝑞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F2.12.5.m5.1d">P_{q}</annotation></semantics></math> and <math id="S1.F2.13.6.m6.1" class="ltx_Math" alttext="W_{q}" display="inline"><semantics id="S1.F2.13.6.m6.1b"><msub id="S1.F2.13.6.m6.1.1" xref="S1.F2.13.6.m6.1.1.cmml"><mi mathcolor="#6C8EBF" id="S1.F2.13.6.m6.1.1.2" xref="S1.F2.13.6.m6.1.1.2.cmml">W</mi><mi mathcolor="#6C8EBF" id="S1.F2.13.6.m6.1.1.3" xref="S1.F2.13.6.m6.1.1.3.cmml">q</mi></msub><annotation-xml encoding="MathML-Content" id="S1.F2.13.6.m6.1c"><apply id="S1.F2.13.6.m6.1.1.cmml" xref="S1.F2.13.6.m6.1.1"><csymbol cd="ambiguous" id="S1.F2.13.6.m6.1.1.1.cmml" xref="S1.F2.13.6.m6.1.1">subscript</csymbol><ci id="S1.F2.13.6.m6.1.1.2.cmml" xref="S1.F2.13.6.m6.1.1.2">𝑊</ci><ci id="S1.F2.13.6.m6.1.1.3.cmml" xref="S1.F2.13.6.m6.1.1.3">𝑞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F2.13.6.m6.1d">W_{q}</annotation></semantics></math> are kept quantized and frozen, over an exponentially increasing interval until <math id="S1.F2.14.7.m7.1" class="ltx_Math" alttext="T_{i}" display="inline"><semantics id="S1.F2.14.7.m7.1b"><msub id="S1.F2.14.7.m7.1.1" xref="S1.F2.14.7.m7.1.1.cmml"><mi mathcolor="#B266FF" id="S1.F2.14.7.m7.1.1.2" xref="S1.F2.14.7.m7.1.1.2.cmml">T</mi><mi mathcolor="#B266FF" id="S1.F2.14.7.m7.1.1.3" xref="S1.F2.14.7.m7.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S1.F2.14.7.m7.1c"><apply id="S1.F2.14.7.m7.1.1.cmml" xref="S1.F2.14.7.m7.1.1"><csymbol cd="ambiguous" id="S1.F2.14.7.m7.1.1.1.cmml" xref="S1.F2.14.7.m7.1.1">subscript</csymbol><ci id="S1.F2.14.7.m7.1.1.2.cmml" xref="S1.F2.14.7.m7.1.1.2">𝑇</ci><ci id="S1.F2.14.7.m7.1.1.3.cmml" xref="S1.F2.14.7.m7.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F2.14.7.m7.1d">T_{i}</annotation></semantics></math>, (3) the low-rank factors are merged back into the quantized model. The process is repeated until training halts.</span></figcaption>
</figure>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.9" class="ltx_p">To address these shortcomings, we propose a new method, <span id="S1.p3.9.1" class="ltx_text ltx_font_smallcaps">LoQT</span>. <span id="S1.p3.9.2" class="ltx_text ltx_font_smallcaps">LoQT</span> initializes two low-rank factors for each weight matrix <math id="S1.p3.1.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S1.p3.1.m1.1a"><mi id="S1.p3.1.m1.1.1" xref="S1.p3.1.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S1.p3.1.m1.1b"><ci id="S1.p3.1.m1.1.1.cmml" xref="S1.p3.1.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.1.m1.1c">W</annotation></semantics></math>: 1) <math id="S1.p3.2.m2.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S1.p3.2.m2.1a"><mi id="S1.p3.2.m2.1.1" xref="S1.p3.2.m2.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S1.p3.2.m2.1b"><ci id="S1.p3.2.m2.1.1.cmml" xref="S1.p3.2.m2.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.2.m2.1c">P</annotation></semantics></math>, using a projection of <math id="S1.p3.3.m3.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S1.p3.3.m3.1a"><mi id="S1.p3.3.m3.1.1" xref="S1.p3.3.m3.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S1.p3.3.m3.1b"><ci id="S1.p3.3.m3.1.1.cmml" xref="S1.p3.3.m3.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.3.m3.1c">W</annotation></semantics></math>’s gradients into a low-rank subspace, and 2) <math id="S1.p3.4.m4.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S1.p3.4.m4.1a"><mi id="S1.p3.4.m4.1.1" xref="S1.p3.4.m4.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S1.p3.4.m4.1b"><ci id="S1.p3.4.m4.1.1.cmml" xref="S1.p3.4.m4.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.4.m4.1c">B</annotation></semantics></math>, initialized to minimize the error of quantizing and represents the only low-rank matrix being trained. Only training <math id="S1.p3.5.m5.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S1.p3.5.m5.1a"><mi id="S1.p3.5.m5.1.1" xref="S1.p3.5.m5.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S1.p3.5.m5.1b"><ci id="S1.p3.5.m5.1.1.cmml" xref="S1.p3.5.m5.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.5.m5.1c">B</annotation></semantics></math> means that the optimizer state can be shrunk significantly. The product <math id="S1.p3.6.m6.1" class="ltx_Math" alttext="PB" display="inline"><semantics id="S1.p3.6.m6.1a"><mrow id="S1.p3.6.m6.1.1" xref="S1.p3.6.m6.1.1.cmml"><mi id="S1.p3.6.m6.1.1.2" xref="S1.p3.6.m6.1.1.2.cmml">P</mi><mo lspace="0em" rspace="0em" id="S1.p3.6.m6.1.1.1" xref="S1.p3.6.m6.1.1.1.cmml">​</mo><mi id="S1.p3.6.m6.1.1.3" xref="S1.p3.6.m6.1.1.3.cmml">B</mi></mrow><annotation-xml encoding="MathML-Content" id="S1.p3.6.m6.1b"><apply id="S1.p3.6.m6.1.1.cmml" xref="S1.p3.6.m6.1.1"><times id="S1.p3.6.m6.1.1.1.cmml" xref="S1.p3.6.m6.1.1.1"></times><ci id="S1.p3.6.m6.1.1.2.cmml" xref="S1.p3.6.m6.1.1.2">𝑃</ci><ci id="S1.p3.6.m6.1.1.3.cmml" xref="S1.p3.6.m6.1.1.3">𝐵</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.6.m6.1c">PB</annotation></semantics></math> is periodically merged into the full rank matrix <math id="S1.p3.7.m7.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S1.p3.7.m7.1a"><mi id="S1.p3.7.m7.1.1" xref="S1.p3.7.m7.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S1.p3.7.m7.1b"><ci id="S1.p3.7.m7.1.1.cmml" xref="S1.p3.7.m7.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.7.m7.1c">W</annotation></semantics></math> with exponentially increasing scheduling. As <math id="S1.p3.8.m8.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S1.p3.8.m8.1a"><mi id="S1.p3.8.m8.1.1" xref="S1.p3.8.m8.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S1.p3.8.m8.1b"><ci id="S1.p3.8.m8.1.1.cmml" xref="S1.p3.8.m8.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.8.m8.1c">W</annotation></semantics></math> and <math id="S1.p3.9.m9.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S1.p3.9.m9.1a"><mi id="S1.p3.9.m9.1.1" xref="S1.p3.9.m9.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S1.p3.9.m9.1b"><ci id="S1.p3.9.m9.1.1.cmml" xref="S1.p3.9.m9.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.9.m9.1c">P</annotation></semantics></math> do not receive gradient updates, they can be quantized, thus optimizing memory usage even further. We stress that it is the large accumulated updates that make it possible to update a quantized model as the addition of smaller changes would not register in the quantized state. A high-level overview is given in Fig. <a href="#S1.F2" title="Figure 2 ‣ 1 Introduction ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">We show that <span id="S1.p4.1.1" class="ltx_text ltx_font_smallcaps">LoQT</span>  works well with and without a quantized model, enabling not only a lower memory footprint in the optimizer state but also over the model parameters. Our results show that we get competitive performance to prior methods using significantly less memory, in particular when using quantization of the model weights in an application such as training an LLM. We also demonstrate superior performance when fine-tuning pre-trained models, by training and evaluating on the GLUE <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib36" title="" class="ltx_ref">36</a>]</cite> benchmark for natural language understanding. Finally, we ablate several properties of the suggested approach and we find that an exponentially increasing projection gap is beneficial, not only to our work but also for GaLore. This is particularly crucial for the training of quantized models. <span id="S1.p4.1.2" class="ltx_text ltx_font_smallcaps">LoQT</span>  enables efficient training of 7B models on consumer-grade hardware with 24GB of memory, and makes it feasible to train models with up to 13 billion parameters without model parallel by making use of per-layer gradient updates <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib25" title="" class="ltx_ref">25</a>]</cite> on the same hardware as shown in Fig. <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work and Background</h2>

<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Neural Network Quantization and NF4</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.5" class="ltx_p">Quantization compresses neural networks by converting high-precision values into lower-precision formats, significantly reducing storage requirements <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib43" title="" class="ltx_ref">43</a>, <a href="#bib.bib33" title="" class="ltx_ref">33</a>, <a href="#bib.bib1" title="" class="ltx_ref">1</a>, <a href="#bib.bib4" title="" class="ltx_ref">4</a>]</cite>. The process involves taking a datatype of high precision, such as 32-bit, requiring 4 bytes of memory, and converting it into a representation with increasing rounding errors but lower memory cost.
In this work, we use NF4 quantization <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite>, since it is a 4-bit code it only contains <math id="S2.SS1.p1.1.m1.1" class="ltx_Math" alttext="2^{4}" display="inline"><semantics id="S2.SS1.p1.1.m1.1a"><msup id="S2.SS1.p1.1.m1.1.1" xref="S2.SS1.p1.1.m1.1.1.cmml"><mn id="S2.SS1.p1.1.m1.1.1.2" xref="S2.SS1.p1.1.m1.1.1.2.cmml">2</mn><mn id="S2.SS1.p1.1.m1.1.1.3" xref="S2.SS1.p1.1.m1.1.1.3.cmml">4</mn></msup><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.1.m1.1b"><apply id="S2.SS1.p1.1.m1.1.1.cmml" xref="S2.SS1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.1.m1.1.1.1.cmml" xref="S2.SS1.p1.1.m1.1.1">superscript</csymbol><cn type="integer" id="S2.SS1.p1.1.m1.1.1.2.cmml" xref="S2.SS1.p1.1.m1.1.1.2">2</cn><cn type="integer" id="S2.SS1.p1.1.m1.1.1.3.cmml" xref="S2.SS1.p1.1.m1.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.1.m1.1c">2^{4}</annotation></semantics></math> different values. NF4 works by first normalizing values onto the interval <math id="S2.SS1.p1.2.m2.1" class="ltx_Math" alttext="[-1:1]" display="inline"><semantics id="S2.SS1.p1.2.m2.1a"><mrow id="S2.SS1.p1.2.m2.1.1.1" xref="S2.SS1.p1.2.m2.1.1.2.cmml"><mo stretchy="false" id="S2.SS1.p1.2.m2.1.1.1.2" xref="S2.SS1.p1.2.m2.1.1.2.1.cmml">[</mo><mrow id="S2.SS1.p1.2.m2.1.1.1.1" xref="S2.SS1.p1.2.m2.1.1.1.1.cmml"><mrow id="S2.SS1.p1.2.m2.1.1.1.1.2" xref="S2.SS1.p1.2.m2.1.1.1.1.2.cmml"><mo id="S2.SS1.p1.2.m2.1.1.1.1.2a" xref="S2.SS1.p1.2.m2.1.1.1.1.2.cmml">−</mo><mn id="S2.SS1.p1.2.m2.1.1.1.1.2.2" xref="S2.SS1.p1.2.m2.1.1.1.1.2.2.cmml">1</mn></mrow><mo lspace="0.278em" rspace="0.278em" id="S2.SS1.p1.2.m2.1.1.1.1.1" xref="S2.SS1.p1.2.m2.1.1.1.1.1.cmml">:</mo><mn id="S2.SS1.p1.2.m2.1.1.1.1.3" xref="S2.SS1.p1.2.m2.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="S2.SS1.p1.2.m2.1.1.1.3" xref="S2.SS1.p1.2.m2.1.1.2.1.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.2.m2.1b"><apply id="S2.SS1.p1.2.m2.1.1.2.cmml" xref="S2.SS1.p1.2.m2.1.1.1"><csymbol cd="latexml" id="S2.SS1.p1.2.m2.1.1.2.1.cmml" xref="S2.SS1.p1.2.m2.1.1.1.2">delimited-[]</csymbol><apply id="S2.SS1.p1.2.m2.1.1.1.1.cmml" xref="S2.SS1.p1.2.m2.1.1.1.1"><ci id="S2.SS1.p1.2.m2.1.1.1.1.1.cmml" xref="S2.SS1.p1.2.m2.1.1.1.1.1">:</ci><apply id="S2.SS1.p1.2.m2.1.1.1.1.2.cmml" xref="S2.SS1.p1.2.m2.1.1.1.1.2"><minus id="S2.SS1.p1.2.m2.1.1.1.1.2.1.cmml" xref="S2.SS1.p1.2.m2.1.1.1.1.2"></minus><cn type="integer" id="S2.SS1.p1.2.m2.1.1.1.1.2.2.cmml" xref="S2.SS1.p1.2.m2.1.1.1.1.2.2">1</cn></apply><cn type="integer" id="S2.SS1.p1.2.m2.1.1.1.1.3.cmml" xref="S2.SS1.p1.2.m2.1.1.1.1.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.2.m2.1c">[-1:1]</annotation></semantics></math>, these are then discretized onto quantiles of the normal distribution, <math id="S2.SS1.p1.3.m3.1" class="ltx_Math" alttext="(q_{i})_{i=1}^{16}" display="inline"><semantics id="S2.SS1.p1.3.m3.1a"><msubsup id="S2.SS1.p1.3.m3.1.1" xref="S2.SS1.p1.3.m3.1.1.cmml"><mrow id="S2.SS1.p1.3.m3.1.1.1.1.1" xref="S2.SS1.p1.3.m3.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS1.p1.3.m3.1.1.1.1.1.2" xref="S2.SS1.p1.3.m3.1.1.1.1.1.1.cmml">(</mo><msub id="S2.SS1.p1.3.m3.1.1.1.1.1.1" xref="S2.SS1.p1.3.m3.1.1.1.1.1.1.cmml"><mi id="S2.SS1.p1.3.m3.1.1.1.1.1.1.2" xref="S2.SS1.p1.3.m3.1.1.1.1.1.1.2.cmml">q</mi><mi id="S2.SS1.p1.3.m3.1.1.1.1.1.1.3" xref="S2.SS1.p1.3.m3.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S2.SS1.p1.3.m3.1.1.1.1.1.3" xref="S2.SS1.p1.3.m3.1.1.1.1.1.1.cmml">)</mo></mrow><mrow id="S2.SS1.p1.3.m3.1.1.1.3" xref="S2.SS1.p1.3.m3.1.1.1.3.cmml"><mi id="S2.SS1.p1.3.m3.1.1.1.3.2" xref="S2.SS1.p1.3.m3.1.1.1.3.2.cmml">i</mi><mo id="S2.SS1.p1.3.m3.1.1.1.3.1" xref="S2.SS1.p1.3.m3.1.1.1.3.1.cmml">=</mo><mn id="S2.SS1.p1.3.m3.1.1.1.3.3" xref="S2.SS1.p1.3.m3.1.1.1.3.3.cmml">1</mn></mrow><mn id="S2.SS1.p1.3.m3.1.1.3" xref="S2.SS1.p1.3.m3.1.1.3.cmml">16</mn></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.3.m3.1b"><apply id="S2.SS1.p1.3.m3.1.1.cmml" xref="S2.SS1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.3.m3.1.1.2.cmml" xref="S2.SS1.p1.3.m3.1.1">superscript</csymbol><apply id="S2.SS1.p1.3.m3.1.1.1.cmml" xref="S2.SS1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.3.m3.1.1.1.2.cmml" xref="S2.SS1.p1.3.m3.1.1">subscript</csymbol><apply id="S2.SS1.p1.3.m3.1.1.1.1.1.1.cmml" xref="S2.SS1.p1.3.m3.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.3.m3.1.1.1.1.1.1.1.cmml" xref="S2.SS1.p1.3.m3.1.1.1.1.1">subscript</csymbol><ci id="S2.SS1.p1.3.m3.1.1.1.1.1.1.2.cmml" xref="S2.SS1.p1.3.m3.1.1.1.1.1.1.2">𝑞</ci><ci id="S2.SS1.p1.3.m3.1.1.1.1.1.1.3.cmml" xref="S2.SS1.p1.3.m3.1.1.1.1.1.1.3">𝑖</ci></apply><apply id="S2.SS1.p1.3.m3.1.1.1.3.cmml" xref="S2.SS1.p1.3.m3.1.1.1.3"><eq id="S2.SS1.p1.3.m3.1.1.1.3.1.cmml" xref="S2.SS1.p1.3.m3.1.1.1.3.1"></eq><ci id="S2.SS1.p1.3.m3.1.1.1.3.2.cmml" xref="S2.SS1.p1.3.m3.1.1.1.3.2">𝑖</ci><cn type="integer" id="S2.SS1.p1.3.m3.1.1.1.3.3.cmml" xref="S2.SS1.p1.3.m3.1.1.1.3.3">1</cn></apply></apply><cn type="integer" id="S2.SS1.p1.3.m3.1.1.3.cmml" xref="S2.SS1.p1.3.m3.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.3.m3.1c">(q_{i})_{i=1}^{16}</annotation></semantics></math> (see <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite> for details). The elements of a layer are divided into blocks of 64 weights. Each block <math id="S2.SS1.p1.4.m4.1" class="ltx_Math" alttext="\beta" display="inline"><semantics id="S2.SS1.p1.4.m4.1a"><mi id="S2.SS1.p1.4.m4.1.1" xref="S2.SS1.p1.4.m4.1.1.cmml">β</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.4.m4.1b"><ci id="S2.SS1.p1.4.m4.1.1.cmml" xref="S2.SS1.p1.4.m4.1.1">𝛽</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.4.m4.1c">\beta</annotation></semantics></math> has a scaling factor <math id="S2.SS1.p1.5.m5.1" class="ltx_Math" alttext="\mathcal{M}_{\beta}=\text{max}_{w\in\beta}|w_{32}|" display="inline"><semantics id="S2.SS1.p1.5.m5.1a"><mrow id="S2.SS1.p1.5.m5.1.1" xref="S2.SS1.p1.5.m5.1.1.cmml"><msub id="S2.SS1.p1.5.m5.1.1.3" xref="S2.SS1.p1.5.m5.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.SS1.p1.5.m5.1.1.3.2" xref="S2.SS1.p1.5.m5.1.1.3.2.cmml">ℳ</mi><mi id="S2.SS1.p1.5.m5.1.1.3.3" xref="S2.SS1.p1.5.m5.1.1.3.3.cmml">β</mi></msub><mo id="S2.SS1.p1.5.m5.1.1.2" xref="S2.SS1.p1.5.m5.1.1.2.cmml">=</mo><mrow id="S2.SS1.p1.5.m5.1.1.1" xref="S2.SS1.p1.5.m5.1.1.1.cmml"><msub id="S2.SS1.p1.5.m5.1.1.1.3" xref="S2.SS1.p1.5.m5.1.1.1.3.cmml"><mtext id="S2.SS1.p1.5.m5.1.1.1.3.2" xref="S2.SS1.p1.5.m5.1.1.1.3.2a.cmml">max</mtext><mrow id="S2.SS1.p1.5.m5.1.1.1.3.3" xref="S2.SS1.p1.5.m5.1.1.1.3.3.cmml"><mi id="S2.SS1.p1.5.m5.1.1.1.3.3.2" xref="S2.SS1.p1.5.m5.1.1.1.3.3.2.cmml">w</mi><mo id="S2.SS1.p1.5.m5.1.1.1.3.3.1" xref="S2.SS1.p1.5.m5.1.1.1.3.3.1.cmml">∈</mo><mi id="S2.SS1.p1.5.m5.1.1.1.3.3.3" xref="S2.SS1.p1.5.m5.1.1.1.3.3.3.cmml">β</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S2.SS1.p1.5.m5.1.1.1.2" xref="S2.SS1.p1.5.m5.1.1.1.2.cmml">​</mo><mrow id="S2.SS1.p1.5.m5.1.1.1.1.1" xref="S2.SS1.p1.5.m5.1.1.1.1.2.cmml"><mo stretchy="false" id="S2.SS1.p1.5.m5.1.1.1.1.1.2" xref="S2.SS1.p1.5.m5.1.1.1.1.2.1.cmml">|</mo><msub id="S2.SS1.p1.5.m5.1.1.1.1.1.1" xref="S2.SS1.p1.5.m5.1.1.1.1.1.1.cmml"><mi id="S2.SS1.p1.5.m5.1.1.1.1.1.1.2" xref="S2.SS1.p1.5.m5.1.1.1.1.1.1.2.cmml">w</mi><mn id="S2.SS1.p1.5.m5.1.1.1.1.1.1.3" xref="S2.SS1.p1.5.m5.1.1.1.1.1.1.3.cmml">32</mn></msub><mo stretchy="false" id="S2.SS1.p1.5.m5.1.1.1.1.1.3" xref="S2.SS1.p1.5.m5.1.1.1.1.2.1.cmml">|</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.5.m5.1b"><apply id="S2.SS1.p1.5.m5.1.1.cmml" xref="S2.SS1.p1.5.m5.1.1"><eq id="S2.SS1.p1.5.m5.1.1.2.cmml" xref="S2.SS1.p1.5.m5.1.1.2"></eq><apply id="S2.SS1.p1.5.m5.1.1.3.cmml" xref="S2.SS1.p1.5.m5.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p1.5.m5.1.1.3.1.cmml" xref="S2.SS1.p1.5.m5.1.1.3">subscript</csymbol><ci id="S2.SS1.p1.5.m5.1.1.3.2.cmml" xref="S2.SS1.p1.5.m5.1.1.3.2">ℳ</ci><ci id="S2.SS1.p1.5.m5.1.1.3.3.cmml" xref="S2.SS1.p1.5.m5.1.1.3.3">𝛽</ci></apply><apply id="S2.SS1.p1.5.m5.1.1.1.cmml" xref="S2.SS1.p1.5.m5.1.1.1"><times id="S2.SS1.p1.5.m5.1.1.1.2.cmml" xref="S2.SS1.p1.5.m5.1.1.1.2"></times><apply id="S2.SS1.p1.5.m5.1.1.1.3.cmml" xref="S2.SS1.p1.5.m5.1.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p1.5.m5.1.1.1.3.1.cmml" xref="S2.SS1.p1.5.m5.1.1.1.3">subscript</csymbol><ci id="S2.SS1.p1.5.m5.1.1.1.3.2a.cmml" xref="S2.SS1.p1.5.m5.1.1.1.3.2"><mtext id="S2.SS1.p1.5.m5.1.1.1.3.2.cmml" xref="S2.SS1.p1.5.m5.1.1.1.3.2">max</mtext></ci><apply id="S2.SS1.p1.5.m5.1.1.1.3.3.cmml" xref="S2.SS1.p1.5.m5.1.1.1.3.3"><in id="S2.SS1.p1.5.m5.1.1.1.3.3.1.cmml" xref="S2.SS1.p1.5.m5.1.1.1.3.3.1"></in><ci id="S2.SS1.p1.5.m5.1.1.1.3.3.2.cmml" xref="S2.SS1.p1.5.m5.1.1.1.3.3.2">𝑤</ci><ci id="S2.SS1.p1.5.m5.1.1.1.3.3.3.cmml" xref="S2.SS1.p1.5.m5.1.1.1.3.3.3">𝛽</ci></apply></apply><apply id="S2.SS1.p1.5.m5.1.1.1.1.2.cmml" xref="S2.SS1.p1.5.m5.1.1.1.1.1"><abs id="S2.SS1.p1.5.m5.1.1.1.1.2.1.cmml" xref="S2.SS1.p1.5.m5.1.1.1.1.1.2"></abs><apply id="S2.SS1.p1.5.m5.1.1.1.1.1.1.cmml" xref="S2.SS1.p1.5.m5.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.5.m5.1.1.1.1.1.1.1.cmml" xref="S2.SS1.p1.5.m5.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS1.p1.5.m5.1.1.1.1.1.1.2.cmml" xref="S2.SS1.p1.5.m5.1.1.1.1.1.1.2">𝑤</ci><cn type="integer" id="S2.SS1.p1.5.m5.1.1.1.1.1.1.3.cmml" xref="S2.SS1.p1.5.m5.1.1.1.1.1.1.3">32</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.5.m5.1c">\mathcal{M}_{\beta}=\text{max}_{w\in\beta}|w_{32}|</annotation></semantics></math>.</p>
<table id="A3.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S2.E1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.E1.m1.1" class="ltx_Math" alttext="\displaystyle w_{\text{NF4}}" display="inline"><semantics id="S2.E1.m1.1a"><msub id="S2.E1.m1.1.1" xref="S2.E1.m1.1.1.cmml"><mi id="S2.E1.m1.1.1.2" xref="S2.E1.m1.1.1.2.cmml">w</mi><mtext id="S2.E1.m1.1.1.3" xref="S2.E1.m1.1.1.3a.cmml">NF4</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.E1.m1.1b"><apply id="S2.E1.m1.1.1.cmml" xref="S2.E1.m1.1.1"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.cmml" xref="S2.E1.m1.1.1">subscript</csymbol><ci id="S2.E1.m1.1.1.2.cmml" xref="S2.E1.m1.1.1.2">𝑤</ci><ci id="S2.E1.m1.1.1.3a.cmml" xref="S2.E1.m1.1.1.3"><mtext mathsize="70%" id="S2.E1.m1.1.1.3.cmml" xref="S2.E1.m1.1.1.3">NF4</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1.m1.1c">\displaystyle w_{\text{NF4}}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.E1.m2.2" class="ltx_Math" alttext="\displaystyle=q_{\text{NF4}}(w,\mathcal{M}_{\beta})\coloneqq\text{argmin}_{q_{i}}|w/\mathcal{M}_{\beta}-q_{i}|," display="inline"><semantics id="S2.E1.m2.2a"><mrow id="S2.E1.m2.2.2.1" xref="S2.E1.m2.2.2.1.1.cmml"><mrow id="S2.E1.m2.2.2.1.1" xref="S2.E1.m2.2.2.1.1.cmml"><mi id="S2.E1.m2.2.2.1.1.4" xref="S2.E1.m2.2.2.1.1.4.cmml"></mi><mo id="S2.E1.m2.2.2.1.1.5" xref="S2.E1.m2.2.2.1.1.5.cmml">=</mo><mrow id="S2.E1.m2.2.2.1.1.1" xref="S2.E1.m2.2.2.1.1.1.cmml"><msub id="S2.E1.m2.2.2.1.1.1.3" xref="S2.E1.m2.2.2.1.1.1.3.cmml"><mi id="S2.E1.m2.2.2.1.1.1.3.2" xref="S2.E1.m2.2.2.1.1.1.3.2.cmml">q</mi><mtext id="S2.E1.m2.2.2.1.1.1.3.3" xref="S2.E1.m2.2.2.1.1.1.3.3a.cmml">NF4</mtext></msub><mo lspace="0em" rspace="0em" id="S2.E1.m2.2.2.1.1.1.2" xref="S2.E1.m2.2.2.1.1.1.2.cmml">​</mo><mrow id="S2.E1.m2.2.2.1.1.1.1.1" xref="S2.E1.m2.2.2.1.1.1.1.2.cmml"><mo stretchy="false" id="S2.E1.m2.2.2.1.1.1.1.1.2" xref="S2.E1.m2.2.2.1.1.1.1.2.cmml">(</mo><mi id="S2.E1.m2.1.1" xref="S2.E1.m2.1.1.cmml">w</mi><mo id="S2.E1.m2.2.2.1.1.1.1.1.3" xref="S2.E1.m2.2.2.1.1.1.1.2.cmml">,</mo><msub id="S2.E1.m2.2.2.1.1.1.1.1.1" xref="S2.E1.m2.2.2.1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.E1.m2.2.2.1.1.1.1.1.1.2" xref="S2.E1.m2.2.2.1.1.1.1.1.1.2.cmml">ℳ</mi><mi id="S2.E1.m2.2.2.1.1.1.1.1.1.3" xref="S2.E1.m2.2.2.1.1.1.1.1.1.3.cmml">β</mi></msub><mo stretchy="false" id="S2.E1.m2.2.2.1.1.1.1.1.4" xref="S2.E1.m2.2.2.1.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="S2.E1.m2.2.2.1.1.6" xref="S2.E1.m2.2.2.1.1.6.cmml">≔</mo><mrow id="S2.E1.m2.2.2.1.1.2" xref="S2.E1.m2.2.2.1.1.2.cmml"><msub id="S2.E1.m2.2.2.1.1.2.3" xref="S2.E1.m2.2.2.1.1.2.3.cmml"><mtext id="S2.E1.m2.2.2.1.1.2.3.2" xref="S2.E1.m2.2.2.1.1.2.3.2a.cmml">argmin</mtext><msub id="S2.E1.m2.2.2.1.1.2.3.3" xref="S2.E1.m2.2.2.1.1.2.3.3.cmml"><mi id="S2.E1.m2.2.2.1.1.2.3.3.2" xref="S2.E1.m2.2.2.1.1.2.3.3.2.cmml">q</mi><mi id="S2.E1.m2.2.2.1.1.2.3.3.3" xref="S2.E1.m2.2.2.1.1.2.3.3.3.cmml">i</mi></msub></msub><mo lspace="0em" rspace="0em" id="S2.E1.m2.2.2.1.1.2.2" xref="S2.E1.m2.2.2.1.1.2.2.cmml">​</mo><mrow id="S2.E1.m2.2.2.1.1.2.1.1" xref="S2.E1.m2.2.2.1.1.2.1.2.cmml"><mo stretchy="false" id="S2.E1.m2.2.2.1.1.2.1.1.2" xref="S2.E1.m2.2.2.1.1.2.1.2.1.cmml">|</mo><mrow id="S2.E1.m2.2.2.1.1.2.1.1.1" xref="S2.E1.m2.2.2.1.1.2.1.1.1.cmml"><mrow id="S2.E1.m2.2.2.1.1.2.1.1.1.2" xref="S2.E1.m2.2.2.1.1.2.1.1.1.2.cmml"><mi id="S2.E1.m2.2.2.1.1.2.1.1.1.2.2" xref="S2.E1.m2.2.2.1.1.2.1.1.1.2.2.cmml">w</mi><mo id="S2.E1.m2.2.2.1.1.2.1.1.1.2.1" xref="S2.E1.m2.2.2.1.1.2.1.1.1.2.1.cmml">/</mo><msub id="S2.E1.m2.2.2.1.1.2.1.1.1.2.3" xref="S2.E1.m2.2.2.1.1.2.1.1.1.2.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.E1.m2.2.2.1.1.2.1.1.1.2.3.2" xref="S2.E1.m2.2.2.1.1.2.1.1.1.2.3.2.cmml">ℳ</mi><mi id="S2.E1.m2.2.2.1.1.2.1.1.1.2.3.3" xref="S2.E1.m2.2.2.1.1.2.1.1.1.2.3.3.cmml">β</mi></msub></mrow><mo id="S2.E1.m2.2.2.1.1.2.1.1.1.1" xref="S2.E1.m2.2.2.1.1.2.1.1.1.1.cmml">−</mo><msub id="S2.E1.m2.2.2.1.1.2.1.1.1.3" xref="S2.E1.m2.2.2.1.1.2.1.1.1.3.cmml"><mi id="S2.E1.m2.2.2.1.1.2.1.1.1.3.2" xref="S2.E1.m2.2.2.1.1.2.1.1.1.3.2.cmml">q</mi><mi id="S2.E1.m2.2.2.1.1.2.1.1.1.3.3" xref="S2.E1.m2.2.2.1.1.2.1.1.1.3.3.cmml">i</mi></msub></mrow><mo stretchy="false" id="S2.E1.m2.2.2.1.1.2.1.1.3" xref="S2.E1.m2.2.2.1.1.2.1.2.1.cmml">|</mo></mrow></mrow></mrow><mo id="S2.E1.m2.2.2.1.2" xref="S2.E1.m2.2.2.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E1.m2.2b"><apply id="S2.E1.m2.2.2.1.1.cmml" xref="S2.E1.m2.2.2.1"><and id="S2.E1.m2.2.2.1.1a.cmml" xref="S2.E1.m2.2.2.1"></and><apply id="S2.E1.m2.2.2.1.1b.cmml" xref="S2.E1.m2.2.2.1"><eq id="S2.E1.m2.2.2.1.1.5.cmml" xref="S2.E1.m2.2.2.1.1.5"></eq><csymbol cd="latexml" id="S2.E1.m2.2.2.1.1.4.cmml" xref="S2.E1.m2.2.2.1.1.4">absent</csymbol><apply id="S2.E1.m2.2.2.1.1.1.cmml" xref="S2.E1.m2.2.2.1.1.1"><times id="S2.E1.m2.2.2.1.1.1.2.cmml" xref="S2.E1.m2.2.2.1.1.1.2"></times><apply id="S2.E1.m2.2.2.1.1.1.3.cmml" xref="S2.E1.m2.2.2.1.1.1.3"><csymbol cd="ambiguous" id="S2.E1.m2.2.2.1.1.1.3.1.cmml" xref="S2.E1.m2.2.2.1.1.1.3">subscript</csymbol><ci id="S2.E1.m2.2.2.1.1.1.3.2.cmml" xref="S2.E1.m2.2.2.1.1.1.3.2">𝑞</ci><ci id="S2.E1.m2.2.2.1.1.1.3.3a.cmml" xref="S2.E1.m2.2.2.1.1.1.3.3"><mtext mathsize="70%" id="S2.E1.m2.2.2.1.1.1.3.3.cmml" xref="S2.E1.m2.2.2.1.1.1.3.3">NF4</mtext></ci></apply><interval closure="open" id="S2.E1.m2.2.2.1.1.1.1.2.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1"><ci id="S2.E1.m2.1.1.cmml" xref="S2.E1.m2.1.1">𝑤</ci><apply id="S2.E1.m2.2.2.1.1.1.1.1.1.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E1.m2.2.2.1.1.1.1.1.1.1.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E1.m2.2.2.1.1.1.1.1.1.2.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.1.2">ℳ</ci><ci id="S2.E1.m2.2.2.1.1.1.1.1.1.3.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.1.3">𝛽</ci></apply></interval></apply></apply><apply id="S2.E1.m2.2.2.1.1c.cmml" xref="S2.E1.m2.2.2.1"><ci id="S2.E1.m2.2.2.1.1.6.cmml" xref="S2.E1.m2.2.2.1.1.6">≔</ci><share href="#S2.E1.m2.2.2.1.1.1.cmml" id="S2.E1.m2.2.2.1.1d.cmml" xref="S2.E1.m2.2.2.1"></share><apply id="S2.E1.m2.2.2.1.1.2.cmml" xref="S2.E1.m2.2.2.1.1.2"><times id="S2.E1.m2.2.2.1.1.2.2.cmml" xref="S2.E1.m2.2.2.1.1.2.2"></times><apply id="S2.E1.m2.2.2.1.1.2.3.cmml" xref="S2.E1.m2.2.2.1.1.2.3"><csymbol cd="ambiguous" id="S2.E1.m2.2.2.1.1.2.3.1.cmml" xref="S2.E1.m2.2.2.1.1.2.3">subscript</csymbol><ci id="S2.E1.m2.2.2.1.1.2.3.2a.cmml" xref="S2.E1.m2.2.2.1.1.2.3.2"><mtext id="S2.E1.m2.2.2.1.1.2.3.2.cmml" xref="S2.E1.m2.2.2.1.1.2.3.2">argmin</mtext></ci><apply id="S2.E1.m2.2.2.1.1.2.3.3.cmml" xref="S2.E1.m2.2.2.1.1.2.3.3"><csymbol cd="ambiguous" id="S2.E1.m2.2.2.1.1.2.3.3.1.cmml" xref="S2.E1.m2.2.2.1.1.2.3.3">subscript</csymbol><ci id="S2.E1.m2.2.2.1.1.2.3.3.2.cmml" xref="S2.E1.m2.2.2.1.1.2.3.3.2">𝑞</ci><ci id="S2.E1.m2.2.2.1.1.2.3.3.3.cmml" xref="S2.E1.m2.2.2.1.1.2.3.3.3">𝑖</ci></apply></apply><apply id="S2.E1.m2.2.2.1.1.2.1.2.cmml" xref="S2.E1.m2.2.2.1.1.2.1.1"><abs id="S2.E1.m2.2.2.1.1.2.1.2.1.cmml" xref="S2.E1.m2.2.2.1.1.2.1.1.2"></abs><apply id="S2.E1.m2.2.2.1.1.2.1.1.1.cmml" xref="S2.E1.m2.2.2.1.1.2.1.1.1"><minus id="S2.E1.m2.2.2.1.1.2.1.1.1.1.cmml" xref="S2.E1.m2.2.2.1.1.2.1.1.1.1"></minus><apply id="S2.E1.m2.2.2.1.1.2.1.1.1.2.cmml" xref="S2.E1.m2.2.2.1.1.2.1.1.1.2"><divide id="S2.E1.m2.2.2.1.1.2.1.1.1.2.1.cmml" xref="S2.E1.m2.2.2.1.1.2.1.1.1.2.1"></divide><ci id="S2.E1.m2.2.2.1.1.2.1.1.1.2.2.cmml" xref="S2.E1.m2.2.2.1.1.2.1.1.1.2.2">𝑤</ci><apply id="S2.E1.m2.2.2.1.1.2.1.1.1.2.3.cmml" xref="S2.E1.m2.2.2.1.1.2.1.1.1.2.3"><csymbol cd="ambiguous" id="S2.E1.m2.2.2.1.1.2.1.1.1.2.3.1.cmml" xref="S2.E1.m2.2.2.1.1.2.1.1.1.2.3">subscript</csymbol><ci id="S2.E1.m2.2.2.1.1.2.1.1.1.2.3.2.cmml" xref="S2.E1.m2.2.2.1.1.2.1.1.1.2.3.2">ℳ</ci><ci id="S2.E1.m2.2.2.1.1.2.1.1.1.2.3.3.cmml" xref="S2.E1.m2.2.2.1.1.2.1.1.1.2.3.3">𝛽</ci></apply></apply><apply id="S2.E1.m2.2.2.1.1.2.1.1.1.3.cmml" xref="S2.E1.m2.2.2.1.1.2.1.1.1.3"><csymbol cd="ambiguous" id="S2.E1.m2.2.2.1.1.2.1.1.1.3.1.cmml" xref="S2.E1.m2.2.2.1.1.2.1.1.1.3">subscript</csymbol><ci id="S2.E1.m2.2.2.1.1.2.1.1.1.3.2.cmml" xref="S2.E1.m2.2.2.1.1.2.1.1.1.3.2">𝑞</ci><ci id="S2.E1.m2.2.2.1.1.2.1.1.1.3.3.cmml" xref="S2.E1.m2.2.2.1.1.2.1.1.1.3.3">𝑖</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1.m2.2c">\displaystyle=q_{\text{NF4}}(w,\mathcal{M}_{\beta})\coloneqq\text{argmin}_{q_{i}}|w/\mathcal{M}_{\beta}-q_{i}|,</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
<tbody id="S2.E2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.E2.m1.1" class="ltx_Math" alttext="\displaystyle w" display="inline"><semantics id="S2.E2.m1.1a"><mi id="S2.E2.m1.1.1" xref="S2.E2.m1.1.1.cmml">w</mi><annotation-xml encoding="MathML-Content" id="S2.E2.m1.1b"><ci id="S2.E2.m1.1.1.cmml" xref="S2.E2.m1.1.1">𝑤</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.E2.m1.1c">\displaystyle w</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.E2.m2.1" class="ltx_Math" alttext="\displaystyle=q_{\text{NF4}}^{-1}(w_{\text{NF4}},\mathcal{M}_{\beta})\coloneqq\mathcal{M}_{\beta}\cdot w_{\text{NF4}}." display="inline"><semantics id="S2.E2.m2.1a"><mrow id="S2.E2.m2.1.1.1" xref="S2.E2.m2.1.1.1.1.cmml"><mrow id="S2.E2.m2.1.1.1.1" xref="S2.E2.m2.1.1.1.1.cmml"><mi id="S2.E2.m2.1.1.1.1.4" xref="S2.E2.m2.1.1.1.1.4.cmml"></mi><mo id="S2.E2.m2.1.1.1.1.5" xref="S2.E2.m2.1.1.1.1.5.cmml">=</mo><mrow id="S2.E2.m2.1.1.1.1.2" xref="S2.E2.m2.1.1.1.1.2.cmml"><msubsup id="S2.E2.m2.1.1.1.1.2.4" xref="S2.E2.m2.1.1.1.1.2.4.cmml"><mi id="S2.E2.m2.1.1.1.1.2.4.2.2" xref="S2.E2.m2.1.1.1.1.2.4.2.2.cmml">q</mi><mtext id="S2.E2.m2.1.1.1.1.2.4.2.3" xref="S2.E2.m2.1.1.1.1.2.4.2.3a.cmml">NF4</mtext><mrow id="S2.E2.m2.1.1.1.1.2.4.3" xref="S2.E2.m2.1.1.1.1.2.4.3.cmml"><mo id="S2.E2.m2.1.1.1.1.2.4.3a" xref="S2.E2.m2.1.1.1.1.2.4.3.cmml">−</mo><mn id="S2.E2.m2.1.1.1.1.2.4.3.2" xref="S2.E2.m2.1.1.1.1.2.4.3.2.cmml">1</mn></mrow></msubsup><mo lspace="0em" rspace="0em" id="S2.E2.m2.1.1.1.1.2.3" xref="S2.E2.m2.1.1.1.1.2.3.cmml">​</mo><mrow id="S2.E2.m2.1.1.1.1.2.2.2" xref="S2.E2.m2.1.1.1.1.2.2.3.cmml"><mo stretchy="false" id="S2.E2.m2.1.1.1.1.2.2.2.3" xref="S2.E2.m2.1.1.1.1.2.2.3.cmml">(</mo><msub id="S2.E2.m2.1.1.1.1.1.1.1.1" xref="S2.E2.m2.1.1.1.1.1.1.1.1.cmml"><mi id="S2.E2.m2.1.1.1.1.1.1.1.1.2" xref="S2.E2.m2.1.1.1.1.1.1.1.1.2.cmml">w</mi><mtext id="S2.E2.m2.1.1.1.1.1.1.1.1.3" xref="S2.E2.m2.1.1.1.1.1.1.1.1.3a.cmml">NF4</mtext></msub><mo id="S2.E2.m2.1.1.1.1.2.2.2.4" xref="S2.E2.m2.1.1.1.1.2.2.3.cmml">,</mo><msub id="S2.E2.m2.1.1.1.1.2.2.2.2" xref="S2.E2.m2.1.1.1.1.2.2.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.E2.m2.1.1.1.1.2.2.2.2.2" xref="S2.E2.m2.1.1.1.1.2.2.2.2.2.cmml">ℳ</mi><mi id="S2.E2.m2.1.1.1.1.2.2.2.2.3" xref="S2.E2.m2.1.1.1.1.2.2.2.2.3.cmml">β</mi></msub><mo stretchy="false" id="S2.E2.m2.1.1.1.1.2.2.2.5" xref="S2.E2.m2.1.1.1.1.2.2.3.cmml">)</mo></mrow></mrow><mo id="S2.E2.m2.1.1.1.1.6" xref="S2.E2.m2.1.1.1.1.6.cmml">≔</mo><mrow id="S2.E2.m2.1.1.1.1.7" xref="S2.E2.m2.1.1.1.1.7.cmml"><msub id="S2.E2.m2.1.1.1.1.7.2" xref="S2.E2.m2.1.1.1.1.7.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.E2.m2.1.1.1.1.7.2.2" xref="S2.E2.m2.1.1.1.1.7.2.2.cmml">ℳ</mi><mi id="S2.E2.m2.1.1.1.1.7.2.3" xref="S2.E2.m2.1.1.1.1.7.2.3.cmml">β</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S2.E2.m2.1.1.1.1.7.1" xref="S2.E2.m2.1.1.1.1.7.1.cmml">⋅</mo><msub id="S2.E2.m2.1.1.1.1.7.3" xref="S2.E2.m2.1.1.1.1.7.3.cmml"><mi id="S2.E2.m2.1.1.1.1.7.3.2" xref="S2.E2.m2.1.1.1.1.7.3.2.cmml">w</mi><mtext id="S2.E2.m2.1.1.1.1.7.3.3" xref="S2.E2.m2.1.1.1.1.7.3.3a.cmml">NF4</mtext></msub></mrow></mrow><mo lspace="0em" id="S2.E2.m2.1.1.1.2" xref="S2.E2.m2.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E2.m2.1b"><apply id="S2.E2.m2.1.1.1.1.cmml" xref="S2.E2.m2.1.1.1"><and id="S2.E2.m2.1.1.1.1a.cmml" xref="S2.E2.m2.1.1.1"></and><apply id="S2.E2.m2.1.1.1.1b.cmml" xref="S2.E2.m2.1.1.1"><eq id="S2.E2.m2.1.1.1.1.5.cmml" xref="S2.E2.m2.1.1.1.1.5"></eq><csymbol cd="latexml" id="S2.E2.m2.1.1.1.1.4.cmml" xref="S2.E2.m2.1.1.1.1.4">absent</csymbol><apply id="S2.E2.m2.1.1.1.1.2.cmml" xref="S2.E2.m2.1.1.1.1.2"><times id="S2.E2.m2.1.1.1.1.2.3.cmml" xref="S2.E2.m2.1.1.1.1.2.3"></times><apply id="S2.E2.m2.1.1.1.1.2.4.cmml" xref="S2.E2.m2.1.1.1.1.2.4"><csymbol cd="ambiguous" id="S2.E2.m2.1.1.1.1.2.4.1.cmml" xref="S2.E2.m2.1.1.1.1.2.4">superscript</csymbol><apply id="S2.E2.m2.1.1.1.1.2.4.2.cmml" xref="S2.E2.m2.1.1.1.1.2.4"><csymbol cd="ambiguous" id="S2.E2.m2.1.1.1.1.2.4.2.1.cmml" xref="S2.E2.m2.1.1.1.1.2.4">subscript</csymbol><ci id="S2.E2.m2.1.1.1.1.2.4.2.2.cmml" xref="S2.E2.m2.1.1.1.1.2.4.2.2">𝑞</ci><ci id="S2.E2.m2.1.1.1.1.2.4.2.3a.cmml" xref="S2.E2.m2.1.1.1.1.2.4.2.3"><mtext mathsize="70%" id="S2.E2.m2.1.1.1.1.2.4.2.3.cmml" xref="S2.E2.m2.1.1.1.1.2.4.2.3">NF4</mtext></ci></apply><apply id="S2.E2.m2.1.1.1.1.2.4.3.cmml" xref="S2.E2.m2.1.1.1.1.2.4.3"><minus id="S2.E2.m2.1.1.1.1.2.4.3.1.cmml" xref="S2.E2.m2.1.1.1.1.2.4.3"></minus><cn type="integer" id="S2.E2.m2.1.1.1.1.2.4.3.2.cmml" xref="S2.E2.m2.1.1.1.1.2.4.3.2">1</cn></apply></apply><interval closure="open" id="S2.E2.m2.1.1.1.1.2.2.3.cmml" xref="S2.E2.m2.1.1.1.1.2.2.2"><apply id="S2.E2.m2.1.1.1.1.1.1.1.1.cmml" xref="S2.E2.m2.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E2.m2.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E2.m2.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E2.m2.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E2.m2.1.1.1.1.1.1.1.1.2">𝑤</ci><ci id="S2.E2.m2.1.1.1.1.1.1.1.1.3a.cmml" xref="S2.E2.m2.1.1.1.1.1.1.1.1.3"><mtext mathsize="70%" id="S2.E2.m2.1.1.1.1.1.1.1.1.3.cmml" xref="S2.E2.m2.1.1.1.1.1.1.1.1.3">NF4</mtext></ci></apply><apply id="S2.E2.m2.1.1.1.1.2.2.2.2.cmml" xref="S2.E2.m2.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S2.E2.m2.1.1.1.1.2.2.2.2.1.cmml" xref="S2.E2.m2.1.1.1.1.2.2.2.2">subscript</csymbol><ci id="S2.E2.m2.1.1.1.1.2.2.2.2.2.cmml" xref="S2.E2.m2.1.1.1.1.2.2.2.2.2">ℳ</ci><ci id="S2.E2.m2.1.1.1.1.2.2.2.2.3.cmml" xref="S2.E2.m2.1.1.1.1.2.2.2.2.3">𝛽</ci></apply></interval></apply></apply><apply id="S2.E2.m2.1.1.1.1c.cmml" xref="S2.E2.m2.1.1.1"><ci id="S2.E2.m2.1.1.1.1.6.cmml" xref="S2.E2.m2.1.1.1.1.6">≔</ci><share href="#S2.E2.m2.1.1.1.1.2.cmml" id="S2.E2.m2.1.1.1.1d.cmml" xref="S2.E2.m2.1.1.1"></share><apply id="S2.E2.m2.1.1.1.1.7.cmml" xref="S2.E2.m2.1.1.1.1.7"><ci id="S2.E2.m2.1.1.1.1.7.1.cmml" xref="S2.E2.m2.1.1.1.1.7.1">⋅</ci><apply id="S2.E2.m2.1.1.1.1.7.2.cmml" xref="S2.E2.m2.1.1.1.1.7.2"><csymbol cd="ambiguous" id="S2.E2.m2.1.1.1.1.7.2.1.cmml" xref="S2.E2.m2.1.1.1.1.7.2">subscript</csymbol><ci id="S2.E2.m2.1.1.1.1.7.2.2.cmml" xref="S2.E2.m2.1.1.1.1.7.2.2">ℳ</ci><ci id="S2.E2.m2.1.1.1.1.7.2.3.cmml" xref="S2.E2.m2.1.1.1.1.7.2.3">𝛽</ci></apply><apply id="S2.E2.m2.1.1.1.1.7.3.cmml" xref="S2.E2.m2.1.1.1.1.7.3"><csymbol cd="ambiguous" id="S2.E2.m2.1.1.1.1.7.3.1.cmml" xref="S2.E2.m2.1.1.1.1.7.3">subscript</csymbol><ci id="S2.E2.m2.1.1.1.1.7.3.2.cmml" xref="S2.E2.m2.1.1.1.1.7.3.2">𝑤</ci><ci id="S2.E2.m2.1.1.1.1.7.3.3a.cmml" xref="S2.E2.m2.1.1.1.1.7.3.3"><mtext mathsize="70%" id="S2.E2.m2.1.1.1.1.7.3.3.cmml" xref="S2.E2.m2.1.1.1.1.7.3.3">NF4</mtext></ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E2.m2.1c">\displaystyle=q_{\text{NF4}}^{-1}(w_{\text{NF4}},\mathcal{M}_{\beta})\coloneqq\mathcal{M}_{\beta}\cdot w_{\text{NF4}}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
</div>
<div id="S2.SS1.p2" class="ltx_para">
<p id="S2.SS1.p2.1" class="ltx_p">We provide an overview of different categories of quantization techniques, and how they relate to <span id="S2.SS1.p2.1.1" class="ltx_text ltx_font_smallcaps">LoQT</span>, in Appendix <a href="#A1" title="Appendix A Quantization Methods ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>. Compared to prior approaches, <span id="S2.SS1.p2.1.2" class="ltx_text ltx_font_smallcaps">LoQT</span> retains the benefits of reduced memory usage while minimizing accuracy loss, using high-precision updates on a low-rank representation. This allows for efficient model updates without the overhead of full matrix storage and re-quantization.</p>
</div>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Adaptation of Pretrained Networks</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.1" class="ltx_p">Low-Rank Adaptation (LoRA) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite> enables fine-tuning of pre-trained models using low-rank adaptors, effectively reducing the memory footprint by only training weight adaptors for targeted layers. However, simple low-rank training using LoRA factor matrices has not been shown to work for pre-training<cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib22" title="" class="ltx_ref">22</a>]</cite>.</p>
</div>
<div id="S2.SS2.p2" class="ltx_para">
<p id="S2.SS2.p2.14" class="ltx_p">LoRA employs trainable low-rank matrices <math id="S2.SS2.p2.1.m1.1" class="ltx_Math" alttext="A" display="inline"><semantics id="S2.SS2.p2.1.m1.1a"><mi id="S2.SS2.p2.1.m1.1.1" xref="S2.SS2.p2.1.m1.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.1.m1.1b"><ci id="S2.SS2.p2.1.m1.1.1.cmml" xref="S2.SS2.p2.1.m1.1.1">𝐴</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.1.m1.1c">A</annotation></semantics></math> and <math id="S2.SS2.p2.2.m2.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S2.SS2.p2.2.m2.1a"><mi id="S2.SS2.p2.2.m2.1.1" xref="S2.SS2.p2.2.m2.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.2.m2.1b"><ci id="S2.SS2.p2.2.m2.1.1.cmml" xref="S2.SS2.p2.2.m2.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.2.m2.1c">B</annotation></semantics></math> that are used to update <math id="S2.SS2.p2.3.m3.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.SS2.p2.3.m3.1a"><mi id="S2.SS2.p2.3.m3.1.1" xref="S2.SS2.p2.3.m3.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.3.m3.1b"><ci id="S2.SS2.p2.3.m3.1.1.cmml" xref="S2.SS2.p2.3.m3.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.3.m3.1c">W</annotation></semantics></math> following <math id="S2.SS2.p2.4.m4.1" class="ltx_Math" alttext="W_{t}=W_{t-1}+AB" display="inline"><semantics id="S2.SS2.p2.4.m4.1a"><mrow id="S2.SS2.p2.4.m4.1.1" xref="S2.SS2.p2.4.m4.1.1.cmml"><msub id="S2.SS2.p2.4.m4.1.1.2" xref="S2.SS2.p2.4.m4.1.1.2.cmml"><mi id="S2.SS2.p2.4.m4.1.1.2.2" xref="S2.SS2.p2.4.m4.1.1.2.2.cmml">W</mi><mi id="S2.SS2.p2.4.m4.1.1.2.3" xref="S2.SS2.p2.4.m4.1.1.2.3.cmml">t</mi></msub><mo id="S2.SS2.p2.4.m4.1.1.1" xref="S2.SS2.p2.4.m4.1.1.1.cmml">=</mo><mrow id="S2.SS2.p2.4.m4.1.1.3" xref="S2.SS2.p2.4.m4.1.1.3.cmml"><msub id="S2.SS2.p2.4.m4.1.1.3.2" xref="S2.SS2.p2.4.m4.1.1.3.2.cmml"><mi id="S2.SS2.p2.4.m4.1.1.3.2.2" xref="S2.SS2.p2.4.m4.1.1.3.2.2.cmml">W</mi><mrow id="S2.SS2.p2.4.m4.1.1.3.2.3" xref="S2.SS2.p2.4.m4.1.1.3.2.3.cmml"><mi id="S2.SS2.p2.4.m4.1.1.3.2.3.2" xref="S2.SS2.p2.4.m4.1.1.3.2.3.2.cmml">t</mi><mo id="S2.SS2.p2.4.m4.1.1.3.2.3.1" xref="S2.SS2.p2.4.m4.1.1.3.2.3.1.cmml">−</mo><mn id="S2.SS2.p2.4.m4.1.1.3.2.3.3" xref="S2.SS2.p2.4.m4.1.1.3.2.3.3.cmml">1</mn></mrow></msub><mo id="S2.SS2.p2.4.m4.1.1.3.1" xref="S2.SS2.p2.4.m4.1.1.3.1.cmml">+</mo><mrow id="S2.SS2.p2.4.m4.1.1.3.3" xref="S2.SS2.p2.4.m4.1.1.3.3.cmml"><mi id="S2.SS2.p2.4.m4.1.1.3.3.2" xref="S2.SS2.p2.4.m4.1.1.3.3.2.cmml">A</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p2.4.m4.1.1.3.3.1" xref="S2.SS2.p2.4.m4.1.1.3.3.1.cmml">​</mo><mi id="S2.SS2.p2.4.m4.1.1.3.3.3" xref="S2.SS2.p2.4.m4.1.1.3.3.3.cmml">B</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.4.m4.1b"><apply id="S2.SS2.p2.4.m4.1.1.cmml" xref="S2.SS2.p2.4.m4.1.1"><eq id="S2.SS2.p2.4.m4.1.1.1.cmml" xref="S2.SS2.p2.4.m4.1.1.1"></eq><apply id="S2.SS2.p2.4.m4.1.1.2.cmml" xref="S2.SS2.p2.4.m4.1.1.2"><csymbol cd="ambiguous" id="S2.SS2.p2.4.m4.1.1.2.1.cmml" xref="S2.SS2.p2.4.m4.1.1.2">subscript</csymbol><ci id="S2.SS2.p2.4.m4.1.1.2.2.cmml" xref="S2.SS2.p2.4.m4.1.1.2.2">𝑊</ci><ci id="S2.SS2.p2.4.m4.1.1.2.3.cmml" xref="S2.SS2.p2.4.m4.1.1.2.3">𝑡</ci></apply><apply id="S2.SS2.p2.4.m4.1.1.3.cmml" xref="S2.SS2.p2.4.m4.1.1.3"><plus id="S2.SS2.p2.4.m4.1.1.3.1.cmml" xref="S2.SS2.p2.4.m4.1.1.3.1"></plus><apply id="S2.SS2.p2.4.m4.1.1.3.2.cmml" xref="S2.SS2.p2.4.m4.1.1.3.2"><csymbol cd="ambiguous" id="S2.SS2.p2.4.m4.1.1.3.2.1.cmml" xref="S2.SS2.p2.4.m4.1.1.3.2">subscript</csymbol><ci id="S2.SS2.p2.4.m4.1.1.3.2.2.cmml" xref="S2.SS2.p2.4.m4.1.1.3.2.2">𝑊</ci><apply id="S2.SS2.p2.4.m4.1.1.3.2.3.cmml" xref="S2.SS2.p2.4.m4.1.1.3.2.3"><minus id="S2.SS2.p2.4.m4.1.1.3.2.3.1.cmml" xref="S2.SS2.p2.4.m4.1.1.3.2.3.1"></minus><ci id="S2.SS2.p2.4.m4.1.1.3.2.3.2.cmml" xref="S2.SS2.p2.4.m4.1.1.3.2.3.2">𝑡</ci><cn type="integer" id="S2.SS2.p2.4.m4.1.1.3.2.3.3.cmml" xref="S2.SS2.p2.4.m4.1.1.3.2.3.3">1</cn></apply></apply><apply id="S2.SS2.p2.4.m4.1.1.3.3.cmml" xref="S2.SS2.p2.4.m4.1.1.3.3"><times id="S2.SS2.p2.4.m4.1.1.3.3.1.cmml" xref="S2.SS2.p2.4.m4.1.1.3.3.1"></times><ci id="S2.SS2.p2.4.m4.1.1.3.3.2.cmml" xref="S2.SS2.p2.4.m4.1.1.3.3.2">𝐴</ci><ci id="S2.SS2.p2.4.m4.1.1.3.3.3.cmml" xref="S2.SS2.p2.4.m4.1.1.3.3.3">𝐵</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.4.m4.1c">W_{t}=W_{t-1}+AB</annotation></semantics></math>, where <math id="S2.SS2.p2.5.m5.1" class="ltx_Math" alttext="W_{t-1}" display="inline"><semantics id="S2.SS2.p2.5.m5.1a"><msub id="S2.SS2.p2.5.m5.1.1" xref="S2.SS2.p2.5.m5.1.1.cmml"><mi id="S2.SS2.p2.5.m5.1.1.2" xref="S2.SS2.p2.5.m5.1.1.2.cmml">W</mi><mrow id="S2.SS2.p2.5.m5.1.1.3" xref="S2.SS2.p2.5.m5.1.1.3.cmml"><mi id="S2.SS2.p2.5.m5.1.1.3.2" xref="S2.SS2.p2.5.m5.1.1.3.2.cmml">t</mi><mo id="S2.SS2.p2.5.m5.1.1.3.1" xref="S2.SS2.p2.5.m5.1.1.3.1.cmml">−</mo><mn id="S2.SS2.p2.5.m5.1.1.3.3" xref="S2.SS2.p2.5.m5.1.1.3.3.cmml">1</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.5.m5.1b"><apply id="S2.SS2.p2.5.m5.1.1.cmml" xref="S2.SS2.p2.5.m5.1.1"><csymbol cd="ambiguous" id="S2.SS2.p2.5.m5.1.1.1.cmml" xref="S2.SS2.p2.5.m5.1.1">subscript</csymbol><ci id="S2.SS2.p2.5.m5.1.1.2.cmml" xref="S2.SS2.p2.5.m5.1.1.2">𝑊</ci><apply id="S2.SS2.p2.5.m5.1.1.3.cmml" xref="S2.SS2.p2.5.m5.1.1.3"><minus id="S2.SS2.p2.5.m5.1.1.3.1.cmml" xref="S2.SS2.p2.5.m5.1.1.3.1"></minus><ci id="S2.SS2.p2.5.m5.1.1.3.2.cmml" xref="S2.SS2.p2.5.m5.1.1.3.2">𝑡</ci><cn type="integer" id="S2.SS2.p2.5.m5.1.1.3.3.cmml" xref="S2.SS2.p2.5.m5.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.5.m5.1c">W_{t-1}</annotation></semantics></math> is frozen to enable precise adjustments within a low-rank framework. Since LoRA only trains <math id="S2.SS2.p2.6.m6.1" class="ltx_Math" alttext="A" display="inline"><semantics id="S2.SS2.p2.6.m6.1a"><mi id="S2.SS2.p2.6.m6.1.1" xref="S2.SS2.p2.6.m6.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.6.m6.1b"><ci id="S2.SS2.p2.6.m6.1.1.cmml" xref="S2.SS2.p2.6.m6.1.1">𝐴</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.6.m6.1c">A</annotation></semantics></math> and <math id="S2.SS2.p2.7.m7.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S2.SS2.p2.7.m7.1a"><mi id="S2.SS2.p2.7.m7.1.1" xref="S2.SS2.p2.7.m7.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.7.m7.1b"><ci id="S2.SS2.p2.7.m7.1.1.cmml" xref="S2.SS2.p2.7.m7.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.7.m7.1c">B</annotation></semantics></math> and keeps <math id="S2.SS2.p2.8.m8.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.SS2.p2.8.m8.1a"><mi id="S2.SS2.p2.8.m8.1.1" xref="S2.SS2.p2.8.m8.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.8.m8.1b"><ci id="S2.SS2.p2.8.m8.1.1.cmml" xref="S2.SS2.p2.8.m8.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.8.m8.1c">W</annotation></semantics></math> fixed, QLoRA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite> explore quantizing <math id="S2.SS2.p2.9.m9.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.SS2.p2.9.m9.1a"><mi id="S2.SS2.p2.9.m9.1.1" xref="S2.SS2.p2.9.m9.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.9.m9.1b"><ci id="S2.SS2.p2.9.m9.1.1.cmml" xref="S2.SS2.p2.9.m9.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.9.m9.1c">W</annotation></semantics></math>. They fine-tune a quantized model <math id="S2.SS2.p2.10.m10.1" class="ltx_Math" alttext="q(W)=W_{q}" display="inline"><semantics id="S2.SS2.p2.10.m10.1a"><mrow id="S2.SS2.p2.10.m10.1.2" xref="S2.SS2.p2.10.m10.1.2.cmml"><mrow id="S2.SS2.p2.10.m10.1.2.2" xref="S2.SS2.p2.10.m10.1.2.2.cmml"><mi id="S2.SS2.p2.10.m10.1.2.2.2" xref="S2.SS2.p2.10.m10.1.2.2.2.cmml">q</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p2.10.m10.1.2.2.1" xref="S2.SS2.p2.10.m10.1.2.2.1.cmml">​</mo><mrow id="S2.SS2.p2.10.m10.1.2.2.3.2" xref="S2.SS2.p2.10.m10.1.2.2.cmml"><mo stretchy="false" id="S2.SS2.p2.10.m10.1.2.2.3.2.1" xref="S2.SS2.p2.10.m10.1.2.2.cmml">(</mo><mi id="S2.SS2.p2.10.m10.1.1" xref="S2.SS2.p2.10.m10.1.1.cmml">W</mi><mo stretchy="false" id="S2.SS2.p2.10.m10.1.2.2.3.2.2" xref="S2.SS2.p2.10.m10.1.2.2.cmml">)</mo></mrow></mrow><mo id="S2.SS2.p2.10.m10.1.2.1" xref="S2.SS2.p2.10.m10.1.2.1.cmml">=</mo><msub id="S2.SS2.p2.10.m10.1.2.3" xref="S2.SS2.p2.10.m10.1.2.3.cmml"><mi id="S2.SS2.p2.10.m10.1.2.3.2" xref="S2.SS2.p2.10.m10.1.2.3.2.cmml">W</mi><mi id="S2.SS2.p2.10.m10.1.2.3.3" xref="S2.SS2.p2.10.m10.1.2.3.3.cmml">q</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.10.m10.1b"><apply id="S2.SS2.p2.10.m10.1.2.cmml" xref="S2.SS2.p2.10.m10.1.2"><eq id="S2.SS2.p2.10.m10.1.2.1.cmml" xref="S2.SS2.p2.10.m10.1.2.1"></eq><apply id="S2.SS2.p2.10.m10.1.2.2.cmml" xref="S2.SS2.p2.10.m10.1.2.2"><times id="S2.SS2.p2.10.m10.1.2.2.1.cmml" xref="S2.SS2.p2.10.m10.1.2.2.1"></times><ci id="S2.SS2.p2.10.m10.1.2.2.2.cmml" xref="S2.SS2.p2.10.m10.1.2.2.2">𝑞</ci><ci id="S2.SS2.p2.10.m10.1.1.cmml" xref="S2.SS2.p2.10.m10.1.1">𝑊</ci></apply><apply id="S2.SS2.p2.10.m10.1.2.3.cmml" xref="S2.SS2.p2.10.m10.1.2.3"><csymbol cd="ambiguous" id="S2.SS2.p2.10.m10.1.2.3.1.cmml" xref="S2.SS2.p2.10.m10.1.2.3">subscript</csymbol><ci id="S2.SS2.p2.10.m10.1.2.3.2.cmml" xref="S2.SS2.p2.10.m10.1.2.3.2">𝑊</ci><ci id="S2.SS2.p2.10.m10.1.2.3.3.cmml" xref="S2.SS2.p2.10.m10.1.2.3.3">𝑞</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.10.m10.1c">q(W)=W_{q}</annotation></semantics></math> with 4-bit precision using randomly initialized 16-bit precision factors <math id="S2.SS2.p2.11.m11.1" class="ltx_Math" alttext="A" display="inline"><semantics id="S2.SS2.p2.11.m11.1a"><mi id="S2.SS2.p2.11.m11.1.1" xref="S2.SS2.p2.11.m11.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.11.m11.1b"><ci id="S2.SS2.p2.11.m11.1.1.cmml" xref="S2.SS2.p2.11.m11.1.1">𝐴</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.11.m11.1c">A</annotation></semantics></math> and <math id="S2.SS2.p2.12.m12.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S2.SS2.p2.12.m12.1a"><mi id="S2.SS2.p2.12.m12.1.1" xref="S2.SS2.p2.12.m12.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.12.m12.1b"><ci id="S2.SS2.p2.12.m12.1.1.cmml" xref="S2.SS2.p2.12.m12.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.12.m12.1c">B</annotation></semantics></math>. To address quantization errors <math id="S2.SS2.p2.13.m13.1" class="ltx_Math" alttext="\mathcal{E}=|W_{q}-W|" display="inline"><semantics id="S2.SS2.p2.13.m13.1a"><mrow id="S2.SS2.p2.13.m13.1.1" xref="S2.SS2.p2.13.m13.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.SS2.p2.13.m13.1.1.3" xref="S2.SS2.p2.13.m13.1.1.3.cmml">ℰ</mi><mo id="S2.SS2.p2.13.m13.1.1.2" xref="S2.SS2.p2.13.m13.1.1.2.cmml">=</mo><mrow id="S2.SS2.p2.13.m13.1.1.1.1" xref="S2.SS2.p2.13.m13.1.1.1.2.cmml"><mo stretchy="false" id="S2.SS2.p2.13.m13.1.1.1.1.2" xref="S2.SS2.p2.13.m13.1.1.1.2.1.cmml">|</mo><mrow id="S2.SS2.p2.13.m13.1.1.1.1.1" xref="S2.SS2.p2.13.m13.1.1.1.1.1.cmml"><msub id="S2.SS2.p2.13.m13.1.1.1.1.1.2" xref="S2.SS2.p2.13.m13.1.1.1.1.1.2.cmml"><mi id="S2.SS2.p2.13.m13.1.1.1.1.1.2.2" xref="S2.SS2.p2.13.m13.1.1.1.1.1.2.2.cmml">W</mi><mi id="S2.SS2.p2.13.m13.1.1.1.1.1.2.3" xref="S2.SS2.p2.13.m13.1.1.1.1.1.2.3.cmml">q</mi></msub><mo id="S2.SS2.p2.13.m13.1.1.1.1.1.1" xref="S2.SS2.p2.13.m13.1.1.1.1.1.1.cmml">−</mo><mi id="S2.SS2.p2.13.m13.1.1.1.1.1.3" xref="S2.SS2.p2.13.m13.1.1.1.1.1.3.cmml">W</mi></mrow><mo stretchy="false" id="S2.SS2.p2.13.m13.1.1.1.1.3" xref="S2.SS2.p2.13.m13.1.1.1.2.1.cmml">|</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.13.m13.1b"><apply id="S2.SS2.p2.13.m13.1.1.cmml" xref="S2.SS2.p2.13.m13.1.1"><eq id="S2.SS2.p2.13.m13.1.1.2.cmml" xref="S2.SS2.p2.13.m13.1.1.2"></eq><ci id="S2.SS2.p2.13.m13.1.1.3.cmml" xref="S2.SS2.p2.13.m13.1.1.3">ℰ</ci><apply id="S2.SS2.p2.13.m13.1.1.1.2.cmml" xref="S2.SS2.p2.13.m13.1.1.1.1"><abs id="S2.SS2.p2.13.m13.1.1.1.2.1.cmml" xref="S2.SS2.p2.13.m13.1.1.1.1.2"></abs><apply id="S2.SS2.p2.13.m13.1.1.1.1.1.cmml" xref="S2.SS2.p2.13.m13.1.1.1.1.1"><minus id="S2.SS2.p2.13.m13.1.1.1.1.1.1.cmml" xref="S2.SS2.p2.13.m13.1.1.1.1.1.1"></minus><apply id="S2.SS2.p2.13.m13.1.1.1.1.1.2.cmml" xref="S2.SS2.p2.13.m13.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.SS2.p2.13.m13.1.1.1.1.1.2.1.cmml" xref="S2.SS2.p2.13.m13.1.1.1.1.1.2">subscript</csymbol><ci id="S2.SS2.p2.13.m13.1.1.1.1.1.2.2.cmml" xref="S2.SS2.p2.13.m13.1.1.1.1.1.2.2">𝑊</ci><ci id="S2.SS2.p2.13.m13.1.1.1.1.1.2.3.cmml" xref="S2.SS2.p2.13.m13.1.1.1.1.1.2.3">𝑞</ci></apply><ci id="S2.SS2.p2.13.m13.1.1.1.1.1.3.cmml" xref="S2.SS2.p2.13.m13.1.1.1.1.1.3">𝑊</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.13.m13.1c">\mathcal{E}=|W_{q}-W|</annotation></semantics></math>, low-rank factors of the quantization error <math id="S2.SS2.p2.14.m14.1" class="ltx_Math" alttext="\mathcal{E}" display="inline"><semantics id="S2.SS2.p2.14.m14.1a"><mi class="ltx_font_mathcaligraphic" id="S2.SS2.p2.14.m14.1.1" xref="S2.SS2.p2.14.m14.1.1.cmml">ℰ</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.14.m14.1b"><ci id="S2.SS2.p2.14.m14.1.1.cmml" xref="S2.SS2.p2.14.m14.1.1">ℰ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.14.m14.1c">\mathcal{E}</annotation></semantics></math> have been used <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib21" title="" class="ltx_ref">21</a>]</cite>.</p>
</div>
<div id="S2.SS2.p3" class="ltx_para">
<p id="S2.SS2.p3.8" class="ltx_p"><span id="S2.SS2.p3.8.1" class="ltx_text ltx_font_smallcaps">LoQT</span> extends LoRA to both pretraining and fine-tuning. Unlike traditional LoRA, <span id="S2.SS2.p3.8.2" class="ltx_text ltx_font_smallcaps">LoQT</span> uses <math id="S2.SS2.p3.1.m1.1" class="ltx_Math" alttext="A" display="inline"><semantics id="S2.SS2.p3.1.m1.1a"><mi id="S2.SS2.p3.1.m1.1.1" xref="S2.SS2.p3.1.m1.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.1.m1.1b"><ci id="S2.SS2.p3.1.m1.1.1.cmml" xref="S2.SS2.p3.1.m1.1.1">𝐴</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.1.m1.1c">A</annotation></semantics></math> and <math id="S2.SS2.p3.2.m2.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S2.SS2.p3.2.m2.1a"><mi id="S2.SS2.p3.2.m2.1.1" xref="S2.SS2.p3.2.m2.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.2.m2.1b"><ci id="S2.SS2.p3.2.m2.1.1.cmml" xref="S2.SS2.p3.2.m2.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.2.m2.1c">B</annotation></semantics></math> to refine <math id="S2.SS2.p3.3.m3.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.SS2.p3.3.m3.1a"><mi id="S2.SS2.p3.3.m3.1.1" xref="S2.SS2.p3.3.m3.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.3.m3.1b"><ci id="S2.SS2.p3.3.m3.1.1.cmml" xref="S2.SS2.p3.3.m3.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.3.m3.1c">W</annotation></semantics></math> throughout training, with <math id="S2.SS2.p3.4.m4.1" class="ltx_Math" alttext="A" display="inline"><semantics id="S2.SS2.p3.4.m4.1a"><mi id="S2.SS2.p3.4.m4.1.1" xref="S2.SS2.p3.4.m4.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.4.m4.1b"><ci id="S2.SS2.p3.4.m4.1.1.cmml" xref="S2.SS2.p3.4.m4.1.1">𝐴</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.4.m4.1c">A</annotation></semantics></math> initialized from <math id="S2.SS2.p3.5.m5.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.SS2.p3.5.m5.1a"><mi id="S2.SS2.p3.5.m5.1.1" xref="S2.SS2.p3.5.m5.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.5.m5.1b"><ci id="S2.SS2.p3.5.m5.1.1.cmml" xref="S2.SS2.p3.5.m5.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.5.m5.1c">W</annotation></semantics></math>’s gradient projection and <math id="S2.SS2.p3.6.m6.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S2.SS2.p3.6.m6.1a"><mi id="S2.SS2.p3.6.m6.1.1" xref="S2.SS2.p3.6.m6.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.6.m6.1b"><ci id="S2.SS2.p3.6.m6.1.1.cmml" xref="S2.SS2.p3.6.m6.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.6.m6.1c">B</annotation></semantics></math> trained along this gradient path. <span id="S2.SS2.p3.8.3" class="ltx_text ltx_font_smallcaps">LoQT</span> also incorporates quantization and targeted optimization iterations similar in spirit to LoftQ <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib21" title="" class="ltx_ref">21</a>]</cite>, correcting for quantization errors in <math id="S2.SS2.p3.7.m7.1" class="ltx_Math" alttext="W_{q}" display="inline"><semantics id="S2.SS2.p3.7.m7.1a"><msub id="S2.SS2.p3.7.m7.1.1" xref="S2.SS2.p3.7.m7.1.1.cmml"><mi id="S2.SS2.p3.7.m7.1.1.2" xref="S2.SS2.p3.7.m7.1.1.2.cmml">W</mi><mi id="S2.SS2.p3.7.m7.1.1.3" xref="S2.SS2.p3.7.m7.1.1.3.cmml">q</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.7.m7.1b"><apply id="S2.SS2.p3.7.m7.1.1.cmml" xref="S2.SS2.p3.7.m7.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.7.m7.1.1.1.cmml" xref="S2.SS2.p3.7.m7.1.1">subscript</csymbol><ci id="S2.SS2.p3.7.m7.1.1.2.cmml" xref="S2.SS2.p3.7.m7.1.1.2">𝑊</ci><ci id="S2.SS2.p3.7.m7.1.1.3.cmml" xref="S2.SS2.p3.7.m7.1.1.3">𝑞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.7.m7.1c">W_{q}</annotation></semantics></math>, thus better aligning it with the original non-quantized <math id="S2.SS2.p3.8.m8.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.SS2.p3.8.m8.1a"><mi id="S2.SS2.p3.8.m8.1.1" xref="S2.SS2.p3.8.m8.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.8.m8.1b"><ci id="S2.SS2.p3.8.m8.1.1.cmml" xref="S2.SS2.p3.8.m8.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.8.m8.1c">W</annotation></semantics></math>.</p>
</div>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3 </span>Memory Efficient Optimization</h3>

<section id="S2.SS3.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Optimizer memory consumption</h4>

<div id="S2.SS3.SSS0.Px1.p1" class="ltx_para">
<p id="S2.SS3.SSS0.Px1.p1.1" class="ltx_p">A significant portion of the memory needed to train neural networks is typically consumed by optimizer states. Notably, Adam <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib18" title="" class="ltx_ref">18</a>]</cite>, one of the most widely used optimizers, uses double the amount of memory as the gradient matrix to maintain first and second-order gradient statistics.
Efforts to reduce this overhead have led to the development of adaptive optimization algorithms like Adafactor <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib32" title="" class="ltx_ref">32</a>]</cite>, which achieves sub-linear memory costs by factorizing the second-order statistics into a row-column outer product. GaLore <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite> expands on this concept by using low-rank factorization and projecting low-rank gradients up to a full-rank when updating model weights.</p>
</div>
</section>
<section id="S2.SS3.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Periodic updating of weight matrices</h4>

<div id="S2.SS3.SSS0.Px2.p1" class="ltx_para">
<p id="S2.SS3.SSS0.Px2.p1.1" class="ltx_p">ReLoRA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib22" title="" class="ltx_ref">22</a>]</cite> combines low-rank updates with initial full-rank training. They find that doing one-third of the training in full-rank, and the subsequent two-thirds in low-rank (see §<a href="#S2.SS2" title="2.2 Adaptation of Pretrained Networks ‣ 2 Related Work and Background ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.2</span></a>) results in comparable performance to standard training methods.</p>
</div>
</section>
<section id="S2.SS3.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Low-rank gradients</h4>

<div id="S2.SS3.SSS0.Px3.p1" class="ltx_para">
<p id="S2.SS3.SSS0.Px3.p1.3" class="ltx_p">GaLore <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite>, focuses on the structure of the gradients, projecting them into a low-rank space using factors <math id="S2.SS3.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S2.SS3.SSS0.Px3.p1.1.m1.1a"><mi id="S2.SS3.SSS0.Px3.p1.1.m1.1.1" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.SSS0.Px3.p1.1.m1.1b"><ci id="S2.SS3.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S2.SS3.SSS0.Px3.p1.1.m1.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.SSS0.Px3.p1.1.m1.1c">P</annotation></semantics></math> and <math id="S2.SS3.SSS0.Px3.p1.2.m2.1" class="ltx_Math" alttext="Q" display="inline"><semantics id="S2.SS3.SSS0.Px3.p1.2.m2.1a"><mi id="S2.SS3.SSS0.Px3.p1.2.m2.1.1" xref="S2.SS3.SSS0.Px3.p1.2.m2.1.1.cmml">Q</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.SSS0.Px3.p1.2.m2.1b"><ci id="S2.SS3.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S2.SS3.SSS0.Px3.p1.2.m2.1.1">𝑄</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.SSS0.Px3.p1.2.m2.1c">Q</annotation></semantics></math>, which are derived from a truncated singular value decomposition (SVD) of the weight matrix gradient, <math id="S2.SS3.SSS0.Px3.p1.3.m3.1" class="ltx_Math" alttext="G_{W}\approx P_{r}\Sigma_{r}Q_{r}" display="inline"><semantics id="S2.SS3.SSS0.Px3.p1.3.m3.1a"><mrow id="S2.SS3.SSS0.Px3.p1.3.m3.1.1" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.cmml"><msub id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.2" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.2.cmml"><mi id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.2.2" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.2.2.cmml">G</mi><mi id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.2.3" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.2.3.cmml">W</mi></msub><mo id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.1" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.1.cmml">≈</mo><mrow id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.cmml"><msub id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.2" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.2.cmml"><mi id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.2.2" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.2.2.cmml">P</mi><mi id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.2.3" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.2.3.cmml">r</mi></msub><mo lspace="0em" rspace="0em" id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.1" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.1.cmml">​</mo><msub id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.3" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.3.cmml"><mi mathvariant="normal" id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.3.2" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.3.2.cmml">Σ</mi><mi id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.3.3" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.3.3.cmml">r</mi></msub><mo lspace="0em" rspace="0em" id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.1a" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.1.cmml">​</mo><msub id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.4" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.4.cmml"><mi id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.4.2" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.4.2.cmml">Q</mi><mi id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.4.3" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.4.3.cmml">r</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.SSS0.Px3.p1.3.m3.1b"><apply id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.cmml" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1"><approx id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.1.cmml" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.1"></approx><apply id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.2.cmml" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.2"><csymbol cd="ambiguous" id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.2.1.cmml" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.2">subscript</csymbol><ci id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.2.2.cmml" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.2.2">𝐺</ci><ci id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.2.3.cmml" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.2.3">𝑊</ci></apply><apply id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.cmml" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3"><times id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.1.cmml" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.1"></times><apply id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.2.cmml" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.2"><csymbol cd="ambiguous" id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.2.1.cmml" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.2">subscript</csymbol><ci id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.2.2.cmml" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.2.2">𝑃</ci><ci id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.2.3.cmml" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.2.3">𝑟</ci></apply><apply id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.3.cmml" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.3"><csymbol cd="ambiguous" id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.3.1.cmml" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.3">subscript</csymbol><ci id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.3.2.cmml" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.3.2">Σ</ci><ci id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.3.3.cmml" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.3.3">𝑟</ci></apply><apply id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.4.cmml" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.4"><csymbol cd="ambiguous" id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.4.1.cmml" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.4">subscript</csymbol><ci id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.4.2.cmml" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.4.2">𝑄</ci><ci id="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.4.3.cmml" xref="S2.SS3.SSS0.Px3.p1.3.m3.1.1.3.4.3">𝑟</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.SSS0.Px3.p1.3.m3.1c">G_{W}\approx P_{r}\Sigma_{r}Q_{r}</annotation></semantics></math>. This reduces memory costs associated with storing the optimizer states and aligns with findings from recent studies which suggest that learning primarily occurs within a low-dimensional subspace at a given time <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib19" title="" class="ltx_ref">19</a>, <a href="#bib.bib11" title="" class="ltx_ref">11</a>]</cite>. This can be further combined with applying per-layer gradient updates, reducing the memory needed for storing the gradients for the full model at once <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib25" title="" class="ltx_ref">25</a>]</cite>.</p>
</div>
<div id="S2.SS3.SSS0.Px3.p2" class="ltx_para">
<p id="S2.SS3.SSS0.Px3.p2.1" class="ltx_p"><span id="S2.SS3.SSS0.Px3.p2.1.1" class="ltx_text ltx_font_smallcaps">LoQT</span> builds on GaLore’s gradient projection (§<a href="#S3.SS1" title="3.1 Background: GaLore ‣ 3 Efficient Pretraining With LoQT ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a>) to initialize LoRA factors while updating the full matrix following a schedule inspired by ReLora, while only training one low-rank matrix per layer. We achieve comparable quality to GaLore and better performance than ReLoRA while reducing tunable parameters and memory usage compared to both approaches.</p>
</div>
</section>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Efficient Pretraining With <span id="S3.1.1" class="ltx_text ltx_font_smallcaps">LoQT</span>
</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.14" class="ltx_p"><span id="S3.p1.14.1" class="ltx_text ltx_font_smallcaps">LoQT</span> works by initializing and training low-rank adapters obtained by taking the SVD of a given layer’s gradients. Let <math id="S3.p1.1.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S3.p1.1.m1.1a"><mi id="S3.p1.1.m1.1.1" xref="S3.p1.1.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.p1.1.m1.1b"><ci id="S3.p1.1.m1.1.1.cmml" xref="S3.p1.1.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.1.m1.1c">W</annotation></semantics></math> indicate the full weights matrix of a given layer, <math id="S3.p1.2.m2.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S3.p1.2.m2.1a"><mi id="S3.p1.2.m2.1.1" xref="S3.p1.2.m2.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S3.p1.2.m2.1b"><ci id="S3.p1.2.m2.1.1.cmml" xref="S3.p1.2.m2.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.2.m2.1c">P</annotation></semantics></math> be the left factor constructed from the SVD decomposition of the gradients matrix: <math id="S3.p1.3.m3.1" class="ltx_Math" alttext="\nabla W=U\Sigma V^{\top}" display="inline"><semantics id="S3.p1.3.m3.1a"><mrow id="S3.p1.3.m3.1.1" xref="S3.p1.3.m3.1.1.cmml"><mrow id="S3.p1.3.m3.1.1.2" xref="S3.p1.3.m3.1.1.2.cmml"><mo rspace="0.167em" id="S3.p1.3.m3.1.1.2.1" xref="S3.p1.3.m3.1.1.2.1.cmml">∇</mo><mi id="S3.p1.3.m3.1.1.2.2" xref="S3.p1.3.m3.1.1.2.2.cmml">W</mi></mrow><mo id="S3.p1.3.m3.1.1.1" xref="S3.p1.3.m3.1.1.1.cmml">=</mo><mrow id="S3.p1.3.m3.1.1.3" xref="S3.p1.3.m3.1.1.3.cmml"><mi id="S3.p1.3.m3.1.1.3.2" xref="S3.p1.3.m3.1.1.3.2.cmml">U</mi><mo lspace="0em" rspace="0em" id="S3.p1.3.m3.1.1.3.1" xref="S3.p1.3.m3.1.1.3.1.cmml">​</mo><mi mathvariant="normal" id="S3.p1.3.m3.1.1.3.3" xref="S3.p1.3.m3.1.1.3.3.cmml">Σ</mi><mo lspace="0em" rspace="0em" id="S3.p1.3.m3.1.1.3.1a" xref="S3.p1.3.m3.1.1.3.1.cmml">​</mo><msup id="S3.p1.3.m3.1.1.3.4" xref="S3.p1.3.m3.1.1.3.4.cmml"><mi id="S3.p1.3.m3.1.1.3.4.2" xref="S3.p1.3.m3.1.1.3.4.2.cmml">V</mi><mo id="S3.p1.3.m3.1.1.3.4.3" xref="S3.p1.3.m3.1.1.3.4.3.cmml">⊤</mo></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.3.m3.1b"><apply id="S3.p1.3.m3.1.1.cmml" xref="S3.p1.3.m3.1.1"><eq id="S3.p1.3.m3.1.1.1.cmml" xref="S3.p1.3.m3.1.1.1"></eq><apply id="S3.p1.3.m3.1.1.2.cmml" xref="S3.p1.3.m3.1.1.2"><ci id="S3.p1.3.m3.1.1.2.1.cmml" xref="S3.p1.3.m3.1.1.2.1">∇</ci><ci id="S3.p1.3.m3.1.1.2.2.cmml" xref="S3.p1.3.m3.1.1.2.2">𝑊</ci></apply><apply id="S3.p1.3.m3.1.1.3.cmml" xref="S3.p1.3.m3.1.1.3"><times id="S3.p1.3.m3.1.1.3.1.cmml" xref="S3.p1.3.m3.1.1.3.1"></times><ci id="S3.p1.3.m3.1.1.3.2.cmml" xref="S3.p1.3.m3.1.1.3.2">𝑈</ci><ci id="S3.p1.3.m3.1.1.3.3.cmml" xref="S3.p1.3.m3.1.1.3.3">Σ</ci><apply id="S3.p1.3.m3.1.1.3.4.cmml" xref="S3.p1.3.m3.1.1.3.4"><csymbol cd="ambiguous" id="S3.p1.3.m3.1.1.3.4.1.cmml" xref="S3.p1.3.m3.1.1.3.4">superscript</csymbol><ci id="S3.p1.3.m3.1.1.3.4.2.cmml" xref="S3.p1.3.m3.1.1.3.4.2">𝑉</ci><csymbol cd="latexml" id="S3.p1.3.m3.1.1.3.4.3.cmml" xref="S3.p1.3.m3.1.1.3.4.3">top</csymbol></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.3.m3.1c">\nabla W=U\Sigma V^{\top}</annotation></semantics></math>; i.e. <math id="S3.p1.4.m4.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S3.p1.4.m4.1a"><mi id="S3.p1.4.m4.1.1" xref="S3.p1.4.m4.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S3.p1.4.m4.1b"><ci id="S3.p1.4.m4.1.1.cmml" xref="S3.p1.4.m4.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.4.m4.1c">P</annotation></semantics></math> consists of the first <math id="S3.p1.5.m5.1" class="ltx_Math" alttext="r" display="inline"><semantics id="S3.p1.5.m5.1a"><mi id="S3.p1.5.m5.1.1" xref="S3.p1.5.m5.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="S3.p1.5.m5.1b"><ci id="S3.p1.5.m5.1.1.cmml" xref="S3.p1.5.m5.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.5.m5.1c">r</annotation></semantics></math> columns of <math id="S3.p1.6.m6.1" class="ltx_Math" alttext="U" display="inline"><semantics id="S3.p1.6.m6.1a"><mi id="S3.p1.6.m6.1.1" xref="S3.p1.6.m6.1.1.cmml">U</mi><annotation-xml encoding="MathML-Content" id="S3.p1.6.m6.1b"><ci id="S3.p1.6.m6.1.1.cmml" xref="S3.p1.6.m6.1.1">𝑈</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.6.m6.1c">U</annotation></semantics></math> corresponding to the singular vectors with the <math id="S3.p1.7.m7.1" class="ltx_Math" alttext="r" display="inline"><semantics id="S3.p1.7.m7.1a"><mi id="S3.p1.7.m7.1.1" xref="S3.p1.7.m7.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="S3.p1.7.m7.1b"><ci id="S3.p1.7.m7.1.1.cmml" xref="S3.p1.7.m7.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.7.m7.1c">r</annotation></semantics></math> largest singular values of <math id="S3.p1.8.m8.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S3.p1.8.m8.1a"><mi id="S3.p1.8.m8.1.1" xref="S3.p1.8.m8.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.p1.8.m8.1b"><ci id="S3.p1.8.m8.1.1.cmml" xref="S3.p1.8.m8.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.8.m8.1c">W</annotation></semantics></math>.
The update rule for an interval <math id="S3.p1.9.m9.2" class="ltx_Math" alttext="[T_{i-1},T_{i}]" display="inline"><semantics id="S3.p1.9.m9.2a"><mrow id="S3.p1.9.m9.2.2.2" xref="S3.p1.9.m9.2.2.3.cmml"><mo stretchy="false" id="S3.p1.9.m9.2.2.2.3" xref="S3.p1.9.m9.2.2.3.cmml">[</mo><msub id="S3.p1.9.m9.1.1.1.1" xref="S3.p1.9.m9.1.1.1.1.cmml"><mi id="S3.p1.9.m9.1.1.1.1.2" xref="S3.p1.9.m9.1.1.1.1.2.cmml">T</mi><mrow id="S3.p1.9.m9.1.1.1.1.3" xref="S3.p1.9.m9.1.1.1.1.3.cmml"><mi id="S3.p1.9.m9.1.1.1.1.3.2" xref="S3.p1.9.m9.1.1.1.1.3.2.cmml">i</mi><mo id="S3.p1.9.m9.1.1.1.1.3.1" xref="S3.p1.9.m9.1.1.1.1.3.1.cmml">−</mo><mn id="S3.p1.9.m9.1.1.1.1.3.3" xref="S3.p1.9.m9.1.1.1.1.3.3.cmml">1</mn></mrow></msub><mo id="S3.p1.9.m9.2.2.2.4" xref="S3.p1.9.m9.2.2.3.cmml">,</mo><msub id="S3.p1.9.m9.2.2.2.2" xref="S3.p1.9.m9.2.2.2.2.cmml"><mi id="S3.p1.9.m9.2.2.2.2.2" xref="S3.p1.9.m9.2.2.2.2.2.cmml">T</mi><mi id="S3.p1.9.m9.2.2.2.2.3" xref="S3.p1.9.m9.2.2.2.2.3.cmml">i</mi></msub><mo stretchy="false" id="S3.p1.9.m9.2.2.2.5" xref="S3.p1.9.m9.2.2.3.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.9.m9.2b"><interval closure="closed" id="S3.p1.9.m9.2.2.3.cmml" xref="S3.p1.9.m9.2.2.2"><apply id="S3.p1.9.m9.1.1.1.1.cmml" xref="S3.p1.9.m9.1.1.1.1"><csymbol cd="ambiguous" id="S3.p1.9.m9.1.1.1.1.1.cmml" xref="S3.p1.9.m9.1.1.1.1">subscript</csymbol><ci id="S3.p1.9.m9.1.1.1.1.2.cmml" xref="S3.p1.9.m9.1.1.1.1.2">𝑇</ci><apply id="S3.p1.9.m9.1.1.1.1.3.cmml" xref="S3.p1.9.m9.1.1.1.1.3"><minus id="S3.p1.9.m9.1.1.1.1.3.1.cmml" xref="S3.p1.9.m9.1.1.1.1.3.1"></minus><ci id="S3.p1.9.m9.1.1.1.1.3.2.cmml" xref="S3.p1.9.m9.1.1.1.1.3.2">𝑖</ci><cn type="integer" id="S3.p1.9.m9.1.1.1.1.3.3.cmml" xref="S3.p1.9.m9.1.1.1.1.3.3">1</cn></apply></apply><apply id="S3.p1.9.m9.2.2.2.2.cmml" xref="S3.p1.9.m9.2.2.2.2"><csymbol cd="ambiguous" id="S3.p1.9.m9.2.2.2.2.1.cmml" xref="S3.p1.9.m9.2.2.2.2">subscript</csymbol><ci id="S3.p1.9.m9.2.2.2.2.2.cmml" xref="S3.p1.9.m9.2.2.2.2.2">𝑇</ci><ci id="S3.p1.9.m9.2.2.2.2.3.cmml" xref="S3.p1.9.m9.2.2.2.2.3">𝑖</ci></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.9.m9.2c">[T_{i-1},T_{i}]</annotation></semantics></math> is then given by <math id="S3.p1.10.m10.1" class="ltx_Math" alttext="W_{T_{i}}=W_{T_{i-1}}+PB" display="inline"><semantics id="S3.p1.10.m10.1a"><mrow id="S3.p1.10.m10.1.1" xref="S3.p1.10.m10.1.1.cmml"><msub id="S3.p1.10.m10.1.1.2" xref="S3.p1.10.m10.1.1.2.cmml"><mi id="S3.p1.10.m10.1.1.2.2" xref="S3.p1.10.m10.1.1.2.2.cmml">W</mi><msub id="S3.p1.10.m10.1.1.2.3" xref="S3.p1.10.m10.1.1.2.3.cmml"><mi id="S3.p1.10.m10.1.1.2.3.2" xref="S3.p1.10.m10.1.1.2.3.2.cmml">T</mi><mi id="S3.p1.10.m10.1.1.2.3.3" xref="S3.p1.10.m10.1.1.2.3.3.cmml">i</mi></msub></msub><mo id="S3.p1.10.m10.1.1.1" xref="S3.p1.10.m10.1.1.1.cmml">=</mo><mrow id="S3.p1.10.m10.1.1.3" xref="S3.p1.10.m10.1.1.3.cmml"><msub id="S3.p1.10.m10.1.1.3.2" xref="S3.p1.10.m10.1.1.3.2.cmml"><mi id="S3.p1.10.m10.1.1.3.2.2" xref="S3.p1.10.m10.1.1.3.2.2.cmml">W</mi><msub id="S3.p1.10.m10.1.1.3.2.3" xref="S3.p1.10.m10.1.1.3.2.3.cmml"><mi id="S3.p1.10.m10.1.1.3.2.3.2" xref="S3.p1.10.m10.1.1.3.2.3.2.cmml">T</mi><mrow id="S3.p1.10.m10.1.1.3.2.3.3" xref="S3.p1.10.m10.1.1.3.2.3.3.cmml"><mi id="S3.p1.10.m10.1.1.3.2.3.3.2" xref="S3.p1.10.m10.1.1.3.2.3.3.2.cmml">i</mi><mo id="S3.p1.10.m10.1.1.3.2.3.3.1" xref="S3.p1.10.m10.1.1.3.2.3.3.1.cmml">−</mo><mn id="S3.p1.10.m10.1.1.3.2.3.3.3" xref="S3.p1.10.m10.1.1.3.2.3.3.3.cmml">1</mn></mrow></msub></msub><mo id="S3.p1.10.m10.1.1.3.1" xref="S3.p1.10.m10.1.1.3.1.cmml">+</mo><mrow id="S3.p1.10.m10.1.1.3.3" xref="S3.p1.10.m10.1.1.3.3.cmml"><mi id="S3.p1.10.m10.1.1.3.3.2" xref="S3.p1.10.m10.1.1.3.3.2.cmml">P</mi><mo lspace="0em" rspace="0em" id="S3.p1.10.m10.1.1.3.3.1" xref="S3.p1.10.m10.1.1.3.3.1.cmml">​</mo><mi id="S3.p1.10.m10.1.1.3.3.3" xref="S3.p1.10.m10.1.1.3.3.3.cmml">B</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.10.m10.1b"><apply id="S3.p1.10.m10.1.1.cmml" xref="S3.p1.10.m10.1.1"><eq id="S3.p1.10.m10.1.1.1.cmml" xref="S3.p1.10.m10.1.1.1"></eq><apply id="S3.p1.10.m10.1.1.2.cmml" xref="S3.p1.10.m10.1.1.2"><csymbol cd="ambiguous" id="S3.p1.10.m10.1.1.2.1.cmml" xref="S3.p1.10.m10.1.1.2">subscript</csymbol><ci id="S3.p1.10.m10.1.1.2.2.cmml" xref="S3.p1.10.m10.1.1.2.2">𝑊</ci><apply id="S3.p1.10.m10.1.1.2.3.cmml" xref="S3.p1.10.m10.1.1.2.3"><csymbol cd="ambiguous" id="S3.p1.10.m10.1.1.2.3.1.cmml" xref="S3.p1.10.m10.1.1.2.3">subscript</csymbol><ci id="S3.p1.10.m10.1.1.2.3.2.cmml" xref="S3.p1.10.m10.1.1.2.3.2">𝑇</ci><ci id="S3.p1.10.m10.1.1.2.3.3.cmml" xref="S3.p1.10.m10.1.1.2.3.3">𝑖</ci></apply></apply><apply id="S3.p1.10.m10.1.1.3.cmml" xref="S3.p1.10.m10.1.1.3"><plus id="S3.p1.10.m10.1.1.3.1.cmml" xref="S3.p1.10.m10.1.1.3.1"></plus><apply id="S3.p1.10.m10.1.1.3.2.cmml" xref="S3.p1.10.m10.1.1.3.2"><csymbol cd="ambiguous" id="S3.p1.10.m10.1.1.3.2.1.cmml" xref="S3.p1.10.m10.1.1.3.2">subscript</csymbol><ci id="S3.p1.10.m10.1.1.3.2.2.cmml" xref="S3.p1.10.m10.1.1.3.2.2">𝑊</ci><apply id="S3.p1.10.m10.1.1.3.2.3.cmml" xref="S3.p1.10.m10.1.1.3.2.3"><csymbol cd="ambiguous" id="S3.p1.10.m10.1.1.3.2.3.1.cmml" xref="S3.p1.10.m10.1.1.3.2.3">subscript</csymbol><ci id="S3.p1.10.m10.1.1.3.2.3.2.cmml" xref="S3.p1.10.m10.1.1.3.2.3.2">𝑇</ci><apply id="S3.p1.10.m10.1.1.3.2.3.3.cmml" xref="S3.p1.10.m10.1.1.3.2.3.3"><minus id="S3.p1.10.m10.1.1.3.2.3.3.1.cmml" xref="S3.p1.10.m10.1.1.3.2.3.3.1"></minus><ci id="S3.p1.10.m10.1.1.3.2.3.3.2.cmml" xref="S3.p1.10.m10.1.1.3.2.3.3.2">𝑖</ci><cn type="integer" id="S3.p1.10.m10.1.1.3.2.3.3.3.cmml" xref="S3.p1.10.m10.1.1.3.2.3.3.3">1</cn></apply></apply></apply><apply id="S3.p1.10.m10.1.1.3.3.cmml" xref="S3.p1.10.m10.1.1.3.3"><times id="S3.p1.10.m10.1.1.3.3.1.cmml" xref="S3.p1.10.m10.1.1.3.3.1"></times><ci id="S3.p1.10.m10.1.1.3.3.2.cmml" xref="S3.p1.10.m10.1.1.3.3.2">𝑃</ci><ci id="S3.p1.10.m10.1.1.3.3.3.cmml" xref="S3.p1.10.m10.1.1.3.3.3">𝐵</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.10.m10.1c">W_{T_{i}}=W_{T_{i-1}}+PB</annotation></semantics></math>, where only the weights of <math id="S3.p1.11.m11.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S3.p1.11.m11.1a"><mi id="S3.p1.11.m11.1.1" xref="S3.p1.11.m11.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S3.p1.11.m11.1b"><ci id="S3.p1.11.m11.1.1.cmml" xref="S3.p1.11.m11.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.11.m11.1c">B</annotation></semantics></math> are updated. <math id="S3.p1.12.m12.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S3.p1.12.m12.1a"><mi id="S3.p1.12.m12.1.1" xref="S3.p1.12.m12.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S3.p1.12.m12.1b"><ci id="S3.p1.12.m12.1.1.cmml" xref="S3.p1.12.m12.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.12.m12.1c">P</annotation></semantics></math> and <math id="S3.p1.13.m13.1" class="ltx_Math" alttext="W_{T_{i-1}}" display="inline"><semantics id="S3.p1.13.m13.1a"><msub id="S3.p1.13.m13.1.1" xref="S3.p1.13.m13.1.1.cmml"><mi id="S3.p1.13.m13.1.1.2" xref="S3.p1.13.m13.1.1.2.cmml">W</mi><msub id="S3.p1.13.m13.1.1.3" xref="S3.p1.13.m13.1.1.3.cmml"><mi id="S3.p1.13.m13.1.1.3.2" xref="S3.p1.13.m13.1.1.3.2.cmml">T</mi><mrow id="S3.p1.13.m13.1.1.3.3" xref="S3.p1.13.m13.1.1.3.3.cmml"><mi id="S3.p1.13.m13.1.1.3.3.2" xref="S3.p1.13.m13.1.1.3.3.2.cmml">i</mi><mo id="S3.p1.13.m13.1.1.3.3.1" xref="S3.p1.13.m13.1.1.3.3.1.cmml">−</mo><mn id="S3.p1.13.m13.1.1.3.3.3" xref="S3.p1.13.m13.1.1.3.3.3.cmml">1</mn></mrow></msub></msub><annotation-xml encoding="MathML-Content" id="S3.p1.13.m13.1b"><apply id="S3.p1.13.m13.1.1.cmml" xref="S3.p1.13.m13.1.1"><csymbol cd="ambiguous" id="S3.p1.13.m13.1.1.1.cmml" xref="S3.p1.13.m13.1.1">subscript</csymbol><ci id="S3.p1.13.m13.1.1.2.cmml" xref="S3.p1.13.m13.1.1.2">𝑊</ci><apply id="S3.p1.13.m13.1.1.3.cmml" xref="S3.p1.13.m13.1.1.3"><csymbol cd="ambiguous" id="S3.p1.13.m13.1.1.3.1.cmml" xref="S3.p1.13.m13.1.1.3">subscript</csymbol><ci id="S3.p1.13.m13.1.1.3.2.cmml" xref="S3.p1.13.m13.1.1.3.2">𝑇</ci><apply id="S3.p1.13.m13.1.1.3.3.cmml" xref="S3.p1.13.m13.1.1.3.3"><minus id="S3.p1.13.m13.1.1.3.3.1.cmml" xref="S3.p1.13.m13.1.1.3.3.1"></minus><ci id="S3.p1.13.m13.1.1.3.3.2.cmml" xref="S3.p1.13.m13.1.1.3.3.2">𝑖</ci><cn type="integer" id="S3.p1.13.m13.1.1.3.3.3.cmml" xref="S3.p1.13.m13.1.1.3.3.3">1</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.13.m13.1c">W_{T_{i-1}}</annotation></semantics></math> are kept constant over the time interval. We describe this in more detail below, followed by a discussion on periodic updating of the factor <math id="S3.p1.14.m14.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S3.p1.14.m14.1a"><mi id="S3.p1.14.m14.1.1" xref="S3.p1.14.m14.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S3.p1.14.m14.1b"><ci id="S3.p1.14.m14.1.1.cmml" xref="S3.p1.14.m14.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.14.m14.1c">P</annotation></semantics></math>, the enabling of quantized pre-training, error compensation, and exponential update intervals. Pseudo-code for <span id="S3.p1.14.2" class="ltx_text ltx_font_smallcaps">LoQT</span> is shown in Fig. <a href="#S3.F3" title="Figure 3 ‣ 3.4 Quantized Training ‣ 3 Efficient Pretraining With LoQT ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>.</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Background: GaLore</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">Zhao et al. <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite> show that gradients exhibit a low-rank structure during training. They exploit this insight by projecting the gradient to a low-rank subspace and applying the Adam optimizer before projecting back to the original dimensions. By doing this, the memory-intensive optimizer states required by Adam are shrunk significantly for low enough ranks.</p>
</div>
<div id="S3.Thmdefinition1" class="ltx_theorem ltx_theorem_definition">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="S3.Thmdefinition1.1.1.1" class="ltx_text ltx_font_bold">Definition 3.1</span></span><span id="S3.Thmdefinition1.2.2" class="ltx_text ltx_font_bold"> </span>(Gradient Low-rank Projection, def. 3.4 in <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite>)<span id="S3.Thmdefinition1.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="S3.Thmdefinition1.p1" class="ltx_para">
<p id="S3.Thmdefinition1.p1.4" class="ltx_p">Gradient low-rank projection (<span id="S3.Thmdefinition1.p1.4.1" class="ltx_text ltx_font_bold">GaLore</span>) denotes the following gradient update rules, where <math id="S3.Thmdefinition1.p1.1.m1.1" class="ltx_Math" alttext="\eta" display="inline"><semantics id="S3.Thmdefinition1.p1.1.m1.1a"><mi id="S3.Thmdefinition1.p1.1.m1.1.1" xref="S3.Thmdefinition1.p1.1.m1.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S3.Thmdefinition1.p1.1.m1.1b"><ci id="S3.Thmdefinition1.p1.1.m1.1.1.cmml" xref="S3.Thmdefinition1.p1.1.m1.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.Thmdefinition1.p1.1.m1.1c">\eta</annotation></semantics></math> is the learning rate, <math id="S3.Thmdefinition1.p1.2.m2.1" class="ltx_Math" alttext="\rho" display="inline"><semantics id="S3.Thmdefinition1.p1.2.m2.1a"><mi id="S3.Thmdefinition1.p1.2.m2.1.1" xref="S3.Thmdefinition1.p1.2.m2.1.1.cmml">ρ</mi><annotation-xml encoding="MathML-Content" id="S3.Thmdefinition1.p1.2.m2.1b"><ci id="S3.Thmdefinition1.p1.2.m2.1.1.cmml" xref="S3.Thmdefinition1.p1.2.m2.1.1">𝜌</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.Thmdefinition1.p1.2.m2.1c">\rho</annotation></semantics></math> is the Adam optimizer, and <math id="S3.Thmdefinition1.p1.3.m3.1" class="ltx_Math" alttext="W\in R^{m\times n}" display="inline"><semantics id="S3.Thmdefinition1.p1.3.m3.1a"><mrow id="S3.Thmdefinition1.p1.3.m3.1.1" xref="S3.Thmdefinition1.p1.3.m3.1.1.cmml"><mi id="S3.Thmdefinition1.p1.3.m3.1.1.2" xref="S3.Thmdefinition1.p1.3.m3.1.1.2.cmml">W</mi><mo id="S3.Thmdefinition1.p1.3.m3.1.1.1" xref="S3.Thmdefinition1.p1.3.m3.1.1.1.cmml">∈</mo><msup id="S3.Thmdefinition1.p1.3.m3.1.1.3" xref="S3.Thmdefinition1.p1.3.m3.1.1.3.cmml"><mi id="S3.Thmdefinition1.p1.3.m3.1.1.3.2" xref="S3.Thmdefinition1.p1.3.m3.1.1.3.2.cmml">R</mi><mrow id="S3.Thmdefinition1.p1.3.m3.1.1.3.3" xref="S3.Thmdefinition1.p1.3.m3.1.1.3.3.cmml"><mi id="S3.Thmdefinition1.p1.3.m3.1.1.3.3.2" xref="S3.Thmdefinition1.p1.3.m3.1.1.3.3.2.cmml">m</mi><mo lspace="0.222em" rspace="0.222em" id="S3.Thmdefinition1.p1.3.m3.1.1.3.3.1" xref="S3.Thmdefinition1.p1.3.m3.1.1.3.3.1.cmml">×</mo><mi id="S3.Thmdefinition1.p1.3.m3.1.1.3.3.3" xref="S3.Thmdefinition1.p1.3.m3.1.1.3.3.3.cmml">n</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.Thmdefinition1.p1.3.m3.1b"><apply id="S3.Thmdefinition1.p1.3.m3.1.1.cmml" xref="S3.Thmdefinition1.p1.3.m3.1.1"><in id="S3.Thmdefinition1.p1.3.m3.1.1.1.cmml" xref="S3.Thmdefinition1.p1.3.m3.1.1.1"></in><ci id="S3.Thmdefinition1.p1.3.m3.1.1.2.cmml" xref="S3.Thmdefinition1.p1.3.m3.1.1.2">𝑊</ci><apply id="S3.Thmdefinition1.p1.3.m3.1.1.3.cmml" xref="S3.Thmdefinition1.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="S3.Thmdefinition1.p1.3.m3.1.1.3.1.cmml" xref="S3.Thmdefinition1.p1.3.m3.1.1.3">superscript</csymbol><ci id="S3.Thmdefinition1.p1.3.m3.1.1.3.2.cmml" xref="S3.Thmdefinition1.p1.3.m3.1.1.3.2">𝑅</ci><apply id="S3.Thmdefinition1.p1.3.m3.1.1.3.3.cmml" xref="S3.Thmdefinition1.p1.3.m3.1.1.3.3"><times id="S3.Thmdefinition1.p1.3.m3.1.1.3.3.1.cmml" xref="S3.Thmdefinition1.p1.3.m3.1.1.3.3.1"></times><ci id="S3.Thmdefinition1.p1.3.m3.1.1.3.3.2.cmml" xref="S3.Thmdefinition1.p1.3.m3.1.1.3.3.2">𝑚</ci><ci id="S3.Thmdefinition1.p1.3.m3.1.1.3.3.3.cmml" xref="S3.Thmdefinition1.p1.3.m3.1.1.3.3.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.Thmdefinition1.p1.3.m3.1c">W\in R^{m\times n}</annotation></semantics></math> is the weight matrix being trained, and <math id="S3.Thmdefinition1.p1.4.m4.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S3.Thmdefinition1.p1.4.m4.1a"><mi id="S3.Thmdefinition1.p1.4.m4.1.1" xref="S3.Thmdefinition1.p1.4.m4.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S3.Thmdefinition1.p1.4.m4.1b"><ci id="S3.Thmdefinition1.p1.4.m4.1.1.cmml" xref="S3.Thmdefinition1.p1.4.m4.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.Thmdefinition1.p1.4.m4.1c">T</annotation></semantics></math> represents the total number of training iterations between recomputing the projection matrices:</p>
<table id="S3.Ex1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.Ex1.m1.2" class="ltx_Math" alttext="W_{T}=W_{0}+\eta\sum_{t=0}^{T-1}\tilde{G}_{t},\text{ where}\hskip 10.00002pt\tilde{G}_{t}=P_{t}\rho_{t}(P_{t}^{\top}G_{t}Q_{t})Q_{t}^{\top}," display="block"><semantics id="S3.Ex1.m1.2a"><mrow id="S3.Ex1.m1.2.2.1"><mrow id="S3.Ex1.m1.2.2.1.1.2" xref="S3.Ex1.m1.2.2.1.1.3.cmml"><mrow id="S3.Ex1.m1.2.2.1.1.1.1" xref="S3.Ex1.m1.2.2.1.1.1.1.cmml"><msub id="S3.Ex1.m1.2.2.1.1.1.1.3" xref="S3.Ex1.m1.2.2.1.1.1.1.3.cmml"><mi id="S3.Ex1.m1.2.2.1.1.1.1.3.2" xref="S3.Ex1.m1.2.2.1.1.1.1.3.2.cmml">W</mi><mi id="S3.Ex1.m1.2.2.1.1.1.1.3.3" xref="S3.Ex1.m1.2.2.1.1.1.1.3.3.cmml">T</mi></msub><mo id="S3.Ex1.m1.2.2.1.1.1.1.2" xref="S3.Ex1.m1.2.2.1.1.1.1.2.cmml">=</mo><mrow id="S3.Ex1.m1.2.2.1.1.1.1.1.1" xref="S3.Ex1.m1.2.2.1.1.1.1.1.2.cmml"><mrow id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.cmml"><msub id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.2" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.2.cmml"><mi id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.2.2" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.2.2.cmml">W</mi><mn id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.2.3" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.2.3.cmml">0</mn></msub><mo id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.1" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.cmml"><mi id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.2" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.2.cmml">η</mi><mo lspace="0em" rspace="0em" id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.1" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.1.cmml">​</mo><mrow id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.cmml"><munderover id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.cmml"><mo movablelimits="false" id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.2" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.2.cmml">∑</mo><mrow id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.3" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.3.cmml"><mi id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.3.2" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.3.2.cmml">t</mi><mo id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.3.1" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.3.1.cmml">=</mo><mn id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.3.3" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.3.3.cmml">0</mn></mrow><mrow id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.3" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.3.cmml"><mi id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.3.2" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.3.2.cmml">T</mi><mo id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.3.1" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.3.1.cmml">−</mo><mn id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.3.3" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.3.3.cmml">1</mn></mrow></munderover><msub id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2.cmml"><mover accent="true" id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2.2" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2.2.cmml"><mi id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2.2.2" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2.2.2.cmml">G</mi><mo id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2.2.1" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2.2.1.cmml">~</mo></mover><mi id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2.3" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2.3.cmml">t</mi></msub></mrow></mrow></mrow><mo id="S3.Ex1.m1.2.2.1.1.1.1.1.1.2" xref="S3.Ex1.m1.2.2.1.1.1.1.1.2.cmml">,</mo><mtext id="S3.Ex1.m1.1.1" xref="S3.Ex1.m1.1.1a.cmml"> where</mtext></mrow></mrow><mspace width="1em" id="S3.Ex1.m1.2.2.1.1.2.3" xref="S3.Ex1.m1.2.2.1.1.3a.cmml"></mspace><mrow id="S3.Ex1.m1.2.2.1.1.2.2" xref="S3.Ex1.m1.2.2.1.1.2.2.cmml"><msub id="S3.Ex1.m1.2.2.1.1.2.2.3" xref="S3.Ex1.m1.2.2.1.1.2.2.3.cmml"><mover accent="true" id="S3.Ex1.m1.2.2.1.1.2.2.3.2" xref="S3.Ex1.m1.2.2.1.1.2.2.3.2.cmml"><mi id="S3.Ex1.m1.2.2.1.1.2.2.3.2.2" xref="S3.Ex1.m1.2.2.1.1.2.2.3.2.2.cmml">G</mi><mo id="S3.Ex1.m1.2.2.1.1.2.2.3.2.1" xref="S3.Ex1.m1.2.2.1.1.2.2.3.2.1.cmml">~</mo></mover><mi id="S3.Ex1.m1.2.2.1.1.2.2.3.3" xref="S3.Ex1.m1.2.2.1.1.2.2.3.3.cmml">t</mi></msub><mo id="S3.Ex1.m1.2.2.1.1.2.2.2" xref="S3.Ex1.m1.2.2.1.1.2.2.2.cmml">=</mo><mrow id="S3.Ex1.m1.2.2.1.1.2.2.1" xref="S3.Ex1.m1.2.2.1.1.2.2.1.cmml"><msub id="S3.Ex1.m1.2.2.1.1.2.2.1.3" xref="S3.Ex1.m1.2.2.1.1.2.2.1.3.cmml"><mi id="S3.Ex1.m1.2.2.1.1.2.2.1.3.2" xref="S3.Ex1.m1.2.2.1.1.2.2.1.3.2.cmml">P</mi><mi id="S3.Ex1.m1.2.2.1.1.2.2.1.3.3" xref="S3.Ex1.m1.2.2.1.1.2.2.1.3.3.cmml">t</mi></msub><mo lspace="0em" rspace="0em" id="S3.Ex1.m1.2.2.1.1.2.2.1.2" xref="S3.Ex1.m1.2.2.1.1.2.2.1.2.cmml">​</mo><msub id="S3.Ex1.m1.2.2.1.1.2.2.1.4" xref="S3.Ex1.m1.2.2.1.1.2.2.1.4.cmml"><mi id="S3.Ex1.m1.2.2.1.1.2.2.1.4.2" xref="S3.Ex1.m1.2.2.1.1.2.2.1.4.2.cmml">ρ</mi><mi id="S3.Ex1.m1.2.2.1.1.2.2.1.4.3" xref="S3.Ex1.m1.2.2.1.1.2.2.1.4.3.cmml">t</mi></msub><mo lspace="0em" rspace="0em" id="S3.Ex1.m1.2.2.1.1.2.2.1.2a" xref="S3.Ex1.m1.2.2.1.1.2.2.1.2.cmml">​</mo><mrow id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.cmml"><mo stretchy="false" id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.2" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.cmml">(</mo><mrow id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.cmml"><msubsup id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2.cmml"><mi id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2.2.2" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2.2.2.cmml">P</mi><mi id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2.2.3" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2.2.3.cmml">t</mi><mo id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2.3" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2.3.cmml">⊤</mo></msubsup><mo lspace="0em" rspace="0em" id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.1" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.1.cmml">​</mo><msub id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.3" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.3.cmml"><mi id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.3.2" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.3.2.cmml">G</mi><mi id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.3.3" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.3.3.cmml">t</mi></msub><mo lspace="0em" rspace="0em" id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.1a" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.1.cmml">​</mo><msub id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.4" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.4.cmml"><mi id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.4.2" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.4.2.cmml">Q</mi><mi id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.4.3" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.4.3.cmml">t</mi></msub></mrow><mo stretchy="false" id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.3" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S3.Ex1.m1.2.2.1.1.2.2.1.2b" xref="S3.Ex1.m1.2.2.1.1.2.2.1.2.cmml">​</mo><msubsup id="S3.Ex1.m1.2.2.1.1.2.2.1.5" xref="S3.Ex1.m1.2.2.1.1.2.2.1.5.cmml"><mi id="S3.Ex1.m1.2.2.1.1.2.2.1.5.2.2" xref="S3.Ex1.m1.2.2.1.1.2.2.1.5.2.2.cmml">Q</mi><mi id="S3.Ex1.m1.2.2.1.1.2.2.1.5.2.3" xref="S3.Ex1.m1.2.2.1.1.2.2.1.5.2.3.cmml">t</mi><mo id="S3.Ex1.m1.2.2.1.1.2.2.1.5.3" xref="S3.Ex1.m1.2.2.1.1.2.2.1.5.3.cmml">⊤</mo></msubsup></mrow></mrow></mrow><mo id="S3.Ex1.m1.2.2.1.2">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.Ex1.m1.2b"><apply id="S3.Ex1.m1.2.2.1.1.3.cmml" xref="S3.Ex1.m1.2.2.1.1.2"><csymbol cd="ambiguous" id="S3.Ex1.m1.2.2.1.1.3a.cmml" xref="S3.Ex1.m1.2.2.1.1.2.3">formulae-sequence</csymbol><apply id="S3.Ex1.m1.2.2.1.1.1.1.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1"><eq id="S3.Ex1.m1.2.2.1.1.1.1.2.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.2"></eq><apply id="S3.Ex1.m1.2.2.1.1.1.1.3.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.Ex1.m1.2.2.1.1.1.1.3.1.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.3">subscript</csymbol><ci id="S3.Ex1.m1.2.2.1.1.1.1.3.2.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.3.2">𝑊</ci><ci id="S3.Ex1.m1.2.2.1.1.1.1.3.3.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.3.3">𝑇</ci></apply><list id="S3.Ex1.m1.2.2.1.1.1.1.1.2.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1"><apply id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1"><plus id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.1.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.1"></plus><apply id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.2.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.2.1.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.2.2.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.2.2">𝑊</ci><cn type="integer" id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.2.3.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.2.3">0</cn></apply><apply id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3"><times id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.1.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.1"></times><ci id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.2.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.2">𝜂</ci><apply id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3"><apply id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1"><csymbol cd="ambiguous" id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.1.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1">superscript</csymbol><apply id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1"><csymbol cd="ambiguous" id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.1.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1">subscript</csymbol><sum id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.2.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.2"></sum><apply id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.3.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.3"><eq id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.3.1.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.3.1"></eq><ci id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.3.2.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.3.2">𝑡</ci><cn type="integer" id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.3.3.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.2.3.3">0</cn></apply></apply><apply id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.3.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.3"><minus id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.3.1.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.3.1"></minus><ci id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.3.2.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.3.2">𝑇</ci><cn type="integer" id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.3.3.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.1.3.3">1</cn></apply></apply><apply id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2"><csymbol cd="ambiguous" id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2.1.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2">subscript</csymbol><apply id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2.2.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2.2"><ci id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2.2.1.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2.2.1">~</ci><ci id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2.2.2.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2.2.2">𝐺</ci></apply><ci id="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2.3.cmml" xref="S3.Ex1.m1.2.2.1.1.1.1.1.1.1.3.3.2.3">𝑡</ci></apply></apply></apply></apply><ci id="S3.Ex1.m1.1.1a.cmml" xref="S3.Ex1.m1.1.1"><mtext id="S3.Ex1.m1.1.1.cmml" xref="S3.Ex1.m1.1.1"> where</mtext></ci></list></apply><apply id="S3.Ex1.m1.2.2.1.1.2.2.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2"><eq id="S3.Ex1.m1.2.2.1.1.2.2.2.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.2"></eq><apply id="S3.Ex1.m1.2.2.1.1.2.2.3.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.3"><csymbol cd="ambiguous" id="S3.Ex1.m1.2.2.1.1.2.2.3.1.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.3">subscript</csymbol><apply id="S3.Ex1.m1.2.2.1.1.2.2.3.2.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.3.2"><ci id="S3.Ex1.m1.2.2.1.1.2.2.3.2.1.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.3.2.1">~</ci><ci id="S3.Ex1.m1.2.2.1.1.2.2.3.2.2.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.3.2.2">𝐺</ci></apply><ci id="S3.Ex1.m1.2.2.1.1.2.2.3.3.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.3.3">𝑡</ci></apply><apply id="S3.Ex1.m1.2.2.1.1.2.2.1.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1"><times id="S3.Ex1.m1.2.2.1.1.2.2.1.2.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.2"></times><apply id="S3.Ex1.m1.2.2.1.1.2.2.1.3.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.3"><csymbol cd="ambiguous" id="S3.Ex1.m1.2.2.1.1.2.2.1.3.1.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.3">subscript</csymbol><ci id="S3.Ex1.m1.2.2.1.1.2.2.1.3.2.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.3.2">𝑃</ci><ci id="S3.Ex1.m1.2.2.1.1.2.2.1.3.3.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.3.3">𝑡</ci></apply><apply id="S3.Ex1.m1.2.2.1.1.2.2.1.4.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.4"><csymbol cd="ambiguous" id="S3.Ex1.m1.2.2.1.1.2.2.1.4.1.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.4">subscript</csymbol><ci id="S3.Ex1.m1.2.2.1.1.2.2.1.4.2.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.4.2">𝜌</ci><ci id="S3.Ex1.m1.2.2.1.1.2.2.1.4.3.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.4.3">𝑡</ci></apply><apply id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1"><times id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.1.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.1"></times><apply id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2.1.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2">superscript</csymbol><apply id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2.2.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2.2.1.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2">subscript</csymbol><ci id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2.2.2.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2.2.2">𝑃</ci><ci id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2.2.3.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2.2.3">𝑡</ci></apply><csymbol cd="latexml" id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2.3.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.2.3">top</csymbol></apply><apply id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.3.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.3.1.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.3">subscript</csymbol><ci id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.3.2.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.3.2">𝐺</ci><ci id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.3.3.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.3.3">𝑡</ci></apply><apply id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.4.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.4.1.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.4">subscript</csymbol><ci id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.4.2.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.4.2">𝑄</ci><ci id="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.4.3.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.1.1.1.4.3">𝑡</ci></apply></apply><apply id="S3.Ex1.m1.2.2.1.1.2.2.1.5.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.5"><csymbol cd="ambiguous" id="S3.Ex1.m1.2.2.1.1.2.2.1.5.1.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.5">superscript</csymbol><apply id="S3.Ex1.m1.2.2.1.1.2.2.1.5.2.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.5"><csymbol cd="ambiguous" id="S3.Ex1.m1.2.2.1.1.2.2.1.5.2.1.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.5">subscript</csymbol><ci id="S3.Ex1.m1.2.2.1.1.2.2.1.5.2.2.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.5.2.2">𝑄</ci><ci id="S3.Ex1.m1.2.2.1.1.2.2.1.5.2.3.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.5.2.3">𝑡</ci></apply><csymbol cd="latexml" id="S3.Ex1.m1.2.2.1.1.2.2.1.5.3.cmml" xref="S3.Ex1.m1.2.2.1.1.2.2.1.5.3">top</csymbol></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.Ex1.m1.2c">W_{T}=W_{0}+\eta\sum_{t=0}^{T-1}\tilde{G}_{t},\text{ where}\hskip 10.00002pt\tilde{G}_{t}=P_{t}\rho_{t}(P_{t}^{\top}G_{t}Q_{t})Q_{t}^{\top},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S3.Thmdefinition1.p1.8" class="ltx_p">where <math id="S3.Thmdefinition1.p1.5.m1.1" class="ltx_Math" alttext="P_{t}\in R^{m\times r}" display="inline"><semantics id="S3.Thmdefinition1.p1.5.m1.1a"><mrow id="S3.Thmdefinition1.p1.5.m1.1.1" xref="S3.Thmdefinition1.p1.5.m1.1.1.cmml"><msub id="S3.Thmdefinition1.p1.5.m1.1.1.2" xref="S3.Thmdefinition1.p1.5.m1.1.1.2.cmml"><mi id="S3.Thmdefinition1.p1.5.m1.1.1.2.2" xref="S3.Thmdefinition1.p1.5.m1.1.1.2.2.cmml">P</mi><mi id="S3.Thmdefinition1.p1.5.m1.1.1.2.3" xref="S3.Thmdefinition1.p1.5.m1.1.1.2.3.cmml">t</mi></msub><mo id="S3.Thmdefinition1.p1.5.m1.1.1.1" xref="S3.Thmdefinition1.p1.5.m1.1.1.1.cmml">∈</mo><msup id="S3.Thmdefinition1.p1.5.m1.1.1.3" xref="S3.Thmdefinition1.p1.5.m1.1.1.3.cmml"><mi id="S3.Thmdefinition1.p1.5.m1.1.1.3.2" xref="S3.Thmdefinition1.p1.5.m1.1.1.3.2.cmml">R</mi><mrow id="S3.Thmdefinition1.p1.5.m1.1.1.3.3" xref="S3.Thmdefinition1.p1.5.m1.1.1.3.3.cmml"><mi id="S3.Thmdefinition1.p1.5.m1.1.1.3.3.2" xref="S3.Thmdefinition1.p1.5.m1.1.1.3.3.2.cmml">m</mi><mo lspace="0.222em" rspace="0.222em" id="S3.Thmdefinition1.p1.5.m1.1.1.3.3.1" xref="S3.Thmdefinition1.p1.5.m1.1.1.3.3.1.cmml">×</mo><mi id="S3.Thmdefinition1.p1.5.m1.1.1.3.3.3" xref="S3.Thmdefinition1.p1.5.m1.1.1.3.3.3.cmml">r</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.Thmdefinition1.p1.5.m1.1b"><apply id="S3.Thmdefinition1.p1.5.m1.1.1.cmml" xref="S3.Thmdefinition1.p1.5.m1.1.1"><in id="S3.Thmdefinition1.p1.5.m1.1.1.1.cmml" xref="S3.Thmdefinition1.p1.5.m1.1.1.1"></in><apply id="S3.Thmdefinition1.p1.5.m1.1.1.2.cmml" xref="S3.Thmdefinition1.p1.5.m1.1.1.2"><csymbol cd="ambiguous" id="S3.Thmdefinition1.p1.5.m1.1.1.2.1.cmml" xref="S3.Thmdefinition1.p1.5.m1.1.1.2">subscript</csymbol><ci id="S3.Thmdefinition1.p1.5.m1.1.1.2.2.cmml" xref="S3.Thmdefinition1.p1.5.m1.1.1.2.2">𝑃</ci><ci id="S3.Thmdefinition1.p1.5.m1.1.1.2.3.cmml" xref="S3.Thmdefinition1.p1.5.m1.1.1.2.3">𝑡</ci></apply><apply id="S3.Thmdefinition1.p1.5.m1.1.1.3.cmml" xref="S3.Thmdefinition1.p1.5.m1.1.1.3"><csymbol cd="ambiguous" id="S3.Thmdefinition1.p1.5.m1.1.1.3.1.cmml" xref="S3.Thmdefinition1.p1.5.m1.1.1.3">superscript</csymbol><ci id="S3.Thmdefinition1.p1.5.m1.1.1.3.2.cmml" xref="S3.Thmdefinition1.p1.5.m1.1.1.3.2">𝑅</ci><apply id="S3.Thmdefinition1.p1.5.m1.1.1.3.3.cmml" xref="S3.Thmdefinition1.p1.5.m1.1.1.3.3"><times id="S3.Thmdefinition1.p1.5.m1.1.1.3.3.1.cmml" xref="S3.Thmdefinition1.p1.5.m1.1.1.3.3.1"></times><ci id="S3.Thmdefinition1.p1.5.m1.1.1.3.3.2.cmml" xref="S3.Thmdefinition1.p1.5.m1.1.1.3.3.2">𝑚</ci><ci id="S3.Thmdefinition1.p1.5.m1.1.1.3.3.3.cmml" xref="S3.Thmdefinition1.p1.5.m1.1.1.3.3.3">𝑟</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.Thmdefinition1.p1.5.m1.1c">P_{t}\in R^{m\times r}</annotation></semantics></math> and <math id="S3.Thmdefinition1.p1.6.m2.1" class="ltx_Math" alttext="Q_{t}\in R^{n\times r}" display="inline"><semantics id="S3.Thmdefinition1.p1.6.m2.1a"><mrow id="S3.Thmdefinition1.p1.6.m2.1.1" xref="S3.Thmdefinition1.p1.6.m2.1.1.cmml"><msub id="S3.Thmdefinition1.p1.6.m2.1.1.2" xref="S3.Thmdefinition1.p1.6.m2.1.1.2.cmml"><mi id="S3.Thmdefinition1.p1.6.m2.1.1.2.2" xref="S3.Thmdefinition1.p1.6.m2.1.1.2.2.cmml">Q</mi><mi id="S3.Thmdefinition1.p1.6.m2.1.1.2.3" xref="S3.Thmdefinition1.p1.6.m2.1.1.2.3.cmml">t</mi></msub><mo id="S3.Thmdefinition1.p1.6.m2.1.1.1" xref="S3.Thmdefinition1.p1.6.m2.1.1.1.cmml">∈</mo><msup id="S3.Thmdefinition1.p1.6.m2.1.1.3" xref="S3.Thmdefinition1.p1.6.m2.1.1.3.cmml"><mi id="S3.Thmdefinition1.p1.6.m2.1.1.3.2" xref="S3.Thmdefinition1.p1.6.m2.1.1.3.2.cmml">R</mi><mrow id="S3.Thmdefinition1.p1.6.m2.1.1.3.3" xref="S3.Thmdefinition1.p1.6.m2.1.1.3.3.cmml"><mi id="S3.Thmdefinition1.p1.6.m2.1.1.3.3.2" xref="S3.Thmdefinition1.p1.6.m2.1.1.3.3.2.cmml">n</mi><mo lspace="0.222em" rspace="0.222em" id="S3.Thmdefinition1.p1.6.m2.1.1.3.3.1" xref="S3.Thmdefinition1.p1.6.m2.1.1.3.3.1.cmml">×</mo><mi id="S3.Thmdefinition1.p1.6.m2.1.1.3.3.3" xref="S3.Thmdefinition1.p1.6.m2.1.1.3.3.3.cmml">r</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.Thmdefinition1.p1.6.m2.1b"><apply id="S3.Thmdefinition1.p1.6.m2.1.1.cmml" xref="S3.Thmdefinition1.p1.6.m2.1.1"><in id="S3.Thmdefinition1.p1.6.m2.1.1.1.cmml" xref="S3.Thmdefinition1.p1.6.m2.1.1.1"></in><apply id="S3.Thmdefinition1.p1.6.m2.1.1.2.cmml" xref="S3.Thmdefinition1.p1.6.m2.1.1.2"><csymbol cd="ambiguous" id="S3.Thmdefinition1.p1.6.m2.1.1.2.1.cmml" xref="S3.Thmdefinition1.p1.6.m2.1.1.2">subscript</csymbol><ci id="S3.Thmdefinition1.p1.6.m2.1.1.2.2.cmml" xref="S3.Thmdefinition1.p1.6.m2.1.1.2.2">𝑄</ci><ci id="S3.Thmdefinition1.p1.6.m2.1.1.2.3.cmml" xref="S3.Thmdefinition1.p1.6.m2.1.1.2.3">𝑡</ci></apply><apply id="S3.Thmdefinition1.p1.6.m2.1.1.3.cmml" xref="S3.Thmdefinition1.p1.6.m2.1.1.3"><csymbol cd="ambiguous" id="S3.Thmdefinition1.p1.6.m2.1.1.3.1.cmml" xref="S3.Thmdefinition1.p1.6.m2.1.1.3">superscript</csymbol><ci id="S3.Thmdefinition1.p1.6.m2.1.1.3.2.cmml" xref="S3.Thmdefinition1.p1.6.m2.1.1.3.2">𝑅</ci><apply id="S3.Thmdefinition1.p1.6.m2.1.1.3.3.cmml" xref="S3.Thmdefinition1.p1.6.m2.1.1.3.3"><times id="S3.Thmdefinition1.p1.6.m2.1.1.3.3.1.cmml" xref="S3.Thmdefinition1.p1.6.m2.1.1.3.3.1"></times><ci id="S3.Thmdefinition1.p1.6.m2.1.1.3.3.2.cmml" xref="S3.Thmdefinition1.p1.6.m2.1.1.3.3.2">𝑛</ci><ci id="S3.Thmdefinition1.p1.6.m2.1.1.3.3.3.cmml" xref="S3.Thmdefinition1.p1.6.m2.1.1.3.3.3">𝑟</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.Thmdefinition1.p1.6.m2.1c">Q_{t}\in R^{n\times r}</annotation></semantics></math> are are the top-<math id="S3.Thmdefinition1.p1.7.m3.1" class="ltx_Math" alttext="r" display="inline"><semantics id="S3.Thmdefinition1.p1.7.m3.1a"><mi id="S3.Thmdefinition1.p1.7.m3.1.1" xref="S3.Thmdefinition1.p1.7.m3.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="S3.Thmdefinition1.p1.7.m3.1b"><ci id="S3.Thmdefinition1.p1.7.m3.1.1.cmml" xref="S3.Thmdefinition1.p1.7.m3.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.Thmdefinition1.p1.7.m3.1c">r</annotation></semantics></math> singular vectors from the SVD decomposition of the gradient matrix at each iteration <math id="S3.Thmdefinition1.p1.8.m4.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S3.Thmdefinition1.p1.8.m4.1a"><mi id="S3.Thmdefinition1.p1.8.m4.1.1" xref="S3.Thmdefinition1.p1.8.m4.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S3.Thmdefinition1.p1.8.m4.1b"><ci id="S3.Thmdefinition1.p1.8.m4.1.1.cmml" xref="S3.Thmdefinition1.p1.8.m4.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.Thmdefinition1.p1.8.m4.1c">t</annotation></semantics></math>. In practice, this can be approximated by only applying a one-sided projection, as in</p>
<table id="S3.Ex2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.Ex2.m1.1" class="ltx_Math" alttext="W_{T}^{\prime}=W_{0}+\eta\sum_{t=0}^{T-1}P_{t}\rho_{t}(P_{t}^{\top}G_{t})\hskip 5.0pt\text{ or }\hskip 5.0ptW_{T}^{\prime}=W_{0}+\eta\sum_{t=0}^{T-1}\rho_{t}(G_{t}Q_{t})Q_{t}^{\top}." display="block"><semantics id="S3.Ex2.m1.1a"><mrow id="S3.Ex2.m1.1.1.1" xref="S3.Ex2.m1.1.1.1.1.cmml"><mrow id="S3.Ex2.m1.1.1.1.1" xref="S3.Ex2.m1.1.1.1.1.cmml"><msubsup id="S3.Ex2.m1.1.1.1.1.4" xref="S3.Ex2.m1.1.1.1.1.4.cmml"><mi id="S3.Ex2.m1.1.1.1.1.4.2.2" xref="S3.Ex2.m1.1.1.1.1.4.2.2.cmml">W</mi><mi id="S3.Ex2.m1.1.1.1.1.4.2.3" xref="S3.Ex2.m1.1.1.1.1.4.2.3.cmml">T</mi><mo id="S3.Ex2.m1.1.1.1.1.4.3" xref="S3.Ex2.m1.1.1.1.1.4.3.cmml">′</mo></msubsup><mo id="S3.Ex2.m1.1.1.1.1.5" xref="S3.Ex2.m1.1.1.1.1.5.cmml">=</mo><mrow id="S3.Ex2.m1.1.1.1.1.1" xref="S3.Ex2.m1.1.1.1.1.1.cmml"><msub id="S3.Ex2.m1.1.1.1.1.1.3" xref="S3.Ex2.m1.1.1.1.1.1.3.cmml"><mi id="S3.Ex2.m1.1.1.1.1.1.3.2" xref="S3.Ex2.m1.1.1.1.1.1.3.2.cmml">W</mi><mn id="S3.Ex2.m1.1.1.1.1.1.3.3" xref="S3.Ex2.m1.1.1.1.1.1.3.3.cmml">0</mn></msub><mo id="S3.Ex2.m1.1.1.1.1.1.2" xref="S3.Ex2.m1.1.1.1.1.1.2.cmml">+</mo><mrow id="S3.Ex2.m1.1.1.1.1.1.1" xref="S3.Ex2.m1.1.1.1.1.1.1.cmml"><mi id="S3.Ex2.m1.1.1.1.1.1.1.3" xref="S3.Ex2.m1.1.1.1.1.1.1.3.cmml">η</mi><mo lspace="0em" rspace="0em" id="S3.Ex2.m1.1.1.1.1.1.1.2" xref="S3.Ex2.m1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.Ex2.m1.1.1.1.1.1.1.1" xref="S3.Ex2.m1.1.1.1.1.1.1.1.cmml"><munderover id="S3.Ex2.m1.1.1.1.1.1.1.1.2" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2.cmml"><mo movablelimits="false" id="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.2" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.2.cmml">∑</mo><mrow id="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.3" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.3.cmml"><mi id="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.3.2" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.3.2.cmml">t</mi><mo id="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.3.1" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.3.1.cmml">=</mo><mn id="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.3.3" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.3.3.cmml">0</mn></mrow><mrow id="S3.Ex2.m1.1.1.1.1.1.1.1.2.3" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2.3.cmml"><mi id="S3.Ex2.m1.1.1.1.1.1.1.1.2.3.2" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2.3.2.cmml">T</mi><mo id="S3.Ex2.m1.1.1.1.1.1.1.1.2.3.1" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2.3.1.cmml">−</mo><mn id="S3.Ex2.m1.1.1.1.1.1.1.1.2.3.3" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2.3.3.cmml">1</mn></mrow></munderover><mrow id="S3.Ex2.m1.1.1.1.1.1.1.1.1" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.cmml"><msub id="S3.Ex2.m1.1.1.1.1.1.1.1.1.3" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S3.Ex2.m1.1.1.1.1.1.1.1.1.3.2" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.3.2.cmml">P</mi><mi id="S3.Ex2.m1.1.1.1.1.1.1.1.1.3.3" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.3.3.cmml">t</mi></msub><mo lspace="0em" rspace="0em" id="S3.Ex2.m1.1.1.1.1.1.1.1.1.2" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.2.cmml">​</mo><msub id="S3.Ex2.m1.1.1.1.1.1.1.1.1.4" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.4.cmml"><mi id="S3.Ex2.m1.1.1.1.1.1.1.1.1.4.2" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.4.2.cmml">ρ</mi><mi id="S3.Ex2.m1.1.1.1.1.1.1.1.1.4.3" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.4.3.cmml">t</mi></msub><mo lspace="0em" rspace="0em" id="S3.Ex2.m1.1.1.1.1.1.1.1.1.2a" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.2" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.cmml"><msubsup id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.2" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.2.cmml">P</mi><mi id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.3" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.3.cmml">t</mi><mo id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2.3" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml">⊤</mo></msubsup><mo lspace="0em" rspace="0em" id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">​</mo><msub id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.3.2" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml">G</mi><mi id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.3.3" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml">t</mi></msub></mrow><mo stretchy="false" id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0.500em" rspace="0em" id="S3.Ex2.m1.1.1.1.1.1.1.1.1.2b" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.2.cmml">​</mo><mtext id="S3.Ex2.m1.1.1.1.1.1.1.1.1.5" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.5a.cmml"> or </mtext><mo lspace="0.500em" rspace="0em" id="S3.Ex2.m1.1.1.1.1.1.1.1.1.2c" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.2.cmml">​</mo><msubsup id="S3.Ex2.m1.1.1.1.1.1.1.1.1.6" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.6.cmml"><mi id="S3.Ex2.m1.1.1.1.1.1.1.1.1.6.2.2" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.6.2.2.cmml">W</mi><mi id="S3.Ex2.m1.1.1.1.1.1.1.1.1.6.2.3" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.6.2.3.cmml">T</mi><mo id="S3.Ex2.m1.1.1.1.1.1.1.1.1.6.3" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.6.3.cmml">′</mo></msubsup></mrow></mrow></mrow></mrow><mo id="S3.Ex2.m1.1.1.1.1.6" xref="S3.Ex2.m1.1.1.1.1.6.cmml">=</mo><mrow id="S3.Ex2.m1.1.1.1.1.2" xref="S3.Ex2.m1.1.1.1.1.2.cmml"><msub id="S3.Ex2.m1.1.1.1.1.2.3" xref="S3.Ex2.m1.1.1.1.1.2.3.cmml"><mi id="S3.Ex2.m1.1.1.1.1.2.3.2" xref="S3.Ex2.m1.1.1.1.1.2.3.2.cmml">W</mi><mn id="S3.Ex2.m1.1.1.1.1.2.3.3" xref="S3.Ex2.m1.1.1.1.1.2.3.3.cmml">0</mn></msub><mo id="S3.Ex2.m1.1.1.1.1.2.2" xref="S3.Ex2.m1.1.1.1.1.2.2.cmml">+</mo><mrow id="S3.Ex2.m1.1.1.1.1.2.1" xref="S3.Ex2.m1.1.1.1.1.2.1.cmml"><mi id="S3.Ex2.m1.1.1.1.1.2.1.3" xref="S3.Ex2.m1.1.1.1.1.2.1.3.cmml">η</mi><mo lspace="0em" rspace="0em" id="S3.Ex2.m1.1.1.1.1.2.1.2" xref="S3.Ex2.m1.1.1.1.1.2.1.2.cmml">​</mo><mrow id="S3.Ex2.m1.1.1.1.1.2.1.1" xref="S3.Ex2.m1.1.1.1.1.2.1.1.cmml"><munderover id="S3.Ex2.m1.1.1.1.1.2.1.1.2" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2.cmml"><mo movablelimits="false" id="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.2" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.2.cmml">∑</mo><mrow id="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.3" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.3.cmml"><mi id="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.3.2" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.3.2.cmml">t</mi><mo id="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.3.1" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.3.1.cmml">=</mo><mn id="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.3.3" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.3.3.cmml">0</mn></mrow><mrow id="S3.Ex2.m1.1.1.1.1.2.1.1.2.3" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2.3.cmml"><mi id="S3.Ex2.m1.1.1.1.1.2.1.1.2.3.2" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2.3.2.cmml">T</mi><mo id="S3.Ex2.m1.1.1.1.1.2.1.1.2.3.1" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2.3.1.cmml">−</mo><mn id="S3.Ex2.m1.1.1.1.1.2.1.1.2.3.3" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2.3.3.cmml">1</mn></mrow></munderover><mrow id="S3.Ex2.m1.1.1.1.1.2.1.1.1" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.cmml"><msub id="S3.Ex2.m1.1.1.1.1.2.1.1.1.3" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.3.cmml"><mi id="S3.Ex2.m1.1.1.1.1.2.1.1.1.3.2" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.3.2.cmml">ρ</mi><mi id="S3.Ex2.m1.1.1.1.1.2.1.1.1.3.3" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.3.3.cmml">t</mi></msub><mo lspace="0em" rspace="0em" id="S3.Ex2.m1.1.1.1.1.2.1.1.1.2" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.2.cmml">​</mo><mrow id="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.2" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.cmml"><msub id="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.2" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.2.cmml"><mi id="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.2.2" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.2.2.cmml">G</mi><mi id="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.2.3" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.2.3.cmml">t</mi></msub><mo lspace="0em" rspace="0em" id="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.1" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.1.cmml">​</mo><msub id="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.3" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.3.cmml"><mi id="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.3.2" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.3.2.cmml">Q</mi><mi id="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.3.3" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.3.3.cmml">t</mi></msub></mrow><mo stretchy="false" id="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.3" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S3.Ex2.m1.1.1.1.1.2.1.1.1.2a" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.2.cmml">​</mo><msubsup id="S3.Ex2.m1.1.1.1.1.2.1.1.1.4" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.4.cmml"><mi id="S3.Ex2.m1.1.1.1.1.2.1.1.1.4.2.2" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.4.2.2.cmml">Q</mi><mi id="S3.Ex2.m1.1.1.1.1.2.1.1.1.4.2.3" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.4.2.3.cmml">t</mi><mo id="S3.Ex2.m1.1.1.1.1.2.1.1.1.4.3" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.4.3.cmml">⊤</mo></msubsup></mrow></mrow></mrow></mrow></mrow><mo lspace="0em" id="S3.Ex2.m1.1.1.1.2" xref="S3.Ex2.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.Ex2.m1.1b"><apply id="S3.Ex2.m1.1.1.1.1.cmml" xref="S3.Ex2.m1.1.1.1"><and id="S3.Ex2.m1.1.1.1.1a.cmml" xref="S3.Ex2.m1.1.1.1"></and><apply id="S3.Ex2.m1.1.1.1.1b.cmml" xref="S3.Ex2.m1.1.1.1"><eq id="S3.Ex2.m1.1.1.1.1.5.cmml" xref="S3.Ex2.m1.1.1.1.1.5"></eq><apply id="S3.Ex2.m1.1.1.1.1.4.cmml" xref="S3.Ex2.m1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.4.1.cmml" xref="S3.Ex2.m1.1.1.1.1.4">superscript</csymbol><apply id="S3.Ex2.m1.1.1.1.1.4.2.cmml" xref="S3.Ex2.m1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.4.2.1.cmml" xref="S3.Ex2.m1.1.1.1.1.4">subscript</csymbol><ci id="S3.Ex2.m1.1.1.1.1.4.2.2.cmml" xref="S3.Ex2.m1.1.1.1.1.4.2.2">𝑊</ci><ci id="S3.Ex2.m1.1.1.1.1.4.2.3.cmml" xref="S3.Ex2.m1.1.1.1.1.4.2.3">𝑇</ci></apply><ci id="S3.Ex2.m1.1.1.1.1.4.3.cmml" xref="S3.Ex2.m1.1.1.1.1.4.3">′</ci></apply><apply id="S3.Ex2.m1.1.1.1.1.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1"><plus id="S3.Ex2.m1.1.1.1.1.1.2.cmml" xref="S3.Ex2.m1.1.1.1.1.1.2"></plus><apply id="S3.Ex2.m1.1.1.1.1.1.3.cmml" xref="S3.Ex2.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.1.3.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.Ex2.m1.1.1.1.1.1.3.2.cmml" xref="S3.Ex2.m1.1.1.1.1.1.3.2">𝑊</ci><cn type="integer" id="S3.Ex2.m1.1.1.1.1.1.3.3.cmml" xref="S3.Ex2.m1.1.1.1.1.1.3.3">0</cn></apply><apply id="S3.Ex2.m1.1.1.1.1.1.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1"><times id="S3.Ex2.m1.1.1.1.1.1.1.2.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.2"></times><ci id="S3.Ex2.m1.1.1.1.1.1.1.3.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.3">𝜂</ci><apply id="S3.Ex2.m1.1.1.1.1.1.1.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1"><apply id="S3.Ex2.m1.1.1.1.1.1.1.1.2.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2">superscript</csymbol><apply id="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2">subscript</csymbol><sum id="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.2.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.2"></sum><apply id="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.3.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.3"><eq id="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.3.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.3.1"></eq><ci id="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.3.2.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.3.2">𝑡</ci><cn type="integer" id="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.3.3.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2.2.3.3">0</cn></apply></apply><apply id="S3.Ex2.m1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2.3"><minus id="S3.Ex2.m1.1.1.1.1.1.1.1.2.3.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2.3.1"></minus><ci id="S3.Ex2.m1.1.1.1.1.1.1.1.2.3.2.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2.3.2">𝑇</ci><cn type="integer" id="S3.Ex2.m1.1.1.1.1.1.1.1.2.3.3.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.2.3.3">1</cn></apply></apply><apply id="S3.Ex2.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1"><times id="S3.Ex2.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.2"></times><apply id="S3.Ex2.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.Ex2.m1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.3.2">𝑃</ci><ci id="S3.Ex2.m1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.3.3">𝑡</ci></apply><apply id="S3.Ex2.m1.1.1.1.1.1.1.1.1.4.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.1.1.1.1.4.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.4">subscript</csymbol><ci id="S3.Ex2.m1.1.1.1.1.1.1.1.1.4.2.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.4.2">𝜌</ci><ci id="S3.Ex2.m1.1.1.1.1.1.1.1.1.4.3.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.4.3">𝑡</ci></apply><apply id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1"><times id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.1"></times><apply id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2">superscript</csymbol><apply id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.2.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.2">𝑃</ci><ci id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.3.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.3">𝑡</ci></apply><csymbol cd="latexml" id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.2.3">top</csymbol></apply><apply id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.3.2">𝐺</ci><ci id="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.1.1.1.3.3">𝑡</ci></apply></apply><ci id="S3.Ex2.m1.1.1.1.1.1.1.1.1.5a.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.5"><mtext id="S3.Ex2.m1.1.1.1.1.1.1.1.1.5.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.5"> or </mtext></ci><apply id="S3.Ex2.m1.1.1.1.1.1.1.1.1.6.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.6"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.1.1.1.1.6.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.6">superscript</csymbol><apply id="S3.Ex2.m1.1.1.1.1.1.1.1.1.6.2.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.6"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.1.1.1.1.6.2.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.6">subscript</csymbol><ci id="S3.Ex2.m1.1.1.1.1.1.1.1.1.6.2.2.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.6.2.2">𝑊</ci><ci id="S3.Ex2.m1.1.1.1.1.1.1.1.1.6.2.3.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.6.2.3">𝑇</ci></apply><ci id="S3.Ex2.m1.1.1.1.1.1.1.1.1.6.3.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.1.1.6.3">′</ci></apply></apply></apply></apply></apply></apply><apply id="S3.Ex2.m1.1.1.1.1c.cmml" xref="S3.Ex2.m1.1.1.1"><eq id="S3.Ex2.m1.1.1.1.1.6.cmml" xref="S3.Ex2.m1.1.1.1.1.6"></eq><share href="#S3.Ex2.m1.1.1.1.1.1.cmml" id="S3.Ex2.m1.1.1.1.1d.cmml" xref="S3.Ex2.m1.1.1.1"></share><apply id="S3.Ex2.m1.1.1.1.1.2.cmml" xref="S3.Ex2.m1.1.1.1.1.2"><plus id="S3.Ex2.m1.1.1.1.1.2.2.cmml" xref="S3.Ex2.m1.1.1.1.1.2.2"></plus><apply id="S3.Ex2.m1.1.1.1.1.2.3.cmml" xref="S3.Ex2.m1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.2.3.1.cmml" xref="S3.Ex2.m1.1.1.1.1.2.3">subscript</csymbol><ci id="S3.Ex2.m1.1.1.1.1.2.3.2.cmml" xref="S3.Ex2.m1.1.1.1.1.2.3.2">𝑊</ci><cn type="integer" id="S3.Ex2.m1.1.1.1.1.2.3.3.cmml" xref="S3.Ex2.m1.1.1.1.1.2.3.3">0</cn></apply><apply id="S3.Ex2.m1.1.1.1.1.2.1.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1"><times id="S3.Ex2.m1.1.1.1.1.2.1.2.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.2"></times><ci id="S3.Ex2.m1.1.1.1.1.2.1.3.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.3">𝜂</ci><apply id="S3.Ex2.m1.1.1.1.1.2.1.1.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1"><apply id="S3.Ex2.m1.1.1.1.1.2.1.1.2.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.2.1.1.2.1.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2">superscript</csymbol><apply id="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.1.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2">subscript</csymbol><sum id="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.2.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.2"></sum><apply id="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.3.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.3"><eq id="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.3.1.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.3.1"></eq><ci id="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.3.2.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.3.2">𝑡</ci><cn type="integer" id="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.3.3.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2.2.3.3">0</cn></apply></apply><apply id="S3.Ex2.m1.1.1.1.1.2.1.1.2.3.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2.3"><minus id="S3.Ex2.m1.1.1.1.1.2.1.1.2.3.1.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2.3.1"></minus><ci id="S3.Ex2.m1.1.1.1.1.2.1.1.2.3.2.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2.3.2">𝑇</ci><cn type="integer" id="S3.Ex2.m1.1.1.1.1.2.1.1.2.3.3.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.2.3.3">1</cn></apply></apply><apply id="S3.Ex2.m1.1.1.1.1.2.1.1.1.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1"><times id="S3.Ex2.m1.1.1.1.1.2.1.1.1.2.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.2"></times><apply id="S3.Ex2.m1.1.1.1.1.2.1.1.1.3.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.3"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.2.1.1.1.3.1.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.3">subscript</csymbol><ci id="S3.Ex2.m1.1.1.1.1.2.1.1.1.3.2.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.3.2">𝜌</ci><ci id="S3.Ex2.m1.1.1.1.1.2.1.1.1.3.3.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.3.3">𝑡</ci></apply><apply id="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1"><times id="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.1.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.1"></times><apply id="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.2.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.2.1.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.2.2.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.2.2">𝐺</ci><ci id="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.2.3.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.2.3">𝑡</ci></apply><apply id="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.3.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.3.1.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.3.2.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.3.2">𝑄</ci><ci id="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.3.3.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.1.1.1.3.3">𝑡</ci></apply></apply><apply id="S3.Ex2.m1.1.1.1.1.2.1.1.1.4.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.4"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.2.1.1.1.4.1.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.4">superscript</csymbol><apply id="S3.Ex2.m1.1.1.1.1.2.1.1.1.4.2.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.4"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.2.1.1.1.4.2.1.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.4">subscript</csymbol><ci id="S3.Ex2.m1.1.1.1.1.2.1.1.1.4.2.2.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.4.2.2">𝑄</ci><ci id="S3.Ex2.m1.1.1.1.1.2.1.1.1.4.2.3.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.4.2.3">𝑡</ci></apply><csymbol cd="latexml" id="S3.Ex2.m1.1.1.1.1.2.1.1.1.4.3.cmml" xref="S3.Ex2.m1.1.1.1.1.2.1.1.1.4.3">top</csymbol></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.Ex2.m1.1c">W_{T}^{\prime}=W_{0}+\eta\sum_{t=0}^{T-1}P_{t}\rho_{t}(P_{t}^{\top}G_{t})\hskip 5.0pt\text{ or }\hskip 5.0ptW_{T}^{\prime}=W_{0}+\eta\sum_{t=0}^{T-1}\rho_{t}(G_{t}Q_{t})Q_{t}^{\top}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
</div>
<div id="S3.SS1.p2" class="ltx_para">
<p id="S3.SS1.p2.1" class="ltx_p">GaLore demonstrates that it is sufficient to keep the projection matrix fixed and only update it once every <math id="S3.SS1.p2.1.m1.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S3.SS1.p2.1.m1.1a"><mi id="S3.SS1.p2.1.m1.1.1" xref="S3.SS1.p2.1.m1.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.1.m1.1b"><ci id="S3.SS1.p2.1.m1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.1.m1.1c">T</annotation></semantics></math> iterations, which we use in the following.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Low-rank Gradients as Adapters</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.6" class="ltx_p">We now describe the process by which we initialize the parameters we optimize in <span id="S3.SS2.p1.6.1" class="ltx_text ltx_font_smallcaps">LoQT</span>. We adopt the memory-performance trade-off of using only a one-sided projection. We compute <math id="S3.SS2.p1.1.m1.1" class="ltx_Math" alttext="P^{\top}G" display="inline"><semantics id="S3.SS2.p1.1.m1.1a"><mrow id="S3.SS2.p1.1.m1.1.1" xref="S3.SS2.p1.1.m1.1.1.cmml"><msup id="S3.SS2.p1.1.m1.1.1.2" xref="S3.SS2.p1.1.m1.1.1.2.cmml"><mi id="S3.SS2.p1.1.m1.1.1.2.2" xref="S3.SS2.p1.1.m1.1.1.2.2.cmml">P</mi><mo id="S3.SS2.p1.1.m1.1.1.2.3" xref="S3.SS2.p1.1.m1.1.1.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="S3.SS2.p1.1.m1.1.1.1" xref="S3.SS2.p1.1.m1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.1.m1.1.1.3" xref="S3.SS2.p1.1.m1.1.1.3.cmml">G</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.1.m1.1b"><apply id="S3.SS2.p1.1.m1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1"><times id="S3.SS2.p1.1.m1.1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1.1"></times><apply id="S3.SS2.p1.1.m1.1.1.2.cmml" xref="S3.SS2.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p1.1.m1.1.1.2.1.cmml" xref="S3.SS2.p1.1.m1.1.1.2">superscript</csymbol><ci id="S3.SS2.p1.1.m1.1.1.2.2.cmml" xref="S3.SS2.p1.1.m1.1.1.2.2">𝑃</ci><csymbol cd="latexml" id="S3.SS2.p1.1.m1.1.1.2.3.cmml" xref="S3.SS2.p1.1.m1.1.1.2.3">top</csymbol></apply><ci id="S3.SS2.p1.1.m1.1.1.3.cmml" xref="S3.SS2.p1.1.m1.1.1.3">𝐺</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.1.m1.1c">P^{\top}G</annotation></semantics></math> if <math id="S3.SS2.p1.2.m2.1" class="ltx_Math" alttext="m\leq n" display="inline"><semantics id="S3.SS2.p1.2.m2.1a"><mrow id="S3.SS2.p1.2.m2.1.1" xref="S3.SS2.p1.2.m2.1.1.cmml"><mi id="S3.SS2.p1.2.m2.1.1.2" xref="S3.SS2.p1.2.m2.1.1.2.cmml">m</mi><mo id="S3.SS2.p1.2.m2.1.1.1" xref="S3.SS2.p1.2.m2.1.1.1.cmml">≤</mo><mi id="S3.SS2.p1.2.m2.1.1.3" xref="S3.SS2.p1.2.m2.1.1.3.cmml">n</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.2.m2.1b"><apply id="S3.SS2.p1.2.m2.1.1.cmml" xref="S3.SS2.p1.2.m2.1.1"><leq id="S3.SS2.p1.2.m2.1.1.1.cmml" xref="S3.SS2.p1.2.m2.1.1.1"></leq><ci id="S3.SS2.p1.2.m2.1.1.2.cmml" xref="S3.SS2.p1.2.m2.1.1.2">𝑚</ci><ci id="S3.SS2.p1.2.m2.1.1.3.cmml" xref="S3.SS2.p1.2.m2.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.2.m2.1c">m\leq n</annotation></semantics></math> and <math id="S3.SS2.p1.3.m3.1" class="ltx_Math" alttext="GQ" display="inline"><semantics id="S3.SS2.p1.3.m3.1a"><mrow id="S3.SS2.p1.3.m3.1.1" xref="S3.SS2.p1.3.m3.1.1.cmml"><mi id="S3.SS2.p1.3.m3.1.1.2" xref="S3.SS2.p1.3.m3.1.1.2.cmml">G</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.3.m3.1.1.1" xref="S3.SS2.p1.3.m3.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.3.m3.1.1.3" xref="S3.SS2.p1.3.m3.1.1.3.cmml">Q</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.3.m3.1b"><apply id="S3.SS2.p1.3.m3.1.1.cmml" xref="S3.SS2.p1.3.m3.1.1"><times id="S3.SS2.p1.3.m3.1.1.1.cmml" xref="S3.SS2.p1.3.m3.1.1.1"></times><ci id="S3.SS2.p1.3.m3.1.1.2.cmml" xref="S3.SS2.p1.3.m3.1.1.2">𝐺</ci><ci id="S3.SS2.p1.3.m3.1.1.3.cmml" xref="S3.SS2.p1.3.m3.1.1.3">𝑄</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.3.m3.1c">GQ</annotation></semantics></math> otherwise. We want to achieve a separation between trainable weights and static weights, which we achieve by rewriting GaLore in terms of low-rank adaptors. Assume without loss of generality that <math id="S3.SS2.p1.4.m4.1" class="ltx_Math" alttext="m\leq n" display="inline"><semantics id="S3.SS2.p1.4.m4.1a"><mrow id="S3.SS2.p1.4.m4.1.1" xref="S3.SS2.p1.4.m4.1.1.cmml"><mi id="S3.SS2.p1.4.m4.1.1.2" xref="S3.SS2.p1.4.m4.1.1.2.cmml">m</mi><mo id="S3.SS2.p1.4.m4.1.1.1" xref="S3.SS2.p1.4.m4.1.1.1.cmml">≤</mo><mi id="S3.SS2.p1.4.m4.1.1.3" xref="S3.SS2.p1.4.m4.1.1.3.cmml">n</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.4.m4.1b"><apply id="S3.SS2.p1.4.m4.1.1.cmml" xref="S3.SS2.p1.4.m4.1.1"><leq id="S3.SS2.p1.4.m4.1.1.1.cmml" xref="S3.SS2.p1.4.m4.1.1.1"></leq><ci id="S3.SS2.p1.4.m4.1.1.2.cmml" xref="S3.SS2.p1.4.m4.1.1.2">𝑚</ci><ci id="S3.SS2.p1.4.m4.1.1.3.cmml" xref="S3.SS2.p1.4.m4.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.4.m4.1c">m\leq n</annotation></semantics></math>.
Using the fact that <math id="S3.SS2.p1.5.m5.1" class="ltx_Math" alttext="P_{t}" display="inline"><semantics id="S3.SS2.p1.5.m5.1a"><msub id="S3.SS2.p1.5.m5.1.1" xref="S3.SS2.p1.5.m5.1.1.cmml"><mi id="S3.SS2.p1.5.m5.1.1.2" xref="S3.SS2.p1.5.m5.1.1.2.cmml">P</mi><mi id="S3.SS2.p1.5.m5.1.1.3" xref="S3.SS2.p1.5.m5.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.5.m5.1b"><apply id="S3.SS2.p1.5.m5.1.1.cmml" xref="S3.SS2.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.5.m5.1.1.1.cmml" xref="S3.SS2.p1.5.m5.1.1">subscript</csymbol><ci id="S3.SS2.p1.5.m5.1.1.2.cmml" xref="S3.SS2.p1.5.m5.1.1.2">𝑃</ci><ci id="S3.SS2.p1.5.m5.1.1.3.cmml" xref="S3.SS2.p1.5.m5.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.5.m5.1c">P_{t}</annotation></semantics></math> is fixed in the interval <math id="S3.SS2.p1.6.m6.2" class="ltx_Math" alttext="[0,T]" display="inline"><semantics id="S3.SS2.p1.6.m6.2a"><mrow id="S3.SS2.p1.6.m6.2.3.2" xref="S3.SS2.p1.6.m6.2.3.1.cmml"><mo stretchy="false" id="S3.SS2.p1.6.m6.2.3.2.1" xref="S3.SS2.p1.6.m6.2.3.1.cmml">[</mo><mn id="S3.SS2.p1.6.m6.1.1" xref="S3.SS2.p1.6.m6.1.1.cmml">0</mn><mo id="S3.SS2.p1.6.m6.2.3.2.2" xref="S3.SS2.p1.6.m6.2.3.1.cmml">,</mo><mi id="S3.SS2.p1.6.m6.2.2" xref="S3.SS2.p1.6.m6.2.2.cmml">T</mi><mo stretchy="false" id="S3.SS2.p1.6.m6.2.3.2.3" xref="S3.SS2.p1.6.m6.2.3.1.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.6.m6.2b"><interval closure="closed" id="S3.SS2.p1.6.m6.2.3.1.cmml" xref="S3.SS2.p1.6.m6.2.3.2"><cn type="integer" id="S3.SS2.p1.6.m6.1.1.cmml" xref="S3.SS2.p1.6.m6.1.1">0</cn><ci id="S3.SS2.p1.6.m6.2.2.cmml" xref="S3.SS2.p1.6.m6.2.2">𝑇</ci></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.6.m6.2c">[0,T]</annotation></semantics></math> we have that</p>
<table id="A3.EGx2" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E3"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E3.m1.1" class="ltx_Math" alttext="\displaystyle W_{T}" display="inline"><semantics id="S3.E3.m1.1a"><msub id="S3.E3.m1.1.1" xref="S3.E3.m1.1.1.cmml"><mi id="S3.E3.m1.1.1.2" xref="S3.E3.m1.1.1.2.cmml">W</mi><mi id="S3.E3.m1.1.1.3" xref="S3.E3.m1.1.1.3.cmml">T</mi></msub><annotation-xml encoding="MathML-Content" id="S3.E3.m1.1b"><apply id="S3.E3.m1.1.1.cmml" xref="S3.E3.m1.1.1"><csymbol cd="ambiguous" id="S3.E3.m1.1.1.1.cmml" xref="S3.E3.m1.1.1">subscript</csymbol><ci id="S3.E3.m1.1.1.2.cmml" xref="S3.E3.m1.1.1.2">𝑊</ci><ci id="S3.E3.m1.1.1.3.cmml" xref="S3.E3.m1.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3.m1.1c">\displaystyle W_{T}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E3.m2.1" class="ltx_Math" alttext="\displaystyle=W_{0}+\eta\sum_{t=0}^{T-1}P\rho_{t}(P^{\top}G_{t})" display="inline"><semantics id="S3.E3.m2.1a"><mrow id="S3.E3.m2.1.1" xref="S3.E3.m2.1.1.cmml"><mi id="S3.E3.m2.1.1.3" xref="S3.E3.m2.1.1.3.cmml"></mi><mo id="S3.E3.m2.1.1.2" xref="S3.E3.m2.1.1.2.cmml">=</mo><mrow id="S3.E3.m2.1.1.1" xref="S3.E3.m2.1.1.1.cmml"><msub id="S3.E3.m2.1.1.1.3" xref="S3.E3.m2.1.1.1.3.cmml"><mi id="S3.E3.m2.1.1.1.3.2" xref="S3.E3.m2.1.1.1.3.2.cmml">W</mi><mn id="S3.E3.m2.1.1.1.3.3" xref="S3.E3.m2.1.1.1.3.3.cmml">0</mn></msub><mo id="S3.E3.m2.1.1.1.2" xref="S3.E3.m2.1.1.1.2.cmml">+</mo><mrow id="S3.E3.m2.1.1.1.1" xref="S3.E3.m2.1.1.1.1.cmml"><mi id="S3.E3.m2.1.1.1.1.3" xref="S3.E3.m2.1.1.1.1.3.cmml">η</mi><mo lspace="0em" rspace="0em" id="S3.E3.m2.1.1.1.1.2" xref="S3.E3.m2.1.1.1.1.2.cmml">​</mo><mrow id="S3.E3.m2.1.1.1.1.1" xref="S3.E3.m2.1.1.1.1.1.cmml"><mstyle displaystyle="true" id="S3.E3.m2.1.1.1.1.1.2" xref="S3.E3.m2.1.1.1.1.1.2.cmml"><munderover id="S3.E3.m2.1.1.1.1.1.2a" xref="S3.E3.m2.1.1.1.1.1.2.cmml"><mo movablelimits="false" id="S3.E3.m2.1.1.1.1.1.2.2.2" xref="S3.E3.m2.1.1.1.1.1.2.2.2.cmml">∑</mo><mrow id="S3.E3.m2.1.1.1.1.1.2.2.3" xref="S3.E3.m2.1.1.1.1.1.2.2.3.cmml"><mi id="S3.E3.m2.1.1.1.1.1.2.2.3.2" xref="S3.E3.m2.1.1.1.1.1.2.2.3.2.cmml">t</mi><mo id="S3.E3.m2.1.1.1.1.1.2.2.3.1" xref="S3.E3.m2.1.1.1.1.1.2.2.3.1.cmml">=</mo><mn id="S3.E3.m2.1.1.1.1.1.2.2.3.3" xref="S3.E3.m2.1.1.1.1.1.2.2.3.3.cmml">0</mn></mrow><mrow id="S3.E3.m2.1.1.1.1.1.2.3" xref="S3.E3.m2.1.1.1.1.1.2.3.cmml"><mi id="S3.E3.m2.1.1.1.1.1.2.3.2" xref="S3.E3.m2.1.1.1.1.1.2.3.2.cmml">T</mi><mo id="S3.E3.m2.1.1.1.1.1.2.3.1" xref="S3.E3.m2.1.1.1.1.1.2.3.1.cmml">−</mo><mn id="S3.E3.m2.1.1.1.1.1.2.3.3" xref="S3.E3.m2.1.1.1.1.1.2.3.3.cmml">1</mn></mrow></munderover></mstyle><mrow id="S3.E3.m2.1.1.1.1.1.1" xref="S3.E3.m2.1.1.1.1.1.1.cmml"><mi id="S3.E3.m2.1.1.1.1.1.1.3" xref="S3.E3.m2.1.1.1.1.1.1.3.cmml">P</mi><mo lspace="0em" rspace="0em" id="S3.E3.m2.1.1.1.1.1.1.2" xref="S3.E3.m2.1.1.1.1.1.1.2.cmml">​</mo><msub id="S3.E3.m2.1.1.1.1.1.1.4" xref="S3.E3.m2.1.1.1.1.1.1.4.cmml"><mi id="S3.E3.m2.1.1.1.1.1.1.4.2" xref="S3.E3.m2.1.1.1.1.1.1.4.2.cmml">ρ</mi><mi id="S3.E3.m2.1.1.1.1.1.1.4.3" xref="S3.E3.m2.1.1.1.1.1.1.4.3.cmml">t</mi></msub><mo lspace="0em" rspace="0em" id="S3.E3.m2.1.1.1.1.1.1.2a" xref="S3.E3.m2.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.E3.m2.1.1.1.1.1.1.1.1" xref="S3.E3.m2.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E3.m2.1.1.1.1.1.1.1.1.2" xref="S3.E3.m2.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E3.m2.1.1.1.1.1.1.1.1.1" xref="S3.E3.m2.1.1.1.1.1.1.1.1.1.cmml"><msup id="S3.E3.m2.1.1.1.1.1.1.1.1.1.2" xref="S3.E3.m2.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="S3.E3.m2.1.1.1.1.1.1.1.1.1.2.2" xref="S3.E3.m2.1.1.1.1.1.1.1.1.1.2.2.cmml">P</mi><mo id="S3.E3.m2.1.1.1.1.1.1.1.1.1.2.3" xref="S3.E3.m2.1.1.1.1.1.1.1.1.1.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="S3.E3.m2.1.1.1.1.1.1.1.1.1.1" xref="S3.E3.m2.1.1.1.1.1.1.1.1.1.1.cmml">​</mo><msub id="S3.E3.m2.1.1.1.1.1.1.1.1.1.3" xref="S3.E3.m2.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S3.E3.m2.1.1.1.1.1.1.1.1.1.3.2" xref="S3.E3.m2.1.1.1.1.1.1.1.1.1.3.2.cmml">G</mi><mi id="S3.E3.m2.1.1.1.1.1.1.1.1.1.3.3" xref="S3.E3.m2.1.1.1.1.1.1.1.1.1.3.3.cmml">t</mi></msub></mrow><mo stretchy="false" id="S3.E3.m2.1.1.1.1.1.1.1.1.3" xref="S3.E3.m2.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E3.m2.1b"><apply id="S3.E3.m2.1.1.cmml" xref="S3.E3.m2.1.1"><eq id="S3.E3.m2.1.1.2.cmml" xref="S3.E3.m2.1.1.2"></eq><csymbol cd="latexml" id="S3.E3.m2.1.1.3.cmml" xref="S3.E3.m2.1.1.3">absent</csymbol><apply id="S3.E3.m2.1.1.1.cmml" xref="S3.E3.m2.1.1.1"><plus id="S3.E3.m2.1.1.1.2.cmml" xref="S3.E3.m2.1.1.1.2"></plus><apply id="S3.E3.m2.1.1.1.3.cmml" xref="S3.E3.m2.1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m2.1.1.1.3.1.cmml" xref="S3.E3.m2.1.1.1.3">subscript</csymbol><ci id="S3.E3.m2.1.1.1.3.2.cmml" xref="S3.E3.m2.1.1.1.3.2">𝑊</ci><cn type="integer" id="S3.E3.m2.1.1.1.3.3.cmml" xref="S3.E3.m2.1.1.1.3.3">0</cn></apply><apply id="S3.E3.m2.1.1.1.1.cmml" xref="S3.E3.m2.1.1.1.1"><times id="S3.E3.m2.1.1.1.1.2.cmml" xref="S3.E3.m2.1.1.1.1.2"></times><ci id="S3.E3.m2.1.1.1.1.3.cmml" xref="S3.E3.m2.1.1.1.1.3">𝜂</ci><apply id="S3.E3.m2.1.1.1.1.1.cmml" xref="S3.E3.m2.1.1.1.1.1"><apply id="S3.E3.m2.1.1.1.1.1.2.cmml" xref="S3.E3.m2.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E3.m2.1.1.1.1.1.2.1.cmml" xref="S3.E3.m2.1.1.1.1.1.2">superscript</csymbol><apply id="S3.E3.m2.1.1.1.1.1.2.2.cmml" xref="S3.E3.m2.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E3.m2.1.1.1.1.1.2.2.1.cmml" xref="S3.E3.m2.1.1.1.1.1.2">subscript</csymbol><sum id="S3.E3.m2.1.1.1.1.1.2.2.2.cmml" xref="S3.E3.m2.1.1.1.1.1.2.2.2"></sum><apply id="S3.E3.m2.1.1.1.1.1.2.2.3.cmml" xref="S3.E3.m2.1.1.1.1.1.2.2.3"><eq id="S3.E3.m2.1.1.1.1.1.2.2.3.1.cmml" xref="S3.E3.m2.1.1.1.1.1.2.2.3.1"></eq><ci id="S3.E3.m2.1.1.1.1.1.2.2.3.2.cmml" xref="S3.E3.m2.1.1.1.1.1.2.2.3.2">𝑡</ci><cn type="integer" id="S3.E3.m2.1.1.1.1.1.2.2.3.3.cmml" xref="S3.E3.m2.1.1.1.1.1.2.2.3.3">0</cn></apply></apply><apply id="S3.E3.m2.1.1.1.1.1.2.3.cmml" xref="S3.E3.m2.1.1.1.1.1.2.3"><minus id="S3.E3.m2.1.1.1.1.1.2.3.1.cmml" xref="S3.E3.m2.1.1.1.1.1.2.3.1"></minus><ci id="S3.E3.m2.1.1.1.1.1.2.3.2.cmml" xref="S3.E3.m2.1.1.1.1.1.2.3.2">𝑇</ci><cn type="integer" id="S3.E3.m2.1.1.1.1.1.2.3.3.cmml" xref="S3.E3.m2.1.1.1.1.1.2.3.3">1</cn></apply></apply><apply id="S3.E3.m2.1.1.1.1.1.1.cmml" xref="S3.E3.m2.1.1.1.1.1.1"><times id="S3.E3.m2.1.1.1.1.1.1.2.cmml" xref="S3.E3.m2.1.1.1.1.1.1.2"></times><ci id="S3.E3.m2.1.1.1.1.1.1.3.cmml" xref="S3.E3.m2.1.1.1.1.1.1.3">𝑃</ci><apply id="S3.E3.m2.1.1.1.1.1.1.4.cmml" xref="S3.E3.m2.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.E3.m2.1.1.1.1.1.1.4.1.cmml" xref="S3.E3.m2.1.1.1.1.1.1.4">subscript</csymbol><ci id="S3.E3.m2.1.1.1.1.1.1.4.2.cmml" xref="S3.E3.m2.1.1.1.1.1.1.4.2">𝜌</ci><ci id="S3.E3.m2.1.1.1.1.1.1.4.3.cmml" xref="S3.E3.m2.1.1.1.1.1.1.4.3">𝑡</ci></apply><apply id="S3.E3.m2.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E3.m2.1.1.1.1.1.1.1.1"><times id="S3.E3.m2.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E3.m2.1.1.1.1.1.1.1.1.1.1"></times><apply id="S3.E3.m2.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E3.m2.1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E3.m2.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E3.m2.1.1.1.1.1.1.1.1.1.2">superscript</csymbol><ci id="S3.E3.m2.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E3.m2.1.1.1.1.1.1.1.1.1.2.2">𝑃</ci><csymbol cd="latexml" id="S3.E3.m2.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E3.m2.1.1.1.1.1.1.1.1.1.2.3">top</csymbol></apply><apply id="S3.E3.m2.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E3.m2.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m2.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E3.m2.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E3.m2.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E3.m2.1.1.1.1.1.1.1.1.1.3.2">𝐺</ci><ci id="S3.E3.m2.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.E3.m2.1.1.1.1.1.1.1.1.1.3.3">𝑡</ci></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3.m2.1c">\displaystyle=W_{0}+\eta\sum_{t=0}^{T-1}P\rho_{t}(P^{\top}G_{t})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
<tbody id="S3.E4"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E4.m1.1" class="ltx_Math" alttext="\displaystyle=W_{0}+\eta\underbrace{P\vphantom{\sum_{t=0}^{T-1}}}_{\in\mathbb{R}^{m\times r}}\!\underbrace{\sum_{t=0}^{T-1}\rho(P^{\top}G_{t})}_{B\in\mathbb{R}^{r\times n}}" display="inline"><semantics id="S3.E4.m1.1a"><mrow id="S3.E4.m1.1.2" xref="S3.E4.m1.1.2.cmml"><mi id="S3.E4.m1.1.2.2" xref="S3.E4.m1.1.2.2.cmml"></mi><mo id="S3.E4.m1.1.2.1" xref="S3.E4.m1.1.2.1.cmml">=</mo><mrow id="S3.E4.m1.1.2.3" xref="S3.E4.m1.1.2.3.cmml"><msub id="S3.E4.m1.1.2.3.2" xref="S3.E4.m1.1.2.3.2.cmml"><mi id="S3.E4.m1.1.2.3.2.2" xref="S3.E4.m1.1.2.3.2.2.cmml">W</mi><mn id="S3.E4.m1.1.2.3.2.3" xref="S3.E4.m1.1.2.3.2.3.cmml">0</mn></msub><mo id="S3.E4.m1.1.2.3.1" xref="S3.E4.m1.1.2.3.1.cmml">+</mo><mrow id="S3.E4.m1.1.2.3.3" xref="S3.E4.m1.1.2.3.3.cmml"><mi id="S3.E4.m1.1.2.3.3.2" xref="S3.E4.m1.1.2.3.3.2.cmml">η</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.1.2.3.3.1" xref="S3.E4.m1.1.2.3.3.1.cmml">​</mo><munder id="S3.E4.m1.1.2.3.3.3" xref="S3.E4.m1.1.2.3.3.3.cmml"><munder accentunder="true" id="S3.E4.m1.1.2.3.3.3.2" xref="S3.E4.m1.1.2.3.3.3.2.cmml"><mi id="S3.E4.m1.1.2.3.3.3.2.2" xref="S3.E4.m1.1.2.3.3.3.2.2.cmml">P</mi><mo id="S3.E4.m1.1.2.3.3.3.2.1" xref="S3.E4.m1.1.2.3.3.3.2.1.cmml">⏟</mo></munder><mrow id="S3.E4.m1.1.2.3.3.3.3" xref="S3.E4.m1.1.2.3.3.3.3.cmml"><mi id="S3.E4.m1.1.2.3.3.3.3.2" xref="S3.E4.m1.1.2.3.3.3.3.2.cmml"></mi><mo id="S3.E4.m1.1.2.3.3.3.3.1" xref="S3.E4.m1.1.2.3.3.3.3.1.cmml">∈</mo><msup id="S3.E4.m1.1.2.3.3.3.3.3" xref="S3.E4.m1.1.2.3.3.3.3.3.cmml"><mi id="S3.E4.m1.1.2.3.3.3.3.3.2" xref="S3.E4.m1.1.2.3.3.3.3.3.2.cmml">ℝ</mi><mrow id="S3.E4.m1.1.2.3.3.3.3.3.3" xref="S3.E4.m1.1.2.3.3.3.3.3.3.cmml"><mi id="S3.E4.m1.1.2.3.3.3.3.3.3.2" xref="S3.E4.m1.1.2.3.3.3.3.3.3.2.cmml">m</mi><mo lspace="0.222em" rspace="0.222em" id="S3.E4.m1.1.2.3.3.3.3.3.3.1" xref="S3.E4.m1.1.2.3.3.3.3.3.3.1.cmml">×</mo><mi id="S3.E4.m1.1.2.3.3.3.3.3.3.3" xref="S3.E4.m1.1.2.3.3.3.3.3.3.3.cmml">r</mi></mrow></msup></mrow></munder><mo lspace="0em" rspace="0em" id="S3.E4.m1.1.2.3.3.1a" xref="S3.E4.m1.1.2.3.3.1.cmml">​</mo><munder id="S3.E4.m1.1.2.3.3.4" xref="S3.E4.m1.1.2.3.3.4.cmml"><munder accentunder="true" id="S3.E4.m1.1.1" xref="S3.E4.m1.1.1.cmml"><mrow id="S3.E4.m1.1.1.1" xref="S3.E4.m1.1.1.1.cmml"><mstyle displaystyle="true" id="S3.E4.m1.1.1.1.2" xref="S3.E4.m1.1.1.1.2.cmml"><munderover id="S3.E4.m1.1.1.1.2a" xref="S3.E4.m1.1.1.1.2.cmml"><mo movablelimits="false" id="S3.E4.m1.1.1.1.2.2.2" xref="S3.E4.m1.1.1.1.2.2.2.cmml">∑</mo><mrow id="S3.E4.m1.1.1.1.2.2.3" xref="S3.E4.m1.1.1.1.2.2.3.cmml"><mi id="S3.E4.m1.1.1.1.2.2.3.2" xref="S3.E4.m1.1.1.1.2.2.3.2.cmml">t</mi><mo id="S3.E4.m1.1.1.1.2.2.3.1" xref="S3.E4.m1.1.1.1.2.2.3.1.cmml">=</mo><mn id="S3.E4.m1.1.1.1.2.2.3.3" xref="S3.E4.m1.1.1.1.2.2.3.3.cmml">0</mn></mrow><mrow id="S3.E4.m1.1.1.1.2.3" xref="S3.E4.m1.1.1.1.2.3.cmml"><mi id="S3.E4.m1.1.1.1.2.3.2" xref="S3.E4.m1.1.1.1.2.3.2.cmml">T</mi><mo id="S3.E4.m1.1.1.1.2.3.1" xref="S3.E4.m1.1.1.1.2.3.1.cmml">−</mo><mn id="S3.E4.m1.1.1.1.2.3.3" xref="S3.E4.m1.1.1.1.2.3.3.cmml">1</mn></mrow></munderover></mstyle><mrow id="S3.E4.m1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.cmml"><mi id="S3.E4.m1.1.1.1.1.3" xref="S3.E4.m1.1.1.1.1.3.cmml">ρ</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.1.1.1.1.2" xref="S3.E4.m1.1.1.1.1.2.cmml">​</mo><mrow id="S3.E4.m1.1.1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E4.m1.1.1.1.1.1.1.2" xref="S3.E4.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E4.m1.1.1.1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.1.1.1.cmml"><msup id="S3.E4.m1.1.1.1.1.1.1.1.2" xref="S3.E4.m1.1.1.1.1.1.1.1.2.cmml"><mi id="S3.E4.m1.1.1.1.1.1.1.1.2.2" xref="S3.E4.m1.1.1.1.1.1.1.1.2.2.cmml">P</mi><mo id="S3.E4.m1.1.1.1.1.1.1.1.2.3" xref="S3.E4.m1.1.1.1.1.1.1.1.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="S3.E4.m1.1.1.1.1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.1.1.1.1.cmml">​</mo><msub id="S3.E4.m1.1.1.1.1.1.1.1.3" xref="S3.E4.m1.1.1.1.1.1.1.1.3.cmml"><mi id="S3.E4.m1.1.1.1.1.1.1.1.3.2" xref="S3.E4.m1.1.1.1.1.1.1.1.3.2.cmml">G</mi><mi id="S3.E4.m1.1.1.1.1.1.1.1.3.3" xref="S3.E4.m1.1.1.1.1.1.1.1.3.3.cmml">t</mi></msub></mrow><mo stretchy="false" id="S3.E4.m1.1.1.1.1.1.1.3" xref="S3.E4.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo id="S3.E4.m1.1.1.2" xref="S3.E4.m1.1.1.2.cmml">⏟</mo></munder><mrow id="S3.E4.m1.1.2.3.3.4.2" xref="S3.E4.m1.1.2.3.3.4.2.cmml"><mi id="S3.E4.m1.1.2.3.3.4.2.2" xref="S3.E4.m1.1.2.3.3.4.2.2.cmml">B</mi><mo id="S3.E4.m1.1.2.3.3.4.2.1" xref="S3.E4.m1.1.2.3.3.4.2.1.cmml">∈</mo><msup id="S3.E4.m1.1.2.3.3.4.2.3" xref="S3.E4.m1.1.2.3.3.4.2.3.cmml"><mi id="S3.E4.m1.1.2.3.3.4.2.3.2" xref="S3.E4.m1.1.2.3.3.4.2.3.2.cmml">ℝ</mi><mrow id="S3.E4.m1.1.2.3.3.4.2.3.3" xref="S3.E4.m1.1.2.3.3.4.2.3.3.cmml"><mi id="S3.E4.m1.1.2.3.3.4.2.3.3.2" xref="S3.E4.m1.1.2.3.3.4.2.3.3.2.cmml">r</mi><mo lspace="0.222em" rspace="0.222em" id="S3.E4.m1.1.2.3.3.4.2.3.3.1" xref="S3.E4.m1.1.2.3.3.4.2.3.3.1.cmml">×</mo><mi id="S3.E4.m1.1.2.3.3.4.2.3.3.3" xref="S3.E4.m1.1.2.3.3.4.2.3.3.3.cmml">n</mi></mrow></msup></mrow></munder></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E4.m1.1b"><apply id="S3.E4.m1.1.2.cmml" xref="S3.E4.m1.1.2"><eq id="S3.E4.m1.1.2.1.cmml" xref="S3.E4.m1.1.2.1"></eq><csymbol cd="latexml" id="S3.E4.m1.1.2.2.cmml" xref="S3.E4.m1.1.2.2">absent</csymbol><apply id="S3.E4.m1.1.2.3.cmml" xref="S3.E4.m1.1.2.3"><plus id="S3.E4.m1.1.2.3.1.cmml" xref="S3.E4.m1.1.2.3.1"></plus><apply id="S3.E4.m1.1.2.3.2.cmml" xref="S3.E4.m1.1.2.3.2"><csymbol cd="ambiguous" id="S3.E4.m1.1.2.3.2.1.cmml" xref="S3.E4.m1.1.2.3.2">subscript</csymbol><ci id="S3.E4.m1.1.2.3.2.2.cmml" xref="S3.E4.m1.1.2.3.2.2">𝑊</ci><cn type="integer" id="S3.E4.m1.1.2.3.2.3.cmml" xref="S3.E4.m1.1.2.3.2.3">0</cn></apply><apply id="S3.E4.m1.1.2.3.3.cmml" xref="S3.E4.m1.1.2.3.3"><times id="S3.E4.m1.1.2.3.3.1.cmml" xref="S3.E4.m1.1.2.3.3.1"></times><ci id="S3.E4.m1.1.2.3.3.2.cmml" xref="S3.E4.m1.1.2.3.3.2">𝜂</ci><apply id="S3.E4.m1.1.2.3.3.3.cmml" xref="S3.E4.m1.1.2.3.3.3"><csymbol cd="ambiguous" id="S3.E4.m1.1.2.3.3.3.1.cmml" xref="S3.E4.m1.1.2.3.3.3">subscript</csymbol><apply id="S3.E4.m1.1.2.3.3.3.2.cmml" xref="S3.E4.m1.1.2.3.3.3.2"><ci id="S3.E4.m1.1.2.3.3.3.2.1.cmml" xref="S3.E4.m1.1.2.3.3.3.2.1">⏟</ci><ci id="S3.E4.m1.1.2.3.3.3.2.2.cmml" xref="S3.E4.m1.1.2.3.3.3.2.2">𝑃</ci></apply><apply id="S3.E4.m1.1.2.3.3.3.3.cmml" xref="S3.E4.m1.1.2.3.3.3.3"><in id="S3.E4.m1.1.2.3.3.3.3.1.cmml" xref="S3.E4.m1.1.2.3.3.3.3.1"></in><csymbol cd="latexml" id="S3.E4.m1.1.2.3.3.3.3.2.cmml" xref="S3.E4.m1.1.2.3.3.3.3.2">absent</csymbol><apply id="S3.E4.m1.1.2.3.3.3.3.3.cmml" xref="S3.E4.m1.1.2.3.3.3.3.3"><csymbol cd="ambiguous" id="S3.E4.m1.1.2.3.3.3.3.3.1.cmml" xref="S3.E4.m1.1.2.3.3.3.3.3">superscript</csymbol><ci id="S3.E4.m1.1.2.3.3.3.3.3.2.cmml" xref="S3.E4.m1.1.2.3.3.3.3.3.2">ℝ</ci><apply id="S3.E4.m1.1.2.3.3.3.3.3.3.cmml" xref="S3.E4.m1.1.2.3.3.3.3.3.3"><times id="S3.E4.m1.1.2.3.3.3.3.3.3.1.cmml" xref="S3.E4.m1.1.2.3.3.3.3.3.3.1"></times><ci id="S3.E4.m1.1.2.3.3.3.3.3.3.2.cmml" xref="S3.E4.m1.1.2.3.3.3.3.3.3.2">𝑚</ci><ci id="S3.E4.m1.1.2.3.3.3.3.3.3.3.cmml" xref="S3.E4.m1.1.2.3.3.3.3.3.3.3">𝑟</ci></apply></apply></apply></apply><apply id="S3.E4.m1.1.2.3.3.4.cmml" xref="S3.E4.m1.1.2.3.3.4"><csymbol cd="ambiguous" id="S3.E4.m1.1.2.3.3.4.1.cmml" xref="S3.E4.m1.1.2.3.3.4">subscript</csymbol><apply id="S3.E4.m1.1.1.cmml" xref="S3.E4.m1.1.1"><ci id="S3.E4.m1.1.1.2.cmml" xref="S3.E4.m1.1.1.2">⏟</ci><apply id="S3.E4.m1.1.1.1.cmml" xref="S3.E4.m1.1.1.1"><apply id="S3.E4.m1.1.1.1.2.cmml" xref="S3.E4.m1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.1.2.1.cmml" xref="S3.E4.m1.1.1.1.2">superscript</csymbol><apply id="S3.E4.m1.1.1.1.2.2.cmml" xref="S3.E4.m1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.1.2.2.1.cmml" xref="S3.E4.m1.1.1.1.2">subscript</csymbol><sum id="S3.E4.m1.1.1.1.2.2.2.cmml" xref="S3.E4.m1.1.1.1.2.2.2"></sum><apply id="S3.E4.m1.1.1.1.2.2.3.cmml" xref="S3.E4.m1.1.1.1.2.2.3"><eq id="S3.E4.m1.1.1.1.2.2.3.1.cmml" xref="S3.E4.m1.1.1.1.2.2.3.1"></eq><ci id="S3.E4.m1.1.1.1.2.2.3.2.cmml" xref="S3.E4.m1.1.1.1.2.2.3.2">𝑡</ci><cn type="integer" id="S3.E4.m1.1.1.1.2.2.3.3.cmml" xref="S3.E4.m1.1.1.1.2.2.3.3">0</cn></apply></apply><apply id="S3.E4.m1.1.1.1.2.3.cmml" xref="S3.E4.m1.1.1.1.2.3"><minus id="S3.E4.m1.1.1.1.2.3.1.cmml" xref="S3.E4.m1.1.1.1.2.3.1"></minus><ci id="S3.E4.m1.1.1.1.2.3.2.cmml" xref="S3.E4.m1.1.1.1.2.3.2">𝑇</ci><cn type="integer" id="S3.E4.m1.1.1.1.2.3.3.cmml" xref="S3.E4.m1.1.1.1.2.3.3">1</cn></apply></apply><apply id="S3.E4.m1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1"><times id="S3.E4.m1.1.1.1.1.2.cmml" xref="S3.E4.m1.1.1.1.1.2"></times><ci id="S3.E4.m1.1.1.1.1.3.cmml" xref="S3.E4.m1.1.1.1.1.3">𝜌</ci><apply id="S3.E4.m1.1.1.1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1"><times id="S3.E4.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.1"></times><apply id="S3.E4.m1.1.1.1.1.1.1.1.2.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.2">superscript</csymbol><ci id="S3.E4.m1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.2.2">𝑃</ci><csymbol cd="latexml" id="S3.E4.m1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.2.3">top</csymbol></apply><apply id="S3.E4.m1.1.1.1.1.1.1.1.3.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E4.m1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.3.2">𝐺</ci><ci id="S3.E4.m1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.3.3">𝑡</ci></apply></apply></apply></apply></apply><apply id="S3.E4.m1.1.2.3.3.4.2.cmml" xref="S3.E4.m1.1.2.3.3.4.2"><in id="S3.E4.m1.1.2.3.3.4.2.1.cmml" xref="S3.E4.m1.1.2.3.3.4.2.1"></in><ci id="S3.E4.m1.1.2.3.3.4.2.2.cmml" xref="S3.E4.m1.1.2.3.3.4.2.2">𝐵</ci><apply id="S3.E4.m1.1.2.3.3.4.2.3.cmml" xref="S3.E4.m1.1.2.3.3.4.2.3"><csymbol cd="ambiguous" id="S3.E4.m1.1.2.3.3.4.2.3.1.cmml" xref="S3.E4.m1.1.2.3.3.4.2.3">superscript</csymbol><ci id="S3.E4.m1.1.2.3.3.4.2.3.2.cmml" xref="S3.E4.m1.1.2.3.3.4.2.3.2">ℝ</ci><apply id="S3.E4.m1.1.2.3.3.4.2.3.3.cmml" xref="S3.E4.m1.1.2.3.3.4.2.3.3"><times id="S3.E4.m1.1.2.3.3.4.2.3.3.1.cmml" xref="S3.E4.m1.1.2.3.3.4.2.3.3.1"></times><ci id="S3.E4.m1.1.2.3.3.4.2.3.3.2.cmml" xref="S3.E4.m1.1.2.3.3.4.2.3.3.2">𝑟</ci><ci id="S3.E4.m1.1.2.3.3.4.2.3.3.3.cmml" xref="S3.E4.m1.1.2.3.3.4.2.3.3.3">𝑛</ci></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E4.m1.1c">\displaystyle=W_{0}+\eta\underbrace{P\vphantom{\sum_{t=0}^{T-1}}}_{\in\mathbb{R}^{m\times r}}\!\underbrace{\sum_{t=0}^{T-1}\rho(P^{\top}G_{t})}_{B\in\mathbb{R}^{r\times n}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
<p id="S3.SS2.p1.12" class="ltx_p">From (4) it is clear that we can keep track of low-rank updates using rank-<math id="S3.SS2.p1.7.m1.1" class="ltx_Math" alttext="r" display="inline"><semantics id="S3.SS2.p1.7.m1.1a"><mi id="S3.SS2.p1.7.m1.1.1" xref="S3.SS2.p1.7.m1.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.7.m1.1b"><ci id="S3.SS2.p1.7.m1.1.1.cmml" xref="S3.SS2.p1.7.m1.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.7.m1.1c">r</annotation></semantics></math> low-rank adaptors. We note that in the interval <math id="S3.SS2.p1.8.m2.2" class="ltx_Math" alttext="[0,T]" display="inline"><semantics id="S3.SS2.p1.8.m2.2a"><mrow id="S3.SS2.p1.8.m2.2.3.2" xref="S3.SS2.p1.8.m2.2.3.1.cmml"><mo stretchy="false" id="S3.SS2.p1.8.m2.2.3.2.1" xref="S3.SS2.p1.8.m2.2.3.1.cmml">[</mo><mn id="S3.SS2.p1.8.m2.1.1" xref="S3.SS2.p1.8.m2.1.1.cmml">0</mn><mo id="S3.SS2.p1.8.m2.2.3.2.2" xref="S3.SS2.p1.8.m2.2.3.1.cmml">,</mo><mi id="S3.SS2.p1.8.m2.2.2" xref="S3.SS2.p1.8.m2.2.2.cmml">T</mi><mo stretchy="false" id="S3.SS2.p1.8.m2.2.3.2.3" xref="S3.SS2.p1.8.m2.2.3.1.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.8.m2.2b"><interval closure="closed" id="S3.SS2.p1.8.m2.2.3.1.cmml" xref="S3.SS2.p1.8.m2.2.3.2"><cn type="integer" id="S3.SS2.p1.8.m2.1.1.cmml" xref="S3.SS2.p1.8.m2.1.1">0</cn><ci id="S3.SS2.p1.8.m2.2.2.cmml" xref="S3.SS2.p1.8.m2.2.2">𝑇</ci></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.8.m2.2c">[0,T]</annotation></semantics></math> only <math id="S3.SS2.p1.9.m3.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S3.SS2.p1.9.m3.1a"><mi id="S3.SS2.p1.9.m3.1.1" xref="S3.SS2.p1.9.m3.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.9.m3.1b"><ci id="S3.SS2.p1.9.m3.1.1.cmml" xref="S3.SS2.p1.9.m3.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.9.m3.1c">B</annotation></semantics></math> is updated, creating the desired separation. If implemented directly, we would need to compute the gradient with respect to <math id="S3.SS2.p1.10.m4.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S3.SS2.p1.10.m4.1a"><mi id="S3.SS2.p1.10.m4.1.1" xref="S3.SS2.p1.10.m4.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.10.m4.1b"><ci id="S3.SS2.p1.10.m4.1.1.cmml" xref="S3.SS2.p1.10.m4.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.10.m4.1c">W</annotation></semantics></math> and then project it down using <math id="S3.SS2.p1.11.m5.1" class="ltx_Math" alttext="P^{\top}G_{t}" display="inline"><semantics id="S3.SS2.p1.11.m5.1a"><mrow id="S3.SS2.p1.11.m5.1.1" xref="S3.SS2.p1.11.m5.1.1.cmml"><msup id="S3.SS2.p1.11.m5.1.1.2" xref="S3.SS2.p1.11.m5.1.1.2.cmml"><mi id="S3.SS2.p1.11.m5.1.1.2.2" xref="S3.SS2.p1.11.m5.1.1.2.2.cmml">P</mi><mo id="S3.SS2.p1.11.m5.1.1.2.3" xref="S3.SS2.p1.11.m5.1.1.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="S3.SS2.p1.11.m5.1.1.1" xref="S3.SS2.p1.11.m5.1.1.1.cmml">​</mo><msub id="S3.SS2.p1.11.m5.1.1.3" xref="S3.SS2.p1.11.m5.1.1.3.cmml"><mi id="S3.SS2.p1.11.m5.1.1.3.2" xref="S3.SS2.p1.11.m5.1.1.3.2.cmml">G</mi><mi id="S3.SS2.p1.11.m5.1.1.3.3" xref="S3.SS2.p1.11.m5.1.1.3.3.cmml">t</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.11.m5.1b"><apply id="S3.SS2.p1.11.m5.1.1.cmml" xref="S3.SS2.p1.11.m5.1.1"><times id="S3.SS2.p1.11.m5.1.1.1.cmml" xref="S3.SS2.p1.11.m5.1.1.1"></times><apply id="S3.SS2.p1.11.m5.1.1.2.cmml" xref="S3.SS2.p1.11.m5.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p1.11.m5.1.1.2.1.cmml" xref="S3.SS2.p1.11.m5.1.1.2">superscript</csymbol><ci id="S3.SS2.p1.11.m5.1.1.2.2.cmml" xref="S3.SS2.p1.11.m5.1.1.2.2">𝑃</ci><csymbol cd="latexml" id="S3.SS2.p1.11.m5.1.1.2.3.cmml" xref="S3.SS2.p1.11.m5.1.1.2.3">top</csymbol></apply><apply id="S3.SS2.p1.11.m5.1.1.3.cmml" xref="S3.SS2.p1.11.m5.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p1.11.m5.1.1.3.1.cmml" xref="S3.SS2.p1.11.m5.1.1.3">subscript</csymbol><ci id="S3.SS2.p1.11.m5.1.1.3.2.cmml" xref="S3.SS2.p1.11.m5.1.1.3.2">𝐺</ci><ci id="S3.SS2.p1.11.m5.1.1.3.3.cmml" xref="S3.SS2.p1.11.m5.1.1.3.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.11.m5.1c">P^{\top}G_{t}</annotation></semantics></math>.
We find that this step is unnecessary; it is sufficient to train <math id="S3.SS2.p1.12.m6.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S3.SS2.p1.12.m6.1a"><mi id="S3.SS2.p1.12.m6.1.1" xref="S3.SS2.p1.12.m6.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.12.m6.1b"><ci id="S3.SS2.p1.12.m6.1.1.cmml" xref="S3.SS2.p1.12.m6.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.12.m6.1c">B</annotation></semantics></math> using standard gradient descent.</p>
</div>
<div id="S3.SS2.p2" class="ltx_para">
<p id="S3.SS2.p2.19" class="ltx_p">We now show that training the <math id="S3.SS2.p2.1.m1.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S3.SS2.p2.1.m1.1a"><mi id="S3.SS2.p2.1.m1.1.1" xref="S3.SS2.p2.1.m1.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.1.m1.1b"><ci id="S3.SS2.p2.1.m1.1.1.cmml" xref="S3.SS2.p2.1.m1.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.1.m1.1c">B</annotation></semantics></math> matrix using gradient descent is equivalent to training w.r.t. <math id="S3.SS2.p2.2.m2.1" class="ltx_Math" alttext="W_{t}" display="inline"><semantics id="S3.SS2.p2.2.m2.1a"><msub id="S3.SS2.p2.2.m2.1.1" xref="S3.SS2.p2.2.m2.1.1.cmml"><mi id="S3.SS2.p2.2.m2.1.1.2" xref="S3.SS2.p2.2.m2.1.1.2.cmml">W</mi><mi id="S3.SS2.p2.2.m2.1.1.3" xref="S3.SS2.p2.2.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.2.m2.1b"><apply id="S3.SS2.p2.2.m2.1.1.cmml" xref="S3.SS2.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.2.m2.1.1.1.cmml" xref="S3.SS2.p2.2.m2.1.1">subscript</csymbol><ci id="S3.SS2.p2.2.m2.1.1.2.cmml" xref="S3.SS2.p2.2.m2.1.1.2">𝑊</ci><ci id="S3.SS2.p2.2.m2.1.1.3.cmml" xref="S3.SS2.p2.2.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.2.m2.1c">W_{t}</annotation></semantics></math> as in definition <a href="#S3.Thmdefinition1" title="Definition 3.1 (Gradient Low-rank Projection, def. 3.4 in [44]). ‣ 3.1 Background: GaLore ‣ 3 Efficient Pretraining With LoQT ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a> 
Let <math id="S3.SS2.p2.3.m3.1" class="ltx_Math" alttext="G^{W}" display="inline"><semantics id="S3.SS2.p2.3.m3.1a"><msup id="S3.SS2.p2.3.m3.1.1" xref="S3.SS2.p2.3.m3.1.1.cmml"><mi id="S3.SS2.p2.3.m3.1.1.2" xref="S3.SS2.p2.3.m3.1.1.2.cmml">G</mi><mi id="S3.SS2.p2.3.m3.1.1.3" xref="S3.SS2.p2.3.m3.1.1.3.cmml">W</mi></msup><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.3.m3.1b"><apply id="S3.SS2.p2.3.m3.1.1.cmml" xref="S3.SS2.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.3.m3.1.1.1.cmml" xref="S3.SS2.p2.3.m3.1.1">superscript</csymbol><ci id="S3.SS2.p2.3.m3.1.1.2.cmml" xref="S3.SS2.p2.3.m3.1.1.2">𝐺</ci><ci id="S3.SS2.p2.3.m3.1.1.3.cmml" xref="S3.SS2.p2.3.m3.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.3.m3.1c">G^{W}</annotation></semantics></math> indicate the gradient of the loss with respect to <math id="S3.SS2.p2.4.m4.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S3.SS2.p2.4.m4.1a"><mi id="S3.SS2.p2.4.m4.1.1" xref="S3.SS2.p2.4.m4.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.4.m4.1b"><ci id="S3.SS2.p2.4.m4.1.1.cmml" xref="S3.SS2.p2.4.m4.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.4.m4.1c">W</annotation></semantics></math>, and <math id="S3.SS2.p2.5.m5.1" class="ltx_Math" alttext="G^{B}" display="inline"><semantics id="S3.SS2.p2.5.m5.1a"><msup id="S3.SS2.p2.5.m5.1.1" xref="S3.SS2.p2.5.m5.1.1.cmml"><mi id="S3.SS2.p2.5.m5.1.1.2" xref="S3.SS2.p2.5.m5.1.1.2.cmml">G</mi><mi id="S3.SS2.p2.5.m5.1.1.3" xref="S3.SS2.p2.5.m5.1.1.3.cmml">B</mi></msup><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.5.m5.1b"><apply id="S3.SS2.p2.5.m5.1.1.cmml" xref="S3.SS2.p2.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.5.m5.1.1.1.cmml" xref="S3.SS2.p2.5.m5.1.1">superscript</csymbol><ci id="S3.SS2.p2.5.m5.1.1.2.cmml" xref="S3.SS2.p2.5.m5.1.1.2">𝐺</ci><ci id="S3.SS2.p2.5.m5.1.1.3.cmml" xref="S3.SS2.p2.5.m5.1.1.3">𝐵</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.5.m5.1c">G^{B}</annotation></semantics></math> for the gradient of the loss with respect to <math id="S3.SS2.p2.6.m6.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S3.SS2.p2.6.m6.1a"><mi id="S3.SS2.p2.6.m6.1.1" xref="S3.SS2.p2.6.m6.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.6.m6.1b"><ci id="S3.SS2.p2.6.m6.1.1.cmml" xref="S3.SS2.p2.6.m6.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.6.m6.1c">B</annotation></semantics></math>. Given a weight matrix <math id="S3.SS2.p2.7.m7.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S3.SS2.p2.7.m7.1a"><mi id="S3.SS2.p2.7.m7.1.1" xref="S3.SS2.p2.7.m7.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.7.m7.1b"><ci id="S3.SS2.p2.7.m7.1.1.cmml" xref="S3.SS2.p2.7.m7.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.7.m7.1c">W</annotation></semantics></math>, an factor <math id="S3.SS2.p2.8.m8.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S3.SS2.p2.8.m8.1a"><mi id="S3.SS2.p2.8.m8.1.1" xref="S3.SS2.p2.8.m8.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.8.m8.1b"><ci id="S3.SS2.p2.8.m8.1.1.cmml" xref="S3.SS2.p2.8.m8.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.8.m8.1c">P</annotation></semantics></math> and a matrix <math id="S3.SS2.p2.9.m9.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S3.SS2.p2.9.m9.1a"><mi id="S3.SS2.p2.9.m9.1.1" xref="S3.SS2.p2.9.m9.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.9.m9.1b"><ci id="S3.SS2.p2.9.m9.1.1.cmml" xref="S3.SS2.p2.9.m9.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.9.m9.1c">B</annotation></semantics></math>, when computing the forward pass <math id="S3.SS2.p2.10.m10.1" class="ltx_Math" alttext="y=xW+xPB" display="inline"><semantics id="S3.SS2.p2.10.m10.1a"><mrow id="S3.SS2.p2.10.m10.1.1" xref="S3.SS2.p2.10.m10.1.1.cmml"><mi id="S3.SS2.p2.10.m10.1.1.2" xref="S3.SS2.p2.10.m10.1.1.2.cmml">y</mi><mo id="S3.SS2.p2.10.m10.1.1.1" xref="S3.SS2.p2.10.m10.1.1.1.cmml">=</mo><mrow id="S3.SS2.p2.10.m10.1.1.3" xref="S3.SS2.p2.10.m10.1.1.3.cmml"><mrow id="S3.SS2.p2.10.m10.1.1.3.2" xref="S3.SS2.p2.10.m10.1.1.3.2.cmml"><mi id="S3.SS2.p2.10.m10.1.1.3.2.2" xref="S3.SS2.p2.10.m10.1.1.3.2.2.cmml">x</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.10.m10.1.1.3.2.1" xref="S3.SS2.p2.10.m10.1.1.3.2.1.cmml">​</mo><mi id="S3.SS2.p2.10.m10.1.1.3.2.3" xref="S3.SS2.p2.10.m10.1.1.3.2.3.cmml">W</mi></mrow><mo id="S3.SS2.p2.10.m10.1.1.3.1" xref="S3.SS2.p2.10.m10.1.1.3.1.cmml">+</mo><mrow id="S3.SS2.p2.10.m10.1.1.3.3" xref="S3.SS2.p2.10.m10.1.1.3.3.cmml"><mi id="S3.SS2.p2.10.m10.1.1.3.3.2" xref="S3.SS2.p2.10.m10.1.1.3.3.2.cmml">x</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.10.m10.1.1.3.3.1" xref="S3.SS2.p2.10.m10.1.1.3.3.1.cmml">​</mo><mi id="S3.SS2.p2.10.m10.1.1.3.3.3" xref="S3.SS2.p2.10.m10.1.1.3.3.3.cmml">P</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.10.m10.1.1.3.3.1a" xref="S3.SS2.p2.10.m10.1.1.3.3.1.cmml">​</mo><mi id="S3.SS2.p2.10.m10.1.1.3.3.4" xref="S3.SS2.p2.10.m10.1.1.3.3.4.cmml">B</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.10.m10.1b"><apply id="S3.SS2.p2.10.m10.1.1.cmml" xref="S3.SS2.p2.10.m10.1.1"><eq id="S3.SS2.p2.10.m10.1.1.1.cmml" xref="S3.SS2.p2.10.m10.1.1.1"></eq><ci id="S3.SS2.p2.10.m10.1.1.2.cmml" xref="S3.SS2.p2.10.m10.1.1.2">𝑦</ci><apply id="S3.SS2.p2.10.m10.1.1.3.cmml" xref="S3.SS2.p2.10.m10.1.1.3"><plus id="S3.SS2.p2.10.m10.1.1.3.1.cmml" xref="S3.SS2.p2.10.m10.1.1.3.1"></plus><apply id="S3.SS2.p2.10.m10.1.1.3.2.cmml" xref="S3.SS2.p2.10.m10.1.1.3.2"><times id="S3.SS2.p2.10.m10.1.1.3.2.1.cmml" xref="S3.SS2.p2.10.m10.1.1.3.2.1"></times><ci id="S3.SS2.p2.10.m10.1.1.3.2.2.cmml" xref="S3.SS2.p2.10.m10.1.1.3.2.2">𝑥</ci><ci id="S3.SS2.p2.10.m10.1.1.3.2.3.cmml" xref="S3.SS2.p2.10.m10.1.1.3.2.3">𝑊</ci></apply><apply id="S3.SS2.p2.10.m10.1.1.3.3.cmml" xref="S3.SS2.p2.10.m10.1.1.3.3"><times id="S3.SS2.p2.10.m10.1.1.3.3.1.cmml" xref="S3.SS2.p2.10.m10.1.1.3.3.1"></times><ci id="S3.SS2.p2.10.m10.1.1.3.3.2.cmml" xref="S3.SS2.p2.10.m10.1.1.3.3.2">𝑥</ci><ci id="S3.SS2.p2.10.m10.1.1.3.3.3.cmml" xref="S3.SS2.p2.10.m10.1.1.3.3.3">𝑃</ci><ci id="S3.SS2.p2.10.m10.1.1.3.3.4.cmml" xref="S3.SS2.p2.10.m10.1.1.3.3.4">𝐵</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.10.m10.1c">y=xW+xPB</annotation></semantics></math>, the gradient of a loss function <math id="S3.SS2.p2.11.m11.1" class="ltx_Math" alttext="\mathcal{L}" display="inline"><semantics id="S3.SS2.p2.11.m11.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p2.11.m11.1.1" xref="S3.SS2.p2.11.m11.1.1.cmml">ℒ</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.11.m11.1b"><ci id="S3.SS2.p2.11.m11.1.1.cmml" xref="S3.SS2.p2.11.m11.1.1">ℒ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.11.m11.1c">\mathcal{L}</annotation></semantics></math> w.r.t. <math id="S3.SS2.p2.12.m12.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S3.SS2.p2.12.m12.1a"><mi id="S3.SS2.p2.12.m12.1.1" xref="S3.SS2.p2.12.m12.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.12.m12.1b"><ci id="S3.SS2.p2.12.m12.1.1.cmml" xref="S3.SS2.p2.12.m12.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.12.m12.1c">B</annotation></semantics></math> is <math id="S3.SS2.p2.13.m13.1" class="ltx_Math" alttext="G^{B}=P^{\top}G^{W}" display="inline"><semantics id="S3.SS2.p2.13.m13.1a"><mrow id="S3.SS2.p2.13.m13.1.1" xref="S3.SS2.p2.13.m13.1.1.cmml"><msup id="S3.SS2.p2.13.m13.1.1.2" xref="S3.SS2.p2.13.m13.1.1.2.cmml"><mi id="S3.SS2.p2.13.m13.1.1.2.2" xref="S3.SS2.p2.13.m13.1.1.2.2.cmml">G</mi><mi id="S3.SS2.p2.13.m13.1.1.2.3" xref="S3.SS2.p2.13.m13.1.1.2.3.cmml">B</mi></msup><mo id="S3.SS2.p2.13.m13.1.1.1" xref="S3.SS2.p2.13.m13.1.1.1.cmml">=</mo><mrow id="S3.SS2.p2.13.m13.1.1.3" xref="S3.SS2.p2.13.m13.1.1.3.cmml"><msup id="S3.SS2.p2.13.m13.1.1.3.2" xref="S3.SS2.p2.13.m13.1.1.3.2.cmml"><mi id="S3.SS2.p2.13.m13.1.1.3.2.2" xref="S3.SS2.p2.13.m13.1.1.3.2.2.cmml">P</mi><mo id="S3.SS2.p2.13.m13.1.1.3.2.3" xref="S3.SS2.p2.13.m13.1.1.3.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="S3.SS2.p2.13.m13.1.1.3.1" xref="S3.SS2.p2.13.m13.1.1.3.1.cmml">​</mo><msup id="S3.SS2.p2.13.m13.1.1.3.3" xref="S3.SS2.p2.13.m13.1.1.3.3.cmml"><mi id="S3.SS2.p2.13.m13.1.1.3.3.2" xref="S3.SS2.p2.13.m13.1.1.3.3.2.cmml">G</mi><mi id="S3.SS2.p2.13.m13.1.1.3.3.3" xref="S3.SS2.p2.13.m13.1.1.3.3.3.cmml">W</mi></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.13.m13.1b"><apply id="S3.SS2.p2.13.m13.1.1.cmml" xref="S3.SS2.p2.13.m13.1.1"><eq id="S3.SS2.p2.13.m13.1.1.1.cmml" xref="S3.SS2.p2.13.m13.1.1.1"></eq><apply id="S3.SS2.p2.13.m13.1.1.2.cmml" xref="S3.SS2.p2.13.m13.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p2.13.m13.1.1.2.1.cmml" xref="S3.SS2.p2.13.m13.1.1.2">superscript</csymbol><ci id="S3.SS2.p2.13.m13.1.1.2.2.cmml" xref="S3.SS2.p2.13.m13.1.1.2.2">𝐺</ci><ci id="S3.SS2.p2.13.m13.1.1.2.3.cmml" xref="S3.SS2.p2.13.m13.1.1.2.3">𝐵</ci></apply><apply id="S3.SS2.p2.13.m13.1.1.3.cmml" xref="S3.SS2.p2.13.m13.1.1.3"><times id="S3.SS2.p2.13.m13.1.1.3.1.cmml" xref="S3.SS2.p2.13.m13.1.1.3.1"></times><apply id="S3.SS2.p2.13.m13.1.1.3.2.cmml" xref="S3.SS2.p2.13.m13.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS2.p2.13.m13.1.1.3.2.1.cmml" xref="S3.SS2.p2.13.m13.1.1.3.2">superscript</csymbol><ci id="S3.SS2.p2.13.m13.1.1.3.2.2.cmml" xref="S3.SS2.p2.13.m13.1.1.3.2.2">𝑃</ci><csymbol cd="latexml" id="S3.SS2.p2.13.m13.1.1.3.2.3.cmml" xref="S3.SS2.p2.13.m13.1.1.3.2.3">top</csymbol></apply><apply id="S3.SS2.p2.13.m13.1.1.3.3.cmml" xref="S3.SS2.p2.13.m13.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS2.p2.13.m13.1.1.3.3.1.cmml" xref="S3.SS2.p2.13.m13.1.1.3.3">superscript</csymbol><ci id="S3.SS2.p2.13.m13.1.1.3.3.2.cmml" xref="S3.SS2.p2.13.m13.1.1.3.3.2">𝐺</ci><ci id="S3.SS2.p2.13.m13.1.1.3.3.3.cmml" xref="S3.SS2.p2.13.m13.1.1.3.3.3">𝑊</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.13.m13.1c">G^{B}=P^{\top}G^{W}</annotation></semantics></math>.
This can be seen by applying the chain rule to get <math id="S3.SS2.p2.14.m14.1" class="ltx_Math" alttext="G^{W}=x^{\top}\frac{\partial\mathcal{L}}{\partial y}" display="inline"><semantics id="S3.SS2.p2.14.m14.1a"><mrow id="S3.SS2.p2.14.m14.1.1" xref="S3.SS2.p2.14.m14.1.1.cmml"><msup id="S3.SS2.p2.14.m14.1.1.2" xref="S3.SS2.p2.14.m14.1.1.2.cmml"><mi id="S3.SS2.p2.14.m14.1.1.2.2" xref="S3.SS2.p2.14.m14.1.1.2.2.cmml">G</mi><mi id="S3.SS2.p2.14.m14.1.1.2.3" xref="S3.SS2.p2.14.m14.1.1.2.3.cmml">W</mi></msup><mo id="S3.SS2.p2.14.m14.1.1.1" xref="S3.SS2.p2.14.m14.1.1.1.cmml">=</mo><mrow id="S3.SS2.p2.14.m14.1.1.3" xref="S3.SS2.p2.14.m14.1.1.3.cmml"><msup id="S3.SS2.p2.14.m14.1.1.3.2" xref="S3.SS2.p2.14.m14.1.1.3.2.cmml"><mi id="S3.SS2.p2.14.m14.1.1.3.2.2" xref="S3.SS2.p2.14.m14.1.1.3.2.2.cmml">x</mi><mo id="S3.SS2.p2.14.m14.1.1.3.2.3" xref="S3.SS2.p2.14.m14.1.1.3.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="S3.SS2.p2.14.m14.1.1.3.1" xref="S3.SS2.p2.14.m14.1.1.3.1.cmml">​</mo><mfrac id="S3.SS2.p2.14.m14.1.1.3.3" xref="S3.SS2.p2.14.m14.1.1.3.3.cmml"><mrow id="S3.SS2.p2.14.m14.1.1.3.3.2" xref="S3.SS2.p2.14.m14.1.1.3.3.2.cmml"><mo rspace="0em" id="S3.SS2.p2.14.m14.1.1.3.3.2.1" xref="S3.SS2.p2.14.m14.1.1.3.3.2.1.cmml">∂</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p2.14.m14.1.1.3.3.2.2" xref="S3.SS2.p2.14.m14.1.1.3.3.2.2.cmml">ℒ</mi></mrow><mrow id="S3.SS2.p2.14.m14.1.1.3.3.3" xref="S3.SS2.p2.14.m14.1.1.3.3.3.cmml"><mo rspace="0em" id="S3.SS2.p2.14.m14.1.1.3.3.3.1" xref="S3.SS2.p2.14.m14.1.1.3.3.3.1.cmml">∂</mo><mi id="S3.SS2.p2.14.m14.1.1.3.3.3.2" xref="S3.SS2.p2.14.m14.1.1.3.3.3.2.cmml">y</mi></mrow></mfrac></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.14.m14.1b"><apply id="S3.SS2.p2.14.m14.1.1.cmml" xref="S3.SS2.p2.14.m14.1.1"><eq id="S3.SS2.p2.14.m14.1.1.1.cmml" xref="S3.SS2.p2.14.m14.1.1.1"></eq><apply id="S3.SS2.p2.14.m14.1.1.2.cmml" xref="S3.SS2.p2.14.m14.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p2.14.m14.1.1.2.1.cmml" xref="S3.SS2.p2.14.m14.1.1.2">superscript</csymbol><ci id="S3.SS2.p2.14.m14.1.1.2.2.cmml" xref="S3.SS2.p2.14.m14.1.1.2.2">𝐺</ci><ci id="S3.SS2.p2.14.m14.1.1.2.3.cmml" xref="S3.SS2.p2.14.m14.1.1.2.3">𝑊</ci></apply><apply id="S3.SS2.p2.14.m14.1.1.3.cmml" xref="S3.SS2.p2.14.m14.1.1.3"><times id="S3.SS2.p2.14.m14.1.1.3.1.cmml" xref="S3.SS2.p2.14.m14.1.1.3.1"></times><apply id="S3.SS2.p2.14.m14.1.1.3.2.cmml" xref="S3.SS2.p2.14.m14.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS2.p2.14.m14.1.1.3.2.1.cmml" xref="S3.SS2.p2.14.m14.1.1.3.2">superscript</csymbol><ci id="S3.SS2.p2.14.m14.1.1.3.2.2.cmml" xref="S3.SS2.p2.14.m14.1.1.3.2.2">𝑥</ci><csymbol cd="latexml" id="S3.SS2.p2.14.m14.1.1.3.2.3.cmml" xref="S3.SS2.p2.14.m14.1.1.3.2.3">top</csymbol></apply><apply id="S3.SS2.p2.14.m14.1.1.3.3.cmml" xref="S3.SS2.p2.14.m14.1.1.3.3"><divide id="S3.SS2.p2.14.m14.1.1.3.3.1.cmml" xref="S3.SS2.p2.14.m14.1.1.3.3"></divide><apply id="S3.SS2.p2.14.m14.1.1.3.3.2.cmml" xref="S3.SS2.p2.14.m14.1.1.3.3.2"><partialdiff id="S3.SS2.p2.14.m14.1.1.3.3.2.1.cmml" xref="S3.SS2.p2.14.m14.1.1.3.3.2.1"></partialdiff><ci id="S3.SS2.p2.14.m14.1.1.3.3.2.2.cmml" xref="S3.SS2.p2.14.m14.1.1.3.3.2.2">ℒ</ci></apply><apply id="S3.SS2.p2.14.m14.1.1.3.3.3.cmml" xref="S3.SS2.p2.14.m14.1.1.3.3.3"><partialdiff id="S3.SS2.p2.14.m14.1.1.3.3.3.1.cmml" xref="S3.SS2.p2.14.m14.1.1.3.3.3.1"></partialdiff><ci id="S3.SS2.p2.14.m14.1.1.3.3.3.2.cmml" xref="S3.SS2.p2.14.m14.1.1.3.3.3.2">𝑦</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.14.m14.1c">G^{W}=x^{\top}\frac{\partial\mathcal{L}}{\partial y}</annotation></semantics></math>.
The vector multiplied onto <math id="S3.SS2.p2.15.m15.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S3.SS2.p2.15.m15.1a"><mi id="S3.SS2.p2.15.m15.1.1" xref="S3.SS2.p2.15.m15.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.15.m15.1b"><ci id="S3.SS2.p2.15.m15.1.1.cmml" xref="S3.SS2.p2.15.m15.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.15.m15.1c">B</annotation></semantics></math> is <math id="S3.SS2.p2.16.m16.1" class="ltx_Math" alttext="xP" display="inline"><semantics id="S3.SS2.p2.16.m16.1a"><mrow id="S3.SS2.p2.16.m16.1.1" xref="S3.SS2.p2.16.m16.1.1.cmml"><mi id="S3.SS2.p2.16.m16.1.1.2" xref="S3.SS2.p2.16.m16.1.1.2.cmml">x</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.16.m16.1.1.1" xref="S3.SS2.p2.16.m16.1.1.1.cmml">​</mo><mi id="S3.SS2.p2.16.m16.1.1.3" xref="S3.SS2.p2.16.m16.1.1.3.cmml">P</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.16.m16.1b"><apply id="S3.SS2.p2.16.m16.1.1.cmml" xref="S3.SS2.p2.16.m16.1.1"><times id="S3.SS2.p2.16.m16.1.1.1.cmml" xref="S3.SS2.p2.16.m16.1.1.1"></times><ci id="S3.SS2.p2.16.m16.1.1.2.cmml" xref="S3.SS2.p2.16.m16.1.1.2">𝑥</ci><ci id="S3.SS2.p2.16.m16.1.1.3.cmml" xref="S3.SS2.p2.16.m16.1.1.3">𝑃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.16.m16.1c">xP</annotation></semantics></math> giving
<math id="S3.SS2.p2.17.m17.1" class="ltx_Math" alttext="G^{B}=(xP)^{\top}\frac{\partial\mathcal{L}}{\partial y}=P^{\top}x^{\top}\frac{\partial L}{\partial y}=P_{t}^{\top}G^{w}" display="inline"><semantics id="S3.SS2.p2.17.m17.1a"><mrow id="S3.SS2.p2.17.m17.1.1" xref="S3.SS2.p2.17.m17.1.1.cmml"><msup id="S3.SS2.p2.17.m17.1.1.3" xref="S3.SS2.p2.17.m17.1.1.3.cmml"><mi id="S3.SS2.p2.17.m17.1.1.3.2" xref="S3.SS2.p2.17.m17.1.1.3.2.cmml">G</mi><mi id="S3.SS2.p2.17.m17.1.1.3.3" xref="S3.SS2.p2.17.m17.1.1.3.3.cmml">B</mi></msup><mo id="S3.SS2.p2.17.m17.1.1.4" xref="S3.SS2.p2.17.m17.1.1.4.cmml">=</mo><mrow id="S3.SS2.p2.17.m17.1.1.1" xref="S3.SS2.p2.17.m17.1.1.1.cmml"><msup id="S3.SS2.p2.17.m17.1.1.1.1" xref="S3.SS2.p2.17.m17.1.1.1.1.cmml"><mrow id="S3.SS2.p2.17.m17.1.1.1.1.1.1" xref="S3.SS2.p2.17.m17.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p2.17.m17.1.1.1.1.1.1.2" xref="S3.SS2.p2.17.m17.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p2.17.m17.1.1.1.1.1.1.1" xref="S3.SS2.p2.17.m17.1.1.1.1.1.1.1.cmml"><mi id="S3.SS2.p2.17.m17.1.1.1.1.1.1.1.2" xref="S3.SS2.p2.17.m17.1.1.1.1.1.1.1.2.cmml">x</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.17.m17.1.1.1.1.1.1.1.1" xref="S3.SS2.p2.17.m17.1.1.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p2.17.m17.1.1.1.1.1.1.1.3" xref="S3.SS2.p2.17.m17.1.1.1.1.1.1.1.3.cmml">P</mi></mrow><mo stretchy="false" id="S3.SS2.p2.17.m17.1.1.1.1.1.1.3" xref="S3.SS2.p2.17.m17.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo id="S3.SS2.p2.17.m17.1.1.1.1.3" xref="S3.SS2.p2.17.m17.1.1.1.1.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="S3.SS2.p2.17.m17.1.1.1.2" xref="S3.SS2.p2.17.m17.1.1.1.2.cmml">​</mo><mfrac id="S3.SS2.p2.17.m17.1.1.1.3" xref="S3.SS2.p2.17.m17.1.1.1.3.cmml"><mrow id="S3.SS2.p2.17.m17.1.1.1.3.2" xref="S3.SS2.p2.17.m17.1.1.1.3.2.cmml"><mo rspace="0em" id="S3.SS2.p2.17.m17.1.1.1.3.2.1" xref="S3.SS2.p2.17.m17.1.1.1.3.2.1.cmml">∂</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p2.17.m17.1.1.1.3.2.2" xref="S3.SS2.p2.17.m17.1.1.1.3.2.2.cmml">ℒ</mi></mrow><mrow id="S3.SS2.p2.17.m17.1.1.1.3.3" xref="S3.SS2.p2.17.m17.1.1.1.3.3.cmml"><mo rspace="0em" id="S3.SS2.p2.17.m17.1.1.1.3.3.1" xref="S3.SS2.p2.17.m17.1.1.1.3.3.1.cmml">∂</mo><mi id="S3.SS2.p2.17.m17.1.1.1.3.3.2" xref="S3.SS2.p2.17.m17.1.1.1.3.3.2.cmml">y</mi></mrow></mfrac></mrow><mo id="S3.SS2.p2.17.m17.1.1.5" xref="S3.SS2.p2.17.m17.1.1.5.cmml">=</mo><mrow id="S3.SS2.p2.17.m17.1.1.6" xref="S3.SS2.p2.17.m17.1.1.6.cmml"><msup id="S3.SS2.p2.17.m17.1.1.6.2" xref="S3.SS2.p2.17.m17.1.1.6.2.cmml"><mi id="S3.SS2.p2.17.m17.1.1.6.2.2" xref="S3.SS2.p2.17.m17.1.1.6.2.2.cmml">P</mi><mo id="S3.SS2.p2.17.m17.1.1.6.2.3" xref="S3.SS2.p2.17.m17.1.1.6.2.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="S3.SS2.p2.17.m17.1.1.6.1" xref="S3.SS2.p2.17.m17.1.1.6.1.cmml">​</mo><msup id="S3.SS2.p2.17.m17.1.1.6.3" xref="S3.SS2.p2.17.m17.1.1.6.3.cmml"><mi id="S3.SS2.p2.17.m17.1.1.6.3.2" xref="S3.SS2.p2.17.m17.1.1.6.3.2.cmml">x</mi><mo id="S3.SS2.p2.17.m17.1.1.6.3.3" xref="S3.SS2.p2.17.m17.1.1.6.3.3.cmml">⊤</mo></msup><mo lspace="0em" rspace="0em" id="S3.SS2.p2.17.m17.1.1.6.1a" xref="S3.SS2.p2.17.m17.1.1.6.1.cmml">​</mo><mfrac id="S3.SS2.p2.17.m17.1.1.6.4" xref="S3.SS2.p2.17.m17.1.1.6.4.cmml"><mrow id="S3.SS2.p2.17.m17.1.1.6.4.2" xref="S3.SS2.p2.17.m17.1.1.6.4.2.cmml"><mo rspace="0em" id="S3.SS2.p2.17.m17.1.1.6.4.2.1" xref="S3.SS2.p2.17.m17.1.1.6.4.2.1.cmml">∂</mo><mi id="S3.SS2.p2.17.m17.1.1.6.4.2.2" xref="S3.SS2.p2.17.m17.1.1.6.4.2.2.cmml">L</mi></mrow><mrow id="S3.SS2.p2.17.m17.1.1.6.4.3" xref="S3.SS2.p2.17.m17.1.1.6.4.3.cmml"><mo rspace="0em" id="S3.SS2.p2.17.m17.1.1.6.4.3.1" xref="S3.SS2.p2.17.m17.1.1.6.4.3.1.cmml">∂</mo><mi id="S3.SS2.p2.17.m17.1.1.6.4.3.2" xref="S3.SS2.p2.17.m17.1.1.6.4.3.2.cmml">y</mi></mrow></mfrac></mrow><mo id="S3.SS2.p2.17.m17.1.1.7" xref="S3.SS2.p2.17.m17.1.1.7.cmml">=</mo><mrow id="S3.SS2.p2.17.m17.1.1.8" xref="S3.SS2.p2.17.m17.1.1.8.cmml"><msubsup id="S3.SS2.p2.17.m17.1.1.8.2" xref="S3.SS2.p2.17.m17.1.1.8.2.cmml"><mi id="S3.SS2.p2.17.m17.1.1.8.2.2.2" xref="S3.SS2.p2.17.m17.1.1.8.2.2.2.cmml">P</mi><mi id="S3.SS2.p2.17.m17.1.1.8.2.2.3" xref="S3.SS2.p2.17.m17.1.1.8.2.2.3.cmml">t</mi><mo id="S3.SS2.p2.17.m17.1.1.8.2.3" xref="S3.SS2.p2.17.m17.1.1.8.2.3.cmml">⊤</mo></msubsup><mo lspace="0em" rspace="0em" id="S3.SS2.p2.17.m17.1.1.8.1" xref="S3.SS2.p2.17.m17.1.1.8.1.cmml">​</mo><msup id="S3.SS2.p2.17.m17.1.1.8.3" xref="S3.SS2.p2.17.m17.1.1.8.3.cmml"><mi id="S3.SS2.p2.17.m17.1.1.8.3.2" xref="S3.SS2.p2.17.m17.1.1.8.3.2.cmml">G</mi><mi id="S3.SS2.p2.17.m17.1.1.8.3.3" xref="S3.SS2.p2.17.m17.1.1.8.3.3.cmml">w</mi></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.17.m17.1b"><apply id="S3.SS2.p2.17.m17.1.1.cmml" xref="S3.SS2.p2.17.m17.1.1"><and id="S3.SS2.p2.17.m17.1.1a.cmml" xref="S3.SS2.p2.17.m17.1.1"></and><apply id="S3.SS2.p2.17.m17.1.1b.cmml" xref="S3.SS2.p2.17.m17.1.1"><eq id="S3.SS2.p2.17.m17.1.1.4.cmml" xref="S3.SS2.p2.17.m17.1.1.4"></eq><apply id="S3.SS2.p2.17.m17.1.1.3.cmml" xref="S3.SS2.p2.17.m17.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p2.17.m17.1.1.3.1.cmml" xref="S3.SS2.p2.17.m17.1.1.3">superscript</csymbol><ci id="S3.SS2.p2.17.m17.1.1.3.2.cmml" xref="S3.SS2.p2.17.m17.1.1.3.2">𝐺</ci><ci id="S3.SS2.p2.17.m17.1.1.3.3.cmml" xref="S3.SS2.p2.17.m17.1.1.3.3">𝐵</ci></apply><apply id="S3.SS2.p2.17.m17.1.1.1.cmml" xref="S3.SS2.p2.17.m17.1.1.1"><times id="S3.SS2.p2.17.m17.1.1.1.2.cmml" xref="S3.SS2.p2.17.m17.1.1.1.2"></times><apply id="S3.SS2.p2.17.m17.1.1.1.1.cmml" xref="S3.SS2.p2.17.m17.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.17.m17.1.1.1.1.2.cmml" xref="S3.SS2.p2.17.m17.1.1.1.1">superscript</csymbol><apply id="S3.SS2.p2.17.m17.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.17.m17.1.1.1.1.1.1"><times id="S3.SS2.p2.17.m17.1.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.17.m17.1.1.1.1.1.1.1.1"></times><ci id="S3.SS2.p2.17.m17.1.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p2.17.m17.1.1.1.1.1.1.1.2">𝑥</ci><ci id="S3.SS2.p2.17.m17.1.1.1.1.1.1.1.3.cmml" xref="S3.SS2.p2.17.m17.1.1.1.1.1.1.1.3">𝑃</ci></apply><csymbol cd="latexml" id="S3.SS2.p2.17.m17.1.1.1.1.3.cmml" xref="S3.SS2.p2.17.m17.1.1.1.1.3">top</csymbol></apply><apply id="S3.SS2.p2.17.m17.1.1.1.3.cmml" xref="S3.SS2.p2.17.m17.1.1.1.3"><divide id="S3.SS2.p2.17.m17.1.1.1.3.1.cmml" xref="S3.SS2.p2.17.m17.1.1.1.3"></divide><apply id="S3.SS2.p2.17.m17.1.1.1.3.2.cmml" xref="S3.SS2.p2.17.m17.1.1.1.3.2"><partialdiff id="S3.SS2.p2.17.m17.1.1.1.3.2.1.cmml" xref="S3.SS2.p2.17.m17.1.1.1.3.2.1"></partialdiff><ci id="S3.SS2.p2.17.m17.1.1.1.3.2.2.cmml" xref="S3.SS2.p2.17.m17.1.1.1.3.2.2">ℒ</ci></apply><apply id="S3.SS2.p2.17.m17.1.1.1.3.3.cmml" xref="S3.SS2.p2.17.m17.1.1.1.3.3"><partialdiff id="S3.SS2.p2.17.m17.1.1.1.3.3.1.cmml" xref="S3.SS2.p2.17.m17.1.1.1.3.3.1"></partialdiff><ci id="S3.SS2.p2.17.m17.1.1.1.3.3.2.cmml" xref="S3.SS2.p2.17.m17.1.1.1.3.3.2">𝑦</ci></apply></apply></apply></apply><apply id="S3.SS2.p2.17.m17.1.1c.cmml" xref="S3.SS2.p2.17.m17.1.1"><eq id="S3.SS2.p2.17.m17.1.1.5.cmml" xref="S3.SS2.p2.17.m17.1.1.5"></eq><share href="#S3.SS2.p2.17.m17.1.1.1.cmml" id="S3.SS2.p2.17.m17.1.1d.cmml" xref="S3.SS2.p2.17.m17.1.1"></share><apply id="S3.SS2.p2.17.m17.1.1.6.cmml" xref="S3.SS2.p2.17.m17.1.1.6"><times id="S3.SS2.p2.17.m17.1.1.6.1.cmml" xref="S3.SS2.p2.17.m17.1.1.6.1"></times><apply id="S3.SS2.p2.17.m17.1.1.6.2.cmml" xref="S3.SS2.p2.17.m17.1.1.6.2"><csymbol cd="ambiguous" id="S3.SS2.p2.17.m17.1.1.6.2.1.cmml" xref="S3.SS2.p2.17.m17.1.1.6.2">superscript</csymbol><ci id="S3.SS2.p2.17.m17.1.1.6.2.2.cmml" xref="S3.SS2.p2.17.m17.1.1.6.2.2">𝑃</ci><csymbol cd="latexml" id="S3.SS2.p2.17.m17.1.1.6.2.3.cmml" xref="S3.SS2.p2.17.m17.1.1.6.2.3">top</csymbol></apply><apply id="S3.SS2.p2.17.m17.1.1.6.3.cmml" xref="S3.SS2.p2.17.m17.1.1.6.3"><csymbol cd="ambiguous" id="S3.SS2.p2.17.m17.1.1.6.3.1.cmml" xref="S3.SS2.p2.17.m17.1.1.6.3">superscript</csymbol><ci id="S3.SS2.p2.17.m17.1.1.6.3.2.cmml" xref="S3.SS2.p2.17.m17.1.1.6.3.2">𝑥</ci><csymbol cd="latexml" id="S3.SS2.p2.17.m17.1.1.6.3.3.cmml" xref="S3.SS2.p2.17.m17.1.1.6.3.3">top</csymbol></apply><apply id="S3.SS2.p2.17.m17.1.1.6.4.cmml" xref="S3.SS2.p2.17.m17.1.1.6.4"><divide id="S3.SS2.p2.17.m17.1.1.6.4.1.cmml" xref="S3.SS2.p2.17.m17.1.1.6.4"></divide><apply id="S3.SS2.p2.17.m17.1.1.6.4.2.cmml" xref="S3.SS2.p2.17.m17.1.1.6.4.2"><partialdiff id="S3.SS2.p2.17.m17.1.1.6.4.2.1.cmml" xref="S3.SS2.p2.17.m17.1.1.6.4.2.1"></partialdiff><ci id="S3.SS2.p2.17.m17.1.1.6.4.2.2.cmml" xref="S3.SS2.p2.17.m17.1.1.6.4.2.2">𝐿</ci></apply><apply id="S3.SS2.p2.17.m17.1.1.6.4.3.cmml" xref="S3.SS2.p2.17.m17.1.1.6.4.3"><partialdiff id="S3.SS2.p2.17.m17.1.1.6.4.3.1.cmml" xref="S3.SS2.p2.17.m17.1.1.6.4.3.1"></partialdiff><ci id="S3.SS2.p2.17.m17.1.1.6.4.3.2.cmml" xref="S3.SS2.p2.17.m17.1.1.6.4.3.2">𝑦</ci></apply></apply></apply></apply><apply id="S3.SS2.p2.17.m17.1.1e.cmml" xref="S3.SS2.p2.17.m17.1.1"><eq id="S3.SS2.p2.17.m17.1.1.7.cmml" xref="S3.SS2.p2.17.m17.1.1.7"></eq><share href="#S3.SS2.p2.17.m17.1.1.6.cmml" id="S3.SS2.p2.17.m17.1.1f.cmml" xref="S3.SS2.p2.17.m17.1.1"></share><apply id="S3.SS2.p2.17.m17.1.1.8.cmml" xref="S3.SS2.p2.17.m17.1.1.8"><times id="S3.SS2.p2.17.m17.1.1.8.1.cmml" xref="S3.SS2.p2.17.m17.1.1.8.1"></times><apply id="S3.SS2.p2.17.m17.1.1.8.2.cmml" xref="S3.SS2.p2.17.m17.1.1.8.2"><csymbol cd="ambiguous" id="S3.SS2.p2.17.m17.1.1.8.2.1.cmml" xref="S3.SS2.p2.17.m17.1.1.8.2">superscript</csymbol><apply id="S3.SS2.p2.17.m17.1.1.8.2.2.cmml" xref="S3.SS2.p2.17.m17.1.1.8.2"><csymbol cd="ambiguous" id="S3.SS2.p2.17.m17.1.1.8.2.2.1.cmml" xref="S3.SS2.p2.17.m17.1.1.8.2">subscript</csymbol><ci id="S3.SS2.p2.17.m17.1.1.8.2.2.2.cmml" xref="S3.SS2.p2.17.m17.1.1.8.2.2.2">𝑃</ci><ci id="S3.SS2.p2.17.m17.1.1.8.2.2.3.cmml" xref="S3.SS2.p2.17.m17.1.1.8.2.2.3">𝑡</ci></apply><csymbol cd="latexml" id="S3.SS2.p2.17.m17.1.1.8.2.3.cmml" xref="S3.SS2.p2.17.m17.1.1.8.2.3">top</csymbol></apply><apply id="S3.SS2.p2.17.m17.1.1.8.3.cmml" xref="S3.SS2.p2.17.m17.1.1.8.3"><csymbol cd="ambiguous" id="S3.SS2.p2.17.m17.1.1.8.3.1.cmml" xref="S3.SS2.p2.17.m17.1.1.8.3">superscript</csymbol><ci id="S3.SS2.p2.17.m17.1.1.8.3.2.cmml" xref="S3.SS2.p2.17.m17.1.1.8.3.2">𝐺</ci><ci id="S3.SS2.p2.17.m17.1.1.8.3.3.cmml" xref="S3.SS2.p2.17.m17.1.1.8.3.3">𝑤</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.17.m17.1c">G^{B}=(xP)^{\top}\frac{\partial\mathcal{L}}{\partial y}=P^{\top}x^{\top}\frac{\partial L}{\partial y}=P_{t}^{\top}G^{w}</annotation></semantics></math>.
This shows that calculating the gradient w.r.t <math id="S3.SS2.p2.18.m18.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S3.SS2.p2.18.m18.1a"><mi id="S3.SS2.p2.18.m18.1.1" xref="S3.SS2.p2.18.m18.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.18.m18.1b"><ci id="S3.SS2.p2.18.m18.1.1.cmml" xref="S3.SS2.p2.18.m18.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.18.m18.1c">B</annotation></semantics></math> gives the same as projecting the gradient w.r.t <math id="S3.SS2.p2.19.m19.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S3.SS2.p2.19.m19.1a"><mi id="S3.SS2.p2.19.m19.1.1" xref="S3.SS2.p2.19.m19.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.19.m19.1b"><ci id="S3.SS2.p2.19.m19.1.1.cmml" xref="S3.SS2.p2.19.m19.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.19.m19.1c">W</annotation></semantics></math>. It is thus clear that GaLore’s low-rank gradient updates should be the same as those obtained using backpropagation through LoRA.</p>
</div>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Enabling pretraining with LoRA</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">Previous work has shown that training low-rank weight matrices works well for fine-tuning pre-trained weights. However, it has been shown that training low-rank factors, and periodically merging them into frozen <math id="S3.SS3.p1.1.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S3.SS3.p1.1.m1.1a"><mi id="S3.SS3.p1.1.m1.1.1" xref="S3.SS3.p1.1.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.1.m1.1b"><ci id="S3.SS3.p1.1.m1.1.1.cmml" xref="S3.SS3.p1.1.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.1.m1.1c">W</annotation></semantics></math>, does not work when starting with a randomly initialized matrix <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib22" title="" class="ltx_ref">22</a>]</cite>. Here we address this shortcoming to enable full training using low-rank weight matrices.</p>
</div>
<div id="S3.SS3.p2" class="ltx_para">
<p id="S3.SS3.p2.3" class="ltx_p">Inspired by prior work <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib22" title="" class="ltx_ref">22</a>, <a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite>, we periodically update a given layer <math id="S3.SS3.p2.1.m1.1" class="ltx_Math" alttext="W_{T+1}=W_{T}+P_{T}B_{T}" display="inline"><semantics id="S3.SS3.p2.1.m1.1a"><mrow id="S3.SS3.p2.1.m1.1.1" xref="S3.SS3.p2.1.m1.1.1.cmml"><msub id="S3.SS3.p2.1.m1.1.1.2" xref="S3.SS3.p2.1.m1.1.1.2.cmml"><mi id="S3.SS3.p2.1.m1.1.1.2.2" xref="S3.SS3.p2.1.m1.1.1.2.2.cmml">W</mi><mrow id="S3.SS3.p2.1.m1.1.1.2.3" xref="S3.SS3.p2.1.m1.1.1.2.3.cmml"><mi id="S3.SS3.p2.1.m1.1.1.2.3.2" xref="S3.SS3.p2.1.m1.1.1.2.3.2.cmml">T</mi><mo id="S3.SS3.p2.1.m1.1.1.2.3.1" xref="S3.SS3.p2.1.m1.1.1.2.3.1.cmml">+</mo><mn id="S3.SS3.p2.1.m1.1.1.2.3.3" xref="S3.SS3.p2.1.m1.1.1.2.3.3.cmml">1</mn></mrow></msub><mo id="S3.SS3.p2.1.m1.1.1.1" xref="S3.SS3.p2.1.m1.1.1.1.cmml">=</mo><mrow id="S3.SS3.p2.1.m1.1.1.3" xref="S3.SS3.p2.1.m1.1.1.3.cmml"><msub id="S3.SS3.p2.1.m1.1.1.3.2" xref="S3.SS3.p2.1.m1.1.1.3.2.cmml"><mi id="S3.SS3.p2.1.m1.1.1.3.2.2" xref="S3.SS3.p2.1.m1.1.1.3.2.2.cmml">W</mi><mi id="S3.SS3.p2.1.m1.1.1.3.2.3" xref="S3.SS3.p2.1.m1.1.1.3.2.3.cmml">T</mi></msub><mo id="S3.SS3.p2.1.m1.1.1.3.1" xref="S3.SS3.p2.1.m1.1.1.3.1.cmml">+</mo><mrow id="S3.SS3.p2.1.m1.1.1.3.3" xref="S3.SS3.p2.1.m1.1.1.3.3.cmml"><msub id="S3.SS3.p2.1.m1.1.1.3.3.2" xref="S3.SS3.p2.1.m1.1.1.3.3.2.cmml"><mi id="S3.SS3.p2.1.m1.1.1.3.3.2.2" xref="S3.SS3.p2.1.m1.1.1.3.3.2.2.cmml">P</mi><mi id="S3.SS3.p2.1.m1.1.1.3.3.2.3" xref="S3.SS3.p2.1.m1.1.1.3.3.2.3.cmml">T</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS3.p2.1.m1.1.1.3.3.1" xref="S3.SS3.p2.1.m1.1.1.3.3.1.cmml">​</mo><msub id="S3.SS3.p2.1.m1.1.1.3.3.3" xref="S3.SS3.p2.1.m1.1.1.3.3.3.cmml"><mi id="S3.SS3.p2.1.m1.1.1.3.3.3.2" xref="S3.SS3.p2.1.m1.1.1.3.3.3.2.cmml">B</mi><mi id="S3.SS3.p2.1.m1.1.1.3.3.3.3" xref="S3.SS3.p2.1.m1.1.1.3.3.3.3.cmml">T</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.1.m1.1b"><apply id="S3.SS3.p2.1.m1.1.1.cmml" xref="S3.SS3.p2.1.m1.1.1"><eq id="S3.SS3.p2.1.m1.1.1.1.cmml" xref="S3.SS3.p2.1.m1.1.1.1"></eq><apply id="S3.SS3.p2.1.m1.1.1.2.cmml" xref="S3.SS3.p2.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p2.1.m1.1.1.2.1.cmml" xref="S3.SS3.p2.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS3.p2.1.m1.1.1.2.2.cmml" xref="S3.SS3.p2.1.m1.1.1.2.2">𝑊</ci><apply id="S3.SS3.p2.1.m1.1.1.2.3.cmml" xref="S3.SS3.p2.1.m1.1.1.2.3"><plus id="S3.SS3.p2.1.m1.1.1.2.3.1.cmml" xref="S3.SS3.p2.1.m1.1.1.2.3.1"></plus><ci id="S3.SS3.p2.1.m1.1.1.2.3.2.cmml" xref="S3.SS3.p2.1.m1.1.1.2.3.2">𝑇</ci><cn type="integer" id="S3.SS3.p2.1.m1.1.1.2.3.3.cmml" xref="S3.SS3.p2.1.m1.1.1.2.3.3">1</cn></apply></apply><apply id="S3.SS3.p2.1.m1.1.1.3.cmml" xref="S3.SS3.p2.1.m1.1.1.3"><plus id="S3.SS3.p2.1.m1.1.1.3.1.cmml" xref="S3.SS3.p2.1.m1.1.1.3.1"></plus><apply id="S3.SS3.p2.1.m1.1.1.3.2.cmml" xref="S3.SS3.p2.1.m1.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS3.p2.1.m1.1.1.3.2.1.cmml" xref="S3.SS3.p2.1.m1.1.1.3.2">subscript</csymbol><ci id="S3.SS3.p2.1.m1.1.1.3.2.2.cmml" xref="S3.SS3.p2.1.m1.1.1.3.2.2">𝑊</ci><ci id="S3.SS3.p2.1.m1.1.1.3.2.3.cmml" xref="S3.SS3.p2.1.m1.1.1.3.2.3">𝑇</ci></apply><apply id="S3.SS3.p2.1.m1.1.1.3.3.cmml" xref="S3.SS3.p2.1.m1.1.1.3.3"><times id="S3.SS3.p2.1.m1.1.1.3.3.1.cmml" xref="S3.SS3.p2.1.m1.1.1.3.3.1"></times><apply id="S3.SS3.p2.1.m1.1.1.3.3.2.cmml" xref="S3.SS3.p2.1.m1.1.1.3.3.2"><csymbol cd="ambiguous" id="S3.SS3.p2.1.m1.1.1.3.3.2.1.cmml" xref="S3.SS3.p2.1.m1.1.1.3.3.2">subscript</csymbol><ci id="S3.SS3.p2.1.m1.1.1.3.3.2.2.cmml" xref="S3.SS3.p2.1.m1.1.1.3.3.2.2">𝑃</ci><ci id="S3.SS3.p2.1.m1.1.1.3.3.2.3.cmml" xref="S3.SS3.p2.1.m1.1.1.3.3.2.3">𝑇</ci></apply><apply id="S3.SS3.p2.1.m1.1.1.3.3.3.cmml" xref="S3.SS3.p2.1.m1.1.1.3.3.3"><csymbol cd="ambiguous" id="S3.SS3.p2.1.m1.1.1.3.3.3.1.cmml" xref="S3.SS3.p2.1.m1.1.1.3.3.3">subscript</csymbol><ci id="S3.SS3.p2.1.m1.1.1.3.3.3.2.cmml" xref="S3.SS3.p2.1.m1.1.1.3.3.3.2">𝐵</ci><ci id="S3.SS3.p2.1.m1.1.1.3.3.3.3.cmml" xref="S3.SS3.p2.1.m1.1.1.3.3.3.3">𝑇</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.1.m1.1c">W_{T+1}=W_{T}+P_{T}B_{T}</annotation></semantics></math> at fixed steps <math id="S3.SS3.p2.2.m2.1" class="ltx_Math" alttext="T\in\mathcal{T}" display="inline"><semantics id="S3.SS3.p2.2.m2.1a"><mrow id="S3.SS3.p2.2.m2.1.1" xref="S3.SS3.p2.2.m2.1.1.cmml"><mi id="S3.SS3.p2.2.m2.1.1.2" xref="S3.SS3.p2.2.m2.1.1.2.cmml">T</mi><mo id="S3.SS3.p2.2.m2.1.1.1" xref="S3.SS3.p2.2.m2.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.2.m2.1.1.3" xref="S3.SS3.p2.2.m2.1.1.3.cmml">𝒯</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.2.m2.1b"><apply id="S3.SS3.p2.2.m2.1.1.cmml" xref="S3.SS3.p2.2.m2.1.1"><in id="S3.SS3.p2.2.m2.1.1.1.cmml" xref="S3.SS3.p2.2.m2.1.1.1"></in><ci id="S3.SS3.p2.2.m2.1.1.2.cmml" xref="S3.SS3.p2.2.m2.1.1.2">𝑇</ci><ci id="S3.SS3.p2.2.m2.1.1.3.cmml" xref="S3.SS3.p2.2.m2.1.1.3">𝒯</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.2.m2.1c">T\in\mathcal{T}</annotation></semantics></math>. This approach allows <math id="S3.SS3.p2.3.m3.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S3.SS3.p2.3.m3.1a"><mi id="S3.SS3.p2.3.m3.1.1" xref="S3.SS3.p2.3.m3.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.3.m3.1b"><ci id="S3.SS3.p2.3.m3.1.1.cmml" xref="S3.SS3.p2.3.m3.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.3.m3.1c">W</annotation></semantics></math> to evolve as a sum of low-rank matrices aligning with GaLore’s strategy of updating the gradient subspace during training:</p>
<table id="S3.Ex3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.Ex3.m1.1" class="ltx_Math" alttext="W_{t}=W_{0}+\Delta W_{T_{1}}+\Delta W_{T_{2}}+\ldots+\Delta W_{T_{n}}," display="block"><semantics id="S3.Ex3.m1.1a"><mrow id="S3.Ex3.m1.1.1.1" xref="S3.Ex3.m1.1.1.1.1.cmml"><mrow id="S3.Ex3.m1.1.1.1.1" xref="S3.Ex3.m1.1.1.1.1.cmml"><msub id="S3.Ex3.m1.1.1.1.1.2" xref="S3.Ex3.m1.1.1.1.1.2.cmml"><mi id="S3.Ex3.m1.1.1.1.1.2.2" xref="S3.Ex3.m1.1.1.1.1.2.2.cmml">W</mi><mi id="S3.Ex3.m1.1.1.1.1.2.3" xref="S3.Ex3.m1.1.1.1.1.2.3.cmml">t</mi></msub><mo id="S3.Ex3.m1.1.1.1.1.1" xref="S3.Ex3.m1.1.1.1.1.1.cmml">=</mo><mrow id="S3.Ex3.m1.1.1.1.1.3" xref="S3.Ex3.m1.1.1.1.1.3.cmml"><msub id="S3.Ex3.m1.1.1.1.1.3.2" xref="S3.Ex3.m1.1.1.1.1.3.2.cmml"><mi id="S3.Ex3.m1.1.1.1.1.3.2.2" xref="S3.Ex3.m1.1.1.1.1.3.2.2.cmml">W</mi><mn id="S3.Ex3.m1.1.1.1.1.3.2.3" xref="S3.Ex3.m1.1.1.1.1.3.2.3.cmml">0</mn></msub><mo id="S3.Ex3.m1.1.1.1.1.3.1" xref="S3.Ex3.m1.1.1.1.1.3.1.cmml">+</mo><mrow id="S3.Ex3.m1.1.1.1.1.3.3" xref="S3.Ex3.m1.1.1.1.1.3.3.cmml"><mi mathvariant="normal" id="S3.Ex3.m1.1.1.1.1.3.3.2" xref="S3.Ex3.m1.1.1.1.1.3.3.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S3.Ex3.m1.1.1.1.1.3.3.1" xref="S3.Ex3.m1.1.1.1.1.3.3.1.cmml">​</mo><msub id="S3.Ex3.m1.1.1.1.1.3.3.3" xref="S3.Ex3.m1.1.1.1.1.3.3.3.cmml"><mi id="S3.Ex3.m1.1.1.1.1.3.3.3.2" xref="S3.Ex3.m1.1.1.1.1.3.3.3.2.cmml">W</mi><msub id="S3.Ex3.m1.1.1.1.1.3.3.3.3" xref="S3.Ex3.m1.1.1.1.1.3.3.3.3.cmml"><mi id="S3.Ex3.m1.1.1.1.1.3.3.3.3.2" xref="S3.Ex3.m1.1.1.1.1.3.3.3.3.2.cmml">T</mi><mn id="S3.Ex3.m1.1.1.1.1.3.3.3.3.3" xref="S3.Ex3.m1.1.1.1.1.3.3.3.3.3.cmml">1</mn></msub></msub></mrow><mo id="S3.Ex3.m1.1.1.1.1.3.1a" xref="S3.Ex3.m1.1.1.1.1.3.1.cmml">+</mo><mrow id="S3.Ex3.m1.1.1.1.1.3.4" xref="S3.Ex3.m1.1.1.1.1.3.4.cmml"><mi mathvariant="normal" id="S3.Ex3.m1.1.1.1.1.3.4.2" xref="S3.Ex3.m1.1.1.1.1.3.4.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S3.Ex3.m1.1.1.1.1.3.4.1" xref="S3.Ex3.m1.1.1.1.1.3.4.1.cmml">​</mo><msub id="S3.Ex3.m1.1.1.1.1.3.4.3" xref="S3.Ex3.m1.1.1.1.1.3.4.3.cmml"><mi id="S3.Ex3.m1.1.1.1.1.3.4.3.2" xref="S3.Ex3.m1.1.1.1.1.3.4.3.2.cmml">W</mi><msub id="S3.Ex3.m1.1.1.1.1.3.4.3.3" xref="S3.Ex3.m1.1.1.1.1.3.4.3.3.cmml"><mi id="S3.Ex3.m1.1.1.1.1.3.4.3.3.2" xref="S3.Ex3.m1.1.1.1.1.3.4.3.3.2.cmml">T</mi><mn id="S3.Ex3.m1.1.1.1.1.3.4.3.3.3" xref="S3.Ex3.m1.1.1.1.1.3.4.3.3.3.cmml">2</mn></msub></msub></mrow><mo id="S3.Ex3.m1.1.1.1.1.3.1b" xref="S3.Ex3.m1.1.1.1.1.3.1.cmml">+</mo><mi mathvariant="normal" id="S3.Ex3.m1.1.1.1.1.3.5" xref="S3.Ex3.m1.1.1.1.1.3.5.cmml">…</mi><mo id="S3.Ex3.m1.1.1.1.1.3.1c" xref="S3.Ex3.m1.1.1.1.1.3.1.cmml">+</mo><mrow id="S3.Ex3.m1.1.1.1.1.3.6" xref="S3.Ex3.m1.1.1.1.1.3.6.cmml"><mi mathvariant="normal" id="S3.Ex3.m1.1.1.1.1.3.6.2" xref="S3.Ex3.m1.1.1.1.1.3.6.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S3.Ex3.m1.1.1.1.1.3.6.1" xref="S3.Ex3.m1.1.1.1.1.3.6.1.cmml">​</mo><msub id="S3.Ex3.m1.1.1.1.1.3.6.3" xref="S3.Ex3.m1.1.1.1.1.3.6.3.cmml"><mi id="S3.Ex3.m1.1.1.1.1.3.6.3.2" xref="S3.Ex3.m1.1.1.1.1.3.6.3.2.cmml">W</mi><msub id="S3.Ex3.m1.1.1.1.1.3.6.3.3" xref="S3.Ex3.m1.1.1.1.1.3.6.3.3.cmml"><mi id="S3.Ex3.m1.1.1.1.1.3.6.3.3.2" xref="S3.Ex3.m1.1.1.1.1.3.6.3.3.2.cmml">T</mi><mi id="S3.Ex3.m1.1.1.1.1.3.6.3.3.3" xref="S3.Ex3.m1.1.1.1.1.3.6.3.3.3.cmml">n</mi></msub></msub></mrow></mrow></mrow><mo id="S3.Ex3.m1.1.1.1.2" xref="S3.Ex3.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.Ex3.m1.1b"><apply id="S3.Ex3.m1.1.1.1.1.cmml" xref="S3.Ex3.m1.1.1.1"><eq id="S3.Ex3.m1.1.1.1.1.1.cmml" xref="S3.Ex3.m1.1.1.1.1.1"></eq><apply id="S3.Ex3.m1.1.1.1.1.2.cmml" xref="S3.Ex3.m1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.Ex3.m1.1.1.1.1.2.1.cmml" xref="S3.Ex3.m1.1.1.1.1.2">subscript</csymbol><ci id="S3.Ex3.m1.1.1.1.1.2.2.cmml" xref="S3.Ex3.m1.1.1.1.1.2.2">𝑊</ci><ci id="S3.Ex3.m1.1.1.1.1.2.3.cmml" xref="S3.Ex3.m1.1.1.1.1.2.3">𝑡</ci></apply><apply id="S3.Ex3.m1.1.1.1.1.3.cmml" xref="S3.Ex3.m1.1.1.1.1.3"><plus id="S3.Ex3.m1.1.1.1.1.3.1.cmml" xref="S3.Ex3.m1.1.1.1.1.3.1"></plus><apply id="S3.Ex3.m1.1.1.1.1.3.2.cmml" xref="S3.Ex3.m1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S3.Ex3.m1.1.1.1.1.3.2.1.cmml" xref="S3.Ex3.m1.1.1.1.1.3.2">subscript</csymbol><ci id="S3.Ex3.m1.1.1.1.1.3.2.2.cmml" xref="S3.Ex3.m1.1.1.1.1.3.2.2">𝑊</ci><cn type="integer" id="S3.Ex3.m1.1.1.1.1.3.2.3.cmml" xref="S3.Ex3.m1.1.1.1.1.3.2.3">0</cn></apply><apply id="S3.Ex3.m1.1.1.1.1.3.3.cmml" xref="S3.Ex3.m1.1.1.1.1.3.3"><times id="S3.Ex3.m1.1.1.1.1.3.3.1.cmml" xref="S3.Ex3.m1.1.1.1.1.3.3.1"></times><ci id="S3.Ex3.m1.1.1.1.1.3.3.2.cmml" xref="S3.Ex3.m1.1.1.1.1.3.3.2">Δ</ci><apply id="S3.Ex3.m1.1.1.1.1.3.3.3.cmml" xref="S3.Ex3.m1.1.1.1.1.3.3.3"><csymbol cd="ambiguous" id="S3.Ex3.m1.1.1.1.1.3.3.3.1.cmml" xref="S3.Ex3.m1.1.1.1.1.3.3.3">subscript</csymbol><ci id="S3.Ex3.m1.1.1.1.1.3.3.3.2.cmml" xref="S3.Ex3.m1.1.1.1.1.3.3.3.2">𝑊</ci><apply id="S3.Ex3.m1.1.1.1.1.3.3.3.3.cmml" xref="S3.Ex3.m1.1.1.1.1.3.3.3.3"><csymbol cd="ambiguous" id="S3.Ex3.m1.1.1.1.1.3.3.3.3.1.cmml" xref="S3.Ex3.m1.1.1.1.1.3.3.3.3">subscript</csymbol><ci id="S3.Ex3.m1.1.1.1.1.3.3.3.3.2.cmml" xref="S3.Ex3.m1.1.1.1.1.3.3.3.3.2">𝑇</ci><cn type="integer" id="S3.Ex3.m1.1.1.1.1.3.3.3.3.3.cmml" xref="S3.Ex3.m1.1.1.1.1.3.3.3.3.3">1</cn></apply></apply></apply><apply id="S3.Ex3.m1.1.1.1.1.3.4.cmml" xref="S3.Ex3.m1.1.1.1.1.3.4"><times id="S3.Ex3.m1.1.1.1.1.3.4.1.cmml" xref="S3.Ex3.m1.1.1.1.1.3.4.1"></times><ci id="S3.Ex3.m1.1.1.1.1.3.4.2.cmml" xref="S3.Ex3.m1.1.1.1.1.3.4.2">Δ</ci><apply id="S3.Ex3.m1.1.1.1.1.3.4.3.cmml" xref="S3.Ex3.m1.1.1.1.1.3.4.3"><csymbol cd="ambiguous" id="S3.Ex3.m1.1.1.1.1.3.4.3.1.cmml" xref="S3.Ex3.m1.1.1.1.1.3.4.3">subscript</csymbol><ci id="S3.Ex3.m1.1.1.1.1.3.4.3.2.cmml" xref="S3.Ex3.m1.1.1.1.1.3.4.3.2">𝑊</ci><apply id="S3.Ex3.m1.1.1.1.1.3.4.3.3.cmml" xref="S3.Ex3.m1.1.1.1.1.3.4.3.3"><csymbol cd="ambiguous" id="S3.Ex3.m1.1.1.1.1.3.4.3.3.1.cmml" xref="S3.Ex3.m1.1.1.1.1.3.4.3.3">subscript</csymbol><ci id="S3.Ex3.m1.1.1.1.1.3.4.3.3.2.cmml" xref="S3.Ex3.m1.1.1.1.1.3.4.3.3.2">𝑇</ci><cn type="integer" id="S3.Ex3.m1.1.1.1.1.3.4.3.3.3.cmml" xref="S3.Ex3.m1.1.1.1.1.3.4.3.3.3">2</cn></apply></apply></apply><ci id="S3.Ex3.m1.1.1.1.1.3.5.cmml" xref="S3.Ex3.m1.1.1.1.1.3.5">…</ci><apply id="S3.Ex3.m1.1.1.1.1.3.6.cmml" xref="S3.Ex3.m1.1.1.1.1.3.6"><times id="S3.Ex3.m1.1.1.1.1.3.6.1.cmml" xref="S3.Ex3.m1.1.1.1.1.3.6.1"></times><ci id="S3.Ex3.m1.1.1.1.1.3.6.2.cmml" xref="S3.Ex3.m1.1.1.1.1.3.6.2">Δ</ci><apply id="S3.Ex3.m1.1.1.1.1.3.6.3.cmml" xref="S3.Ex3.m1.1.1.1.1.3.6.3"><csymbol cd="ambiguous" id="S3.Ex3.m1.1.1.1.1.3.6.3.1.cmml" xref="S3.Ex3.m1.1.1.1.1.3.6.3">subscript</csymbol><ci id="S3.Ex3.m1.1.1.1.1.3.6.3.2.cmml" xref="S3.Ex3.m1.1.1.1.1.3.6.3.2">𝑊</ci><apply id="S3.Ex3.m1.1.1.1.1.3.6.3.3.cmml" xref="S3.Ex3.m1.1.1.1.1.3.6.3.3"><csymbol cd="ambiguous" id="S3.Ex3.m1.1.1.1.1.3.6.3.3.1.cmml" xref="S3.Ex3.m1.1.1.1.1.3.6.3.3">subscript</csymbol><ci id="S3.Ex3.m1.1.1.1.1.3.6.3.3.2.cmml" xref="S3.Ex3.m1.1.1.1.1.3.6.3.3.2">𝑇</ci><ci id="S3.Ex3.m1.1.1.1.1.3.6.3.3.3.cmml" xref="S3.Ex3.m1.1.1.1.1.3.6.3.3.3">𝑛</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.Ex3.m1.1c">W_{t}=W_{0}+\Delta W_{T_{1}}+\Delta W_{T_{2}}+\ldots+\Delta W_{T_{n}},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S3.SS3.p2.14" class="ltx_p">where <math id="S3.SS3.p2.4.m1.1" class="ltx_Math" alttext="t=\sum_{i=1}^{|\mathcal{T}|}T_{i}" display="inline"><semantics id="S3.SS3.p2.4.m1.1a"><mrow id="S3.SS3.p2.4.m1.1.2" xref="S3.SS3.p2.4.m1.1.2.cmml"><mi id="S3.SS3.p2.4.m1.1.2.2" xref="S3.SS3.p2.4.m1.1.2.2.cmml">t</mi><mo rspace="0.111em" id="S3.SS3.p2.4.m1.1.2.1" xref="S3.SS3.p2.4.m1.1.2.1.cmml">=</mo><mrow id="S3.SS3.p2.4.m1.1.2.3" xref="S3.SS3.p2.4.m1.1.2.3.cmml"><msubsup id="S3.SS3.p2.4.m1.1.2.3.1" xref="S3.SS3.p2.4.m1.1.2.3.1.cmml"><mo id="S3.SS3.p2.4.m1.1.2.3.1.2.2" xref="S3.SS3.p2.4.m1.1.2.3.1.2.2.cmml">∑</mo><mrow id="S3.SS3.p2.4.m1.1.2.3.1.2.3" xref="S3.SS3.p2.4.m1.1.2.3.1.2.3.cmml"><mi id="S3.SS3.p2.4.m1.1.2.3.1.2.3.2" xref="S3.SS3.p2.4.m1.1.2.3.1.2.3.2.cmml">i</mi><mo id="S3.SS3.p2.4.m1.1.2.3.1.2.3.1" xref="S3.SS3.p2.4.m1.1.2.3.1.2.3.1.cmml">=</mo><mn id="S3.SS3.p2.4.m1.1.2.3.1.2.3.3" xref="S3.SS3.p2.4.m1.1.2.3.1.2.3.3.cmml">1</mn></mrow><mrow id="S3.SS3.p2.4.m1.1.1.1.3" xref="S3.SS3.p2.4.m1.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS3.p2.4.m1.1.1.1.3.1" xref="S3.SS3.p2.4.m1.1.1.1.2.1.cmml">|</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.4.m1.1.1.1.1" xref="S3.SS3.p2.4.m1.1.1.1.1.cmml">𝒯</mi><mo stretchy="false" id="S3.SS3.p2.4.m1.1.1.1.3.2" xref="S3.SS3.p2.4.m1.1.1.1.2.1.cmml">|</mo></mrow></msubsup><msub id="S3.SS3.p2.4.m1.1.2.3.2" xref="S3.SS3.p2.4.m1.1.2.3.2.cmml"><mi id="S3.SS3.p2.4.m1.1.2.3.2.2" xref="S3.SS3.p2.4.m1.1.2.3.2.2.cmml">T</mi><mi id="S3.SS3.p2.4.m1.1.2.3.2.3" xref="S3.SS3.p2.4.m1.1.2.3.2.3.cmml">i</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.4.m1.1b"><apply id="S3.SS3.p2.4.m1.1.2.cmml" xref="S3.SS3.p2.4.m1.1.2"><eq id="S3.SS3.p2.4.m1.1.2.1.cmml" xref="S3.SS3.p2.4.m1.1.2.1"></eq><ci id="S3.SS3.p2.4.m1.1.2.2.cmml" xref="S3.SS3.p2.4.m1.1.2.2">𝑡</ci><apply id="S3.SS3.p2.4.m1.1.2.3.cmml" xref="S3.SS3.p2.4.m1.1.2.3"><apply id="S3.SS3.p2.4.m1.1.2.3.1.cmml" xref="S3.SS3.p2.4.m1.1.2.3.1"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m1.1.2.3.1.1.cmml" xref="S3.SS3.p2.4.m1.1.2.3.1">superscript</csymbol><apply id="S3.SS3.p2.4.m1.1.2.3.1.2.cmml" xref="S3.SS3.p2.4.m1.1.2.3.1"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m1.1.2.3.1.2.1.cmml" xref="S3.SS3.p2.4.m1.1.2.3.1">subscript</csymbol><sum id="S3.SS3.p2.4.m1.1.2.3.1.2.2.cmml" xref="S3.SS3.p2.4.m1.1.2.3.1.2.2"></sum><apply id="S3.SS3.p2.4.m1.1.2.3.1.2.3.cmml" xref="S3.SS3.p2.4.m1.1.2.3.1.2.3"><eq id="S3.SS3.p2.4.m1.1.2.3.1.2.3.1.cmml" xref="S3.SS3.p2.4.m1.1.2.3.1.2.3.1"></eq><ci id="S3.SS3.p2.4.m1.1.2.3.1.2.3.2.cmml" xref="S3.SS3.p2.4.m1.1.2.3.1.2.3.2">𝑖</ci><cn type="integer" id="S3.SS3.p2.4.m1.1.2.3.1.2.3.3.cmml" xref="S3.SS3.p2.4.m1.1.2.3.1.2.3.3">1</cn></apply></apply><apply id="S3.SS3.p2.4.m1.1.1.1.2.cmml" xref="S3.SS3.p2.4.m1.1.1.1.3"><abs id="S3.SS3.p2.4.m1.1.1.1.2.1.cmml" xref="S3.SS3.p2.4.m1.1.1.1.3.1"></abs><ci id="S3.SS3.p2.4.m1.1.1.1.1.cmml" xref="S3.SS3.p2.4.m1.1.1.1.1">𝒯</ci></apply></apply><apply id="S3.SS3.p2.4.m1.1.2.3.2.cmml" xref="S3.SS3.p2.4.m1.1.2.3.2"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m1.1.2.3.2.1.cmml" xref="S3.SS3.p2.4.m1.1.2.3.2">subscript</csymbol><ci id="S3.SS3.p2.4.m1.1.2.3.2.2.cmml" xref="S3.SS3.p2.4.m1.1.2.3.2.2">𝑇</ci><ci id="S3.SS3.p2.4.m1.1.2.3.2.3.cmml" xref="S3.SS3.p2.4.m1.1.2.3.2.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.4.m1.1c">t=\sum_{i=1}^{|\mathcal{T}|}T_{i}</annotation></semantics></math> and <math id="S3.SS3.p2.5.m2.1" class="ltx_Math" alttext="\Delta W_{T_{i}}=P_{T_{i}}B_{T_{i}}" display="inline"><semantics id="S3.SS3.p2.5.m2.1a"><mrow id="S3.SS3.p2.5.m2.1.1" xref="S3.SS3.p2.5.m2.1.1.cmml"><mrow id="S3.SS3.p2.5.m2.1.1.2" xref="S3.SS3.p2.5.m2.1.1.2.cmml"><mi mathvariant="normal" id="S3.SS3.p2.5.m2.1.1.2.2" xref="S3.SS3.p2.5.m2.1.1.2.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.5.m2.1.1.2.1" xref="S3.SS3.p2.5.m2.1.1.2.1.cmml">​</mo><msub id="S3.SS3.p2.5.m2.1.1.2.3" xref="S3.SS3.p2.5.m2.1.1.2.3.cmml"><mi id="S3.SS3.p2.5.m2.1.1.2.3.2" xref="S3.SS3.p2.5.m2.1.1.2.3.2.cmml">W</mi><msub id="S3.SS3.p2.5.m2.1.1.2.3.3" xref="S3.SS3.p2.5.m2.1.1.2.3.3.cmml"><mi id="S3.SS3.p2.5.m2.1.1.2.3.3.2" xref="S3.SS3.p2.5.m2.1.1.2.3.3.2.cmml">T</mi><mi id="S3.SS3.p2.5.m2.1.1.2.3.3.3" xref="S3.SS3.p2.5.m2.1.1.2.3.3.3.cmml">i</mi></msub></msub></mrow><mo id="S3.SS3.p2.5.m2.1.1.1" xref="S3.SS3.p2.5.m2.1.1.1.cmml">=</mo><mrow id="S3.SS3.p2.5.m2.1.1.3" xref="S3.SS3.p2.5.m2.1.1.3.cmml"><msub id="S3.SS3.p2.5.m2.1.1.3.2" xref="S3.SS3.p2.5.m2.1.1.3.2.cmml"><mi id="S3.SS3.p2.5.m2.1.1.3.2.2" xref="S3.SS3.p2.5.m2.1.1.3.2.2.cmml">P</mi><msub id="S3.SS3.p2.5.m2.1.1.3.2.3" xref="S3.SS3.p2.5.m2.1.1.3.2.3.cmml"><mi id="S3.SS3.p2.5.m2.1.1.3.2.3.2" xref="S3.SS3.p2.5.m2.1.1.3.2.3.2.cmml">T</mi><mi id="S3.SS3.p2.5.m2.1.1.3.2.3.3" xref="S3.SS3.p2.5.m2.1.1.3.2.3.3.cmml">i</mi></msub></msub><mo lspace="0em" rspace="0em" id="S3.SS3.p2.5.m2.1.1.3.1" xref="S3.SS3.p2.5.m2.1.1.3.1.cmml">​</mo><msub id="S3.SS3.p2.5.m2.1.1.3.3" xref="S3.SS3.p2.5.m2.1.1.3.3.cmml"><mi id="S3.SS3.p2.5.m2.1.1.3.3.2" xref="S3.SS3.p2.5.m2.1.1.3.3.2.cmml">B</mi><msub id="S3.SS3.p2.5.m2.1.1.3.3.3" xref="S3.SS3.p2.5.m2.1.1.3.3.3.cmml"><mi id="S3.SS3.p2.5.m2.1.1.3.3.3.2" xref="S3.SS3.p2.5.m2.1.1.3.3.3.2.cmml">T</mi><mi id="S3.SS3.p2.5.m2.1.1.3.3.3.3" xref="S3.SS3.p2.5.m2.1.1.3.3.3.3.cmml">i</mi></msub></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.5.m2.1b"><apply id="S3.SS3.p2.5.m2.1.1.cmml" xref="S3.SS3.p2.5.m2.1.1"><eq id="S3.SS3.p2.5.m2.1.1.1.cmml" xref="S3.SS3.p2.5.m2.1.1.1"></eq><apply id="S3.SS3.p2.5.m2.1.1.2.cmml" xref="S3.SS3.p2.5.m2.1.1.2"><times id="S3.SS3.p2.5.m2.1.1.2.1.cmml" xref="S3.SS3.p2.5.m2.1.1.2.1"></times><ci id="S3.SS3.p2.5.m2.1.1.2.2.cmml" xref="S3.SS3.p2.5.m2.1.1.2.2">Δ</ci><apply id="S3.SS3.p2.5.m2.1.1.2.3.cmml" xref="S3.SS3.p2.5.m2.1.1.2.3"><csymbol cd="ambiguous" id="S3.SS3.p2.5.m2.1.1.2.3.1.cmml" xref="S3.SS3.p2.5.m2.1.1.2.3">subscript</csymbol><ci id="S3.SS3.p2.5.m2.1.1.2.3.2.cmml" xref="S3.SS3.p2.5.m2.1.1.2.3.2">𝑊</ci><apply id="S3.SS3.p2.5.m2.1.1.2.3.3.cmml" xref="S3.SS3.p2.5.m2.1.1.2.3.3"><csymbol cd="ambiguous" id="S3.SS3.p2.5.m2.1.1.2.3.3.1.cmml" xref="S3.SS3.p2.5.m2.1.1.2.3.3">subscript</csymbol><ci id="S3.SS3.p2.5.m2.1.1.2.3.3.2.cmml" xref="S3.SS3.p2.5.m2.1.1.2.3.3.2">𝑇</ci><ci id="S3.SS3.p2.5.m2.1.1.2.3.3.3.cmml" xref="S3.SS3.p2.5.m2.1.1.2.3.3.3">𝑖</ci></apply></apply></apply><apply id="S3.SS3.p2.5.m2.1.1.3.cmml" xref="S3.SS3.p2.5.m2.1.1.3"><times id="S3.SS3.p2.5.m2.1.1.3.1.cmml" xref="S3.SS3.p2.5.m2.1.1.3.1"></times><apply id="S3.SS3.p2.5.m2.1.1.3.2.cmml" xref="S3.SS3.p2.5.m2.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS3.p2.5.m2.1.1.3.2.1.cmml" xref="S3.SS3.p2.5.m2.1.1.3.2">subscript</csymbol><ci id="S3.SS3.p2.5.m2.1.1.3.2.2.cmml" xref="S3.SS3.p2.5.m2.1.1.3.2.2">𝑃</ci><apply id="S3.SS3.p2.5.m2.1.1.3.2.3.cmml" xref="S3.SS3.p2.5.m2.1.1.3.2.3"><csymbol cd="ambiguous" id="S3.SS3.p2.5.m2.1.1.3.2.3.1.cmml" xref="S3.SS3.p2.5.m2.1.1.3.2.3">subscript</csymbol><ci id="S3.SS3.p2.5.m2.1.1.3.2.3.2.cmml" xref="S3.SS3.p2.5.m2.1.1.3.2.3.2">𝑇</ci><ci id="S3.SS3.p2.5.m2.1.1.3.2.3.3.cmml" xref="S3.SS3.p2.5.m2.1.1.3.2.3.3">𝑖</ci></apply></apply><apply id="S3.SS3.p2.5.m2.1.1.3.3.cmml" xref="S3.SS3.p2.5.m2.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS3.p2.5.m2.1.1.3.3.1.cmml" xref="S3.SS3.p2.5.m2.1.1.3.3">subscript</csymbol><ci id="S3.SS3.p2.5.m2.1.1.3.3.2.cmml" xref="S3.SS3.p2.5.m2.1.1.3.3.2">𝐵</ci><apply id="S3.SS3.p2.5.m2.1.1.3.3.3.cmml" xref="S3.SS3.p2.5.m2.1.1.3.3.3"><csymbol cd="ambiguous" id="S3.SS3.p2.5.m2.1.1.3.3.3.1.cmml" xref="S3.SS3.p2.5.m2.1.1.3.3.3">subscript</csymbol><ci id="S3.SS3.p2.5.m2.1.1.3.3.3.2.cmml" xref="S3.SS3.p2.5.m2.1.1.3.3.3.2">𝑇</ci><ci id="S3.SS3.p2.5.m2.1.1.3.3.3.3.cmml" xref="S3.SS3.p2.5.m2.1.1.3.3.3.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.5.m2.1c">\Delta W_{T_{i}}=P_{T_{i}}B_{T_{i}}</annotation></semantics></math> represents the product of the learning from <math id="S3.SS3.p2.6.m3.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S3.SS3.p2.6.m3.1a"><mi id="S3.SS3.p2.6.m3.1.1" xref="S3.SS3.p2.6.m3.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.6.m3.1b"><ci id="S3.SS3.p2.6.m3.1.1.cmml" xref="S3.SS3.p2.6.m3.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.6.m3.1c">B</annotation></semantics></math> during the interval <math id="S3.SS3.p2.7.m4.1" class="ltx_Math" alttext="T_{i}-T_{i-1}" display="inline"><semantics id="S3.SS3.p2.7.m4.1a"><mrow id="S3.SS3.p2.7.m4.1.1" xref="S3.SS3.p2.7.m4.1.1.cmml"><msub id="S3.SS3.p2.7.m4.1.1.2" xref="S3.SS3.p2.7.m4.1.1.2.cmml"><mi id="S3.SS3.p2.7.m4.1.1.2.2" xref="S3.SS3.p2.7.m4.1.1.2.2.cmml">T</mi><mi id="S3.SS3.p2.7.m4.1.1.2.3" xref="S3.SS3.p2.7.m4.1.1.2.3.cmml">i</mi></msub><mo id="S3.SS3.p2.7.m4.1.1.1" xref="S3.SS3.p2.7.m4.1.1.1.cmml">−</mo><msub id="S3.SS3.p2.7.m4.1.1.3" xref="S3.SS3.p2.7.m4.1.1.3.cmml"><mi id="S3.SS3.p2.7.m4.1.1.3.2" xref="S3.SS3.p2.7.m4.1.1.3.2.cmml">T</mi><mrow id="S3.SS3.p2.7.m4.1.1.3.3" xref="S3.SS3.p2.7.m4.1.1.3.3.cmml"><mi id="S3.SS3.p2.7.m4.1.1.3.3.2" xref="S3.SS3.p2.7.m4.1.1.3.3.2.cmml">i</mi><mo id="S3.SS3.p2.7.m4.1.1.3.3.1" xref="S3.SS3.p2.7.m4.1.1.3.3.1.cmml">−</mo><mn id="S3.SS3.p2.7.m4.1.1.3.3.3" xref="S3.SS3.p2.7.m4.1.1.3.3.3.cmml">1</mn></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.7.m4.1b"><apply id="S3.SS3.p2.7.m4.1.1.cmml" xref="S3.SS3.p2.7.m4.1.1"><minus id="S3.SS3.p2.7.m4.1.1.1.cmml" xref="S3.SS3.p2.7.m4.1.1.1"></minus><apply id="S3.SS3.p2.7.m4.1.1.2.cmml" xref="S3.SS3.p2.7.m4.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p2.7.m4.1.1.2.1.cmml" xref="S3.SS3.p2.7.m4.1.1.2">subscript</csymbol><ci id="S3.SS3.p2.7.m4.1.1.2.2.cmml" xref="S3.SS3.p2.7.m4.1.1.2.2">𝑇</ci><ci id="S3.SS3.p2.7.m4.1.1.2.3.cmml" xref="S3.SS3.p2.7.m4.1.1.2.3">𝑖</ci></apply><apply id="S3.SS3.p2.7.m4.1.1.3.cmml" xref="S3.SS3.p2.7.m4.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p2.7.m4.1.1.3.1.cmml" xref="S3.SS3.p2.7.m4.1.1.3">subscript</csymbol><ci id="S3.SS3.p2.7.m4.1.1.3.2.cmml" xref="S3.SS3.p2.7.m4.1.1.3.2">𝑇</ci><apply id="S3.SS3.p2.7.m4.1.1.3.3.cmml" xref="S3.SS3.p2.7.m4.1.1.3.3"><minus id="S3.SS3.p2.7.m4.1.1.3.3.1.cmml" xref="S3.SS3.p2.7.m4.1.1.3.3.1"></minus><ci id="S3.SS3.p2.7.m4.1.1.3.3.2.cmml" xref="S3.SS3.p2.7.m4.1.1.3.3.2">𝑖</ci><cn type="integer" id="S3.SS3.p2.7.m4.1.1.3.3.3.cmml" xref="S3.SS3.p2.7.m4.1.1.3.3.3">1</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.7.m4.1c">T_{i}-T_{i-1}</annotation></semantics></math> scaled by the learning rate <math id="S3.SS3.p2.8.m5.1" class="ltx_Math" alttext="\eta" display="inline"><semantics id="S3.SS3.p2.8.m5.1a"><mi id="S3.SS3.p2.8.m5.1.1" xref="S3.SS3.p2.8.m5.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.8.m5.1b"><ci id="S3.SS3.p2.8.m5.1.1.cmml" xref="S3.SS3.p2.8.m5.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.8.m5.1c">\eta</annotation></semantics></math> and modulated by the gradient projection matrix <math id="S3.SS3.p2.9.m6.1" class="ltx_Math" alttext="P_{T_{i}}" display="inline"><semantics id="S3.SS3.p2.9.m6.1a"><msub id="S3.SS3.p2.9.m6.1.1" xref="S3.SS3.p2.9.m6.1.1.cmml"><mi id="S3.SS3.p2.9.m6.1.1.2" xref="S3.SS3.p2.9.m6.1.1.2.cmml">P</mi><msub id="S3.SS3.p2.9.m6.1.1.3" xref="S3.SS3.p2.9.m6.1.1.3.cmml"><mi id="S3.SS3.p2.9.m6.1.1.3.2" xref="S3.SS3.p2.9.m6.1.1.3.2.cmml">T</mi><mi id="S3.SS3.p2.9.m6.1.1.3.3" xref="S3.SS3.p2.9.m6.1.1.3.3.cmml">i</mi></msub></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.9.m6.1b"><apply id="S3.SS3.p2.9.m6.1.1.cmml" xref="S3.SS3.p2.9.m6.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.9.m6.1.1.1.cmml" xref="S3.SS3.p2.9.m6.1.1">subscript</csymbol><ci id="S3.SS3.p2.9.m6.1.1.2.cmml" xref="S3.SS3.p2.9.m6.1.1.2">𝑃</ci><apply id="S3.SS3.p2.9.m6.1.1.3.cmml" xref="S3.SS3.p2.9.m6.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p2.9.m6.1.1.3.1.cmml" xref="S3.SS3.p2.9.m6.1.1.3">subscript</csymbol><ci id="S3.SS3.p2.9.m6.1.1.3.2.cmml" xref="S3.SS3.p2.9.m6.1.1.3.2">𝑇</ci><ci id="S3.SS3.p2.9.m6.1.1.3.3.cmml" xref="S3.SS3.p2.9.m6.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.9.m6.1c">P_{T_{i}}</annotation></semantics></math>.
After each update at iteration <math id="S3.SS3.p2.10.m7.1" class="ltx_Math" alttext="T_{i}\in\mathcal{T}" display="inline"><semantics id="S3.SS3.p2.10.m7.1a"><mrow id="S3.SS3.p2.10.m7.1.1" xref="S3.SS3.p2.10.m7.1.1.cmml"><msub id="S3.SS3.p2.10.m7.1.1.2" xref="S3.SS3.p2.10.m7.1.1.2.cmml"><mi id="S3.SS3.p2.10.m7.1.1.2.2" xref="S3.SS3.p2.10.m7.1.1.2.2.cmml">T</mi><mi id="S3.SS3.p2.10.m7.1.1.2.3" xref="S3.SS3.p2.10.m7.1.1.2.3.cmml">i</mi></msub><mo id="S3.SS3.p2.10.m7.1.1.1" xref="S3.SS3.p2.10.m7.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.10.m7.1.1.3" xref="S3.SS3.p2.10.m7.1.1.3.cmml">𝒯</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.10.m7.1b"><apply id="S3.SS3.p2.10.m7.1.1.cmml" xref="S3.SS3.p2.10.m7.1.1"><in id="S3.SS3.p2.10.m7.1.1.1.cmml" xref="S3.SS3.p2.10.m7.1.1.1"></in><apply id="S3.SS3.p2.10.m7.1.1.2.cmml" xref="S3.SS3.p2.10.m7.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p2.10.m7.1.1.2.1.cmml" xref="S3.SS3.p2.10.m7.1.1.2">subscript</csymbol><ci id="S3.SS3.p2.10.m7.1.1.2.2.cmml" xref="S3.SS3.p2.10.m7.1.1.2.2">𝑇</ci><ci id="S3.SS3.p2.10.m7.1.1.2.3.cmml" xref="S3.SS3.p2.10.m7.1.1.2.3">𝑖</ci></apply><ci id="S3.SS3.p2.10.m7.1.1.3.cmml" xref="S3.SS3.p2.10.m7.1.1.3">𝒯</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.10.m7.1c">T_{i}\in\mathcal{T}</annotation></semantics></math>, we reinitialize the low-rank factors <math id="S3.SS3.p2.11.m8.1" class="ltx_Math" alttext="P_{T}" display="inline"><semantics id="S3.SS3.p2.11.m8.1a"><msub id="S3.SS3.p2.11.m8.1.1" xref="S3.SS3.p2.11.m8.1.1.cmml"><mi id="S3.SS3.p2.11.m8.1.1.2" xref="S3.SS3.p2.11.m8.1.1.2.cmml">P</mi><mi id="S3.SS3.p2.11.m8.1.1.3" xref="S3.SS3.p2.11.m8.1.1.3.cmml">T</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.11.m8.1b"><apply id="S3.SS3.p2.11.m8.1.1.cmml" xref="S3.SS3.p2.11.m8.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.11.m8.1.1.1.cmml" xref="S3.SS3.p2.11.m8.1.1">subscript</csymbol><ci id="S3.SS3.p2.11.m8.1.1.2.cmml" xref="S3.SS3.p2.11.m8.1.1.2">𝑃</ci><ci id="S3.SS3.p2.11.m8.1.1.3.cmml" xref="S3.SS3.p2.11.m8.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.11.m8.1c">P_{T}</annotation></semantics></math> and <math id="S3.SS3.p2.12.m9.1" class="ltx_Math" alttext="B_{T}" display="inline"><semantics id="S3.SS3.p2.12.m9.1a"><msub id="S3.SS3.p2.12.m9.1.1" xref="S3.SS3.p2.12.m9.1.1.cmml"><mi id="S3.SS3.p2.12.m9.1.1.2" xref="S3.SS3.p2.12.m9.1.1.2.cmml">B</mi><mi id="S3.SS3.p2.12.m9.1.1.3" xref="S3.SS3.p2.12.m9.1.1.3.cmml">T</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.12.m9.1b"><apply id="S3.SS3.p2.12.m9.1.1.cmml" xref="S3.SS3.p2.12.m9.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.12.m9.1.1.1.cmml" xref="S3.SS3.p2.12.m9.1.1">subscript</csymbol><ci id="S3.SS3.p2.12.m9.1.1.2.cmml" xref="S3.SS3.p2.12.m9.1.1.2">𝐵</ci><ci id="S3.SS3.p2.12.m9.1.1.3.cmml" xref="S3.SS3.p2.12.m9.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.12.m9.1c">B_{T}</annotation></semantics></math>. As in <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite>, we compute the gradient of <math id="S3.SS3.p2.13.m10.1" class="ltx_Math" alttext="W_{T}" display="inline"><semantics id="S3.SS3.p2.13.m10.1a"><msub id="S3.SS3.p2.13.m10.1.1" xref="S3.SS3.p2.13.m10.1.1.cmml"><mi id="S3.SS3.p2.13.m10.1.1.2" xref="S3.SS3.p2.13.m10.1.1.2.cmml">W</mi><mi id="S3.SS3.p2.13.m10.1.1.3" xref="S3.SS3.p2.13.m10.1.1.3.cmml">T</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.13.m10.1b"><apply id="S3.SS3.p2.13.m10.1.1.cmml" xref="S3.SS3.p2.13.m10.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.13.m10.1.1.1.cmml" xref="S3.SS3.p2.13.m10.1.1">subscript</csymbol><ci id="S3.SS3.p2.13.m10.1.1.2.cmml" xref="S3.SS3.p2.13.m10.1.1.2">𝑊</ci><ci id="S3.SS3.p2.13.m10.1.1.3.cmml" xref="S3.SS3.p2.13.m10.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.13.m10.1c">W_{T}</annotation></semantics></math> over a single batch, focusing only on <math id="S3.SS3.p2.14.m11.1" class="ltx_Math" alttext="\nabla W_{T}" display="inline"><semantics id="S3.SS3.p2.14.m11.1a"><mrow id="S3.SS3.p2.14.m11.1.1" xref="S3.SS3.p2.14.m11.1.1.cmml"><mo rspace="0.167em" id="S3.SS3.p2.14.m11.1.1.1" xref="S3.SS3.p2.14.m11.1.1.1.cmml">∇</mo><msub id="S3.SS3.p2.14.m11.1.1.2" xref="S3.SS3.p2.14.m11.1.1.2.cmml"><mi id="S3.SS3.p2.14.m11.1.1.2.2" xref="S3.SS3.p2.14.m11.1.1.2.2.cmml">W</mi><mi id="S3.SS3.p2.14.m11.1.1.2.3" xref="S3.SS3.p2.14.m11.1.1.2.3.cmml">T</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.14.m11.1b"><apply id="S3.SS3.p2.14.m11.1.1.cmml" xref="S3.SS3.p2.14.m11.1.1"><ci id="S3.SS3.p2.14.m11.1.1.1.cmml" xref="S3.SS3.p2.14.m11.1.1.1">∇</ci><apply id="S3.SS3.p2.14.m11.1.1.2.cmml" xref="S3.SS3.p2.14.m11.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p2.14.m11.1.1.2.1.cmml" xref="S3.SS3.p2.14.m11.1.1.2">subscript</csymbol><ci id="S3.SS3.p2.14.m11.1.1.2.2.cmml" xref="S3.SS3.p2.14.m11.1.1.2.2">𝑊</ci><ci id="S3.SS3.p2.14.m11.1.1.2.3.cmml" xref="S3.SS3.p2.14.m11.1.1.2.3">𝑇</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.14.m11.1c">\nabla W_{T}</annotation></semantics></math> without needing full optimizer states. Not requiring optimizer states reduces the memory increase compared to full-rank training.</p>
</div>
<div id="S3.SS3.p3" class="ltx_para">
<p id="S3.SS3.p3.11" class="ltx_p">With the updated <math id="S3.SS3.p3.1.m1.1" class="ltx_Math" alttext="W_{t}" display="inline"><semantics id="S3.SS3.p3.1.m1.1a"><msub id="S3.SS3.p3.1.m1.1.1" xref="S3.SS3.p3.1.m1.1.1.cmml"><mi id="S3.SS3.p3.1.m1.1.1.2" xref="S3.SS3.p3.1.m1.1.1.2.cmml">W</mi><mi id="S3.SS3.p3.1.m1.1.1.3" xref="S3.SS3.p3.1.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.1.m1.1b"><apply id="S3.SS3.p3.1.m1.1.1.cmml" xref="S3.SS3.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p3.1.m1.1.1.1.cmml" xref="S3.SS3.p3.1.m1.1.1">subscript</csymbol><ci id="S3.SS3.p3.1.m1.1.1.2.cmml" xref="S3.SS3.p3.1.m1.1.1.2">𝑊</ci><ci id="S3.SS3.p3.1.m1.1.1.3.cmml" xref="S3.SS3.p3.1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.1.m1.1c">W_{t}</annotation></semantics></math> and reinitialized <math id="S3.SS3.p3.2.m2.1" class="ltx_Math" alttext="P_{t}" display="inline"><semantics id="S3.SS3.p3.2.m2.1a"><msub id="S3.SS3.p3.2.m2.1.1" xref="S3.SS3.p3.2.m2.1.1.cmml"><mi id="S3.SS3.p3.2.m2.1.1.2" xref="S3.SS3.p3.2.m2.1.1.2.cmml">P</mi><mi id="S3.SS3.p3.2.m2.1.1.3" xref="S3.SS3.p3.2.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.2.m2.1b"><apply id="S3.SS3.p3.2.m2.1.1.cmml" xref="S3.SS3.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS3.p3.2.m2.1.1.1.cmml" xref="S3.SS3.p3.2.m2.1.1">subscript</csymbol><ci id="S3.SS3.p3.2.m2.1.1.2.cmml" xref="S3.SS3.p3.2.m2.1.1.2">𝑃</ci><ci id="S3.SS3.p3.2.m2.1.1.3.cmml" xref="S3.SS3.p3.2.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.2.m2.1c">P_{t}</annotation></semantics></math> and <math id="S3.SS3.p3.3.m3.1" class="ltx_Math" alttext="B_{t}" display="inline"><semantics id="S3.SS3.p3.3.m3.1a"><msub id="S3.SS3.p3.3.m3.1.1" xref="S3.SS3.p3.3.m3.1.1.cmml"><mi id="S3.SS3.p3.3.m3.1.1.2" xref="S3.SS3.p3.3.m3.1.1.2.cmml">B</mi><mi id="S3.SS3.p3.3.m3.1.1.3" xref="S3.SS3.p3.3.m3.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.3.m3.1b"><apply id="S3.SS3.p3.3.m3.1.1.cmml" xref="S3.SS3.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS3.p3.3.m3.1.1.1.cmml" xref="S3.SS3.p3.3.m3.1.1">subscript</csymbol><ci id="S3.SS3.p3.3.m3.1.1.2.cmml" xref="S3.SS3.p3.3.m3.1.1.2">𝐵</ci><ci id="S3.SS3.p3.3.m3.1.1.3.cmml" xref="S3.SS3.p3.3.m3.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.3.m3.1c">B_{t}</annotation></semantics></math>, a new gradient subspace is established for exploring the next <math id="S3.SS3.p3.4.m4.1" class="ltx_Math" alttext="T_{i+1}-T_{i}" display="inline"><semantics id="S3.SS3.p3.4.m4.1a"><mrow id="S3.SS3.p3.4.m4.1.1" xref="S3.SS3.p3.4.m4.1.1.cmml"><msub id="S3.SS3.p3.4.m4.1.1.2" xref="S3.SS3.p3.4.m4.1.1.2.cmml"><mi id="S3.SS3.p3.4.m4.1.1.2.2" xref="S3.SS3.p3.4.m4.1.1.2.2.cmml">T</mi><mrow id="S3.SS3.p3.4.m4.1.1.2.3" xref="S3.SS3.p3.4.m4.1.1.2.3.cmml"><mi id="S3.SS3.p3.4.m4.1.1.2.3.2" xref="S3.SS3.p3.4.m4.1.1.2.3.2.cmml">i</mi><mo id="S3.SS3.p3.4.m4.1.1.2.3.1" xref="S3.SS3.p3.4.m4.1.1.2.3.1.cmml">+</mo><mn id="S3.SS3.p3.4.m4.1.1.2.3.3" xref="S3.SS3.p3.4.m4.1.1.2.3.3.cmml">1</mn></mrow></msub><mo id="S3.SS3.p3.4.m4.1.1.1" xref="S3.SS3.p3.4.m4.1.1.1.cmml">−</mo><msub id="S3.SS3.p3.4.m4.1.1.3" xref="S3.SS3.p3.4.m4.1.1.3.cmml"><mi id="S3.SS3.p3.4.m4.1.1.3.2" xref="S3.SS3.p3.4.m4.1.1.3.2.cmml">T</mi><mi id="S3.SS3.p3.4.m4.1.1.3.3" xref="S3.SS3.p3.4.m4.1.1.3.3.cmml">i</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.4.m4.1b"><apply id="S3.SS3.p3.4.m4.1.1.cmml" xref="S3.SS3.p3.4.m4.1.1"><minus id="S3.SS3.p3.4.m4.1.1.1.cmml" xref="S3.SS3.p3.4.m4.1.1.1"></minus><apply id="S3.SS3.p3.4.m4.1.1.2.cmml" xref="S3.SS3.p3.4.m4.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p3.4.m4.1.1.2.1.cmml" xref="S3.SS3.p3.4.m4.1.1.2">subscript</csymbol><ci id="S3.SS3.p3.4.m4.1.1.2.2.cmml" xref="S3.SS3.p3.4.m4.1.1.2.2">𝑇</ci><apply id="S3.SS3.p3.4.m4.1.1.2.3.cmml" xref="S3.SS3.p3.4.m4.1.1.2.3"><plus id="S3.SS3.p3.4.m4.1.1.2.3.1.cmml" xref="S3.SS3.p3.4.m4.1.1.2.3.1"></plus><ci id="S3.SS3.p3.4.m4.1.1.2.3.2.cmml" xref="S3.SS3.p3.4.m4.1.1.2.3.2">𝑖</ci><cn type="integer" id="S3.SS3.p3.4.m4.1.1.2.3.3.cmml" xref="S3.SS3.p3.4.m4.1.1.2.3.3">1</cn></apply></apply><apply id="S3.SS3.p3.4.m4.1.1.3.cmml" xref="S3.SS3.p3.4.m4.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p3.4.m4.1.1.3.1.cmml" xref="S3.SS3.p3.4.m4.1.1.3">subscript</csymbol><ci id="S3.SS3.p3.4.m4.1.1.3.2.cmml" xref="S3.SS3.p3.4.m4.1.1.3.2">𝑇</ci><ci id="S3.SS3.p3.4.m4.1.1.3.3.cmml" xref="S3.SS3.p3.4.m4.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.4.m4.1c">T_{i+1}-T_{i}</annotation></semantics></math> steps. Our method treats <math id="S3.SS3.p3.5.m5.1" class="ltx_Math" alttext="W_{t}" display="inline"><semantics id="S3.SS3.p3.5.m5.1a"><msub id="S3.SS3.p3.5.m5.1.1" xref="S3.SS3.p3.5.m5.1.1.cmml"><mi id="S3.SS3.p3.5.m5.1.1.2" xref="S3.SS3.p3.5.m5.1.1.2.cmml">W</mi><mi id="S3.SS3.p3.5.m5.1.1.3" xref="S3.SS3.p3.5.m5.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.5.m5.1b"><apply id="S3.SS3.p3.5.m5.1.1.cmml" xref="S3.SS3.p3.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS3.p3.5.m5.1.1.1.cmml" xref="S3.SS3.p3.5.m5.1.1">subscript</csymbol><ci id="S3.SS3.p3.5.m5.1.1.2.cmml" xref="S3.SS3.p3.5.m5.1.1.2">𝑊</ci><ci id="S3.SS3.p3.5.m5.1.1.3.cmml" xref="S3.SS3.p3.5.m5.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.5.m5.1c">W_{t}</annotation></semantics></math> as the full-rank repository of accumulated updates. Although it is periodically updated, <math id="S3.SS3.p3.6.m6.1" class="ltx_Math" alttext="W_{t}" display="inline"><semantics id="S3.SS3.p3.6.m6.1a"><msub id="S3.SS3.p3.6.m6.1.1" xref="S3.SS3.p3.6.m6.1.1.cmml"><mi id="S3.SS3.p3.6.m6.1.1.2" xref="S3.SS3.p3.6.m6.1.1.2.cmml">W</mi><mi id="S3.SS3.p3.6.m6.1.1.3" xref="S3.SS3.p3.6.m6.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.6.m6.1b"><apply id="S3.SS3.p3.6.m6.1.1.cmml" xref="S3.SS3.p3.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS3.p3.6.m6.1.1.1.cmml" xref="S3.SS3.p3.6.m6.1.1">subscript</csymbol><ci id="S3.SS3.p3.6.m6.1.1.2.cmml" xref="S3.SS3.p3.6.m6.1.1.2">𝑊</ci><ci id="S3.SS3.p3.6.m6.1.1.3.cmml" xref="S3.SS3.p3.6.m6.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.6.m6.1c">W_{t}</annotation></semantics></math> is not part of the optimizer state computations and the gradients during the single forward pass are offloaded to cpu. Since the SVD calculations are done layerwise only the current layer is needed on GPU, or the SVD can be calculated on CPU. <math id="S3.SS3.p3.7.m7.1" class="ltx_Math" alttext="P_{t}" display="inline"><semantics id="S3.SS3.p3.7.m7.1a"><msub id="S3.SS3.p3.7.m7.1.1" xref="S3.SS3.p3.7.m7.1.1.cmml"><mi id="S3.SS3.p3.7.m7.1.1.2" xref="S3.SS3.p3.7.m7.1.1.2.cmml">P</mi><mi id="S3.SS3.p3.7.m7.1.1.3" xref="S3.SS3.p3.7.m7.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.7.m7.1b"><apply id="S3.SS3.p3.7.m7.1.1.cmml" xref="S3.SS3.p3.7.m7.1.1"><csymbol cd="ambiguous" id="S3.SS3.p3.7.m7.1.1.1.cmml" xref="S3.SS3.p3.7.m7.1.1">subscript</csymbol><ci id="S3.SS3.p3.7.m7.1.1.2.cmml" xref="S3.SS3.p3.7.m7.1.1.2">𝑃</ci><ci id="S3.SS3.p3.7.m7.1.1.3.cmml" xref="S3.SS3.p3.7.m7.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.7.m7.1c">P_{t}</annotation></semantics></math> defines the general gradient subspace and trajectory for the upcoming <math id="S3.SS3.p3.8.m8.1" class="ltx_Math" alttext="T_{i+1}-T_{i}" display="inline"><semantics id="S3.SS3.p3.8.m8.1a"><mrow id="S3.SS3.p3.8.m8.1.1" xref="S3.SS3.p3.8.m8.1.1.cmml"><msub id="S3.SS3.p3.8.m8.1.1.2" xref="S3.SS3.p3.8.m8.1.1.2.cmml"><mi id="S3.SS3.p3.8.m8.1.1.2.2" xref="S3.SS3.p3.8.m8.1.1.2.2.cmml">T</mi><mrow id="S3.SS3.p3.8.m8.1.1.2.3" xref="S3.SS3.p3.8.m8.1.1.2.3.cmml"><mi id="S3.SS3.p3.8.m8.1.1.2.3.2" xref="S3.SS3.p3.8.m8.1.1.2.3.2.cmml">i</mi><mo id="S3.SS3.p3.8.m8.1.1.2.3.1" xref="S3.SS3.p3.8.m8.1.1.2.3.1.cmml">+</mo><mn id="S3.SS3.p3.8.m8.1.1.2.3.3" xref="S3.SS3.p3.8.m8.1.1.2.3.3.cmml">1</mn></mrow></msub><mo id="S3.SS3.p3.8.m8.1.1.1" xref="S3.SS3.p3.8.m8.1.1.1.cmml">−</mo><msub id="S3.SS3.p3.8.m8.1.1.3" xref="S3.SS3.p3.8.m8.1.1.3.cmml"><mi id="S3.SS3.p3.8.m8.1.1.3.2" xref="S3.SS3.p3.8.m8.1.1.3.2.cmml">T</mi><mi id="S3.SS3.p3.8.m8.1.1.3.3" xref="S3.SS3.p3.8.m8.1.1.3.3.cmml">i</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.8.m8.1b"><apply id="S3.SS3.p3.8.m8.1.1.cmml" xref="S3.SS3.p3.8.m8.1.1"><minus id="S3.SS3.p3.8.m8.1.1.1.cmml" xref="S3.SS3.p3.8.m8.1.1.1"></minus><apply id="S3.SS3.p3.8.m8.1.1.2.cmml" xref="S3.SS3.p3.8.m8.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p3.8.m8.1.1.2.1.cmml" xref="S3.SS3.p3.8.m8.1.1.2">subscript</csymbol><ci id="S3.SS3.p3.8.m8.1.1.2.2.cmml" xref="S3.SS3.p3.8.m8.1.1.2.2">𝑇</ci><apply id="S3.SS3.p3.8.m8.1.1.2.3.cmml" xref="S3.SS3.p3.8.m8.1.1.2.3"><plus id="S3.SS3.p3.8.m8.1.1.2.3.1.cmml" xref="S3.SS3.p3.8.m8.1.1.2.3.1"></plus><ci id="S3.SS3.p3.8.m8.1.1.2.3.2.cmml" xref="S3.SS3.p3.8.m8.1.1.2.3.2">𝑖</ci><cn type="integer" id="S3.SS3.p3.8.m8.1.1.2.3.3.cmml" xref="S3.SS3.p3.8.m8.1.1.2.3.3">1</cn></apply></apply><apply id="S3.SS3.p3.8.m8.1.1.3.cmml" xref="S3.SS3.p3.8.m8.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p3.8.m8.1.1.3.1.cmml" xref="S3.SS3.p3.8.m8.1.1.3">subscript</csymbol><ci id="S3.SS3.p3.8.m8.1.1.3.2.cmml" xref="S3.SS3.p3.8.m8.1.1.3.2">𝑇</ci><ci id="S3.SS3.p3.8.m8.1.1.3.3.cmml" xref="S3.SS3.p3.8.m8.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.8.m8.1c">T_{i+1}-T_{i}</annotation></semantics></math> steps, and <math id="S3.SS3.p3.9.m9.1" class="ltx_Math" alttext="B_{t}" display="inline"><semantics id="S3.SS3.p3.9.m9.1a"><msub id="S3.SS3.p3.9.m9.1.1" xref="S3.SS3.p3.9.m9.1.1.cmml"><mi id="S3.SS3.p3.9.m9.1.1.2" xref="S3.SS3.p3.9.m9.1.1.2.cmml">B</mi><mi id="S3.SS3.p3.9.m9.1.1.3" xref="S3.SS3.p3.9.m9.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.9.m9.1b"><apply id="S3.SS3.p3.9.m9.1.1.cmml" xref="S3.SS3.p3.9.m9.1.1"><csymbol cd="ambiguous" id="S3.SS3.p3.9.m9.1.1.1.cmml" xref="S3.SS3.p3.9.m9.1.1">subscript</csymbol><ci id="S3.SS3.p3.9.m9.1.1.2.cmml" xref="S3.SS3.p3.9.m9.1.1.2">𝐵</ci><ci id="S3.SS3.p3.9.m9.1.1.3.cmml" xref="S3.SS3.p3.9.m9.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.9.m9.1c">B_{t}</annotation></semantics></math> is adjusted to navigate optimally within the direction set by <math id="S3.SS3.p3.10.m10.1" class="ltx_Math" alttext="P_{t}" display="inline"><semantics id="S3.SS3.p3.10.m10.1a"><msub id="S3.SS3.p3.10.m10.1.1" xref="S3.SS3.p3.10.m10.1.1.cmml"><mi id="S3.SS3.p3.10.m10.1.1.2" xref="S3.SS3.p3.10.m10.1.1.2.cmml">P</mi><mi id="S3.SS3.p3.10.m10.1.1.3" xref="S3.SS3.p3.10.m10.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.10.m10.1b"><apply id="S3.SS3.p3.10.m10.1.1.cmml" xref="S3.SS3.p3.10.m10.1.1"><csymbol cd="ambiguous" id="S3.SS3.p3.10.m10.1.1.1.cmml" xref="S3.SS3.p3.10.m10.1.1">subscript</csymbol><ci id="S3.SS3.p3.10.m10.1.1.2.cmml" xref="S3.SS3.p3.10.m10.1.1.2">𝑃</ci><ci id="S3.SS3.p3.10.m10.1.1.3.cmml" xref="S3.SS3.p3.10.m10.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.10.m10.1c">P_{t}</annotation></semantics></math>.
As only <math id="S3.SS3.p3.11.m11.1" class="ltx_Math" alttext="B_{t}" display="inline"><semantics id="S3.SS3.p3.11.m11.1a"><msub id="S3.SS3.p3.11.m11.1.1" xref="S3.SS3.p3.11.m11.1.1.cmml"><mi id="S3.SS3.p3.11.m11.1.1.2" xref="S3.SS3.p3.11.m11.1.1.2.cmml">B</mi><mi id="S3.SS3.p3.11.m11.1.1.3" xref="S3.SS3.p3.11.m11.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.11.m11.1b"><apply id="S3.SS3.p3.11.m11.1.1.cmml" xref="S3.SS3.p3.11.m11.1.1"><csymbol cd="ambiguous" id="S3.SS3.p3.11.m11.1.1.1.cmml" xref="S3.SS3.p3.11.m11.1.1">subscript</csymbol><ci id="S3.SS3.p3.11.m11.1.1.2.cmml" xref="S3.SS3.p3.11.m11.1.1.2">𝐵</ci><ci id="S3.SS3.p3.11.m11.1.1.3.cmml" xref="S3.SS3.p3.11.m11.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.11.m11.1c">B_{t}</annotation></semantics></math> is trained, the number of parameters needing optimizer states is drastically reduced.</p>
</div>
</section>
<section id="S3.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.4 </span>Quantized Training</h3>

<div id="S3.SS4.p1" class="ltx_para">
<p id="S3.SS4.p1.5" class="ltx_p">Given that <math id="S3.SS4.p1.1.m1.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S3.SS4.p1.1.m1.1a"><mi id="S3.SS4.p1.1.m1.1.1" xref="S3.SS4.p1.1.m1.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.1.m1.1b"><ci id="S3.SS4.p1.1.m1.1.1.cmml" xref="S3.SS4.p1.1.m1.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.1.m1.1c">B</annotation></semantics></math> is the only matrix accumulating gradients and undergoing changes, the other matrices
<math id="S3.SS4.p1.2.m2.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S3.SS4.p1.2.m2.1a"><mi id="S3.SS4.p1.2.m2.1.1" xref="S3.SS4.p1.2.m2.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.2.m2.1b"><ci id="S3.SS4.p1.2.m2.1.1.cmml" xref="S3.SS4.p1.2.m2.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.2.m2.1c">W</annotation></semantics></math> and <math id="S3.SS4.p1.3.m3.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S3.SS4.p1.3.m3.1a"><mi id="S3.SS4.p1.3.m3.1.1" xref="S3.SS4.p1.3.m3.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.3.m3.1b"><ci id="S3.SS4.p1.3.m3.1.1.cmml" xref="S3.SS4.p1.3.m3.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.3.m3.1c">P</annotation></semantics></math> can be kept quantized. This approach allows storing the weights in NF4 precision without requiring high-precision gradient and weights to update <math id="S3.SS4.p1.4.m4.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S3.SS4.p1.4.m4.1a"><mi id="S3.SS4.p1.4.m4.1.1" xref="S3.SS4.p1.4.m4.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.4.m4.1b"><ci id="S3.SS4.p1.4.m4.1.1.cmml" xref="S3.SS4.p1.4.m4.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.4.m4.1c">W</annotation></semantics></math> and <math id="S3.SS4.p1.5.m5.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S3.SS4.p1.5.m5.1a"><mi id="S3.SS4.p1.5.m5.1.1" xref="S3.SS4.p1.5.m5.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.5.m5.1b"><ci id="S3.SS4.p1.5.m5.1.1.cmml" xref="S3.SS4.p1.5.m5.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.5.m5.1c">P</annotation></semantics></math>. To the best of our knowledge, we are the first to enable efficient 4-bit quantized training using gradient descent without storing the weights in full precision.</p>
</div>
<div id="S3.SS4.p2" class="ltx_para">
<p id="S3.SS4.p2.8" class="ltx_p">We quantize weights <math id="S3.SS4.p2.1.m1.1" class="ltx_Math" alttext="q_{\text{NF4}}(W)=W_{q}" display="inline"><semantics id="S3.SS4.p2.1.m1.1a"><mrow id="S3.SS4.p2.1.m1.1.2" xref="S3.SS4.p2.1.m1.1.2.cmml"><mrow id="S3.SS4.p2.1.m1.1.2.2" xref="S3.SS4.p2.1.m1.1.2.2.cmml"><msub id="S3.SS4.p2.1.m1.1.2.2.2" xref="S3.SS4.p2.1.m1.1.2.2.2.cmml"><mi id="S3.SS4.p2.1.m1.1.2.2.2.2" xref="S3.SS4.p2.1.m1.1.2.2.2.2.cmml">q</mi><mtext id="S3.SS4.p2.1.m1.1.2.2.2.3" xref="S3.SS4.p2.1.m1.1.2.2.2.3a.cmml">NF4</mtext></msub><mo lspace="0em" rspace="0em" id="S3.SS4.p2.1.m1.1.2.2.1" xref="S3.SS4.p2.1.m1.1.2.2.1.cmml">​</mo><mrow id="S3.SS4.p2.1.m1.1.2.2.3.2" xref="S3.SS4.p2.1.m1.1.2.2.cmml"><mo stretchy="false" id="S3.SS4.p2.1.m1.1.2.2.3.2.1" xref="S3.SS4.p2.1.m1.1.2.2.cmml">(</mo><mi id="S3.SS4.p2.1.m1.1.1" xref="S3.SS4.p2.1.m1.1.1.cmml">W</mi><mo stretchy="false" id="S3.SS4.p2.1.m1.1.2.2.3.2.2" xref="S3.SS4.p2.1.m1.1.2.2.cmml">)</mo></mrow></mrow><mo id="S3.SS4.p2.1.m1.1.2.1" xref="S3.SS4.p2.1.m1.1.2.1.cmml">=</mo><msub id="S3.SS4.p2.1.m1.1.2.3" xref="S3.SS4.p2.1.m1.1.2.3.cmml"><mi id="S3.SS4.p2.1.m1.1.2.3.2" xref="S3.SS4.p2.1.m1.1.2.3.2.cmml">W</mi><mi id="S3.SS4.p2.1.m1.1.2.3.3" xref="S3.SS4.p2.1.m1.1.2.3.3.cmml">q</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.1.m1.1b"><apply id="S3.SS4.p2.1.m1.1.2.cmml" xref="S3.SS4.p2.1.m1.1.2"><eq id="S3.SS4.p2.1.m1.1.2.1.cmml" xref="S3.SS4.p2.1.m1.1.2.1"></eq><apply id="S3.SS4.p2.1.m1.1.2.2.cmml" xref="S3.SS4.p2.1.m1.1.2.2"><times id="S3.SS4.p2.1.m1.1.2.2.1.cmml" xref="S3.SS4.p2.1.m1.1.2.2.1"></times><apply id="S3.SS4.p2.1.m1.1.2.2.2.cmml" xref="S3.SS4.p2.1.m1.1.2.2.2"><csymbol cd="ambiguous" id="S3.SS4.p2.1.m1.1.2.2.2.1.cmml" xref="S3.SS4.p2.1.m1.1.2.2.2">subscript</csymbol><ci id="S3.SS4.p2.1.m1.1.2.2.2.2.cmml" xref="S3.SS4.p2.1.m1.1.2.2.2.2">𝑞</ci><ci id="S3.SS4.p2.1.m1.1.2.2.2.3a.cmml" xref="S3.SS4.p2.1.m1.1.2.2.2.3"><mtext mathsize="70%" id="S3.SS4.p2.1.m1.1.2.2.2.3.cmml" xref="S3.SS4.p2.1.m1.1.2.2.2.3">NF4</mtext></ci></apply><ci id="S3.SS4.p2.1.m1.1.1.cmml" xref="S3.SS4.p2.1.m1.1.1">𝑊</ci></apply><apply id="S3.SS4.p2.1.m1.1.2.3.cmml" xref="S3.SS4.p2.1.m1.1.2.3"><csymbol cd="ambiguous" id="S3.SS4.p2.1.m1.1.2.3.1.cmml" xref="S3.SS4.p2.1.m1.1.2.3">subscript</csymbol><ci id="S3.SS4.p2.1.m1.1.2.3.2.cmml" xref="S3.SS4.p2.1.m1.1.2.3.2">𝑊</ci><ci id="S3.SS4.p2.1.m1.1.2.3.3.cmml" xref="S3.SS4.p2.1.m1.1.2.3.3">𝑞</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.1.m1.1c">q_{\text{NF4}}(W)=W_{q}</annotation></semantics></math> and <math id="S3.SS4.p2.2.m2.1" class="ltx_Math" alttext="q_{\text{NF4}}(P)=P_{q}" display="inline"><semantics id="S3.SS4.p2.2.m2.1a"><mrow id="S3.SS4.p2.2.m2.1.2" xref="S3.SS4.p2.2.m2.1.2.cmml"><mrow id="S3.SS4.p2.2.m2.1.2.2" xref="S3.SS4.p2.2.m2.1.2.2.cmml"><msub id="S3.SS4.p2.2.m2.1.2.2.2" xref="S3.SS4.p2.2.m2.1.2.2.2.cmml"><mi id="S3.SS4.p2.2.m2.1.2.2.2.2" xref="S3.SS4.p2.2.m2.1.2.2.2.2.cmml">q</mi><mtext id="S3.SS4.p2.2.m2.1.2.2.2.3" xref="S3.SS4.p2.2.m2.1.2.2.2.3a.cmml">NF4</mtext></msub><mo lspace="0em" rspace="0em" id="S3.SS4.p2.2.m2.1.2.2.1" xref="S3.SS4.p2.2.m2.1.2.2.1.cmml">​</mo><mrow id="S3.SS4.p2.2.m2.1.2.2.3.2" xref="S3.SS4.p2.2.m2.1.2.2.cmml"><mo stretchy="false" id="S3.SS4.p2.2.m2.1.2.2.3.2.1" xref="S3.SS4.p2.2.m2.1.2.2.cmml">(</mo><mi id="S3.SS4.p2.2.m2.1.1" xref="S3.SS4.p2.2.m2.1.1.cmml">P</mi><mo stretchy="false" id="S3.SS4.p2.2.m2.1.2.2.3.2.2" xref="S3.SS4.p2.2.m2.1.2.2.cmml">)</mo></mrow></mrow><mo id="S3.SS4.p2.2.m2.1.2.1" xref="S3.SS4.p2.2.m2.1.2.1.cmml">=</mo><msub id="S3.SS4.p2.2.m2.1.2.3" xref="S3.SS4.p2.2.m2.1.2.3.cmml"><mi id="S3.SS4.p2.2.m2.1.2.3.2" xref="S3.SS4.p2.2.m2.1.2.3.2.cmml">P</mi><mi id="S3.SS4.p2.2.m2.1.2.3.3" xref="S3.SS4.p2.2.m2.1.2.3.3.cmml">q</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.2.m2.1b"><apply id="S3.SS4.p2.2.m2.1.2.cmml" xref="S3.SS4.p2.2.m2.1.2"><eq id="S3.SS4.p2.2.m2.1.2.1.cmml" xref="S3.SS4.p2.2.m2.1.2.1"></eq><apply id="S3.SS4.p2.2.m2.1.2.2.cmml" xref="S3.SS4.p2.2.m2.1.2.2"><times id="S3.SS4.p2.2.m2.1.2.2.1.cmml" xref="S3.SS4.p2.2.m2.1.2.2.1"></times><apply id="S3.SS4.p2.2.m2.1.2.2.2.cmml" xref="S3.SS4.p2.2.m2.1.2.2.2"><csymbol cd="ambiguous" id="S3.SS4.p2.2.m2.1.2.2.2.1.cmml" xref="S3.SS4.p2.2.m2.1.2.2.2">subscript</csymbol><ci id="S3.SS4.p2.2.m2.1.2.2.2.2.cmml" xref="S3.SS4.p2.2.m2.1.2.2.2.2">𝑞</ci><ci id="S3.SS4.p2.2.m2.1.2.2.2.3a.cmml" xref="S3.SS4.p2.2.m2.1.2.2.2.3"><mtext mathsize="70%" id="S3.SS4.p2.2.m2.1.2.2.2.3.cmml" xref="S3.SS4.p2.2.m2.1.2.2.2.3">NF4</mtext></ci></apply><ci id="S3.SS4.p2.2.m2.1.1.cmml" xref="S3.SS4.p2.2.m2.1.1">𝑃</ci></apply><apply id="S3.SS4.p2.2.m2.1.2.3.cmml" xref="S3.SS4.p2.2.m2.1.2.3"><csymbol cd="ambiguous" id="S3.SS4.p2.2.m2.1.2.3.1.cmml" xref="S3.SS4.p2.2.m2.1.2.3">subscript</csymbol><ci id="S3.SS4.p2.2.m2.1.2.3.2.cmml" xref="S3.SS4.p2.2.m2.1.2.3.2">𝑃</ci><ci id="S3.SS4.p2.2.m2.1.2.3.3.cmml" xref="S3.SS4.p2.2.m2.1.2.3.3">𝑞</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.2.m2.1c">q_{\text{NF4}}(P)=P_{q}</annotation></semantics></math> as described in §<a href="#S2.SS1" title="2.1 Neural Network Quantization and NF4 ‣ 2 Related Work and Background ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.1</span></a>. During periodic updates at interval time steps <math id="S3.SS4.p2.3.m3.1" class="ltx_Math" alttext="(\sum_{i=1}^{n}T_{i})_{n=1}^{\text{max}}" display="inline"><semantics id="S3.SS4.p2.3.m3.1a"><msubsup id="S3.SS4.p2.3.m3.1.1" xref="S3.SS4.p2.3.m3.1.1.cmml"><mrow id="S3.SS4.p2.3.m3.1.1.1.1.1" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS4.p2.3.m3.1.1.1.1.1.2" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS4.p2.3.m3.1.1.1.1.1.1" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.cmml"><msubsup id="S3.SS4.p2.3.m3.1.1.1.1.1.1.1" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.cmml"><mo lspace="0em" id="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.2" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.2.cmml">∑</mo><mrow id="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.3" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.3.cmml"><mi id="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.3.2" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.3.2.cmml">i</mi><mo id="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.3.1" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.3.1.cmml">=</mo><mn id="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.3.3" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.3.3.cmml">1</mn></mrow><mi id="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.3" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.3.cmml">n</mi></msubsup><msub id="S3.SS4.p2.3.m3.1.1.1.1.1.1.2" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.2.cmml"><mi id="S3.SS4.p2.3.m3.1.1.1.1.1.1.2.2" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.2.2.cmml">T</mi><mi id="S3.SS4.p2.3.m3.1.1.1.1.1.1.2.3" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.2.3.cmml">i</mi></msub></mrow><mo stretchy="false" id="S3.SS4.p2.3.m3.1.1.1.1.1.3" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.cmml">)</mo></mrow><mrow id="S3.SS4.p2.3.m3.1.1.1.3" xref="S3.SS4.p2.3.m3.1.1.1.3.cmml"><mi id="S3.SS4.p2.3.m3.1.1.1.3.2" xref="S3.SS4.p2.3.m3.1.1.1.3.2.cmml">n</mi><mo id="S3.SS4.p2.3.m3.1.1.1.3.1" xref="S3.SS4.p2.3.m3.1.1.1.3.1.cmml">=</mo><mn id="S3.SS4.p2.3.m3.1.1.1.3.3" xref="S3.SS4.p2.3.m3.1.1.1.3.3.cmml">1</mn></mrow><mtext id="S3.SS4.p2.3.m3.1.1.3" xref="S3.SS4.p2.3.m3.1.1.3a.cmml">max</mtext></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.3.m3.1b"><apply id="S3.SS4.p2.3.m3.1.1.cmml" xref="S3.SS4.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS4.p2.3.m3.1.1.2.cmml" xref="S3.SS4.p2.3.m3.1.1">superscript</csymbol><apply id="S3.SS4.p2.3.m3.1.1.1.cmml" xref="S3.SS4.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS4.p2.3.m3.1.1.1.2.cmml" xref="S3.SS4.p2.3.m3.1.1">subscript</csymbol><apply id="S3.SS4.p2.3.m3.1.1.1.1.1.1.cmml" xref="S3.SS4.p2.3.m3.1.1.1.1.1"><apply id="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.cmml" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.1.cmml" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.1">superscript</csymbol><apply id="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.cmml" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.1.cmml" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.1">subscript</csymbol><sum id="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.2.cmml" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.2"></sum><apply id="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.3.cmml" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.3"><eq id="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.3.1.cmml" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.3.1"></eq><ci id="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.3.2.cmml" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.3.2">𝑖</ci><cn type="integer" id="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.3.3.cmml" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.2.3.3">1</cn></apply></apply><ci id="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.3.cmml" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.1.3">𝑛</ci></apply><apply id="S3.SS4.p2.3.m3.1.1.1.1.1.1.2.cmml" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS4.p2.3.m3.1.1.1.1.1.1.2.1.cmml" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.SS4.p2.3.m3.1.1.1.1.1.1.2.2.cmml" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.2.2">𝑇</ci><ci id="S3.SS4.p2.3.m3.1.1.1.1.1.1.2.3.cmml" xref="S3.SS4.p2.3.m3.1.1.1.1.1.1.2.3">𝑖</ci></apply></apply><apply id="S3.SS4.p2.3.m3.1.1.1.3.cmml" xref="S3.SS4.p2.3.m3.1.1.1.3"><eq id="S3.SS4.p2.3.m3.1.1.1.3.1.cmml" xref="S3.SS4.p2.3.m3.1.1.1.3.1"></eq><ci id="S3.SS4.p2.3.m3.1.1.1.3.2.cmml" xref="S3.SS4.p2.3.m3.1.1.1.3.2">𝑛</ci><cn type="integer" id="S3.SS4.p2.3.m3.1.1.1.3.3.cmml" xref="S3.SS4.p2.3.m3.1.1.1.3.3">1</cn></apply></apply><ci id="S3.SS4.p2.3.m3.1.1.3a.cmml" xref="S3.SS4.p2.3.m3.1.1.3"><mtext mathsize="70%" id="S3.SS4.p2.3.m3.1.1.3.cmml" xref="S3.SS4.p2.3.m3.1.1.3">max</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.3.m3.1c">(\sum_{i=1}^{n}T_{i})_{n=1}^{\text{max}}</annotation></semantics></math>, <math id="S3.SS4.p2.4.m4.1" class="ltx_Math" alttext="P_{q}" display="inline"><semantics id="S3.SS4.p2.4.m4.1a"><msub id="S3.SS4.p2.4.m4.1.1" xref="S3.SS4.p2.4.m4.1.1.cmml"><mi id="S3.SS4.p2.4.m4.1.1.2" xref="S3.SS4.p2.4.m4.1.1.2.cmml">P</mi><mi id="S3.SS4.p2.4.m4.1.1.3" xref="S3.SS4.p2.4.m4.1.1.3.cmml">q</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.4.m4.1b"><apply id="S3.SS4.p2.4.m4.1.1.cmml" xref="S3.SS4.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS4.p2.4.m4.1.1.1.cmml" xref="S3.SS4.p2.4.m4.1.1">subscript</csymbol><ci id="S3.SS4.p2.4.m4.1.1.2.cmml" xref="S3.SS4.p2.4.m4.1.1.2">𝑃</ci><ci id="S3.SS4.p2.4.m4.1.1.3.cmml" xref="S3.SS4.p2.4.m4.1.1.3">𝑞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.4.m4.1c">P_{q}</annotation></semantics></math> and <math id="S3.SS4.p2.5.m5.1" class="ltx_Math" alttext="W_{q}" display="inline"><semantics id="S3.SS4.p2.5.m5.1a"><msub id="S3.SS4.p2.5.m5.1.1" xref="S3.SS4.p2.5.m5.1.1.cmml"><mi id="S3.SS4.p2.5.m5.1.1.2" xref="S3.SS4.p2.5.m5.1.1.2.cmml">W</mi><mi id="S3.SS4.p2.5.m5.1.1.3" xref="S3.SS4.p2.5.m5.1.1.3.cmml">q</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.5.m5.1b"><apply id="S3.SS4.p2.5.m5.1.1.cmml" xref="S3.SS4.p2.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS4.p2.5.m5.1.1.1.cmml" xref="S3.SS4.p2.5.m5.1.1">subscript</csymbol><ci id="S3.SS4.p2.5.m5.1.1.2.cmml" xref="S3.SS4.p2.5.m5.1.1.2">𝑊</ci><ci id="S3.SS4.p2.5.m5.1.1.3.cmml" xref="S3.SS4.p2.5.m5.1.1.3">𝑞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.5.m5.1c">W_{q}</annotation></semantics></math> are dequantized using the inverse function, <math id="S3.SS4.p2.6.m6.1" class="ltx_Math" alttext="P_{\text{BF16}}=q_{\text{NF4}}^{-1}(P_{\text{NF4}})" display="inline"><semantics id="S3.SS4.p2.6.m6.1a"><mrow id="S3.SS4.p2.6.m6.1.1" xref="S3.SS4.p2.6.m6.1.1.cmml"><msub id="S3.SS4.p2.6.m6.1.1.3" xref="S3.SS4.p2.6.m6.1.1.3.cmml"><mi id="S3.SS4.p2.6.m6.1.1.3.2" xref="S3.SS4.p2.6.m6.1.1.3.2.cmml">P</mi><mtext id="S3.SS4.p2.6.m6.1.1.3.3" xref="S3.SS4.p2.6.m6.1.1.3.3a.cmml">BF16</mtext></msub><mo id="S3.SS4.p2.6.m6.1.1.2" xref="S3.SS4.p2.6.m6.1.1.2.cmml">=</mo><mrow id="S3.SS4.p2.6.m6.1.1.1" xref="S3.SS4.p2.6.m6.1.1.1.cmml"><msubsup id="S3.SS4.p2.6.m6.1.1.1.3" xref="S3.SS4.p2.6.m6.1.1.1.3.cmml"><mi id="S3.SS4.p2.6.m6.1.1.1.3.2.2" xref="S3.SS4.p2.6.m6.1.1.1.3.2.2.cmml">q</mi><mtext id="S3.SS4.p2.6.m6.1.1.1.3.2.3" xref="S3.SS4.p2.6.m6.1.1.1.3.2.3a.cmml">NF4</mtext><mrow id="S3.SS4.p2.6.m6.1.1.1.3.3" xref="S3.SS4.p2.6.m6.1.1.1.3.3.cmml"><mo id="S3.SS4.p2.6.m6.1.1.1.3.3a" xref="S3.SS4.p2.6.m6.1.1.1.3.3.cmml">−</mo><mn id="S3.SS4.p2.6.m6.1.1.1.3.3.2" xref="S3.SS4.p2.6.m6.1.1.1.3.3.2.cmml">1</mn></mrow></msubsup><mo lspace="0em" rspace="0em" id="S3.SS4.p2.6.m6.1.1.1.2" xref="S3.SS4.p2.6.m6.1.1.1.2.cmml">​</mo><mrow id="S3.SS4.p2.6.m6.1.1.1.1.1" xref="S3.SS4.p2.6.m6.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS4.p2.6.m6.1.1.1.1.1.2" xref="S3.SS4.p2.6.m6.1.1.1.1.1.1.cmml">(</mo><msub id="S3.SS4.p2.6.m6.1.1.1.1.1.1" xref="S3.SS4.p2.6.m6.1.1.1.1.1.1.cmml"><mi id="S3.SS4.p2.6.m6.1.1.1.1.1.1.2" xref="S3.SS4.p2.6.m6.1.1.1.1.1.1.2.cmml">P</mi><mtext id="S3.SS4.p2.6.m6.1.1.1.1.1.1.3" xref="S3.SS4.p2.6.m6.1.1.1.1.1.1.3a.cmml">NF4</mtext></msub><mo stretchy="false" id="S3.SS4.p2.6.m6.1.1.1.1.1.3" xref="S3.SS4.p2.6.m6.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.6.m6.1b"><apply id="S3.SS4.p2.6.m6.1.1.cmml" xref="S3.SS4.p2.6.m6.1.1"><eq id="S3.SS4.p2.6.m6.1.1.2.cmml" xref="S3.SS4.p2.6.m6.1.1.2"></eq><apply id="S3.SS4.p2.6.m6.1.1.3.cmml" xref="S3.SS4.p2.6.m6.1.1.3"><csymbol cd="ambiguous" id="S3.SS4.p2.6.m6.1.1.3.1.cmml" xref="S3.SS4.p2.6.m6.1.1.3">subscript</csymbol><ci id="S3.SS4.p2.6.m6.1.1.3.2.cmml" xref="S3.SS4.p2.6.m6.1.1.3.2">𝑃</ci><ci id="S3.SS4.p2.6.m6.1.1.3.3a.cmml" xref="S3.SS4.p2.6.m6.1.1.3.3"><mtext mathsize="70%" id="S3.SS4.p2.6.m6.1.1.3.3.cmml" xref="S3.SS4.p2.6.m6.1.1.3.3">BF16</mtext></ci></apply><apply id="S3.SS4.p2.6.m6.1.1.1.cmml" xref="S3.SS4.p2.6.m6.1.1.1"><times id="S3.SS4.p2.6.m6.1.1.1.2.cmml" xref="S3.SS4.p2.6.m6.1.1.1.2"></times><apply id="S3.SS4.p2.6.m6.1.1.1.3.cmml" xref="S3.SS4.p2.6.m6.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS4.p2.6.m6.1.1.1.3.1.cmml" xref="S3.SS4.p2.6.m6.1.1.1.3">superscript</csymbol><apply id="S3.SS4.p2.6.m6.1.1.1.3.2.cmml" xref="S3.SS4.p2.6.m6.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS4.p2.6.m6.1.1.1.3.2.1.cmml" xref="S3.SS4.p2.6.m6.1.1.1.3">subscript</csymbol><ci id="S3.SS4.p2.6.m6.1.1.1.3.2.2.cmml" xref="S3.SS4.p2.6.m6.1.1.1.3.2.2">𝑞</ci><ci id="S3.SS4.p2.6.m6.1.1.1.3.2.3a.cmml" xref="S3.SS4.p2.6.m6.1.1.1.3.2.3"><mtext mathsize="70%" id="S3.SS4.p2.6.m6.1.1.1.3.2.3.cmml" xref="S3.SS4.p2.6.m6.1.1.1.3.2.3">NF4</mtext></ci></apply><apply id="S3.SS4.p2.6.m6.1.1.1.3.3.cmml" xref="S3.SS4.p2.6.m6.1.1.1.3.3"><minus id="S3.SS4.p2.6.m6.1.1.1.3.3.1.cmml" xref="S3.SS4.p2.6.m6.1.1.1.3.3"></minus><cn type="integer" id="S3.SS4.p2.6.m6.1.1.1.3.3.2.cmml" xref="S3.SS4.p2.6.m6.1.1.1.3.3.2">1</cn></apply></apply><apply id="S3.SS4.p2.6.m6.1.1.1.1.1.1.cmml" xref="S3.SS4.p2.6.m6.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS4.p2.6.m6.1.1.1.1.1.1.1.cmml" xref="S3.SS4.p2.6.m6.1.1.1.1.1">subscript</csymbol><ci id="S3.SS4.p2.6.m6.1.1.1.1.1.1.2.cmml" xref="S3.SS4.p2.6.m6.1.1.1.1.1.1.2">𝑃</ci><ci id="S3.SS4.p2.6.m6.1.1.1.1.1.1.3a.cmml" xref="S3.SS4.p2.6.m6.1.1.1.1.1.1.3"><mtext mathsize="70%" id="S3.SS4.p2.6.m6.1.1.1.1.1.1.3.cmml" xref="S3.SS4.p2.6.m6.1.1.1.1.1.1.3">NF4</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.6.m6.1c">P_{\text{BF16}}=q_{\text{NF4}}^{-1}(P_{\text{NF4}})</annotation></semantics></math> and <math id="S3.SS4.p2.7.m7.1" class="ltx_Math" alttext="W_{BF16}=q_{\text{NF4}}^{-1}(W_{\text{NF4}})" display="inline"><semantics id="S3.SS4.p2.7.m7.1a"><mrow id="S3.SS4.p2.7.m7.1.1" xref="S3.SS4.p2.7.m7.1.1.cmml"><msub id="S3.SS4.p2.7.m7.1.1.3" xref="S3.SS4.p2.7.m7.1.1.3.cmml"><mi id="S3.SS4.p2.7.m7.1.1.3.2" xref="S3.SS4.p2.7.m7.1.1.3.2.cmml">W</mi><mrow id="S3.SS4.p2.7.m7.1.1.3.3" xref="S3.SS4.p2.7.m7.1.1.3.3.cmml"><mi id="S3.SS4.p2.7.m7.1.1.3.3.2" xref="S3.SS4.p2.7.m7.1.1.3.3.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S3.SS4.p2.7.m7.1.1.3.3.1" xref="S3.SS4.p2.7.m7.1.1.3.3.1.cmml">​</mo><mi id="S3.SS4.p2.7.m7.1.1.3.3.3" xref="S3.SS4.p2.7.m7.1.1.3.3.3.cmml">F</mi><mo lspace="0em" rspace="0em" id="S3.SS4.p2.7.m7.1.1.3.3.1a" xref="S3.SS4.p2.7.m7.1.1.3.3.1.cmml">​</mo><mn id="S3.SS4.p2.7.m7.1.1.3.3.4" xref="S3.SS4.p2.7.m7.1.1.3.3.4.cmml">16</mn></mrow></msub><mo id="S3.SS4.p2.7.m7.1.1.2" xref="S3.SS4.p2.7.m7.1.1.2.cmml">=</mo><mrow id="S3.SS4.p2.7.m7.1.1.1" xref="S3.SS4.p2.7.m7.1.1.1.cmml"><msubsup id="S3.SS4.p2.7.m7.1.1.1.3" xref="S3.SS4.p2.7.m7.1.1.1.3.cmml"><mi id="S3.SS4.p2.7.m7.1.1.1.3.2.2" xref="S3.SS4.p2.7.m7.1.1.1.3.2.2.cmml">q</mi><mtext id="S3.SS4.p2.7.m7.1.1.1.3.2.3" xref="S3.SS4.p2.7.m7.1.1.1.3.2.3a.cmml">NF4</mtext><mrow id="S3.SS4.p2.7.m7.1.1.1.3.3" xref="S3.SS4.p2.7.m7.1.1.1.3.3.cmml"><mo id="S3.SS4.p2.7.m7.1.1.1.3.3a" xref="S3.SS4.p2.7.m7.1.1.1.3.3.cmml">−</mo><mn id="S3.SS4.p2.7.m7.1.1.1.3.3.2" xref="S3.SS4.p2.7.m7.1.1.1.3.3.2.cmml">1</mn></mrow></msubsup><mo lspace="0em" rspace="0em" id="S3.SS4.p2.7.m7.1.1.1.2" xref="S3.SS4.p2.7.m7.1.1.1.2.cmml">​</mo><mrow id="S3.SS4.p2.7.m7.1.1.1.1.1" xref="S3.SS4.p2.7.m7.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS4.p2.7.m7.1.1.1.1.1.2" xref="S3.SS4.p2.7.m7.1.1.1.1.1.1.cmml">(</mo><msub id="S3.SS4.p2.7.m7.1.1.1.1.1.1" xref="S3.SS4.p2.7.m7.1.1.1.1.1.1.cmml"><mi id="S3.SS4.p2.7.m7.1.1.1.1.1.1.2" xref="S3.SS4.p2.7.m7.1.1.1.1.1.1.2.cmml">W</mi><mtext id="S3.SS4.p2.7.m7.1.1.1.1.1.1.3" xref="S3.SS4.p2.7.m7.1.1.1.1.1.1.3a.cmml">NF4</mtext></msub><mo stretchy="false" id="S3.SS4.p2.7.m7.1.1.1.1.1.3" xref="S3.SS4.p2.7.m7.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.7.m7.1b"><apply id="S3.SS4.p2.7.m7.1.1.cmml" xref="S3.SS4.p2.7.m7.1.1"><eq id="S3.SS4.p2.7.m7.1.1.2.cmml" xref="S3.SS4.p2.7.m7.1.1.2"></eq><apply id="S3.SS4.p2.7.m7.1.1.3.cmml" xref="S3.SS4.p2.7.m7.1.1.3"><csymbol cd="ambiguous" id="S3.SS4.p2.7.m7.1.1.3.1.cmml" xref="S3.SS4.p2.7.m7.1.1.3">subscript</csymbol><ci id="S3.SS4.p2.7.m7.1.1.3.2.cmml" xref="S3.SS4.p2.7.m7.1.1.3.2">𝑊</ci><apply id="S3.SS4.p2.7.m7.1.1.3.3.cmml" xref="S3.SS4.p2.7.m7.1.1.3.3"><times id="S3.SS4.p2.7.m7.1.1.3.3.1.cmml" xref="S3.SS4.p2.7.m7.1.1.3.3.1"></times><ci id="S3.SS4.p2.7.m7.1.1.3.3.2.cmml" xref="S3.SS4.p2.7.m7.1.1.3.3.2">𝐵</ci><ci id="S3.SS4.p2.7.m7.1.1.3.3.3.cmml" xref="S3.SS4.p2.7.m7.1.1.3.3.3">𝐹</ci><cn type="integer" id="S3.SS4.p2.7.m7.1.1.3.3.4.cmml" xref="S3.SS4.p2.7.m7.1.1.3.3.4">16</cn></apply></apply><apply id="S3.SS4.p2.7.m7.1.1.1.cmml" xref="S3.SS4.p2.7.m7.1.1.1"><times id="S3.SS4.p2.7.m7.1.1.1.2.cmml" xref="S3.SS4.p2.7.m7.1.1.1.2"></times><apply id="S3.SS4.p2.7.m7.1.1.1.3.cmml" xref="S3.SS4.p2.7.m7.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS4.p2.7.m7.1.1.1.3.1.cmml" xref="S3.SS4.p2.7.m7.1.1.1.3">superscript</csymbol><apply id="S3.SS4.p2.7.m7.1.1.1.3.2.cmml" xref="S3.SS4.p2.7.m7.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS4.p2.7.m7.1.1.1.3.2.1.cmml" xref="S3.SS4.p2.7.m7.1.1.1.3">subscript</csymbol><ci id="S3.SS4.p2.7.m7.1.1.1.3.2.2.cmml" xref="S3.SS4.p2.7.m7.1.1.1.3.2.2">𝑞</ci><ci id="S3.SS4.p2.7.m7.1.1.1.3.2.3a.cmml" xref="S3.SS4.p2.7.m7.1.1.1.3.2.3"><mtext mathsize="70%" id="S3.SS4.p2.7.m7.1.1.1.3.2.3.cmml" xref="S3.SS4.p2.7.m7.1.1.1.3.2.3">NF4</mtext></ci></apply><apply id="S3.SS4.p2.7.m7.1.1.1.3.3.cmml" xref="S3.SS4.p2.7.m7.1.1.1.3.3"><minus id="S3.SS4.p2.7.m7.1.1.1.3.3.1.cmml" xref="S3.SS4.p2.7.m7.1.1.1.3.3"></minus><cn type="integer" id="S3.SS4.p2.7.m7.1.1.1.3.3.2.cmml" xref="S3.SS4.p2.7.m7.1.1.1.3.3.2">1</cn></apply></apply><apply id="S3.SS4.p2.7.m7.1.1.1.1.1.1.cmml" xref="S3.SS4.p2.7.m7.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS4.p2.7.m7.1.1.1.1.1.1.1.cmml" xref="S3.SS4.p2.7.m7.1.1.1.1.1">subscript</csymbol><ci id="S3.SS4.p2.7.m7.1.1.1.1.1.1.2.cmml" xref="S3.SS4.p2.7.m7.1.1.1.1.1.1.2">𝑊</ci><ci id="S3.SS4.p2.7.m7.1.1.1.1.1.1.3a.cmml" xref="S3.SS4.p2.7.m7.1.1.1.1.1.1.3"><mtext mathsize="70%" id="S3.SS4.p2.7.m7.1.1.1.1.1.1.3.cmml" xref="S3.SS4.p2.7.m7.1.1.1.1.1.1.3">NF4</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.7.m7.1c">W_{BF16}=q_{\text{NF4}}^{-1}(W_{\text{NF4}})</annotation></semantics></math>. After this, <math id="S3.SS4.p2.8.m8.1" class="ltx_Math" alttext="W_{T_{i}}=W_{T_{i-1}}+P_{T_{i-1}}B_{T_{i-1}}" display="inline"><semantics id="S3.SS4.p2.8.m8.1a"><mrow id="S3.SS4.p2.8.m8.1.1" xref="S3.SS4.p2.8.m8.1.1.cmml"><msub id="S3.SS4.p2.8.m8.1.1.2" xref="S3.SS4.p2.8.m8.1.1.2.cmml"><mi id="S3.SS4.p2.8.m8.1.1.2.2" xref="S3.SS4.p2.8.m8.1.1.2.2.cmml">W</mi><msub id="S3.SS4.p2.8.m8.1.1.2.3" xref="S3.SS4.p2.8.m8.1.1.2.3.cmml"><mi id="S3.SS4.p2.8.m8.1.1.2.3.2" xref="S3.SS4.p2.8.m8.1.1.2.3.2.cmml">T</mi><mi id="S3.SS4.p2.8.m8.1.1.2.3.3" xref="S3.SS4.p2.8.m8.1.1.2.3.3.cmml">i</mi></msub></msub><mo id="S3.SS4.p2.8.m8.1.1.1" xref="S3.SS4.p2.8.m8.1.1.1.cmml">=</mo><mrow id="S3.SS4.p2.8.m8.1.1.3" xref="S3.SS4.p2.8.m8.1.1.3.cmml"><msub id="S3.SS4.p2.8.m8.1.1.3.2" xref="S3.SS4.p2.8.m8.1.1.3.2.cmml"><mi id="S3.SS4.p2.8.m8.1.1.3.2.2" xref="S3.SS4.p2.8.m8.1.1.3.2.2.cmml">W</mi><msub id="S3.SS4.p2.8.m8.1.1.3.2.3" xref="S3.SS4.p2.8.m8.1.1.3.2.3.cmml"><mi id="S3.SS4.p2.8.m8.1.1.3.2.3.2" xref="S3.SS4.p2.8.m8.1.1.3.2.3.2.cmml">T</mi><mrow id="S3.SS4.p2.8.m8.1.1.3.2.3.3" xref="S3.SS4.p2.8.m8.1.1.3.2.3.3.cmml"><mi id="S3.SS4.p2.8.m8.1.1.3.2.3.3.2" xref="S3.SS4.p2.8.m8.1.1.3.2.3.3.2.cmml">i</mi><mo id="S3.SS4.p2.8.m8.1.1.3.2.3.3.1" xref="S3.SS4.p2.8.m8.1.1.3.2.3.3.1.cmml">−</mo><mn id="S3.SS4.p2.8.m8.1.1.3.2.3.3.3" xref="S3.SS4.p2.8.m8.1.1.3.2.3.3.3.cmml">1</mn></mrow></msub></msub><mo id="S3.SS4.p2.8.m8.1.1.3.1" xref="S3.SS4.p2.8.m8.1.1.3.1.cmml">+</mo><mrow id="S3.SS4.p2.8.m8.1.1.3.3" xref="S3.SS4.p2.8.m8.1.1.3.3.cmml"><msub id="S3.SS4.p2.8.m8.1.1.3.3.2" xref="S3.SS4.p2.8.m8.1.1.3.3.2.cmml"><mi id="S3.SS4.p2.8.m8.1.1.3.3.2.2" xref="S3.SS4.p2.8.m8.1.1.3.3.2.2.cmml">P</mi><msub id="S3.SS4.p2.8.m8.1.1.3.3.2.3" xref="S3.SS4.p2.8.m8.1.1.3.3.2.3.cmml"><mi id="S3.SS4.p2.8.m8.1.1.3.3.2.3.2" xref="S3.SS4.p2.8.m8.1.1.3.3.2.3.2.cmml">T</mi><mrow id="S3.SS4.p2.8.m8.1.1.3.3.2.3.3" xref="S3.SS4.p2.8.m8.1.1.3.3.2.3.3.cmml"><mi id="S3.SS4.p2.8.m8.1.1.3.3.2.3.3.2" xref="S3.SS4.p2.8.m8.1.1.3.3.2.3.3.2.cmml">i</mi><mo id="S3.SS4.p2.8.m8.1.1.3.3.2.3.3.1" xref="S3.SS4.p2.8.m8.1.1.3.3.2.3.3.1.cmml">−</mo><mn id="S3.SS4.p2.8.m8.1.1.3.3.2.3.3.3" xref="S3.SS4.p2.8.m8.1.1.3.3.2.3.3.3.cmml">1</mn></mrow></msub></msub><mo lspace="0em" rspace="0em" id="S3.SS4.p2.8.m8.1.1.3.3.1" xref="S3.SS4.p2.8.m8.1.1.3.3.1.cmml">​</mo><msub id="S3.SS4.p2.8.m8.1.1.3.3.3" xref="S3.SS4.p2.8.m8.1.1.3.3.3.cmml"><mi id="S3.SS4.p2.8.m8.1.1.3.3.3.2" xref="S3.SS4.p2.8.m8.1.1.3.3.3.2.cmml">B</mi><msub id="S3.SS4.p2.8.m8.1.1.3.3.3.3" xref="S3.SS4.p2.8.m8.1.1.3.3.3.3.cmml"><mi id="S3.SS4.p2.8.m8.1.1.3.3.3.3.2" xref="S3.SS4.p2.8.m8.1.1.3.3.3.3.2.cmml">T</mi><mrow id="S3.SS4.p2.8.m8.1.1.3.3.3.3.3" xref="S3.SS4.p2.8.m8.1.1.3.3.3.3.3.cmml"><mi id="S3.SS4.p2.8.m8.1.1.3.3.3.3.3.2" xref="S3.SS4.p2.8.m8.1.1.3.3.3.3.3.2.cmml">i</mi><mo id="S3.SS4.p2.8.m8.1.1.3.3.3.3.3.1" xref="S3.SS4.p2.8.m8.1.1.3.3.3.3.3.1.cmml">−</mo><mn id="S3.SS4.p2.8.m8.1.1.3.3.3.3.3.3" xref="S3.SS4.p2.8.m8.1.1.3.3.3.3.3.3.cmml">1</mn></mrow></msub></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.8.m8.1b"><apply id="S3.SS4.p2.8.m8.1.1.cmml" xref="S3.SS4.p2.8.m8.1.1"><eq id="S3.SS4.p2.8.m8.1.1.1.cmml" xref="S3.SS4.p2.8.m8.1.1.1"></eq><apply id="S3.SS4.p2.8.m8.1.1.2.cmml" xref="S3.SS4.p2.8.m8.1.1.2"><csymbol cd="ambiguous" id="S3.SS4.p2.8.m8.1.1.2.1.cmml" xref="S3.SS4.p2.8.m8.1.1.2">subscript</csymbol><ci id="S3.SS4.p2.8.m8.1.1.2.2.cmml" xref="S3.SS4.p2.8.m8.1.1.2.2">𝑊</ci><apply id="S3.SS4.p2.8.m8.1.1.2.3.cmml" xref="S3.SS4.p2.8.m8.1.1.2.3"><csymbol cd="ambiguous" id="S3.SS4.p2.8.m8.1.1.2.3.1.cmml" xref="S3.SS4.p2.8.m8.1.1.2.3">subscript</csymbol><ci id="S3.SS4.p2.8.m8.1.1.2.3.2.cmml" xref="S3.SS4.p2.8.m8.1.1.2.3.2">𝑇</ci><ci id="S3.SS4.p2.8.m8.1.1.2.3.3.cmml" xref="S3.SS4.p2.8.m8.1.1.2.3.3">𝑖</ci></apply></apply><apply id="S3.SS4.p2.8.m8.1.1.3.cmml" xref="S3.SS4.p2.8.m8.1.1.3"><plus id="S3.SS4.p2.8.m8.1.1.3.1.cmml" xref="S3.SS4.p2.8.m8.1.1.3.1"></plus><apply id="S3.SS4.p2.8.m8.1.1.3.2.cmml" xref="S3.SS4.p2.8.m8.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS4.p2.8.m8.1.1.3.2.1.cmml" xref="S3.SS4.p2.8.m8.1.1.3.2">subscript</csymbol><ci id="S3.SS4.p2.8.m8.1.1.3.2.2.cmml" xref="S3.SS4.p2.8.m8.1.1.3.2.2">𝑊</ci><apply id="S3.SS4.p2.8.m8.1.1.3.2.3.cmml" xref="S3.SS4.p2.8.m8.1.1.3.2.3"><csymbol cd="ambiguous" id="S3.SS4.p2.8.m8.1.1.3.2.3.1.cmml" xref="S3.SS4.p2.8.m8.1.1.3.2.3">subscript</csymbol><ci id="S3.SS4.p2.8.m8.1.1.3.2.3.2.cmml" xref="S3.SS4.p2.8.m8.1.1.3.2.3.2">𝑇</ci><apply id="S3.SS4.p2.8.m8.1.1.3.2.3.3.cmml" xref="S3.SS4.p2.8.m8.1.1.3.2.3.3"><minus id="S3.SS4.p2.8.m8.1.1.3.2.3.3.1.cmml" xref="S3.SS4.p2.8.m8.1.1.3.2.3.3.1"></minus><ci id="S3.SS4.p2.8.m8.1.1.3.2.3.3.2.cmml" xref="S3.SS4.p2.8.m8.1.1.3.2.3.3.2">𝑖</ci><cn type="integer" id="S3.SS4.p2.8.m8.1.1.3.2.3.3.3.cmml" xref="S3.SS4.p2.8.m8.1.1.3.2.3.3.3">1</cn></apply></apply></apply><apply id="S3.SS4.p2.8.m8.1.1.3.3.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3"><times id="S3.SS4.p2.8.m8.1.1.3.3.1.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3.1"></times><apply id="S3.SS4.p2.8.m8.1.1.3.3.2.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3.2"><csymbol cd="ambiguous" id="S3.SS4.p2.8.m8.1.1.3.3.2.1.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3.2">subscript</csymbol><ci id="S3.SS4.p2.8.m8.1.1.3.3.2.2.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3.2.2">𝑃</ci><apply id="S3.SS4.p2.8.m8.1.1.3.3.2.3.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3.2.3"><csymbol cd="ambiguous" id="S3.SS4.p2.8.m8.1.1.3.3.2.3.1.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3.2.3">subscript</csymbol><ci id="S3.SS4.p2.8.m8.1.1.3.3.2.3.2.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3.2.3.2">𝑇</ci><apply id="S3.SS4.p2.8.m8.1.1.3.3.2.3.3.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3.2.3.3"><minus id="S3.SS4.p2.8.m8.1.1.3.3.2.3.3.1.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3.2.3.3.1"></minus><ci id="S3.SS4.p2.8.m8.1.1.3.3.2.3.3.2.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3.2.3.3.2">𝑖</ci><cn type="integer" id="S3.SS4.p2.8.m8.1.1.3.3.2.3.3.3.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3.2.3.3.3">1</cn></apply></apply></apply><apply id="S3.SS4.p2.8.m8.1.1.3.3.3.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3.3"><csymbol cd="ambiguous" id="S3.SS4.p2.8.m8.1.1.3.3.3.1.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3.3">subscript</csymbol><ci id="S3.SS4.p2.8.m8.1.1.3.3.3.2.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3.3.2">𝐵</ci><apply id="S3.SS4.p2.8.m8.1.1.3.3.3.3.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3.3.3"><csymbol cd="ambiguous" id="S3.SS4.p2.8.m8.1.1.3.3.3.3.1.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3.3.3">subscript</csymbol><ci id="S3.SS4.p2.8.m8.1.1.3.3.3.3.2.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3.3.3.2">𝑇</ci><apply id="S3.SS4.p2.8.m8.1.1.3.3.3.3.3.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3.3.3.3"><minus id="S3.SS4.p2.8.m8.1.1.3.3.3.3.3.1.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3.3.3.3.1"></minus><ci id="S3.SS4.p2.8.m8.1.1.3.3.3.3.3.2.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3.3.3.3.2">𝑖</ci><cn type="integer" id="S3.SS4.p2.8.m8.1.1.3.3.3.3.3.3.cmml" xref="S3.SS4.p2.8.m8.1.1.3.3.3.3.3.3">1</cn></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.8.m8.1c">W_{T_{i}}=W_{T_{i-1}}+P_{T_{i-1}}B_{T_{i-1}}</annotation></semantics></math> is computed and quantized. The quantization and dequantization processes are applied layer by layer, ensuring that not all layers are simultaneously in a non-quantized state to reduce memory usage. Moreover, the quantization state itself is re-quantized for further efficiency following <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite>. We implement <span id="S3.SS4.p2.8.1" class="ltx_text ltx_font_smallcaps">LoQT</span> using weight-only quantization, meaning the quantized weights are loaded into memory and then dequantized before computing the matrix multiplications.</p>
</div>
<figure id="S3.F3" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="alg1" class="ltx_float ltx_figure_panel ltx_minipage ltx_align_center ltx_align_top ltx_framed ltx_framed_top" style="width:195.1pt;">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg1.3.1.1" class="ltx_text ltx_font_bold">Algorithm 1</span> </span> <span id="alg1.4.2" class="ltx_text ltx_font_smallcaps">LoQT</span>: Low Rank Adapters for Quantized Training</figcaption>
<div id="alg1.5" class="ltx_listing ltx_listing">
<div id="alg1.l0" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l0.1.1.1" class="ltx_text" style="font-size:80%;">0:</span></span>  <math id="alg1.l0.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="alg1.l0.m1.1a"><mi id="alg1.l0.m1.1.1" xref="alg1.l0.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="alg1.l0.m1.1b"><ci id="alg1.l0.m1.1.1.cmml" xref="alg1.l0.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l0.m1.1c">W</annotation></semantics></math>: Weight, <math id="alg1.l0.m2.1" class="ltx_Math" alttext="T" display="inline"><semantics id="alg1.l0.m2.1a"><mi id="alg1.l0.m2.1.1" xref="alg1.l0.m2.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="alg1.l0.m2.1b"><ci id="alg1.l0.m2.1.1.cmml" xref="alg1.l0.m2.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l0.m2.1c">T</annotation></semantics></math>: Update steps, <math id="alg1.l0.m3.1" class="ltx_Math" alttext="\eta" display="inline"><semantics id="alg1.l0.m3.1a"><mi id="alg1.l0.m3.1.1" xref="alg1.l0.m3.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="alg1.l0.m3.1b"><ci id="alg1.l0.m3.1.1.cmml" xref="alg1.l0.m3.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l0.m3.1c">\eta</annotation></semantics></math>: LR, <math id="alg1.l0.m4.1" class="ltx_Math" alttext="r" display="inline"><semantics id="alg1.l0.m4.1a"><mi id="alg1.l0.m4.1.1" xref="alg1.l0.m4.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="alg1.l0.m4.1b"><ci id="alg1.l0.m4.1.1.cmml" xref="alg1.l0.m4.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l0.m4.1c">r</annotation></semantics></math>: rank, <math id="alg1.l0.m5.1" class="ltx_Math" alttext="q_{N}(\cdot)" display="inline"><semantics id="alg1.l0.m5.1a"><mrow id="alg1.l0.m5.1.2" xref="alg1.l0.m5.1.2.cmml"><msub id="alg1.l0.m5.1.2.2" xref="alg1.l0.m5.1.2.2.cmml"><mi id="alg1.l0.m5.1.2.2.2" xref="alg1.l0.m5.1.2.2.2.cmml">q</mi><mi id="alg1.l0.m5.1.2.2.3" xref="alg1.l0.m5.1.2.2.3.cmml">N</mi></msub><mo lspace="0em" rspace="0em" id="alg1.l0.m5.1.2.1" xref="alg1.l0.m5.1.2.1.cmml">​</mo><mrow id="alg1.l0.m5.1.2.3.2" xref="alg1.l0.m5.1.2.cmml"><mo stretchy="false" id="alg1.l0.m5.1.2.3.2.1" xref="alg1.l0.m5.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="alg1.l0.m5.1.1" xref="alg1.l0.m5.1.1.cmml">⋅</mo><mo stretchy="false" id="alg1.l0.m5.1.2.3.2.2" xref="alg1.l0.m5.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l0.m5.1b"><apply id="alg1.l0.m5.1.2.cmml" xref="alg1.l0.m5.1.2"><times id="alg1.l0.m5.1.2.1.cmml" xref="alg1.l0.m5.1.2.1"></times><apply id="alg1.l0.m5.1.2.2.cmml" xref="alg1.l0.m5.1.2.2"><csymbol cd="ambiguous" id="alg1.l0.m5.1.2.2.1.cmml" xref="alg1.l0.m5.1.2.2">subscript</csymbol><ci id="alg1.l0.m5.1.2.2.2.cmml" xref="alg1.l0.m5.1.2.2.2">𝑞</ci><ci id="alg1.l0.m5.1.2.2.3.cmml" xref="alg1.l0.m5.1.2.2.3">𝑁</ci></apply><ci id="alg1.l0.m5.1.1.cmml" xref="alg1.l0.m5.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l0.m5.1c">q_{N}(\cdot)</annotation></semantics></math> and <math id="alg1.l0.m6.1" class="ltx_Math" alttext="deq_{N}(\cdot)" display="inline"><semantics id="alg1.l0.m6.1a"><mrow id="alg1.l0.m6.1.2" xref="alg1.l0.m6.1.2.cmml"><mi id="alg1.l0.m6.1.2.2" xref="alg1.l0.m6.1.2.2.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg1.l0.m6.1.2.1" xref="alg1.l0.m6.1.2.1.cmml">​</mo><mi id="alg1.l0.m6.1.2.3" xref="alg1.l0.m6.1.2.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l0.m6.1.2.1a" xref="alg1.l0.m6.1.2.1.cmml">​</mo><msub id="alg1.l0.m6.1.2.4" xref="alg1.l0.m6.1.2.4.cmml"><mi id="alg1.l0.m6.1.2.4.2" xref="alg1.l0.m6.1.2.4.2.cmml">q</mi><mi id="alg1.l0.m6.1.2.4.3" xref="alg1.l0.m6.1.2.4.3.cmml">N</mi></msub><mo lspace="0em" rspace="0em" id="alg1.l0.m6.1.2.1b" xref="alg1.l0.m6.1.2.1.cmml">​</mo><mrow id="alg1.l0.m6.1.2.5.2" xref="alg1.l0.m6.1.2.cmml"><mo stretchy="false" id="alg1.l0.m6.1.2.5.2.1" xref="alg1.l0.m6.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="alg1.l0.m6.1.1" xref="alg1.l0.m6.1.1.cmml">⋅</mo><mo stretchy="false" id="alg1.l0.m6.1.2.5.2.2" xref="alg1.l0.m6.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l0.m6.1b"><apply id="alg1.l0.m6.1.2.cmml" xref="alg1.l0.m6.1.2"><times id="alg1.l0.m6.1.2.1.cmml" xref="alg1.l0.m6.1.2.1"></times><ci id="alg1.l0.m6.1.2.2.cmml" xref="alg1.l0.m6.1.2.2">𝑑</ci><ci id="alg1.l0.m6.1.2.3.cmml" xref="alg1.l0.m6.1.2.3">𝑒</ci><apply id="alg1.l0.m6.1.2.4.cmml" xref="alg1.l0.m6.1.2.4"><csymbol cd="ambiguous" id="alg1.l0.m6.1.2.4.1.cmml" xref="alg1.l0.m6.1.2.4">subscript</csymbol><ci id="alg1.l0.m6.1.2.4.2.cmml" xref="alg1.l0.m6.1.2.4.2">𝑞</ci><ci id="alg1.l0.m6.1.2.4.3.cmml" xref="alg1.l0.m6.1.2.4.3">𝑁</ci></apply><ci id="alg1.l0.m6.1.1.cmml" xref="alg1.l0.m6.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l0.m6.1c">deq_{N}(\cdot)</annotation></semantics></math>: N-bit quant and dequant functions.

</div>
<div id="alg1.l1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l1.1.1.1" class="ltx_text" style="font-size:80%;">1:</span></span>  <math id="alg1.l1.m1.1" class="ltx_Math" alttext="G_{W}\leftarrow\nabla_{W}\mathcal{L}(W)" display="inline"><semantics id="alg1.l1.m1.1a"><mrow id="alg1.l1.m1.1.2" xref="alg1.l1.m1.1.2.cmml"><msub id="alg1.l1.m1.1.2.2" xref="alg1.l1.m1.1.2.2.cmml"><mi id="alg1.l1.m1.1.2.2.2" xref="alg1.l1.m1.1.2.2.2.cmml">G</mi><mi id="alg1.l1.m1.1.2.2.3" xref="alg1.l1.m1.1.2.2.3.cmml">W</mi></msub><mo stretchy="false" id="alg1.l1.m1.1.2.1" xref="alg1.l1.m1.1.2.1.cmml">←</mo><mrow id="alg1.l1.m1.1.2.3" xref="alg1.l1.m1.1.2.3.cmml"><mrow id="alg1.l1.m1.1.2.3.2" xref="alg1.l1.m1.1.2.3.2.cmml"><msub id="alg1.l1.m1.1.2.3.2.1" xref="alg1.l1.m1.1.2.3.2.1.cmml"><mo rspace="0.167em" id="alg1.l1.m1.1.2.3.2.1.2" xref="alg1.l1.m1.1.2.3.2.1.2.cmml">∇</mo><mi id="alg1.l1.m1.1.2.3.2.1.3" xref="alg1.l1.m1.1.2.3.2.1.3.cmml">W</mi></msub><mi class="ltx_font_mathcaligraphic" id="alg1.l1.m1.1.2.3.2.2" xref="alg1.l1.m1.1.2.3.2.2.cmml">ℒ</mi></mrow><mo lspace="0em" rspace="0em" id="alg1.l1.m1.1.2.3.1" xref="alg1.l1.m1.1.2.3.1.cmml">​</mo><mrow id="alg1.l1.m1.1.2.3.3.2" xref="alg1.l1.m1.1.2.3.cmml"><mo stretchy="false" id="alg1.l1.m1.1.2.3.3.2.1" xref="alg1.l1.m1.1.2.3.cmml">(</mo><mi id="alg1.l1.m1.1.1" xref="alg1.l1.m1.1.1.cmml">W</mi><mo stretchy="false" id="alg1.l1.m1.1.2.3.3.2.2" xref="alg1.l1.m1.1.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l1.m1.1b"><apply id="alg1.l1.m1.1.2.cmml" xref="alg1.l1.m1.1.2"><ci id="alg1.l1.m1.1.2.1.cmml" xref="alg1.l1.m1.1.2.1">←</ci><apply id="alg1.l1.m1.1.2.2.cmml" xref="alg1.l1.m1.1.2.2"><csymbol cd="ambiguous" id="alg1.l1.m1.1.2.2.1.cmml" xref="alg1.l1.m1.1.2.2">subscript</csymbol><ci id="alg1.l1.m1.1.2.2.2.cmml" xref="alg1.l1.m1.1.2.2.2">𝐺</ci><ci id="alg1.l1.m1.1.2.2.3.cmml" xref="alg1.l1.m1.1.2.2.3">𝑊</ci></apply><apply id="alg1.l1.m1.1.2.3.cmml" xref="alg1.l1.m1.1.2.3"><times id="alg1.l1.m1.1.2.3.1.cmml" xref="alg1.l1.m1.1.2.3.1"></times><apply id="alg1.l1.m1.1.2.3.2.cmml" xref="alg1.l1.m1.1.2.3.2"><apply id="alg1.l1.m1.1.2.3.2.1.cmml" xref="alg1.l1.m1.1.2.3.2.1"><csymbol cd="ambiguous" id="alg1.l1.m1.1.2.3.2.1.1.cmml" xref="alg1.l1.m1.1.2.3.2.1">subscript</csymbol><ci id="alg1.l1.m1.1.2.3.2.1.2.cmml" xref="alg1.l1.m1.1.2.3.2.1.2">∇</ci><ci id="alg1.l1.m1.1.2.3.2.1.3.cmml" xref="alg1.l1.m1.1.2.3.2.1.3">𝑊</ci></apply><ci id="alg1.l1.m1.1.2.3.2.2.cmml" xref="alg1.l1.m1.1.2.3.2.2">ℒ</ci></apply><ci id="alg1.l1.m1.1.1.cmml" xref="alg1.l1.m1.1.1">𝑊</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m1.1c">G_{W}\leftarrow\nabla_{W}\mathcal{L}(W)</annotation></semantics></math>

</div>
<div id="alg1.l2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l2.1.1.1" class="ltx_text" style="font-size:80%;">2:</span></span>  <math id="alg1.l2.m1.5" class="ltx_Math" alttext="W_{Q},P_{Q},B\leftarrow\text{Initialize}(W,G_{W})" display="inline"><semantics id="alg1.l2.m1.5a"><mrow id="alg1.l2.m1.5.5" xref="alg1.l2.m1.5.5.cmml"><mrow id="alg1.l2.m1.4.4.2.2" xref="alg1.l2.m1.4.4.2.3.cmml"><msub id="alg1.l2.m1.3.3.1.1.1" xref="alg1.l2.m1.3.3.1.1.1.cmml"><mi id="alg1.l2.m1.3.3.1.1.1.2" xref="alg1.l2.m1.3.3.1.1.1.2.cmml">W</mi><mi id="alg1.l2.m1.3.3.1.1.1.3" xref="alg1.l2.m1.3.3.1.1.1.3.cmml">Q</mi></msub><mo id="alg1.l2.m1.4.4.2.2.3" xref="alg1.l2.m1.4.4.2.3.cmml">,</mo><msub id="alg1.l2.m1.4.4.2.2.2" xref="alg1.l2.m1.4.4.2.2.2.cmml"><mi id="alg1.l2.m1.4.4.2.2.2.2" xref="alg1.l2.m1.4.4.2.2.2.2.cmml">P</mi><mi id="alg1.l2.m1.4.4.2.2.2.3" xref="alg1.l2.m1.4.4.2.2.2.3.cmml">Q</mi></msub><mo id="alg1.l2.m1.4.4.2.2.4" xref="alg1.l2.m1.4.4.2.3.cmml">,</mo><mi id="alg1.l2.m1.2.2" xref="alg1.l2.m1.2.2.cmml">B</mi></mrow><mo stretchy="false" id="alg1.l2.m1.5.5.4" xref="alg1.l2.m1.5.5.4.cmml">←</mo><mrow id="alg1.l2.m1.5.5.3" xref="alg1.l2.m1.5.5.3.cmml"><mtext id="alg1.l2.m1.5.5.3.3" xref="alg1.l2.m1.5.5.3.3a.cmml">Initialize</mtext><mo lspace="0em" rspace="0em" id="alg1.l2.m1.5.5.3.2" xref="alg1.l2.m1.5.5.3.2.cmml">​</mo><mrow id="alg1.l2.m1.5.5.3.1.1" xref="alg1.l2.m1.5.5.3.1.2.cmml"><mo stretchy="false" id="alg1.l2.m1.5.5.3.1.1.2" xref="alg1.l2.m1.5.5.3.1.2.cmml">(</mo><mi id="alg1.l2.m1.1.1" xref="alg1.l2.m1.1.1.cmml">W</mi><mo id="alg1.l2.m1.5.5.3.1.1.3" xref="alg1.l2.m1.5.5.3.1.2.cmml">,</mo><msub id="alg1.l2.m1.5.5.3.1.1.1" xref="alg1.l2.m1.5.5.3.1.1.1.cmml"><mi id="alg1.l2.m1.5.5.3.1.1.1.2" xref="alg1.l2.m1.5.5.3.1.1.1.2.cmml">G</mi><mi id="alg1.l2.m1.5.5.3.1.1.1.3" xref="alg1.l2.m1.5.5.3.1.1.1.3.cmml">W</mi></msub><mo stretchy="false" id="alg1.l2.m1.5.5.3.1.1.4" xref="alg1.l2.m1.5.5.3.1.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l2.m1.5b"><apply id="alg1.l2.m1.5.5.cmml" xref="alg1.l2.m1.5.5"><ci id="alg1.l2.m1.5.5.4.cmml" xref="alg1.l2.m1.5.5.4">←</ci><list id="alg1.l2.m1.4.4.2.3.cmml" xref="alg1.l2.m1.4.4.2.2"><apply id="alg1.l2.m1.3.3.1.1.1.cmml" xref="alg1.l2.m1.3.3.1.1.1"><csymbol cd="ambiguous" id="alg1.l2.m1.3.3.1.1.1.1.cmml" xref="alg1.l2.m1.3.3.1.1.1">subscript</csymbol><ci id="alg1.l2.m1.3.3.1.1.1.2.cmml" xref="alg1.l2.m1.3.3.1.1.1.2">𝑊</ci><ci id="alg1.l2.m1.3.3.1.1.1.3.cmml" xref="alg1.l2.m1.3.3.1.1.1.3">𝑄</ci></apply><apply id="alg1.l2.m1.4.4.2.2.2.cmml" xref="alg1.l2.m1.4.4.2.2.2"><csymbol cd="ambiguous" id="alg1.l2.m1.4.4.2.2.2.1.cmml" xref="alg1.l2.m1.4.4.2.2.2">subscript</csymbol><ci id="alg1.l2.m1.4.4.2.2.2.2.cmml" xref="alg1.l2.m1.4.4.2.2.2.2">𝑃</ci><ci id="alg1.l2.m1.4.4.2.2.2.3.cmml" xref="alg1.l2.m1.4.4.2.2.2.3">𝑄</ci></apply><ci id="alg1.l2.m1.2.2.cmml" xref="alg1.l2.m1.2.2">𝐵</ci></list><apply id="alg1.l2.m1.5.5.3.cmml" xref="alg1.l2.m1.5.5.3"><times id="alg1.l2.m1.5.5.3.2.cmml" xref="alg1.l2.m1.5.5.3.2"></times><ci id="alg1.l2.m1.5.5.3.3a.cmml" xref="alg1.l2.m1.5.5.3.3"><mtext id="alg1.l2.m1.5.5.3.3.cmml" xref="alg1.l2.m1.5.5.3.3">Initialize</mtext></ci><interval closure="open" id="alg1.l2.m1.5.5.3.1.2.cmml" xref="alg1.l2.m1.5.5.3.1.1"><ci id="alg1.l2.m1.1.1.cmml" xref="alg1.l2.m1.1.1">𝑊</ci><apply id="alg1.l2.m1.5.5.3.1.1.1.cmml" xref="alg1.l2.m1.5.5.3.1.1.1"><csymbol cd="ambiguous" id="alg1.l2.m1.5.5.3.1.1.1.1.cmml" xref="alg1.l2.m1.5.5.3.1.1.1">subscript</csymbol><ci id="alg1.l2.m1.5.5.3.1.1.1.2.cmml" xref="alg1.l2.m1.5.5.3.1.1.1.2">𝐺</ci><ci id="alg1.l2.m1.5.5.3.1.1.1.3.cmml" xref="alg1.l2.m1.5.5.3.1.1.1.3">𝑊</ci></apply></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l2.m1.5c">W_{Q},P_{Q},B\leftarrow\text{Initialize}(W,G_{W})</annotation></semantics></math>

</div>
<div id="alg1.l3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l3.1.1.1" class="ltx_text" style="font-size:80%;">3:</span></span>  <span id="alg1.l3.2" class="ltx_text ltx_font_bold">for</span> each <math id="alg1.l3.m1.1" class="ltx_Math" alttext="t" display="inline"><semantics id="alg1.l3.m1.1a"><mi id="alg1.l3.m1.1.1" xref="alg1.l3.m1.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="alg1.l3.m1.1b"><ci id="alg1.l3.m1.1.1.cmml" xref="alg1.l3.m1.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l3.m1.1c">t</annotation></semantics></math> in training steps <span id="alg1.l3.3" class="ltx_text ltx_font_bold">do</span>



</div>
<div id="alg1.l4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l4.1.1.1" class="ltx_text" style="font-size:80%;">4:</span></span>     <span id="alg1.l4.2" class="ltx_text ltx_font_bold">if</span> <math id="alg1.l4.m1.1" class="ltx_Math" alttext="t\in T" display="inline"><semantics id="alg1.l4.m1.1a"><mrow id="alg1.l4.m1.1.1" xref="alg1.l4.m1.1.1.cmml"><mi id="alg1.l4.m1.1.1.2" xref="alg1.l4.m1.1.1.2.cmml">t</mi><mo id="alg1.l4.m1.1.1.1" xref="alg1.l4.m1.1.1.1.cmml">∈</mo><mi id="alg1.l4.m1.1.1.3" xref="alg1.l4.m1.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="alg1.l4.m1.1b"><apply id="alg1.l4.m1.1.1.cmml" xref="alg1.l4.m1.1.1"><in id="alg1.l4.m1.1.1.1.cmml" xref="alg1.l4.m1.1.1.1"></in><ci id="alg1.l4.m1.1.1.2.cmml" xref="alg1.l4.m1.1.1.2">𝑡</ci><ci id="alg1.l4.m1.1.1.3.cmml" xref="alg1.l4.m1.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l4.m1.1c">t\in T</annotation></semantics></math> <span id="alg1.l4.3" class="ltx_text ltx_font_bold">then</span>



</div>
<div id="alg1.l5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l5.1.1.1" class="ltx_text" style="font-size:80%;">5:</span></span>        <math id="alg1.l5.m1.1" class="ltx_Math" alttext="W\leftarrow W_{Q}+s\cdot P_{Q}\cdot B_{t}" display="inline"><semantics id="alg1.l5.m1.1a"><mrow id="alg1.l5.m1.1.1" xref="alg1.l5.m1.1.1.cmml"><mi id="alg1.l5.m1.1.1.2" xref="alg1.l5.m1.1.1.2.cmml">W</mi><mo stretchy="false" id="alg1.l5.m1.1.1.1" xref="alg1.l5.m1.1.1.1.cmml">←</mo><mrow id="alg1.l5.m1.1.1.3" xref="alg1.l5.m1.1.1.3.cmml"><msub id="alg1.l5.m1.1.1.3.2" xref="alg1.l5.m1.1.1.3.2.cmml"><mi id="alg1.l5.m1.1.1.3.2.2" xref="alg1.l5.m1.1.1.3.2.2.cmml">W</mi><mi id="alg1.l5.m1.1.1.3.2.3" xref="alg1.l5.m1.1.1.3.2.3.cmml">Q</mi></msub><mo id="alg1.l5.m1.1.1.3.1" xref="alg1.l5.m1.1.1.3.1.cmml">+</mo><mrow id="alg1.l5.m1.1.1.3.3" xref="alg1.l5.m1.1.1.3.3.cmml"><mi id="alg1.l5.m1.1.1.3.3.2" xref="alg1.l5.m1.1.1.3.3.2.cmml">s</mi><mo lspace="0.222em" rspace="0.222em" id="alg1.l5.m1.1.1.3.3.1" xref="alg1.l5.m1.1.1.3.3.1.cmml">⋅</mo><msub id="alg1.l5.m1.1.1.3.3.3" xref="alg1.l5.m1.1.1.3.3.3.cmml"><mi id="alg1.l5.m1.1.1.3.3.3.2" xref="alg1.l5.m1.1.1.3.3.3.2.cmml">P</mi><mi id="alg1.l5.m1.1.1.3.3.3.3" xref="alg1.l5.m1.1.1.3.3.3.3.cmml">Q</mi></msub><mo lspace="0.222em" rspace="0.222em" id="alg1.l5.m1.1.1.3.3.1a" xref="alg1.l5.m1.1.1.3.3.1.cmml">⋅</mo><msub id="alg1.l5.m1.1.1.3.3.4" xref="alg1.l5.m1.1.1.3.3.4.cmml"><mi id="alg1.l5.m1.1.1.3.3.4.2" xref="alg1.l5.m1.1.1.3.3.4.2.cmml">B</mi><mi id="alg1.l5.m1.1.1.3.3.4.3" xref="alg1.l5.m1.1.1.3.3.4.3.cmml">t</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l5.m1.1b"><apply id="alg1.l5.m1.1.1.cmml" xref="alg1.l5.m1.1.1"><ci id="alg1.l5.m1.1.1.1.cmml" xref="alg1.l5.m1.1.1.1">←</ci><ci id="alg1.l5.m1.1.1.2.cmml" xref="alg1.l5.m1.1.1.2">𝑊</ci><apply id="alg1.l5.m1.1.1.3.cmml" xref="alg1.l5.m1.1.1.3"><plus id="alg1.l5.m1.1.1.3.1.cmml" xref="alg1.l5.m1.1.1.3.1"></plus><apply id="alg1.l5.m1.1.1.3.2.cmml" xref="alg1.l5.m1.1.1.3.2"><csymbol cd="ambiguous" id="alg1.l5.m1.1.1.3.2.1.cmml" xref="alg1.l5.m1.1.1.3.2">subscript</csymbol><ci id="alg1.l5.m1.1.1.3.2.2.cmml" xref="alg1.l5.m1.1.1.3.2.2">𝑊</ci><ci id="alg1.l5.m1.1.1.3.2.3.cmml" xref="alg1.l5.m1.1.1.3.2.3">𝑄</ci></apply><apply id="alg1.l5.m1.1.1.3.3.cmml" xref="alg1.l5.m1.1.1.3.3"><ci id="alg1.l5.m1.1.1.3.3.1.cmml" xref="alg1.l5.m1.1.1.3.3.1">⋅</ci><ci id="alg1.l5.m1.1.1.3.3.2.cmml" xref="alg1.l5.m1.1.1.3.3.2">𝑠</ci><apply id="alg1.l5.m1.1.1.3.3.3.cmml" xref="alg1.l5.m1.1.1.3.3.3"><csymbol cd="ambiguous" id="alg1.l5.m1.1.1.3.3.3.1.cmml" xref="alg1.l5.m1.1.1.3.3.3">subscript</csymbol><ci id="alg1.l5.m1.1.1.3.3.3.2.cmml" xref="alg1.l5.m1.1.1.3.3.3.2">𝑃</ci><ci id="alg1.l5.m1.1.1.3.3.3.3.cmml" xref="alg1.l5.m1.1.1.3.3.3.3">𝑄</ci></apply><apply id="alg1.l5.m1.1.1.3.3.4.cmml" xref="alg1.l5.m1.1.1.3.3.4"><csymbol cd="ambiguous" id="alg1.l5.m1.1.1.3.3.4.1.cmml" xref="alg1.l5.m1.1.1.3.3.4">subscript</csymbol><ci id="alg1.l5.m1.1.1.3.3.4.2.cmml" xref="alg1.l5.m1.1.1.3.3.4.2">𝐵</ci><ci id="alg1.l5.m1.1.1.3.3.4.3.cmml" xref="alg1.l5.m1.1.1.3.3.4.3">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m1.1c">W\leftarrow W_{Q}+s\cdot P_{Q}\cdot B_{t}</annotation></semantics></math>

</div>
<div id="alg1.l6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l6.1.1.1" class="ltx_text" style="font-size:80%;">6:</span></span>        <math id="alg1.l6.m1.1" class="ltx_Math" alttext="G^{W}\leftarrow\nabla_{W}\mathcal{L}(W)" display="inline"><semantics id="alg1.l6.m1.1a"><mrow id="alg1.l6.m1.1.2" xref="alg1.l6.m1.1.2.cmml"><msup id="alg1.l6.m1.1.2.2" xref="alg1.l6.m1.1.2.2.cmml"><mi id="alg1.l6.m1.1.2.2.2" xref="alg1.l6.m1.1.2.2.2.cmml">G</mi><mi id="alg1.l6.m1.1.2.2.3" xref="alg1.l6.m1.1.2.2.3.cmml">W</mi></msup><mo stretchy="false" id="alg1.l6.m1.1.2.1" xref="alg1.l6.m1.1.2.1.cmml">←</mo><mrow id="alg1.l6.m1.1.2.3" xref="alg1.l6.m1.1.2.3.cmml"><mrow id="alg1.l6.m1.1.2.3.2" xref="alg1.l6.m1.1.2.3.2.cmml"><msub id="alg1.l6.m1.1.2.3.2.1" xref="alg1.l6.m1.1.2.3.2.1.cmml"><mo rspace="0.167em" id="alg1.l6.m1.1.2.3.2.1.2" xref="alg1.l6.m1.1.2.3.2.1.2.cmml">∇</mo><mi id="alg1.l6.m1.1.2.3.2.1.3" xref="alg1.l6.m1.1.2.3.2.1.3.cmml">W</mi></msub><mi class="ltx_font_mathcaligraphic" id="alg1.l6.m1.1.2.3.2.2" xref="alg1.l6.m1.1.2.3.2.2.cmml">ℒ</mi></mrow><mo lspace="0em" rspace="0em" id="alg1.l6.m1.1.2.3.1" xref="alg1.l6.m1.1.2.3.1.cmml">​</mo><mrow id="alg1.l6.m1.1.2.3.3.2" xref="alg1.l6.m1.1.2.3.cmml"><mo stretchy="false" id="alg1.l6.m1.1.2.3.3.2.1" xref="alg1.l6.m1.1.2.3.cmml">(</mo><mi id="alg1.l6.m1.1.1" xref="alg1.l6.m1.1.1.cmml">W</mi><mo stretchy="false" id="alg1.l6.m1.1.2.3.3.2.2" xref="alg1.l6.m1.1.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l6.m1.1b"><apply id="alg1.l6.m1.1.2.cmml" xref="alg1.l6.m1.1.2"><ci id="alg1.l6.m1.1.2.1.cmml" xref="alg1.l6.m1.1.2.1">←</ci><apply id="alg1.l6.m1.1.2.2.cmml" xref="alg1.l6.m1.1.2.2"><csymbol cd="ambiguous" id="alg1.l6.m1.1.2.2.1.cmml" xref="alg1.l6.m1.1.2.2">superscript</csymbol><ci id="alg1.l6.m1.1.2.2.2.cmml" xref="alg1.l6.m1.1.2.2.2">𝐺</ci><ci id="alg1.l6.m1.1.2.2.3.cmml" xref="alg1.l6.m1.1.2.2.3">𝑊</ci></apply><apply id="alg1.l6.m1.1.2.3.cmml" xref="alg1.l6.m1.1.2.3"><times id="alg1.l6.m1.1.2.3.1.cmml" xref="alg1.l6.m1.1.2.3.1"></times><apply id="alg1.l6.m1.1.2.3.2.cmml" xref="alg1.l6.m1.1.2.3.2"><apply id="alg1.l6.m1.1.2.3.2.1.cmml" xref="alg1.l6.m1.1.2.3.2.1"><csymbol cd="ambiguous" id="alg1.l6.m1.1.2.3.2.1.1.cmml" xref="alg1.l6.m1.1.2.3.2.1">subscript</csymbol><ci id="alg1.l6.m1.1.2.3.2.1.2.cmml" xref="alg1.l6.m1.1.2.3.2.1.2">∇</ci><ci id="alg1.l6.m1.1.2.3.2.1.3.cmml" xref="alg1.l6.m1.1.2.3.2.1.3">𝑊</ci></apply><ci id="alg1.l6.m1.1.2.3.2.2.cmml" xref="alg1.l6.m1.1.2.3.2.2">ℒ</ci></apply><ci id="alg1.l6.m1.1.1.cmml" xref="alg1.l6.m1.1.1">𝑊</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l6.m1.1c">G^{W}\leftarrow\nabla_{W}\mathcal{L}(W)</annotation></semantics></math>

</div>
<div id="alg1.l7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l7.1.1.1" class="ltx_text" style="font-size:80%;">7:</span></span>        <math id="alg1.l7.m1.5" class="ltx_Math" alttext="W_{Q},P_{Q},B_{t}\leftarrow\text{Initialize}(W,G^{W})" display="inline"><semantics id="alg1.l7.m1.5a"><mrow id="alg1.l7.m1.5.5" xref="alg1.l7.m1.5.5.cmml"><mrow id="alg1.l7.m1.4.4.3.3" xref="alg1.l7.m1.4.4.3.4.cmml"><msub id="alg1.l7.m1.2.2.1.1.1" xref="alg1.l7.m1.2.2.1.1.1.cmml"><mi id="alg1.l7.m1.2.2.1.1.1.2" xref="alg1.l7.m1.2.2.1.1.1.2.cmml">W</mi><mi id="alg1.l7.m1.2.2.1.1.1.3" xref="alg1.l7.m1.2.2.1.1.1.3.cmml">Q</mi></msub><mo id="alg1.l7.m1.4.4.3.3.4" xref="alg1.l7.m1.4.4.3.4.cmml">,</mo><msub id="alg1.l7.m1.3.3.2.2.2" xref="alg1.l7.m1.3.3.2.2.2.cmml"><mi id="alg1.l7.m1.3.3.2.2.2.2" xref="alg1.l7.m1.3.3.2.2.2.2.cmml">P</mi><mi id="alg1.l7.m1.3.3.2.2.2.3" xref="alg1.l7.m1.3.3.2.2.2.3.cmml">Q</mi></msub><mo id="alg1.l7.m1.4.4.3.3.5" xref="alg1.l7.m1.4.4.3.4.cmml">,</mo><msub id="alg1.l7.m1.4.4.3.3.3" xref="alg1.l7.m1.4.4.3.3.3.cmml"><mi id="alg1.l7.m1.4.4.3.3.3.2" xref="alg1.l7.m1.4.4.3.3.3.2.cmml">B</mi><mi id="alg1.l7.m1.4.4.3.3.3.3" xref="alg1.l7.m1.4.4.3.3.3.3.cmml">t</mi></msub></mrow><mo stretchy="false" id="alg1.l7.m1.5.5.5" xref="alg1.l7.m1.5.5.5.cmml">←</mo><mrow id="alg1.l7.m1.5.5.4" xref="alg1.l7.m1.5.5.4.cmml"><mtext id="alg1.l7.m1.5.5.4.3" xref="alg1.l7.m1.5.5.4.3a.cmml">Initialize</mtext><mo lspace="0em" rspace="0em" id="alg1.l7.m1.5.5.4.2" xref="alg1.l7.m1.5.5.4.2.cmml">​</mo><mrow id="alg1.l7.m1.5.5.4.1.1" xref="alg1.l7.m1.5.5.4.1.2.cmml"><mo stretchy="false" id="alg1.l7.m1.5.5.4.1.1.2" xref="alg1.l7.m1.5.5.4.1.2.cmml">(</mo><mi id="alg1.l7.m1.1.1" xref="alg1.l7.m1.1.1.cmml">W</mi><mo id="alg1.l7.m1.5.5.4.1.1.3" xref="alg1.l7.m1.5.5.4.1.2.cmml">,</mo><msup id="alg1.l7.m1.5.5.4.1.1.1" xref="alg1.l7.m1.5.5.4.1.1.1.cmml"><mi id="alg1.l7.m1.5.5.4.1.1.1.2" xref="alg1.l7.m1.5.5.4.1.1.1.2.cmml">G</mi><mi id="alg1.l7.m1.5.5.4.1.1.1.3" xref="alg1.l7.m1.5.5.4.1.1.1.3.cmml">W</mi></msup><mo stretchy="false" id="alg1.l7.m1.5.5.4.1.1.4" xref="alg1.l7.m1.5.5.4.1.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l7.m1.5b"><apply id="alg1.l7.m1.5.5.cmml" xref="alg1.l7.m1.5.5"><ci id="alg1.l7.m1.5.5.5.cmml" xref="alg1.l7.m1.5.5.5">←</ci><list id="alg1.l7.m1.4.4.3.4.cmml" xref="alg1.l7.m1.4.4.3.3"><apply id="alg1.l7.m1.2.2.1.1.1.cmml" xref="alg1.l7.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="alg1.l7.m1.2.2.1.1.1.1.cmml" xref="alg1.l7.m1.2.2.1.1.1">subscript</csymbol><ci id="alg1.l7.m1.2.2.1.1.1.2.cmml" xref="alg1.l7.m1.2.2.1.1.1.2">𝑊</ci><ci id="alg1.l7.m1.2.2.1.1.1.3.cmml" xref="alg1.l7.m1.2.2.1.1.1.3">𝑄</ci></apply><apply id="alg1.l7.m1.3.3.2.2.2.cmml" xref="alg1.l7.m1.3.3.2.2.2"><csymbol cd="ambiguous" id="alg1.l7.m1.3.3.2.2.2.1.cmml" xref="alg1.l7.m1.3.3.2.2.2">subscript</csymbol><ci id="alg1.l7.m1.3.3.2.2.2.2.cmml" xref="alg1.l7.m1.3.3.2.2.2.2">𝑃</ci><ci id="alg1.l7.m1.3.3.2.2.2.3.cmml" xref="alg1.l7.m1.3.3.2.2.2.3">𝑄</ci></apply><apply id="alg1.l7.m1.4.4.3.3.3.cmml" xref="alg1.l7.m1.4.4.3.3.3"><csymbol cd="ambiguous" id="alg1.l7.m1.4.4.3.3.3.1.cmml" xref="alg1.l7.m1.4.4.3.3.3">subscript</csymbol><ci id="alg1.l7.m1.4.4.3.3.3.2.cmml" xref="alg1.l7.m1.4.4.3.3.3.2">𝐵</ci><ci id="alg1.l7.m1.4.4.3.3.3.3.cmml" xref="alg1.l7.m1.4.4.3.3.3.3">𝑡</ci></apply></list><apply id="alg1.l7.m1.5.5.4.cmml" xref="alg1.l7.m1.5.5.4"><times id="alg1.l7.m1.5.5.4.2.cmml" xref="alg1.l7.m1.5.5.4.2"></times><ci id="alg1.l7.m1.5.5.4.3a.cmml" xref="alg1.l7.m1.5.5.4.3"><mtext id="alg1.l7.m1.5.5.4.3.cmml" xref="alg1.l7.m1.5.5.4.3">Initialize</mtext></ci><interval closure="open" id="alg1.l7.m1.5.5.4.1.2.cmml" xref="alg1.l7.m1.5.5.4.1.1"><ci id="alg1.l7.m1.1.1.cmml" xref="alg1.l7.m1.1.1">𝑊</ci><apply id="alg1.l7.m1.5.5.4.1.1.1.cmml" xref="alg1.l7.m1.5.5.4.1.1.1"><csymbol cd="ambiguous" id="alg1.l7.m1.5.5.4.1.1.1.1.cmml" xref="alg1.l7.m1.5.5.4.1.1.1">superscript</csymbol><ci id="alg1.l7.m1.5.5.4.1.1.1.2.cmml" xref="alg1.l7.m1.5.5.4.1.1.1.2">𝐺</ci><ci id="alg1.l7.m1.5.5.4.1.1.1.3.cmml" xref="alg1.l7.m1.5.5.4.1.1.1.3">𝑊</ci></apply></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l7.m1.5c">W_{Q},P_{Q},B_{t}\leftarrow\text{Initialize}(W,G^{W})</annotation></semantics></math>

</div>
<div id="alg1.l8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l8.1.1.1" class="ltx_text" style="font-size:80%;">8:</span></span>     <span id="alg1.l8.2" class="ltx_text ltx_font_bold">else</span>


</div>
<div id="alg1.l9" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l9.1.1.1" class="ltx_text" style="font-size:80%;">9:</span></span>        <math id="alg1.l9.m1.1" class="ltx_Math" alttext="B_{t+1}\leftarrow B_{t}-\rho(G_{t}^{B})" display="inline"><semantics id="alg1.l9.m1.1a"><mrow id="alg1.l9.m1.1.1" xref="alg1.l9.m1.1.1.cmml"><msub id="alg1.l9.m1.1.1.3" xref="alg1.l9.m1.1.1.3.cmml"><mi id="alg1.l9.m1.1.1.3.2" xref="alg1.l9.m1.1.1.3.2.cmml">B</mi><mrow id="alg1.l9.m1.1.1.3.3" xref="alg1.l9.m1.1.1.3.3.cmml"><mi id="alg1.l9.m1.1.1.3.3.2" xref="alg1.l9.m1.1.1.3.3.2.cmml">t</mi><mo id="alg1.l9.m1.1.1.3.3.1" xref="alg1.l9.m1.1.1.3.3.1.cmml">+</mo><mn id="alg1.l9.m1.1.1.3.3.3" xref="alg1.l9.m1.1.1.3.3.3.cmml">1</mn></mrow></msub><mo stretchy="false" id="alg1.l9.m1.1.1.2" xref="alg1.l9.m1.1.1.2.cmml">←</mo><mrow id="alg1.l9.m1.1.1.1" xref="alg1.l9.m1.1.1.1.cmml"><msub id="alg1.l9.m1.1.1.1.3" xref="alg1.l9.m1.1.1.1.3.cmml"><mi id="alg1.l9.m1.1.1.1.3.2" xref="alg1.l9.m1.1.1.1.3.2.cmml">B</mi><mi id="alg1.l9.m1.1.1.1.3.3" xref="alg1.l9.m1.1.1.1.3.3.cmml">t</mi></msub><mo id="alg1.l9.m1.1.1.1.2" xref="alg1.l9.m1.1.1.1.2.cmml">−</mo><mrow id="alg1.l9.m1.1.1.1.1" xref="alg1.l9.m1.1.1.1.1.cmml"><mi id="alg1.l9.m1.1.1.1.1.3" xref="alg1.l9.m1.1.1.1.1.3.cmml">ρ</mi><mo lspace="0em" rspace="0em" id="alg1.l9.m1.1.1.1.1.2" xref="alg1.l9.m1.1.1.1.1.2.cmml">​</mo><mrow id="alg1.l9.m1.1.1.1.1.1.1" xref="alg1.l9.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l9.m1.1.1.1.1.1.1.2" xref="alg1.l9.m1.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="alg1.l9.m1.1.1.1.1.1.1.1" xref="alg1.l9.m1.1.1.1.1.1.1.1.cmml"><mi id="alg1.l9.m1.1.1.1.1.1.1.1.2.2" xref="alg1.l9.m1.1.1.1.1.1.1.1.2.2.cmml">G</mi><mi id="alg1.l9.m1.1.1.1.1.1.1.1.2.3" xref="alg1.l9.m1.1.1.1.1.1.1.1.2.3.cmml">t</mi><mi id="alg1.l9.m1.1.1.1.1.1.1.1.3" xref="alg1.l9.m1.1.1.1.1.1.1.1.3.cmml">B</mi></msubsup><mo stretchy="false" id="alg1.l9.m1.1.1.1.1.1.1.3" xref="alg1.l9.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l9.m1.1b"><apply id="alg1.l9.m1.1.1.cmml" xref="alg1.l9.m1.1.1"><ci id="alg1.l9.m1.1.1.2.cmml" xref="alg1.l9.m1.1.1.2">←</ci><apply id="alg1.l9.m1.1.1.3.cmml" xref="alg1.l9.m1.1.1.3"><csymbol cd="ambiguous" id="alg1.l9.m1.1.1.3.1.cmml" xref="alg1.l9.m1.1.1.3">subscript</csymbol><ci id="alg1.l9.m1.1.1.3.2.cmml" xref="alg1.l9.m1.1.1.3.2">𝐵</ci><apply id="alg1.l9.m1.1.1.3.3.cmml" xref="alg1.l9.m1.1.1.3.3"><plus id="alg1.l9.m1.1.1.3.3.1.cmml" xref="alg1.l9.m1.1.1.3.3.1"></plus><ci id="alg1.l9.m1.1.1.3.3.2.cmml" xref="alg1.l9.m1.1.1.3.3.2">𝑡</ci><cn type="integer" id="alg1.l9.m1.1.1.3.3.3.cmml" xref="alg1.l9.m1.1.1.3.3.3">1</cn></apply></apply><apply id="alg1.l9.m1.1.1.1.cmml" xref="alg1.l9.m1.1.1.1"><minus id="alg1.l9.m1.1.1.1.2.cmml" xref="alg1.l9.m1.1.1.1.2"></minus><apply id="alg1.l9.m1.1.1.1.3.cmml" xref="alg1.l9.m1.1.1.1.3"><csymbol cd="ambiguous" id="alg1.l9.m1.1.1.1.3.1.cmml" xref="alg1.l9.m1.1.1.1.3">subscript</csymbol><ci id="alg1.l9.m1.1.1.1.3.2.cmml" xref="alg1.l9.m1.1.1.1.3.2">𝐵</ci><ci id="alg1.l9.m1.1.1.1.3.3.cmml" xref="alg1.l9.m1.1.1.1.3.3">𝑡</ci></apply><apply id="alg1.l9.m1.1.1.1.1.cmml" xref="alg1.l9.m1.1.1.1.1"><times id="alg1.l9.m1.1.1.1.1.2.cmml" xref="alg1.l9.m1.1.1.1.1.2"></times><ci id="alg1.l9.m1.1.1.1.1.3.cmml" xref="alg1.l9.m1.1.1.1.1.3">𝜌</ci><apply id="alg1.l9.m1.1.1.1.1.1.1.1.cmml" xref="alg1.l9.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l9.m1.1.1.1.1.1.1.1.1.cmml" xref="alg1.l9.m1.1.1.1.1.1.1">superscript</csymbol><apply id="alg1.l9.m1.1.1.1.1.1.1.1.2.cmml" xref="alg1.l9.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l9.m1.1.1.1.1.1.1.1.2.1.cmml" xref="alg1.l9.m1.1.1.1.1.1.1">subscript</csymbol><ci id="alg1.l9.m1.1.1.1.1.1.1.1.2.2.cmml" xref="alg1.l9.m1.1.1.1.1.1.1.1.2.2">𝐺</ci><ci id="alg1.l9.m1.1.1.1.1.1.1.1.2.3.cmml" xref="alg1.l9.m1.1.1.1.1.1.1.1.2.3">𝑡</ci></apply><ci id="alg1.l9.m1.1.1.1.1.1.1.1.3.cmml" xref="alg1.l9.m1.1.1.1.1.1.1.1.3">𝐵</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l9.m1.1c">B_{t+1}\leftarrow B_{t}-\rho(G_{t}^{B})</annotation></semantics></math>

</div>
<div id="alg1.l10" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l10.1.1.1" class="ltx_text" style="font-size:80%;">10:</span></span>  <span id="alg1.l10.2" class="ltx_text ltx_font_bold">return</span>  <math id="alg1.l10.m1.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="alg1.l10.m1.1a"><mi id="alg1.l10.m1.1.1" xref="alg1.l10.m1.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="alg1.l10.m1.1b"><ci id="alg1.l10.m1.1.1.cmml" xref="alg1.l10.m1.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l10.m1.1c">\theta</annotation></semantics></math>

</div>
</div>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="alg2" class="ltx_float ltx_figure_panel ltx_minipage ltx_align_center ltx_align_top ltx_framed ltx_framed_top" style="width:195.1pt;">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg2.2.1.1" class="ltx_text ltx_font_bold">Algorithm 2</span> </span> Initialization Procedure</figcaption>
<div id="alg2.3" class="ltx_listing ltx_listing">
<div id="alg2.l1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l1.2.1.1" class="ltx_text" style="font-size:80%;">1:</span></span>  <span id="alg2.l1.1" class="ltx_text ltx_font_bold">Initialize<math id="alg2.l1.1.m1.2" class="ltx_Math" alttext="(W,G^{W})" display="inline"><semantics id="alg2.l1.1.m1.2a"><mrow id="alg2.l1.1.m1.2.2.1" xref="alg2.l1.1.m1.2.2.2.cmml"><mo stretchy="false" id="alg2.l1.1.m1.2.2.1.2" xref="alg2.l1.1.m1.2.2.2.cmml">(</mo><mi id="alg2.l1.1.m1.1.1" xref="alg2.l1.1.m1.1.1.cmml">W</mi><mo id="alg2.l1.1.m1.2.2.1.3" xref="alg2.l1.1.m1.2.2.2.cmml">,</mo><msup id="alg2.l1.1.m1.2.2.1.1" xref="alg2.l1.1.m1.2.2.1.1.cmml"><mi id="alg2.l1.1.m1.2.2.1.1.2" xref="alg2.l1.1.m1.2.2.1.1.2.cmml">G</mi><mi id="alg2.l1.1.m1.2.2.1.1.3" xref="alg2.l1.1.m1.2.2.1.1.3.cmml">W</mi></msup><mo stretchy="false" id="alg2.l1.1.m1.2.2.1.4" xref="alg2.l1.1.m1.2.2.2.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="alg2.l1.1.m1.2b"><interval closure="open" id="alg2.l1.1.m1.2.2.2.cmml" xref="alg2.l1.1.m1.2.2.1"><ci id="alg2.l1.1.m1.1.1.cmml" xref="alg2.l1.1.m1.1.1">𝑊</ci><apply id="alg2.l1.1.m1.2.2.1.1.cmml" xref="alg2.l1.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="alg2.l1.1.m1.2.2.1.1.1.cmml" xref="alg2.l1.1.m1.2.2.1.1">superscript</csymbol><ci id="alg2.l1.1.m1.2.2.1.1.2.cmml" xref="alg2.l1.1.m1.2.2.1.1.2">𝐺</ci><ci id="alg2.l1.1.m1.2.2.1.1.3.cmml" xref="alg2.l1.1.m1.2.2.1.1.3">𝑊</ci></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="alg2.l1.1.m1.2c">(W,G^{W})</annotation></semantics></math></span>:

</div>
<div id="alg2.l2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l2.1.1.1" class="ltx_text" style="font-size:80%;">2:</span></span>  <math id="alg2.l2.m1.4" class="ltx_Math" alttext="U,S,V^{T}\leftarrow\text{SVD}(G^{W})" display="inline"><semantics id="alg2.l2.m1.4a"><mrow id="alg2.l2.m1.4.4" xref="alg2.l2.m1.4.4.cmml"><mrow id="alg2.l2.m1.3.3.1.1" xref="alg2.l2.m1.3.3.1.2.cmml"><mi id="alg2.l2.m1.1.1" xref="alg2.l2.m1.1.1.cmml">U</mi><mo id="alg2.l2.m1.3.3.1.1.2" xref="alg2.l2.m1.3.3.1.2.cmml">,</mo><mi id="alg2.l2.m1.2.2" xref="alg2.l2.m1.2.2.cmml">S</mi><mo id="alg2.l2.m1.3.3.1.1.3" xref="alg2.l2.m1.3.3.1.2.cmml">,</mo><msup id="alg2.l2.m1.3.3.1.1.1" xref="alg2.l2.m1.3.3.1.1.1.cmml"><mi id="alg2.l2.m1.3.3.1.1.1.2" xref="alg2.l2.m1.3.3.1.1.1.2.cmml">V</mi><mi id="alg2.l2.m1.3.3.1.1.1.3" xref="alg2.l2.m1.3.3.1.1.1.3.cmml">T</mi></msup></mrow><mo stretchy="false" id="alg2.l2.m1.4.4.3" xref="alg2.l2.m1.4.4.3.cmml">←</mo><mrow id="alg2.l2.m1.4.4.2" xref="alg2.l2.m1.4.4.2.cmml"><mtext id="alg2.l2.m1.4.4.2.3" xref="alg2.l2.m1.4.4.2.3a.cmml">SVD</mtext><mo lspace="0em" rspace="0em" id="alg2.l2.m1.4.4.2.2" xref="alg2.l2.m1.4.4.2.2.cmml">​</mo><mrow id="alg2.l2.m1.4.4.2.1.1" xref="alg2.l2.m1.4.4.2.1.1.1.cmml"><mo stretchy="false" id="alg2.l2.m1.4.4.2.1.1.2" xref="alg2.l2.m1.4.4.2.1.1.1.cmml">(</mo><msup id="alg2.l2.m1.4.4.2.1.1.1" xref="alg2.l2.m1.4.4.2.1.1.1.cmml"><mi id="alg2.l2.m1.4.4.2.1.1.1.2" xref="alg2.l2.m1.4.4.2.1.1.1.2.cmml">G</mi><mi id="alg2.l2.m1.4.4.2.1.1.1.3" xref="alg2.l2.m1.4.4.2.1.1.1.3.cmml">W</mi></msup><mo stretchy="false" id="alg2.l2.m1.4.4.2.1.1.3" xref="alg2.l2.m1.4.4.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l2.m1.4b"><apply id="alg2.l2.m1.4.4.cmml" xref="alg2.l2.m1.4.4"><ci id="alg2.l2.m1.4.4.3.cmml" xref="alg2.l2.m1.4.4.3">←</ci><list id="alg2.l2.m1.3.3.1.2.cmml" xref="alg2.l2.m1.3.3.1.1"><ci id="alg2.l2.m1.1.1.cmml" xref="alg2.l2.m1.1.1">𝑈</ci><ci id="alg2.l2.m1.2.2.cmml" xref="alg2.l2.m1.2.2">𝑆</ci><apply id="alg2.l2.m1.3.3.1.1.1.cmml" xref="alg2.l2.m1.3.3.1.1.1"><csymbol cd="ambiguous" id="alg2.l2.m1.3.3.1.1.1.1.cmml" xref="alg2.l2.m1.3.3.1.1.1">superscript</csymbol><ci id="alg2.l2.m1.3.3.1.1.1.2.cmml" xref="alg2.l2.m1.3.3.1.1.1.2">𝑉</ci><ci id="alg2.l2.m1.3.3.1.1.1.3.cmml" xref="alg2.l2.m1.3.3.1.1.1.3">𝑇</ci></apply></list><apply id="alg2.l2.m1.4.4.2.cmml" xref="alg2.l2.m1.4.4.2"><times id="alg2.l2.m1.4.4.2.2.cmml" xref="alg2.l2.m1.4.4.2.2"></times><ci id="alg2.l2.m1.4.4.2.3a.cmml" xref="alg2.l2.m1.4.4.2.3"><mtext id="alg2.l2.m1.4.4.2.3.cmml" xref="alg2.l2.m1.4.4.2.3">SVD</mtext></ci><apply id="alg2.l2.m1.4.4.2.1.1.1.cmml" xref="alg2.l2.m1.4.4.2.1.1"><csymbol cd="ambiguous" id="alg2.l2.m1.4.4.2.1.1.1.1.cmml" xref="alg2.l2.m1.4.4.2.1.1">superscript</csymbol><ci id="alg2.l2.m1.4.4.2.1.1.1.2.cmml" xref="alg2.l2.m1.4.4.2.1.1.1.2">𝐺</ci><ci id="alg2.l2.m1.4.4.2.1.1.1.3.cmml" xref="alg2.l2.m1.4.4.2.1.1.1.3">𝑊</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l2.m1.4c">U,S,V^{T}\leftarrow\text{SVD}(G^{W})</annotation></semantics></math>

</div>
<div id="alg2.l3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l3.1.1.1" class="ltx_text" style="font-size:80%;">3:</span></span>  <math id="alg2.l3.m1.1" class="ltx_math_unparsed" alttext="P\leftarrow U[:,:r]" display="inline"><semantics id="alg2.l3.m1.1a"><mrow id="alg2.l3.m1.1b"><mi id="alg2.l3.m1.1.1">P</mi><mo stretchy="false" id="alg2.l3.m1.1.2">←</mo><mi id="alg2.l3.m1.1.3">U</mi><mrow id="alg2.l3.m1.1.4"><mo stretchy="false" id="alg2.l3.m1.1.4.1">[</mo><mo rspace="0em" id="alg2.l3.m1.1.4.2">:</mo><mo id="alg2.l3.m1.1.4.3">,</mo><mo rspace="0.278em" id="alg2.l3.m1.1.4.4">:</mo><mi id="alg2.l3.m1.1.4.5">r</mi><mo stretchy="false" id="alg2.l3.m1.1.4.6">]</mo></mrow></mrow><annotation encoding="application/x-tex" id="alg2.l3.m1.1c">P\leftarrow U[:,:r]</annotation></semantics></math> {First <math id="alg2.l3.m2.1" class="ltx_Math" alttext="r" display="inline"><semantics id="alg2.l3.m2.1a"><mi id="alg2.l3.m2.1.1" xref="alg2.l3.m2.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="alg2.l3.m2.1b"><ci id="alg2.l3.m2.1.1.cmml" xref="alg2.l3.m2.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l3.m2.1c">r</annotation></semantics></math> singular vectors}

</div>
<div id="alg2.l4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l4.1.1.1" class="ltx_text" style="font-size:80%;">4:</span></span>  <math id="alg2.l4.m1.1" class="ltx_Math" alttext="P_{q}\leftarrow q_{N}(P)" display="inline"><semantics id="alg2.l4.m1.1a"><mrow id="alg2.l4.m1.1.2" xref="alg2.l4.m1.1.2.cmml"><msub id="alg2.l4.m1.1.2.2" xref="alg2.l4.m1.1.2.2.cmml"><mi id="alg2.l4.m1.1.2.2.2" xref="alg2.l4.m1.1.2.2.2.cmml">P</mi><mi id="alg2.l4.m1.1.2.2.3" xref="alg2.l4.m1.1.2.2.3.cmml">q</mi></msub><mo stretchy="false" id="alg2.l4.m1.1.2.1" xref="alg2.l4.m1.1.2.1.cmml">←</mo><mrow id="alg2.l4.m1.1.2.3" xref="alg2.l4.m1.1.2.3.cmml"><msub id="alg2.l4.m1.1.2.3.2" xref="alg2.l4.m1.1.2.3.2.cmml"><mi id="alg2.l4.m1.1.2.3.2.2" xref="alg2.l4.m1.1.2.3.2.2.cmml">q</mi><mi id="alg2.l4.m1.1.2.3.2.3" xref="alg2.l4.m1.1.2.3.2.3.cmml">N</mi></msub><mo lspace="0em" rspace="0em" id="alg2.l4.m1.1.2.3.1" xref="alg2.l4.m1.1.2.3.1.cmml">​</mo><mrow id="alg2.l4.m1.1.2.3.3.2" xref="alg2.l4.m1.1.2.3.cmml"><mo stretchy="false" id="alg2.l4.m1.1.2.3.3.2.1" xref="alg2.l4.m1.1.2.3.cmml">(</mo><mi id="alg2.l4.m1.1.1" xref="alg2.l4.m1.1.1.cmml">P</mi><mo stretchy="false" id="alg2.l4.m1.1.2.3.3.2.2" xref="alg2.l4.m1.1.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l4.m1.1b"><apply id="alg2.l4.m1.1.2.cmml" xref="alg2.l4.m1.1.2"><ci id="alg2.l4.m1.1.2.1.cmml" xref="alg2.l4.m1.1.2.1">←</ci><apply id="alg2.l4.m1.1.2.2.cmml" xref="alg2.l4.m1.1.2.2"><csymbol cd="ambiguous" id="alg2.l4.m1.1.2.2.1.cmml" xref="alg2.l4.m1.1.2.2">subscript</csymbol><ci id="alg2.l4.m1.1.2.2.2.cmml" xref="alg2.l4.m1.1.2.2.2">𝑃</ci><ci id="alg2.l4.m1.1.2.2.3.cmml" xref="alg2.l4.m1.1.2.2.3">𝑞</ci></apply><apply id="alg2.l4.m1.1.2.3.cmml" xref="alg2.l4.m1.1.2.3"><times id="alg2.l4.m1.1.2.3.1.cmml" xref="alg2.l4.m1.1.2.3.1"></times><apply id="alg2.l4.m1.1.2.3.2.cmml" xref="alg2.l4.m1.1.2.3.2"><csymbol cd="ambiguous" id="alg2.l4.m1.1.2.3.2.1.cmml" xref="alg2.l4.m1.1.2.3.2">subscript</csymbol><ci id="alg2.l4.m1.1.2.3.2.2.cmml" xref="alg2.l4.m1.1.2.3.2.2">𝑞</ci><ci id="alg2.l4.m1.1.2.3.2.3.cmml" xref="alg2.l4.m1.1.2.3.2.3">𝑁</ci></apply><ci id="alg2.l4.m1.1.1.cmml" xref="alg2.l4.m1.1.1">𝑃</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l4.m1.1c">P_{q}\leftarrow q_{N}(P)</annotation></semantics></math>

</div>
<div id="alg2.l5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l5.1.1.1" class="ltx_text" style="font-size:80%;">5:</span></span>  <math id="alg2.l5.m1.1" class="ltx_Math" alttext="B\leftarrow 0" display="inline"><semantics id="alg2.l5.m1.1a"><mrow id="alg2.l5.m1.1.1" xref="alg2.l5.m1.1.1.cmml"><mi id="alg2.l5.m1.1.1.2" xref="alg2.l5.m1.1.1.2.cmml">B</mi><mo stretchy="false" id="alg2.l5.m1.1.1.1" xref="alg2.l5.m1.1.1.1.cmml">←</mo><mn id="alg2.l5.m1.1.1.3" xref="alg2.l5.m1.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="alg2.l5.m1.1b"><apply id="alg2.l5.m1.1.1.cmml" xref="alg2.l5.m1.1.1"><ci id="alg2.l5.m1.1.1.1.cmml" xref="alg2.l5.m1.1.1.1">←</ci><ci id="alg2.l5.m1.1.1.2.cmml" xref="alg2.l5.m1.1.1.2">𝐵</ci><cn type="integer" id="alg2.l5.m1.1.1.3.cmml" xref="alg2.l5.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l5.m1.1c">B\leftarrow 0</annotation></semantics></math>

</div>
<div id="alg2.l6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l6.1.1.1" class="ltx_text" style="font-size:80%;">6:</span></span>  <math id="alg2.l6.m1.1" class="ltx_Math" alttext="\hat{W}\leftarrow W" display="inline"><semantics id="alg2.l6.m1.1a"><mrow id="alg2.l6.m1.1.1" xref="alg2.l6.m1.1.1.cmml"><mover accent="true" id="alg2.l6.m1.1.1.2" xref="alg2.l6.m1.1.1.2.cmml"><mi id="alg2.l6.m1.1.1.2.2" xref="alg2.l6.m1.1.1.2.2.cmml">W</mi><mo id="alg2.l6.m1.1.1.2.1" xref="alg2.l6.m1.1.1.2.1.cmml">^</mo></mover><mo stretchy="false" id="alg2.l6.m1.1.1.1" xref="alg2.l6.m1.1.1.1.cmml">←</mo><mi id="alg2.l6.m1.1.1.3" xref="alg2.l6.m1.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="alg2.l6.m1.1b"><apply id="alg2.l6.m1.1.1.cmml" xref="alg2.l6.m1.1.1"><ci id="alg2.l6.m1.1.1.1.cmml" xref="alg2.l6.m1.1.1.1">←</ci><apply id="alg2.l6.m1.1.1.2.cmml" xref="alg2.l6.m1.1.1.2"><ci id="alg2.l6.m1.1.1.2.1.cmml" xref="alg2.l6.m1.1.1.2.1">^</ci><ci id="alg2.l6.m1.1.1.2.2.cmml" xref="alg2.l6.m1.1.1.2.2">𝑊</ci></apply><ci id="alg2.l6.m1.1.1.3.cmml" xref="alg2.l6.m1.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l6.m1.1c">\hat{W}\leftarrow W</annotation></semantics></math>

</div>
<div id="alg2.l7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l7.1.1.1" class="ltx_text" style="font-size:80%;">7:</span></span>  <span id="alg2.l7.2" class="ltx_text ltx_font_bold">for</span> each <math id="alg2.l7.m1.1" class="ltx_Math" alttext="c" display="inline"><semantics id="alg2.l7.m1.1a"><mi id="alg2.l7.m1.1.1" xref="alg2.l7.m1.1.1.cmml">c</mi><annotation-xml encoding="MathML-Content" id="alg2.l7.m1.1b"><ci id="alg2.l7.m1.1.1.cmml" xref="alg2.l7.m1.1.1">𝑐</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l7.m1.1c">c</annotation></semantics></math> in compensation steps <math id="alg2.l7.m2.1" class="ltx_Math" alttext="C" display="inline"><semantics id="alg2.l7.m2.1a"><mi id="alg2.l7.m2.1.1" xref="alg2.l7.m2.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="alg2.l7.m2.1b"><ci id="alg2.l7.m2.1.1.cmml" xref="alg2.l7.m2.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l7.m2.1c">C</annotation></semantics></math> <span id="alg2.l7.3" class="ltx_text ltx_font_bold">do</span>



</div>
<div id="alg2.l8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l8.1.1.1" class="ltx_text" style="font-size:80%;">8:</span></span>     <math id="alg2.l8.m1.1" class="ltx_Math" alttext="Q_{c}\leftarrow q_{N}(\hat{W})" display="inline"><semantics id="alg2.l8.m1.1a"><mrow id="alg2.l8.m1.1.2" xref="alg2.l8.m1.1.2.cmml"><msub id="alg2.l8.m1.1.2.2" xref="alg2.l8.m1.1.2.2.cmml"><mi id="alg2.l8.m1.1.2.2.2" xref="alg2.l8.m1.1.2.2.2.cmml">Q</mi><mi id="alg2.l8.m1.1.2.2.3" xref="alg2.l8.m1.1.2.2.3.cmml">c</mi></msub><mo stretchy="false" id="alg2.l8.m1.1.2.1" xref="alg2.l8.m1.1.2.1.cmml">←</mo><mrow id="alg2.l8.m1.1.2.3" xref="alg2.l8.m1.1.2.3.cmml"><msub id="alg2.l8.m1.1.2.3.2" xref="alg2.l8.m1.1.2.3.2.cmml"><mi id="alg2.l8.m1.1.2.3.2.2" xref="alg2.l8.m1.1.2.3.2.2.cmml">q</mi><mi id="alg2.l8.m1.1.2.3.2.3" xref="alg2.l8.m1.1.2.3.2.3.cmml">N</mi></msub><mo lspace="0em" rspace="0em" id="alg2.l8.m1.1.2.3.1" xref="alg2.l8.m1.1.2.3.1.cmml">​</mo><mrow id="alg2.l8.m1.1.2.3.3.2" xref="alg2.l8.m1.1.1.cmml"><mo stretchy="false" id="alg2.l8.m1.1.2.3.3.2.1" xref="alg2.l8.m1.1.1.cmml">(</mo><mover accent="true" id="alg2.l8.m1.1.1" xref="alg2.l8.m1.1.1.cmml"><mi id="alg2.l8.m1.1.1.2" xref="alg2.l8.m1.1.1.2.cmml">W</mi><mo id="alg2.l8.m1.1.1.1" xref="alg2.l8.m1.1.1.1.cmml">^</mo></mover><mo stretchy="false" id="alg2.l8.m1.1.2.3.3.2.2" xref="alg2.l8.m1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l8.m1.1b"><apply id="alg2.l8.m1.1.2.cmml" xref="alg2.l8.m1.1.2"><ci id="alg2.l8.m1.1.2.1.cmml" xref="alg2.l8.m1.1.2.1">←</ci><apply id="alg2.l8.m1.1.2.2.cmml" xref="alg2.l8.m1.1.2.2"><csymbol cd="ambiguous" id="alg2.l8.m1.1.2.2.1.cmml" xref="alg2.l8.m1.1.2.2">subscript</csymbol><ci id="alg2.l8.m1.1.2.2.2.cmml" xref="alg2.l8.m1.1.2.2.2">𝑄</ci><ci id="alg2.l8.m1.1.2.2.3.cmml" xref="alg2.l8.m1.1.2.2.3">𝑐</ci></apply><apply id="alg2.l8.m1.1.2.3.cmml" xref="alg2.l8.m1.1.2.3"><times id="alg2.l8.m1.1.2.3.1.cmml" xref="alg2.l8.m1.1.2.3.1"></times><apply id="alg2.l8.m1.1.2.3.2.cmml" xref="alg2.l8.m1.1.2.3.2"><csymbol cd="ambiguous" id="alg2.l8.m1.1.2.3.2.1.cmml" xref="alg2.l8.m1.1.2.3.2">subscript</csymbol><ci id="alg2.l8.m1.1.2.3.2.2.cmml" xref="alg2.l8.m1.1.2.3.2.2">𝑞</ci><ci id="alg2.l8.m1.1.2.3.2.3.cmml" xref="alg2.l8.m1.1.2.3.2.3">𝑁</ci></apply><apply id="alg2.l8.m1.1.1.cmml" xref="alg2.l8.m1.1.2.3.3.2"><ci id="alg2.l8.m1.1.1.1.cmml" xref="alg2.l8.m1.1.1.1">^</ci><ci id="alg2.l8.m1.1.1.2.cmml" xref="alg2.l8.m1.1.1.2">𝑊</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l8.m1.1c">Q_{c}\leftarrow q_{N}(\hat{W})</annotation></semantics></math>

</div>
<div id="alg2.l9" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l9.1.1.1" class="ltx_text" style="font-size:80%;">9:</span></span>     <math id="alg2.l9.m1.1" class="ltx_Math" alttext="B\leftarrow P^{+}(\hat{W}-Q_{c})" display="inline"><semantics id="alg2.l9.m1.1a"><mrow id="alg2.l9.m1.1.1" xref="alg2.l9.m1.1.1.cmml"><mi id="alg2.l9.m1.1.1.3" xref="alg2.l9.m1.1.1.3.cmml">B</mi><mo stretchy="false" id="alg2.l9.m1.1.1.2" xref="alg2.l9.m1.1.1.2.cmml">←</mo><mrow id="alg2.l9.m1.1.1.1" xref="alg2.l9.m1.1.1.1.cmml"><msup id="alg2.l9.m1.1.1.1.3" xref="alg2.l9.m1.1.1.1.3.cmml"><mi id="alg2.l9.m1.1.1.1.3.2" xref="alg2.l9.m1.1.1.1.3.2.cmml">P</mi><mo id="alg2.l9.m1.1.1.1.3.3" xref="alg2.l9.m1.1.1.1.3.3.cmml">+</mo></msup><mo lspace="0em" rspace="0em" id="alg2.l9.m1.1.1.1.2" xref="alg2.l9.m1.1.1.1.2.cmml">​</mo><mrow id="alg2.l9.m1.1.1.1.1.1" xref="alg2.l9.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg2.l9.m1.1.1.1.1.1.2" xref="alg2.l9.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="alg2.l9.m1.1.1.1.1.1.1" xref="alg2.l9.m1.1.1.1.1.1.1.cmml"><mover accent="true" id="alg2.l9.m1.1.1.1.1.1.1.2" xref="alg2.l9.m1.1.1.1.1.1.1.2.cmml"><mi id="alg2.l9.m1.1.1.1.1.1.1.2.2" xref="alg2.l9.m1.1.1.1.1.1.1.2.2.cmml">W</mi><mo id="alg2.l9.m1.1.1.1.1.1.1.2.1" xref="alg2.l9.m1.1.1.1.1.1.1.2.1.cmml">^</mo></mover><mo id="alg2.l9.m1.1.1.1.1.1.1.1" xref="alg2.l9.m1.1.1.1.1.1.1.1.cmml">−</mo><msub id="alg2.l9.m1.1.1.1.1.1.1.3" xref="alg2.l9.m1.1.1.1.1.1.1.3.cmml"><mi id="alg2.l9.m1.1.1.1.1.1.1.3.2" xref="alg2.l9.m1.1.1.1.1.1.1.3.2.cmml">Q</mi><mi id="alg2.l9.m1.1.1.1.1.1.1.3.3" xref="alg2.l9.m1.1.1.1.1.1.1.3.3.cmml">c</mi></msub></mrow><mo stretchy="false" id="alg2.l9.m1.1.1.1.1.1.3" xref="alg2.l9.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l9.m1.1b"><apply id="alg2.l9.m1.1.1.cmml" xref="alg2.l9.m1.1.1"><ci id="alg2.l9.m1.1.1.2.cmml" xref="alg2.l9.m1.1.1.2">←</ci><ci id="alg2.l9.m1.1.1.3.cmml" xref="alg2.l9.m1.1.1.3">𝐵</ci><apply id="alg2.l9.m1.1.1.1.cmml" xref="alg2.l9.m1.1.1.1"><times id="alg2.l9.m1.1.1.1.2.cmml" xref="alg2.l9.m1.1.1.1.2"></times><apply id="alg2.l9.m1.1.1.1.3.cmml" xref="alg2.l9.m1.1.1.1.3"><csymbol cd="ambiguous" id="alg2.l9.m1.1.1.1.3.1.cmml" xref="alg2.l9.m1.1.1.1.3">superscript</csymbol><ci id="alg2.l9.m1.1.1.1.3.2.cmml" xref="alg2.l9.m1.1.1.1.3.2">𝑃</ci><plus id="alg2.l9.m1.1.1.1.3.3.cmml" xref="alg2.l9.m1.1.1.1.3.3"></plus></apply><apply id="alg2.l9.m1.1.1.1.1.1.1.cmml" xref="alg2.l9.m1.1.1.1.1.1"><minus id="alg2.l9.m1.1.1.1.1.1.1.1.cmml" xref="alg2.l9.m1.1.1.1.1.1.1.1"></minus><apply id="alg2.l9.m1.1.1.1.1.1.1.2.cmml" xref="alg2.l9.m1.1.1.1.1.1.1.2"><ci id="alg2.l9.m1.1.1.1.1.1.1.2.1.cmml" xref="alg2.l9.m1.1.1.1.1.1.1.2.1">^</ci><ci id="alg2.l9.m1.1.1.1.1.1.1.2.2.cmml" xref="alg2.l9.m1.1.1.1.1.1.1.2.2">𝑊</ci></apply><apply id="alg2.l9.m1.1.1.1.1.1.1.3.cmml" xref="alg2.l9.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="alg2.l9.m1.1.1.1.1.1.1.3.1.cmml" xref="alg2.l9.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="alg2.l9.m1.1.1.1.1.1.1.3.2.cmml" xref="alg2.l9.m1.1.1.1.1.1.1.3.2">𝑄</ci><ci id="alg2.l9.m1.1.1.1.1.1.1.3.3.cmml" xref="alg2.l9.m1.1.1.1.1.1.1.3.3">𝑐</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l9.m1.1c">B\leftarrow P^{+}(\hat{W}-Q_{c})</annotation></semantics></math>

</div>
<div id="alg2.l10" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l10.1.1.1" class="ltx_text" style="font-size:80%;">10:</span></span>     <math id="alg2.l10.m1.1" class="ltx_Math" alttext="\hat{W}\leftarrow W-PB" display="inline"><semantics id="alg2.l10.m1.1a"><mrow id="alg2.l10.m1.1.1" xref="alg2.l10.m1.1.1.cmml"><mover accent="true" id="alg2.l10.m1.1.1.2" xref="alg2.l10.m1.1.1.2.cmml"><mi id="alg2.l10.m1.1.1.2.2" xref="alg2.l10.m1.1.1.2.2.cmml">W</mi><mo id="alg2.l10.m1.1.1.2.1" xref="alg2.l10.m1.1.1.2.1.cmml">^</mo></mover><mo stretchy="false" id="alg2.l10.m1.1.1.1" xref="alg2.l10.m1.1.1.1.cmml">←</mo><mrow id="alg2.l10.m1.1.1.3" xref="alg2.l10.m1.1.1.3.cmml"><mi id="alg2.l10.m1.1.1.3.2" xref="alg2.l10.m1.1.1.3.2.cmml">W</mi><mo id="alg2.l10.m1.1.1.3.1" xref="alg2.l10.m1.1.1.3.1.cmml">−</mo><mrow id="alg2.l10.m1.1.1.3.3" xref="alg2.l10.m1.1.1.3.3.cmml"><mi id="alg2.l10.m1.1.1.3.3.2" xref="alg2.l10.m1.1.1.3.3.2.cmml">P</mi><mo lspace="0em" rspace="0em" id="alg2.l10.m1.1.1.3.3.1" xref="alg2.l10.m1.1.1.3.3.1.cmml">​</mo><mi id="alg2.l10.m1.1.1.3.3.3" xref="alg2.l10.m1.1.1.3.3.3.cmml">B</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l10.m1.1b"><apply id="alg2.l10.m1.1.1.cmml" xref="alg2.l10.m1.1.1"><ci id="alg2.l10.m1.1.1.1.cmml" xref="alg2.l10.m1.1.1.1">←</ci><apply id="alg2.l10.m1.1.1.2.cmml" xref="alg2.l10.m1.1.1.2"><ci id="alg2.l10.m1.1.1.2.1.cmml" xref="alg2.l10.m1.1.1.2.1">^</ci><ci id="alg2.l10.m1.1.1.2.2.cmml" xref="alg2.l10.m1.1.1.2.2">𝑊</ci></apply><apply id="alg2.l10.m1.1.1.3.cmml" xref="alg2.l10.m1.1.1.3"><minus id="alg2.l10.m1.1.1.3.1.cmml" xref="alg2.l10.m1.1.1.3.1"></minus><ci id="alg2.l10.m1.1.1.3.2.cmml" xref="alg2.l10.m1.1.1.3.2">𝑊</ci><apply id="alg2.l10.m1.1.1.3.3.cmml" xref="alg2.l10.m1.1.1.3.3"><times id="alg2.l10.m1.1.1.3.3.1.cmml" xref="alg2.l10.m1.1.1.3.3.1"></times><ci id="alg2.l10.m1.1.1.3.3.2.cmml" xref="alg2.l10.m1.1.1.3.3.2">𝑃</ci><ci id="alg2.l10.m1.1.1.3.3.3.cmml" xref="alg2.l10.m1.1.1.3.3.3">𝐵</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l10.m1.1c">\hat{W}\leftarrow W-PB</annotation></semantics></math>

</div>
<div id="alg2.l11" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l11.1.1.1" class="ltx_text" style="font-size:80%;">11:</span></span>  <span id="alg2.l11.2" class="ltx_text ltx_font_bold">return</span>  <math id="alg2.l11.m1.3" class="ltx_Math" alttext="Q_{c},B,P_{q}" display="inline"><semantics id="alg2.l11.m1.3a"><mrow id="alg2.l11.m1.3.3.2" xref="alg2.l11.m1.3.3.3.cmml"><msub id="alg2.l11.m1.2.2.1.1" xref="alg2.l11.m1.2.2.1.1.cmml"><mi id="alg2.l11.m1.2.2.1.1.2" xref="alg2.l11.m1.2.2.1.1.2.cmml">Q</mi><mi id="alg2.l11.m1.2.2.1.1.3" xref="alg2.l11.m1.2.2.1.1.3.cmml">c</mi></msub><mo id="alg2.l11.m1.3.3.2.3" xref="alg2.l11.m1.3.3.3.cmml">,</mo><mi id="alg2.l11.m1.1.1" xref="alg2.l11.m1.1.1.cmml">B</mi><mo id="alg2.l11.m1.3.3.2.4" xref="alg2.l11.m1.3.3.3.cmml">,</mo><msub id="alg2.l11.m1.3.3.2.2" xref="alg2.l11.m1.3.3.2.2.cmml"><mi id="alg2.l11.m1.3.3.2.2.2" xref="alg2.l11.m1.3.3.2.2.2.cmml">P</mi><mi id="alg2.l11.m1.3.3.2.2.3" xref="alg2.l11.m1.3.3.2.2.3.cmml">q</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="alg2.l11.m1.3b"><list id="alg2.l11.m1.3.3.3.cmml" xref="alg2.l11.m1.3.3.2"><apply id="alg2.l11.m1.2.2.1.1.cmml" xref="alg2.l11.m1.2.2.1.1"><csymbol cd="ambiguous" id="alg2.l11.m1.2.2.1.1.1.cmml" xref="alg2.l11.m1.2.2.1.1">subscript</csymbol><ci id="alg2.l11.m1.2.2.1.1.2.cmml" xref="alg2.l11.m1.2.2.1.1.2">𝑄</ci><ci id="alg2.l11.m1.2.2.1.1.3.cmml" xref="alg2.l11.m1.2.2.1.1.3">𝑐</ci></apply><ci id="alg2.l11.m1.1.1.cmml" xref="alg2.l11.m1.1.1">𝐵</ci><apply id="alg2.l11.m1.3.3.2.2.cmml" xref="alg2.l11.m1.3.3.2.2"><csymbol cd="ambiguous" id="alg2.l11.m1.3.3.2.2.1.cmml" xref="alg2.l11.m1.3.3.2.2">subscript</csymbol><ci id="alg2.l11.m1.3.3.2.2.2.cmml" xref="alg2.l11.m1.3.3.2.2.2">𝑃</ci><ci id="alg2.l11.m1.3.3.2.2.3.cmml" xref="alg2.l11.m1.3.3.2.2.3">𝑞</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg2.l11.m1.3c">Q_{c},B,P_{q}</annotation></semantics></math>

</div>
</div>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S3.F3.3.1.1" class="ltx_text" style="font-size:90%;">Figure 3</span>: </span><span id="S3.F3.4.2" class="ltx_text" style="font-size:90%;">Pseudo-code for <span id="S3.F3.4.2.1" class="ltx_text ltx_font_smallcaps">LoQT</span>.</span></figcaption>
</figure>
</section>
<section id="S3.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.5 </span>Compensating for Errors Introduced by Quantization</h3>

<div id="S3.SS5.p1" class="ltx_para">
<p id="S3.SS5.p1.1" class="ltx_p">As the quantization process inevitably leads to a discrepancy between the non-quantized and quantized versions of <math id="S3.SS5.p1.1.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S3.SS5.p1.1.m1.1a"><mi id="S3.SS5.p1.1.m1.1.1" xref="S3.SS5.p1.1.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.SS5.p1.1.m1.1b"><ci id="S3.SS5.p1.1.m1.1.1.cmml" xref="S3.SS5.p1.1.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p1.1.m1.1c">W</annotation></semantics></math> we wish to reduce this effect as much as possible. While compensating for quantization errors has been done before <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib21" title="" class="ltx_ref">21</a>]</cite>, we need a tailored solution for <span id="S3.SS5.p1.1.1" class="ltx_text ltx_font_smallcaps">LoQT</span>.</p>
</div>
<div id="S3.SS5.p2" class="ltx_para">
<p id="S3.SS5.p2.14" class="ltx_p">During the merging phase, we first dequantize to obtain <math id="S3.SS5.p2.1.m1.1" class="ltx_Math" alttext="W_{T-1}" display="inline"><semantics id="S3.SS5.p2.1.m1.1a"><msub id="S3.SS5.p2.1.m1.1.1" xref="S3.SS5.p2.1.m1.1.1.cmml"><mi id="S3.SS5.p2.1.m1.1.1.2" xref="S3.SS5.p2.1.m1.1.1.2.cmml">W</mi><mrow id="S3.SS5.p2.1.m1.1.1.3" xref="S3.SS5.p2.1.m1.1.1.3.cmml"><mi id="S3.SS5.p2.1.m1.1.1.3.2" xref="S3.SS5.p2.1.m1.1.1.3.2.cmml">T</mi><mo id="S3.SS5.p2.1.m1.1.1.3.1" xref="S3.SS5.p2.1.m1.1.1.3.1.cmml">−</mo><mn id="S3.SS5.p2.1.m1.1.1.3.3" xref="S3.SS5.p2.1.m1.1.1.3.3.cmml">1</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS5.p2.1.m1.1b"><apply id="S3.SS5.p2.1.m1.1.1.cmml" xref="S3.SS5.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS5.p2.1.m1.1.1.1.cmml" xref="S3.SS5.p2.1.m1.1.1">subscript</csymbol><ci id="S3.SS5.p2.1.m1.1.1.2.cmml" xref="S3.SS5.p2.1.m1.1.1.2">𝑊</ci><apply id="S3.SS5.p2.1.m1.1.1.3.cmml" xref="S3.SS5.p2.1.m1.1.1.3"><minus id="S3.SS5.p2.1.m1.1.1.3.1.cmml" xref="S3.SS5.p2.1.m1.1.1.3.1"></minus><ci id="S3.SS5.p2.1.m1.1.1.3.2.cmml" xref="S3.SS5.p2.1.m1.1.1.3.2">𝑇</ci><cn type="integer" id="S3.SS5.p2.1.m1.1.1.3.3.cmml" xref="S3.SS5.p2.1.m1.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p2.1.m1.1c">W_{T-1}</annotation></semantics></math> and <math id="S3.SS5.p2.2.m2.1" class="ltx_Math" alttext="P_{T-1}" display="inline"><semantics id="S3.SS5.p2.2.m2.1a"><msub id="S3.SS5.p2.2.m2.1.1" xref="S3.SS5.p2.2.m2.1.1.cmml"><mi id="S3.SS5.p2.2.m2.1.1.2" xref="S3.SS5.p2.2.m2.1.1.2.cmml">P</mi><mrow id="S3.SS5.p2.2.m2.1.1.3" xref="S3.SS5.p2.2.m2.1.1.3.cmml"><mi id="S3.SS5.p2.2.m2.1.1.3.2" xref="S3.SS5.p2.2.m2.1.1.3.2.cmml">T</mi><mo id="S3.SS5.p2.2.m2.1.1.3.1" xref="S3.SS5.p2.2.m2.1.1.3.1.cmml">−</mo><mn id="S3.SS5.p2.2.m2.1.1.3.3" xref="S3.SS5.p2.2.m2.1.1.3.3.cmml">1</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS5.p2.2.m2.1b"><apply id="S3.SS5.p2.2.m2.1.1.cmml" xref="S3.SS5.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS5.p2.2.m2.1.1.1.cmml" xref="S3.SS5.p2.2.m2.1.1">subscript</csymbol><ci id="S3.SS5.p2.2.m2.1.1.2.cmml" xref="S3.SS5.p2.2.m2.1.1.2">𝑃</ci><apply id="S3.SS5.p2.2.m2.1.1.3.cmml" xref="S3.SS5.p2.2.m2.1.1.3"><minus id="S3.SS5.p2.2.m2.1.1.3.1.cmml" xref="S3.SS5.p2.2.m2.1.1.3.1"></minus><ci id="S3.SS5.p2.2.m2.1.1.3.2.cmml" xref="S3.SS5.p2.2.m2.1.1.3.2">𝑇</ci><cn type="integer" id="S3.SS5.p2.2.m2.1.1.3.3.cmml" xref="S3.SS5.p2.2.m2.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p2.2.m2.1c">P_{T-1}</annotation></semantics></math>, and then compute the update <math id="S3.SS5.p2.3.m3.1" class="ltx_Math" alttext="W_{T}=W_{T-1}+P_{T-1}B_{T-1}" display="inline"><semantics id="S3.SS5.p2.3.m3.1a"><mrow id="S3.SS5.p2.3.m3.1.1" xref="S3.SS5.p2.3.m3.1.1.cmml"><msub id="S3.SS5.p2.3.m3.1.1.2" xref="S3.SS5.p2.3.m3.1.1.2.cmml"><mi id="S3.SS5.p2.3.m3.1.1.2.2" xref="S3.SS5.p2.3.m3.1.1.2.2.cmml">W</mi><mi id="S3.SS5.p2.3.m3.1.1.2.3" xref="S3.SS5.p2.3.m3.1.1.2.3.cmml">T</mi></msub><mo id="S3.SS5.p2.3.m3.1.1.1" xref="S3.SS5.p2.3.m3.1.1.1.cmml">=</mo><mrow id="S3.SS5.p2.3.m3.1.1.3" xref="S3.SS5.p2.3.m3.1.1.3.cmml"><msub id="S3.SS5.p2.3.m3.1.1.3.2" xref="S3.SS5.p2.3.m3.1.1.3.2.cmml"><mi id="S3.SS5.p2.3.m3.1.1.3.2.2" xref="S3.SS5.p2.3.m3.1.1.3.2.2.cmml">W</mi><mrow id="S3.SS5.p2.3.m3.1.1.3.2.3" xref="S3.SS5.p2.3.m3.1.1.3.2.3.cmml"><mi id="S3.SS5.p2.3.m3.1.1.3.2.3.2" xref="S3.SS5.p2.3.m3.1.1.3.2.3.2.cmml">T</mi><mo id="S3.SS5.p2.3.m3.1.1.3.2.3.1" xref="S3.SS5.p2.3.m3.1.1.3.2.3.1.cmml">−</mo><mn id="S3.SS5.p2.3.m3.1.1.3.2.3.3" xref="S3.SS5.p2.3.m3.1.1.3.2.3.3.cmml">1</mn></mrow></msub><mo id="S3.SS5.p2.3.m3.1.1.3.1" xref="S3.SS5.p2.3.m3.1.1.3.1.cmml">+</mo><mrow id="S3.SS5.p2.3.m3.1.1.3.3" xref="S3.SS5.p2.3.m3.1.1.3.3.cmml"><msub id="S3.SS5.p2.3.m3.1.1.3.3.2" xref="S3.SS5.p2.3.m3.1.1.3.3.2.cmml"><mi id="S3.SS5.p2.3.m3.1.1.3.3.2.2" xref="S3.SS5.p2.3.m3.1.1.3.3.2.2.cmml">P</mi><mrow id="S3.SS5.p2.3.m3.1.1.3.3.2.3" xref="S3.SS5.p2.3.m3.1.1.3.3.2.3.cmml"><mi id="S3.SS5.p2.3.m3.1.1.3.3.2.3.2" xref="S3.SS5.p2.3.m3.1.1.3.3.2.3.2.cmml">T</mi><mo id="S3.SS5.p2.3.m3.1.1.3.3.2.3.1" xref="S3.SS5.p2.3.m3.1.1.3.3.2.3.1.cmml">−</mo><mn id="S3.SS5.p2.3.m3.1.1.3.3.2.3.3" xref="S3.SS5.p2.3.m3.1.1.3.3.2.3.3.cmml">1</mn></mrow></msub><mo lspace="0em" rspace="0em" id="S3.SS5.p2.3.m3.1.1.3.3.1" xref="S3.SS5.p2.3.m3.1.1.3.3.1.cmml">​</mo><msub id="S3.SS5.p2.3.m3.1.1.3.3.3" xref="S3.SS5.p2.3.m3.1.1.3.3.3.cmml"><mi id="S3.SS5.p2.3.m3.1.1.3.3.3.2" xref="S3.SS5.p2.3.m3.1.1.3.3.3.2.cmml">B</mi><mrow id="S3.SS5.p2.3.m3.1.1.3.3.3.3" xref="S3.SS5.p2.3.m3.1.1.3.3.3.3.cmml"><mi id="S3.SS5.p2.3.m3.1.1.3.3.3.3.2" xref="S3.SS5.p2.3.m3.1.1.3.3.3.3.2.cmml">T</mi><mo id="S3.SS5.p2.3.m3.1.1.3.3.3.3.1" xref="S3.SS5.p2.3.m3.1.1.3.3.3.3.1.cmml">−</mo><mn id="S3.SS5.p2.3.m3.1.1.3.3.3.3.3" xref="S3.SS5.p2.3.m3.1.1.3.3.3.3.3.cmml">1</mn></mrow></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS5.p2.3.m3.1b"><apply id="S3.SS5.p2.3.m3.1.1.cmml" xref="S3.SS5.p2.3.m3.1.1"><eq id="S3.SS5.p2.3.m3.1.1.1.cmml" xref="S3.SS5.p2.3.m3.1.1.1"></eq><apply id="S3.SS5.p2.3.m3.1.1.2.cmml" xref="S3.SS5.p2.3.m3.1.1.2"><csymbol cd="ambiguous" id="S3.SS5.p2.3.m3.1.1.2.1.cmml" xref="S3.SS5.p2.3.m3.1.1.2">subscript</csymbol><ci id="S3.SS5.p2.3.m3.1.1.2.2.cmml" xref="S3.SS5.p2.3.m3.1.1.2.2">𝑊</ci><ci id="S3.SS5.p2.3.m3.1.1.2.3.cmml" xref="S3.SS5.p2.3.m3.1.1.2.3">𝑇</ci></apply><apply id="S3.SS5.p2.3.m3.1.1.3.cmml" xref="S3.SS5.p2.3.m3.1.1.3"><plus id="S3.SS5.p2.3.m3.1.1.3.1.cmml" xref="S3.SS5.p2.3.m3.1.1.3.1"></plus><apply id="S3.SS5.p2.3.m3.1.1.3.2.cmml" xref="S3.SS5.p2.3.m3.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS5.p2.3.m3.1.1.3.2.1.cmml" xref="S3.SS5.p2.3.m3.1.1.3.2">subscript</csymbol><ci id="S3.SS5.p2.3.m3.1.1.3.2.2.cmml" xref="S3.SS5.p2.3.m3.1.1.3.2.2">𝑊</ci><apply id="S3.SS5.p2.3.m3.1.1.3.2.3.cmml" xref="S3.SS5.p2.3.m3.1.1.3.2.3"><minus id="S3.SS5.p2.3.m3.1.1.3.2.3.1.cmml" xref="S3.SS5.p2.3.m3.1.1.3.2.3.1"></minus><ci id="S3.SS5.p2.3.m3.1.1.3.2.3.2.cmml" xref="S3.SS5.p2.3.m3.1.1.3.2.3.2">𝑇</ci><cn type="integer" id="S3.SS5.p2.3.m3.1.1.3.2.3.3.cmml" xref="S3.SS5.p2.3.m3.1.1.3.2.3.3">1</cn></apply></apply><apply id="S3.SS5.p2.3.m3.1.1.3.3.cmml" xref="S3.SS5.p2.3.m3.1.1.3.3"><times id="S3.SS5.p2.3.m3.1.1.3.3.1.cmml" xref="S3.SS5.p2.3.m3.1.1.3.3.1"></times><apply id="S3.SS5.p2.3.m3.1.1.3.3.2.cmml" xref="S3.SS5.p2.3.m3.1.1.3.3.2"><csymbol cd="ambiguous" id="S3.SS5.p2.3.m3.1.1.3.3.2.1.cmml" xref="S3.SS5.p2.3.m3.1.1.3.3.2">subscript</csymbol><ci id="S3.SS5.p2.3.m3.1.1.3.3.2.2.cmml" xref="S3.SS5.p2.3.m3.1.1.3.3.2.2">𝑃</ci><apply id="S3.SS5.p2.3.m3.1.1.3.3.2.3.cmml" xref="S3.SS5.p2.3.m3.1.1.3.3.2.3"><minus id="S3.SS5.p2.3.m3.1.1.3.3.2.3.1.cmml" xref="S3.SS5.p2.3.m3.1.1.3.3.2.3.1"></minus><ci id="S3.SS5.p2.3.m3.1.1.3.3.2.3.2.cmml" xref="S3.SS5.p2.3.m3.1.1.3.3.2.3.2">𝑇</ci><cn type="integer" id="S3.SS5.p2.3.m3.1.1.3.3.2.3.3.cmml" xref="S3.SS5.p2.3.m3.1.1.3.3.2.3.3">1</cn></apply></apply><apply id="S3.SS5.p2.3.m3.1.1.3.3.3.cmml" xref="S3.SS5.p2.3.m3.1.1.3.3.3"><csymbol cd="ambiguous" id="S3.SS5.p2.3.m3.1.1.3.3.3.1.cmml" xref="S3.SS5.p2.3.m3.1.1.3.3.3">subscript</csymbol><ci id="S3.SS5.p2.3.m3.1.1.3.3.3.2.cmml" xref="S3.SS5.p2.3.m3.1.1.3.3.3.2">𝐵</ci><apply id="S3.SS5.p2.3.m3.1.1.3.3.3.3.cmml" xref="S3.SS5.p2.3.m3.1.1.3.3.3.3"><minus id="S3.SS5.p2.3.m3.1.1.3.3.3.3.1.cmml" xref="S3.SS5.p2.3.m3.1.1.3.3.3.3.1"></minus><ci id="S3.SS5.p2.3.m3.1.1.3.3.3.3.2.cmml" xref="S3.SS5.p2.3.m3.1.1.3.3.3.3.2">𝑇</ci><cn type="integer" id="S3.SS5.p2.3.m3.1.1.3.3.3.3.3.cmml" xref="S3.SS5.p2.3.m3.1.1.3.3.3.3.3">1</cn></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p2.3.m3.1c">W_{T}=W_{T-1}+P_{T-1}B_{T-1}</annotation></semantics></math>. This is immediately followed by re-quantizing to get <math id="S3.SS5.p2.4.m4.1" class="ltx_Math" alttext="Q_{T}=q_{\text{NF4}}(W_{T})" display="inline"><semantics id="S3.SS5.p2.4.m4.1a"><mrow id="S3.SS5.p2.4.m4.1.1" xref="S3.SS5.p2.4.m4.1.1.cmml"><msub id="S3.SS5.p2.4.m4.1.1.3" xref="S3.SS5.p2.4.m4.1.1.3.cmml"><mi id="S3.SS5.p2.4.m4.1.1.3.2" xref="S3.SS5.p2.4.m4.1.1.3.2.cmml">Q</mi><mi id="S3.SS5.p2.4.m4.1.1.3.3" xref="S3.SS5.p2.4.m4.1.1.3.3.cmml">T</mi></msub><mo id="S3.SS5.p2.4.m4.1.1.2" xref="S3.SS5.p2.4.m4.1.1.2.cmml">=</mo><mrow id="S3.SS5.p2.4.m4.1.1.1" xref="S3.SS5.p2.4.m4.1.1.1.cmml"><msub id="S3.SS5.p2.4.m4.1.1.1.3" xref="S3.SS5.p2.4.m4.1.1.1.3.cmml"><mi id="S3.SS5.p2.4.m4.1.1.1.3.2" xref="S3.SS5.p2.4.m4.1.1.1.3.2.cmml">q</mi><mtext id="S3.SS5.p2.4.m4.1.1.1.3.3" xref="S3.SS5.p2.4.m4.1.1.1.3.3a.cmml">NF4</mtext></msub><mo lspace="0em" rspace="0em" id="S3.SS5.p2.4.m4.1.1.1.2" xref="S3.SS5.p2.4.m4.1.1.1.2.cmml">​</mo><mrow id="S3.SS5.p2.4.m4.1.1.1.1.1" xref="S3.SS5.p2.4.m4.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS5.p2.4.m4.1.1.1.1.1.2" xref="S3.SS5.p2.4.m4.1.1.1.1.1.1.cmml">(</mo><msub id="S3.SS5.p2.4.m4.1.1.1.1.1.1" xref="S3.SS5.p2.4.m4.1.1.1.1.1.1.cmml"><mi id="S3.SS5.p2.4.m4.1.1.1.1.1.1.2" xref="S3.SS5.p2.4.m4.1.1.1.1.1.1.2.cmml">W</mi><mi id="S3.SS5.p2.4.m4.1.1.1.1.1.1.3" xref="S3.SS5.p2.4.m4.1.1.1.1.1.1.3.cmml">T</mi></msub><mo stretchy="false" id="S3.SS5.p2.4.m4.1.1.1.1.1.3" xref="S3.SS5.p2.4.m4.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS5.p2.4.m4.1b"><apply id="S3.SS5.p2.4.m4.1.1.cmml" xref="S3.SS5.p2.4.m4.1.1"><eq id="S3.SS5.p2.4.m4.1.1.2.cmml" xref="S3.SS5.p2.4.m4.1.1.2"></eq><apply id="S3.SS5.p2.4.m4.1.1.3.cmml" xref="S3.SS5.p2.4.m4.1.1.3"><csymbol cd="ambiguous" id="S3.SS5.p2.4.m4.1.1.3.1.cmml" xref="S3.SS5.p2.4.m4.1.1.3">subscript</csymbol><ci id="S3.SS5.p2.4.m4.1.1.3.2.cmml" xref="S3.SS5.p2.4.m4.1.1.3.2">𝑄</ci><ci id="S3.SS5.p2.4.m4.1.1.3.3.cmml" xref="S3.SS5.p2.4.m4.1.1.3.3">𝑇</ci></apply><apply id="S3.SS5.p2.4.m4.1.1.1.cmml" xref="S3.SS5.p2.4.m4.1.1.1"><times id="S3.SS5.p2.4.m4.1.1.1.2.cmml" xref="S3.SS5.p2.4.m4.1.1.1.2"></times><apply id="S3.SS5.p2.4.m4.1.1.1.3.cmml" xref="S3.SS5.p2.4.m4.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS5.p2.4.m4.1.1.1.3.1.cmml" xref="S3.SS5.p2.4.m4.1.1.1.3">subscript</csymbol><ci id="S3.SS5.p2.4.m4.1.1.1.3.2.cmml" xref="S3.SS5.p2.4.m4.1.1.1.3.2">𝑞</ci><ci id="S3.SS5.p2.4.m4.1.1.1.3.3a.cmml" xref="S3.SS5.p2.4.m4.1.1.1.3.3"><mtext mathsize="70%" id="S3.SS5.p2.4.m4.1.1.1.3.3.cmml" xref="S3.SS5.p2.4.m4.1.1.1.3.3">NF4</mtext></ci></apply><apply id="S3.SS5.p2.4.m4.1.1.1.1.1.1.cmml" xref="S3.SS5.p2.4.m4.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS5.p2.4.m4.1.1.1.1.1.1.1.cmml" xref="S3.SS5.p2.4.m4.1.1.1.1.1">subscript</csymbol><ci id="S3.SS5.p2.4.m4.1.1.1.1.1.1.2.cmml" xref="S3.SS5.p2.4.m4.1.1.1.1.1.1.2">𝑊</ci><ci id="S3.SS5.p2.4.m4.1.1.1.1.1.1.3.cmml" xref="S3.SS5.p2.4.m4.1.1.1.1.1.1.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p2.4.m4.1c">Q_{T}=q_{\text{NF4}}(W_{T})</annotation></semantics></math>. Our goal is thus to minimize the quantization error <math id="S3.SS5.p2.5.m5.1" class="ltx_Math" alttext="\|(Q_{T}+P_{T}B_{T})-W_{T}\|" display="inline"><semantics id="S3.SS5.p2.5.m5.1a"><mrow id="S3.SS5.p2.5.m5.1.1.1" xref="S3.SS5.p2.5.m5.1.1.2.cmml"><mo stretchy="false" id="S3.SS5.p2.5.m5.1.1.1.2" xref="S3.SS5.p2.5.m5.1.1.2.1.cmml">‖</mo><mrow id="S3.SS5.p2.5.m5.1.1.1.1" xref="S3.SS5.p2.5.m5.1.1.1.1.cmml"><mrow id="S3.SS5.p2.5.m5.1.1.1.1.1.1" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS5.p2.5.m5.1.1.1.1.1.1.2" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.cmml"><msub id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.2" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.2.cmml"><mi id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.2.2" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.2.2.cmml">Q</mi><mi id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.2.3" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.2.3.cmml">T</mi></msub><mo id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.1" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.cmml"><msub id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.2" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.2.cmml"><mi id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.2.2" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.2.2.cmml">P</mi><mi id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.2.3" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.2.3.cmml">T</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.1" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.3" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.3.cmml"><mi id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.3.2" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.3.2.cmml">B</mi><mi id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.3.3" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.3.3.cmml">T</mi></msub></mrow></mrow><mo stretchy="false" id="S3.SS5.p2.5.m5.1.1.1.1.1.1.3" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo id="S3.SS5.p2.5.m5.1.1.1.1.2" xref="S3.SS5.p2.5.m5.1.1.1.1.2.cmml">−</mo><msub id="S3.SS5.p2.5.m5.1.1.1.1.3" xref="S3.SS5.p2.5.m5.1.1.1.1.3.cmml"><mi id="S3.SS5.p2.5.m5.1.1.1.1.3.2" xref="S3.SS5.p2.5.m5.1.1.1.1.3.2.cmml">W</mi><mi id="S3.SS5.p2.5.m5.1.1.1.1.3.3" xref="S3.SS5.p2.5.m5.1.1.1.1.3.3.cmml">T</mi></msub></mrow><mo stretchy="false" id="S3.SS5.p2.5.m5.1.1.1.3" xref="S3.SS5.p2.5.m5.1.1.2.1.cmml">‖</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS5.p2.5.m5.1b"><apply id="S3.SS5.p2.5.m5.1.1.2.cmml" xref="S3.SS5.p2.5.m5.1.1.1"><csymbol cd="latexml" id="S3.SS5.p2.5.m5.1.1.2.1.cmml" xref="S3.SS5.p2.5.m5.1.1.1.2">norm</csymbol><apply id="S3.SS5.p2.5.m5.1.1.1.1.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1"><minus id="S3.SS5.p2.5.m5.1.1.1.1.2.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1.2"></minus><apply id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1"><plus id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.1.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.1"></plus><apply id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.2.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.2.1.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.2.2.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.2.2">𝑄</ci><ci id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.2.3.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.2.3">𝑇</ci></apply><apply id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3"><times id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.1.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.1"></times><apply id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.2.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.2.1.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.2">subscript</csymbol><ci id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.2.2.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.2.2">𝑃</ci><ci id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.2.3.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.2.3">𝑇</ci></apply><apply id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.3.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.3.1.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.3">subscript</csymbol><ci id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.3.2.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.3.2">𝐵</ci><ci id="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.3.3.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1.1.1.1.3.3.3">𝑇</ci></apply></apply></apply><apply id="S3.SS5.p2.5.m5.1.1.1.1.3.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS5.p2.5.m5.1.1.1.1.3.1.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1.3">subscript</csymbol><ci id="S3.SS5.p2.5.m5.1.1.1.1.3.2.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1.3.2">𝑊</ci><ci id="S3.SS5.p2.5.m5.1.1.1.1.3.3.cmml" xref="S3.SS5.p2.5.m5.1.1.1.1.3.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p2.5.m5.1c">\|(Q_{T}+P_{T}B_{T})-W_{T}\|</annotation></semantics></math>. To achieve this, we solve for <math id="S3.SS5.p2.6.m6.1" class="ltx_Math" alttext="B_{T}" display="inline"><semantics id="S3.SS5.p2.6.m6.1a"><msub id="S3.SS5.p2.6.m6.1.1" xref="S3.SS5.p2.6.m6.1.1.cmml"><mi id="S3.SS5.p2.6.m6.1.1.2" xref="S3.SS5.p2.6.m6.1.1.2.cmml">B</mi><mi id="S3.SS5.p2.6.m6.1.1.3" xref="S3.SS5.p2.6.m6.1.1.3.cmml">T</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS5.p2.6.m6.1b"><apply id="S3.SS5.p2.6.m6.1.1.cmml" xref="S3.SS5.p2.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS5.p2.6.m6.1.1.1.cmml" xref="S3.SS5.p2.6.m6.1.1">subscript</csymbol><ci id="S3.SS5.p2.6.m6.1.1.2.cmml" xref="S3.SS5.p2.6.m6.1.1.2">𝐵</ci><ci id="S3.SS5.p2.6.m6.1.1.3.cmml" xref="S3.SS5.p2.6.m6.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p2.6.m6.1c">B_{T}</annotation></semantics></math> in the merging step, initializing <math id="S3.SS5.p2.7.m7.1" class="ltx_Math" alttext="B_{T}" display="inline"><semantics id="S3.SS5.p2.7.m7.1a"><msub id="S3.SS5.p2.7.m7.1.1" xref="S3.SS5.p2.7.m7.1.1.cmml"><mi id="S3.SS5.p2.7.m7.1.1.2" xref="S3.SS5.p2.7.m7.1.1.2.cmml">B</mi><mi id="S3.SS5.p2.7.m7.1.1.3" xref="S3.SS5.p2.7.m7.1.1.3.cmml">T</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS5.p2.7.m7.1b"><apply id="S3.SS5.p2.7.m7.1.1.cmml" xref="S3.SS5.p2.7.m7.1.1"><csymbol cd="ambiguous" id="S3.SS5.p2.7.m7.1.1.1.cmml" xref="S3.SS5.p2.7.m7.1.1">subscript</csymbol><ci id="S3.SS5.p2.7.m7.1.1.2.cmml" xref="S3.SS5.p2.7.m7.1.1.2">𝐵</ci><ci id="S3.SS5.p2.7.m7.1.1.3.cmml" xref="S3.SS5.p2.7.m7.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p2.7.m7.1c">B_{T}</annotation></semantics></math> as <math id="S3.SS5.p2.8.m8.1" class="ltx_Math" alttext="B_{T}:=P_{T}^{+}(Q_{T}-W_{T})" display="inline"><semantics id="S3.SS5.p2.8.m8.1a"><mrow id="S3.SS5.p2.8.m8.1.1" xref="S3.SS5.p2.8.m8.1.1.cmml"><msub id="S3.SS5.p2.8.m8.1.1.3" xref="S3.SS5.p2.8.m8.1.1.3.cmml"><mi id="S3.SS5.p2.8.m8.1.1.3.2" xref="S3.SS5.p2.8.m8.1.1.3.2.cmml">B</mi><mi id="S3.SS5.p2.8.m8.1.1.3.3" xref="S3.SS5.p2.8.m8.1.1.3.3.cmml">T</mi></msub><mo lspace="0.278em" rspace="0.278em" id="S3.SS5.p2.8.m8.1.1.2" xref="S3.SS5.p2.8.m8.1.1.2.cmml">:=</mo><mrow id="S3.SS5.p2.8.m8.1.1.1" xref="S3.SS5.p2.8.m8.1.1.1.cmml"><msubsup id="S3.SS5.p2.8.m8.1.1.1.3" xref="S3.SS5.p2.8.m8.1.1.1.3.cmml"><mi id="S3.SS5.p2.8.m8.1.1.1.3.2.2" xref="S3.SS5.p2.8.m8.1.1.1.3.2.2.cmml">P</mi><mi id="S3.SS5.p2.8.m8.1.1.1.3.2.3" xref="S3.SS5.p2.8.m8.1.1.1.3.2.3.cmml">T</mi><mo id="S3.SS5.p2.8.m8.1.1.1.3.3" xref="S3.SS5.p2.8.m8.1.1.1.3.3.cmml">+</mo></msubsup><mo lspace="0em" rspace="0em" id="S3.SS5.p2.8.m8.1.1.1.2" xref="S3.SS5.p2.8.m8.1.1.1.2.cmml">​</mo><mrow id="S3.SS5.p2.8.m8.1.1.1.1.1" xref="S3.SS5.p2.8.m8.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS5.p2.8.m8.1.1.1.1.1.2" xref="S3.SS5.p2.8.m8.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS5.p2.8.m8.1.1.1.1.1.1" xref="S3.SS5.p2.8.m8.1.1.1.1.1.1.cmml"><msub id="S3.SS5.p2.8.m8.1.1.1.1.1.1.2" xref="S3.SS5.p2.8.m8.1.1.1.1.1.1.2.cmml"><mi id="S3.SS5.p2.8.m8.1.1.1.1.1.1.2.2" xref="S3.SS5.p2.8.m8.1.1.1.1.1.1.2.2.cmml">Q</mi><mi id="S3.SS5.p2.8.m8.1.1.1.1.1.1.2.3" xref="S3.SS5.p2.8.m8.1.1.1.1.1.1.2.3.cmml">T</mi></msub><mo id="S3.SS5.p2.8.m8.1.1.1.1.1.1.1" xref="S3.SS5.p2.8.m8.1.1.1.1.1.1.1.cmml">−</mo><msub id="S3.SS5.p2.8.m8.1.1.1.1.1.1.3" xref="S3.SS5.p2.8.m8.1.1.1.1.1.1.3.cmml"><mi id="S3.SS5.p2.8.m8.1.1.1.1.1.1.3.2" xref="S3.SS5.p2.8.m8.1.1.1.1.1.1.3.2.cmml">W</mi><mi id="S3.SS5.p2.8.m8.1.1.1.1.1.1.3.3" xref="S3.SS5.p2.8.m8.1.1.1.1.1.1.3.3.cmml">T</mi></msub></mrow><mo stretchy="false" id="S3.SS5.p2.8.m8.1.1.1.1.1.3" xref="S3.SS5.p2.8.m8.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS5.p2.8.m8.1b"><apply id="S3.SS5.p2.8.m8.1.1.cmml" xref="S3.SS5.p2.8.m8.1.1"><csymbol cd="latexml" id="S3.SS5.p2.8.m8.1.1.2.cmml" xref="S3.SS5.p2.8.m8.1.1.2">assign</csymbol><apply id="S3.SS5.p2.8.m8.1.1.3.cmml" xref="S3.SS5.p2.8.m8.1.1.3"><csymbol cd="ambiguous" id="S3.SS5.p2.8.m8.1.1.3.1.cmml" xref="S3.SS5.p2.8.m8.1.1.3">subscript</csymbol><ci id="S3.SS5.p2.8.m8.1.1.3.2.cmml" xref="S3.SS5.p2.8.m8.1.1.3.2">𝐵</ci><ci id="S3.SS5.p2.8.m8.1.1.3.3.cmml" xref="S3.SS5.p2.8.m8.1.1.3.3">𝑇</ci></apply><apply id="S3.SS5.p2.8.m8.1.1.1.cmml" xref="S3.SS5.p2.8.m8.1.1.1"><times id="S3.SS5.p2.8.m8.1.1.1.2.cmml" xref="S3.SS5.p2.8.m8.1.1.1.2"></times><apply id="S3.SS5.p2.8.m8.1.1.1.3.cmml" xref="S3.SS5.p2.8.m8.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS5.p2.8.m8.1.1.1.3.1.cmml" xref="S3.SS5.p2.8.m8.1.1.1.3">superscript</csymbol><apply id="S3.SS5.p2.8.m8.1.1.1.3.2.cmml" xref="S3.SS5.p2.8.m8.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS5.p2.8.m8.1.1.1.3.2.1.cmml" xref="S3.SS5.p2.8.m8.1.1.1.3">subscript</csymbol><ci id="S3.SS5.p2.8.m8.1.1.1.3.2.2.cmml" xref="S3.SS5.p2.8.m8.1.1.1.3.2.2">𝑃</ci><ci id="S3.SS5.p2.8.m8.1.1.1.3.2.3.cmml" xref="S3.SS5.p2.8.m8.1.1.1.3.2.3">𝑇</ci></apply><plus id="S3.SS5.p2.8.m8.1.1.1.3.3.cmml" xref="S3.SS5.p2.8.m8.1.1.1.3.3"></plus></apply><apply id="S3.SS5.p2.8.m8.1.1.1.1.1.1.cmml" xref="S3.SS5.p2.8.m8.1.1.1.1.1"><minus id="S3.SS5.p2.8.m8.1.1.1.1.1.1.1.cmml" xref="S3.SS5.p2.8.m8.1.1.1.1.1.1.1"></minus><apply id="S3.SS5.p2.8.m8.1.1.1.1.1.1.2.cmml" xref="S3.SS5.p2.8.m8.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS5.p2.8.m8.1.1.1.1.1.1.2.1.cmml" xref="S3.SS5.p2.8.m8.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.SS5.p2.8.m8.1.1.1.1.1.1.2.2.cmml" xref="S3.SS5.p2.8.m8.1.1.1.1.1.1.2.2">𝑄</ci><ci id="S3.SS5.p2.8.m8.1.1.1.1.1.1.2.3.cmml" xref="S3.SS5.p2.8.m8.1.1.1.1.1.1.2.3">𝑇</ci></apply><apply id="S3.SS5.p2.8.m8.1.1.1.1.1.1.3.cmml" xref="S3.SS5.p2.8.m8.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS5.p2.8.m8.1.1.1.1.1.1.3.1.cmml" xref="S3.SS5.p2.8.m8.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.SS5.p2.8.m8.1.1.1.1.1.1.3.2.cmml" xref="S3.SS5.p2.8.m8.1.1.1.1.1.1.3.2">𝑊</ci><ci id="S3.SS5.p2.8.m8.1.1.1.1.1.1.3.3.cmml" xref="S3.SS5.p2.8.m8.1.1.1.1.1.1.3.3">𝑇</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p2.8.m8.1c">B_{T}:=P_{T}^{+}(Q_{T}-W_{T})</annotation></semantics></math>, where <math id="S3.SS5.p2.9.m9.1" class="ltx_Math" alttext="P_{T}^{+}" display="inline"><semantics id="S3.SS5.p2.9.m9.1a"><msubsup id="S3.SS5.p2.9.m9.1.1" xref="S3.SS5.p2.9.m9.1.1.cmml"><mi id="S3.SS5.p2.9.m9.1.1.2.2" xref="S3.SS5.p2.9.m9.1.1.2.2.cmml">P</mi><mi id="S3.SS5.p2.9.m9.1.1.2.3" xref="S3.SS5.p2.9.m9.1.1.2.3.cmml">T</mi><mo id="S3.SS5.p2.9.m9.1.1.3" xref="S3.SS5.p2.9.m9.1.1.3.cmml">+</mo></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS5.p2.9.m9.1b"><apply id="S3.SS5.p2.9.m9.1.1.cmml" xref="S3.SS5.p2.9.m9.1.1"><csymbol cd="ambiguous" id="S3.SS5.p2.9.m9.1.1.1.cmml" xref="S3.SS5.p2.9.m9.1.1">superscript</csymbol><apply id="S3.SS5.p2.9.m9.1.1.2.cmml" xref="S3.SS5.p2.9.m9.1.1"><csymbol cd="ambiguous" id="S3.SS5.p2.9.m9.1.1.2.1.cmml" xref="S3.SS5.p2.9.m9.1.1">subscript</csymbol><ci id="S3.SS5.p2.9.m9.1.1.2.2.cmml" xref="S3.SS5.p2.9.m9.1.1.2.2">𝑃</ci><ci id="S3.SS5.p2.9.m9.1.1.2.3.cmml" xref="S3.SS5.p2.9.m9.1.1.2.3">𝑇</ci></apply><plus id="S3.SS5.p2.9.m9.1.1.3.cmml" xref="S3.SS5.p2.9.m9.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p2.9.m9.1c">P_{T}^{+}</annotation></semantics></math> is the Moore-Penrose pseudo-inverse. This approach avoids initializing <math id="S3.SS5.p2.10.m10.1" class="ltx_Math" alttext="B_{T}" display="inline"><semantics id="S3.SS5.p2.10.m10.1a"><msub id="S3.SS5.p2.10.m10.1.1" xref="S3.SS5.p2.10.m10.1.1.cmml"><mi id="S3.SS5.p2.10.m10.1.1.2" xref="S3.SS5.p2.10.m10.1.1.2.cmml">B</mi><mi id="S3.SS5.p2.10.m10.1.1.3" xref="S3.SS5.p2.10.m10.1.1.3.cmml">T</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS5.p2.10.m10.1b"><apply id="S3.SS5.p2.10.m10.1.1.cmml" xref="S3.SS5.p2.10.m10.1.1"><csymbol cd="ambiguous" id="S3.SS5.p2.10.m10.1.1.1.cmml" xref="S3.SS5.p2.10.m10.1.1">subscript</csymbol><ci id="S3.SS5.p2.10.m10.1.1.2.cmml" xref="S3.SS5.p2.10.m10.1.1.2">𝐵</ci><ci id="S3.SS5.p2.10.m10.1.1.3.cmml" xref="S3.SS5.p2.10.m10.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p2.10.m10.1c">B_{T}</annotation></semantics></math> as zeros and instead uses it for minimizing the quantization error <math id="S3.SS5.p2.11.m11.1" class="ltx_Math" alttext="\|Q_{T}-W_{T}\|" display="inline"><semantics id="S3.SS5.p2.11.m11.1a"><mrow id="S3.SS5.p2.11.m11.1.1.1" xref="S3.SS5.p2.11.m11.1.1.2.cmml"><mo stretchy="false" id="S3.SS5.p2.11.m11.1.1.1.2" xref="S3.SS5.p2.11.m11.1.1.2.1.cmml">‖</mo><mrow id="S3.SS5.p2.11.m11.1.1.1.1" xref="S3.SS5.p2.11.m11.1.1.1.1.cmml"><msub id="S3.SS5.p2.11.m11.1.1.1.1.2" xref="S3.SS5.p2.11.m11.1.1.1.1.2.cmml"><mi id="S3.SS5.p2.11.m11.1.1.1.1.2.2" xref="S3.SS5.p2.11.m11.1.1.1.1.2.2.cmml">Q</mi><mi id="S3.SS5.p2.11.m11.1.1.1.1.2.3" xref="S3.SS5.p2.11.m11.1.1.1.1.2.3.cmml">T</mi></msub><mo id="S3.SS5.p2.11.m11.1.1.1.1.1" xref="S3.SS5.p2.11.m11.1.1.1.1.1.cmml">−</mo><msub id="S3.SS5.p2.11.m11.1.1.1.1.3" xref="S3.SS5.p2.11.m11.1.1.1.1.3.cmml"><mi id="S3.SS5.p2.11.m11.1.1.1.1.3.2" xref="S3.SS5.p2.11.m11.1.1.1.1.3.2.cmml">W</mi><mi id="S3.SS5.p2.11.m11.1.1.1.1.3.3" xref="S3.SS5.p2.11.m11.1.1.1.1.3.3.cmml">T</mi></msub></mrow><mo stretchy="false" id="S3.SS5.p2.11.m11.1.1.1.3" xref="S3.SS5.p2.11.m11.1.1.2.1.cmml">‖</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS5.p2.11.m11.1b"><apply id="S3.SS5.p2.11.m11.1.1.2.cmml" xref="S3.SS5.p2.11.m11.1.1.1"><csymbol cd="latexml" id="S3.SS5.p2.11.m11.1.1.2.1.cmml" xref="S3.SS5.p2.11.m11.1.1.1.2">norm</csymbol><apply id="S3.SS5.p2.11.m11.1.1.1.1.cmml" xref="S3.SS5.p2.11.m11.1.1.1.1"><minus id="S3.SS5.p2.11.m11.1.1.1.1.1.cmml" xref="S3.SS5.p2.11.m11.1.1.1.1.1"></minus><apply id="S3.SS5.p2.11.m11.1.1.1.1.2.cmml" xref="S3.SS5.p2.11.m11.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS5.p2.11.m11.1.1.1.1.2.1.cmml" xref="S3.SS5.p2.11.m11.1.1.1.1.2">subscript</csymbol><ci id="S3.SS5.p2.11.m11.1.1.1.1.2.2.cmml" xref="S3.SS5.p2.11.m11.1.1.1.1.2.2">𝑄</ci><ci id="S3.SS5.p2.11.m11.1.1.1.1.2.3.cmml" xref="S3.SS5.p2.11.m11.1.1.1.1.2.3">𝑇</ci></apply><apply id="S3.SS5.p2.11.m11.1.1.1.1.3.cmml" xref="S3.SS5.p2.11.m11.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS5.p2.11.m11.1.1.1.1.3.1.cmml" xref="S3.SS5.p2.11.m11.1.1.1.1.3">subscript</csymbol><ci id="S3.SS5.p2.11.m11.1.1.1.1.3.2.cmml" xref="S3.SS5.p2.11.m11.1.1.1.1.3.2">𝑊</ci><ci id="S3.SS5.p2.11.m11.1.1.1.1.3.3.cmml" xref="S3.SS5.p2.11.m11.1.1.1.1.3.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p2.11.m11.1c">\|Q_{T}-W_{T}\|</annotation></semantics></math>. We then iteratively refine <math id="S3.SS5.p2.12.m12.1" class="ltx_Math" alttext="B_{T}" display="inline"><semantics id="S3.SS5.p2.12.m12.1a"><msub id="S3.SS5.p2.12.m12.1.1" xref="S3.SS5.p2.12.m12.1.1.cmml"><mi id="S3.SS5.p2.12.m12.1.1.2" xref="S3.SS5.p2.12.m12.1.1.2.cmml">B</mi><mi id="S3.SS5.p2.12.m12.1.1.3" xref="S3.SS5.p2.12.m12.1.1.3.cmml">T</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS5.p2.12.m12.1b"><apply id="S3.SS5.p2.12.m12.1.1.cmml" xref="S3.SS5.p2.12.m12.1.1"><csymbol cd="ambiguous" id="S3.SS5.p2.12.m12.1.1.1.cmml" xref="S3.SS5.p2.12.m12.1.1">subscript</csymbol><ci id="S3.SS5.p2.12.m12.1.1.2.cmml" xref="S3.SS5.p2.12.m12.1.1.2">𝐵</ci><ci id="S3.SS5.p2.12.m12.1.1.3.cmml" xref="S3.SS5.p2.12.m12.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p2.12.m12.1c">B_{T}</annotation></semantics></math>, recomputing <math id="S3.SS5.p2.13.m13.1" class="ltx_Math" alttext="Q_{T}=q_{\text{NF4}}(W_{T}-P_{T}B_{T})" display="inline"><semantics id="S3.SS5.p2.13.m13.1a"><mrow id="S3.SS5.p2.13.m13.1.1" xref="S3.SS5.p2.13.m13.1.1.cmml"><msub id="S3.SS5.p2.13.m13.1.1.3" xref="S3.SS5.p2.13.m13.1.1.3.cmml"><mi id="S3.SS5.p2.13.m13.1.1.3.2" xref="S3.SS5.p2.13.m13.1.1.3.2.cmml">Q</mi><mi id="S3.SS5.p2.13.m13.1.1.3.3" xref="S3.SS5.p2.13.m13.1.1.3.3.cmml">T</mi></msub><mo id="S3.SS5.p2.13.m13.1.1.2" xref="S3.SS5.p2.13.m13.1.1.2.cmml">=</mo><mrow id="S3.SS5.p2.13.m13.1.1.1" xref="S3.SS5.p2.13.m13.1.1.1.cmml"><msub id="S3.SS5.p2.13.m13.1.1.1.3" xref="S3.SS5.p2.13.m13.1.1.1.3.cmml"><mi id="S3.SS5.p2.13.m13.1.1.1.3.2" xref="S3.SS5.p2.13.m13.1.1.1.3.2.cmml">q</mi><mtext id="S3.SS5.p2.13.m13.1.1.1.3.3" xref="S3.SS5.p2.13.m13.1.1.1.3.3a.cmml">NF4</mtext></msub><mo lspace="0em" rspace="0em" id="S3.SS5.p2.13.m13.1.1.1.2" xref="S3.SS5.p2.13.m13.1.1.1.2.cmml">​</mo><mrow id="S3.SS5.p2.13.m13.1.1.1.1.1" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS5.p2.13.m13.1.1.1.1.1.2" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS5.p2.13.m13.1.1.1.1.1.1" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.cmml"><msub id="S3.SS5.p2.13.m13.1.1.1.1.1.1.2" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.2.cmml"><mi id="S3.SS5.p2.13.m13.1.1.1.1.1.1.2.2" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.2.2.cmml">W</mi><mi id="S3.SS5.p2.13.m13.1.1.1.1.1.1.2.3" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.2.3.cmml">T</mi></msub><mo id="S3.SS5.p2.13.m13.1.1.1.1.1.1.1" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.1.cmml">−</mo><mrow id="S3.SS5.p2.13.m13.1.1.1.1.1.1.3" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.cmml"><msub id="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.2" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.2.cmml"><mi id="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.2.2" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.2.2.cmml">P</mi><mi id="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.2.3" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.2.3.cmml">T</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.1" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.3" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.3.cmml"><mi id="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.3.2" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.3.2.cmml">B</mi><mi id="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.3.3" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.3.3.cmml">T</mi></msub></mrow></mrow><mo stretchy="false" id="S3.SS5.p2.13.m13.1.1.1.1.1.3" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS5.p2.13.m13.1b"><apply id="S3.SS5.p2.13.m13.1.1.cmml" xref="S3.SS5.p2.13.m13.1.1"><eq id="S3.SS5.p2.13.m13.1.1.2.cmml" xref="S3.SS5.p2.13.m13.1.1.2"></eq><apply id="S3.SS5.p2.13.m13.1.1.3.cmml" xref="S3.SS5.p2.13.m13.1.1.3"><csymbol cd="ambiguous" id="S3.SS5.p2.13.m13.1.1.3.1.cmml" xref="S3.SS5.p2.13.m13.1.1.3">subscript</csymbol><ci id="S3.SS5.p2.13.m13.1.1.3.2.cmml" xref="S3.SS5.p2.13.m13.1.1.3.2">𝑄</ci><ci id="S3.SS5.p2.13.m13.1.1.3.3.cmml" xref="S3.SS5.p2.13.m13.1.1.3.3">𝑇</ci></apply><apply id="S3.SS5.p2.13.m13.1.1.1.cmml" xref="S3.SS5.p2.13.m13.1.1.1"><times id="S3.SS5.p2.13.m13.1.1.1.2.cmml" xref="S3.SS5.p2.13.m13.1.1.1.2"></times><apply id="S3.SS5.p2.13.m13.1.1.1.3.cmml" xref="S3.SS5.p2.13.m13.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS5.p2.13.m13.1.1.1.3.1.cmml" xref="S3.SS5.p2.13.m13.1.1.1.3">subscript</csymbol><ci id="S3.SS5.p2.13.m13.1.1.1.3.2.cmml" xref="S3.SS5.p2.13.m13.1.1.1.3.2">𝑞</ci><ci id="S3.SS5.p2.13.m13.1.1.1.3.3a.cmml" xref="S3.SS5.p2.13.m13.1.1.1.3.3"><mtext mathsize="70%" id="S3.SS5.p2.13.m13.1.1.1.3.3.cmml" xref="S3.SS5.p2.13.m13.1.1.1.3.3">NF4</mtext></ci></apply><apply id="S3.SS5.p2.13.m13.1.1.1.1.1.1.cmml" xref="S3.SS5.p2.13.m13.1.1.1.1.1"><minus id="S3.SS5.p2.13.m13.1.1.1.1.1.1.1.cmml" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.1"></minus><apply id="S3.SS5.p2.13.m13.1.1.1.1.1.1.2.cmml" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS5.p2.13.m13.1.1.1.1.1.1.2.1.cmml" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.SS5.p2.13.m13.1.1.1.1.1.1.2.2.cmml" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.2.2">𝑊</ci><ci id="S3.SS5.p2.13.m13.1.1.1.1.1.1.2.3.cmml" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.2.3">𝑇</ci></apply><apply id="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.cmml" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.3"><times id="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.1.cmml" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.1"></times><apply id="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.2.cmml" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.2.1.cmml" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.2">subscript</csymbol><ci id="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.2.2.cmml" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.2.2">𝑃</ci><ci id="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.2.3.cmml" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.2.3">𝑇</ci></apply><apply id="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.3.cmml" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.3.1.cmml" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.3">subscript</csymbol><ci id="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.3.2.cmml" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.3.2">𝐵</ci><ci id="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.3.3.cmml" xref="S3.SS5.p2.13.m13.1.1.1.1.1.1.3.3.3">𝑇</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p2.13.m13.1c">Q_{T}=q_{\text{NF4}}(W_{T}-P_{T}B_{T})</annotation></semantics></math>, improving the alignment between the full-precision <math id="S3.SS5.p2.14.m14.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S3.SS5.p2.14.m14.1a"><mi id="S3.SS5.p2.14.m14.1.1" xref="S3.SS5.p2.14.m14.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.SS5.p2.14.m14.1b"><ci id="S3.SS5.p2.14.m14.1.1.cmml" xref="S3.SS5.p2.14.m14.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p2.14.m14.1c">W</annotation></semantics></math> and its quantized state.</p>
</div>
<div id="S3.SS5.p3" class="ltx_para">
<p id="S3.SS5.p3.10" class="ltx_p">As training advances and the learning rate decays, the magnitude of the update <math id="S3.SS5.p3.1.m1.1" class="ltx_Math" alttext="B_{T-1}" display="inline"><semantics id="S3.SS5.p3.1.m1.1a"><msub id="S3.SS5.p3.1.m1.1.1" xref="S3.SS5.p3.1.m1.1.1.cmml"><mi id="S3.SS5.p3.1.m1.1.1.2" xref="S3.SS5.p3.1.m1.1.1.2.cmml">B</mi><mrow id="S3.SS5.p3.1.m1.1.1.3" xref="S3.SS5.p3.1.m1.1.1.3.cmml"><mi id="S3.SS5.p3.1.m1.1.1.3.2" xref="S3.SS5.p3.1.m1.1.1.3.2.cmml">T</mi><mo id="S3.SS5.p3.1.m1.1.1.3.1" xref="S3.SS5.p3.1.m1.1.1.3.1.cmml">−</mo><mn id="S3.SS5.p3.1.m1.1.1.3.3" xref="S3.SS5.p3.1.m1.1.1.3.3.cmml">1</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS5.p3.1.m1.1b"><apply id="S3.SS5.p3.1.m1.1.1.cmml" xref="S3.SS5.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS5.p3.1.m1.1.1.1.cmml" xref="S3.SS5.p3.1.m1.1.1">subscript</csymbol><ci id="S3.SS5.p3.1.m1.1.1.2.cmml" xref="S3.SS5.p3.1.m1.1.1.2">𝐵</ci><apply id="S3.SS5.p3.1.m1.1.1.3.cmml" xref="S3.SS5.p3.1.m1.1.1.3"><minus id="S3.SS5.p3.1.m1.1.1.3.1.cmml" xref="S3.SS5.p3.1.m1.1.1.3.1"></minus><ci id="S3.SS5.p3.1.m1.1.1.3.2.cmml" xref="S3.SS5.p3.1.m1.1.1.3.2">𝑇</ci><cn type="integer" id="S3.SS5.p3.1.m1.1.1.3.3.cmml" xref="S3.SS5.p3.1.m1.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p3.1.m1.1c">B_{T-1}</annotation></semantics></math> to form <math id="S3.SS5.p3.2.m2.1" class="ltx_Math" alttext="W_{T}" display="inline"><semantics id="S3.SS5.p3.2.m2.1a"><msub id="S3.SS5.p3.2.m2.1.1" xref="S3.SS5.p3.2.m2.1.1.cmml"><mi id="S3.SS5.p3.2.m2.1.1.2" xref="S3.SS5.p3.2.m2.1.1.2.cmml">W</mi><mi id="S3.SS5.p3.2.m2.1.1.3" xref="S3.SS5.p3.2.m2.1.1.3.cmml">T</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS5.p3.2.m2.1b"><apply id="S3.SS5.p3.2.m2.1.1.cmml" xref="S3.SS5.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS5.p3.2.m2.1.1.1.cmml" xref="S3.SS5.p3.2.m2.1.1">subscript</csymbol><ci id="S3.SS5.p3.2.m2.1.1.2.cmml" xref="S3.SS5.p3.2.m2.1.1.2">𝑊</ci><ci id="S3.SS5.p3.2.m2.1.1.3.cmml" xref="S3.SS5.p3.2.m2.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p3.2.m2.1c">W_{T}</annotation></semantics></math> decreases. This leads to negligible differences between <math id="S3.SS5.p3.3.m3.1" class="ltx_Math" alttext="\left|q\left(Q_{t}+P_{t}B_{t}\right)-Q_{t}\right|" display="inline"><semantics id="S3.SS5.p3.3.m3.1a"><mrow id="S3.SS5.p3.3.m3.1.1.1" xref="S3.SS5.p3.3.m3.1.1.2.cmml"><mo id="S3.SS5.p3.3.m3.1.1.1.2" xref="S3.SS5.p3.3.m3.1.1.2.1.cmml">|</mo><mrow id="S3.SS5.p3.3.m3.1.1.1.1" xref="S3.SS5.p3.3.m3.1.1.1.1.cmml"><mrow id="S3.SS5.p3.3.m3.1.1.1.1.1" xref="S3.SS5.p3.3.m3.1.1.1.1.1.cmml"><mi id="S3.SS5.p3.3.m3.1.1.1.1.1.3" xref="S3.SS5.p3.3.m3.1.1.1.1.1.3.cmml">q</mi><mo lspace="0em" rspace="0em" id="S3.SS5.p3.3.m3.1.1.1.1.1.2" xref="S3.SS5.p3.3.m3.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.cmml"><mo id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.2" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.cmml"><msub id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.2" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.2.cmml"><mi id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.2.2" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.2.2.cmml">Q</mi><mi id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.2.3" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.2.3.cmml">t</mi></msub><mo id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.1" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.cmml"><msub id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.2" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.2.cmml"><mi id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.2.2" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.2.2.cmml">P</mi><mi id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.2.3" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.2.3.cmml">t</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.1" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.3" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.3.cmml"><mi id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.3.2" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.3.2.cmml">B</mi><mi id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.3.3" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.3.3.cmml">t</mi></msub></mrow></mrow><mo id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.3" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS5.p3.3.m3.1.1.1.1.2" xref="S3.SS5.p3.3.m3.1.1.1.1.2.cmml">−</mo><msub id="S3.SS5.p3.3.m3.1.1.1.1.3" xref="S3.SS5.p3.3.m3.1.1.1.1.3.cmml"><mi id="S3.SS5.p3.3.m3.1.1.1.1.3.2" xref="S3.SS5.p3.3.m3.1.1.1.1.3.2.cmml">Q</mi><mi id="S3.SS5.p3.3.m3.1.1.1.1.3.3" xref="S3.SS5.p3.3.m3.1.1.1.1.3.3.cmml">t</mi></msub></mrow><mo id="S3.SS5.p3.3.m3.1.1.1.3" xref="S3.SS5.p3.3.m3.1.1.2.1.cmml">|</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS5.p3.3.m3.1b"><apply id="S3.SS5.p3.3.m3.1.1.2.cmml" xref="S3.SS5.p3.3.m3.1.1.1"><abs id="S3.SS5.p3.3.m3.1.1.2.1.cmml" xref="S3.SS5.p3.3.m3.1.1.1.2"></abs><apply id="S3.SS5.p3.3.m3.1.1.1.1.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1"><minus id="S3.SS5.p3.3.m3.1.1.1.1.2.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.2"></minus><apply id="S3.SS5.p3.3.m3.1.1.1.1.1.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.1"><times id="S3.SS5.p3.3.m3.1.1.1.1.1.2.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.1.2"></times><ci id="S3.SS5.p3.3.m3.1.1.1.1.1.3.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.1.3">𝑞</ci><apply id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1"><plus id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.1"></plus><apply id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.2.2">𝑄</ci><ci id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.2.3">𝑡</ci></apply><apply id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3"><times id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.1"></times><apply id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.2.1.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.2">subscript</csymbol><ci id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.2.2.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.2.2">𝑃</ci><ci id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.2.3.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.2.3">𝑡</ci></apply><apply id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.3.1.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.3">subscript</csymbol><ci id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.3.2.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.3.2">𝐵</ci><ci id="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.3.3.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.1.1.1.1.3.3.3">𝑡</ci></apply></apply></apply></apply><apply id="S3.SS5.p3.3.m3.1.1.1.1.3.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS5.p3.3.m3.1.1.1.1.3.1.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.3">subscript</csymbol><ci id="S3.SS5.p3.3.m3.1.1.1.1.3.2.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.3.2">𝑄</ci><ci id="S3.SS5.p3.3.m3.1.1.1.1.3.3.cmml" xref="S3.SS5.p3.3.m3.1.1.1.1.3.3">𝑡</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p3.3.m3.1c">\left|q\left(Q_{t}+P_{t}B_{t}\right)-Q_{t}\right|</annotation></semantics></math>, which results in the weights plateauing early, as depicted in Fig. <a href="#S5.F4.sf1" title="In Figure 4 ‣ 5 Ablations ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4(a)</span></a>.
To address this, we implement an exponentially increasing scheduler for updating <math id="S3.SS5.p3.4.m4.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S3.SS5.p3.4.m4.1a"><mi id="S3.SS5.p3.4.m4.1.1" xref="S3.SS5.p3.4.m4.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.SS5.p3.4.m4.1b"><ci id="S3.SS5.p3.4.m4.1.1.cmml" xref="S3.SS5.p3.4.m4.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p3.4.m4.1c">W</annotation></semantics></math>. Drawing from GaLore’s observation on the exponential decay of gradient rank (Lemma 3.1 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite>), we start with a frequency gap <math id="S3.SS5.p3.5.m5.1" class="ltx_Math" alttext="\tau" display="inline"><semantics id="S3.SS5.p3.5.m5.1a"><mi id="S3.SS5.p3.5.m5.1.1" xref="S3.SS5.p3.5.m5.1.1.cmml">τ</mi><annotation-xml encoding="MathML-Content" id="S3.SS5.p3.5.m5.1b"><ci id="S3.SS5.p3.5.m5.1.1.cmml" xref="S3.SS5.p3.5.m5.1.1">𝜏</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p3.5.m5.1c">\tau</annotation></semantics></math> and progressively increase the update intervals by a factor of <math id="S3.SS5.p3.6.m6.1" class="ltx_Math" alttext="\psi" display="inline"><semantics id="S3.SS5.p3.6.m6.1a"><mi id="S3.SS5.p3.6.m6.1.1" xref="S3.SS5.p3.6.m6.1.1.cmml">ψ</mi><annotation-xml encoding="MathML-Content" id="S3.SS5.p3.6.m6.1b"><ci id="S3.SS5.p3.6.m6.1.1.cmml" xref="S3.SS5.p3.6.m6.1.1">𝜓</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p3.6.m6.1c">\psi</annotation></semantics></math>. The sequence of updates is then given by <math id="S3.SS5.p3.7.m7.2" class="ltx_Math" alttext="(T_{i})_{i=0}^{\infty}=(\tau+\psi^{i})_{i=0}^{\infty}" display="inline"><semantics id="S3.SS5.p3.7.m7.2a"><mrow id="S3.SS5.p3.7.m7.2.2" xref="S3.SS5.p3.7.m7.2.2.cmml"><msubsup id="S3.SS5.p3.7.m7.1.1.1" xref="S3.SS5.p3.7.m7.1.1.1.cmml"><mrow id="S3.SS5.p3.7.m7.1.1.1.1.1.1" xref="S3.SS5.p3.7.m7.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS5.p3.7.m7.1.1.1.1.1.1.2" xref="S3.SS5.p3.7.m7.1.1.1.1.1.1.1.cmml">(</mo><msub id="S3.SS5.p3.7.m7.1.1.1.1.1.1.1" xref="S3.SS5.p3.7.m7.1.1.1.1.1.1.1.cmml"><mi id="S3.SS5.p3.7.m7.1.1.1.1.1.1.1.2" xref="S3.SS5.p3.7.m7.1.1.1.1.1.1.1.2.cmml">T</mi><mi id="S3.SS5.p3.7.m7.1.1.1.1.1.1.1.3" xref="S3.SS5.p3.7.m7.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S3.SS5.p3.7.m7.1.1.1.1.1.1.3" xref="S3.SS5.p3.7.m7.1.1.1.1.1.1.1.cmml">)</mo></mrow><mrow id="S3.SS5.p3.7.m7.1.1.1.1.3" xref="S3.SS5.p3.7.m7.1.1.1.1.3.cmml"><mi id="S3.SS5.p3.7.m7.1.1.1.1.3.2" xref="S3.SS5.p3.7.m7.1.1.1.1.3.2.cmml">i</mi><mo id="S3.SS5.p3.7.m7.1.1.1.1.3.1" xref="S3.SS5.p3.7.m7.1.1.1.1.3.1.cmml">=</mo><mn id="S3.SS5.p3.7.m7.1.1.1.1.3.3" xref="S3.SS5.p3.7.m7.1.1.1.1.3.3.cmml">0</mn></mrow><mi mathvariant="normal" id="S3.SS5.p3.7.m7.1.1.1.3" xref="S3.SS5.p3.7.m7.1.1.1.3.cmml">∞</mi></msubsup><mo id="S3.SS5.p3.7.m7.2.2.3" xref="S3.SS5.p3.7.m7.2.2.3.cmml">=</mo><msubsup id="S3.SS5.p3.7.m7.2.2.2" xref="S3.SS5.p3.7.m7.2.2.2.cmml"><mrow id="S3.SS5.p3.7.m7.2.2.2.1.1.1" xref="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS5.p3.7.m7.2.2.2.1.1.1.2" xref="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.cmml">(</mo><mrow id="S3.SS5.p3.7.m7.2.2.2.1.1.1.1" xref="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.cmml"><mi id="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.2" xref="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.2.cmml">τ</mi><mo id="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.1" xref="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.1.cmml">+</mo><msup id="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.3" xref="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.3.cmml"><mi id="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.3.2" xref="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.3.2.cmml">ψ</mi><mi id="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.3.3" xref="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.3.3.cmml">i</mi></msup></mrow><mo stretchy="false" id="S3.SS5.p3.7.m7.2.2.2.1.1.1.3" xref="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.cmml">)</mo></mrow><mrow id="S3.SS5.p3.7.m7.2.2.2.1.3" xref="S3.SS5.p3.7.m7.2.2.2.1.3.cmml"><mi id="S3.SS5.p3.7.m7.2.2.2.1.3.2" xref="S3.SS5.p3.7.m7.2.2.2.1.3.2.cmml">i</mi><mo id="S3.SS5.p3.7.m7.2.2.2.1.3.1" xref="S3.SS5.p3.7.m7.2.2.2.1.3.1.cmml">=</mo><mn id="S3.SS5.p3.7.m7.2.2.2.1.3.3" xref="S3.SS5.p3.7.m7.2.2.2.1.3.3.cmml">0</mn></mrow><mi mathvariant="normal" id="S3.SS5.p3.7.m7.2.2.2.3" xref="S3.SS5.p3.7.m7.2.2.2.3.cmml">∞</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS5.p3.7.m7.2b"><apply id="S3.SS5.p3.7.m7.2.2.cmml" xref="S3.SS5.p3.7.m7.2.2"><eq id="S3.SS5.p3.7.m7.2.2.3.cmml" xref="S3.SS5.p3.7.m7.2.2.3"></eq><apply id="S3.SS5.p3.7.m7.1.1.1.cmml" xref="S3.SS5.p3.7.m7.1.1.1"><csymbol cd="ambiguous" id="S3.SS5.p3.7.m7.1.1.1.2.cmml" xref="S3.SS5.p3.7.m7.1.1.1">superscript</csymbol><apply id="S3.SS5.p3.7.m7.1.1.1.1.cmml" xref="S3.SS5.p3.7.m7.1.1.1"><csymbol cd="ambiguous" id="S3.SS5.p3.7.m7.1.1.1.1.2.cmml" xref="S3.SS5.p3.7.m7.1.1.1">subscript</csymbol><apply id="S3.SS5.p3.7.m7.1.1.1.1.1.1.1.cmml" xref="S3.SS5.p3.7.m7.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS5.p3.7.m7.1.1.1.1.1.1.1.1.cmml" xref="S3.SS5.p3.7.m7.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS5.p3.7.m7.1.1.1.1.1.1.1.2.cmml" xref="S3.SS5.p3.7.m7.1.1.1.1.1.1.1.2">𝑇</ci><ci id="S3.SS5.p3.7.m7.1.1.1.1.1.1.1.3.cmml" xref="S3.SS5.p3.7.m7.1.1.1.1.1.1.1.3">𝑖</ci></apply><apply id="S3.SS5.p3.7.m7.1.1.1.1.3.cmml" xref="S3.SS5.p3.7.m7.1.1.1.1.3"><eq id="S3.SS5.p3.7.m7.1.1.1.1.3.1.cmml" xref="S3.SS5.p3.7.m7.1.1.1.1.3.1"></eq><ci id="S3.SS5.p3.7.m7.1.1.1.1.3.2.cmml" xref="S3.SS5.p3.7.m7.1.1.1.1.3.2">𝑖</ci><cn type="integer" id="S3.SS5.p3.7.m7.1.1.1.1.3.3.cmml" xref="S3.SS5.p3.7.m7.1.1.1.1.3.3">0</cn></apply></apply><infinity id="S3.SS5.p3.7.m7.1.1.1.3.cmml" xref="S3.SS5.p3.7.m7.1.1.1.3"></infinity></apply><apply id="S3.SS5.p3.7.m7.2.2.2.cmml" xref="S3.SS5.p3.7.m7.2.2.2"><csymbol cd="ambiguous" id="S3.SS5.p3.7.m7.2.2.2.2.cmml" xref="S3.SS5.p3.7.m7.2.2.2">superscript</csymbol><apply id="S3.SS5.p3.7.m7.2.2.2.1.cmml" xref="S3.SS5.p3.7.m7.2.2.2"><csymbol cd="ambiguous" id="S3.SS5.p3.7.m7.2.2.2.1.2.cmml" xref="S3.SS5.p3.7.m7.2.2.2">subscript</csymbol><apply id="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.cmml" xref="S3.SS5.p3.7.m7.2.2.2.1.1.1"><plus id="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.1.cmml" xref="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.1"></plus><ci id="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.2.cmml" xref="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.2">𝜏</ci><apply id="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.3.cmml" xref="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.3.1.cmml" xref="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.3">superscript</csymbol><ci id="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.3.2.cmml" xref="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.3.2">𝜓</ci><ci id="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.3.3.cmml" xref="S3.SS5.p3.7.m7.2.2.2.1.1.1.1.3.3">𝑖</ci></apply></apply><apply id="S3.SS5.p3.7.m7.2.2.2.1.3.cmml" xref="S3.SS5.p3.7.m7.2.2.2.1.3"><eq id="S3.SS5.p3.7.m7.2.2.2.1.3.1.cmml" xref="S3.SS5.p3.7.m7.2.2.2.1.3.1"></eq><ci id="S3.SS5.p3.7.m7.2.2.2.1.3.2.cmml" xref="S3.SS5.p3.7.m7.2.2.2.1.3.2">𝑖</ci><cn type="integer" id="S3.SS5.p3.7.m7.2.2.2.1.3.3.cmml" xref="S3.SS5.p3.7.m7.2.2.2.1.3.3">0</cn></apply></apply><infinity id="S3.SS5.p3.7.m7.2.2.2.3.cmml" xref="S3.SS5.p3.7.m7.2.2.2.3"></infinity></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p3.7.m7.2c">(T_{i})_{i=0}^{\infty}=(\tau+\psi^{i})_{i=0}^{\infty}</annotation></semantics></math> Each <math id="S3.SS5.p3.8.m8.1" class="ltx_Math" alttext="T_{i}" display="inline"><semantics id="S3.SS5.p3.8.m8.1a"><msub id="S3.SS5.p3.8.m8.1.1" xref="S3.SS5.p3.8.m8.1.1.cmml"><mi id="S3.SS5.p3.8.m8.1.1.2" xref="S3.SS5.p3.8.m8.1.1.2.cmml">T</mi><mi id="S3.SS5.p3.8.m8.1.1.3" xref="S3.SS5.p3.8.m8.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS5.p3.8.m8.1b"><apply id="S3.SS5.p3.8.m8.1.1.cmml" xref="S3.SS5.p3.8.m8.1.1"><csymbol cd="ambiguous" id="S3.SS5.p3.8.m8.1.1.1.cmml" xref="S3.SS5.p3.8.m8.1.1">subscript</csymbol><ci id="S3.SS5.p3.8.m8.1.1.2.cmml" xref="S3.SS5.p3.8.m8.1.1.2">𝑇</ci><ci id="S3.SS5.p3.8.m8.1.1.3.cmml" xref="S3.SS5.p3.8.m8.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p3.8.m8.1c">T_{i}</annotation></semantics></math> marks a training step <math id="S3.SS5.p3.9.m9.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S3.SS5.p3.9.m9.1a"><mi id="S3.SS5.p3.9.m9.1.1" xref="S3.SS5.p3.9.m9.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S3.SS5.p3.9.m9.1b"><ci id="S3.SS5.p3.9.m9.1.1.cmml" xref="S3.SS5.p3.9.m9.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p3.9.m9.1c">t</annotation></semantics></math> when <math id="S3.SS5.p3.10.m10.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S3.SS5.p3.10.m10.1a"><mi id="S3.SS5.p3.10.m10.1.1" xref="S3.SS5.p3.10.m10.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.SS5.p3.10.m10.1b"><ci id="S3.SS5.p3.10.m10.1.1.cmml" xref="S3.SS5.p3.10.m10.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p3.10.m10.1c">W</annotation></semantics></math> is updated. This scheduling ensures more frequent updates earlier in training and more well-spaced adjustments later, allowing more accumulated gradients before each update.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Experiments and Results</h2>

<figure id="S4.T1" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S4.T1.18.2.1" class="ltx_text" style="font-size:90%;">Table 1</span>: </span><span id="S4.T1.2.1" class="ltx_text" style="font-size:90%;">Comparison of low-rank pre-training methods for LLaMA2-style language models on the C4 dataset. The table shows validation perplexity, memory estimates, and quantization states for <span id="S4.T1.2.1.1" class="ltx_text ltx_font_smallcaps">LoQT</span>. The rank ratio <math id="S4.T1.2.1.m1.1" class="ltx_Math" alttext="r/d_{model}" display="inline"><semantics id="S4.T1.2.1.m1.1b"><mrow id="S4.T1.2.1.m1.1.1" xref="S4.T1.2.1.m1.1.1.cmml"><mi id="S4.T1.2.1.m1.1.1.2" xref="S4.T1.2.1.m1.1.1.2.cmml">r</mi><mo id="S4.T1.2.1.m1.1.1.1" xref="S4.T1.2.1.m1.1.1.1.cmml">/</mo><msub id="S4.T1.2.1.m1.1.1.3" xref="S4.T1.2.1.m1.1.1.3.cmml"><mi id="S4.T1.2.1.m1.1.1.3.2" xref="S4.T1.2.1.m1.1.1.3.2.cmml">d</mi><mrow id="S4.T1.2.1.m1.1.1.3.3" xref="S4.T1.2.1.m1.1.1.3.3.cmml"><mi id="S4.T1.2.1.m1.1.1.3.3.2" xref="S4.T1.2.1.m1.1.1.3.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="S4.T1.2.1.m1.1.1.3.3.1" xref="S4.T1.2.1.m1.1.1.3.3.1.cmml">​</mo><mi id="S4.T1.2.1.m1.1.1.3.3.3" xref="S4.T1.2.1.m1.1.1.3.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S4.T1.2.1.m1.1.1.3.3.1b" xref="S4.T1.2.1.m1.1.1.3.3.1.cmml">​</mo><mi id="S4.T1.2.1.m1.1.1.3.3.4" xref="S4.T1.2.1.m1.1.1.3.3.4.cmml">d</mi><mo lspace="0em" rspace="0em" id="S4.T1.2.1.m1.1.1.3.3.1c" xref="S4.T1.2.1.m1.1.1.3.3.1.cmml">​</mo><mi id="S4.T1.2.1.m1.1.1.3.3.5" xref="S4.T1.2.1.m1.1.1.3.3.5.cmml">e</mi><mo lspace="0em" rspace="0em" id="S4.T1.2.1.m1.1.1.3.3.1d" xref="S4.T1.2.1.m1.1.1.3.3.1.cmml">​</mo><mi id="S4.T1.2.1.m1.1.1.3.3.6" xref="S4.T1.2.1.m1.1.1.3.3.6.cmml">l</mi></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.2.1.m1.1c"><apply id="S4.T1.2.1.m1.1.1.cmml" xref="S4.T1.2.1.m1.1.1"><divide id="S4.T1.2.1.m1.1.1.1.cmml" xref="S4.T1.2.1.m1.1.1.1"></divide><ci id="S4.T1.2.1.m1.1.1.2.cmml" xref="S4.T1.2.1.m1.1.1.2">𝑟</ci><apply id="S4.T1.2.1.m1.1.1.3.cmml" xref="S4.T1.2.1.m1.1.1.3"><csymbol cd="ambiguous" id="S4.T1.2.1.m1.1.1.3.1.cmml" xref="S4.T1.2.1.m1.1.1.3">subscript</csymbol><ci id="S4.T1.2.1.m1.1.1.3.2.cmml" xref="S4.T1.2.1.m1.1.1.3.2">𝑑</ci><apply id="S4.T1.2.1.m1.1.1.3.3.cmml" xref="S4.T1.2.1.m1.1.1.3.3"><times id="S4.T1.2.1.m1.1.1.3.3.1.cmml" xref="S4.T1.2.1.m1.1.1.3.3.1"></times><ci id="S4.T1.2.1.m1.1.1.3.3.2.cmml" xref="S4.T1.2.1.m1.1.1.3.3.2">𝑚</ci><ci id="S4.T1.2.1.m1.1.1.3.3.3.cmml" xref="S4.T1.2.1.m1.1.1.3.3.3">𝑜</ci><ci id="S4.T1.2.1.m1.1.1.3.3.4.cmml" xref="S4.T1.2.1.m1.1.1.3.3.4">𝑑</ci><ci id="S4.T1.2.1.m1.1.1.3.3.5.cmml" xref="S4.T1.2.1.m1.1.1.3.3.5">𝑒</ci><ci id="S4.T1.2.1.m1.1.1.3.3.6.cmml" xref="S4.T1.2.1.m1.1.1.3.3.6">𝑙</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.2.1.m1.1d">r/d_{model}</annotation></semantics></math> is relative to the largest weight matrix dimension. Perplexity values are averaged over three seeds showing mean and standard error. (*) Denotes results from GaLore <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite>. Only one seed was used for the 1B experiment due to compute constraints.</span></figcaption>
<div id="S4.T1.15" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:141pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-32.3pt,10.5pt) scale(0.870381283622591,0.870381283622591) ;">
<table id="S4.T1.15.13" class="ltx_tabular ltx_align_middle">
<tr id="S4.T1.15.13.14" class="ltx_tr">
<td id="S4.T1.15.13.14.1" class="ltx_td ltx_border_tt"></td>
<td id="S4.T1.15.13.14.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T1.15.13.14.2.1" class="ltx_text ltx_font_bold">60M</span></td>
<td id="S4.T1.15.13.14.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T1.15.13.14.3.1" class="ltx_text ltx_font_bold">130M</span></td>
<td id="S4.T1.15.13.14.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T1.15.13.14.4.1" class="ltx_text ltx_font_bold">350M</span></td>
<td id="S4.T1.15.13.14.5" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T1.15.13.14.5.1" class="ltx_text ltx_font_bold">1B</span></td>
</tr>
<tr id="S4.T1.5.3.3" class="ltx_tr">
<td id="S4.T1.5.3.3.4" class="ltx_td ltx_align_left ltx_border_t">Full</td>
<td id="S4.T1.3.1.1.1" class="ltx_td ltx_align_center ltx_border_t">33.32 <math id="S4.T1.3.1.1.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T1.3.1.1.1.m1.1a"><mo id="S4.T1.3.1.1.1.m1.1.1" xref="S4.T1.3.1.1.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T1.3.1.1.1.m1.1b"><csymbol cd="latexml" id="S4.T1.3.1.1.1.m1.1.1.cmml" xref="S4.T1.3.1.1.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.3.1.1.1.m1.1c">\pm</annotation></semantics></math> 0.22 (0.36G)</td>
<td id="S4.T1.4.2.2.2" class="ltx_td ltx_align_center ltx_border_t">24.51 <math id="S4.T1.4.2.2.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T1.4.2.2.2.m1.1a"><mo id="S4.T1.4.2.2.2.m1.1.1" xref="S4.T1.4.2.2.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T1.4.2.2.2.m1.1b"><csymbol cd="latexml" id="S4.T1.4.2.2.2.m1.1.1.cmml" xref="S4.T1.4.2.2.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.4.2.2.2.m1.1c">\pm</annotation></semantics></math> 0.03 (0.76G)</td>
<td id="S4.T1.5.3.3.3" class="ltx_td ltx_align_center ltx_border_t">18.87 <math id="S4.T1.5.3.3.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T1.5.3.3.3.m1.1a"><mo id="S4.T1.5.3.3.3.m1.1.1" xref="S4.T1.5.3.3.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T1.5.3.3.3.m1.1b"><csymbol cd="latexml" id="S4.T1.5.3.3.3.m1.1.1.cmml" xref="S4.T1.5.3.3.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.5.3.3.3.m1.1c">\pm</annotation></semantics></math> 0.18 (2.06G)</td>
<td id="S4.T1.5.3.3.5" class="ltx_td ltx_align_center ltx_border_t">15.56* (7.80G)</td>
</tr>
<tr id="S4.T1.8.6.6" class="ltx_tr">
<td id="S4.T1.8.6.6.4" class="ltx_td ltx_align_left ltx_border_t">
<span id="S4.T1.8.6.6.4.1" class="ltx_text ltx_font_smallcaps">LoQT</span> (Ours)</td>
<td id="S4.T1.6.4.4.1" class="ltx_td ltx_align_center ltx_border_t">33.98 <math id="S4.T1.6.4.4.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T1.6.4.4.1.m1.1a"><mo id="S4.T1.6.4.4.1.m1.1.1" xref="S4.T1.6.4.4.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T1.6.4.4.1.m1.1b"><csymbol cd="latexml" id="S4.T1.6.4.4.1.m1.1.1.cmml" xref="S4.T1.6.4.4.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.6.4.4.1.m1.1c">\pm</annotation></semantics></math> 0.15 (0.23G)</td>
<td id="S4.T1.7.5.5.2" class="ltx_td ltx_align_center ltx_border_t">24.57 <math id="S4.T1.7.5.5.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T1.7.5.5.2.m1.1a"><mo id="S4.T1.7.5.5.2.m1.1.1" xref="S4.T1.7.5.5.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T1.7.5.5.2.m1.1b"><csymbol cd="latexml" id="S4.T1.7.5.5.2.m1.1.1.cmml" xref="S4.T1.7.5.5.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.7.5.5.2.m1.1c">\pm</annotation></semantics></math> 0.01 (0.49G)</td>
<td id="S4.T1.8.6.6.3" class="ltx_td ltx_align_center ltx_border_t">19.12 <math id="S4.T1.8.6.6.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T1.8.6.6.3.m1.1a"><mo id="S4.T1.8.6.6.3.m1.1.1" xref="S4.T1.8.6.6.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T1.8.6.6.3.m1.1b"><csymbol cd="latexml" id="S4.T1.8.6.6.3.m1.1.1.cmml" xref="S4.T1.8.6.6.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.8.6.6.3.m1.1c">\pm</annotation></semantics></math> 0.01 (0.98G)</td>
<td id="S4.T1.8.6.6.5" class="ltx_td ltx_align_center ltx_border_t">15.55 (3.16G)</td>
</tr>
<tr id="S4.T1.11.9.9" class="ltx_tr">
<td id="S4.T1.11.9.9.4" class="ltx_td ltx_align_left">
<span id="S4.T1.11.9.9.4.1" class="ltx_text ltx_font_smallcaps">LoQT</span>-nq (No quant.)</td>
<td id="S4.T1.9.7.7.1" class="ltx_td ltx_align_center">33.55 <math id="S4.T1.9.7.7.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T1.9.7.7.1.m1.1a"><mo id="S4.T1.9.7.7.1.m1.1.1" xref="S4.T1.9.7.7.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T1.9.7.7.1.m1.1b"><csymbol cd="latexml" id="S4.T1.9.7.7.1.m1.1.1.cmml" xref="S4.T1.9.7.7.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.9.7.7.1.m1.1c">\pm</annotation></semantics></math> 0.03 (0.28G)</td>
<td id="S4.T1.10.8.8.2" class="ltx_td ltx_align_center">24.37 <math id="S4.T1.10.8.8.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T1.10.8.8.2.m1.1a"><mo id="S4.T1.10.8.8.2.m1.1.1" xref="S4.T1.10.8.8.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T1.10.8.8.2.m1.1b"><csymbol cd="latexml" id="S4.T1.10.8.8.2.m1.1.1.cmml" xref="S4.T1.10.8.8.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.10.8.8.2.m1.1c">\pm</annotation></semantics></math> 0.02 (0.63G)</td>
<td id="S4.T1.11.9.9.3" class="ltx_td ltx_align_center">18.85 <math id="S4.T1.11.9.9.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T1.11.9.9.3.m1.1a"><mo id="S4.T1.11.9.9.3.m1.1.1" xref="S4.T1.11.9.9.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T1.11.9.9.3.m1.1b"><csymbol cd="latexml" id="S4.T1.11.9.9.3.m1.1.1.cmml" xref="S4.T1.11.9.9.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.11.9.9.3.m1.1c">\pm</annotation></semantics></math> 0.01 (1.47G)</td>
<td id="S4.T1.11.9.9.5" class="ltx_td ltx_align_center">15.20 (5.11G)</td>
</tr>
<tr id="S4.T1.14.12.12" class="ltx_tr">
<td id="S4.T1.14.12.12.4" class="ltx_td ltx_align_left">GaLore</td>
<td id="S4.T1.12.10.10.1" class="ltx_td ltx_align_center">34.15 <math id="S4.T1.12.10.10.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T1.12.10.10.1.m1.1a"><mo id="S4.T1.12.10.10.1.m1.1.1" xref="S4.T1.12.10.10.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T1.12.10.10.1.m1.1b"><csymbol cd="latexml" id="S4.T1.12.10.10.1.m1.1.1.cmml" xref="S4.T1.12.10.10.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.12.10.10.1.m1.1c">\pm</annotation></semantics></math> 0.24 (0.24G)</td>
<td id="S4.T1.13.11.11.2" class="ltx_td ltx_align_center">24.81 <math id="S4.T1.13.11.11.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T1.13.11.11.2.m1.1a"><mo id="S4.T1.13.11.11.2.m1.1.1" xref="S4.T1.13.11.11.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T1.13.11.11.2.m1.1b"><csymbol cd="latexml" id="S4.T1.13.11.11.2.m1.1.1.cmml" xref="S4.T1.13.11.11.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.13.11.11.2.m1.1c">\pm</annotation></semantics></math> 0.04 (0.52G)</td>
<td id="S4.T1.14.12.12.3" class="ltx_td ltx_align_center">19.47 <math id="S4.T1.14.12.12.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T1.14.12.12.3.m1.1a"><mo id="S4.T1.14.12.12.3.m1.1.1" xref="S4.T1.14.12.12.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T1.14.12.12.3.m1.1b"><csymbol cd="latexml" id="S4.T1.14.12.12.3.m1.1.1.cmml" xref="S4.T1.14.12.12.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.14.12.12.3.m1.1c">\pm</annotation></semantics></math> 0.01 (1.22G)</td>
<td id="S4.T1.14.12.12.5" class="ltx_td ltx_align_center">15.64* (4.38G)</td>
</tr>
<tr id="S4.T1.15.13.15" class="ltx_tr">
<td id="S4.T1.15.13.15.1" class="ltx_td ltx_align_left">LoRA</td>
<td id="S4.T1.15.13.15.2" class="ltx_td ltx_align_center">34.99* (0.36G)</td>
<td id="S4.T1.15.13.15.3" class="ltx_td ltx_align_center">33.92* (0.80G)</td>
<td id="S4.T1.15.13.15.4" class="ltx_td ltx_align_center">25.58* (1.76G)</td>
<td id="S4.T1.15.13.15.5" class="ltx_td ltx_align_center">19.21* (6.17G)</td>
</tr>
<tr id="S4.T1.15.13.16" class="ltx_tr">
<td id="S4.T1.15.13.16.1" class="ltx_td ltx_align_left">ReLoRA</td>
<td id="S4.T1.15.13.16.2" class="ltx_td ltx_align_center">37.04* (0.36G)</td>
<td id="S4.T1.15.13.16.3" class="ltx_td ltx_align_center">29.37* (0.80G)</td>
<td id="S4.T1.15.13.16.4" class="ltx_td ltx_align_center">29.08* (1.76G)</td>
<td id="S4.T1.15.13.16.5" class="ltx_td ltx_align_center">18.33* (6.17G)</td>
</tr>
<tr id="S4.T1.15.13.13" class="ltx_tr">
<td id="S4.T1.15.13.13.1" class="ltx_td ltx_align_left ltx_border_t"><math id="S4.T1.15.13.13.1.m1.1" class="ltx_Math" alttext="r/d_{model}" display="inline"><semantics id="S4.T1.15.13.13.1.m1.1a"><mrow id="S4.T1.15.13.13.1.m1.1.1" xref="S4.T1.15.13.13.1.m1.1.1.cmml"><mi id="S4.T1.15.13.13.1.m1.1.1.2" xref="S4.T1.15.13.13.1.m1.1.1.2.cmml">r</mi><mo id="S4.T1.15.13.13.1.m1.1.1.1" xref="S4.T1.15.13.13.1.m1.1.1.1.cmml">/</mo><msub id="S4.T1.15.13.13.1.m1.1.1.3" xref="S4.T1.15.13.13.1.m1.1.1.3.cmml"><mi id="S4.T1.15.13.13.1.m1.1.1.3.2" xref="S4.T1.15.13.13.1.m1.1.1.3.2.cmml">d</mi><mrow id="S4.T1.15.13.13.1.m1.1.1.3.3" xref="S4.T1.15.13.13.1.m1.1.1.3.3.cmml"><mi id="S4.T1.15.13.13.1.m1.1.1.3.3.2" xref="S4.T1.15.13.13.1.m1.1.1.3.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="S4.T1.15.13.13.1.m1.1.1.3.3.1" xref="S4.T1.15.13.13.1.m1.1.1.3.3.1.cmml">​</mo><mi id="S4.T1.15.13.13.1.m1.1.1.3.3.3" xref="S4.T1.15.13.13.1.m1.1.1.3.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S4.T1.15.13.13.1.m1.1.1.3.3.1a" xref="S4.T1.15.13.13.1.m1.1.1.3.3.1.cmml">​</mo><mi id="S4.T1.15.13.13.1.m1.1.1.3.3.4" xref="S4.T1.15.13.13.1.m1.1.1.3.3.4.cmml">d</mi><mo lspace="0em" rspace="0em" id="S4.T1.15.13.13.1.m1.1.1.3.3.1b" xref="S4.T1.15.13.13.1.m1.1.1.3.3.1.cmml">​</mo><mi id="S4.T1.15.13.13.1.m1.1.1.3.3.5" xref="S4.T1.15.13.13.1.m1.1.1.3.3.5.cmml">e</mi><mo lspace="0em" rspace="0em" id="S4.T1.15.13.13.1.m1.1.1.3.3.1c" xref="S4.T1.15.13.13.1.m1.1.1.3.3.1.cmml">​</mo><mi id="S4.T1.15.13.13.1.m1.1.1.3.3.6" xref="S4.T1.15.13.13.1.m1.1.1.3.3.6.cmml">l</mi></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="S4.T1.15.13.13.1.m1.1b"><apply id="S4.T1.15.13.13.1.m1.1.1.cmml" xref="S4.T1.15.13.13.1.m1.1.1"><divide id="S4.T1.15.13.13.1.m1.1.1.1.cmml" xref="S4.T1.15.13.13.1.m1.1.1.1"></divide><ci id="S4.T1.15.13.13.1.m1.1.1.2.cmml" xref="S4.T1.15.13.13.1.m1.1.1.2">𝑟</ci><apply id="S4.T1.15.13.13.1.m1.1.1.3.cmml" xref="S4.T1.15.13.13.1.m1.1.1.3"><csymbol cd="ambiguous" id="S4.T1.15.13.13.1.m1.1.1.3.1.cmml" xref="S4.T1.15.13.13.1.m1.1.1.3">subscript</csymbol><ci id="S4.T1.15.13.13.1.m1.1.1.3.2.cmml" xref="S4.T1.15.13.13.1.m1.1.1.3.2">𝑑</ci><apply id="S4.T1.15.13.13.1.m1.1.1.3.3.cmml" xref="S4.T1.15.13.13.1.m1.1.1.3.3"><times id="S4.T1.15.13.13.1.m1.1.1.3.3.1.cmml" xref="S4.T1.15.13.13.1.m1.1.1.3.3.1"></times><ci id="S4.T1.15.13.13.1.m1.1.1.3.3.2.cmml" xref="S4.T1.15.13.13.1.m1.1.1.3.3.2">𝑚</ci><ci id="S4.T1.15.13.13.1.m1.1.1.3.3.3.cmml" xref="S4.T1.15.13.13.1.m1.1.1.3.3.3">𝑜</ci><ci id="S4.T1.15.13.13.1.m1.1.1.3.3.4.cmml" xref="S4.T1.15.13.13.1.m1.1.1.3.3.4">𝑑</ci><ci id="S4.T1.15.13.13.1.m1.1.1.3.3.5.cmml" xref="S4.T1.15.13.13.1.m1.1.1.3.3.5">𝑒</ci><ci id="S4.T1.15.13.13.1.m1.1.1.3.3.6.cmml" xref="S4.T1.15.13.13.1.m1.1.1.3.3.6">𝑙</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.15.13.13.1.m1.1c">r/d_{model}</annotation></semantics></math></td>
<td id="S4.T1.15.13.13.2" class="ltx_td ltx_align_center ltx_border_t">128 / 256</td>
<td id="S4.T1.15.13.13.3" class="ltx_td ltx_align_center ltx_border_t">256 / 768</td>
<td id="S4.T1.15.13.13.4" class="ltx_td ltx_align_center ltx_border_t">256 / 1024</td>
<td id="S4.T1.15.13.13.5" class="ltx_td ltx_align_center ltx_border_t">512 / 2048</td>
</tr>
<tr id="S4.T1.15.13.17" class="ltx_tr">
<td id="S4.T1.15.13.17.1" class="ltx_td ltx_align_left ltx_border_bb">Training Tokens</td>
<td id="S4.T1.15.13.17.2" class="ltx_td ltx_align_center ltx_border_bb">1.1B</td>
<td id="S4.T1.15.13.17.3" class="ltx_td ltx_align_center ltx_border_bb">2.2B</td>
<td id="S4.T1.15.13.17.4" class="ltx_td ltx_align_center ltx_border_bb">6.4B</td>
<td id="S4.T1.15.13.17.5" class="ltx_td ltx_align_center ltx_border_bb">13.1B</td>
</tr>
</table>
</span></div>
</figure>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Experimental Setup</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">We evaluate <span id="S4.SS1.p1.1.1" class="ltx_text ltx_font_smallcaps">LoQT</span> by training LLaMA-based<cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib34" title="" class="ltx_ref">34</a>]</cite> language models on the C4 dataset <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib30" title="" class="ltx_ref">30</a>]</cite>, a collection of processed text in English that was scraped from the internet <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib30" title="" class="ltx_ref">30</a>]</cite>. We train models of sizes of 60M, 130M, 350M, and 1B parameters, adhering to single-epoch training cycles determined by Chinchilla Scaling Laws <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite>. While <span id="S4.SS1.p1.1.2" class="ltx_text ltx_font_smallcaps">LoQT</span> is capable of training models up to 13 billion parameters on consumer GPUs, compute limits prevent us from training to convergence for sizes above 1B. We also benchmark <span id="S4.SS1.p1.1.3" class="ltx_text ltx_font_smallcaps">LoQT</span> on the GLUE test-suite for natural language understanding <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib37" title="" class="ltx_ref">37</a>]</cite>. Runs were conducted on up to 4x 40GB NVIDIA A100s 2x 80GB NVIDIA H100s, or a single 24GB NVIDIA RTX 3090. The longest run was the training of the 1B models, taking approximately four days on the four A100s. The 3090 was used for throughput and to empirically verify memory claims.</p>
</div>
<div id="S4.SS1.p2" class="ltx_para">
<p id="S4.SS1.p2.1" class="ltx_p">Hyperparameters are consistent across model sizes, with experiments in BF16 format for memory efficiency. All models use a maximum sequence length of <math id="S4.SS1.p2.1.m1.1" class="ltx_Math" alttext="256" display="inline"><semantics id="S4.SS1.p2.1.m1.1a"><mn id="S4.SS1.p2.1.m1.1.1" xref="S4.SS1.p2.1.m1.1.1.cmml">256</mn><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.1.m1.1b"><cn type="integer" id="S4.SS1.p2.1.m1.1.1.cmml" xref="S4.SS1.p2.1.m1.1.1">256</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.1.m1.1c">256</annotation></semantics></math>, a total token batch size of 131K tokens, and a learning rate warmup for the first 10% of the training steps, followed by cosine annealing to 10% of the initial learning rate. Full experimental details, including the specific hyperparameters for each task, are provided in Appendix <a href="#A2" title="Appendix B Hyperparamters ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a>.</p>
</div>
<section id="S4.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Baselines</h4>

<div id="S4.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS1.SSS0.Px1.p1.1" class="ltx_p">For pre-training, we compare <span id="S4.SS1.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_smallcaps">LoQT</span> against LoRA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite>, ReLoRA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib22" title="" class="ltx_ref">22</a>]</cite>, GaLore <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite>, and non-quantized version of <span id="S4.SS1.SSS0.Px1.p1.1.2" class="ltx_text ltx_font_smallcaps">LoQT</span>, <span id="S4.SS1.SSS0.Px1.p1.1.3" class="ltx_text ltx_font_smallcaps">LoQT</span>-nq. In our experiments, <span id="S4.SS1.SSS0.Px1.p1.1.4" class="ltx_text ltx_font_smallcaps">LoQT</span>, LoRA, and ReLoRA modify attention and fully connected layers while maintaining full-rank embeddings and normalization layers. This contrasts with GaLore, which keeps weights full-rank but projects gradients to low-rank, and standard full-rank training.
For fine-tuning, we benchmark <span id="S4.SS1.SSS0.Px1.p1.1.5" class="ltx_text ltx_font_smallcaps">LoQT</span> against GaLore, LoftQ <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib21" title="" class="ltx_ref">21</a>]</cite>, LoRA and <span id="S4.SS1.SSS0.Px1.p1.1.6" class="ltx_text ltx_font_smallcaps">LoQT</span>-nq. All models use identical update frequencies for GaLore, ReLoRA, <span id="S4.SS1.SSS0.Px1.p1.1.7" class="ltx_text ltx_font_smallcaps">LoQT</span>-nq, and <span id="S4.SS1.SSS0.Px1.p1.1.8" class="ltx_text ltx_font_smallcaps">LoQT</span>, starting with an update frequency of <math id="S4.SS1.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="T=100" display="inline"><semantics id="S4.SS1.SSS0.Px1.p1.1.m1.1a"><mrow id="S4.SS1.SSS0.Px1.p1.1.m1.1.1" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.cmml"><mi id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.cmml">T</mi><mo id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.1" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.1.cmml">=</mo><mn id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.3" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.3.cmml">100</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p1.1.m1.1b"><apply id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1"><eq id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.1"></eq><ci id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2">𝑇</ci><cn type="integer" id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.3">100</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p1.1.m1.1c">T=100</annotation></semantics></math> and then with exponentially increasing update frequencies. This means that we do more frequent updates early and fewer as the model stabilizes (see Section <a href="#S5.F4.sf2" title="In Figure 4 ‣ 5 Ablations ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4(b)</span></a> for more details).
All models are trained using the Adam optimizer, except GaLore which uses GaLoreAdam for gradient projection.</p>
</div>
</section>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Pre-training of Generative Language Models</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">Results and details for pretraining of language models of sizes 60M, 130M, 350M and 1B parameters are shown in Table <a href="#S4.T1" title="Table 1 ‣ 4 Experiments and Results ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>. Model sizes are calculated based on the full models without any low-rank methods. We see that <span id="S4.SS2.p1.1.1" class="ltx_text ltx_font_smallcaps">LoQT</span> and <span id="S4.SS2.p1.1.2" class="ltx_text ltx_font_smallcaps">LoQT</span>-nq both perform very close to full rank pretraining and GaLore, while using significantly less memory by keeping most of the model weights in a quantized state. For the 60M model, full training is only slightly better than <span id="S4.SS2.p1.1.3" class="ltx_text ltx_font_smallcaps">LoQT</span>, while we see results improve or be within the standard error for the other sizes. We also notice a slight drop in performance from quantizing the original weight matrix, comparing <span id="S4.SS2.p1.1.4" class="ltx_text ltx_font_smallcaps">LoQT</span> and <span id="S4.SS2.p1.1.5" class="ltx_text ltx_font_smallcaps">LoQT</span>-nq. The key difference between the approaches are the theoretical memory estimaes, e.g. where <span id="S4.SS2.p1.1.6" class="ltx_text ltx_font_smallcaps">LoQT</span> requires 59% less memory for the 1B model in full precision and 28% less memory than GaLore.</p>
</div>
<figure id="S4.T2" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S4.T2.44.1.1" class="ltx_text" style="font-size:90%;">Table 2</span>: </span><span id="S4.T2.45.2" class="ltx_text" style="font-size:90%;">Results with <span id="S4.T2.45.2.1" class="ltx_text ltx_font_smallcaps">LoQT</span>, <span id="S4.T2.45.2.2" class="ltx_text ltx_font_smallcaps">LoQT</span>-nq, and GaLore of DeBERTaV3-base models on the GLUE development set. We report mean and standard error over three seeds. The best results on each dataset are shown in <span id="S4.T2.45.2.3" class="ltx_text ltx_font_bold">bold</span>.</span></figcaption>
<div id="S4.T2.39.39" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:94.6pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-72.0pt,15.7pt) scale(0.750747161673296,0.750747161673296) ;">
<table id="S4.T2.39.39.39" class="ltx_tabular ltx_align_middle">
<tr id="S4.T2.39.39.39.40" class="ltx_tr">
<td id="S4.T2.39.39.39.40.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="S4.T2.39.39.39.40.1.1" class="ltx_text ltx_font_bold">Rank</span></td>
<td id="S4.T2.39.39.39.40.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="S4.T2.39.39.39.40.2.1" class="ltx_text ltx_font_bold">Method</span></td>
<td id="S4.T2.39.39.39.40.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T2.39.39.39.40.3.1" class="ltx_text ltx_font_bold">MNLI</span></td>
<td id="S4.T2.39.39.39.40.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T2.39.39.39.40.4.1" class="ltx_text ltx_font_bold">QNLI</span></td>
<td id="S4.T2.39.39.39.40.5" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T2.39.39.39.40.5.1" class="ltx_text ltx_font_bold">RTE</span></td>
<td id="S4.T2.39.39.39.40.6" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T2.39.39.39.40.6.1" class="ltx_text ltx_font_bold">SST</span></td>
<td id="S4.T2.39.39.39.40.7" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T2.39.39.39.40.7.1" class="ltx_text ltx_font_bold">MRPC</span></td>
<td id="S4.T2.39.39.39.40.8" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T2.39.39.39.40.8.1" class="ltx_text ltx_font_bold">CoLA</span></td>
<td id="S4.T2.39.39.39.40.9" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T2.39.39.39.40.9.1" class="ltx_text ltx_font_bold">QQP</span></td>
<td id="S4.T2.39.39.39.40.10" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="S4.T2.39.39.39.40.10.1" class="ltx_text ltx_font_bold">STSB</span></td>
<td id="S4.T2.39.39.39.40.11" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T2.39.39.39.40.11.1" class="ltx_text ltx_font_bold">Average</span></td>
</tr>
<tr id="S4.T2.39.39.39.41" class="ltx_tr">
<td id="S4.T2.39.39.39.41.1" class="ltx_td ltx_border_r"></td>
<td id="S4.T2.39.39.39.41.2" class="ltx_td ltx_border_r"></td>
<td id="S4.T2.39.39.39.41.3" class="ltx_td ltx_align_center">Acc</td>
<td id="S4.T2.39.39.39.41.4" class="ltx_td ltx_align_center">Acc</td>
<td id="S4.T2.39.39.39.41.5" class="ltx_td ltx_align_center">Acc</td>
<td id="S4.T2.39.39.39.41.6" class="ltx_td ltx_align_center">Acc</td>
<td id="S4.T2.39.39.39.41.7" class="ltx_td ltx_align_center">f1</td>
<td id="S4.T2.39.39.39.41.8" class="ltx_td ltx_align_center">Matt</td>
<td id="S4.T2.39.39.39.41.9" class="ltx_td ltx_align_center">f1</td>
<td id="S4.T2.39.39.39.41.10" class="ltx_td ltx_align_center ltx_border_r">PCorr</td>
<td id="S4.T2.39.39.39.41.11" class="ltx_td"></td>
</tr>
<tr id="S4.T2.8.8.8.8" class="ltx_tr">
<td id="S4.T2.8.8.8.8.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">32</td>
<td id="S4.T2.8.8.8.8.10" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">
<span id="S4.T2.8.8.8.8.10.1" class="ltx_text ltx_font_smallcaps">LoQT</span>-nq</td>
<td id="S4.T2.1.1.1.1.1" class="ltx_td ltx_align_center ltx_border_t">90.0<math id="S4.T2.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.1.1.1.1.1.m1.1a"><mo id="S4.T2.1.1.1.1.1.m1.1.1" xref="S4.T2.1.1.1.1.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.1.1.1.1.1.m1.1b"><csymbol cd="latexml" id="S4.T2.1.1.1.1.1.m1.1.1.cmml" xref="S4.T2.1.1.1.1.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.1.1.1.1.1.m1.1c">\pm</annotation></semantics></math>0.10</td>
<td id="S4.T2.2.2.2.2.2" class="ltx_td ltx_align_center ltx_border_t">94.2<math id="S4.T2.2.2.2.2.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.2.2.2.2.2.m1.1a"><mo id="S4.T2.2.2.2.2.2.m1.1.1" xref="S4.T2.2.2.2.2.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.2.2.2.2.2.m1.1b"><csymbol cd="latexml" id="S4.T2.2.2.2.2.2.m1.1.1.cmml" xref="S4.T2.2.2.2.2.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.2.2.2.2.2.m1.1c">\pm</annotation></semantics></math>0.06</td>
<td id="S4.T2.3.3.3.3.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T2.3.3.3.3.3.1" class="ltx_text ltx_font_bold">84.8<math id="S4.T2.3.3.3.3.3.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.3.3.3.3.3.1.m1.1a"><mo id="S4.T2.3.3.3.3.3.1.m1.1.1" xref="S4.T2.3.3.3.3.3.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.3.3.3.3.3.1.m1.1b"><csymbol cd="latexml" id="S4.T2.3.3.3.3.3.1.m1.1.1.cmml" xref="S4.T2.3.3.3.3.3.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.3.3.3.3.3.1.m1.1c">\pm</annotation></semantics></math>0.75</span></td>
<td id="S4.T2.4.4.4.4.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T2.4.4.4.4.4.1" class="ltx_text ltx_font_bold">95.9<math id="S4.T2.4.4.4.4.4.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.4.4.4.4.4.1.m1.1a"><mo id="S4.T2.4.4.4.4.4.1.m1.1.1" xref="S4.T2.4.4.4.4.4.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.4.4.4.4.4.1.m1.1b"><csymbol cd="latexml" id="S4.T2.4.4.4.4.4.1.m1.1.1.cmml" xref="S4.T2.4.4.4.4.4.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.4.4.4.4.4.1.m1.1c">\pm</annotation></semantics></math>0.06</span></td>
<td id="S4.T2.5.5.5.5.5" class="ltx_td ltx_align_center ltx_border_t">94.1<math id="S4.T2.5.5.5.5.5.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.5.5.5.5.5.m1.1a"><mo id="S4.T2.5.5.5.5.5.m1.1.1" xref="S4.T2.5.5.5.5.5.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.5.5.5.5.5.m1.1b"><csymbol cd="latexml" id="S4.T2.5.5.5.5.5.m1.1.1.cmml" xref="S4.T2.5.5.5.5.5.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.5.5.5.5.5.m1.1c">\pm</annotation></semantics></math>0.25</td>
<td id="S4.T2.6.6.6.6.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T2.6.6.6.6.6.1" class="ltx_text ltx_font_bold">72.5<math id="S4.T2.6.6.6.6.6.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.6.6.6.6.6.1.m1.1a"><mo id="S4.T2.6.6.6.6.6.1.m1.1.1" xref="S4.T2.6.6.6.6.6.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.6.6.6.6.6.1.m1.1b"><csymbol cd="latexml" id="S4.T2.6.6.6.6.6.1.m1.1.1.cmml" xref="S4.T2.6.6.6.6.6.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.6.6.6.6.6.1.m1.1c">\pm</annotation></semantics></math>0.41</span></td>
<td id="S4.T2.7.7.7.7.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T2.7.7.7.7.7.1" class="ltx_text ltx_font_bold">90.0<math id="S4.T2.7.7.7.7.7.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.7.7.7.7.7.1.m1.1a"><mo id="S4.T2.7.7.7.7.7.1.m1.1.1" xref="S4.T2.7.7.7.7.7.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.7.7.7.7.7.1.m1.1b"><csymbol cd="latexml" id="S4.T2.7.7.7.7.7.1.m1.1.1.cmml" xref="S4.T2.7.7.7.7.7.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.7.7.7.7.7.1.m1.1c">\pm</annotation></semantics></math>0.06</span></td>
<td id="S4.T2.8.8.8.8.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">91.5<math id="S4.T2.8.8.8.8.8.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.8.8.8.8.8.m1.1a"><mo id="S4.T2.8.8.8.8.8.m1.1.1" xref="S4.T2.8.8.8.8.8.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.8.8.8.8.8.m1.1b"><csymbol cd="latexml" id="S4.T2.8.8.8.8.8.m1.1.1.cmml" xref="S4.T2.8.8.8.8.8.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.8.8.8.8.8.m1.1c">\pm</annotation></semantics></math>0.07</td>
<td id="S4.T2.8.8.8.8.11" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T2.8.8.8.8.11.1" class="ltx_text ltx_font_bold">89.1</span></td>
</tr>
<tr id="S4.T2.16.16.16.16" class="ltx_tr">
<td id="S4.T2.16.16.16.16.9" class="ltx_td ltx_align_center ltx_border_r">32</td>
<td id="S4.T2.16.16.16.16.10" class="ltx_td ltx_align_center ltx_border_r"><span id="S4.T2.16.16.16.16.10.1" class="ltx_text ltx_font_smallcaps">LoQT</span></td>
<td id="S4.T2.9.9.9.9.1" class="ltx_td ltx_align_center">90.0<math id="S4.T2.9.9.9.9.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.9.9.9.9.1.m1.1a"><mo id="S4.T2.9.9.9.9.1.m1.1.1" xref="S4.T2.9.9.9.9.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.9.9.9.9.1.m1.1b"><csymbol cd="latexml" id="S4.T2.9.9.9.9.1.m1.1.1.cmml" xref="S4.T2.9.9.9.9.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.9.9.9.9.1.m1.1c">\pm</annotation></semantics></math>0.09</td>
<td id="S4.T2.10.10.10.10.2" class="ltx_td ltx_align_center"><span id="S4.T2.10.10.10.10.2.1" class="ltx_text ltx_font_bold">94.3<math id="S4.T2.10.10.10.10.2.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.10.10.10.10.2.1.m1.1a"><mo id="S4.T2.10.10.10.10.2.1.m1.1.1" xref="S4.T2.10.10.10.10.2.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.10.10.10.10.2.1.m1.1b"><csymbol cd="latexml" id="S4.T2.10.10.10.10.2.1.m1.1.1.cmml" xref="S4.T2.10.10.10.10.2.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.10.10.10.10.2.1.m1.1c">\pm</annotation></semantics></math>0.04</span></td>
<td id="S4.T2.11.11.11.11.3" class="ltx_td ltx_align_center">84.1<math id="S4.T2.11.11.11.11.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.11.11.11.11.3.m1.1a"><mo id="S4.T2.11.11.11.11.3.m1.1.1" xref="S4.T2.11.11.11.11.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.11.11.11.11.3.m1.1b"><csymbol cd="latexml" id="S4.T2.11.11.11.11.3.m1.1.1.cmml" xref="S4.T2.11.11.11.11.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.11.11.11.11.3.m1.1c">\pm</annotation></semantics></math>0.91</td>
<td id="S4.T2.12.12.12.12.4" class="ltx_td ltx_align_center">95.5<math id="S4.T2.12.12.12.12.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.12.12.12.12.4.m1.1a"><mo id="S4.T2.12.12.12.12.4.m1.1.1" xref="S4.T2.12.12.12.12.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.12.12.12.12.4.m1.1b"><csymbol cd="latexml" id="S4.T2.12.12.12.12.4.m1.1.1.cmml" xref="S4.T2.12.12.12.12.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.12.12.12.12.4.m1.1c">\pm</annotation></semantics></math>0.10</td>
<td id="S4.T2.13.13.13.13.5" class="ltx_td ltx_align_center"><span id="S4.T2.13.13.13.13.5.1" class="ltx_text ltx_font_bold">94.4<math id="S4.T2.13.13.13.13.5.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.13.13.13.13.5.1.m1.1a"><mo id="S4.T2.13.13.13.13.5.1.m1.1.1" xref="S4.T2.13.13.13.13.5.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.13.13.13.13.5.1.m1.1b"><csymbol cd="latexml" id="S4.T2.13.13.13.13.5.1.m1.1.1.cmml" xref="S4.T2.13.13.13.13.5.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.13.13.13.13.5.1.m1.1c">\pm</annotation></semantics></math>0.20</span></td>
<td id="S4.T2.14.14.14.14.6" class="ltx_td ltx_align_center">70.5<math id="S4.T2.14.14.14.14.6.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.14.14.14.14.6.m1.1a"><mo id="S4.T2.14.14.14.14.6.m1.1.1" xref="S4.T2.14.14.14.14.6.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.14.14.14.14.6.m1.1b"><csymbol cd="latexml" id="S4.T2.14.14.14.14.6.m1.1.1.cmml" xref="S4.T2.14.14.14.14.6.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.14.14.14.14.6.m1.1c">\pm</annotation></semantics></math>0.35</td>
<td id="S4.T2.15.15.15.15.7" class="ltx_td ltx_align_center">89.2<math id="S4.T2.15.15.15.15.7.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.15.15.15.15.7.m1.1a"><mo id="S4.T2.15.15.15.15.7.m1.1.1" xref="S4.T2.15.15.15.15.7.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.15.15.15.15.7.m1.1b"><csymbol cd="latexml" id="S4.T2.15.15.15.15.7.m1.1.1.cmml" xref="S4.T2.15.15.15.15.7.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.15.15.15.15.7.m1.1c">\pm</annotation></semantics></math>0.02</td>
<td id="S4.T2.16.16.16.16.8" class="ltx_td ltx_align_center ltx_border_r"><span id="S4.T2.16.16.16.16.8.1" class="ltx_text ltx_font_bold">91.5<math id="S4.T2.16.16.16.16.8.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.16.16.16.16.8.1.m1.1a"><mo id="S4.T2.16.16.16.16.8.1.m1.1.1" xref="S4.T2.16.16.16.16.8.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.16.16.16.16.8.1.m1.1b"><csymbol cd="latexml" id="S4.T2.16.16.16.16.8.1.m1.1.1.cmml" xref="S4.T2.16.16.16.16.8.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.16.16.16.16.8.1.m1.1c">\pm</annotation></semantics></math>0.13</span></td>
<td id="S4.T2.16.16.16.16.11" class="ltx_td ltx_align_center">88.7</td>
</tr>
<tr id="S4.T2.24.24.24.24" class="ltx_tr">
<td id="S4.T2.24.24.24.24.9" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">32</td>
<td id="S4.T2.24.24.24.24.10" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">LoRA</td>
<td id="S4.T2.17.17.17.17.1" class="ltx_td ltx_align_center ltx_border_t">89.9<math id="S4.T2.17.17.17.17.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.17.17.17.17.1.m1.1a"><mo id="S4.T2.17.17.17.17.1.m1.1.1" xref="S4.T2.17.17.17.17.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.17.17.17.17.1.m1.1b"><csymbol cd="latexml" id="S4.T2.17.17.17.17.1.m1.1.1.cmml" xref="S4.T2.17.17.17.17.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.17.17.17.17.1.m1.1c">\pm</annotation></semantics></math>0.03</td>
<td id="S4.T2.18.18.18.18.2" class="ltx_td ltx_align_center ltx_border_t">94.0<math id="S4.T2.18.18.18.18.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.18.18.18.18.2.m1.1a"><mo id="S4.T2.18.18.18.18.2.m1.1.1" xref="S4.T2.18.18.18.18.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.18.18.18.18.2.m1.1b"><csymbol cd="latexml" id="S4.T2.18.18.18.18.2.m1.1.1.cmml" xref="S4.T2.18.18.18.18.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.18.18.18.18.2.m1.1c">\pm</annotation></semantics></math>0.09</td>
<td id="S4.T2.19.19.19.19.3" class="ltx_td ltx_align_center ltx_border_t">83.6<math id="S4.T2.19.19.19.19.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.19.19.19.19.3.m1.1a"><mo id="S4.T2.19.19.19.19.3.m1.1.1" xref="S4.T2.19.19.19.19.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.19.19.19.19.3.m1.1b"><csymbol cd="latexml" id="S4.T2.19.19.19.19.3.m1.1.1.cmml" xref="S4.T2.19.19.19.19.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.19.19.19.19.3.m1.1c">\pm</annotation></semantics></math>0.12</td>
<td id="S4.T2.20.20.20.20.4" class="ltx_td ltx_align_center ltx_border_t">95.7<math id="S4.T2.20.20.20.20.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.20.20.20.20.4.m1.1a"><mo id="S4.T2.20.20.20.20.4.m1.1.1" xref="S4.T2.20.20.20.20.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.20.20.20.20.4.m1.1b"><csymbol cd="latexml" id="S4.T2.20.20.20.20.4.m1.1.1.cmml" xref="S4.T2.20.20.20.20.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.20.20.20.20.4.m1.1c">\pm</annotation></semantics></math>0.15</td>
<td id="S4.T2.21.21.21.21.5" class="ltx_td ltx_align_center ltx_border_t">93.5<math id="S4.T2.21.21.21.21.5.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.21.21.21.21.5.m1.1a"><mo id="S4.T2.21.21.21.21.5.m1.1.1" xref="S4.T2.21.21.21.21.5.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.21.21.21.21.5.m1.1b"><csymbol cd="latexml" id="S4.T2.21.21.21.21.5.m1.1.1.cmml" xref="S4.T2.21.21.21.21.5.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.21.21.21.21.5.m1.1c">\pm</annotation></semantics></math>0.26</td>
<td id="S4.T2.22.22.22.22.6" class="ltx_td ltx_align_center ltx_border_t">69.3<math id="S4.T2.22.22.22.22.6.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.22.22.22.22.6.m1.1a"><mo id="S4.T2.22.22.22.22.6.m1.1.1" xref="S4.T2.22.22.22.22.6.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.22.22.22.22.6.m1.1b"><csymbol cd="latexml" id="S4.T2.22.22.22.22.6.m1.1.1.cmml" xref="S4.T2.22.22.22.22.6.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.22.22.22.22.6.m1.1c">\pm</annotation></semantics></math>0.47</td>
<td id="S4.T2.23.23.23.23.7" class="ltx_td ltx_align_center ltx_border_t">89.8<math id="S4.T2.23.23.23.23.7.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.23.23.23.23.7.m1.1a"><mo id="S4.T2.23.23.23.23.7.m1.1.1" xref="S4.T2.23.23.23.23.7.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.23.23.23.23.7.m1.1b"><csymbol cd="latexml" id="S4.T2.23.23.23.23.7.m1.1.1.cmml" xref="S4.T2.23.23.23.23.7.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.23.23.23.23.7.m1.1c">\pm</annotation></semantics></math>0.11</td>
<td id="S4.T2.24.24.24.24.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">90.7<math id="S4.T2.24.24.24.24.8.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.24.24.24.24.8.m1.1a"><mo id="S4.T2.24.24.24.24.8.m1.1.1" xref="S4.T2.24.24.24.24.8.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.24.24.24.24.8.m1.1b"><csymbol cd="latexml" id="S4.T2.24.24.24.24.8.m1.1.1.cmml" xref="S4.T2.24.24.24.24.8.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.24.24.24.24.8.m1.1c">\pm</annotation></semantics></math>0.22</td>
<td id="S4.T2.24.24.24.24.11" class="ltx_td ltx_align_center ltx_border_t">88.3</td>
</tr>
<tr id="S4.T2.31.31.31.31" class="ltx_tr">
<td id="S4.T2.31.31.31.31.8" class="ltx_td ltx_align_center ltx_border_r">32</td>
<td id="S4.T2.31.31.31.31.9" class="ltx_td ltx_align_center ltx_border_r">LoftQ</td>
<td id="S4.T2.25.25.25.25.1" class="ltx_td ltx_align_center">90.4<math id="S4.T2.25.25.25.25.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.25.25.25.25.1.m1.1a"><mo id="S4.T2.25.25.25.25.1.m1.1.1" xref="S4.T2.25.25.25.25.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.25.25.25.25.1.m1.1b"><csymbol cd="latexml" id="S4.T2.25.25.25.25.1.m1.1.1.cmml" xref="S4.T2.25.25.25.25.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.25.25.25.25.1.m1.1c">\pm</annotation></semantics></math>0.09</td>
<td id="S4.T2.26.26.26.26.2" class="ltx_td ltx_align_center">93.2<math id="S4.T2.26.26.26.26.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.26.26.26.26.2.m1.1a"><mo id="S4.T2.26.26.26.26.2.m1.1.1" xref="S4.T2.26.26.26.26.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.26.26.26.26.2.m1.1b"><csymbol cd="latexml" id="S4.T2.26.26.26.26.2.m1.1.1.cmml" xref="S4.T2.26.26.26.26.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.26.26.26.26.2.m1.1c">\pm</annotation></semantics></math>0.02</td>
<td id="S4.T2.27.27.27.27.3" class="ltx_td ltx_align_center">83.8<math id="S4.T2.27.27.27.27.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.27.27.27.27.3.m1.1a"><mo id="S4.T2.27.27.27.27.3.m1.1.1" xref="S4.T2.27.27.27.27.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.27.27.27.27.3.m1.1b"><csymbol cd="latexml" id="S4.T2.27.27.27.27.3.m1.1.1.cmml" xref="S4.T2.27.27.27.27.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.27.27.27.27.3.m1.1c">\pm</annotation></semantics></math>0.63</td>
<td id="S4.T2.31.31.31.31.10" class="ltx_td ltx_align_center">94.7*</td>
<td id="S4.T2.28.28.28.28.4" class="ltx_td ltx_align_center">93.2<math id="S4.T2.28.28.28.28.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.28.28.28.28.4.m1.1a"><mo id="S4.T2.28.28.28.28.4.m1.1.1" xref="S4.T2.28.28.28.28.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.28.28.28.28.4.m1.1b"><csymbol cd="latexml" id="S4.T2.28.28.28.28.4.m1.1.1.cmml" xref="S4.T2.28.28.28.28.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.28.28.28.28.4.m1.1c">\pm</annotation></semantics></math>0.14</td>
<td id="S4.T2.29.29.29.29.5" class="ltx_td ltx_align_center">71.1<math id="S4.T2.29.29.29.29.5.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.29.29.29.29.5.m1.1a"><mo id="S4.T2.29.29.29.29.5.m1.1.1" xref="S4.T2.29.29.29.29.5.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.29.29.29.29.5.m1.1b"><csymbol cd="latexml" id="S4.T2.29.29.29.29.5.m1.1.1.cmml" xref="S4.T2.29.29.29.29.5.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.29.29.29.29.5.m1.1c">\pm</annotation></semantics></math>0.28</td>
<td id="S4.T2.30.30.30.30.6" class="ltx_td ltx_align_center">89.6<math id="S4.T2.30.30.30.30.6.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.30.30.30.30.6.m1.1a"><mo id="S4.T2.30.30.30.30.6.m1.1.1" xref="S4.T2.30.30.30.30.6.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.30.30.30.30.6.m1.1b"><csymbol cd="latexml" id="S4.T2.30.30.30.30.6.m1.1.1.cmml" xref="S4.T2.30.30.30.30.6.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.30.30.30.30.6.m1.1c">\pm</annotation></semantics></math>0.12</td>
<td id="S4.T2.31.31.31.31.7" class="ltx_td ltx_align_center ltx_border_r">91.0<math id="S4.T2.31.31.31.31.7.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.31.31.31.31.7.m1.1a"><mo id="S4.T2.31.31.31.31.7.m1.1.1" xref="S4.T2.31.31.31.31.7.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.31.31.31.31.7.m1.1b"><csymbol cd="latexml" id="S4.T2.31.31.31.31.7.m1.1.1.cmml" xref="S4.T2.31.31.31.31.7.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.31.31.31.31.7.m1.1c">\pm</annotation></semantics></math>0.09</td>
<td id="S4.T2.31.31.31.31.11" class="ltx_td ltx_align_center">88.4</td>
</tr>
<tr id="S4.T2.39.39.39.39" class="ltx_tr">
<td id="S4.T2.39.39.39.39.9" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">32</td>
<td id="S4.T2.39.39.39.39.10" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">GaLore</td>
<td id="S4.T2.32.32.32.32.1" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T2.32.32.32.32.1.1" class="ltx_text ltx_font_bold">90.3<math id="S4.T2.32.32.32.32.1.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.32.32.32.32.1.1.m1.1a"><mo id="S4.T2.32.32.32.32.1.1.m1.1.1" xref="S4.T2.32.32.32.32.1.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.32.32.32.32.1.1.m1.1b"><csymbol cd="latexml" id="S4.T2.32.32.32.32.1.1.m1.1.1.cmml" xref="S4.T2.32.32.32.32.1.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.32.32.32.32.1.1.m1.1c">\pm</annotation></semantics></math>0.07</span></td>
<td id="S4.T2.33.33.33.33.2" class="ltx_td ltx_align_center ltx_border_bb">94.0<math id="S4.T2.33.33.33.33.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.33.33.33.33.2.m1.1a"><mo id="S4.T2.33.33.33.33.2.m1.1.1" xref="S4.T2.33.33.33.33.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.33.33.33.33.2.m1.1b"><csymbol cd="latexml" id="S4.T2.33.33.33.33.2.m1.1.1.cmml" xref="S4.T2.33.33.33.33.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.33.33.33.33.2.m1.1c">\pm</annotation></semantics></math>0.04</td>
<td id="S4.T2.34.34.34.34.3" class="ltx_td ltx_align_center ltx_border_bb">83.7<math id="S4.T2.34.34.34.34.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.34.34.34.34.3.m1.1a"><mo id="S4.T2.34.34.34.34.3.m1.1.1" xref="S4.T2.34.34.34.34.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.34.34.34.34.3.m1.1b"><csymbol cd="latexml" id="S4.T2.34.34.34.34.3.m1.1.1.cmml" xref="S4.T2.34.34.34.34.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.34.34.34.34.3.m1.1c">\pm</annotation></semantics></math>0.79</td>
<td id="S4.T2.35.35.35.35.4" class="ltx_td ltx_align_center ltx_border_bb">95.6<math id="S4.T2.35.35.35.35.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.35.35.35.35.4.m1.1a"><mo id="S4.T2.35.35.35.35.4.m1.1.1" xref="S4.T2.35.35.35.35.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.35.35.35.35.4.m1.1b"><csymbol cd="latexml" id="S4.T2.35.35.35.35.4.m1.1.1.cmml" xref="S4.T2.35.35.35.35.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.35.35.35.35.4.m1.1c">\pm</annotation></semantics></math>0.07</td>
<td id="S4.T2.36.36.36.36.5" class="ltx_td ltx_align_center ltx_border_bb">93.4<math id="S4.T2.36.36.36.36.5.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.36.36.36.36.5.m1.1a"><mo id="S4.T2.36.36.36.36.5.m1.1.1" xref="S4.T2.36.36.36.36.5.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.36.36.36.36.5.m1.1b"><csymbol cd="latexml" id="S4.T2.36.36.36.36.5.m1.1.1.cmml" xref="S4.T2.36.36.36.36.5.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.36.36.36.36.5.m1.1c">\pm</annotation></semantics></math>0.38</td>
<td id="S4.T2.37.37.37.37.6" class="ltx_td ltx_align_center ltx_border_bb">70.7<math id="S4.T2.37.37.37.37.6.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.37.37.37.37.6.m1.1a"><mo id="S4.T2.37.37.37.37.6.m1.1.1" xref="S4.T2.37.37.37.37.6.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.37.37.37.37.6.m1.1b"><csymbol cd="latexml" id="S4.T2.37.37.37.37.6.m1.1.1.cmml" xref="S4.T2.37.37.37.37.6.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.37.37.37.37.6.m1.1c">\pm</annotation></semantics></math>0.24</td>
<td id="S4.T2.38.38.38.38.7" class="ltx_td ltx_align_center ltx_border_bb">89.8<math id="S4.T2.38.38.38.38.7.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.38.38.38.38.7.m1.1a"><mo id="S4.T2.38.38.38.38.7.m1.1.1" xref="S4.T2.38.38.38.38.7.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.38.38.38.38.7.m1.1b"><csymbol cd="latexml" id="S4.T2.38.38.38.38.7.m1.1.1.cmml" xref="S4.T2.38.38.38.38.7.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.38.38.38.38.7.m1.1c">\pm</annotation></semantics></math>0.05</td>
<td id="S4.T2.39.39.39.39.8" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">90.6<math id="S4.T2.39.39.39.39.8.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.T2.39.39.39.39.8.m1.1a"><mo id="S4.T2.39.39.39.39.8.m1.1.1" xref="S4.T2.39.39.39.39.8.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.T2.39.39.39.39.8.m1.1b"><csymbol cd="latexml" id="S4.T2.39.39.39.39.8.m1.1.1.cmml" xref="S4.T2.39.39.39.39.8.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.39.39.39.39.8.m1.1c">\pm</annotation></semantics></math>0.01</td>
<td id="S4.T2.39.39.39.39.11" class="ltx_td ltx_align_center ltx_border_bb">88.5</td>
</tr>
</table>
</span></div>
</figure>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Memory-efficient finetuning</h3>

<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p">We fine-tune the pre-trained DeBERTa-V3-base<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>From <a target="_blank" href="https://huggingface.co/microsoft/deberta-v3-base" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/microsoft/deberta-v3-base</a>.</span></span></span> <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib13" title="" class="ltx_ref">13</a>]</cite> model on GLUE tasks using <span id="S4.SS3.p1.1.1" class="ltx_text ltx_font_smallcaps">LoQT</span> and compare its performance with a full fine-tuning baseline, LoRA, LoftQ, and GaLore. See Appendix <a href="#A2.T5" title="Table 5 ‣ B.2 Fine-tuning ‣ Appendix B Hyperparamters ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> for details on hyperparameters. Results are given in Table <a href="#S4.T2" title="Table 2 ‣ 4.2 Pre-training of Generative Language Models ‣ 4 Experiments and Results ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.</p>
</div>
<div id="S4.SS3.p2" class="ltx_para">
<p id="S4.SS3.p2.1" class="ltx_p">We find that both <span id="S4.SS3.p2.1.1" class="ltx_text ltx_font_smallcaps">LoQT</span>-nq and <span id="S4.SS3.p2.1.2" class="ltx_text ltx_font_smallcaps">LoQT</span> perform well. And somewhat surprisingly, it sometimes surpasses GaLore, LoftQ, and LoRA.This may indicate that initializing the LoRA factors with information about the gradient of <math id="S4.SS3.p2.1.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S4.SS3.p2.1.m1.1a"><mi id="S4.SS3.p2.1.m1.1.1" xref="S4.SS3.p2.1.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.1.m1.1b"><ci id="S4.SS3.p2.1.m1.1.1.cmml" xref="S4.SS3.p2.1.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.1.m1.1c">W</annotation></semantics></math> could be a beneficial starting point compared to standard initialization methods. As the goal of this work is to limit memory consumption, we leave out further comparisons that could verify these findings to future work.</p>
</div>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.4 </span>Memory and Throughput</h3>

<figure id="S4.T3" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S4.T3.17.4.1" class="ltx_text" style="font-size:90%;">Table 3</span>: </span><span id="S4.T3.6.3" class="ltx_text" style="font-size:90%;">Comparison of memory usage between GaLore, LoRA, and <span id="S4.T3.6.3.1" class="ltx_text ltx_font_smallcaps">LoQT</span>. <math id="S4.T3.4.1.m1.1" class="ltx_Math" alttext="W\in\mathbb{R}^{m\times n}" display="inline"><semantics id="S4.T3.4.1.m1.1b"><mrow id="S4.T3.4.1.m1.1.1" xref="S4.T3.4.1.m1.1.1.cmml"><mi id="S4.T3.4.1.m1.1.1.2" xref="S4.T3.4.1.m1.1.1.2.cmml">W</mi><mo id="S4.T3.4.1.m1.1.1.1" xref="S4.T3.4.1.m1.1.1.1.cmml">∈</mo><msup id="S4.T3.4.1.m1.1.1.3" xref="S4.T3.4.1.m1.1.1.3.cmml"><mi id="S4.T3.4.1.m1.1.1.3.2" xref="S4.T3.4.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="S4.T3.4.1.m1.1.1.3.3" xref="S4.T3.4.1.m1.1.1.3.3.cmml"><mi id="S4.T3.4.1.m1.1.1.3.3.2" xref="S4.T3.4.1.m1.1.1.3.3.2.cmml">m</mi><mo lspace="0.222em" rspace="0.222em" id="S4.T3.4.1.m1.1.1.3.3.1" xref="S4.T3.4.1.m1.1.1.3.3.1.cmml">×</mo><mi id="S4.T3.4.1.m1.1.1.3.3.3" xref="S4.T3.4.1.m1.1.1.3.3.3.cmml">n</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.T3.4.1.m1.1c"><apply id="S4.T3.4.1.m1.1.1.cmml" xref="S4.T3.4.1.m1.1.1"><in id="S4.T3.4.1.m1.1.1.1.cmml" xref="S4.T3.4.1.m1.1.1.1"></in><ci id="S4.T3.4.1.m1.1.1.2.cmml" xref="S4.T3.4.1.m1.1.1.2">𝑊</ci><apply id="S4.T3.4.1.m1.1.1.3.cmml" xref="S4.T3.4.1.m1.1.1.3"><csymbol cd="ambiguous" id="S4.T3.4.1.m1.1.1.3.1.cmml" xref="S4.T3.4.1.m1.1.1.3">superscript</csymbol><ci id="S4.T3.4.1.m1.1.1.3.2.cmml" xref="S4.T3.4.1.m1.1.1.3.2">ℝ</ci><apply id="S4.T3.4.1.m1.1.1.3.3.cmml" xref="S4.T3.4.1.m1.1.1.3.3"><times id="S4.T3.4.1.m1.1.1.3.3.1.cmml" xref="S4.T3.4.1.m1.1.1.3.3.1"></times><ci id="S4.T3.4.1.m1.1.1.3.3.2.cmml" xref="S4.T3.4.1.m1.1.1.3.3.2">𝑚</ci><ci id="S4.T3.4.1.m1.1.1.3.3.3.cmml" xref="S4.T3.4.1.m1.1.1.3.3.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.4.1.m1.1d">W\in\mathbb{R}^{m\times n}</annotation></semantics></math> (<math id="S4.T3.5.2.m2.1" class="ltx_Math" alttext="m\leq n" display="inline"><semantics id="S4.T3.5.2.m2.1b"><mrow id="S4.T3.5.2.m2.1.1" xref="S4.T3.5.2.m2.1.1.cmml"><mi id="S4.T3.5.2.m2.1.1.2" xref="S4.T3.5.2.m2.1.1.2.cmml">m</mi><mo id="S4.T3.5.2.m2.1.1.1" xref="S4.T3.5.2.m2.1.1.1.cmml">≤</mo><mi id="S4.T3.5.2.m2.1.1.3" xref="S4.T3.5.2.m2.1.1.3.cmml">n</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.T3.5.2.m2.1c"><apply id="S4.T3.5.2.m2.1.1.cmml" xref="S4.T3.5.2.m2.1.1"><leq id="S4.T3.5.2.m2.1.1.1.cmml" xref="S4.T3.5.2.m2.1.1.1"></leq><ci id="S4.T3.5.2.m2.1.1.2.cmml" xref="S4.T3.5.2.m2.1.1.2">𝑚</ci><ci id="S4.T3.5.2.m2.1.1.3.cmml" xref="S4.T3.5.2.m2.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.5.2.m2.1d">m\leq n</annotation></semantics></math>), rank <math id="S4.T3.6.3.m3.1" class="ltx_Math" alttext="r" display="inline"><semantics id="S4.T3.6.3.m3.1b"><mi id="S4.T3.6.3.m3.1.1" xref="S4.T3.6.3.m3.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="S4.T3.6.3.m3.1c"><ci id="S4.T3.6.3.m3.1.1.cmml" xref="S4.T3.6.3.m3.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.6.3.m3.1d">r</annotation></semantics></math>.</span></figcaption>
<table id="S4.T3.15.9" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S4.T3.15.9.10" class="ltx_tr">
<td id="S4.T3.15.9.10.1" class="ltx_td ltx_border_tt"></td>
<td id="S4.T3.15.9.10.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T3.15.9.10.2.1" class="ltx_text" style="font-size:90%;">GaLore</span></td>
<td id="S4.T3.15.9.10.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T3.15.9.10.3.1" class="ltx_text" style="font-size:90%;">LoRA</span></td>
<td id="S4.T3.15.9.10.4" class="ltx_td ltx_align_center ltx_border_tt">
<span id="S4.T3.15.9.10.4.1" class="ltx_text ltx_font_smallcaps" style="font-size:90%;">LoQT</span><span id="S4.T3.15.9.10.4.2" class="ltx_text" style="font-size:90%;"> (Ours)</span>
</td>
</tr>
<tr id="S4.T3.9.3.3" class="ltx_tr">
<td id="S4.T3.9.3.3.4" class="ltx_td ltx_align_left ltx_border_t"><span id="S4.T3.9.3.3.4.1" class="ltx_text" style="font-size:90%;">Weights</span></td>
<td id="S4.T3.7.1.1.1" class="ltx_td ltx_align_center ltx_border_t"><math id="S4.T3.7.1.1.1.m1.1" class="ltx_Math" alttext="mn" display="inline"><semantics id="S4.T3.7.1.1.1.m1.1a"><mrow id="S4.T3.7.1.1.1.m1.1.1" xref="S4.T3.7.1.1.1.m1.1.1.cmml"><mi mathsize="90%" id="S4.T3.7.1.1.1.m1.1.1.2" xref="S4.T3.7.1.1.1.m1.1.1.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="S4.T3.7.1.1.1.m1.1.1.1" xref="S4.T3.7.1.1.1.m1.1.1.1.cmml">​</mo><mi mathsize="90%" id="S4.T3.7.1.1.1.m1.1.1.3" xref="S4.T3.7.1.1.1.m1.1.1.3.cmml">n</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.T3.7.1.1.1.m1.1b"><apply id="S4.T3.7.1.1.1.m1.1.1.cmml" xref="S4.T3.7.1.1.1.m1.1.1"><times id="S4.T3.7.1.1.1.m1.1.1.1.cmml" xref="S4.T3.7.1.1.1.m1.1.1.1"></times><ci id="S4.T3.7.1.1.1.m1.1.1.2.cmml" xref="S4.T3.7.1.1.1.m1.1.1.2">𝑚</ci><ci id="S4.T3.7.1.1.1.m1.1.1.3.cmml" xref="S4.T3.7.1.1.1.m1.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.7.1.1.1.m1.1c">mn</annotation></semantics></math></td>
<td id="S4.T3.8.2.2.2" class="ltx_td ltx_align_center ltx_border_t"><math id="S4.T3.8.2.2.2.m1.1" class="ltx_Math" alttext="mn+mr+nr" display="inline"><semantics id="S4.T3.8.2.2.2.m1.1a"><mrow id="S4.T3.8.2.2.2.m1.1.1" xref="S4.T3.8.2.2.2.m1.1.1.cmml"><mrow id="S4.T3.8.2.2.2.m1.1.1.2" xref="S4.T3.8.2.2.2.m1.1.1.2.cmml"><mi mathsize="90%" id="S4.T3.8.2.2.2.m1.1.1.2.2" xref="S4.T3.8.2.2.2.m1.1.1.2.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="S4.T3.8.2.2.2.m1.1.1.2.1" xref="S4.T3.8.2.2.2.m1.1.1.2.1.cmml">​</mo><mi mathsize="90%" id="S4.T3.8.2.2.2.m1.1.1.2.3" xref="S4.T3.8.2.2.2.m1.1.1.2.3.cmml">n</mi></mrow><mo mathsize="90%" id="S4.T3.8.2.2.2.m1.1.1.1" xref="S4.T3.8.2.2.2.m1.1.1.1.cmml">+</mo><mrow id="S4.T3.8.2.2.2.m1.1.1.3" xref="S4.T3.8.2.2.2.m1.1.1.3.cmml"><mi mathsize="90%" id="S4.T3.8.2.2.2.m1.1.1.3.2" xref="S4.T3.8.2.2.2.m1.1.1.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="S4.T3.8.2.2.2.m1.1.1.3.1" xref="S4.T3.8.2.2.2.m1.1.1.3.1.cmml">​</mo><mi mathsize="90%" id="S4.T3.8.2.2.2.m1.1.1.3.3" xref="S4.T3.8.2.2.2.m1.1.1.3.3.cmml">r</mi></mrow><mo mathsize="90%" id="S4.T3.8.2.2.2.m1.1.1.1a" xref="S4.T3.8.2.2.2.m1.1.1.1.cmml">+</mo><mrow id="S4.T3.8.2.2.2.m1.1.1.4" xref="S4.T3.8.2.2.2.m1.1.1.4.cmml"><mi mathsize="90%" id="S4.T3.8.2.2.2.m1.1.1.4.2" xref="S4.T3.8.2.2.2.m1.1.1.4.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="S4.T3.8.2.2.2.m1.1.1.4.1" xref="S4.T3.8.2.2.2.m1.1.1.4.1.cmml">​</mo><mi mathsize="90%" id="S4.T3.8.2.2.2.m1.1.1.4.3" xref="S4.T3.8.2.2.2.m1.1.1.4.3.cmml">r</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T3.8.2.2.2.m1.1b"><apply id="S4.T3.8.2.2.2.m1.1.1.cmml" xref="S4.T3.8.2.2.2.m1.1.1"><plus id="S4.T3.8.2.2.2.m1.1.1.1.cmml" xref="S4.T3.8.2.2.2.m1.1.1.1"></plus><apply id="S4.T3.8.2.2.2.m1.1.1.2.cmml" xref="S4.T3.8.2.2.2.m1.1.1.2"><times id="S4.T3.8.2.2.2.m1.1.1.2.1.cmml" xref="S4.T3.8.2.2.2.m1.1.1.2.1"></times><ci id="S4.T3.8.2.2.2.m1.1.1.2.2.cmml" xref="S4.T3.8.2.2.2.m1.1.1.2.2">𝑚</ci><ci id="S4.T3.8.2.2.2.m1.1.1.2.3.cmml" xref="S4.T3.8.2.2.2.m1.1.1.2.3">𝑛</ci></apply><apply id="S4.T3.8.2.2.2.m1.1.1.3.cmml" xref="S4.T3.8.2.2.2.m1.1.1.3"><times id="S4.T3.8.2.2.2.m1.1.1.3.1.cmml" xref="S4.T3.8.2.2.2.m1.1.1.3.1"></times><ci id="S4.T3.8.2.2.2.m1.1.1.3.2.cmml" xref="S4.T3.8.2.2.2.m1.1.1.3.2">𝑚</ci><ci id="S4.T3.8.2.2.2.m1.1.1.3.3.cmml" xref="S4.T3.8.2.2.2.m1.1.1.3.3">𝑟</ci></apply><apply id="S4.T3.8.2.2.2.m1.1.1.4.cmml" xref="S4.T3.8.2.2.2.m1.1.1.4"><times id="S4.T3.8.2.2.2.m1.1.1.4.1.cmml" xref="S4.T3.8.2.2.2.m1.1.1.4.1"></times><ci id="S4.T3.8.2.2.2.m1.1.1.4.2.cmml" xref="S4.T3.8.2.2.2.m1.1.1.4.2">𝑛</ci><ci id="S4.T3.8.2.2.2.m1.1.1.4.3.cmml" xref="S4.T3.8.2.2.2.m1.1.1.4.3">𝑟</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.8.2.2.2.m1.1c">mn+mr+nr</annotation></semantics></math></td>
<td id="S4.T3.9.3.3.3" class="ltx_td ltx_align_center ltx_border_t"><math id="S4.T3.9.3.3.3.m1.1" class="ltx_Math" alttext="mn+mr+nr" display="inline"><semantics id="S4.T3.9.3.3.3.m1.1a"><mrow id="S4.T3.9.3.3.3.m1.1.1" xref="S4.T3.9.3.3.3.m1.1.1.cmml"><mrow id="S4.T3.9.3.3.3.m1.1.1.2" xref="S4.T3.9.3.3.3.m1.1.1.2.cmml"><mi mathsize="90%" id="S4.T3.9.3.3.3.m1.1.1.2.2" xref="S4.T3.9.3.3.3.m1.1.1.2.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="S4.T3.9.3.3.3.m1.1.1.2.1" xref="S4.T3.9.3.3.3.m1.1.1.2.1.cmml">​</mo><mi mathsize="90%" id="S4.T3.9.3.3.3.m1.1.1.2.3" xref="S4.T3.9.3.3.3.m1.1.1.2.3.cmml">n</mi></mrow><mo mathsize="90%" id="S4.T3.9.3.3.3.m1.1.1.1" xref="S4.T3.9.3.3.3.m1.1.1.1.cmml">+</mo><mrow id="S4.T3.9.3.3.3.m1.1.1.3" xref="S4.T3.9.3.3.3.m1.1.1.3.cmml"><mi mathsize="90%" id="S4.T3.9.3.3.3.m1.1.1.3.2" xref="S4.T3.9.3.3.3.m1.1.1.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="S4.T3.9.3.3.3.m1.1.1.3.1" xref="S4.T3.9.3.3.3.m1.1.1.3.1.cmml">​</mo><mi mathsize="90%" id="S4.T3.9.3.3.3.m1.1.1.3.3" xref="S4.T3.9.3.3.3.m1.1.1.3.3.cmml">r</mi></mrow><mo mathsize="90%" id="S4.T3.9.3.3.3.m1.1.1.1a" xref="S4.T3.9.3.3.3.m1.1.1.1.cmml">+</mo><mrow id="S4.T3.9.3.3.3.m1.1.1.4" xref="S4.T3.9.3.3.3.m1.1.1.4.cmml"><mi mathsize="90%" id="S4.T3.9.3.3.3.m1.1.1.4.2" xref="S4.T3.9.3.3.3.m1.1.1.4.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="S4.T3.9.3.3.3.m1.1.1.4.1" xref="S4.T3.9.3.3.3.m1.1.1.4.1.cmml">​</mo><mi mathsize="90%" id="S4.T3.9.3.3.3.m1.1.1.4.3" xref="S4.T3.9.3.3.3.m1.1.1.4.3.cmml">r</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T3.9.3.3.3.m1.1b"><apply id="S4.T3.9.3.3.3.m1.1.1.cmml" xref="S4.T3.9.3.3.3.m1.1.1"><plus id="S4.T3.9.3.3.3.m1.1.1.1.cmml" xref="S4.T3.9.3.3.3.m1.1.1.1"></plus><apply id="S4.T3.9.3.3.3.m1.1.1.2.cmml" xref="S4.T3.9.3.3.3.m1.1.1.2"><times id="S4.T3.9.3.3.3.m1.1.1.2.1.cmml" xref="S4.T3.9.3.3.3.m1.1.1.2.1"></times><ci id="S4.T3.9.3.3.3.m1.1.1.2.2.cmml" xref="S4.T3.9.3.3.3.m1.1.1.2.2">𝑚</ci><ci id="S4.T3.9.3.3.3.m1.1.1.2.3.cmml" xref="S4.T3.9.3.3.3.m1.1.1.2.3">𝑛</ci></apply><apply id="S4.T3.9.3.3.3.m1.1.1.3.cmml" xref="S4.T3.9.3.3.3.m1.1.1.3"><times id="S4.T3.9.3.3.3.m1.1.1.3.1.cmml" xref="S4.T3.9.3.3.3.m1.1.1.3.1"></times><ci id="S4.T3.9.3.3.3.m1.1.1.3.2.cmml" xref="S4.T3.9.3.3.3.m1.1.1.3.2">𝑚</ci><ci id="S4.T3.9.3.3.3.m1.1.1.3.3.cmml" xref="S4.T3.9.3.3.3.m1.1.1.3.3">𝑟</ci></apply><apply id="S4.T3.9.3.3.3.m1.1.1.4.cmml" xref="S4.T3.9.3.3.3.m1.1.1.4"><times id="S4.T3.9.3.3.3.m1.1.1.4.1.cmml" xref="S4.T3.9.3.3.3.m1.1.1.4.1"></times><ci id="S4.T3.9.3.3.3.m1.1.1.4.2.cmml" xref="S4.T3.9.3.3.3.m1.1.1.4.2">𝑛</ci><ci id="S4.T3.9.3.3.3.m1.1.1.4.3.cmml" xref="S4.T3.9.3.3.3.m1.1.1.4.3">𝑟</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.9.3.3.3.m1.1c">mn+mr+nr</annotation></semantics></math></td>
</tr>
<tr id="S4.T3.12.6.6" class="ltx_tr">
<td id="S4.T3.12.6.6.4" class="ltx_td ltx_align_left"><span id="S4.T3.12.6.6.4.1" class="ltx_text" style="font-size:90%;">Optimizer States</span></td>
<td id="S4.T3.10.4.4.1" class="ltx_td ltx_align_center"><math id="S4.T3.10.4.4.1.m1.1" class="ltx_Math" alttext="mr+2nr" display="inline"><semantics id="S4.T3.10.4.4.1.m1.1a"><mrow id="S4.T3.10.4.4.1.m1.1.1" xref="S4.T3.10.4.4.1.m1.1.1.cmml"><mrow id="S4.T3.10.4.4.1.m1.1.1.2" xref="S4.T3.10.4.4.1.m1.1.1.2.cmml"><mi mathsize="90%" id="S4.T3.10.4.4.1.m1.1.1.2.2" xref="S4.T3.10.4.4.1.m1.1.1.2.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="S4.T3.10.4.4.1.m1.1.1.2.1" xref="S4.T3.10.4.4.1.m1.1.1.2.1.cmml">​</mo><mi mathsize="90%" id="S4.T3.10.4.4.1.m1.1.1.2.3" xref="S4.T3.10.4.4.1.m1.1.1.2.3.cmml">r</mi></mrow><mo mathsize="90%" id="S4.T3.10.4.4.1.m1.1.1.1" xref="S4.T3.10.4.4.1.m1.1.1.1.cmml">+</mo><mrow id="S4.T3.10.4.4.1.m1.1.1.3" xref="S4.T3.10.4.4.1.m1.1.1.3.cmml"><mn mathsize="90%" id="S4.T3.10.4.4.1.m1.1.1.3.2" xref="S4.T3.10.4.4.1.m1.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S4.T3.10.4.4.1.m1.1.1.3.1" xref="S4.T3.10.4.4.1.m1.1.1.3.1.cmml">​</mo><mi mathsize="90%" id="S4.T3.10.4.4.1.m1.1.1.3.3" xref="S4.T3.10.4.4.1.m1.1.1.3.3.cmml">n</mi><mo lspace="0em" rspace="0em" id="S4.T3.10.4.4.1.m1.1.1.3.1a" xref="S4.T3.10.4.4.1.m1.1.1.3.1.cmml">​</mo><mi mathsize="90%" id="S4.T3.10.4.4.1.m1.1.1.3.4" xref="S4.T3.10.4.4.1.m1.1.1.3.4.cmml">r</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T3.10.4.4.1.m1.1b"><apply id="S4.T3.10.4.4.1.m1.1.1.cmml" xref="S4.T3.10.4.4.1.m1.1.1"><plus id="S4.T3.10.4.4.1.m1.1.1.1.cmml" xref="S4.T3.10.4.4.1.m1.1.1.1"></plus><apply id="S4.T3.10.4.4.1.m1.1.1.2.cmml" xref="S4.T3.10.4.4.1.m1.1.1.2"><times id="S4.T3.10.4.4.1.m1.1.1.2.1.cmml" xref="S4.T3.10.4.4.1.m1.1.1.2.1"></times><ci id="S4.T3.10.4.4.1.m1.1.1.2.2.cmml" xref="S4.T3.10.4.4.1.m1.1.1.2.2">𝑚</ci><ci id="S4.T3.10.4.4.1.m1.1.1.2.3.cmml" xref="S4.T3.10.4.4.1.m1.1.1.2.3">𝑟</ci></apply><apply id="S4.T3.10.4.4.1.m1.1.1.3.cmml" xref="S4.T3.10.4.4.1.m1.1.1.3"><times id="S4.T3.10.4.4.1.m1.1.1.3.1.cmml" xref="S4.T3.10.4.4.1.m1.1.1.3.1"></times><cn type="integer" id="S4.T3.10.4.4.1.m1.1.1.3.2.cmml" xref="S4.T3.10.4.4.1.m1.1.1.3.2">2</cn><ci id="S4.T3.10.4.4.1.m1.1.1.3.3.cmml" xref="S4.T3.10.4.4.1.m1.1.1.3.3">𝑛</ci><ci id="S4.T3.10.4.4.1.m1.1.1.3.4.cmml" xref="S4.T3.10.4.4.1.m1.1.1.3.4">𝑟</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.10.4.4.1.m1.1c">mr+2nr</annotation></semantics></math></td>
<td id="S4.T3.11.5.5.2" class="ltx_td ltx_align_center"><math id="S4.T3.11.5.5.2.m1.1" class="ltx_Math" alttext="2mr+2nr" display="inline"><semantics id="S4.T3.11.5.5.2.m1.1a"><mrow id="S4.T3.11.5.5.2.m1.1.1" xref="S4.T3.11.5.5.2.m1.1.1.cmml"><mrow id="S4.T3.11.5.5.2.m1.1.1.2" xref="S4.T3.11.5.5.2.m1.1.1.2.cmml"><mn mathsize="90%" id="S4.T3.11.5.5.2.m1.1.1.2.2" xref="S4.T3.11.5.5.2.m1.1.1.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S4.T3.11.5.5.2.m1.1.1.2.1" xref="S4.T3.11.5.5.2.m1.1.1.2.1.cmml">​</mo><mi mathsize="90%" id="S4.T3.11.5.5.2.m1.1.1.2.3" xref="S4.T3.11.5.5.2.m1.1.1.2.3.cmml">m</mi><mo lspace="0em" rspace="0em" id="S4.T3.11.5.5.2.m1.1.1.2.1a" xref="S4.T3.11.5.5.2.m1.1.1.2.1.cmml">​</mo><mi mathsize="90%" id="S4.T3.11.5.5.2.m1.1.1.2.4" xref="S4.T3.11.5.5.2.m1.1.1.2.4.cmml">r</mi></mrow><mo mathsize="90%" id="S4.T3.11.5.5.2.m1.1.1.1" xref="S4.T3.11.5.5.2.m1.1.1.1.cmml">+</mo><mrow id="S4.T3.11.5.5.2.m1.1.1.3" xref="S4.T3.11.5.5.2.m1.1.1.3.cmml"><mn mathsize="90%" id="S4.T3.11.5.5.2.m1.1.1.3.2" xref="S4.T3.11.5.5.2.m1.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S4.T3.11.5.5.2.m1.1.1.3.1" xref="S4.T3.11.5.5.2.m1.1.1.3.1.cmml">​</mo><mi mathsize="90%" id="S4.T3.11.5.5.2.m1.1.1.3.3" xref="S4.T3.11.5.5.2.m1.1.1.3.3.cmml">n</mi><mo lspace="0em" rspace="0em" id="S4.T3.11.5.5.2.m1.1.1.3.1a" xref="S4.T3.11.5.5.2.m1.1.1.3.1.cmml">​</mo><mi mathsize="90%" id="S4.T3.11.5.5.2.m1.1.1.3.4" xref="S4.T3.11.5.5.2.m1.1.1.3.4.cmml">r</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T3.11.5.5.2.m1.1b"><apply id="S4.T3.11.5.5.2.m1.1.1.cmml" xref="S4.T3.11.5.5.2.m1.1.1"><plus id="S4.T3.11.5.5.2.m1.1.1.1.cmml" xref="S4.T3.11.5.5.2.m1.1.1.1"></plus><apply id="S4.T3.11.5.5.2.m1.1.1.2.cmml" xref="S4.T3.11.5.5.2.m1.1.1.2"><times id="S4.T3.11.5.5.2.m1.1.1.2.1.cmml" xref="S4.T3.11.5.5.2.m1.1.1.2.1"></times><cn type="integer" id="S4.T3.11.5.5.2.m1.1.1.2.2.cmml" xref="S4.T3.11.5.5.2.m1.1.1.2.2">2</cn><ci id="S4.T3.11.5.5.2.m1.1.1.2.3.cmml" xref="S4.T3.11.5.5.2.m1.1.1.2.3">𝑚</ci><ci id="S4.T3.11.5.5.2.m1.1.1.2.4.cmml" xref="S4.T3.11.5.5.2.m1.1.1.2.4">𝑟</ci></apply><apply id="S4.T3.11.5.5.2.m1.1.1.3.cmml" xref="S4.T3.11.5.5.2.m1.1.1.3"><times id="S4.T3.11.5.5.2.m1.1.1.3.1.cmml" xref="S4.T3.11.5.5.2.m1.1.1.3.1"></times><cn type="integer" id="S4.T3.11.5.5.2.m1.1.1.3.2.cmml" xref="S4.T3.11.5.5.2.m1.1.1.3.2">2</cn><ci id="S4.T3.11.5.5.2.m1.1.1.3.3.cmml" xref="S4.T3.11.5.5.2.m1.1.1.3.3">𝑛</ci><ci id="S4.T3.11.5.5.2.m1.1.1.3.4.cmml" xref="S4.T3.11.5.5.2.m1.1.1.3.4">𝑟</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.11.5.5.2.m1.1c">2mr+2nr</annotation></semantics></math></td>
<td id="S4.T3.12.6.6.3" class="ltx_td ltx_align_center"><math id="S4.T3.12.6.6.3.m1.1" class="ltx_Math" alttext="2nr" display="inline"><semantics id="S4.T3.12.6.6.3.m1.1a"><mrow id="S4.T3.12.6.6.3.m1.1.1" xref="S4.T3.12.6.6.3.m1.1.1.cmml"><mn mathsize="90%" id="S4.T3.12.6.6.3.m1.1.1.2" xref="S4.T3.12.6.6.3.m1.1.1.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S4.T3.12.6.6.3.m1.1.1.1" xref="S4.T3.12.6.6.3.m1.1.1.1.cmml">​</mo><mi mathsize="90%" id="S4.T3.12.6.6.3.m1.1.1.3" xref="S4.T3.12.6.6.3.m1.1.1.3.cmml">n</mi><mo lspace="0em" rspace="0em" id="S4.T3.12.6.6.3.m1.1.1.1a" xref="S4.T3.12.6.6.3.m1.1.1.1.cmml">​</mo><mi mathsize="90%" id="S4.T3.12.6.6.3.m1.1.1.4" xref="S4.T3.12.6.6.3.m1.1.1.4.cmml">r</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.T3.12.6.6.3.m1.1b"><apply id="S4.T3.12.6.6.3.m1.1.1.cmml" xref="S4.T3.12.6.6.3.m1.1.1"><times id="S4.T3.12.6.6.3.m1.1.1.1.cmml" xref="S4.T3.12.6.6.3.m1.1.1.1"></times><cn type="integer" id="S4.T3.12.6.6.3.m1.1.1.2.cmml" xref="S4.T3.12.6.6.3.m1.1.1.2">2</cn><ci id="S4.T3.12.6.6.3.m1.1.1.3.cmml" xref="S4.T3.12.6.6.3.m1.1.1.3">𝑛</ci><ci id="S4.T3.12.6.6.3.m1.1.1.4.cmml" xref="S4.T3.12.6.6.3.m1.1.1.4">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.12.6.6.3.m1.1c">2nr</annotation></semantics></math></td>
</tr>
<tr id="S4.T3.15.9.9" class="ltx_tr">
<td id="S4.T3.15.9.9.4" class="ltx_td ltx_align_left"><span id="S4.T3.15.9.9.4.1" class="ltx_text" style="font-size:90%;">Gradients</span></td>
<td id="S4.T3.13.7.7.1" class="ltx_td ltx_align_center"><math id="S4.T3.13.7.7.1.m1.1" class="ltx_Math" alttext="mn" display="inline"><semantics id="S4.T3.13.7.7.1.m1.1a"><mrow id="S4.T3.13.7.7.1.m1.1.1" xref="S4.T3.13.7.7.1.m1.1.1.cmml"><mi mathsize="90%" id="S4.T3.13.7.7.1.m1.1.1.2" xref="S4.T3.13.7.7.1.m1.1.1.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="S4.T3.13.7.7.1.m1.1.1.1" xref="S4.T3.13.7.7.1.m1.1.1.1.cmml">​</mo><mi mathsize="90%" id="S4.T3.13.7.7.1.m1.1.1.3" xref="S4.T3.13.7.7.1.m1.1.1.3.cmml">n</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.T3.13.7.7.1.m1.1b"><apply id="S4.T3.13.7.7.1.m1.1.1.cmml" xref="S4.T3.13.7.7.1.m1.1.1"><times id="S4.T3.13.7.7.1.m1.1.1.1.cmml" xref="S4.T3.13.7.7.1.m1.1.1.1"></times><ci id="S4.T3.13.7.7.1.m1.1.1.2.cmml" xref="S4.T3.13.7.7.1.m1.1.1.2">𝑚</ci><ci id="S4.T3.13.7.7.1.m1.1.1.3.cmml" xref="S4.T3.13.7.7.1.m1.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.13.7.7.1.m1.1c">mn</annotation></semantics></math></td>
<td id="S4.T3.14.8.8.2" class="ltx_td ltx_align_center"><math id="S4.T3.14.8.8.2.m1.1" class="ltx_Math" alttext="mr+nr" display="inline"><semantics id="S4.T3.14.8.8.2.m1.1a"><mrow id="S4.T3.14.8.8.2.m1.1.1" xref="S4.T3.14.8.8.2.m1.1.1.cmml"><mrow id="S4.T3.14.8.8.2.m1.1.1.2" xref="S4.T3.14.8.8.2.m1.1.1.2.cmml"><mi mathsize="90%" id="S4.T3.14.8.8.2.m1.1.1.2.2" xref="S4.T3.14.8.8.2.m1.1.1.2.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="S4.T3.14.8.8.2.m1.1.1.2.1" xref="S4.T3.14.8.8.2.m1.1.1.2.1.cmml">​</mo><mi mathsize="90%" id="S4.T3.14.8.8.2.m1.1.1.2.3" xref="S4.T3.14.8.8.2.m1.1.1.2.3.cmml">r</mi></mrow><mo mathsize="90%" id="S4.T3.14.8.8.2.m1.1.1.1" xref="S4.T3.14.8.8.2.m1.1.1.1.cmml">+</mo><mrow id="S4.T3.14.8.8.2.m1.1.1.3" xref="S4.T3.14.8.8.2.m1.1.1.3.cmml"><mi mathsize="90%" id="S4.T3.14.8.8.2.m1.1.1.3.2" xref="S4.T3.14.8.8.2.m1.1.1.3.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="S4.T3.14.8.8.2.m1.1.1.3.1" xref="S4.T3.14.8.8.2.m1.1.1.3.1.cmml">​</mo><mi mathsize="90%" id="S4.T3.14.8.8.2.m1.1.1.3.3" xref="S4.T3.14.8.8.2.m1.1.1.3.3.cmml">r</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.T3.14.8.8.2.m1.1b"><apply id="S4.T3.14.8.8.2.m1.1.1.cmml" xref="S4.T3.14.8.8.2.m1.1.1"><plus id="S4.T3.14.8.8.2.m1.1.1.1.cmml" xref="S4.T3.14.8.8.2.m1.1.1.1"></plus><apply id="S4.T3.14.8.8.2.m1.1.1.2.cmml" xref="S4.T3.14.8.8.2.m1.1.1.2"><times id="S4.T3.14.8.8.2.m1.1.1.2.1.cmml" xref="S4.T3.14.8.8.2.m1.1.1.2.1"></times><ci id="S4.T3.14.8.8.2.m1.1.1.2.2.cmml" xref="S4.T3.14.8.8.2.m1.1.1.2.2">𝑚</ci><ci id="S4.T3.14.8.8.2.m1.1.1.2.3.cmml" xref="S4.T3.14.8.8.2.m1.1.1.2.3">𝑟</ci></apply><apply id="S4.T3.14.8.8.2.m1.1.1.3.cmml" xref="S4.T3.14.8.8.2.m1.1.1.3"><times id="S4.T3.14.8.8.2.m1.1.1.3.1.cmml" xref="S4.T3.14.8.8.2.m1.1.1.3.1"></times><ci id="S4.T3.14.8.8.2.m1.1.1.3.2.cmml" xref="S4.T3.14.8.8.2.m1.1.1.3.2">𝑛</ci><ci id="S4.T3.14.8.8.2.m1.1.1.3.3.cmml" xref="S4.T3.14.8.8.2.m1.1.1.3.3">𝑟</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.14.8.8.2.m1.1c">mr+nr</annotation></semantics></math></td>
<td id="S4.T3.15.9.9.3" class="ltx_td ltx_align_center"><math id="S4.T3.15.9.9.3.m1.1" class="ltx_Math" alttext="nr" display="inline"><semantics id="S4.T3.15.9.9.3.m1.1a"><mrow id="S4.T3.15.9.9.3.m1.1.1" xref="S4.T3.15.9.9.3.m1.1.1.cmml"><mi mathsize="90%" id="S4.T3.15.9.9.3.m1.1.1.2" xref="S4.T3.15.9.9.3.m1.1.1.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="S4.T3.15.9.9.3.m1.1.1.1" xref="S4.T3.15.9.9.3.m1.1.1.1.cmml">​</mo><mi mathsize="90%" id="S4.T3.15.9.9.3.m1.1.1.3" xref="S4.T3.15.9.9.3.m1.1.1.3.cmml">r</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.T3.15.9.9.3.m1.1b"><apply id="S4.T3.15.9.9.3.m1.1.1.cmml" xref="S4.T3.15.9.9.3.m1.1.1"><times id="S4.T3.15.9.9.3.m1.1.1.1.cmml" xref="S4.T3.15.9.9.3.m1.1.1.1"></times><ci id="S4.T3.15.9.9.3.m1.1.1.2.cmml" xref="S4.T3.15.9.9.3.m1.1.1.2">𝑛</ci><ci id="S4.T3.15.9.9.3.m1.1.1.3.cmml" xref="S4.T3.15.9.9.3.m1.1.1.3">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T3.15.9.9.3.m1.1c">nr</annotation></semantics></math></td>
</tr>
<tr id="S4.T3.15.9.11" class="ltx_tr">
<td id="S4.T3.15.9.11.1" class="ltx_td ltx_align_left"><span id="S4.T3.15.9.11.1.1" class="ltx_text" style="font-size:90%;">Pretraining</span></td>
<td id="S4.T3.15.9.11.2" class="ltx_td ltx_align_center"><span id="S4.T3.15.9.11.2.1" class="ltx_text" style="font-size:90%;">Yes</span></td>
<td id="S4.T3.15.9.11.3" class="ltx_td ltx_align_center"><span id="S4.T3.15.9.11.3.1" class="ltx_text" style="font-size:90%;">No</span></td>
<td id="S4.T3.15.9.11.4" class="ltx_td ltx_align_center"><span id="S4.T3.15.9.11.4.1" class="ltx_text" style="font-size:90%;">Yes</span></td>
</tr>
<tr id="S4.T3.15.9.12" class="ltx_tr">
<td id="S4.T3.15.9.12.1" class="ltx_td ltx_align_left"><span id="S4.T3.15.9.12.1.1" class="ltx_text" style="font-size:90%;">Fine-Tuning</span></td>
<td id="S4.T3.15.9.12.2" class="ltx_td ltx_align_center"><span id="S4.T3.15.9.12.2.1" class="ltx_text" style="font-size:90%;">Yes</span></td>
<td id="S4.T3.15.9.12.3" class="ltx_td ltx_align_center"><span id="S4.T3.15.9.12.3.1" class="ltx_text" style="font-size:90%;">Yes</span></td>
<td id="S4.T3.15.9.12.4" class="ltx_td ltx_align_center"><span id="S4.T3.15.9.12.4.1" class="ltx_text" style="font-size:90%;">Yes</span></td>
</tr>
<tr id="S4.T3.15.9.13" class="ltx_tr">
<td id="S4.T3.15.9.13.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="S4.T3.15.9.13.1.1" class="ltx_text" style="font-size:90%;">Quantizeable</span></td>
<td id="S4.T3.15.9.13.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T3.15.9.13.2.1" class="ltx_text" style="font-size:90%;">No</span></td>
<td id="S4.T3.15.9.13.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T3.15.9.13.3.1" class="ltx_text" style="font-size:90%;">Yes</span></td>
<td id="S4.T3.15.9.13.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T3.15.9.13.4.1" class="ltx_text" style="font-size:90%;">Yes</span></td>
</tr>
</table>
</figure>
<section id="S4.SS4.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Memory usage</h4>

<div id="S4.SS4.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS4.SSS0.Px1.p1.1" class="ltx_p">An overview of memory usage for GaLore, LoRA and <span id="S4.SS4.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_smallcaps">LoQT</span> is given in Table <a href="#S4.T3" title="Table 3 ‣ 4.4 Memory and Throughput ‣ 4 Experiments and Results ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. We see that <span id="S4.SS4.SSS0.Px1.p1.1.2" class="ltx_text ltx_font_smallcaps">LoQT</span> makes use of the same number of trainable parameters as LoRA for a given rank while using less memory for the optimizer states and gradients than in both LoRA and GaLore.</p>
</div>
<div id="S4.SS4.SSS0.Px1.p2" class="ltx_para">
<p id="S4.SS4.SSS0.Px1.p2.1" class="ltx_p">We compare the <span id="S4.SS4.SSS0.Px1.p2.1.1" class="ltx_text ltx_font_smallcaps">LoQT</span> to the closest in memory performance approach, GaLore, for 13B in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, and for other model-sizes in Figure <a href="#A3.F6" title="Figure 6 ‣ C.1 Memory Measurements ‣ Appendix C Rank Ablation ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>. We compare three different use cases, using the approaches directly, combining them with an 8-bit Adam optimizer <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib5" title="" class="ltx_ref">5</a>]</cite>, and using per-layer weight updates with offloading (while still using 8-bit Adam). We see from the figures that <span id="S4.SS4.SSS0.Px1.p2.1.2" class="ltx_text ltx_font_smallcaps">LoQT</span> significantly shrinks both the number of trainable parameters and optimizer states compared to GaLore.</p>
</div>
<div id="S4.SS4.SSS0.Px1.p3" class="ltx_para">
<p id="S4.SS4.SSS0.Px1.p3.1" class="ltx_p">Per-layer weight update is essential for GaLore; without it, an additional <math id="S4.SS4.SSS0.Px1.p3.1.m1.1" class="ltx_Math" alttext="\sim\!\!12" display="inline"><semantics id="S4.SS4.SSS0.Px1.p3.1.m1.1a"><mrow id="S4.SS4.SSS0.Px1.p3.1.m1.1.1" xref="S4.SS4.SSS0.Px1.p3.1.m1.1.1.cmml"><mi id="S4.SS4.SSS0.Px1.p3.1.m1.1.1.2" xref="S4.SS4.SSS0.Px1.p3.1.m1.1.1.2.cmml"></mi><mpadded width="0.726em"><mo id="S4.SS4.SSS0.Px1.p3.1.m1.1.1.1" xref="S4.SS4.SSS0.Px1.p3.1.m1.1.1.1.cmml">∼</mo></mpadded><mn id="S4.SS4.SSS0.Px1.p3.1.m1.1.1.3" xref="S4.SS4.SSS0.Px1.p3.1.m1.1.1.3.cmml">12</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.SSS0.Px1.p3.1.m1.1b"><apply id="S4.SS4.SSS0.Px1.p3.1.m1.1.1.cmml" xref="S4.SS4.SSS0.Px1.p3.1.m1.1.1"><csymbol cd="latexml" id="S4.SS4.SSS0.Px1.p3.1.m1.1.1.1.cmml" xref="S4.SS4.SSS0.Px1.p3.1.m1.1.1.1">similar-to</csymbol><csymbol cd="latexml" id="S4.SS4.SSS0.Px1.p3.1.m1.1.1.2.cmml" xref="S4.SS4.SSS0.Px1.p3.1.m1.1.1.2">absent</csymbol><cn type="integer" id="S4.SS4.SSS0.Px1.p3.1.m1.1.1.3.cmml" xref="S4.SS4.SSS0.Px1.p3.1.m1.1.1.3">12</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.SSS0.Px1.p3.1.m1.1c">\sim\!\!12</annotation></semantics></math> GB of VRAM is needed for gradients in a 7B model, making full-parameter fine-tuning impossible on a 24GB GPU. Additionally, the per-layer gradient updates may not work well with DDP (Distributed Data Parallel) and gradient accumulation. With our method, we can get a lower memory than GaLore even when they use per-layer gradient updates. When not using per-layer gradient updates, this difference becomes even bigger as seen for the 7B model in Figure <a href="#A3.F6" title="Figure 6 ‣ C.1 Memory Measurements ‣ Appendix C Rank Ablation ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.</p>
</div>
<div id="S4.SS4.SSS0.Px1.p4" class="ltx_para">
<p id="S4.SS4.SSS0.Px1.p4.2" class="ltx_p">Moreover, our method supports training 7B models without per-layer computations on 24GB GPU. This makes it possible to use multi-GPU training, a capability not possible with the current GaLore approach. Our memory advantage allows for a batch size of <math id="S4.SS4.SSS0.Px1.p4.1.m1.1" class="ltx_Math" alttext="1280" display="inline"><semantics id="S4.SS4.SSS0.Px1.p4.1.m1.1a"><mn id="S4.SS4.SSS0.Px1.p4.1.m1.1.1" xref="S4.SS4.SSS0.Px1.p4.1.m1.1.1.cmml">1280</mn><annotation-xml encoding="MathML-Content" id="S4.SS4.SSS0.Px1.p4.1.m1.1b"><cn type="integer" id="S4.SS4.SSS0.Px1.p4.1.m1.1.1.cmml" xref="S4.SS4.SSS0.Px1.p4.1.m1.1.1">1280</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.SSS0.Px1.p4.1.m1.1c">1280</annotation></semantics></math> tokens compared to GaLore’s <math id="S4.SS4.SSS0.Px1.p4.2.m2.1" class="ltx_Math" alttext="256" display="inline"><semantics id="S4.SS4.SSS0.Px1.p4.2.m2.1a"><mn id="S4.SS4.SSS0.Px1.p4.2.m2.1.1" xref="S4.SS4.SSS0.Px1.p4.2.m2.1.1.cmml">256</mn><annotation-xml encoding="MathML-Content" id="S4.SS4.SSS0.Px1.p4.2.m2.1b"><cn type="integer" id="S4.SS4.SSS0.Px1.p4.2.m2.1.1.cmml" xref="S4.SS4.SSS0.Px1.p4.2.m2.1.1">256</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.SSS0.Px1.p4.2.m2.1c">256</annotation></semantics></math> for the 7B model on the 24GB RTX3090. With per-layer gradient updates, <span id="S4.SS4.SSS0.Px1.p4.2.1" class="ltx_text ltx_font_smallcaps">LoQT</span> can train a 13B model on a single GPU, pushing the limits of hardware efficiency.</p>
</div>
</section>
<section id="S4.SS4.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Throughput</h4>

<div id="S4.SS4.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS4.SSS0.Px2.p1.1" class="ltx_p">We evaluate the throughput with a sample batch size of 16 with a total batch size of 512 using gradient accumulation, which is the largest power of 2 that fits on the GPU. We update the projection matrix <math id="S4.SS4.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S4.SS4.SSS0.Px2.p1.1.m1.1a"><mi id="S4.SS4.SSS0.Px2.p1.1.m1.1.1" xref="S4.SS4.SSS0.Px2.p1.1.m1.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S4.SS4.SSS0.Px2.p1.1.m1.1b"><ci id="S4.SS4.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S4.SS4.SSS0.Px2.p1.1.m1.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.SSS0.Px2.p1.1.m1.1c">P</annotation></semantics></math> for every 200 iterations. The per-layer gradient update algorithms apply a weight update for every mini-batch as they do not support gradient accumulation. For the evaluation, we use a 1B parameter model with rank 512. We find that <span id="S4.SS4.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_smallcaps">LoQT</span> can process 16% fewer tokens per second than only using AdamW, at 3996 tokens/s compared to 4782 tokens/s on the RTX3090.</p>
</div>
</section>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Ablations</h2>

<figure id="S5.F4" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S5.F4.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2405.16528/assets/x3.png" id="S5.F4.sf1.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="372" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S5.F4.sf1.5.2.1" class="ltx_text" style="font-size:90%;">(a)</span> </span><span id="S5.F4.sf1.2.1" class="ltx_text" style="font-size:90%;">EC: Error compensation, EF: Exp<span id="S5.F4.sf1.2.1.1" class="ltx_text ltx_markedasmath">.</span> decreasing update frequency.</span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S5.F4.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2405.16528/assets/x4.png" id="S5.F4.sf2.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="372" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S5.F4.sf2.2.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span><span id="S5.F4.sf2.3.2" class="ltx_text" style="font-size:90%;">Ablation for different update frequencies, with exponentially increasing at the bottom</span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S5.F4.4.2.1" class="ltx_text" style="font-size:90%;">Figure 4</span>: </span><span id="S5.F4.2.1" class="ltx_text" style="font-size:90%;">Ablation results for update frequency, error-compensation, quantization, model size 130m, and rank <math id="S5.F4.2.1.m1.1" class="ltx_Math" alttext="256" display="inline"><semantics id="S5.F4.2.1.m1.1b"><mn id="S5.F4.2.1.m1.1.1" xref="S5.F4.2.1.m1.1.1.cmml">256</mn><annotation-xml encoding="MathML-Content" id="S5.F4.2.1.m1.1c"><cn type="integer" id="S5.F4.2.1.m1.1.1.cmml" xref="S5.F4.2.1.m1.1.1">256</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.F4.2.1.m1.1d">256</annotation></semantics></math>.</span></figcaption>
</figure>
<section id="S5.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Quantization Error Compensation and Initialization</h4>

<div id="S5.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S5.SS0.SSS0.Px1.p1.5" class="ltx_p">To assess the impact of quantization error compensation, we analyze the validation loss curves for a 130 million parameter model. Figure <a href="#S5.F4.sf1" title="In Figure 4 ‣ 5 Ablations ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4(a)</span></a> shows that quantizing <math id="S5.SS0.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S5.SS0.SSS0.Px1.p1.1.m1.1a"><mi id="S5.SS0.SSS0.Px1.p1.1.m1.1.1" xref="S5.SS0.SSS0.Px1.p1.1.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px1.p1.1.m1.1b"><ci id="S5.SS0.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S5.SS0.SSS0.Px1.p1.1.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px1.p1.1.m1.1c">W</annotation></semantics></math> or both <math id="S5.SS0.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S5.SS0.SSS0.Px1.p1.2.m2.1a"><mi id="S5.SS0.SSS0.Px1.p1.2.m2.1.1" xref="S5.SS0.SSS0.Px1.p1.2.m2.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px1.p1.2.m2.1b"><ci id="S5.SS0.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S5.SS0.SSS0.Px1.p1.2.m2.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px1.p1.2.m2.1c">W</annotation></semantics></math> and <math id="S5.SS0.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S5.SS0.SSS0.Px1.p1.3.m3.1a"><mi id="S5.SS0.SSS0.Px1.p1.3.m3.1.1" xref="S5.SS0.SSS0.Px1.p1.3.m3.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px1.p1.3.m3.1b"><ci id="S5.SS0.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S5.SS0.SSS0.Px1.p1.3.m3.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px1.p1.3.m3.1c">P</annotation></semantics></math> without error compensation, or exponential frequency updates, causes the loss to stagnate early. We also note that quantizing <math id="S5.SS0.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S5.SS0.SSS0.Px1.p1.4.m4.1a"><mi id="S5.SS0.SSS0.Px1.p1.4.m4.1.1" xref="S5.SS0.SSS0.Px1.p1.4.m4.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px1.p1.4.m4.1b"><ci id="S5.SS0.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S5.SS0.SSS0.Px1.p1.4.m4.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px1.p1.4.m4.1c">P</annotation></semantics></math> has a much smaller effect on the loss compared to quantizing <math id="S5.SS0.SSS0.Px1.p1.5.m5.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S5.SS0.SSS0.Px1.p1.5.m5.1a"><mi id="S5.SS0.SSS0.Px1.p1.5.m5.1.1" xref="S5.SS0.SSS0.Px1.p1.5.m5.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px1.p1.5.m5.1b"><ci id="S5.SS0.SSS0.Px1.p1.5.m5.1.1.cmml" xref="S5.SS0.SSS0.Px1.p1.5.m5.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px1.p1.5.m5.1c">W</annotation></semantics></math>.
Error compensation significantly improves the model’s performance, resulting in approximately 3.5 points better perplexity. Adding exponentially increasing update frequency improves perplexity by an additional 1.5 points, achieving performance close to that of models without quantization.</p>
</div>
<div id="S5.SS0.SSS0.Px1.p2" class="ltx_para">
<p id="S5.SS0.SSS0.Px1.p2.2" class="ltx_p">Without the quantization error compensation detailed in §<a href="#S3.SS5" title="3.5 Compensating for Errors Introduced by Quantization ‣ 3 Efficient Pretraining With LoQT ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.5</span></a>, <span id="S5.SS0.SSS0.Px1.p2.2.1" class="ltx_text ltx_font_smallcaps">LoQT</span>’s performance stagnates earlier and diverges more from the other models. This demonstrates the effectiveness of our compensation approach in mitigating the quantization errors introduced during the <math id="S5.SS0.SSS0.Px1.p2.1.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S5.SS0.SSS0.Px1.p2.1.m1.1a"><mi id="S5.SS0.SSS0.Px1.p2.1.m1.1.1" xref="S5.SS0.SSS0.Px1.p2.1.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px1.p2.1.m1.1b"><ci id="S5.SS0.SSS0.Px1.p2.1.m1.1.1.cmml" xref="S5.SS0.SSS0.Px1.p2.1.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px1.p2.1.m1.1c">W</annotation></semantics></math> update with <math id="S5.SS0.SSS0.Px1.p2.2.m2.1" class="ltx_Math" alttext="AB" display="inline"><semantics id="S5.SS0.SSS0.Px1.p2.2.m2.1a"><mrow id="S5.SS0.SSS0.Px1.p2.2.m2.1.1" xref="S5.SS0.SSS0.Px1.p2.2.m2.1.1.cmml"><mi id="S5.SS0.SSS0.Px1.p2.2.m2.1.1.2" xref="S5.SS0.SSS0.Px1.p2.2.m2.1.1.2.cmml">A</mi><mo lspace="0em" rspace="0em" id="S5.SS0.SSS0.Px1.p2.2.m2.1.1.1" xref="S5.SS0.SSS0.Px1.p2.2.m2.1.1.1.cmml">​</mo><mi id="S5.SS0.SSS0.Px1.p2.2.m2.1.1.3" xref="S5.SS0.SSS0.Px1.p2.2.m2.1.1.3.cmml">B</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px1.p2.2.m2.1b"><apply id="S5.SS0.SSS0.Px1.p2.2.m2.1.1.cmml" xref="S5.SS0.SSS0.Px1.p2.2.m2.1.1"><times id="S5.SS0.SSS0.Px1.p2.2.m2.1.1.1.cmml" xref="S5.SS0.SSS0.Px1.p2.2.m2.1.1.1"></times><ci id="S5.SS0.SSS0.Px1.p2.2.m2.1.1.2.cmml" xref="S5.SS0.SSS0.Px1.p2.2.m2.1.1.2">𝐴</ci><ci id="S5.SS0.SSS0.Px1.p2.2.m2.1.1.3.cmml" xref="S5.SS0.SSS0.Px1.p2.2.m2.1.1.3">𝐵</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px1.p2.2.m2.1c">AB</annotation></semantics></math> and subsequent quantization steps.</p>
</div>
</section>
<section id="S5.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Projection update frequency</h4>

<div id="S5.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S5.SS0.SSS0.Px2.p1.7" class="ltx_p">Our scheduling approach ensures more frequent updates early in training to facilitate substantial weight adjustments. As training progresses, the update frequency decreases, allowing for the accumulation of larger updates to compensate for smaller updates that might be canceled out by quantization errors.
Figure <a href="#S5.F4.sf2" title="In Figure 4 ‣ 5 Ablations ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4(b)</span></a> presents an ablation study on our method of progressively increasing update frequency starting at <math id="S5.SS0.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="100" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.1.m1.1a"><mn id="S5.SS0.SSS0.Px2.p1.1.m1.1.1" xref="S5.SS0.SSS0.Px2.p1.1.m1.1.1.cmml">100</mn><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.1.m1.1b"><cn type="integer" id="S5.SS0.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.1.m1.1.1">100</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.1.m1.1c">100</annotation></semantics></math> and increasing by a factor of <math id="S5.SS0.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="1.2^{T}" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.2.m2.1a"><msup id="S5.SS0.SSS0.Px2.p1.2.m2.1.1" xref="S5.SS0.SSS0.Px2.p1.2.m2.1.1.cmml"><mn id="S5.SS0.SSS0.Px2.p1.2.m2.1.1.2" xref="S5.SS0.SSS0.Px2.p1.2.m2.1.1.2.cmml">1.2</mn><mi id="S5.SS0.SSS0.Px2.p1.2.m2.1.1.3" xref="S5.SS0.SSS0.Px2.p1.2.m2.1.1.3.cmml">T</mi></msup><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.2.m2.1b"><apply id="S5.SS0.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S5.SS0.SSS0.Px2.p1.2.m2.1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.2.m2.1.1">superscript</csymbol><cn type="float" id="S5.SS0.SSS0.Px2.p1.2.m2.1.1.2.cmml" xref="S5.SS0.SSS0.Px2.p1.2.m2.1.1.2">1.2</cn><ci id="S5.SS0.SSS0.Px2.p1.2.m2.1.1.3.cmml" xref="S5.SS0.SSS0.Px2.p1.2.m2.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.2.m2.1c">1.2^{T}</annotation></semantics></math> up to <math id="S5.SS0.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="2500" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.3.m3.1a"><mn id="S5.SS0.SSS0.Px2.p1.3.m3.1.1" xref="S5.SS0.SSS0.Px2.p1.3.m3.1.1.cmml">2500</mn><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.3.m3.1b"><cn type="integer" id="S5.SS0.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.3.m3.1.1">2500</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.3.m3.1c">2500</annotation></semantics></math>. We show the validation loss curves for fixed update frequencies <math id="S5.SS0.SSS0.Px2.p1.4.m4.1" class="ltx_Math" alttext="200" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.4.m4.1a"><mn id="S5.SS0.SSS0.Px2.p1.4.m4.1.1" xref="S5.SS0.SSS0.Px2.p1.4.m4.1.1.cmml">200</mn><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.4.m4.1b"><cn type="integer" id="S5.SS0.SSS0.Px2.p1.4.m4.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.4.m4.1.1">200</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.4.m4.1c">200</annotation></semantics></math>, <math id="S5.SS0.SSS0.Px2.p1.5.m5.1" class="ltx_Math" alttext="400" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.5.m5.1a"><mn id="S5.SS0.SSS0.Px2.p1.5.m5.1.1" xref="S5.SS0.SSS0.Px2.p1.5.m5.1.1.cmml">400</mn><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.5.m5.1b"><cn type="integer" id="S5.SS0.SSS0.Px2.p1.5.m5.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.5.m5.1.1">400</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.5.m5.1c">400</annotation></semantics></math>, <math id="S5.SS0.SSS0.Px2.p1.6.m6.1" class="ltx_Math" alttext="500" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.6.m6.1a"><mn id="S5.SS0.SSS0.Px2.p1.6.m6.1.1" xref="S5.SS0.SSS0.Px2.p1.6.m6.1.1.cmml">500</mn><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.6.m6.1b"><cn type="integer" id="S5.SS0.SSS0.Px2.p1.6.m6.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.6.m6.1.1">500</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.6.m6.1c">500</annotation></semantics></math>, and <math id="S5.SS0.SSS0.Px2.p1.7.m7.1" class="ltx_Math" alttext="1000" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.7.m7.1a"><mn id="S5.SS0.SSS0.Px2.p1.7.m7.1.1" xref="S5.SS0.SSS0.Px2.p1.7.m7.1.1.cmml">1000</mn><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.7.m7.1b"><cn type="integer" id="S5.SS0.SSS0.Px2.p1.7.m7.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.7.m7.1.1">1000</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.7.m7.1c">1000</annotation></semantics></math>.</p>
</div>
<div id="S5.SS0.SSS0.Px2.p2" class="ltx_para">
<p id="S5.SS0.SSS0.Px2.p2.3" class="ltx_p">The results show that exponentially increasing the update gap is particularly beneficial for models employing quantization, enabling them to achieve the same perplexity as those without quantization while making use of GaLore. Conversely, the performance gains are more subtle for models that do not use quantization and rely solely on GaLore. We hypothesize that even these models might benefit from the larger projection gap intervals. This could be due to the reduction in the accumulation of errors from frequent updates of the projection factor <math id="S5.SS0.SSS0.Px2.p2.1.m1.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S5.SS0.SSS0.Px2.p2.1.m1.1a"><mi id="S5.SS0.SSS0.Px2.p2.1.m1.1.1" xref="S5.SS0.SSS0.Px2.p2.1.m1.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p2.1.m1.1b"><ci id="S5.SS0.SSS0.Px2.p2.1.m1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p2.1.m1.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p2.1.m1.1c">P</annotation></semantics></math>, as the influence of outdated optimizer statistics becomes less prevalent.
Finally, an ablation on the ranks used for <math id="S5.SS0.SSS0.Px2.p2.2.m2.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S5.SS0.SSS0.Px2.p2.2.m2.1a"><mi id="S5.SS0.SSS0.Px2.p2.2.m2.1.1" xref="S5.SS0.SSS0.Px2.p2.2.m2.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p2.2.m2.1b"><ci id="S5.SS0.SSS0.Px2.p2.2.m2.1.1.cmml" xref="S5.SS0.SSS0.Px2.p2.2.m2.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p2.2.m2.1c">P</annotation></semantics></math> and <math id="S5.SS0.SSS0.Px2.p2.3.m3.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S5.SS0.SSS0.Px2.p2.3.m3.1a"><mi id="S5.SS0.SSS0.Px2.p2.3.m3.1.1" xref="S5.SS0.SSS0.Px2.p2.3.m3.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p2.3.m3.1b"><ci id="S5.SS0.SSS0.Px2.p2.3.m3.1.1.cmml" xref="S5.SS0.SSS0.Px2.p2.3.m3.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p2.3.m3.1c">B</annotation></semantics></math> is given in Figure <a href="#A3.F5" title="Figure 5 ‣ Appendix C Rank Ablation ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> in the Appendix.</p>
</div>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Discussion and Conclusion</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">We present <span id="S6.p1.1.1" class="ltx_text ltx_font_smallcaps">LoQT</span>, a method for memory-efficient pretraining and adaptation of quantized models. The key insights behind the approach are the benefits of initializing low-rank factors using the gradient of the weight matrix and using exponentially increasing update gaps that make updating of a quantized model possible.
While our initial goal was to lower memory usage to facilitate the training of models such as LLMs on consumer-grade hardware, we are cautiously excited about the results sometimes being better than the baselines. These evaluations will be explored in more detail in future work.</p>
</div>
<div id="S6.p2" class="ltx_para">
<p id="S6.p2.1" class="ltx_p">Our method is general and opens up new ways of decreasing memory use as well as improving the training throughput. This could be done by implementing kernel fusion and using other quantization methods such as NF2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite> or quantization of activations, making it possible to do the matrix multiplications using modern tensor core formats such as FP8 or INT4.</p>
</div>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Impact and Limitations</h2>

<div id="S7.p1" class="ltx_para">
<p id="S7.p1.1" class="ltx_p">Our work has the potential to have a significant impact on those working in hardware-constrained settings by enabling more efficient training on consumer hardware. We are particularly excited to see the method being applied in single GPU settings. We validate <span id="S7.p1.1.1" class="ltx_text ltx_font_smallcaps">LoQT</span> on several model sizes, by training over many steps and by fine-tuning on a standard benchmark for natural language understanding. While we are confident in our results, further exploration of training duration, data diversity, and hyper-parameter tuning might lead to different results in those settings.</p>
</div>
</section>
<section id="S8" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">8 </span>Acknowledgements</h2>

<div id="S8.p1" class="ltx_para">
<p id="S8.p1.1" class="ltx_p">This work is supported by the Danish Data Science Academy, which is funded by the Novo Nordisk Foundation (NNF21SA0069429) and VILLUM FONDEN (40516).
Serge Belongie and Vésteinn Snæbjarnarson are supported by the Pioneer Centre for AI, DNRF grant number P1. MJK acknowledges support from the Carlsberg Foundation and the Novo Nordisk Foundation.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[1]</span>
<span class="ltx_bibblock">
Haoli Bai, Lu Hou, Lifeng Shang, Xin Jiang, Irwin King, and Michael R. Lyu.

</span>
<span class="ltx_bibblock">Towards efficient post-training quantization of pre-trained language
models, 2021.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[2]</span>
<span class="ltx_bibblock">
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry.

</span>
<span class="ltx_bibblock">Scalable methods for 8-bit training of neural networks, 2018.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[3]</span>
<span class="ltx_bibblock">
Brian Chmiel, Ron Banner, Elad Hoffer, Hilla Ben Yaacov, and Daniel Soudry.

</span>
<span class="ltx_bibblock">Logarithmic unbiased quantization: Simple 4-bit training in deep
learning, 2022.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[4]</span>
<span class="ltx_bibblock">
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer.

</span>
<span class="ltx_bibblock">Llm.int8(): 8-bit matrix multiplication for transformers at scale,
2022.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[5]</span>
<span class="ltx_bibblock">
Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer.

</span>
<span class="ltx_bibblock">8-bit optimizers via block-wise quantization, 2022.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[6]</span>
<span class="ltx_bibblock">
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.

</span>
<span class="ltx_bibblock">Qlora: Efficient finetuning of quantized llms, 2023.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[7]</span>
<span class="ltx_bibblock">
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias
Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan
Alistarh.

</span>
<span class="ltx_bibblock">Spqr: A sparse-quantized representation for near-lossless llm weight
compression, 2023.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[8]</span>
<span class="ltx_bibblock">
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem
Babenko, and Dan Alistarh.

</span>
<span class="ltx_bibblock">Extreme compression of large language models via additive
quantization, 2024.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[9]</span>
<span class="ltx_bibblock">
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh.

</span>
<span class="ltx_bibblock">Gptq: Accurate post-training quantization for generative pre-trained
transformers, 2023.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[10]</span>
<span class="ltx_bibblock">
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt
Keutzer.

</span>
<span class="ltx_bibblock">A survey of quantization methods for efficient neural network
inference.

</span>
<span class="ltx_bibblock"><span id="bib.bib10.1.1" class="ltx_text ltx_font_italic">CoRR</span>, abs/2103.13630, 2021.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[11]</span>
<span class="ltx_bibblock">
Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer.

</span>
<span class="ltx_bibblock">Gradient descent happens in a tiny subspace, 2018.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[12]</span>
<span class="ltx_bibblock">
Soufiane Hayou, Nikhil Ghosh, and Bin Yu.

</span>
<span class="ltx_bibblock">Lora+: Efficient low rank adaptation of large models.

</span>
<span class="ltx_bibblock"><span id="bib.bib12.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2402.12354</span>, 2024.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[13]</span>
<span class="ltx_bibblock">
Pengcheng He, Jianfeng Gao, and Weizhu Chen.

</span>
<span class="ltx_bibblock">Debertav3: Improving deberta using electra-style pre-training with
gradient-disentangled embedding sharing, 2023.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[14]</span>
<span class="ltx_bibblock">
Jung Hwan Heo, Jeonghoon Kim, Beomseok Kwon, Byeongwook Kim, Se Jung Kwon, and
Dongsoo Lee.

</span>
<span class="ltx_bibblock">Rethinking channel dimensions to isolate outliers for low-bit weight
quantization of large language models, 2024.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[15]</span>
<span class="ltx_bibblock">
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor
Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes
Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den
Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich
Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.

</span>
<span class="ltx_bibblock">Training compute-optimal large language models, 2022.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[16]</span>
<span class="ltx_bibblock">
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean
Wang, Lu Wang, and Weizhu Chen.

</span>
<span class="ltx_bibblock">Lora: Low-rank adaptation of large language models, 2021.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[17]</span>
<span class="ltx_bibblock">
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Youngjun Kwak, Jae-Joon
Han, Sung Ju Hwang, and Changkyu Choi.

</span>
<span class="ltx_bibblock">Learning to quantize deep networks by optimizing quantization
intervals with task loss, 2018.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[18]</span>
<span class="ltx_bibblock">
Diederik P. Kingma and Jimmy Ba.

</span>
<span class="ltx_bibblock">Adam: A method for stochastic optimization, 2017.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[19]</span>
<span class="ltx_bibblock">
Brett W. Larsen, Stanislav Fort, Nic Becker, and Surya Ganguli.

</span>
<span class="ltx_bibblock">How many degrees of freedom do we need to train deep networks: a loss
landscape perspective, 2022.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[20]</span>
<span class="ltx_bibblock">
Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lee.

</span>
<span class="ltx_bibblock">FlexRound: Learnable rounding based on element-wise division for
post-training quantization.

</span>
<span class="ltx_bibblock">In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt,
Sivan Sabato, and Jonathan Scarlett, editors, <span id="bib.bib20.1.1" class="ltx_text ltx_font_italic">Proceedings of the 40th
International Conference on Machine Learning</span>, volume 202 of <span id="bib.bib20.2.2" class="ltx_text ltx_font_italic">Proceedings
of Machine Learning Research</span>, pages 18913–18939. PMLR, 23–29 Jul 2023.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[21]</span>
<span class="ltx_bibblock">
Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu
Chen, and Tuo Zhao.

</span>
<span class="ltx_bibblock">Loftq: Lora-fine-tuning-aware quantization for large language models,
2023.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[22]</span>
<span class="ltx_bibblock">
Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky.

</span>
<span class="ltx_bibblock">Relora: High-rank training through low-rank updates, 2023.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[23]</span>
<span class="ltx_bibblock">
Baohao Liao and Christof Monz.

</span>
<span class="ltx_bibblock">Apiq: Finetuning of 2-bit quantized large language model, 2024.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[24]</span>
<span class="ltx_bibblock">
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar
Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra.

</span>
<span class="ltx_bibblock">Llm-qat: Data-free quantization aware training for large language
models.

</span>
<span class="ltx_bibblock"><span id="bib.bib24.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2305.17888</span>, 2023.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[25]</span>
<span class="ltx_bibblock">
Kai Lv, Yuqing Yang, Tengxiao Liu, Qinghui Gao, Qipeng Guo, and Xipeng Qiu.

</span>
<span class="ltx_bibblock">Full parameter fine-tuning for large language models with limited
resources, 2023.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[26]</span>
<span class="ltx_bibblock">
Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang,
Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei.

</span>
<span class="ltx_bibblock">The era of 1-bit llms: All large language models are in 1.58 bits,
2024.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[27]</span>
<span class="ltx_bibblock">
Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok
Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee.

</span>
<span class="ltx_bibblock">Lut-gemm: Quantized matrix multiplication based on luts for efficient
inference in large-scale generative language models, 2024.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[28]</span>
<span class="ltx_bibblock">
Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang, Ze Liu, Yifan
Xiong, Ziyue Yang, Bolin Ni, Jingcheng Hu, Ruihang Li, Miaosen Zhang, Chen
Li, Jia Ning, Ruizhe Wang, Zheng Zhang, Shuguang Liu, Joe Chau, Han Hu, and
Peng Cheng.

</span>
<span class="ltx_bibblock">Fp8-lm: Training fp8 large language models, 2023.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[29]</span>
<span class="ltx_bibblock">
Sergio P. Perez, Yan Zhang, James Briggs, Charlie Blake, Josh Levy-Kramer, Paul
Balanca, Carlo Luschi, Stephen Barlow, and Andrew William Fitzgibbon.

</span>
<span class="ltx_bibblock">Training and inference of large language models using 8-bit floating
point, 2023.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[30]</span>
<span class="ltx_bibblock">
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael
Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.

</span>
<span class="ltx_bibblock">Exploring the limits of transfer learning with a unified text-to-text
transformer.

</span>
<span class="ltx_bibblock"><span id="bib.bib30.1.1" class="ltx_text ltx_font_italic">arXiv e-prints</span>, 2019.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[31]</span>
<span class="ltx_bibblock">
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li,
Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo.

</span>
<span class="ltx_bibblock">Omniquant: Omnidirectionally calibrated quantization for large
language models, 2024.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[32]</span>
<span class="ltx_bibblock">
Noam Shazeer and Mitchell Stern.

</span>
<span class="ltx_bibblock">Adafactor: Adaptive learning rates with sublinear memory cost, 2018.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[33]</span>
<span class="ltx_bibblock">
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami,
Michael W. Mahoney, and Kurt Keutzer.

</span>
<span class="ltx_bibblock">Q-bert: Hessian based ultra low precision quantization of bert, 2019.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[34]</span>
<span class="ltx_bibblock">
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine
Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale,
Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem
Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,
Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar
Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa,
Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux,
Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier
Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew
Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan
Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang,
Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan
Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien
Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.

</span>
<span class="ltx_bibblock">Llama 2: Open foundation and fine-tuned chat models, 2023.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[35]</span>
<span class="ltx_bibblock">
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De
Sa.

</span>
<span class="ltx_bibblock">Quip : Even better llm quantization with hadamard incoherence and
lattice codebooks, 2024.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[36]</span>
<span class="ltx_bibblock">
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel
Bowman.

</span>
<span class="ltx_bibblock">GLUE: A multi-task benchmark and analysis platform for natural
language understanding.

</span>
<span class="ltx_bibblock">In Tal Linzen, Grzegorz Chrupała, and Afra Alishahi, editors, <span id="bib.bib36.1.1" class="ltx_text ltx_font_italic">Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and
Interpreting Neural Networks for NLP</span>, pages 353–355, Brussels, Belgium,
November 2018. Association for Computational Linguistics.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[37]</span>
<span class="ltx_bibblock">
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and
Samuel R. Bowman.

</span>
<span class="ltx_bibblock">Glue: A multi-task benchmark and analysis platform for natural
language understanding, 2019.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[38]</span>
<span class="ltx_bibblock">
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan
Yang, Ruiping Wang, Yi Wu, and Furu Wei.

</span>
<span class="ltx_bibblock">Bitnet: Scaling 1-bit transformers for large language models, 2023.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[39]</span>
<span class="ltx_bibblock">
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, and Kailash
Gopalakrishnan.

</span>
<span class="ltx_bibblock">Training deep neural networks with 8-bit floating point numbers,
2018.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[40]</span>
<span class="ltx_bibblock">
Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and
Ludwig Schmidt.

</span>
<span class="ltx_bibblock">Stable and low-precision training for large-scale vision-language
models, 2023.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[41]</span>
<span class="ltx_bibblock">
Haocheng Xi, Yuxiang Chen, Kang Zhao, Kaijun Zheng, Jianfei Chen, and Jun Zhu.

</span>
<span class="ltx_bibblock">Jetfire: Efficient and accurate transformer pretraining with int8
data flow and per-block quantization, 2024.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[42]</span>
<span class="ltx_bibblock">
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han.

</span>
<span class="ltx_bibblock">Smoothquant: Accurate and efficient post-training quantization for
large language models, 2024.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[43]</span>
<span class="ltx_bibblock">
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat.

</span>
<span class="ltx_bibblock">Q8bert: Quantized 8bit bert.

</span>
<span class="ltx_bibblock">In <span id="bib.bib43.1.1" class="ltx_text ltx_font_italic">2019 Fifth Workshop on Energy Efficient Machine Learning and
Cognitive Computing - NeurIPS Edition (EMC2-NIPS)</span>. IEEE, December 2019.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">[44]</span>
<span class="ltx_bibblock">
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and
Yuandong Tian.

</span>
<span class="ltx_bibblock">Galore: Memory-efficient llm training by gradient low-rank
projection, 2024.

</span>
</li>
</ul>
</section>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Quantization Methods</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.1" class="ltx_p">Quantization methods can be broadly categorized into Quantization-Aware Training (QAT), Post-Training Quantization (PTQ), and Fully Quantized Training (FQT).</p>
</div>
<section id="A1.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Quantization-Aware Training (QAT)</h4>

<div id="A1.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="A1.SS0.SSS0.Px1.p1.1" class="ltx_p">QAT  <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib24" title="" class="ltx_ref">24</a>, <a href="#bib.bib17" title="" class="ltx_ref">17</a>, <a href="#bib.bib8" title="" class="ltx_ref">8</a>, <a href="#bib.bib38" title="" class="ltx_ref">38</a>, <a href="#bib.bib26" title="" class="ltx_ref">26</a>]</cite> integrates quantization in the training process by emulating inference time quantization where the model weights are quantized. By maintaining high precision gradients and optimizer states, QAT allows the model to adapt to quantized weights while preserving accuracy.
These methods predominantly focus on weight-only quantization approaches, which involve converting weight matrices into low-precision formats and then upcasting them just before computation <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib38" title="" class="ltx_ref">38</a>, <a href="#bib.bib26" title="" class="ltx_ref">26</a>]</cite>. This allows the main computation to occur at high precision, effectively preserving model accuracy while significantly compressing the model <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib9" title="" class="ltx_ref">9</a>]</cite>. However, QAT can require significant computational resources due to the need for full precision gradient calculations and large optimization states <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite>.</p>
</div>
</section>
<section id="A1.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Post-Training Quantization (PTQ)</h4>

<div id="A1.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="A1.SS0.SSS0.Px2.p1.1" class="ltx_p">PQT <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib9" title="" class="ltx_ref">9</a>, <a href="#bib.bib7" title="" class="ltx_ref">7</a>, <a href="#bib.bib35" title="" class="ltx_ref">35</a>, <a href="#bib.bib42" title="" class="ltx_ref">42</a>, <a href="#bib.bib27" title="" class="ltx_ref">27</a>, <a href="#bib.bib4" title="" class="ltx_ref">4</a>, <a href="#bib.bib20" title="" class="ltx_ref">20</a>, <a href="#bib.bib14" title="" class="ltx_ref">14</a>, <a href="#bib.bib31" title="" class="ltx_ref">31</a>]</cite> involves converting a pre-trained high-precision model into a lower precision format. This can be done directly or by using a subset of the training data to calibrate the quantization process or fine-tune the quantized weights to adapt the model to the quantization. However, PTQ often results in reduced accuracy compared to QAT because the model does not learn to adjust to quantized weights during training <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib9" title="" class="ltx_ref">9</a>, <a href="#bib.bib42" title="" class="ltx_ref">42</a>]</cite>.</p>
</div>
</section>
<section id="A1.SS0.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Fully Quantized Training (FQT)</h4>

<div id="A1.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="A1.SS0.SSS0.Px3.p1.1" class="ltx_p">FQT aims to minimize memory and accelerate the training process by quantizing both forward and backward passes <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib39" title="" class="ltx_ref">39</a>, <a href="#bib.bib3" title="" class="ltx_ref">3</a>, <a href="#bib.bib2" title="" class="ltx_ref">2</a>, <a href="#bib.bib29" title="" class="ltx_ref">29</a>, <a href="#bib.bib40" title="" class="ltx_ref">40</a>]</cite>. These methods often require specialized hardware <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib28" title="" class="ltx_ref">28</a>, <a href="#bib.bib41" title="" class="ltx_ref">41</a>]</cite> but are promising for efficient training, and current approaches cannot maintain accuracy <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib41" title="" class="ltx_ref">41</a>]</cite>.</p>
</div>
<div id="A1.SS0.SSS0.Px3.p2" class="ltx_para">
<p id="A1.SS0.SSS0.Px3.p2.10" class="ltx_p"><span id="A1.SS0.SSS0.Px3.p2.10.1" class="ltx_text ltx_font_smallcaps">LoQT</span> is a form of QAT that gets close to FQT. As we perform a variant of LoRA (see §<a href="#S2.SS2" title="2.2 Adaptation of Pretrained Networks ‣ 2 Related Work and Background ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.2</span></a>), we factor the layers <math id="A1.SS0.SSS0.Px3.p2.1.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="A1.SS0.SSS0.Px3.p2.1.m1.1a"><mi id="A1.SS0.SSS0.Px3.p2.1.m1.1.1" xref="A1.SS0.SSS0.Px3.p2.1.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="A1.SS0.SSS0.Px3.p2.1.m1.1b"><ci id="A1.SS0.SSS0.Px3.p2.1.m1.1.1.cmml" xref="A1.SS0.SSS0.Px3.p2.1.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS0.SSS0.Px3.p2.1.m1.1c">W</annotation></semantics></math> into two matrices <math id="A1.SS0.SSS0.Px3.p2.2.m2.1" class="ltx_Math" alttext="P" display="inline"><semantics id="A1.SS0.SSS0.Px3.p2.2.m2.1a"><mi id="A1.SS0.SSS0.Px3.p2.2.m2.1.1" xref="A1.SS0.SSS0.Px3.p2.2.m2.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="A1.SS0.SSS0.Px3.p2.2.m2.1b"><ci id="A1.SS0.SSS0.Px3.p2.2.m2.1.1.cmml" xref="A1.SS0.SSS0.Px3.p2.2.m2.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS0.SSS0.Px3.p2.2.m2.1c">P</annotation></semantics></math> and <math id="A1.SS0.SSS0.Px3.p2.3.m3.1" class="ltx_Math" alttext="B" display="inline"><semantics id="A1.SS0.SSS0.Px3.p2.3.m3.1a"><mi id="A1.SS0.SSS0.Px3.p2.3.m3.1.1" xref="A1.SS0.SSS0.Px3.p2.3.m3.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="A1.SS0.SSS0.Px3.p2.3.m3.1b"><ci id="A1.SS0.SSS0.Px3.p2.3.m3.1.1.cmml" xref="A1.SS0.SSS0.Px3.p2.3.m3.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS0.SSS0.Px3.p2.3.m3.1c">B</annotation></semantics></math>. We quantize the <math id="A1.SS0.SSS0.Px3.p2.4.m4.1" class="ltx_Math" alttext="W" display="inline"><semantics id="A1.SS0.SSS0.Px3.p2.4.m4.1a"><mi id="A1.SS0.SSS0.Px3.p2.4.m4.1.1" xref="A1.SS0.SSS0.Px3.p2.4.m4.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="A1.SS0.SSS0.Px3.p2.4.m4.1b"><ci id="A1.SS0.SSS0.Px3.p2.4.m4.1.1.cmml" xref="A1.SS0.SSS0.Px3.p2.4.m4.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS0.SSS0.Px3.p2.4.m4.1c">W</annotation></semantics></math> and <math id="A1.SS0.SSS0.Px3.p2.5.m5.1" class="ltx_Math" alttext="P" display="inline"><semantics id="A1.SS0.SSS0.Px3.p2.5.m5.1a"><mi id="A1.SS0.SSS0.Px3.p2.5.m5.1.1" xref="A1.SS0.SSS0.Px3.p2.5.m5.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="A1.SS0.SSS0.Px3.p2.5.m5.1b"><ci id="A1.SS0.SSS0.Px3.p2.5.m5.1.1.cmml" xref="A1.SS0.SSS0.Px3.p2.5.m5.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS0.SSS0.Px3.p2.5.m5.1c">P</annotation></semantics></math> with NF4, but keep <math id="A1.SS0.SSS0.Px3.p2.6.m6.1" class="ltx_Math" alttext="B" display="inline"><semantics id="A1.SS0.SSS0.Px3.p2.6.m6.1a"><mi id="A1.SS0.SSS0.Px3.p2.6.m6.1.1" xref="A1.SS0.SSS0.Px3.p2.6.m6.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="A1.SS0.SSS0.Px3.p2.6.m6.1b"><ci id="A1.SS0.SSS0.Px3.p2.6.m6.1.1.cmml" xref="A1.SS0.SSS0.Px3.p2.6.m6.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS0.SSS0.Px3.p2.6.m6.1c">B</annotation></semantics></math> in 16-bit precision. We periodically update the <math id="A1.SS0.SSS0.Px3.p2.7.m7.1" class="ltx_Math" alttext="W" display="inline"><semantics id="A1.SS0.SSS0.Px3.p2.7.m7.1a"><mi id="A1.SS0.SSS0.Px3.p2.7.m7.1.1" xref="A1.SS0.SSS0.Px3.p2.7.m7.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="A1.SS0.SSS0.Px3.p2.7.m7.1b"><ci id="A1.SS0.SSS0.Px3.p2.7.m7.1.1.cmml" xref="A1.SS0.SSS0.Px3.p2.7.m7.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS0.SSS0.Px3.p2.7.m7.1c">W</annotation></semantics></math> matrices using the product of the fixed <math id="A1.SS0.SSS0.Px3.p2.8.m8.1" class="ltx_Math" alttext="P" display="inline"><semantics id="A1.SS0.SSS0.Px3.p2.8.m8.1a"><mi id="A1.SS0.SSS0.Px3.p2.8.m8.1.1" xref="A1.SS0.SSS0.Px3.p2.8.m8.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="A1.SS0.SSS0.Px3.p2.8.m8.1b"><ci id="A1.SS0.SSS0.Px3.p2.8.m8.1.1.cmml" xref="A1.SS0.SSS0.Px3.p2.8.m8.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS0.SSS0.Px3.p2.8.m8.1c">P</annotation></semantics></math> and the updated <math id="A1.SS0.SSS0.Px3.p2.9.m9.1" class="ltx_Math" alttext="B" display="inline"><semantics id="A1.SS0.SSS0.Px3.p2.9.m9.1a"><mi id="A1.SS0.SSS0.Px3.p2.9.m9.1.1" xref="A1.SS0.SSS0.Px3.p2.9.m9.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="A1.SS0.SSS0.Px3.p2.9.m9.1b"><ci id="A1.SS0.SSS0.Px3.p2.9.m9.1.1.cmml" xref="A1.SS0.SSS0.Px3.p2.9.m9.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS0.SSS0.Px3.p2.9.m9.1c">B</annotation></semantics></math>s without ever dequantizing it all at once, only layerwise when merging in <math id="A1.SS0.SSS0.Px3.p2.10.m10.1" class="ltx_Math" alttext="PB" display="inline"><semantics id="A1.SS0.SSS0.Px3.p2.10.m10.1a"><mrow id="A1.SS0.SSS0.Px3.p2.10.m10.1.1" xref="A1.SS0.SSS0.Px3.p2.10.m10.1.1.cmml"><mi id="A1.SS0.SSS0.Px3.p2.10.m10.1.1.2" xref="A1.SS0.SSS0.Px3.p2.10.m10.1.1.2.cmml">P</mi><mo lspace="0em" rspace="0em" id="A1.SS0.SSS0.Px3.p2.10.m10.1.1.1" xref="A1.SS0.SSS0.Px3.p2.10.m10.1.1.1.cmml">​</mo><mi id="A1.SS0.SSS0.Px3.p2.10.m10.1.1.3" xref="A1.SS0.SSS0.Px3.p2.10.m10.1.1.3.cmml">B</mi></mrow><annotation-xml encoding="MathML-Content" id="A1.SS0.SSS0.Px3.p2.10.m10.1b"><apply id="A1.SS0.SSS0.Px3.p2.10.m10.1.1.cmml" xref="A1.SS0.SSS0.Px3.p2.10.m10.1.1"><times id="A1.SS0.SSS0.Px3.p2.10.m10.1.1.1.cmml" xref="A1.SS0.SSS0.Px3.p2.10.m10.1.1.1"></times><ci id="A1.SS0.SSS0.Px3.p2.10.m10.1.1.2.cmml" xref="A1.SS0.SSS0.Px3.p2.10.m10.1.1.2">𝑃</ci><ci id="A1.SS0.SSS0.Px3.p2.10.m10.1.1.3.cmml" xref="A1.SS0.SSS0.Px3.p2.10.m10.1.1.3">𝐵</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS0.SSS0.Px3.p2.10.m10.1c">PB</annotation></semantics></math>.
This approach retains the benefits of reduced memory usage while minimizing accuracy loss, focusing high-precision updates on a low-rank representation, and allowing efficient model updates without the overhead of full matrix re-quantization.</p>
</div>
</section>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Hyperparamters</h2>

<section id="A2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.1 </span>Pre-training</h3>

<div id="A2.SS1.p1" class="ltx_para">
<p id="A2.SS1.p1.1" class="ltx_p">For the pre-training results shown in Table <a href="#S4.T1" title="Table 1 ‣ 4 Experiments and Results ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, we adopted configurations from GaLore <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite> and tested pre-training methods on different LLaMA 2 model sizes using the C4 dataset. Training was conducted with optimizer states in BF16 precision, and NF4 precision quantization was used for <span id="A2.SS1.p1.1.1" class="ltx_text ltx_font_smallcaps">LoQT</span>. The model rank was adapted based on the largest layer with specific parameters.</p>
</div>
<div id="A2.SS1.p2" class="ltx_para">
<p id="A2.SS1.p2.1" class="ltx_p">Table <a href="#S4.T1" title="Table 1 ‣ 4 Experiments and Results ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> shows the ratio <math id="A2.SS1.p2.1.m1.1" class="ltx_Math" alttext="r/d_{model}" display="inline"><semantics id="A2.SS1.p2.1.m1.1a"><mrow id="A2.SS1.p2.1.m1.1.1" xref="A2.SS1.p2.1.m1.1.1.cmml"><mi id="A2.SS1.p2.1.m1.1.1.2" xref="A2.SS1.p2.1.m1.1.1.2.cmml">r</mi><mo id="A2.SS1.p2.1.m1.1.1.1" xref="A2.SS1.p2.1.m1.1.1.1.cmml">/</mo><msub id="A2.SS1.p2.1.m1.1.1.3" xref="A2.SS1.p2.1.m1.1.1.3.cmml"><mi id="A2.SS1.p2.1.m1.1.1.3.2" xref="A2.SS1.p2.1.m1.1.1.3.2.cmml">d</mi><mrow id="A2.SS1.p2.1.m1.1.1.3.3" xref="A2.SS1.p2.1.m1.1.1.3.3.cmml"><mi id="A2.SS1.p2.1.m1.1.1.3.3.2" xref="A2.SS1.p2.1.m1.1.1.3.3.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p2.1.m1.1.1.3.3.1" xref="A2.SS1.p2.1.m1.1.1.3.3.1.cmml">​</mo><mi id="A2.SS1.p2.1.m1.1.1.3.3.3" xref="A2.SS1.p2.1.m1.1.1.3.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p2.1.m1.1.1.3.3.1a" xref="A2.SS1.p2.1.m1.1.1.3.3.1.cmml">​</mo><mi id="A2.SS1.p2.1.m1.1.1.3.3.4" xref="A2.SS1.p2.1.m1.1.1.3.3.4.cmml">d</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p2.1.m1.1.1.3.3.1b" xref="A2.SS1.p2.1.m1.1.1.3.3.1.cmml">​</mo><mi id="A2.SS1.p2.1.m1.1.1.3.3.5" xref="A2.SS1.p2.1.m1.1.1.3.3.5.cmml">e</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p2.1.m1.1.1.3.3.1c" xref="A2.SS1.p2.1.m1.1.1.3.3.1.cmml">​</mo><mi id="A2.SS1.p2.1.m1.1.1.3.3.6" xref="A2.SS1.p2.1.m1.1.1.3.3.6.cmml">l</mi></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="A2.SS1.p2.1.m1.1b"><apply id="A2.SS1.p2.1.m1.1.1.cmml" xref="A2.SS1.p2.1.m1.1.1"><divide id="A2.SS1.p2.1.m1.1.1.1.cmml" xref="A2.SS1.p2.1.m1.1.1.1"></divide><ci id="A2.SS1.p2.1.m1.1.1.2.cmml" xref="A2.SS1.p2.1.m1.1.1.2">𝑟</ci><apply id="A2.SS1.p2.1.m1.1.1.3.cmml" xref="A2.SS1.p2.1.m1.1.1.3"><csymbol cd="ambiguous" id="A2.SS1.p2.1.m1.1.1.3.1.cmml" xref="A2.SS1.p2.1.m1.1.1.3">subscript</csymbol><ci id="A2.SS1.p2.1.m1.1.1.3.2.cmml" xref="A2.SS1.p2.1.m1.1.1.3.2">𝑑</ci><apply id="A2.SS1.p2.1.m1.1.1.3.3.cmml" xref="A2.SS1.p2.1.m1.1.1.3.3"><times id="A2.SS1.p2.1.m1.1.1.3.3.1.cmml" xref="A2.SS1.p2.1.m1.1.1.3.3.1"></times><ci id="A2.SS1.p2.1.m1.1.1.3.3.2.cmml" xref="A2.SS1.p2.1.m1.1.1.3.3.2">𝑚</ci><ci id="A2.SS1.p2.1.m1.1.1.3.3.3.cmml" xref="A2.SS1.p2.1.m1.1.1.3.3.3">𝑜</ci><ci id="A2.SS1.p2.1.m1.1.1.3.3.4.cmml" xref="A2.SS1.p2.1.m1.1.1.3.3.4">𝑑</ci><ci id="A2.SS1.p2.1.m1.1.1.3.3.5.cmml" xref="A2.SS1.p2.1.m1.1.1.3.3.5">𝑒</ci><ci id="A2.SS1.p2.1.m1.1.1.3.3.6.cmml" xref="A2.SS1.p2.1.m1.1.1.3.3.6">𝑙</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p2.1.m1.1c">r/d_{model}</annotation></semantics></math>, which denotes the rank relative to the largest weight matrix dimension. All experiments used a maximum sequence length of 256, learning rate warmup for the first 10% of training steps, and cosine annealing for the learning rate schedule, decaying to 10% of the initial rate.</p>
</div>
<div id="A2.SS1.p3" class="ltx_para">
<p id="A2.SS1.p3.3" class="ltx_p">Galore, <span id="A2.SS1.p3.3.1" class="ltx_text ltx_font_smallcaps">LoQT</span>-nq, and <span id="A2.SS1.p3.3.2" class="ltx_text ltx_font_smallcaps">LoQT</span> used exponentially increasing update frequencies starting at 100 and increasing by <math id="A2.SS1.p3.1.m1.1" class="ltx_Math" alttext="100+\psi^{i}" display="inline"><semantics id="A2.SS1.p3.1.m1.1a"><mrow id="A2.SS1.p3.1.m1.1.1" xref="A2.SS1.p3.1.m1.1.1.cmml"><mn id="A2.SS1.p3.1.m1.1.1.2" xref="A2.SS1.p3.1.m1.1.1.2.cmml">100</mn><mo id="A2.SS1.p3.1.m1.1.1.1" xref="A2.SS1.p3.1.m1.1.1.1.cmml">+</mo><msup id="A2.SS1.p3.1.m1.1.1.3" xref="A2.SS1.p3.1.m1.1.1.3.cmml"><mi id="A2.SS1.p3.1.m1.1.1.3.2" xref="A2.SS1.p3.1.m1.1.1.3.2.cmml">ψ</mi><mi id="A2.SS1.p3.1.m1.1.1.3.3" xref="A2.SS1.p3.1.m1.1.1.3.3.cmml">i</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="A2.SS1.p3.1.m1.1b"><apply id="A2.SS1.p3.1.m1.1.1.cmml" xref="A2.SS1.p3.1.m1.1.1"><plus id="A2.SS1.p3.1.m1.1.1.1.cmml" xref="A2.SS1.p3.1.m1.1.1.1"></plus><cn type="integer" id="A2.SS1.p3.1.m1.1.1.2.cmml" xref="A2.SS1.p3.1.m1.1.1.2">100</cn><apply id="A2.SS1.p3.1.m1.1.1.3.cmml" xref="A2.SS1.p3.1.m1.1.1.3"><csymbol cd="ambiguous" id="A2.SS1.p3.1.m1.1.1.3.1.cmml" xref="A2.SS1.p3.1.m1.1.1.3">superscript</csymbol><ci id="A2.SS1.p3.1.m1.1.1.3.2.cmml" xref="A2.SS1.p3.1.m1.1.1.3.2">𝜓</ci><ci id="A2.SS1.p3.1.m1.1.1.3.3.cmml" xref="A2.SS1.p3.1.m1.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p3.1.m1.1c">100+\psi^{i}</annotation></semantics></math>, where <math id="A2.SS1.p3.2.m2.1" class="ltx_Math" alttext="\psi" display="inline"><semantics id="A2.SS1.p3.2.m2.1a"><mi id="A2.SS1.p3.2.m2.1.1" xref="A2.SS1.p3.2.m2.1.1.cmml">ψ</mi><annotation-xml encoding="MathML-Content" id="A2.SS1.p3.2.m2.1b"><ci id="A2.SS1.p3.2.m2.1.1.cmml" xref="A2.SS1.p3.2.m2.1.1">𝜓</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p3.2.m2.1c">\psi</annotation></semantics></math> is 1.2 and <math id="A2.SS1.p3.3.m3.1" class="ltx_Math" alttext="i" display="inline"><semantics id="A2.SS1.p3.3.m3.1a"><mi id="A2.SS1.p3.3.m3.1.1" xref="A2.SS1.p3.3.m3.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="A2.SS1.p3.3.m3.1b"><ci id="A2.SS1.p3.3.m3.1.1.cmml" xref="A2.SS1.p3.3.m3.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p3.3.m3.1c">i</annotation></semantics></math> is the update counter (see Section <a href="#A3.SS1" title="C.1 Memory Measurements ‣ Appendix C Rank Ablation ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C.1</span></a> for more details).</p>
</div>
<div id="A2.SS1.p4" class="ltx_para">
<p id="A2.SS1.p4.14" class="ltx_p">We tested learning rates of <math id="A2.SS1.p4.1.m1.1" class="ltx_Math" alttext="0.01" display="inline"><semantics id="A2.SS1.p4.1.m1.1a"><mn id="A2.SS1.p4.1.m1.1.1" xref="A2.SS1.p4.1.m1.1.1.cmml">0.01</mn><annotation-xml encoding="MathML-Content" id="A2.SS1.p4.1.m1.1b"><cn type="float" id="A2.SS1.p4.1.m1.1.1.cmml" xref="A2.SS1.p4.1.m1.1.1">0.01</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p4.1.m1.1c">0.01</annotation></semantics></math>, <math id="A2.SS1.p4.2.m2.1" class="ltx_Math" alttext="0.005" display="inline"><semantics id="A2.SS1.p4.2.m2.1a"><mn id="A2.SS1.p4.2.m2.1.1" xref="A2.SS1.p4.2.m2.1.1.cmml">0.005</mn><annotation-xml encoding="MathML-Content" id="A2.SS1.p4.2.m2.1b"><cn type="float" id="A2.SS1.p4.2.m2.1.1.cmml" xref="A2.SS1.p4.2.m2.1.1">0.005</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p4.2.m2.1c">0.005</annotation></semantics></math>, <math id="A2.SS1.p4.3.m3.1" class="ltx_Math" alttext="0.001" display="inline"><semantics id="A2.SS1.p4.3.m3.1a"><mn id="A2.SS1.p4.3.m3.1.1" xref="A2.SS1.p4.3.m3.1.1.cmml">0.001</mn><annotation-xml encoding="MathML-Content" id="A2.SS1.p4.3.m3.1b"><cn type="float" id="A2.SS1.p4.3.m3.1.1.cmml" xref="A2.SS1.p4.3.m3.1.1">0.001</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p4.3.m3.1c">0.001</annotation></semantics></math>, and <math id="A2.SS1.p4.4.m4.1" class="ltx_Math" alttext="0.0005" display="inline"><semantics id="A2.SS1.p4.4.m4.1a"><mn id="A2.SS1.p4.4.m4.1.1" xref="A2.SS1.p4.4.m4.1.1.cmml">0.0005</mn><annotation-xml encoding="MathML-Content" id="A2.SS1.p4.4.m4.1b"><cn type="float" id="A2.SS1.p4.4.m4.1.1.cmml" xref="A2.SS1.p4.4.m4.1.1">0.0005</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p4.4.m4.1c">0.0005</annotation></semantics></math> across different model sizes. For models ranging from 60M to 350M parameters, a learning rate of <math id="A2.SS1.p4.5.m5.1" class="ltx_Math" alttext="0.01" display="inline"><semantics id="A2.SS1.p4.5.m5.1a"><mn id="A2.SS1.p4.5.m5.1.1" xref="A2.SS1.p4.5.m5.1.1.cmml">0.01</mn><annotation-xml encoding="MathML-Content" id="A2.SS1.p4.5.m5.1b"><cn type="float" id="A2.SS1.p4.5.m5.1.1.cmml" xref="A2.SS1.p4.5.m5.1.1">0.01</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p4.5.m5.1c">0.01</annotation></semantics></math> yielded the best performance. In contrast, full-rank models required smaller learning rates: <math id="A2.SS1.p4.6.m6.1" class="ltx_Math" alttext="0.001" display="inline"><semantics id="A2.SS1.p4.6.m6.1a"><mn id="A2.SS1.p4.6.m6.1.1" xref="A2.SS1.p4.6.m6.1.1.cmml">0.001</mn><annotation-xml encoding="MathML-Content" id="A2.SS1.p4.6.m6.1b"><cn type="float" id="A2.SS1.p4.6.m6.1.1.cmml" xref="A2.SS1.p4.6.m6.1.1">0.001</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p4.6.m6.1c">0.001</annotation></semantics></math> for 60M to 350M models and <math id="A2.SS1.p4.7.m7.1" class="ltx_Math" alttext="0.0005" display="inline"><semantics id="A2.SS1.p4.7.m7.1a"><mn id="A2.SS1.p4.7.m7.1.1" xref="A2.SS1.p4.7.m7.1.1.cmml">0.0005</mn><annotation-xml encoding="MathML-Content" id="A2.SS1.p4.7.m7.1b"><cn type="float" id="A2.SS1.p4.7.m7.1.1.cmml" xref="A2.SS1.p4.7.m7.1.1">0.0005</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p4.7.m7.1c">0.0005</annotation></semantics></math> for the 1B model.
To scale the learning rates for <span id="A2.SS1.p4.14.1" class="ltx_text ltx_font_smallcaps">LoQT</span>, <span id="A2.SS1.p4.14.2" class="ltx_text ltx_font_smallcaps">LoQT</span>-nq, and Galore, we employed a scale parameter <math id="A2.SS1.p4.8.m8.1" class="ltx_Math" alttext="s" display="inline"><semantics id="A2.SS1.p4.8.m8.1a"><mi id="A2.SS1.p4.8.m8.1.1" xref="A2.SS1.p4.8.m8.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="A2.SS1.p4.8.m8.1b"><ci id="A2.SS1.p4.8.m8.1.1.cmml" xref="A2.SS1.p4.8.m8.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p4.8.m8.1c">s</annotation></semantics></math> set to <math id="A2.SS1.p4.9.m9.1" class="ltx_Math" alttext="0.5" display="inline"><semantics id="A2.SS1.p4.9.m9.1a"><mn id="A2.SS1.p4.9.m9.1.1" xref="A2.SS1.p4.9.m9.1.1.cmml">0.5</mn><annotation-xml encoding="MathML-Content" id="A2.SS1.p4.9.m9.1b"><cn type="float" id="A2.SS1.p4.9.m9.1.1.cmml" xref="A2.SS1.p4.9.m9.1.1">0.5</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p4.9.m9.1c">0.5</annotation></semantics></math> and <math id="A2.SS1.p4.10.m10.1" class="ltx_Math" alttext="0.25" display="inline"><semantics id="A2.SS1.p4.10.m10.1a"><mn id="A2.SS1.p4.10.m10.1.1" xref="A2.SS1.p4.10.m10.1.1.cmml">0.25</mn><annotation-xml encoding="MathML-Content" id="A2.SS1.p4.10.m10.1b"><cn type="float" id="A2.SS1.p4.10.m10.1.1.cmml" xref="A2.SS1.p4.10.m10.1.1">0.25</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p4.10.m10.1c">0.25</annotation></semantics></math> for the 1B model. This parameter functions similarly to the LoRA alpha parameter, determining the weight on the learned factors for <span id="A2.SS1.p4.14.3" class="ltx_text ltx_font_smallcaps">LoQT</span> and <span id="A2.SS1.p4.14.4" class="ltx_text ltx_font_smallcaps">LoQT</span>-nq. For Galore, our experiments indicated that <math id="A2.SS1.p4.11.m11.1" class="ltx_Math" alttext="s=0.5" display="inline"><semantics id="A2.SS1.p4.11.m11.1a"><mrow id="A2.SS1.p4.11.m11.1.1" xref="A2.SS1.p4.11.m11.1.1.cmml"><mi id="A2.SS1.p4.11.m11.1.1.2" xref="A2.SS1.p4.11.m11.1.1.2.cmml">s</mi><mo id="A2.SS1.p4.11.m11.1.1.1" xref="A2.SS1.p4.11.m11.1.1.1.cmml">=</mo><mn id="A2.SS1.p4.11.m11.1.1.3" xref="A2.SS1.p4.11.m11.1.1.3.cmml">0.5</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.SS1.p4.11.m11.1b"><apply id="A2.SS1.p4.11.m11.1.1.cmml" xref="A2.SS1.p4.11.m11.1.1"><eq id="A2.SS1.p4.11.m11.1.1.1.cmml" xref="A2.SS1.p4.11.m11.1.1.1"></eq><ci id="A2.SS1.p4.11.m11.1.1.2.cmml" xref="A2.SS1.p4.11.m11.1.1.2">𝑠</ci><cn type="float" id="A2.SS1.p4.11.m11.1.1.3.cmml" xref="A2.SS1.p4.11.m11.1.1.3">0.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p4.11.m11.1c">s=0.5</annotation></semantics></math> was more effective than the <math id="A2.SS1.p4.12.m12.1" class="ltx_Math" alttext="0.25" display="inline"><semantics id="A2.SS1.p4.12.m12.1a"><mn id="A2.SS1.p4.12.m12.1.1" xref="A2.SS1.p4.12.m12.1.1.cmml">0.25</mn><annotation-xml encoding="MathML-Content" id="A2.SS1.p4.12.m12.1b"><cn type="float" id="A2.SS1.p4.12.m12.1.1.cmml" xref="A2.SS1.p4.12.m12.1.1">0.25</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p4.12.m12.1c">0.25</annotation></semantics></math> reported in <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite>. This scaling approach effectively adjusts the learning rate, resulting in an actual rate of <math id="A2.SS1.p4.13.m13.1" class="ltx_Math" alttext="0.005" display="inline"><semantics id="A2.SS1.p4.13.m13.1a"><mn id="A2.SS1.p4.13.m13.1.1" xref="A2.SS1.p4.13.m13.1.1.cmml">0.005</mn><annotation-xml encoding="MathML-Content" id="A2.SS1.p4.13.m13.1b"><cn type="float" id="A2.SS1.p4.13.m13.1.1.cmml" xref="A2.SS1.p4.13.m13.1.1">0.005</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p4.13.m13.1c">0.005</annotation></semantics></math> for the multi-head attention and feed-forward layers in LLaMA models, which is relatively large compared to the <math id="A2.SS1.p4.14.m14.1" class="ltx_Math" alttext="0.001" display="inline"><semantics id="A2.SS1.p4.14.m14.1a"><mn id="A2.SS1.p4.14.m14.1.1" xref="A2.SS1.p4.14.m14.1.1.cmml">0.001</mn><annotation-xml encoding="MathML-Content" id="A2.SS1.p4.14.m14.1b"><cn type="float" id="A2.SS1.p4.14.m14.1.1.cmml" xref="A2.SS1.p4.14.m14.1.1">0.001</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p4.14.m14.1c">0.001</annotation></semantics></math> used for full-rank models. Higher learning rates led to spikes in the training loss for both full-rank and <span id="A2.SS1.p4.14.5" class="ltx_text ltx_font_smallcaps">LoQT</span> models.</p>
</div>
<figure id="A2.T4" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="A2.T4.6.1.1" class="ltx_text" style="font-size:90%;">Table 4</span>: </span><span id="A2.T4.7.2" class="ltx_text" style="font-size:90%;">Pre-training hyperparameters of LLaMA models for evaluation. (-) Indicates we did not train such a model.</span></figcaption>
<br class="ltx_break">
<table id="A2.T4.4.4" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A2.T4.4.4.5" class="ltx_tr">
<td id="A2.T4.4.4.5.1" class="ltx_td ltx_align_center ltx_border_tt">Model Size</td>
<td id="A2.T4.4.4.5.2" class="ltx_td ltx_align_center ltx_border_tt">Hidden/Intermediate</td>
<td id="A2.T4.4.4.5.3" class="ltx_td ltx_align_center ltx_border_tt">Attention Heads</td>
<td id="A2.T4.4.4.5.4" class="ltx_td ltx_align_center ltx_border_tt">Layers</td>
<td id="A2.T4.4.4.5.5" class="ltx_td ltx_align_center ltx_border_tt">Steps</td>
<td id="A2.T4.4.4.5.6" class="ltx_td ltx_align_center ltx_border_tt">Data Amount</td>
<td id="A2.T4.4.4.5.7" class="ltx_td ltx_align_center ltx_border_tt">Rank</td>
</tr>
<tr id="A2.T4.1.1.1" class="ltx_tr">
<td id="A2.T4.1.1.1.2" class="ltx_td ltx_align_center ltx_border_t">60M</td>
<td id="A2.T4.1.1.1.3" class="ltx_td ltx_align_center ltx_border_t">512 / 1376</td>
<td id="A2.T4.1.1.1.4" class="ltx_td ltx_align_center ltx_border_t">8</td>
<td id="A2.T4.1.1.1.5" class="ltx_td ltx_align_center ltx_border_t">8</td>
<td id="A2.T4.1.1.1.6" class="ltx_td ltx_align_center ltx_border_t">10K</td>
<td id="A2.T4.1.1.1.1" class="ltx_td ltx_align_center ltx_border_t"><math id="A2.T4.1.1.1.1.m1.1" class="ltx_Math" alttext="1.3\mathrm{B}" display="inline"><semantics id="A2.T4.1.1.1.1.m1.1a"><mrow id="A2.T4.1.1.1.1.m1.1.1" xref="A2.T4.1.1.1.1.m1.1.1.cmml"><mn id="A2.T4.1.1.1.1.m1.1.1.2" xref="A2.T4.1.1.1.1.m1.1.1.2.cmml">1.3</mn><mo lspace="0em" rspace="0em" id="A2.T4.1.1.1.1.m1.1.1.1" xref="A2.T4.1.1.1.1.m1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="A2.T4.1.1.1.1.m1.1.1.3" xref="A2.T4.1.1.1.1.m1.1.1.3.cmml">B</mi></mrow><annotation-xml encoding="MathML-Content" id="A2.T4.1.1.1.1.m1.1b"><apply id="A2.T4.1.1.1.1.m1.1.1.cmml" xref="A2.T4.1.1.1.1.m1.1.1"><times id="A2.T4.1.1.1.1.m1.1.1.1.cmml" xref="A2.T4.1.1.1.1.m1.1.1.1"></times><cn type="float" id="A2.T4.1.1.1.1.m1.1.1.2.cmml" xref="A2.T4.1.1.1.1.m1.1.1.2">1.3</cn><ci id="A2.T4.1.1.1.1.m1.1.1.3.cmml" xref="A2.T4.1.1.1.1.m1.1.1.3">B</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T4.1.1.1.1.m1.1c">1.3\mathrm{B}</annotation></semantics></math></td>
<td id="A2.T4.1.1.1.7" class="ltx_td ltx_align_center ltx_border_t">128</td>
</tr>
<tr id="A2.T4.2.2.2" class="ltx_tr">
<td id="A2.T4.2.2.2.2" class="ltx_td ltx_align_center">130M</td>
<td id="A2.T4.2.2.2.3" class="ltx_td ltx_align_center">768 / 2048</td>
<td id="A2.T4.2.2.2.4" class="ltx_td ltx_align_center">12</td>
<td id="A2.T4.2.2.2.5" class="ltx_td ltx_align_center">12</td>
<td id="A2.T4.2.2.2.6" class="ltx_td ltx_align_center">20K</td>
<td id="A2.T4.2.2.2.1" class="ltx_td ltx_align_center"><math id="A2.T4.2.2.2.1.m1.1" class="ltx_Math" alttext="2.6\mathrm{B}" display="inline"><semantics id="A2.T4.2.2.2.1.m1.1a"><mrow id="A2.T4.2.2.2.1.m1.1.1" xref="A2.T4.2.2.2.1.m1.1.1.cmml"><mn id="A2.T4.2.2.2.1.m1.1.1.2" xref="A2.T4.2.2.2.1.m1.1.1.2.cmml">2.6</mn><mo lspace="0em" rspace="0em" id="A2.T4.2.2.2.1.m1.1.1.1" xref="A2.T4.2.2.2.1.m1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="A2.T4.2.2.2.1.m1.1.1.3" xref="A2.T4.2.2.2.1.m1.1.1.3.cmml">B</mi></mrow><annotation-xml encoding="MathML-Content" id="A2.T4.2.2.2.1.m1.1b"><apply id="A2.T4.2.2.2.1.m1.1.1.cmml" xref="A2.T4.2.2.2.1.m1.1.1"><times id="A2.T4.2.2.2.1.m1.1.1.1.cmml" xref="A2.T4.2.2.2.1.m1.1.1.1"></times><cn type="float" id="A2.T4.2.2.2.1.m1.1.1.2.cmml" xref="A2.T4.2.2.2.1.m1.1.1.2">2.6</cn><ci id="A2.T4.2.2.2.1.m1.1.1.3.cmml" xref="A2.T4.2.2.2.1.m1.1.1.3">B</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T4.2.2.2.1.m1.1c">2.6\mathrm{B}</annotation></semantics></math></td>
<td id="A2.T4.2.2.2.7" class="ltx_td ltx_align_center">256</td>
</tr>
<tr id="A2.T4.3.3.3" class="ltx_tr">
<td id="A2.T4.3.3.3.2" class="ltx_td ltx_align_center">350M</td>
<td id="A2.T4.3.3.3.3" class="ltx_td ltx_align_center">1024 / 2736</td>
<td id="A2.T4.3.3.3.4" class="ltx_td ltx_align_center">16</td>
<td id="A2.T4.3.3.3.5" class="ltx_td ltx_align_center">24</td>
<td id="A2.T4.3.3.3.6" class="ltx_td ltx_align_center">60K</td>
<td id="A2.T4.3.3.3.1" class="ltx_td ltx_align_center"><math id="A2.T4.3.3.3.1.m1.1" class="ltx_Math" alttext="7.8\mathrm{B}" display="inline"><semantics id="A2.T4.3.3.3.1.m1.1a"><mrow id="A2.T4.3.3.3.1.m1.1.1" xref="A2.T4.3.3.3.1.m1.1.1.cmml"><mn id="A2.T4.3.3.3.1.m1.1.1.2" xref="A2.T4.3.3.3.1.m1.1.1.2.cmml">7.8</mn><mo lspace="0em" rspace="0em" id="A2.T4.3.3.3.1.m1.1.1.1" xref="A2.T4.3.3.3.1.m1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="A2.T4.3.3.3.1.m1.1.1.3" xref="A2.T4.3.3.3.1.m1.1.1.3.cmml">B</mi></mrow><annotation-xml encoding="MathML-Content" id="A2.T4.3.3.3.1.m1.1b"><apply id="A2.T4.3.3.3.1.m1.1.1.cmml" xref="A2.T4.3.3.3.1.m1.1.1"><times id="A2.T4.3.3.3.1.m1.1.1.1.cmml" xref="A2.T4.3.3.3.1.m1.1.1.1"></times><cn type="float" id="A2.T4.3.3.3.1.m1.1.1.2.cmml" xref="A2.T4.3.3.3.1.m1.1.1.2">7.8</cn><ci id="A2.T4.3.3.3.1.m1.1.1.3.cmml" xref="A2.T4.3.3.3.1.m1.1.1.3">B</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T4.3.3.3.1.m1.1c">7.8\mathrm{B}</annotation></semantics></math></td>
<td id="A2.T4.3.3.3.7" class="ltx_td ltx_align_center">256</td>
</tr>
<tr id="A2.T4.4.4.4" class="ltx_tr">
<td id="A2.T4.4.4.4.2" class="ltx_td ltx_align_center">1B</td>
<td id="A2.T4.4.4.4.3" class="ltx_td ltx_align_center">2048 / 5461</td>
<td id="A2.T4.4.4.4.4" class="ltx_td ltx_align_center">24</td>
<td id="A2.T4.4.4.4.5" class="ltx_td ltx_align_center">32</td>
<td id="A2.T4.4.4.4.6" class="ltx_td ltx_align_center">100K</td>
<td id="A2.T4.4.4.4.1" class="ltx_td ltx_align_center"><math id="A2.T4.4.4.4.1.m1.1" class="ltx_Math" alttext="13.1\mathrm{B}" display="inline"><semantics id="A2.T4.4.4.4.1.m1.1a"><mrow id="A2.T4.4.4.4.1.m1.1.1" xref="A2.T4.4.4.4.1.m1.1.1.cmml"><mn id="A2.T4.4.4.4.1.m1.1.1.2" xref="A2.T4.4.4.4.1.m1.1.1.2.cmml">13.1</mn><mo lspace="0em" rspace="0em" id="A2.T4.4.4.4.1.m1.1.1.1" xref="A2.T4.4.4.4.1.m1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="A2.T4.4.4.4.1.m1.1.1.3" xref="A2.T4.4.4.4.1.m1.1.1.3.cmml">B</mi></mrow><annotation-xml encoding="MathML-Content" id="A2.T4.4.4.4.1.m1.1b"><apply id="A2.T4.4.4.4.1.m1.1.1.cmml" xref="A2.T4.4.4.4.1.m1.1.1"><times id="A2.T4.4.4.4.1.m1.1.1.1.cmml" xref="A2.T4.4.4.4.1.m1.1.1.1"></times><cn type="float" id="A2.T4.4.4.4.1.m1.1.1.2.cmml" xref="A2.T4.4.4.4.1.m1.1.1.2">13.1</cn><ci id="A2.T4.4.4.4.1.m1.1.1.3.cmml" xref="A2.T4.4.4.4.1.m1.1.1.3">B</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T4.4.4.4.1.m1.1c">13.1\mathrm{B}</annotation></semantics></math></td>
<td id="A2.T4.4.4.4.7" class="ltx_td ltx_align_center">512</td>
</tr>
<tr id="A2.T4.4.4.6" class="ltx_tr">
<td id="A2.T4.4.4.6.1" class="ltx_td ltx_align_center">7B</td>
<td id="A2.T4.4.4.6.2" class="ltx_td ltx_align_center">4096/11008</td>
<td id="A2.T4.4.4.6.3" class="ltx_td ltx_align_center">32</td>
<td id="A2.T4.4.4.6.4" class="ltx_td ltx_align_center">32</td>
<td id="A2.T4.4.4.6.5" class="ltx_td ltx_align_center">-</td>
<td id="A2.T4.4.4.6.6" class="ltx_td ltx_align_center">-</td>
<td id="A2.T4.4.4.6.7" class="ltx_td ltx_align_center">1024</td>
</tr>
<tr id="A2.T4.4.4.7" class="ltx_tr">
<td id="A2.T4.4.4.7.1" class="ltx_td ltx_align_center ltx_border_bb">13B</td>
<td id="A2.T4.4.4.7.2" class="ltx_td ltx_align_center ltx_border_bb">5120/13824</td>
<td id="A2.T4.4.4.7.3" class="ltx_td ltx_align_center ltx_border_bb">40</td>
<td id="A2.T4.4.4.7.4" class="ltx_td ltx_align_center ltx_border_bb">40</td>
<td id="A2.T4.4.4.7.5" class="ltx_td ltx_align_center ltx_border_bb">-</td>
<td id="A2.T4.4.4.7.6" class="ltx_td ltx_align_center ltx_border_bb">-</td>
<td id="A2.T4.4.4.7.7" class="ltx_td ltx_align_center ltx_border_bb">1536</td>
</tr>
</table>
</figure>
</section>
<section id="A2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.2 </span>Fine-tuning</h3>

<div id="A2.SS2.p1" class="ltx_para">
<p id="A2.SS2.p1.2" class="ltx_p">We test learning rates in the range of <math id="A2.SS2.p1.1.m1.1" class="ltx_Math" alttext="1\times 10^{-5}" display="inline"><semantics id="A2.SS2.p1.1.m1.1a"><mrow id="A2.SS2.p1.1.m1.1.1" xref="A2.SS2.p1.1.m1.1.1.cmml"><mn id="A2.SS2.p1.1.m1.1.1.2" xref="A2.SS2.p1.1.m1.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="A2.SS2.p1.1.m1.1.1.1" xref="A2.SS2.p1.1.m1.1.1.1.cmml">×</mo><msup id="A2.SS2.p1.1.m1.1.1.3" xref="A2.SS2.p1.1.m1.1.1.3.cmml"><mn id="A2.SS2.p1.1.m1.1.1.3.2" xref="A2.SS2.p1.1.m1.1.1.3.2.cmml">10</mn><mrow id="A2.SS2.p1.1.m1.1.1.3.3" xref="A2.SS2.p1.1.m1.1.1.3.3.cmml"><mo id="A2.SS2.p1.1.m1.1.1.3.3a" xref="A2.SS2.p1.1.m1.1.1.3.3.cmml">−</mo><mn id="A2.SS2.p1.1.m1.1.1.3.3.2" xref="A2.SS2.p1.1.m1.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A2.SS2.p1.1.m1.1b"><apply id="A2.SS2.p1.1.m1.1.1.cmml" xref="A2.SS2.p1.1.m1.1.1"><times id="A2.SS2.p1.1.m1.1.1.1.cmml" xref="A2.SS2.p1.1.m1.1.1.1"></times><cn type="integer" id="A2.SS2.p1.1.m1.1.1.2.cmml" xref="A2.SS2.p1.1.m1.1.1.2">1</cn><apply id="A2.SS2.p1.1.m1.1.1.3.cmml" xref="A2.SS2.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="A2.SS2.p1.1.m1.1.1.3.1.cmml" xref="A2.SS2.p1.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="A2.SS2.p1.1.m1.1.1.3.2.cmml" xref="A2.SS2.p1.1.m1.1.1.3.2">10</cn><apply id="A2.SS2.p1.1.m1.1.1.3.3.cmml" xref="A2.SS2.p1.1.m1.1.1.3.3"><minus id="A2.SS2.p1.1.m1.1.1.3.3.1.cmml" xref="A2.SS2.p1.1.m1.1.1.3.3"></minus><cn type="integer" id="A2.SS2.p1.1.m1.1.1.3.3.2.cmml" xref="A2.SS2.p1.1.m1.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS2.p1.1.m1.1c">1\times 10^{-5}</annotation></semantics></math> to <math id="A2.SS2.p1.2.m2.1" class="ltx_Math" alttext="5\times 10^{-4}" display="inline"><semantics id="A2.SS2.p1.2.m2.1a"><mrow id="A2.SS2.p1.2.m2.1.1" xref="A2.SS2.p1.2.m2.1.1.cmml"><mn id="A2.SS2.p1.2.m2.1.1.2" xref="A2.SS2.p1.2.m2.1.1.2.cmml">5</mn><mo lspace="0.222em" rspace="0.222em" id="A2.SS2.p1.2.m2.1.1.1" xref="A2.SS2.p1.2.m2.1.1.1.cmml">×</mo><msup id="A2.SS2.p1.2.m2.1.1.3" xref="A2.SS2.p1.2.m2.1.1.3.cmml"><mn id="A2.SS2.p1.2.m2.1.1.3.2" xref="A2.SS2.p1.2.m2.1.1.3.2.cmml">10</mn><mrow id="A2.SS2.p1.2.m2.1.1.3.3" xref="A2.SS2.p1.2.m2.1.1.3.3.cmml"><mo id="A2.SS2.p1.2.m2.1.1.3.3a" xref="A2.SS2.p1.2.m2.1.1.3.3.cmml">−</mo><mn id="A2.SS2.p1.2.m2.1.1.3.3.2" xref="A2.SS2.p1.2.m2.1.1.3.3.2.cmml">4</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A2.SS2.p1.2.m2.1b"><apply id="A2.SS2.p1.2.m2.1.1.cmml" xref="A2.SS2.p1.2.m2.1.1"><times id="A2.SS2.p1.2.m2.1.1.1.cmml" xref="A2.SS2.p1.2.m2.1.1.1"></times><cn type="integer" id="A2.SS2.p1.2.m2.1.1.2.cmml" xref="A2.SS2.p1.2.m2.1.1.2">5</cn><apply id="A2.SS2.p1.2.m2.1.1.3.cmml" xref="A2.SS2.p1.2.m2.1.1.3"><csymbol cd="ambiguous" id="A2.SS2.p1.2.m2.1.1.3.1.cmml" xref="A2.SS2.p1.2.m2.1.1.3">superscript</csymbol><cn type="integer" id="A2.SS2.p1.2.m2.1.1.3.2.cmml" xref="A2.SS2.p1.2.m2.1.1.3.2">10</cn><apply id="A2.SS2.p1.2.m2.1.1.3.3.cmml" xref="A2.SS2.p1.2.m2.1.1.3.3"><minus id="A2.SS2.p1.2.m2.1.1.3.3.1.cmml" xref="A2.SS2.p1.2.m2.1.1.3.3"></minus><cn type="integer" id="A2.SS2.p1.2.m2.1.1.3.3.2.cmml" xref="A2.SS2.p1.2.m2.1.1.3.3.2">4</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS2.p1.2.m2.1c">5\times 10^{-4}</annotation></semantics></math>. For <span id="A2.SS2.p1.2.1" class="ltx_text ltx_font_smallcaps">LoQT</span> qLoFT, we employed normal float NF4 quantization and performed 5 iterations of optimizing the error of quantization. We used a batch size of 32 and a maximum sequence length of 256.
Table <a href="#A2.T5" title="Table 5 ‣ B.2 Fine-tuning ‣ Appendix B Hyperparamters ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> summarizes the detailed hyperparameters for tasks in GLUE using the DeBERTaV3-base model. We use a fixed projection gap of 2400 for all runs.</p>
</div>
<figure id="A2.T5" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="A2.T5.20.1.1" class="ltx_text" style="font-size:90%;">Table 5</span>: </span><span id="A2.T5.21.2" class="ltx_text" style="font-size:90%;">Hyperparameter setup for <span id="A2.T5.21.2.1" class="ltx_text ltx_font_smallcaps">LoQT</span>-nq, <span id="A2.T5.21.2.2" class="ltx_text ltx_font_smallcaps">LoQT</span>, LoftQ<cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib21" title="" class="ltx_ref">21</a>]</cite>, LoRA<cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib21" title="" class="ltx_ref">21</a>]</cite>, and Galore across various tasks on the GLUE benchmark.</span></figcaption>
<div id="A2.T5.16" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:75.8pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-40.7pt,7.1pt) scale(0.841889901176652,0.841889901176652) ;">
<table id="A2.T5.16.16" class="ltx_tabular ltx_align_middle">
<tr id="A2.T5.16.16.17" class="ltx_tr">
<td id="A2.T5.16.16.17.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="A2.T5.16.16.17.1.1" class="ltx_text ltx_font_bold">Method</span></td>
<td id="A2.T5.16.16.17.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="A2.T5.16.16.17.2.1" class="ltx_text ltx_font_bold">Hyper-parameter</span></td>
<td id="A2.T5.16.16.17.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="A2.T5.16.16.17.3.1" class="ltx_text ltx_font_bold">MNLI</span></td>
<td id="A2.T5.16.16.17.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="A2.T5.16.16.17.4.1" class="ltx_text ltx_font_bold">RTE</span></td>
<td id="A2.T5.16.16.17.5" class="ltx_td ltx_align_center ltx_border_tt"><span id="A2.T5.16.16.17.5.1" class="ltx_text ltx_font_bold">QNLI</span></td>
<td id="A2.T5.16.16.17.6" class="ltx_td ltx_align_center ltx_border_tt"><span id="A2.T5.16.16.17.6.1" class="ltx_text ltx_font_bold">MRPC</span></td>
<td id="A2.T5.16.16.17.7" class="ltx_td ltx_align_center ltx_border_tt"><span id="A2.T5.16.16.17.7.1" class="ltx_text ltx_font_bold">QQP</span></td>
<td id="A2.T5.16.16.17.8" class="ltx_td ltx_align_center ltx_border_tt"><span id="A2.T5.16.16.17.8.1" class="ltx_text ltx_font_bold">SST-2</span></td>
<td id="A2.T5.16.16.17.9" class="ltx_td ltx_align_center ltx_border_tt"><span id="A2.T5.16.16.17.9.1" class="ltx_text ltx_font_bold">CoLA</span></td>
<td id="A2.T5.16.16.17.10" class="ltx_td ltx_align_center ltx_border_tt"><span id="A2.T5.16.16.17.10.1" class="ltx_text ltx_font_bold">STS-B</span></td>
</tr>
<tr id="A2.T5.16.16.18" class="ltx_tr">
<td id="A2.T5.16.16.18.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" rowspan="2"><span id="A2.T5.16.16.18.1.1" class="ltx_text"><span id="A2.T5.16.16.18.1.1.1" class="ltx_text ltx_font_smallcaps">LoQT</span>, LoFTQ</span></td>
<td id="A2.T5.16.16.18.2" class="ltx_td ltx_align_center ltx_border_t"># Epochs</td>
<td id="A2.T5.16.16.18.3" class="ltx_td ltx_align_center ltx_border_t">5</td>
<td id="A2.T5.16.16.18.4" class="ltx_td ltx_align_center ltx_border_t">20</td>
<td id="A2.T5.16.16.18.5" class="ltx_td ltx_align_center ltx_border_t">10</td>
<td id="A2.T5.16.16.18.6" class="ltx_td ltx_align_center ltx_border_t">60</td>
<td id="A2.T5.16.16.18.7" class="ltx_td ltx_align_center ltx_border_t">10</td>
<td id="A2.T5.16.16.18.8" class="ltx_td ltx_align_center ltx_border_t">10</td>
<td id="A2.T5.16.16.18.9" class="ltx_td ltx_align_center ltx_border_t">20</td>
<td id="A2.T5.16.16.18.10" class="ltx_td ltx_align_center ltx_border_t">60</td>
</tr>
<tr id="A2.T5.8.8.8" class="ltx_tr">
<td id="A2.T5.8.8.8.9" class="ltx_td ltx_align_center">Learning Rate</td>
<td id="A2.T5.1.1.1.1" class="ltx_td ltx_align_center"><math id="A2.T5.1.1.1.1.m1.1" class="ltx_Math" alttext="1\times 10^{-4}" display="inline"><semantics id="A2.T5.1.1.1.1.m1.1a"><mrow id="A2.T5.1.1.1.1.m1.1.1" xref="A2.T5.1.1.1.1.m1.1.1.cmml"><mn id="A2.T5.1.1.1.1.m1.1.1.2" xref="A2.T5.1.1.1.1.m1.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="A2.T5.1.1.1.1.m1.1.1.1" xref="A2.T5.1.1.1.1.m1.1.1.1.cmml">×</mo><msup id="A2.T5.1.1.1.1.m1.1.1.3" xref="A2.T5.1.1.1.1.m1.1.1.3.cmml"><mn id="A2.T5.1.1.1.1.m1.1.1.3.2" xref="A2.T5.1.1.1.1.m1.1.1.3.2.cmml">10</mn><mrow id="A2.T5.1.1.1.1.m1.1.1.3.3" xref="A2.T5.1.1.1.1.m1.1.1.3.3.cmml"><mo id="A2.T5.1.1.1.1.m1.1.1.3.3a" xref="A2.T5.1.1.1.1.m1.1.1.3.3.cmml">−</mo><mn id="A2.T5.1.1.1.1.m1.1.1.3.3.2" xref="A2.T5.1.1.1.1.m1.1.1.3.3.2.cmml">4</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A2.T5.1.1.1.1.m1.1b"><apply id="A2.T5.1.1.1.1.m1.1.1.cmml" xref="A2.T5.1.1.1.1.m1.1.1"><times id="A2.T5.1.1.1.1.m1.1.1.1.cmml" xref="A2.T5.1.1.1.1.m1.1.1.1"></times><cn type="integer" id="A2.T5.1.1.1.1.m1.1.1.2.cmml" xref="A2.T5.1.1.1.1.m1.1.1.2">1</cn><apply id="A2.T5.1.1.1.1.m1.1.1.3.cmml" xref="A2.T5.1.1.1.1.m1.1.1.3"><csymbol cd="ambiguous" id="A2.T5.1.1.1.1.m1.1.1.3.1.cmml" xref="A2.T5.1.1.1.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="A2.T5.1.1.1.1.m1.1.1.3.2.cmml" xref="A2.T5.1.1.1.1.m1.1.1.3.2">10</cn><apply id="A2.T5.1.1.1.1.m1.1.1.3.3.cmml" xref="A2.T5.1.1.1.1.m1.1.1.3.3"><minus id="A2.T5.1.1.1.1.m1.1.1.3.3.1.cmml" xref="A2.T5.1.1.1.1.m1.1.1.3.3"></minus><cn type="integer" id="A2.T5.1.1.1.1.m1.1.1.3.3.2.cmml" xref="A2.T5.1.1.1.1.m1.1.1.3.3.2">4</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.1.1.1.1.m1.1c">1\times 10^{-4}</annotation></semantics></math></td>
<td id="A2.T5.2.2.2.2" class="ltx_td ltx_align_center"><math id="A2.T5.2.2.2.2.m1.1" class="ltx_Math" alttext="5\times 10^{-5}" display="inline"><semantics id="A2.T5.2.2.2.2.m1.1a"><mrow id="A2.T5.2.2.2.2.m1.1.1" xref="A2.T5.2.2.2.2.m1.1.1.cmml"><mn id="A2.T5.2.2.2.2.m1.1.1.2" xref="A2.T5.2.2.2.2.m1.1.1.2.cmml">5</mn><mo lspace="0.222em" rspace="0.222em" id="A2.T5.2.2.2.2.m1.1.1.1" xref="A2.T5.2.2.2.2.m1.1.1.1.cmml">×</mo><msup id="A2.T5.2.2.2.2.m1.1.1.3" xref="A2.T5.2.2.2.2.m1.1.1.3.cmml"><mn id="A2.T5.2.2.2.2.m1.1.1.3.2" xref="A2.T5.2.2.2.2.m1.1.1.3.2.cmml">10</mn><mrow id="A2.T5.2.2.2.2.m1.1.1.3.3" xref="A2.T5.2.2.2.2.m1.1.1.3.3.cmml"><mo id="A2.T5.2.2.2.2.m1.1.1.3.3a" xref="A2.T5.2.2.2.2.m1.1.1.3.3.cmml">−</mo><mn id="A2.T5.2.2.2.2.m1.1.1.3.3.2" xref="A2.T5.2.2.2.2.m1.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A2.T5.2.2.2.2.m1.1b"><apply id="A2.T5.2.2.2.2.m1.1.1.cmml" xref="A2.T5.2.2.2.2.m1.1.1"><times id="A2.T5.2.2.2.2.m1.1.1.1.cmml" xref="A2.T5.2.2.2.2.m1.1.1.1"></times><cn type="integer" id="A2.T5.2.2.2.2.m1.1.1.2.cmml" xref="A2.T5.2.2.2.2.m1.1.1.2">5</cn><apply id="A2.T5.2.2.2.2.m1.1.1.3.cmml" xref="A2.T5.2.2.2.2.m1.1.1.3"><csymbol cd="ambiguous" id="A2.T5.2.2.2.2.m1.1.1.3.1.cmml" xref="A2.T5.2.2.2.2.m1.1.1.3">superscript</csymbol><cn type="integer" id="A2.T5.2.2.2.2.m1.1.1.3.2.cmml" xref="A2.T5.2.2.2.2.m1.1.1.3.2">10</cn><apply id="A2.T5.2.2.2.2.m1.1.1.3.3.cmml" xref="A2.T5.2.2.2.2.m1.1.1.3.3"><minus id="A2.T5.2.2.2.2.m1.1.1.3.3.1.cmml" xref="A2.T5.2.2.2.2.m1.1.1.3.3"></minus><cn type="integer" id="A2.T5.2.2.2.2.m1.1.1.3.3.2.cmml" xref="A2.T5.2.2.2.2.m1.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.2.2.2.2.m1.1c">5\times 10^{-5}</annotation></semantics></math></td>
<td id="A2.T5.3.3.3.3" class="ltx_td ltx_align_center"><math id="A2.T5.3.3.3.3.m1.1" class="ltx_Math" alttext="5\times 10^{-5}" display="inline"><semantics id="A2.T5.3.3.3.3.m1.1a"><mrow id="A2.T5.3.3.3.3.m1.1.1" xref="A2.T5.3.3.3.3.m1.1.1.cmml"><mn id="A2.T5.3.3.3.3.m1.1.1.2" xref="A2.T5.3.3.3.3.m1.1.1.2.cmml">5</mn><mo lspace="0.222em" rspace="0.222em" id="A2.T5.3.3.3.3.m1.1.1.1" xref="A2.T5.3.3.3.3.m1.1.1.1.cmml">×</mo><msup id="A2.T5.3.3.3.3.m1.1.1.3" xref="A2.T5.3.3.3.3.m1.1.1.3.cmml"><mn id="A2.T5.3.3.3.3.m1.1.1.3.2" xref="A2.T5.3.3.3.3.m1.1.1.3.2.cmml">10</mn><mrow id="A2.T5.3.3.3.3.m1.1.1.3.3" xref="A2.T5.3.3.3.3.m1.1.1.3.3.cmml"><mo id="A2.T5.3.3.3.3.m1.1.1.3.3a" xref="A2.T5.3.3.3.3.m1.1.1.3.3.cmml">−</mo><mn id="A2.T5.3.3.3.3.m1.1.1.3.3.2" xref="A2.T5.3.3.3.3.m1.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A2.T5.3.3.3.3.m1.1b"><apply id="A2.T5.3.3.3.3.m1.1.1.cmml" xref="A2.T5.3.3.3.3.m1.1.1"><times id="A2.T5.3.3.3.3.m1.1.1.1.cmml" xref="A2.T5.3.3.3.3.m1.1.1.1"></times><cn type="integer" id="A2.T5.3.3.3.3.m1.1.1.2.cmml" xref="A2.T5.3.3.3.3.m1.1.1.2">5</cn><apply id="A2.T5.3.3.3.3.m1.1.1.3.cmml" xref="A2.T5.3.3.3.3.m1.1.1.3"><csymbol cd="ambiguous" id="A2.T5.3.3.3.3.m1.1.1.3.1.cmml" xref="A2.T5.3.3.3.3.m1.1.1.3">superscript</csymbol><cn type="integer" id="A2.T5.3.3.3.3.m1.1.1.3.2.cmml" xref="A2.T5.3.3.3.3.m1.1.1.3.2">10</cn><apply id="A2.T5.3.3.3.3.m1.1.1.3.3.cmml" xref="A2.T5.3.3.3.3.m1.1.1.3.3"><minus id="A2.T5.3.3.3.3.m1.1.1.3.3.1.cmml" xref="A2.T5.3.3.3.3.m1.1.1.3.3"></minus><cn type="integer" id="A2.T5.3.3.3.3.m1.1.1.3.3.2.cmml" xref="A2.T5.3.3.3.3.m1.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.3.3.3.3.m1.1c">5\times 10^{-5}</annotation></semantics></math></td>
<td id="A2.T5.4.4.4.4" class="ltx_td ltx_align_center"><math id="A2.T5.4.4.4.4.m1.1" class="ltx_Math" alttext="1\times 10^{-4}" display="inline"><semantics id="A2.T5.4.4.4.4.m1.1a"><mrow id="A2.T5.4.4.4.4.m1.1.1" xref="A2.T5.4.4.4.4.m1.1.1.cmml"><mn id="A2.T5.4.4.4.4.m1.1.1.2" xref="A2.T5.4.4.4.4.m1.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="A2.T5.4.4.4.4.m1.1.1.1" xref="A2.T5.4.4.4.4.m1.1.1.1.cmml">×</mo><msup id="A2.T5.4.4.4.4.m1.1.1.3" xref="A2.T5.4.4.4.4.m1.1.1.3.cmml"><mn id="A2.T5.4.4.4.4.m1.1.1.3.2" xref="A2.T5.4.4.4.4.m1.1.1.3.2.cmml">10</mn><mrow id="A2.T5.4.4.4.4.m1.1.1.3.3" xref="A2.T5.4.4.4.4.m1.1.1.3.3.cmml"><mo id="A2.T5.4.4.4.4.m1.1.1.3.3a" xref="A2.T5.4.4.4.4.m1.1.1.3.3.cmml">−</mo><mn id="A2.T5.4.4.4.4.m1.1.1.3.3.2" xref="A2.T5.4.4.4.4.m1.1.1.3.3.2.cmml">4</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A2.T5.4.4.4.4.m1.1b"><apply id="A2.T5.4.4.4.4.m1.1.1.cmml" xref="A2.T5.4.4.4.4.m1.1.1"><times id="A2.T5.4.4.4.4.m1.1.1.1.cmml" xref="A2.T5.4.4.4.4.m1.1.1.1"></times><cn type="integer" id="A2.T5.4.4.4.4.m1.1.1.2.cmml" xref="A2.T5.4.4.4.4.m1.1.1.2">1</cn><apply id="A2.T5.4.4.4.4.m1.1.1.3.cmml" xref="A2.T5.4.4.4.4.m1.1.1.3"><csymbol cd="ambiguous" id="A2.T5.4.4.4.4.m1.1.1.3.1.cmml" xref="A2.T5.4.4.4.4.m1.1.1.3">superscript</csymbol><cn type="integer" id="A2.T5.4.4.4.4.m1.1.1.3.2.cmml" xref="A2.T5.4.4.4.4.m1.1.1.3.2">10</cn><apply id="A2.T5.4.4.4.4.m1.1.1.3.3.cmml" xref="A2.T5.4.4.4.4.m1.1.1.3.3"><minus id="A2.T5.4.4.4.4.m1.1.1.3.3.1.cmml" xref="A2.T5.4.4.4.4.m1.1.1.3.3"></minus><cn type="integer" id="A2.T5.4.4.4.4.m1.1.1.3.3.2.cmml" xref="A2.T5.4.4.4.4.m1.1.1.3.3.2">4</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.4.4.4.4.m1.1c">1\times 10^{-4}</annotation></semantics></math></td>
<td id="A2.T5.5.5.5.5" class="ltx_td ltx_align_center"><math id="A2.T5.5.5.5.5.m1.1" class="ltx_Math" alttext="5\times 10^{-5}" display="inline"><semantics id="A2.T5.5.5.5.5.m1.1a"><mrow id="A2.T5.5.5.5.5.m1.1.1" xref="A2.T5.5.5.5.5.m1.1.1.cmml"><mn id="A2.T5.5.5.5.5.m1.1.1.2" xref="A2.T5.5.5.5.5.m1.1.1.2.cmml">5</mn><mo lspace="0.222em" rspace="0.222em" id="A2.T5.5.5.5.5.m1.1.1.1" xref="A2.T5.5.5.5.5.m1.1.1.1.cmml">×</mo><msup id="A2.T5.5.5.5.5.m1.1.1.3" xref="A2.T5.5.5.5.5.m1.1.1.3.cmml"><mn id="A2.T5.5.5.5.5.m1.1.1.3.2" xref="A2.T5.5.5.5.5.m1.1.1.3.2.cmml">10</mn><mrow id="A2.T5.5.5.5.5.m1.1.1.3.3" xref="A2.T5.5.5.5.5.m1.1.1.3.3.cmml"><mo id="A2.T5.5.5.5.5.m1.1.1.3.3a" xref="A2.T5.5.5.5.5.m1.1.1.3.3.cmml">−</mo><mn id="A2.T5.5.5.5.5.m1.1.1.3.3.2" xref="A2.T5.5.5.5.5.m1.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A2.T5.5.5.5.5.m1.1b"><apply id="A2.T5.5.5.5.5.m1.1.1.cmml" xref="A2.T5.5.5.5.5.m1.1.1"><times id="A2.T5.5.5.5.5.m1.1.1.1.cmml" xref="A2.T5.5.5.5.5.m1.1.1.1"></times><cn type="integer" id="A2.T5.5.5.5.5.m1.1.1.2.cmml" xref="A2.T5.5.5.5.5.m1.1.1.2">5</cn><apply id="A2.T5.5.5.5.5.m1.1.1.3.cmml" xref="A2.T5.5.5.5.5.m1.1.1.3"><csymbol cd="ambiguous" id="A2.T5.5.5.5.5.m1.1.1.3.1.cmml" xref="A2.T5.5.5.5.5.m1.1.1.3">superscript</csymbol><cn type="integer" id="A2.T5.5.5.5.5.m1.1.1.3.2.cmml" xref="A2.T5.5.5.5.5.m1.1.1.3.2">10</cn><apply id="A2.T5.5.5.5.5.m1.1.1.3.3.cmml" xref="A2.T5.5.5.5.5.m1.1.1.3.3"><minus id="A2.T5.5.5.5.5.m1.1.1.3.3.1.cmml" xref="A2.T5.5.5.5.5.m1.1.1.3.3"></minus><cn type="integer" id="A2.T5.5.5.5.5.m1.1.1.3.3.2.cmml" xref="A2.T5.5.5.5.5.m1.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.5.5.5.5.m1.1c">5\times 10^{-5}</annotation></semantics></math></td>
<td id="A2.T5.6.6.6.6" class="ltx_td ltx_align_center"><math id="A2.T5.6.6.6.6.m1.1" class="ltx_Math" alttext="5\times 10^{-5}" display="inline"><semantics id="A2.T5.6.6.6.6.m1.1a"><mrow id="A2.T5.6.6.6.6.m1.1.1" xref="A2.T5.6.6.6.6.m1.1.1.cmml"><mn id="A2.T5.6.6.6.6.m1.1.1.2" xref="A2.T5.6.6.6.6.m1.1.1.2.cmml">5</mn><mo lspace="0.222em" rspace="0.222em" id="A2.T5.6.6.6.6.m1.1.1.1" xref="A2.T5.6.6.6.6.m1.1.1.1.cmml">×</mo><msup id="A2.T5.6.6.6.6.m1.1.1.3" xref="A2.T5.6.6.6.6.m1.1.1.3.cmml"><mn id="A2.T5.6.6.6.6.m1.1.1.3.2" xref="A2.T5.6.6.6.6.m1.1.1.3.2.cmml">10</mn><mrow id="A2.T5.6.6.6.6.m1.1.1.3.3" xref="A2.T5.6.6.6.6.m1.1.1.3.3.cmml"><mo id="A2.T5.6.6.6.6.m1.1.1.3.3a" xref="A2.T5.6.6.6.6.m1.1.1.3.3.cmml">−</mo><mn id="A2.T5.6.6.6.6.m1.1.1.3.3.2" xref="A2.T5.6.6.6.6.m1.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A2.T5.6.6.6.6.m1.1b"><apply id="A2.T5.6.6.6.6.m1.1.1.cmml" xref="A2.T5.6.6.6.6.m1.1.1"><times id="A2.T5.6.6.6.6.m1.1.1.1.cmml" xref="A2.T5.6.6.6.6.m1.1.1.1"></times><cn type="integer" id="A2.T5.6.6.6.6.m1.1.1.2.cmml" xref="A2.T5.6.6.6.6.m1.1.1.2">5</cn><apply id="A2.T5.6.6.6.6.m1.1.1.3.cmml" xref="A2.T5.6.6.6.6.m1.1.1.3"><csymbol cd="ambiguous" id="A2.T5.6.6.6.6.m1.1.1.3.1.cmml" xref="A2.T5.6.6.6.6.m1.1.1.3">superscript</csymbol><cn type="integer" id="A2.T5.6.6.6.6.m1.1.1.3.2.cmml" xref="A2.T5.6.6.6.6.m1.1.1.3.2">10</cn><apply id="A2.T5.6.6.6.6.m1.1.1.3.3.cmml" xref="A2.T5.6.6.6.6.m1.1.1.3.3"><minus id="A2.T5.6.6.6.6.m1.1.1.3.3.1.cmml" xref="A2.T5.6.6.6.6.m1.1.1.3.3"></minus><cn type="integer" id="A2.T5.6.6.6.6.m1.1.1.3.3.2.cmml" xref="A2.T5.6.6.6.6.m1.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.6.6.6.6.m1.1c">5\times 10^{-5}</annotation></semantics></math></td>
<td id="A2.T5.7.7.7.7" class="ltx_td ltx_align_center"><math id="A2.T5.7.7.7.7.m1.1" class="ltx_Math" alttext="1\times 10^{-4}" display="inline"><semantics id="A2.T5.7.7.7.7.m1.1a"><mrow id="A2.T5.7.7.7.7.m1.1.1" xref="A2.T5.7.7.7.7.m1.1.1.cmml"><mn id="A2.T5.7.7.7.7.m1.1.1.2" xref="A2.T5.7.7.7.7.m1.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="A2.T5.7.7.7.7.m1.1.1.1" xref="A2.T5.7.7.7.7.m1.1.1.1.cmml">×</mo><msup id="A2.T5.7.7.7.7.m1.1.1.3" xref="A2.T5.7.7.7.7.m1.1.1.3.cmml"><mn id="A2.T5.7.7.7.7.m1.1.1.3.2" xref="A2.T5.7.7.7.7.m1.1.1.3.2.cmml">10</mn><mrow id="A2.T5.7.7.7.7.m1.1.1.3.3" xref="A2.T5.7.7.7.7.m1.1.1.3.3.cmml"><mo id="A2.T5.7.7.7.7.m1.1.1.3.3a" xref="A2.T5.7.7.7.7.m1.1.1.3.3.cmml">−</mo><mn id="A2.T5.7.7.7.7.m1.1.1.3.3.2" xref="A2.T5.7.7.7.7.m1.1.1.3.3.2.cmml">4</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A2.T5.7.7.7.7.m1.1b"><apply id="A2.T5.7.7.7.7.m1.1.1.cmml" xref="A2.T5.7.7.7.7.m1.1.1"><times id="A2.T5.7.7.7.7.m1.1.1.1.cmml" xref="A2.T5.7.7.7.7.m1.1.1.1"></times><cn type="integer" id="A2.T5.7.7.7.7.m1.1.1.2.cmml" xref="A2.T5.7.7.7.7.m1.1.1.2">1</cn><apply id="A2.T5.7.7.7.7.m1.1.1.3.cmml" xref="A2.T5.7.7.7.7.m1.1.1.3"><csymbol cd="ambiguous" id="A2.T5.7.7.7.7.m1.1.1.3.1.cmml" xref="A2.T5.7.7.7.7.m1.1.1.3">superscript</csymbol><cn type="integer" id="A2.T5.7.7.7.7.m1.1.1.3.2.cmml" xref="A2.T5.7.7.7.7.m1.1.1.3.2">10</cn><apply id="A2.T5.7.7.7.7.m1.1.1.3.3.cmml" xref="A2.T5.7.7.7.7.m1.1.1.3.3"><minus id="A2.T5.7.7.7.7.m1.1.1.3.3.1.cmml" xref="A2.T5.7.7.7.7.m1.1.1.3.3"></minus><cn type="integer" id="A2.T5.7.7.7.7.m1.1.1.3.3.2.cmml" xref="A2.T5.7.7.7.7.m1.1.1.3.3.2">4</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.7.7.7.7.m1.1c">1\times 10^{-4}</annotation></semantics></math></td>
<td id="A2.T5.8.8.8.8" class="ltx_td ltx_align_center"><math id="A2.T5.8.8.8.8.m1.1" class="ltx_Math" alttext="5\times 10^{-5}" display="inline"><semantics id="A2.T5.8.8.8.8.m1.1a"><mrow id="A2.T5.8.8.8.8.m1.1.1" xref="A2.T5.8.8.8.8.m1.1.1.cmml"><mn id="A2.T5.8.8.8.8.m1.1.1.2" xref="A2.T5.8.8.8.8.m1.1.1.2.cmml">5</mn><mo lspace="0.222em" rspace="0.222em" id="A2.T5.8.8.8.8.m1.1.1.1" xref="A2.T5.8.8.8.8.m1.1.1.1.cmml">×</mo><msup id="A2.T5.8.8.8.8.m1.1.1.3" xref="A2.T5.8.8.8.8.m1.1.1.3.cmml"><mn id="A2.T5.8.8.8.8.m1.1.1.3.2" xref="A2.T5.8.8.8.8.m1.1.1.3.2.cmml">10</mn><mrow id="A2.T5.8.8.8.8.m1.1.1.3.3" xref="A2.T5.8.8.8.8.m1.1.1.3.3.cmml"><mo id="A2.T5.8.8.8.8.m1.1.1.3.3a" xref="A2.T5.8.8.8.8.m1.1.1.3.3.cmml">−</mo><mn id="A2.T5.8.8.8.8.m1.1.1.3.3.2" xref="A2.T5.8.8.8.8.m1.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A2.T5.8.8.8.8.m1.1b"><apply id="A2.T5.8.8.8.8.m1.1.1.cmml" xref="A2.T5.8.8.8.8.m1.1.1"><times id="A2.T5.8.8.8.8.m1.1.1.1.cmml" xref="A2.T5.8.8.8.8.m1.1.1.1"></times><cn type="integer" id="A2.T5.8.8.8.8.m1.1.1.2.cmml" xref="A2.T5.8.8.8.8.m1.1.1.2">5</cn><apply id="A2.T5.8.8.8.8.m1.1.1.3.cmml" xref="A2.T5.8.8.8.8.m1.1.1.3"><csymbol cd="ambiguous" id="A2.T5.8.8.8.8.m1.1.1.3.1.cmml" xref="A2.T5.8.8.8.8.m1.1.1.3">superscript</csymbol><cn type="integer" id="A2.T5.8.8.8.8.m1.1.1.3.2.cmml" xref="A2.T5.8.8.8.8.m1.1.1.3.2">10</cn><apply id="A2.T5.8.8.8.8.m1.1.1.3.3.cmml" xref="A2.T5.8.8.8.8.m1.1.1.3.3"><minus id="A2.T5.8.8.8.8.m1.1.1.3.3.1.cmml" xref="A2.T5.8.8.8.8.m1.1.1.3.3"></minus><cn type="integer" id="A2.T5.8.8.8.8.m1.1.1.3.3.2.cmml" xref="A2.T5.8.8.8.8.m1.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.8.8.8.8.m1.1c">5\times 10^{-5}</annotation></semantics></math></td>
</tr>
<tr id="A2.T5.16.16.19" class="ltx_tr">
<td id="A2.T5.16.16.19.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r ltx_border_t" rowspan="2"><span id="A2.T5.16.16.19.1.1" class="ltx_text">LoRA, Galore</span></td>
<td id="A2.T5.16.16.19.2" class="ltx_td ltx_align_center ltx_border_t"># Epochs</td>
<td id="A2.T5.16.16.19.3" class="ltx_td ltx_align_center ltx_border_t">10</td>
<td id="A2.T5.16.16.19.4" class="ltx_td ltx_align_center ltx_border_t">30</td>
<td id="A2.T5.16.16.19.5" class="ltx_td ltx_align_center ltx_border_t">30</td>
<td id="A2.T5.16.16.19.6" class="ltx_td ltx_align_center ltx_border_t">30</td>
<td id="A2.T5.16.16.19.7" class="ltx_td ltx_align_center ltx_border_t">30</td>
<td id="A2.T5.16.16.19.8" class="ltx_td ltx_align_center ltx_border_t">30</td>
<td id="A2.T5.16.16.19.9" class="ltx_td ltx_align_center ltx_border_t">30</td>
<td id="A2.T5.16.16.19.10" class="ltx_td ltx_align_center ltx_border_t">30</td>
</tr>
<tr id="A2.T5.16.16.16" class="ltx_tr">
<td id="A2.T5.16.16.16.9" class="ltx_td ltx_align_center ltx_border_bb">Learning Rate</td>
<td id="A2.T5.9.9.9.1" class="ltx_td ltx_align_center ltx_border_bb"><math id="A2.T5.9.9.9.1.m1.1" class="ltx_Math" alttext="1\times 10^{-5}" display="inline"><semantics id="A2.T5.9.9.9.1.m1.1a"><mrow id="A2.T5.9.9.9.1.m1.1.1" xref="A2.T5.9.9.9.1.m1.1.1.cmml"><mn id="A2.T5.9.9.9.1.m1.1.1.2" xref="A2.T5.9.9.9.1.m1.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="A2.T5.9.9.9.1.m1.1.1.1" xref="A2.T5.9.9.9.1.m1.1.1.1.cmml">×</mo><msup id="A2.T5.9.9.9.1.m1.1.1.3" xref="A2.T5.9.9.9.1.m1.1.1.3.cmml"><mn id="A2.T5.9.9.9.1.m1.1.1.3.2" xref="A2.T5.9.9.9.1.m1.1.1.3.2.cmml">10</mn><mrow id="A2.T5.9.9.9.1.m1.1.1.3.3" xref="A2.T5.9.9.9.1.m1.1.1.3.3.cmml"><mo id="A2.T5.9.9.9.1.m1.1.1.3.3a" xref="A2.T5.9.9.9.1.m1.1.1.3.3.cmml">−</mo><mn id="A2.T5.9.9.9.1.m1.1.1.3.3.2" xref="A2.T5.9.9.9.1.m1.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A2.T5.9.9.9.1.m1.1b"><apply id="A2.T5.9.9.9.1.m1.1.1.cmml" xref="A2.T5.9.9.9.1.m1.1.1"><times id="A2.T5.9.9.9.1.m1.1.1.1.cmml" xref="A2.T5.9.9.9.1.m1.1.1.1"></times><cn type="integer" id="A2.T5.9.9.9.1.m1.1.1.2.cmml" xref="A2.T5.9.9.9.1.m1.1.1.2">1</cn><apply id="A2.T5.9.9.9.1.m1.1.1.3.cmml" xref="A2.T5.9.9.9.1.m1.1.1.3"><csymbol cd="ambiguous" id="A2.T5.9.9.9.1.m1.1.1.3.1.cmml" xref="A2.T5.9.9.9.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="A2.T5.9.9.9.1.m1.1.1.3.2.cmml" xref="A2.T5.9.9.9.1.m1.1.1.3.2">10</cn><apply id="A2.T5.9.9.9.1.m1.1.1.3.3.cmml" xref="A2.T5.9.9.9.1.m1.1.1.3.3"><minus id="A2.T5.9.9.9.1.m1.1.1.3.3.1.cmml" xref="A2.T5.9.9.9.1.m1.1.1.3.3"></minus><cn type="integer" id="A2.T5.9.9.9.1.m1.1.1.3.3.2.cmml" xref="A2.T5.9.9.9.1.m1.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.9.9.9.1.m1.1c">1\times 10^{-5}</annotation></semantics></math></td>
<td id="A2.T5.10.10.10.2" class="ltx_td ltx_align_center ltx_border_bb"><math id="A2.T5.10.10.10.2.m1.1" class="ltx_Math" alttext="2\times 10^{-5}" display="inline"><semantics id="A2.T5.10.10.10.2.m1.1a"><mrow id="A2.T5.10.10.10.2.m1.1.1" xref="A2.T5.10.10.10.2.m1.1.1.cmml"><mn id="A2.T5.10.10.10.2.m1.1.1.2" xref="A2.T5.10.10.10.2.m1.1.1.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="A2.T5.10.10.10.2.m1.1.1.1" xref="A2.T5.10.10.10.2.m1.1.1.1.cmml">×</mo><msup id="A2.T5.10.10.10.2.m1.1.1.3" xref="A2.T5.10.10.10.2.m1.1.1.3.cmml"><mn id="A2.T5.10.10.10.2.m1.1.1.3.2" xref="A2.T5.10.10.10.2.m1.1.1.3.2.cmml">10</mn><mrow id="A2.T5.10.10.10.2.m1.1.1.3.3" xref="A2.T5.10.10.10.2.m1.1.1.3.3.cmml"><mo id="A2.T5.10.10.10.2.m1.1.1.3.3a" xref="A2.T5.10.10.10.2.m1.1.1.3.3.cmml">−</mo><mn id="A2.T5.10.10.10.2.m1.1.1.3.3.2" xref="A2.T5.10.10.10.2.m1.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A2.T5.10.10.10.2.m1.1b"><apply id="A2.T5.10.10.10.2.m1.1.1.cmml" xref="A2.T5.10.10.10.2.m1.1.1"><times id="A2.T5.10.10.10.2.m1.1.1.1.cmml" xref="A2.T5.10.10.10.2.m1.1.1.1"></times><cn type="integer" id="A2.T5.10.10.10.2.m1.1.1.2.cmml" xref="A2.T5.10.10.10.2.m1.1.1.2">2</cn><apply id="A2.T5.10.10.10.2.m1.1.1.3.cmml" xref="A2.T5.10.10.10.2.m1.1.1.3"><csymbol cd="ambiguous" id="A2.T5.10.10.10.2.m1.1.1.3.1.cmml" xref="A2.T5.10.10.10.2.m1.1.1.3">superscript</csymbol><cn type="integer" id="A2.T5.10.10.10.2.m1.1.1.3.2.cmml" xref="A2.T5.10.10.10.2.m1.1.1.3.2">10</cn><apply id="A2.T5.10.10.10.2.m1.1.1.3.3.cmml" xref="A2.T5.10.10.10.2.m1.1.1.3.3"><minus id="A2.T5.10.10.10.2.m1.1.1.3.3.1.cmml" xref="A2.T5.10.10.10.2.m1.1.1.3.3"></minus><cn type="integer" id="A2.T5.10.10.10.2.m1.1.1.3.3.2.cmml" xref="A2.T5.10.10.10.2.m1.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.10.10.10.2.m1.1c">2\times 10^{-5}</annotation></semantics></math></td>
<td id="A2.T5.11.11.11.3" class="ltx_td ltx_align_center ltx_border_bb"><math id="A2.T5.11.11.11.3.m1.1" class="ltx_Math" alttext="1\times 10^{-5}" display="inline"><semantics id="A2.T5.11.11.11.3.m1.1a"><mrow id="A2.T5.11.11.11.3.m1.1.1" xref="A2.T5.11.11.11.3.m1.1.1.cmml"><mn id="A2.T5.11.11.11.3.m1.1.1.2" xref="A2.T5.11.11.11.3.m1.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="A2.T5.11.11.11.3.m1.1.1.1" xref="A2.T5.11.11.11.3.m1.1.1.1.cmml">×</mo><msup id="A2.T5.11.11.11.3.m1.1.1.3" xref="A2.T5.11.11.11.3.m1.1.1.3.cmml"><mn id="A2.T5.11.11.11.3.m1.1.1.3.2" xref="A2.T5.11.11.11.3.m1.1.1.3.2.cmml">10</mn><mrow id="A2.T5.11.11.11.3.m1.1.1.3.3" xref="A2.T5.11.11.11.3.m1.1.1.3.3.cmml"><mo id="A2.T5.11.11.11.3.m1.1.1.3.3a" xref="A2.T5.11.11.11.3.m1.1.1.3.3.cmml">−</mo><mn id="A2.T5.11.11.11.3.m1.1.1.3.3.2" xref="A2.T5.11.11.11.3.m1.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A2.T5.11.11.11.3.m1.1b"><apply id="A2.T5.11.11.11.3.m1.1.1.cmml" xref="A2.T5.11.11.11.3.m1.1.1"><times id="A2.T5.11.11.11.3.m1.1.1.1.cmml" xref="A2.T5.11.11.11.3.m1.1.1.1"></times><cn type="integer" id="A2.T5.11.11.11.3.m1.1.1.2.cmml" xref="A2.T5.11.11.11.3.m1.1.1.2">1</cn><apply id="A2.T5.11.11.11.3.m1.1.1.3.cmml" xref="A2.T5.11.11.11.3.m1.1.1.3"><csymbol cd="ambiguous" id="A2.T5.11.11.11.3.m1.1.1.3.1.cmml" xref="A2.T5.11.11.11.3.m1.1.1.3">superscript</csymbol><cn type="integer" id="A2.T5.11.11.11.3.m1.1.1.3.2.cmml" xref="A2.T5.11.11.11.3.m1.1.1.3.2">10</cn><apply id="A2.T5.11.11.11.3.m1.1.1.3.3.cmml" xref="A2.T5.11.11.11.3.m1.1.1.3.3"><minus id="A2.T5.11.11.11.3.m1.1.1.3.3.1.cmml" xref="A2.T5.11.11.11.3.m1.1.1.3.3"></minus><cn type="integer" id="A2.T5.11.11.11.3.m1.1.1.3.3.2.cmml" xref="A2.T5.11.11.11.3.m1.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.11.11.11.3.m1.1c">1\times 10^{-5}</annotation></semantics></math></td>
<td id="A2.T5.12.12.12.4" class="ltx_td ltx_align_center ltx_border_bb"><math id="A2.T5.12.12.12.4.m1.1" class="ltx_Math" alttext="2\times 10^{-5}" display="inline"><semantics id="A2.T5.12.12.12.4.m1.1a"><mrow id="A2.T5.12.12.12.4.m1.1.1" xref="A2.T5.12.12.12.4.m1.1.1.cmml"><mn id="A2.T5.12.12.12.4.m1.1.1.2" xref="A2.T5.12.12.12.4.m1.1.1.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="A2.T5.12.12.12.4.m1.1.1.1" xref="A2.T5.12.12.12.4.m1.1.1.1.cmml">×</mo><msup id="A2.T5.12.12.12.4.m1.1.1.3" xref="A2.T5.12.12.12.4.m1.1.1.3.cmml"><mn id="A2.T5.12.12.12.4.m1.1.1.3.2" xref="A2.T5.12.12.12.4.m1.1.1.3.2.cmml">10</mn><mrow id="A2.T5.12.12.12.4.m1.1.1.3.3" xref="A2.T5.12.12.12.4.m1.1.1.3.3.cmml"><mo id="A2.T5.12.12.12.4.m1.1.1.3.3a" xref="A2.T5.12.12.12.4.m1.1.1.3.3.cmml">−</mo><mn id="A2.T5.12.12.12.4.m1.1.1.3.3.2" xref="A2.T5.12.12.12.4.m1.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A2.T5.12.12.12.4.m1.1b"><apply id="A2.T5.12.12.12.4.m1.1.1.cmml" xref="A2.T5.12.12.12.4.m1.1.1"><times id="A2.T5.12.12.12.4.m1.1.1.1.cmml" xref="A2.T5.12.12.12.4.m1.1.1.1"></times><cn type="integer" id="A2.T5.12.12.12.4.m1.1.1.2.cmml" xref="A2.T5.12.12.12.4.m1.1.1.2">2</cn><apply id="A2.T5.12.12.12.4.m1.1.1.3.cmml" xref="A2.T5.12.12.12.4.m1.1.1.3"><csymbol cd="ambiguous" id="A2.T5.12.12.12.4.m1.1.1.3.1.cmml" xref="A2.T5.12.12.12.4.m1.1.1.3">superscript</csymbol><cn type="integer" id="A2.T5.12.12.12.4.m1.1.1.3.2.cmml" xref="A2.T5.12.12.12.4.m1.1.1.3.2">10</cn><apply id="A2.T5.12.12.12.4.m1.1.1.3.3.cmml" xref="A2.T5.12.12.12.4.m1.1.1.3.3"><minus id="A2.T5.12.12.12.4.m1.1.1.3.3.1.cmml" xref="A2.T5.12.12.12.4.m1.1.1.3.3"></minus><cn type="integer" id="A2.T5.12.12.12.4.m1.1.1.3.3.2.cmml" xref="A2.T5.12.12.12.4.m1.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.12.12.12.4.m1.1c">2\times 10^{-5}</annotation></semantics></math></td>
<td id="A2.T5.13.13.13.5" class="ltx_td ltx_align_center ltx_border_bb"><math id="A2.T5.13.13.13.5.m1.1" class="ltx_Math" alttext="1\times 10^{-5}" display="inline"><semantics id="A2.T5.13.13.13.5.m1.1a"><mrow id="A2.T5.13.13.13.5.m1.1.1" xref="A2.T5.13.13.13.5.m1.1.1.cmml"><mn id="A2.T5.13.13.13.5.m1.1.1.2" xref="A2.T5.13.13.13.5.m1.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="A2.T5.13.13.13.5.m1.1.1.1" xref="A2.T5.13.13.13.5.m1.1.1.1.cmml">×</mo><msup id="A2.T5.13.13.13.5.m1.1.1.3" xref="A2.T5.13.13.13.5.m1.1.1.3.cmml"><mn id="A2.T5.13.13.13.5.m1.1.1.3.2" xref="A2.T5.13.13.13.5.m1.1.1.3.2.cmml">10</mn><mrow id="A2.T5.13.13.13.5.m1.1.1.3.3" xref="A2.T5.13.13.13.5.m1.1.1.3.3.cmml"><mo id="A2.T5.13.13.13.5.m1.1.1.3.3a" xref="A2.T5.13.13.13.5.m1.1.1.3.3.cmml">−</mo><mn id="A2.T5.13.13.13.5.m1.1.1.3.3.2" xref="A2.T5.13.13.13.5.m1.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A2.T5.13.13.13.5.m1.1b"><apply id="A2.T5.13.13.13.5.m1.1.1.cmml" xref="A2.T5.13.13.13.5.m1.1.1"><times id="A2.T5.13.13.13.5.m1.1.1.1.cmml" xref="A2.T5.13.13.13.5.m1.1.1.1"></times><cn type="integer" id="A2.T5.13.13.13.5.m1.1.1.2.cmml" xref="A2.T5.13.13.13.5.m1.1.1.2">1</cn><apply id="A2.T5.13.13.13.5.m1.1.1.3.cmml" xref="A2.T5.13.13.13.5.m1.1.1.3"><csymbol cd="ambiguous" id="A2.T5.13.13.13.5.m1.1.1.3.1.cmml" xref="A2.T5.13.13.13.5.m1.1.1.3">superscript</csymbol><cn type="integer" id="A2.T5.13.13.13.5.m1.1.1.3.2.cmml" xref="A2.T5.13.13.13.5.m1.1.1.3.2">10</cn><apply id="A2.T5.13.13.13.5.m1.1.1.3.3.cmml" xref="A2.T5.13.13.13.5.m1.1.1.3.3"><minus id="A2.T5.13.13.13.5.m1.1.1.3.3.1.cmml" xref="A2.T5.13.13.13.5.m1.1.1.3.3"></minus><cn type="integer" id="A2.T5.13.13.13.5.m1.1.1.3.3.2.cmml" xref="A2.T5.13.13.13.5.m1.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.13.13.13.5.m1.1c">1\times 10^{-5}</annotation></semantics></math></td>
<td id="A2.T5.14.14.14.6" class="ltx_td ltx_align_center ltx_border_bb"><math id="A2.T5.14.14.14.6.m1.1" class="ltx_Math" alttext="2\times 10^{-5}" display="inline"><semantics id="A2.T5.14.14.14.6.m1.1a"><mrow id="A2.T5.14.14.14.6.m1.1.1" xref="A2.T5.14.14.14.6.m1.1.1.cmml"><mn id="A2.T5.14.14.14.6.m1.1.1.2" xref="A2.T5.14.14.14.6.m1.1.1.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="A2.T5.14.14.14.6.m1.1.1.1" xref="A2.T5.14.14.14.6.m1.1.1.1.cmml">×</mo><msup id="A2.T5.14.14.14.6.m1.1.1.3" xref="A2.T5.14.14.14.6.m1.1.1.3.cmml"><mn id="A2.T5.14.14.14.6.m1.1.1.3.2" xref="A2.T5.14.14.14.6.m1.1.1.3.2.cmml">10</mn><mrow id="A2.T5.14.14.14.6.m1.1.1.3.3" xref="A2.T5.14.14.14.6.m1.1.1.3.3.cmml"><mo id="A2.T5.14.14.14.6.m1.1.1.3.3a" xref="A2.T5.14.14.14.6.m1.1.1.3.3.cmml">−</mo><mn id="A2.T5.14.14.14.6.m1.1.1.3.3.2" xref="A2.T5.14.14.14.6.m1.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A2.T5.14.14.14.6.m1.1b"><apply id="A2.T5.14.14.14.6.m1.1.1.cmml" xref="A2.T5.14.14.14.6.m1.1.1"><times id="A2.T5.14.14.14.6.m1.1.1.1.cmml" xref="A2.T5.14.14.14.6.m1.1.1.1"></times><cn type="integer" id="A2.T5.14.14.14.6.m1.1.1.2.cmml" xref="A2.T5.14.14.14.6.m1.1.1.2">2</cn><apply id="A2.T5.14.14.14.6.m1.1.1.3.cmml" xref="A2.T5.14.14.14.6.m1.1.1.3"><csymbol cd="ambiguous" id="A2.T5.14.14.14.6.m1.1.1.3.1.cmml" xref="A2.T5.14.14.14.6.m1.1.1.3">superscript</csymbol><cn type="integer" id="A2.T5.14.14.14.6.m1.1.1.3.2.cmml" xref="A2.T5.14.14.14.6.m1.1.1.3.2">10</cn><apply id="A2.T5.14.14.14.6.m1.1.1.3.3.cmml" xref="A2.T5.14.14.14.6.m1.1.1.3.3"><minus id="A2.T5.14.14.14.6.m1.1.1.3.3.1.cmml" xref="A2.T5.14.14.14.6.m1.1.1.3.3"></minus><cn type="integer" id="A2.T5.14.14.14.6.m1.1.1.3.3.2.cmml" xref="A2.T5.14.14.14.6.m1.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.14.14.14.6.m1.1c">2\times 10^{-5}</annotation></semantics></math></td>
<td id="A2.T5.15.15.15.7" class="ltx_td ltx_align_center ltx_border_bb"><math id="A2.T5.15.15.15.7.m1.1" class="ltx_Math" alttext="2\times 10^{-5}" display="inline"><semantics id="A2.T5.15.15.15.7.m1.1a"><mrow id="A2.T5.15.15.15.7.m1.1.1" xref="A2.T5.15.15.15.7.m1.1.1.cmml"><mn id="A2.T5.15.15.15.7.m1.1.1.2" xref="A2.T5.15.15.15.7.m1.1.1.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="A2.T5.15.15.15.7.m1.1.1.1" xref="A2.T5.15.15.15.7.m1.1.1.1.cmml">×</mo><msup id="A2.T5.15.15.15.7.m1.1.1.3" xref="A2.T5.15.15.15.7.m1.1.1.3.cmml"><mn id="A2.T5.15.15.15.7.m1.1.1.3.2" xref="A2.T5.15.15.15.7.m1.1.1.3.2.cmml">10</mn><mrow id="A2.T5.15.15.15.7.m1.1.1.3.3" xref="A2.T5.15.15.15.7.m1.1.1.3.3.cmml"><mo id="A2.T5.15.15.15.7.m1.1.1.3.3a" xref="A2.T5.15.15.15.7.m1.1.1.3.3.cmml">−</mo><mn id="A2.T5.15.15.15.7.m1.1.1.3.3.2" xref="A2.T5.15.15.15.7.m1.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A2.T5.15.15.15.7.m1.1b"><apply id="A2.T5.15.15.15.7.m1.1.1.cmml" xref="A2.T5.15.15.15.7.m1.1.1"><times id="A2.T5.15.15.15.7.m1.1.1.1.cmml" xref="A2.T5.15.15.15.7.m1.1.1.1"></times><cn type="integer" id="A2.T5.15.15.15.7.m1.1.1.2.cmml" xref="A2.T5.15.15.15.7.m1.1.1.2">2</cn><apply id="A2.T5.15.15.15.7.m1.1.1.3.cmml" xref="A2.T5.15.15.15.7.m1.1.1.3"><csymbol cd="ambiguous" id="A2.T5.15.15.15.7.m1.1.1.3.1.cmml" xref="A2.T5.15.15.15.7.m1.1.1.3">superscript</csymbol><cn type="integer" id="A2.T5.15.15.15.7.m1.1.1.3.2.cmml" xref="A2.T5.15.15.15.7.m1.1.1.3.2">10</cn><apply id="A2.T5.15.15.15.7.m1.1.1.3.3.cmml" xref="A2.T5.15.15.15.7.m1.1.1.3.3"><minus id="A2.T5.15.15.15.7.m1.1.1.3.3.1.cmml" xref="A2.T5.15.15.15.7.m1.1.1.3.3"></minus><cn type="integer" id="A2.T5.15.15.15.7.m1.1.1.3.3.2.cmml" xref="A2.T5.15.15.15.7.m1.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.15.15.15.7.m1.1c">2\times 10^{-5}</annotation></semantics></math></td>
<td id="A2.T5.16.16.16.8" class="ltx_td ltx_align_center ltx_border_bb"><math id="A2.T5.16.16.16.8.m1.1" class="ltx_Math" alttext="3\times 10^{-5}" display="inline"><semantics id="A2.T5.16.16.16.8.m1.1a"><mrow id="A2.T5.16.16.16.8.m1.1.1" xref="A2.T5.16.16.16.8.m1.1.1.cmml"><mn id="A2.T5.16.16.16.8.m1.1.1.2" xref="A2.T5.16.16.16.8.m1.1.1.2.cmml">3</mn><mo lspace="0.222em" rspace="0.222em" id="A2.T5.16.16.16.8.m1.1.1.1" xref="A2.T5.16.16.16.8.m1.1.1.1.cmml">×</mo><msup id="A2.T5.16.16.16.8.m1.1.1.3" xref="A2.T5.16.16.16.8.m1.1.1.3.cmml"><mn id="A2.T5.16.16.16.8.m1.1.1.3.2" xref="A2.T5.16.16.16.8.m1.1.1.3.2.cmml">10</mn><mrow id="A2.T5.16.16.16.8.m1.1.1.3.3" xref="A2.T5.16.16.16.8.m1.1.1.3.3.cmml"><mo id="A2.T5.16.16.16.8.m1.1.1.3.3a" xref="A2.T5.16.16.16.8.m1.1.1.3.3.cmml">−</mo><mn id="A2.T5.16.16.16.8.m1.1.1.3.3.2" xref="A2.T5.16.16.16.8.m1.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A2.T5.16.16.16.8.m1.1b"><apply id="A2.T5.16.16.16.8.m1.1.1.cmml" xref="A2.T5.16.16.16.8.m1.1.1"><times id="A2.T5.16.16.16.8.m1.1.1.1.cmml" xref="A2.T5.16.16.16.8.m1.1.1.1"></times><cn type="integer" id="A2.T5.16.16.16.8.m1.1.1.2.cmml" xref="A2.T5.16.16.16.8.m1.1.1.2">3</cn><apply id="A2.T5.16.16.16.8.m1.1.1.3.cmml" xref="A2.T5.16.16.16.8.m1.1.1.3"><csymbol cd="ambiguous" id="A2.T5.16.16.16.8.m1.1.1.3.1.cmml" xref="A2.T5.16.16.16.8.m1.1.1.3">superscript</csymbol><cn type="integer" id="A2.T5.16.16.16.8.m1.1.1.3.2.cmml" xref="A2.T5.16.16.16.8.m1.1.1.3.2">10</cn><apply id="A2.T5.16.16.16.8.m1.1.1.3.3.cmml" xref="A2.T5.16.16.16.8.m1.1.1.3.3"><minus id="A2.T5.16.16.16.8.m1.1.1.3.3.1.cmml" xref="A2.T5.16.16.16.8.m1.1.1.3.3"></minus><cn type="integer" id="A2.T5.16.16.16.8.m1.1.1.3.3.2.cmml" xref="A2.T5.16.16.16.8.m1.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.16.16.16.8.m1.1c">3\times 10^{-5}</annotation></semantics></math></td>
</tr>
</table>
</span></div>
</figure>
</section>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Rank Ablation</h2>

<div id="A3.p1" class="ltx_para">
<p id="A3.p1.6" class="ltx_p">Figure <a href="#A3.F5" title="Figure 5 ‣ Appendix C Rank Ablation ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> shows the validation perplexity versus training steps for various ranks using <span id="A3.p1.6.1" class="ltx_text ltx_font_smallcaps">LoQT</span>-nq and <span id="A3.p1.6.2" class="ltx_text ltx_font_smallcaps">LoQT</span> on a <math id="A3.p1.1.m1.1" class="ltx_Math" alttext="130" display="inline"><semantics id="A3.p1.1.m1.1a"><mn id="A3.p1.1.m1.1.1" xref="A3.p1.1.m1.1.1.cmml">130</mn><annotation-xml encoding="MathML-Content" id="A3.p1.1.m1.1b"><cn type="integer" id="A3.p1.1.m1.1.1.cmml" xref="A3.p1.1.m1.1.1">130</cn></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.1.m1.1c">130</annotation></semantics></math> million parameter model over 20,000 iterations. All models employ an exponentially increasing update frequency starting at <math id="A3.p1.2.m2.1" class="ltx_Math" alttext="100" display="inline"><semantics id="A3.p1.2.m2.1a"><mn id="A3.p1.2.m2.1.1" xref="A3.p1.2.m2.1.1.cmml">100</mn><annotation-xml encoding="MathML-Content" id="A3.p1.2.m2.1b"><cn type="integer" id="A3.p1.2.m2.1.1.cmml" xref="A3.p1.2.m2.1.1">100</cn></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.2.m2.1c">100</annotation></semantics></math>, with a factor of <math id="A3.p1.3.m3.1" class="ltx_Math" alttext="1.2^{T_{i}}" display="inline"><semantics id="A3.p1.3.m3.1a"><msup id="A3.p1.3.m3.1.1" xref="A3.p1.3.m3.1.1.cmml"><mn id="A3.p1.3.m3.1.1.2" xref="A3.p1.3.m3.1.1.2.cmml">1.2</mn><msub id="A3.p1.3.m3.1.1.3" xref="A3.p1.3.m3.1.1.3.cmml"><mi id="A3.p1.3.m3.1.1.3.2" xref="A3.p1.3.m3.1.1.3.2.cmml">T</mi><mi id="A3.p1.3.m3.1.1.3.3" xref="A3.p1.3.m3.1.1.3.3.cmml">i</mi></msub></msup><annotation-xml encoding="MathML-Content" id="A3.p1.3.m3.1b"><apply id="A3.p1.3.m3.1.1.cmml" xref="A3.p1.3.m3.1.1"><csymbol cd="ambiguous" id="A3.p1.3.m3.1.1.1.cmml" xref="A3.p1.3.m3.1.1">superscript</csymbol><cn type="float" id="A3.p1.3.m3.1.1.2.cmml" xref="A3.p1.3.m3.1.1.2">1.2</cn><apply id="A3.p1.3.m3.1.1.3.cmml" xref="A3.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="A3.p1.3.m3.1.1.3.1.cmml" xref="A3.p1.3.m3.1.1.3">subscript</csymbol><ci id="A3.p1.3.m3.1.1.3.2.cmml" xref="A3.p1.3.m3.1.1.3.2">𝑇</ci><ci id="A3.p1.3.m3.1.1.3.3.cmml" xref="A3.p1.3.m3.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.3.m3.1c">1.2^{T_{i}}</annotation></semantics></math>.
The results demonstrate that both the quantized (<span id="A3.p1.6.3" class="ltx_text ltx_font_smallcaps">LoQT</span>) and non-quantized (<span id="A3.p1.6.4" class="ltx_text ltx_font_smallcaps">LoQT</span>-nq) models follow a similar trajectory for ranks ranging from <math id="A3.p1.4.m4.1" class="ltx_Math" alttext="64" display="inline"><semantics id="A3.p1.4.m4.1a"><mn id="A3.p1.4.m4.1.1" xref="A3.p1.4.m4.1.1.cmml">64</mn><annotation-xml encoding="MathML-Content" id="A3.p1.4.m4.1b"><cn type="integer" id="A3.p1.4.m4.1.1.cmml" xref="A3.p1.4.m4.1.1">64</cn></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.4.m4.1c">64</annotation></semantics></math> to <math id="A3.p1.5.m5.1" class="ltx_Math" alttext="512" display="inline"><semantics id="A3.p1.5.m5.1a"><mn id="A3.p1.5.m5.1.1" xref="A3.p1.5.m5.1.1.cmml">512</mn><annotation-xml encoding="MathML-Content" id="A3.p1.5.m5.1b"><cn type="integer" id="A3.p1.5.m5.1.1.cmml" xref="A3.p1.5.m5.1.1">512</cn></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.5.m5.1c">512</annotation></semantics></math>. However, for the smaller rank of <math id="A3.p1.6.m6.1" class="ltx_Math" alttext="64" display="inline"><semantics id="A3.p1.6.m6.1a"><mn id="A3.p1.6.m6.1.1" xref="A3.p1.6.m6.1.1.cmml">64</mn><annotation-xml encoding="MathML-Content" id="A3.p1.6.m6.1b"><cn type="integer" id="A3.p1.6.m6.1.1.cmml" xref="A3.p1.6.m6.1.1">64</cn></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.6.m6.1c">64</annotation></semantics></math>, there is a slight divergence between <span id="A3.p1.6.5" class="ltx_text ltx_font_smallcaps">LoQT</span>-nq and <span id="A3.p1.6.6" class="ltx_text ltx_font_smallcaps">LoQT</span>, indicating a limit to how low the rank can be while maintaining accuracy with quantization. This plot highlights the tradeoff between rank and perplexity, suggesting that while our method supports low-rank training, there is a minimum rank threshold needed to achieve results comparable to regular pre-training.</p>
</div>
<figure id="A3.F5" class="ltx_figure"><img src="/html/2405.16528/assets/x5.png" id="A3.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="230" height="181" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A3.F5.4.1.1" class="ltx_text" style="font-size:90%;">Figure 5</span>: </span><span id="A3.F5.5.2" class="ltx_text" style="font-size:90%;">Rank ablation for <span id="A3.F5.5.2.1" class="ltx_text ltx_font_smallcaps">LoQT</span> and <span id="A3.F5.5.2.2" class="ltx_text ltx_font_smallcaps">LoQT</span>-nq showing perplexity as a function of steps.</span></figcaption>
</figure>
<section id="A3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.1 </span>Memory Measurements</h3>

<div id="A3.SS1.p1" class="ltx_para">
<p id="A3.SS1.p1.1" class="ltx_p">Figure <a href="#A3.F6" title="Figure 6 ‣ C.1 Memory Measurements ‣ Appendix C Rank Ablation ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> demonstrates that <span id="A3.SS1.p1.1.1" class="ltx_text ltx_font_smallcaps">LoQT</span> requires less memory than GaLore and Adam, even without using per-layer gradients <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib25" title="" class="ltx_ref">25</a>]</cite> or Adam 8-bit <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib5" title="" class="ltx_ref">5</a>]</cite>. The gap between <span id="A3.SS1.p1.1.2" class="ltx_text ltx_font_smallcaps">LoQT</span> and the baselines increases with larger model sizes. The configurations and ranks for each model are shown in Table <a href="#A2.T4" title="Table 4 ‣ B.1 Pre-training ‣ Appendix B Hyperparamters ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>.
With <span id="A3.SS1.p1.1.3" class="ltx_text ltx_font_smallcaps">LoQT</span> and Adam 8-bit, it is possible to pre-train a 13B model with rank 1024 on a GPU with 24GB of VRAM. This enables training with <span id="A3.SS1.p1.1.4" class="ltx_text ltx_font_smallcaps">LoQT</span> on consumer GPUs, such as the NVIDIA 4090, using a small per-GPU token batch size 256. Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ LoQT: Low Rank Adapters for Quantized Training" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> in the main text provides a detailed breakdown of each memory component for the 13B model. Maximum memory allocated is measured using <span id="A3.SS1.p1.1.5" class="ltx_text ltx_font_typewriter">nvitop</span> (<a target="_blank" href="https://github.com/XuehaiPan/nvitop" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/XuehaiPan/nvitop</a>).</p>
</div>
<figure id="A3.F6" class="ltx_figure"><img src="/html/2405.16528/assets/x6.png" id="A3.F6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="346" height="197" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A3.F6.3.1.1" class="ltx_text" style="font-size:90%;">Figure 6</span>: </span><span id="A3.F6.4.2" class="ltx_text" style="font-size:90%;">Memory usage for <span id="A3.F6.4.2.1" class="ltx_text ltx_font_smallcaps">LoQT</span> vs baselines for different model sizes. LW means per-layer gradient updates as per <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib25" title="" class="ltx_ref">25</a>]</cite>, and A8bit mean with Adam 8-bit. We evaluate using a token batch size of 256.</span></figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2405.16527" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2405.16528" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2405.16528">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2405.16528" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2405.16529" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Wed Jun  5 16:24:33 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2405.16528
Training of large neural networks requires significant computational resources. Despite advances using low-rank adapters and quantization, pretraining of models such as LLMs on consumer hardware has not been possible without model sharding, offloading during training, or per-layer gradient updates. To address these limitations, we proposeLoQT, a method for efficiently training quantized models.LoQTuses gradient-based tensor factorization to initialize low-rank trainable weight matrices that are periodically merged into quantized full-rank weight matrices. Our approach is suitable for both pretraining and fine-tuning models, which we demonstrate experimentally for language modeling and downstream task adaptation. We find thatLoQTenables efficient training of models up to 7B parameters on a consumer-grade 24GB GPU. We also demonstrate the feasibility of training a 13B parameter model using per-layer gradient updates on the same hardware. https://github.com/sebulo/LoQT
abstract
LoQT: Low Rank Adapters for Quantized Training
Abstract
2405.16528
1IntroductionFigure 1:Memory usage of Llama 13B, rank 1024. PL: per-layer gradient updates. A8bit: Adam 8-bitTraining large neural networks requires substantial hardware and energy resources. Reducing these requirements is thus important for cost efficiency and environmental sustainability, while also lowering the entry barrier for researchers and practitioners. The main barriers in training large models are the compute operations required, as well as the memory needed to store those computations, in this paper we focus on the latter. Memory use during training comes primarily from the weights of the model itself as well as the optimizer states used to train the model.
To target the weights, variations on low-rank adaptation (LoRA) have been suggested to decrease the number of trainable parameters, in combination with the use of low precision representations.To target the optimizer, low-rank approaches for projecting gradients to a lower rank have been used. Finally, various applications of quantizationhave been used to decrease memory requirements. In this work, we combine these approaches into a highly memory-efficient training configuration.In typical training setups, the optimizer states take up larger space than the model itself, as methods such as Adamneed to keep track of two parameters for each weight of the model. GaLoresignificantly reduces the number of parameters needed for storing the optimizer states by only keeping track of the optimizer state in a low-rank projection which is then projected up to be applied to the model weights. Combining this method with quantization would further shrink the footprint of the model. However, updating the weights of a highly quantized model directly in low-precision space has not been shown to work. This is mainly due to the higher-precision gradient updates being too small to have an impact on the lower-precision quantized states. Lastly, while LoRA is memory efficient for parameter-efficient fine-tuning of pre-trained models, it does not work as a pretraining method by itself.Figure 2:Overview ofLoQT. (1)  Low-rank factorsandare periodically initialized from the gradient of the dequantized model weights, (2)  then onlyis trained whileandare kept quantized and frozen, over an exponentially increasing interval until, (3)  the low-rank factors are merged back into the quantized model. The process is repeated until training halts.To address these shortcomings, we propose a new method,LoQT.LoQTinitializes two low-rank factors for each weight matrix: 1) , using a projection of’s gradients into a low-rank subspace, and 2) , initialized to minimize the error of quantizing and represents the only low-rank matrix being trained. Only trainingmeans that the optimizer state can be shrunk significantly. The productis periodically merged into the full rank matrixwith exponentially increasing scheduling. Asanddo not receive gradient updates, they can be quantized, thus optimizing memory usage even further. We stress that it is the large accumulated updates that make it possible to update a quantized model as the addition of smaller changes would not register in the quantized state. A high-level overview is given in Fig.2.We show thatLoQTworks well with and without a quantized model, enabling not only a lower memory footprint in the optimizer state but also over the model parameters. Our results show that we get competitive performance to prior methods using significantly less memory, in particular when using quantization of the model weights in an application such as training an LLM. We also demonstrate superior performance when fine-tuning pre-trained models, by training and evaluating on the GLUEbenchmark for natural language understanding. Finally, we ablate several properties of the suggested approach and we find that an exponentially increasing projection gap is beneficial, not only to our work but also for GaLore. This is particularly crucial for the training of quantized models.LoQTenables efficient training of 7B models on consumer-grade hardware with 24GB of memory, and makes it feasible to train models with up to 13 billion parameters without model parallel by making use of per-layer gradient updateson the same hardware as shown in Fig.1.
introduction
LoQT: Low Rank Adapters for Quantized Training
1Introduction
2405.16528
2Related Work and Background2.1Neural Network Quantization and NF4Quantization compresses neural networks by converting high-precision values into lower-precision formats, significantly reducing storage requirements. The process involves taking a datatype of high precision, such as 32-bit, requiring 4 bytes of memory, and converting it into a representation with increasing rounding errors but lower memory cost.
In this work, we use NF4 quantization, since it is a 4-bit code it only containsdifferent values. NF4 works by first normalizing values onto the interval, these are then discretized onto quantiles of the normal distribution,(seefor details) . The elements of a layer are divided into blocks of 64 weights. Each blockhas a scaling factor.(1) (2) We provide an overview of different categories of quantization techniques, and how they relate toLoQT, in AppendixA. Compared to prior approaches,LoQTretains the benefits of reduced memory usage while minimizing accuracy loss, using high-precision updates on a low-rank representation. This allows for efficient model updates without the overhead of full matrix storage and re-quantization.2.2Adaptation of Pretrained NetworksLow-Rank Adaptation (LoRA) enables fine-tuning of pre-trained models using low-rank adaptors, effectively reducing the memory footprint by only training weight adaptors for targeted layers. However, simple low-rank training using LoRA factor matrices has not been shown to work for pre-training.LoRA employs trainable low-rank matricesandthat are used to updatefollowing, whereis frozen to enable precise adjustments within a low-rank framework. Since LoRA only trainsandand keepsfixed, QLoRAexplore quantizing. They fine-tune a quantized modelwith 4-bit precision using randomly initialized 16-bit precision factorsand. To address quantization errors, low-rank factors of the quantization errorhave been used.LoQTextends LoRA to both pretraining and fine-tuning. Unlike traditional LoRA,LoQTusesandto refinethroughout training, withinitialized from’s gradient projection andtrained along this gradient path.LoQTalso incorporates quantization and targeted optimization iterations similar in spirit to LoftQ, correcting for quantization errors in, thus better aligning it with the original non-quantized.2.3Memory Efficient OptimizationOptimizer memory consumptionA significant portion of the memory needed to train neural networks is typically consumed by optimizer states. Notably, Adam, one of the most widely used optimizers, uses double the amount of memory as the gradient matrix to maintain first and second-order gradient statistics.
Efforts to reduce this overhead have led to the development of adaptive optimization algorithms like Adafactor, which achieves sub-linear memory costs by factorizing the second-order statistics into a row-column outer product. GaLoreexpands on this concept by using low-rank factorization and projecting low-rank gradients up to a full-rank when updating model weights.Periodic updating of weight matricesReLoRAcombines low-rank updates with initial full-rank training. They find that doing one-third of the training in full-rank, and the subsequent two-thirds in low-rank (see §2.2)  results in comparable performance to standard training methods.Low-rank gradientsGaLore, focuses on the structure of the gradients, projecting them into a low-rank space using factorsand, which are derived from a truncated singular value decomposition (SVD)  of the weight matrix gradient,. This reduces memory costs associated with storing the optimizer states and aligns with findings from recent studies which suggest that learning primarily occurs within a low-dimensional subspace at a given time. This can be further combined with applying per-layer gradient updates, reducing the memory needed for storing the gradients for the full model at once.LoQTbuilds on GaLore’s gradient projection (§3.1)  to initialize LoRA factors while updating the full matrix following a schedule inspired by ReLora, while only training one low-rank matrix per layer. We achieve comparable quality to GaLore and better performance than ReLoRA while reducing tunable parameters and memory usage compared to both approaches.
body
LoQT: Low Rank Adapters for Quantized Training
2Related Work and Background
2405.16528
3Efficient Pretraining WithLoQTLoQTworks by initializing and training low-rank adapters obtained by taking the SVD of a given layer’s gradients. Letindicate the full weights matrix of a given layer,be the left factor constructed from the SVD decomposition of the gradients matrix:; i.e.consists of the firstcolumns ofcorresponding to the singular vectors with thelargest singular values of.
The update rule for an intervalis then given by, where only the weights ofare updated.andare kept constant over the time interval. We describe this in more detail below, followed by a discussion on periodic updating of the factor, the enabling of quantized pre-training, error compensation, and exponential update intervals. Pseudo-code forLoQTis shown in Fig.3.3.1Background: GaLoreZhao et al.show that gradients exhibit a low-rank structure during training. They exploit this insight by projecting the gradient to a low-rank subspace and applying the Adam optimizer before projecting back to the original dimensions. By doing this, the memory-intensive optimizer states required by Adam are shrunk significantly for low enough ranks.Definition 3.1(Gradient Low-rank Projection, def. 3.4 in) .Gradient low-rank projection (GaLore)  denotes the following gradient update rules, whereis the learning rate,is the Adam optimizer, andis the weight matrix being trained, andrepresents the total number of training iterations between recomputing the projection matrices:whereandare are the top-singular vectors from the SVD decomposition of the gradient matrix at each iteration. In practice, this can be approximated by only applying a one-sided projection, as inGaLore demonstrates that it is sufficient to keep the projection matrix fixed and only update it once everyiterations, which we use in the following.3.2Low-rank Gradients as AdaptersWe now describe the process by which we initialize the parameters we optimize inLoQT. We adopt the memory-performance trade-off of using only a one-sided projection. We computeifandotherwise. We want to achieve a separation between trainable weights and static weights, which we achieve by rewriting GaLore in terms of low-rank adaptors. Assume without loss of generality that.
Using the fact thatis fixed in the intervalwe have that(3) (4) From (4)  it is clear that we can keep track of low-rank updates using rank-low-rank adaptors. We note that in the intervalonlyis updated, creating the desired separation. If implemented directly, we would need to compute the gradient with respect toand then project it down using.
We find that this step is unnecessary; it is sufficient to trainusing standard gradient descent.We now show that training thematrix using gradient descent is equivalent to training w.r.t.as in definition3.1Letindicate the gradient of the loss with respect to, andfor the gradient of the loss with respect to. Given a weight matrix, an factorand a matrix, when computing the forward pass, the gradient of a loss functionw.r.t.is.
This can be seen by applying the chain rule to get.
The vector multiplied ontoisgiving.
This shows that calculating the gradient w.r.tgives the same as projecting the gradient w.r.t. It is thus clear that GaLore’s low-rank gradient updates should be the same as those obtained using backpropagation through LoRA.3.3Enabling pretraining with LoRAPrevious work has shown that training low-rank weight matrices works well for fine-tuning pre-trained weights. However, it has been shown that training low-rank factors, and periodically merging them into frozen, does not work when starting with a randomly initialized matrix. Here we address this shortcoming to enable full training using low-rank weight matrices.Inspired by prior work, we periodically update a given layerat fixed steps. This approach allowsto evolve as a sum of low-rank matrices aligning with GaLore’s strategy of updating the gradient subspace during training:whereandrepresents the product of the learning fromduring the intervalscaled by the learning rateand modulated by the gradient projection matrix.
After each update at iteration, we reinitialize the low-rank factorsand. As in, we compute the gradient ofover a single batch, focusing only onwithout needing full optimizer states. Not requiring optimizer states reduces the memory increase compared to full-rank training.With the updatedand reinitializedand, a new gradient subspace is established for exploring the nextsteps. Our method treatsas the full-rank repository of accumulated updates. Although it is periodically updated,is not part of the optimizer state computations and the gradients during the single forward pass are offloaded to cpu. Since the SVD calculations are done layerwise only the current layer is needed on GPU, or the SVD can be calculated on CPU.defines the general gradient subspace and trajectory for the upcomingsteps, andis adjusted to navigate optimally within the direction set by.
As onlyis trained, the number of parameters needing optimizer states is drastically reduced.3.4Quantized TrainingGiven thatis the only matrix accumulating gradients and undergoing changes, the other matricesandcan be kept quantized. This approach allows storing the weights in NF4 precision without requiring high-precision gradient and weights to updateand. To the best of our knowledge, we are the first to enable efficient 4-bit quantized training using gradient descent without storing the weights in full precision.We quantize weightsandas described in §2.1. During periodic updates at interval time steps,andare dequantized using the inverse function,and. After this,is computed and quantized. The quantization and dequantization processes are applied layer by layer, ensuring that not all layers are simultaneously in a non-quantized state to reduce memory usage. Moreover, the quantization state itself is re-quantized for further efficiency following. We implementLoQTusing weight-only quantization, meaning the quantized weights are loaded into memory and then dequantized before computing the matrix multiplications.Algorithm 1LoQT: Low Rank Adapters for Quantized Training0:: Weight,: Update steps,: LR,: rank,and: N-bit quant and dequant functions.1:2:3:foreachin training stepsdo4:ifthen5:6:7:8:else9:10:returnAlgorithm 2Initialization Procedure1:Initialize:2:3:{Firstsingular vectors}4:5:6:7:foreachin compensation stepsdo8:9:10:11:returnFigure 3:Pseudo-code forLoQT.3.5Compensating for Errors Introduced by QuantizationAs the quantization process inevitably leads to a discrepancy between the non-quantized and quantized versions ofwe wish to reduce this effect as much as possible. While compensating for quantization errors has been done before, we need a tailored solution forLoQT.During the merging phase, we first dequantize to obtainand, and then compute the update. This is immediately followed by re-quantizing to get. Our goal is thus to minimize the quantization error. To achieve this, we solve forin the merging step, initializingas, whereis the Moore-Penrose pseudo-inverse. This approach avoids initializingas zeros and instead uses it for minimizing the quantization error. We then iteratively refine, recomputing, improving the alignment between the full-precisionand its quantized state.As training advances and the learning rate decays, the magnitude of the updateto formdecreases. This leads to negligible differences between, which results in the weights plateauing early, as depicted in Fig.4(a) .
To address this, we implement an exponentially increasing scheduler for updating. Drawing from GaLore’s observation on the exponential decay of gradient rank (Lemma 3.1) , we start with a frequency gapand progressively increase the update intervals by a factor of. The sequence of updates is then given byEachmarks a training stepwhenis updated. This scheduling ensures more frequent updates earlier in training and more well-spaced adjustments later, allowing more accumulated gradients before each update.
body
LoQT: Low Rank Adapters for Quantized Training
3Efficient Pretraining WithLoQT
2405.16528
4Experiments and ResultsTable 1:Comparison of low-rank pre-training methods for LLaMA2-style language models on the C4 dataset. The table shows validation perplexity, memory estimates, and quantization states forLoQT. The rank ratiois relative to the largest weight matrix dimension. Perplexity values are averaged over three seeds showing mean and standard error. (*)  Denotes results from GaLore. Only one seed was used for the 1B experiment due to compute constraints.60M130M350M1BFull33.320.22 (0.36G) 24.510.03 (0.76G) 18.870.18 (2.06G) 15.56* (7.80G) LoQT(Ours) 33.980.15 (0.23G) 24.570.01 (0.49G) 19.120.01 (0.98G) 15.55 (3.16G) LoQT-nq (No quant.) 33.550.03 (0.28G) 24.370.02 (0.63G) 18.850.01 (1.47G) 15.20 (5.11G) GaLore34.150.24 (0.24G) 24.810.04 (0.52G) 19.470.01 (1.22G) 15.64* (4.38G) LoRA34.99* (0.36G) 33.92* (0.80G) 25.58* (1.76G) 19.21* (6.17G) ReLoRA37.04* (0.36G) 29.37* (0.80G) 29.08* (1.76G) 18.33* (6.17G) 128 / 256256 / 768256 / 1024512 / 2048Training Tokens1.1B2.2B6.4B13.1B4.1Experimental SetupWe evaluateLoQTby training LLaMA-basedlanguage models on the C4 dataset, a collection of processed text in English that was scraped from the internet. We train models of sizes of 60M, 130M, 350M, and 1B parameters, adhering to single-epoch training cycles determined by Chinchilla Scaling Laws. WhileLoQTis capable of training models up to 13 billion parameters on consumer GPUs, compute limits prevent us from training to convergence for sizes above 1B. We also benchmarkLoQTon the GLUE test-suite for natural language understanding. Runs were conducted on up to 4x 40GB NVIDIA A100s 2x 80GB NVIDIA H100s, or a single 24GB NVIDIA RTX 3090. The longest run was the training of the 1B models, taking approximately four days on the four A100s. The 3090 was used for throughput and to empirically verify memory claims.Hyperparameters are consistent across model sizes, with experiments in BF16 format for memory efficiency. All models use a maximum sequence length of, a total token batch size of 131K tokens, and a learning rate warmup for the first 10% of the training steps, followed by cosine annealing to 10% of the initial learning rate. Full experimental details, including the specific hyperparameters for each task, are provided in AppendixB.BaselinesFor pre-training, we compareLoQTagainst LoRA, ReLoRA, GaLore, and non-quantized version ofLoQT,LoQT-nq. In our experiments,LoQT, LoRA, and ReLoRA modify attention and fully connected layers while maintaining full-rank embeddings and normalization layers. This contrasts with GaLore, which keeps weights full-rank but projects gradients to low-rank, and standard full-rank training.
For fine-tuning, we benchmarkLoQTagainst GaLore, LoftQ, LoRA andLoQT-nq. All models use identical update frequencies for GaLore, ReLoRA,LoQT-nq, andLoQT, starting with an update frequency ofand then with exponentially increasing update frequencies. This means that we do more frequent updates early and fewer as the model stabilizes (see Section4(b) for more details) .
All models are trained using the Adam optimizer, except GaLore which uses GaLoreAdam for gradient projection.4.2Pre-training of Generative Language ModelsResults and details for pretraining of language models of sizes 60M, 130M, 350M and 1B parameters are shown in Table1. Model sizes are calculated based on the full models without any low-rank methods. We see thatLoQTandLoQT-nq both perform very close to full rank pretraining and GaLore, while using significantly less memory by keeping most of the model weights in a quantized state. For the 60M model, full training is only slightly better thanLoQT, while we see results improve or be within the standard error for the other sizes. We also notice a slight drop in performance from quantizing the original weight matrix, comparingLoQTandLoQT-nq. The key difference between the approaches are the theoretical memory estimaes, e.g. whereLoQTrequires 59% less memory for the 1B model in full precision and 28% less memory than GaLore.Table 2:Results withLoQT,LoQT-nq, and GaLore of DeBERTaV3-base models on the GLUE development set. We report mean and standard error over three seeds. The best results on each dataset are shown inbold.RankMethodMNLIQNLIRTESSTMRPCCoLAQQPSTSBAverageAccAccAccAccf1Mattf1PCorr32LoQT-nq90.00.1094.20.0684.80.7595.90.0694.10.2572.50.4190.00.0691.50.0789.132LoQT90.00.0994.30.0484.10.9195.50.1094.40.2070.50.3589.20.0291.50.1388.732LoRA89.90.0394.00.0983.60.1295.70.1593.50.2669.30.4789.80.1190.70.2288.332LoftQ90.40.0993.20.0283.80.6394.7*93.20.1471.10.2889.60.1291.00.0988.432GaLore90.30.0794.00.0483.70.7995.60.0793.40.3870.70.2489.80.0590.60.0188.54.3Memory-efficient finetuningWe fine-tune the pre-trained DeBERTa-V3-base111Fromhttps://huggingface.co/microsoft/deberta-v3-base.model on GLUE tasks usingLoQTand compare its performance with a full fine-tuning baseline, LoRA, LoftQ, and GaLore. See Appendix5for details on hyperparameters. Results are given in Table2.We find that bothLoQT-nq andLoQTperform well. And somewhat surprisingly, it sometimes surpasses GaLore, LoftQ, and LoRA.This may indicate that initializing the LoRA factors with information about the gradient ofcould be a beneficial starting point compared to standard initialization methods. As the goal of this work is to limit memory consumption, we leave out further comparisons that could verify these findings to future work.4.4Memory and ThroughputTable 3:Comparison of memory usage between GaLore, LoRA, andLoQT.() , rank.GaLoreLoRALoQT(Ours) WeightsOptimizer StatesGradientsPretrainingYesNoYesFine-TuningYesYesYesQuantizeableNoYesYesMemory usageAn overview of memory usage for GaLore, LoRA andLoQTis given in Table3. We see thatLoQTmakes use of the same number of trainable parameters as LoRA for a given rank while using less memory for the optimizer states and gradients than in both LoRA and GaLore.We compare theLoQTto the closest in memory performance approach, GaLore, for 13B in Figure1, and for other model-sizes in Figure6. We compare three different use cases, using the approaches directly, combining them with an 8-bit Adam optimizer, and using per-layer weight updates with offloading (while still using 8-bit Adam) . We see from the figures thatLoQTsignificantly shrinks both the number of trainable parameters and optimizer states compared to GaLore.Per-layer weight update is essential for GaLore; without it, an additionalGB of VRAM is needed for gradients in a 7B model, making full-parameter fine-tuning impossible on a 24GB GPU. Additionally, the per-layer gradient updates may not work well with DDP (Distributed Data Parallel)  and gradient accumulation. With our method, we can get a lower memory than GaLore even when they use per-layer gradient updates. When not using per-layer gradient updates, this difference becomes even bigger as seen for the 7B model in Figure6.Moreover, our method supports training 7B models without per-layer computations on 24GB GPU. This makes it possible to use multi-GPU training, a capability not possible with the current GaLore approach. Our memory advantage allows for a batch size oftokens compared to GaLore’sfor the 7B model on the 24GB RTX3090. With per-layer gradient updates,LoQTcan train a 13B model on a single GPU, pushing the limits of hardware efficiency.ThroughputWe evaluate the throughput with a sample batch size of 16 with a total batch size of 512 using gradient accumulation, which is the largest power of 2 that fits on the GPU. We update the projection matrixfor every 200 iterations. The per-layer gradient update algorithms apply a weight update for every mini-batch as they do not support gradient accumulation. For the evaluation, we use a 1B parameter model with rank 512. We find thatLoQTcan process 16% fewer tokens per second than only using AdamW, at 3996 tokens/s compared to 4782 tokens/s on the RTX3090.
body
LoQT: Low Rank Adapters for Quantized Training
4Experiments and Results
2405.16528
5Ablations(a) EC: Error compensation, EF: Exp.decreasing update frequency.(b) Ablation for different update frequencies, with exponentially increasing at the bottomFigure 4:Ablation results for update frequency, error-compensation, quantization, model size 130m, and rank.Quantization Error Compensation and InitializationTo assess the impact of quantization error compensation, we analyze the validation loss curves for a 130 million parameter model. Figure4(a) shows that quantizingor bothandwithout error compensation, or exponential frequency updates, causes the loss to stagnate early. We also note that quantizinghas a much smaller effect on the loss compared to quantizing.
Error compensation significantly improves the model’s performance, resulting in approximately 3.5 points better perplexity. Adding exponentially increasing update frequency improves perplexity by an additional 1.5 points, achieving performance close to that of models without quantization.Without the quantization error compensation detailed in §3.5,LoQT’s performance stagnates earlier and diverges more from the other models. This demonstrates the effectiveness of our compensation approach in mitigating the quantization errors introduced during theupdate withand subsequent quantization steps.Projection update frequencyOur scheduling approach ensures more frequent updates early in training to facilitate substantial weight adjustments. As training progresses, the update frequency decreases, allowing for the accumulation of larger updates to compensate for smaller updates that might be canceled out by quantization errors.
Figure4(b) presents an ablation study on our method of progressively increasing update frequency starting atand increasing by a factor ofup to. We show the validation loss curves for fixed update frequencies,,, and.The results show that exponentially increasing the update gap is particularly beneficial for models employing quantization, enabling them to achieve the same perplexity as those without quantization while making use of GaLore. Conversely, the performance gains are more subtle for models that do not use quantization and rely solely on GaLore. We hypothesize that even these models might benefit from the larger projection gap intervals. This could be due to the reduction in the accumulation of errors from frequent updates of the projection factor, as the influence of outdated optimizer statistics becomes less prevalent.
Finally, an ablation on the ranks used forandis given in Figure5in the Appendix.
body
LoQT: Low Rank Adapters for Quantized Training
5Ablations
2405.16528
6Discussion and ConclusionWe presentLoQT, a method for memory-efficient pretraining and adaptation of quantized models. The key insights behind the approach are the benefits of initializing low-rank factors using the gradient of the weight matrix and using exponentially increasing update gaps that make updating of a quantized model possible.
While our initial goal was to lower memory usage to facilitate the training of models such as LLMs on consumer-grade hardware, we are cautiously excited about the results sometimes being better than the baselines. These evaluations will be explored in more detail in future work.Our method is general and opens up new ways of decreasing memory use as well as improving the training throughput. This could be done by implementing kernel fusion and using other quantization methods such as NF2or quantization of activations, making it possible to do the matrix multiplications using modern tensor core formats such as FP8 or INT4.
body
LoQT: Low Rank Adapters for Quantized Training
6Discussion and Conclusion
2405.16528
7Impact and LimitationsOur work has the potential to have a significant impact on those working in hardware-constrained settings by enabling more efficient training on consumer hardware. We are particularly excited to see the method being applied in single GPU settings. We validateLoQTon several model sizes, by training over many steps and by fine-tuning on a standard benchmark for natural language understanding. While we are confident in our results, further exploration of training duration, data diversity, and hyper-parameter tuning might lead to different results in those settings.
body
LoQT: Low Rank Adapters for Quantized Training
7Impact and Limitations
2405.16528
8AcknowledgementsThis work is supported by the Danish Data Science Academy, which is funded by the Novo Nordisk Foundation (NNF21SA0069429)  and VILLUM FONDEN (40516) .
Serge Belongie and Vésteinn Snæbjarnarson are supported by the Pioneer Centre for AI, DNRF grant number P1. MJK acknowledges support from the Carlsberg Foundation and the Novo Nordisk Foundation.
conclusion
LoQT: Low Rank Adapters for Quantized Training
8Acknowledgements