<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2405.12130] MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning</title><meta property="og:description" content="Low-rank adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method for large language models (LLMs).
In this paper, we analyze the impact of low-rank updating, as implemented in LoRA. Our findings su…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2405.12130">

<!--Generated on Wed Jun  5 17:15:59 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document">
<h1 class="ltx_title ltx_title_document">MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">
<span id="id2.2.id1" class="ltx_text ltx_font_bold">Ting Jiang<sup id="id2.2.id1.1" class="ltx_sup"><span id="id2.2.id1.1.1" class="ltx_text ltx_font_medium">1</span></sup></span> , <span id="id3.3.id2" class="ltx_text ltx_font_bold">Shaohan Huang<sup id="id3.3.id2.1" class="ltx_sup"><span id="id3.3.id2.1.1" class="ltx_text ltx_font_medium">2</span></sup></span> , <span id="id4.4.id3" class="ltx_text ltx_font_bold">Shengyue Luo</span> <sup id="id5.5.id4" class="ltx_sup">2</sup>, <span id="id6.6.id5" class="ltx_text ltx_font_bold">Zihan Zhang<sup id="id6.6.id5.1" class="ltx_sup"><span id="id6.6.id5.1.1" class="ltx_text ltx_font_medium">2</span></sup></span> , <span id="id7.7.id6" class="ltx_text ltx_font_bold">Haizhen Huang<sup id="id7.7.id6.1" class="ltx_sup"><span id="id7.7.id6.1.1" class="ltx_text ltx_font_medium">2</span></sup>
<br class="ltx_break">Furu Wei<sup id="id7.7.id6.2" class="ltx_sup"><span id="id7.7.id6.2.1" class="ltx_text ltx_font_medium">2</span></sup></span> <span id="id8.8.id7" class="ltx_text ltx_font_bold">,</span> <span id="id9.9.id8" class="ltx_text ltx_font_bold">Weiwei Deng<sup id="id9.9.id8.1" class="ltx_sup"><span id="id9.9.id8.1.1" class="ltx_text ltx_font_medium">2</span></sup></span> <span id="id10.10.id9" class="ltx_text ltx_font_bold">,</span> <span id="id11.11.id10" class="ltx_text ltx_font_bold">Feng Sun<sup id="id11.11.id10.1" class="ltx_sup"><span id="id11.11.id10.1.1" class="ltx_text ltx_font_medium">2</span></sup></span> <span id="id12.12.id11" class="ltx_text ltx_font_bold">,</span> <span id="id13.13.id12" class="ltx_text ltx_font_bold">Qi Zhang<sup id="id13.13.id12.1" class="ltx_sup"><span id="id13.13.id12.1.1" class="ltx_text ltx_font_medium">2</span></sup></span> <span id="id14.14.id13" class="ltx_text ltx_font_bold">,</span> <span id="id1.1.1" class="ltx_text ltx_font_bold">Deqing Wang<sup id="id1.1.1.1" class="ltx_sup"><span id="id1.1.1.1.1" class="ltx_text ltx_font_medium">1</span></sup><sup id="id1.1.1.2" class="ltx_sup"><span id="id1.1.1.2.1" class="ltx_text ltx_font_medium">†</span></sup></span> <span id="id15.15.id14" class="ltx_text ltx_font_bold">,</span> <span id="id16.16.id15" class="ltx_text ltx_font_bold">Fuzhen Zhuang<sup id="id16.16.id15.1" class="ltx_sup"><span id="id16.16.id15.1.1" class="ltx_text ltx_font_medium">1</span></sup></span> 
<br class="ltx_break"><sup id="id17.17.id16" class="ltx_sup">1</sup>Beihang University <sup id="id18.18.id17" class="ltx_sup">2</sup>Microsoft Corporation
<br class="ltx_break"><span id="id19.19.id18" class="ltx_text ltx_font_typewriter">royokong@buaa.edu.cn
<br class="ltx_break"></span>
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id20.id1" class="ltx_p">Low-rank adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method for large language models (LLMs).
In this paper, we analyze the impact of low-rank updating, as implemented in LoRA. Our findings suggest that the low-rank updating mechanism may limit the ability of LLMs to effectively learn and memorize new knowledge.
Inspired by this observation, we propose a new method called MoRA, which employs a square matrix to achieve high-rank updating while maintaining the same number of trainable parameters.
To achieve it,
we introduce the corresponding non-parameter operators to reduce the input dimension and increase the output dimension for the square matrix.
Furthermore, these operators ensure that the weight can be merged back into LLMs, which makes our method can be deployed like LoRA.
We perform a comprehensive evaluation of our method across five tasks: instruction tuning, mathematical reasoning, continual pretraining, memory and pretraining. Our method outperforms LoRA on memory-intensive tasks and achieves comparable performance on other tasks.
Our code will be available at <a target="_blank" href="https://github.com/kongds/MoRA" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/kongds/MoRA</a>.</p>
</div>
<div id="p1" class="ltx_para ltx_noindent">
<div id="p1.1" class="ltx_block ltx_align_bottom">
<p id="p1.1.2" class="ltx_p"><span id="p1.1.2.1" class="ltx_text ltx_font_bold">MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning</span></p>
<br class="ltx_break ltx_centering">
<p id="p1.1.1" class="ltx_p ltx_align_center" style="width:433.6pt;"><span id="p1.1.1.1" class="ltx_text ltx_inline-block" style="width:0.0pt;">
<span id="p1.1.1.1.1" class="ltx_tabular ltx_align_top">
<span class="ltx_tbody">
<span id="p1.1.1.1.1.2.1" class="ltx_tr">
<span id="p1.1.1.1.1.2.1.1" class="ltx_td ltx_align_center"><span id="p1.1.1.1.1.2.1.1.1" class="ltx_text ltx_font_bold">
Ting Jiang<sup id="p1.1.1.1.1.2.1.1.1.1" class="ltx_sup"><span id="p1.1.1.1.1.2.1.1.1.1.1" class="ltx_text ltx_font_medium">1</span></sup> , Shaohan Huang<sup id="p1.1.1.1.1.2.1.1.1.2" class="ltx_sup"><span id="p1.1.1.1.1.2.1.1.1.2.1" class="ltx_text ltx_font_medium">2</span></sup> , Shengyue Luo <sup id="p1.1.1.1.1.2.1.1.1.3" class="ltx_sup"><span id="p1.1.1.1.1.2.1.1.1.3.1" class="ltx_text ltx_font_medium">2</span></sup>, Zihan Zhang<sup id="p1.1.1.1.1.2.1.1.1.4" class="ltx_sup"><span id="p1.1.1.1.1.2.1.1.1.4.1" class="ltx_text ltx_font_medium">2</span></sup> , Haizhen Huang<sup id="p1.1.1.1.1.2.1.1.1.5" class="ltx_sup"><span id="p1.1.1.1.1.2.1.1.1.5.1" class="ltx_text ltx_font_medium">2</span></sup></span></span></span>
<span id="p1.1.1.1.1.1" class="ltx_tr">
<span id="p1.1.1.1.1.1.1" class="ltx_td ltx_align_center"><span id="p1.1.1.1.1.1.1.2" class="ltx_text ltx_font_bold">Furu Wei<sup id="p1.1.1.1.1.1.1.2.1" class="ltx_sup"><span id="p1.1.1.1.1.1.1.2.1.1" class="ltx_text ltx_font_medium">2</span></sup></span> <span id="p1.1.1.1.1.1.1.3" class="ltx_text ltx_font_bold">,</span> <span id="p1.1.1.1.1.1.1.4" class="ltx_text ltx_font_bold">Weiwei Deng<sup id="p1.1.1.1.1.1.1.4.1" class="ltx_sup"><span id="p1.1.1.1.1.1.1.4.1.1" class="ltx_text ltx_font_medium">2</span></sup></span> <span id="p1.1.1.1.1.1.1.5" class="ltx_text ltx_font_bold">,</span> <span id="p1.1.1.1.1.1.1.6" class="ltx_text ltx_font_bold">Feng Sun<sup id="p1.1.1.1.1.1.1.6.1" class="ltx_sup"><span id="p1.1.1.1.1.1.1.6.1.1" class="ltx_text ltx_font_medium">2</span></sup></span> <span id="p1.1.1.1.1.1.1.7" class="ltx_text ltx_font_bold">,</span> <span id="p1.1.1.1.1.1.1.8" class="ltx_text ltx_font_bold">Qi Zhang<sup id="p1.1.1.1.1.1.1.8.1" class="ltx_sup"><span id="p1.1.1.1.1.1.1.8.1.1" class="ltx_text ltx_font_medium">2</span></sup></span> <span id="p1.1.1.1.1.1.1.9" class="ltx_text ltx_font_bold">,</span> <span id="p1.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Deqing Wang<sup id="p1.1.1.1.1.1.1.1.1" class="ltx_sup"><span id="p1.1.1.1.1.1.1.1.1.1" class="ltx_text ltx_font_medium">1</span></sup><sup id="p1.1.1.1.1.1.1.1.2" class="ltx_sup"><span id="p1.1.1.1.1.1.1.1.2.1" class="ltx_text ltx_font_medium">†</span></sup></span> <span id="p1.1.1.1.1.1.1.10" class="ltx_text ltx_font_bold">,</span> <span id="p1.1.1.1.1.1.1.11" class="ltx_text ltx_font_bold">Fuzhen Zhuang<sup id="p1.1.1.1.1.1.1.11.1" class="ltx_sup"><span id="p1.1.1.1.1.1.1.11.1.1" class="ltx_text ltx_font_medium">1</span></sup></span></span></span>
<span id="p1.1.1.1.1.3.2" class="ltx_tr">
<span id="p1.1.1.1.1.3.2.1" class="ltx_td ltx_align_center"><sup id="p1.1.1.1.1.3.2.1.1" class="ltx_sup">1</sup>Beihang University <sup id="p1.1.1.1.1.3.2.1.2" class="ltx_sup">2</sup>Microsoft Corporation</span></span>
<span id="p1.1.1.1.1.4.3" class="ltx_tr">
<span id="p1.1.1.1.1.4.3.1" class="ltx_td ltx_align_center"><span id="p1.1.1.1.1.4.3.1.1" class="ltx_text ltx_font_typewriter">royokong@buaa.edu.cn</span></span></span>
</span>
</span></span></p>
<br class="ltx_break ltx_centering">
</div>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">As the size of language models increases, parameter-efficient fine-tuning (PEFT) <cite class="ltx_cite ltx_citemacro_cite">Houlsby et al. (<a href="#bib.bib14" title="" class="ltx_ref">2019</a>)</cite> has emerged as a popular technique to adapt these models to specific downstream tasks.
Compared to Full Fine-Tuning (FFT), which updates all model parameters, PEFT modifies only a small part of the parameters. For example, it can achieve similar performance with FFT by updating less than 1% of the parameters in some tasks <cite class="ltx_cite ltx_citemacro_cite">Hu et al. (<a href="#bib.bib15" title="" class="ltx_ref">2021</a>)</cite>, which significantly reduces the memory requirements for the optimizer and facilitates the storage and deployment of fine-tuned models.</p>
</div>
<figure id="S1.F1" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S1.F1.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2405.12130/assets/lora.png" id="S1.F1.sf1.g1" class="ltx_graphics ltx_img_portrait" width="263" height="397" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span>LoRA (<math id="S1.F1.sf1.2.m1.1" class="ltx_Math" alttext="r=8" display="inline"><semantics id="S1.F1.sf1.2.m1.1b"><mrow id="S1.F1.sf1.2.m1.1.1" xref="S1.F1.sf1.2.m1.1.1.cmml"><mi mathcolor="#1B65CC" id="S1.F1.sf1.2.m1.1.1.2" xref="S1.F1.sf1.2.m1.1.1.2.cmml">r</mi><mo mathcolor="#1B65CC" id="S1.F1.sf1.2.m1.1.1.1" xref="S1.F1.sf1.2.m1.1.1.1.cmml">=</mo><mn mathcolor="#1B65CC" id="S1.F1.sf1.2.m1.1.1.3" xref="S1.F1.sf1.2.m1.1.1.3.cmml">8</mn></mrow><annotation-xml encoding="MathML-Content" id="S1.F1.sf1.2.m1.1c"><apply id="S1.F1.sf1.2.m1.1.1.cmml" xref="S1.F1.sf1.2.m1.1.1"><eq id="S1.F1.sf1.2.m1.1.1.1.cmml" xref="S1.F1.sf1.2.m1.1.1.1"></eq><ci id="S1.F1.sf1.2.m1.1.1.2.cmml" xref="S1.F1.sf1.2.m1.1.1.2">𝑟</ci><cn type="integer" id="S1.F1.sf1.2.m1.1.1.3.cmml" xref="S1.F1.sf1.2.m1.1.1.3">8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.sf1.2.m1.1d">r=8</annotation></semantics></math>)</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S1.F1.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2405.12130/assets/our.png" id="S1.F1.sf2.g1" class="ltx_graphics ltx_img_portrait" width="276" height="389" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span>MoRA (<math id="S1.F1.sf2.2.m1.1" class="ltx_Math" alttext="r=256" display="inline"><semantics id="S1.F1.sf2.2.m1.1b"><mrow id="S1.F1.sf2.2.m1.1.1" xref="S1.F1.sf2.2.m1.1.1.cmml"><mi mathcolor="#1B65CC" id="S1.F1.sf2.2.m1.1.1.2" xref="S1.F1.sf2.2.m1.1.1.2.cmml">r</mi><mo mathcolor="#1B65CC" id="S1.F1.sf2.2.m1.1.1.1" xref="S1.F1.sf2.2.m1.1.1.1.cmml">=</mo><mn mathcolor="#1B65CC" id="S1.F1.sf2.2.m1.1.1.3" xref="S1.F1.sf2.2.m1.1.1.3.cmml">256</mn></mrow><annotation-xml encoding="MathML-Content" id="S1.F1.sf2.2.m1.1c"><apply id="S1.F1.sf2.2.m1.1.1.cmml" xref="S1.F1.sf2.2.m1.1.1"><eq id="S1.F1.sf2.2.m1.1.1.1.cmml" xref="S1.F1.sf2.2.m1.1.1.1"></eq><ci id="S1.F1.sf2.2.m1.1.1.2.cmml" xref="S1.F1.sf2.2.m1.1.1.2">𝑟</ci><cn type="integer" id="S1.F1.sf2.2.m1.1.1.3.cmml" xref="S1.F1.sf2.2.m1.1.1.3">256</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.sf2.2.m1.1d">r=256</annotation></semantics></math>)</figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>
An overview of our method compared to LoRA under <span id="S1.F1.11.1" class="ltx_text ltx_font_bold">same</span> number of trainable parameters.
<math id="S1.F1.5.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S1.F1.5.m1.1b"><mi id="S1.F1.5.m1.1.1" xref="S1.F1.5.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S1.F1.5.m1.1c"><ci id="S1.F1.5.m1.1.1.cmml" xref="S1.F1.5.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.5.m1.1d">W</annotation></semantics></math> is the frozen weight from model. <math id="S1.F1.6.m2.1" class="ltx_Math" alttext="A" display="inline"><semantics id="S1.F1.6.m2.1b"><mi id="S1.F1.6.m2.1.1" xref="S1.F1.6.m2.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="S1.F1.6.m2.1c"><ci id="S1.F1.6.m2.1.1.cmml" xref="S1.F1.6.m2.1.1">𝐴</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.6.m2.1d">A</annotation></semantics></math> and <math id="S1.F1.7.m3.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S1.F1.7.m3.1b"><mi id="S1.F1.7.m3.1.1" xref="S1.F1.7.m3.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S1.F1.7.m3.1c"><ci id="S1.F1.7.m3.1.1.cmml" xref="S1.F1.7.m3.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.7.m3.1d">B</annotation></semantics></math> are trainable low-rank matrices in LoRA.
<math id="S1.F1.8.m4.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S1.F1.8.m4.1b"><mi id="S1.F1.8.m4.1.1" xref="S1.F1.8.m4.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S1.F1.8.m4.1c"><ci id="S1.F1.8.m4.1.1.cmml" xref="S1.F1.8.m4.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.8.m4.1d">M</annotation></semantics></math> is the trainable matrix in our method.
Gray parts are non-parameter operators to reducing the input dimension and increasing the output dimension.
<span id="S1.F1.12.2" class="ltx_text" style="color:#1B65CC;">r</span> represents the rank in two methods.
</figcaption>
</figure>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">Among the existing PEFT methods, Low-Rank Adaptation (LoRA) <cite class="ltx_cite ltx_citemacro_cite">Hu et al. (<a href="#bib.bib15" title="" class="ltx_ref">2021</a>)</cite> is particularly prevalent for LLMs. LoRA enhances performance over other PEFT methods such as prompt tuning <cite class="ltx_cite ltx_citemacro_cite">Lester et al. (<a href="#bib.bib21" title="" class="ltx_ref">2021</a>)</cite> or adapters <cite class="ltx_cite ltx_citemacro_cite">Houlsby et al. (<a href="#bib.bib14" title="" class="ltx_ref">2019</a>)</cite> by updating parameters via low-rank matrices. These matrices can be merged into the original model parameters, thereby avoiding additional computational costs during inference. There are numerous methods that aim to improve LoRA for LLMs. However,
most methods primarily validate their efficiency based on GLUE <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib35" title="" class="ltx_ref">2018</a>)</cite>, either by achieving better performance or by requiring fewer trainable parameters.
Recent methods <cite class="ltx_cite ltx_citemacro_cite">Liu et al. (<a href="#bib.bib24" title="" class="ltx_ref">2024</a>); Meng et al. (<a href="#bib.bib27" title="" class="ltx_ref">2024</a>); Zhu et al. (<a href="#bib.bib41" title="" class="ltx_ref">2024</a>)</cite> leverage instruction tuning task such as Alpaca <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib36" title="" class="ltx_ref">2024</a>)</cite> or reasoning tasks like GSM8K <cite class="ltx_cite ltx_citemacro_cite">Cobbe et al. (<a href="#bib.bib4" title="" class="ltx_ref">2021</a>)</cite> to better evaluate their performance on LLMs.
However, the diverse settings and datasets used in the evaluation complicate the understanding of their progression.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">In this paper, we conduct a comprehensive evaluation of LoRA across various tasks under the same settings, including instruction tuning, mathematical reasoning, and continual pretraining. We find that LoRA-like methods demonstrate similar performance across these tasks and they perform comparably to FFT in instruction tuning but fall short in mathematical reasoning and continual pretraining. Among these tasks, instruction tuning primarily focuses on interacting with the format, rather than acquiring knowledge and capabilities, which are learned almost entirely during pretraining <cite class="ltx_cite ltx_citemacro_cite">Zhou et al. (<a href="#bib.bib40" title="" class="ltx_ref">2024</a>)</cite>. We observe that LoRA is easily adapted to follow response formats in instruction tuning but struggles with other tasks that require enhancing knowledge and capabilities through fine-tuning.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.2" class="ltx_p">One plausible explanation for this limitation observed with LoRA could be its reliance on low-rank updates <cite class="ltx_cite ltx_citemacro_cite">Lialin et al. (<a href="#bib.bib22" title="" class="ltx_ref">2023</a>)</cite>. The low-rank update matrix, <math id="S1.p4.1.m1.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S1.p4.1.m1.1a"><mrow id="S1.p4.1.m1.1.1" xref="S1.p4.1.m1.1.1.cmml"><mi mathvariant="normal" id="S1.p4.1.m1.1.1.2" xref="S1.p4.1.m1.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S1.p4.1.m1.1.1.1" xref="S1.p4.1.m1.1.1.1.cmml">​</mo><mi id="S1.p4.1.m1.1.1.3" xref="S1.p4.1.m1.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S1.p4.1.m1.1b"><apply id="S1.p4.1.m1.1.1.cmml" xref="S1.p4.1.m1.1.1"><times id="S1.p4.1.m1.1.1.1.cmml" xref="S1.p4.1.m1.1.1.1"></times><ci id="S1.p4.1.m1.1.1.2.cmml" xref="S1.p4.1.m1.1.1.2">Δ</ci><ci id="S1.p4.1.m1.1.1.3.cmml" xref="S1.p4.1.m1.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p4.1.m1.1c">\Delta W</annotation></semantics></math>, struggles to estimate the full-rank updates in FFT, particularly in memory-intensive tasks like continual pretraining that require memorizing domain-specific knowledge.
Since the rank of <math id="S1.p4.2.m2.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S1.p4.2.m2.1a"><mrow id="S1.p4.2.m2.1.1" xref="S1.p4.2.m2.1.1.cmml"><mi mathvariant="normal" id="S1.p4.2.m2.1.1.2" xref="S1.p4.2.m2.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S1.p4.2.m2.1.1.1" xref="S1.p4.2.m2.1.1.1.cmml">​</mo><mi id="S1.p4.2.m2.1.1.3" xref="S1.p4.2.m2.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S1.p4.2.m2.1b"><apply id="S1.p4.2.m2.1.1.cmml" xref="S1.p4.2.m2.1.1"><times id="S1.p4.2.m2.1.1.1.cmml" xref="S1.p4.2.m2.1.1.1"></times><ci id="S1.p4.2.m2.1.1.2.cmml" xref="S1.p4.2.m2.1.1.2">Δ</ci><ci id="S1.p4.2.m2.1.1.3.cmml" xref="S1.p4.2.m2.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p4.2.m2.1c">\Delta W</annotation></semantics></math> is significantly smaller than the full rank, this limitation restricts capacity to store new information via fine-tuning. Moreover, current variants of LoRA cannot alter the inherent characteristic of low-rank updates.
To validate this, we conducted a memorization task using pseudo-data to assess the performance of LoRA in memorizing new knowledge. We found that LoRA performed significantly worse than FFT, even with a large rank such as 256.
</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.9" class="ltx_p">Given these observations, we introduce a method called MoRA, which employs a square matrix as opposed to low-rank matrices, aiming to maximize the rank in <math id="S1.p5.1.m1.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S1.p5.1.m1.1a"><mrow id="S1.p5.1.m1.1.1" xref="S1.p5.1.m1.1.1.cmml"><mi mathvariant="normal" id="S1.p5.1.m1.1.1.2" xref="S1.p5.1.m1.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S1.p5.1.m1.1.1.1" xref="S1.p5.1.m1.1.1.1.cmml">​</mo><mi id="S1.p5.1.m1.1.1.3" xref="S1.p5.1.m1.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S1.p5.1.m1.1b"><apply id="S1.p5.1.m1.1.1.cmml" xref="S1.p5.1.m1.1.1"><times id="S1.p5.1.m1.1.1.1.cmml" xref="S1.p5.1.m1.1.1.1"></times><ci id="S1.p5.1.m1.1.1.2.cmml" xref="S1.p5.1.m1.1.1.2">Δ</ci><ci id="S1.p5.1.m1.1.1.3.cmml" xref="S1.p5.1.m1.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p5.1.m1.1c">\Delta W</annotation></semantics></math> while maintaining the same number of trainable parameters. For instance, when utilizing 8 rank with the hidden size 4096, LoRA employs two low-rank matrices <math id="S1.p5.2.m2.1" class="ltx_Math" alttext="A\in\mathbb{R}^{4096\times 8}" display="inline"><semantics id="S1.p5.2.m2.1a"><mrow id="S1.p5.2.m2.1.1" xref="S1.p5.2.m2.1.1.cmml"><mi id="S1.p5.2.m2.1.1.2" xref="S1.p5.2.m2.1.1.2.cmml">A</mi><mo id="S1.p5.2.m2.1.1.1" xref="S1.p5.2.m2.1.1.1.cmml">∈</mo><msup id="S1.p5.2.m2.1.1.3" xref="S1.p5.2.m2.1.1.3.cmml"><mi id="S1.p5.2.m2.1.1.3.2" xref="S1.p5.2.m2.1.1.3.2.cmml">ℝ</mi><mrow id="S1.p5.2.m2.1.1.3.3" xref="S1.p5.2.m2.1.1.3.3.cmml"><mn id="S1.p5.2.m2.1.1.3.3.2" xref="S1.p5.2.m2.1.1.3.3.2.cmml">4096</mn><mo lspace="0.222em" rspace="0.222em" id="S1.p5.2.m2.1.1.3.3.1" xref="S1.p5.2.m2.1.1.3.3.1.cmml">×</mo><mn id="S1.p5.2.m2.1.1.3.3.3" xref="S1.p5.2.m2.1.1.3.3.3.cmml">8</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S1.p5.2.m2.1b"><apply id="S1.p5.2.m2.1.1.cmml" xref="S1.p5.2.m2.1.1"><in id="S1.p5.2.m2.1.1.1.cmml" xref="S1.p5.2.m2.1.1.1"></in><ci id="S1.p5.2.m2.1.1.2.cmml" xref="S1.p5.2.m2.1.1.2">𝐴</ci><apply id="S1.p5.2.m2.1.1.3.cmml" xref="S1.p5.2.m2.1.1.3"><csymbol cd="ambiguous" id="S1.p5.2.m2.1.1.3.1.cmml" xref="S1.p5.2.m2.1.1.3">superscript</csymbol><ci id="S1.p5.2.m2.1.1.3.2.cmml" xref="S1.p5.2.m2.1.1.3.2">ℝ</ci><apply id="S1.p5.2.m2.1.1.3.3.cmml" xref="S1.p5.2.m2.1.1.3.3"><times id="S1.p5.2.m2.1.1.3.3.1.cmml" xref="S1.p5.2.m2.1.1.3.3.1"></times><cn type="integer" id="S1.p5.2.m2.1.1.3.3.2.cmml" xref="S1.p5.2.m2.1.1.3.3.2">4096</cn><cn type="integer" id="S1.p5.2.m2.1.1.3.3.3.cmml" xref="S1.p5.2.m2.1.1.3.3.3">8</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p5.2.m2.1c">A\in\mathbb{R}^{4096\times 8}</annotation></semantics></math> and <math id="S1.p5.3.m3.1" class="ltx_Math" alttext="B\in\mathbb{R}^{8\times 4096}" display="inline"><semantics id="S1.p5.3.m3.1a"><mrow id="S1.p5.3.m3.1.1" xref="S1.p5.3.m3.1.1.cmml"><mi id="S1.p5.3.m3.1.1.2" xref="S1.p5.3.m3.1.1.2.cmml">B</mi><mo id="S1.p5.3.m3.1.1.1" xref="S1.p5.3.m3.1.1.1.cmml">∈</mo><msup id="S1.p5.3.m3.1.1.3" xref="S1.p5.3.m3.1.1.3.cmml"><mi id="S1.p5.3.m3.1.1.3.2" xref="S1.p5.3.m3.1.1.3.2.cmml">ℝ</mi><mrow id="S1.p5.3.m3.1.1.3.3" xref="S1.p5.3.m3.1.1.3.3.cmml"><mn id="S1.p5.3.m3.1.1.3.3.2" xref="S1.p5.3.m3.1.1.3.3.2.cmml">8</mn><mo lspace="0.222em" rspace="0.222em" id="S1.p5.3.m3.1.1.3.3.1" xref="S1.p5.3.m3.1.1.3.3.1.cmml">×</mo><mn id="S1.p5.3.m3.1.1.3.3.3" xref="S1.p5.3.m3.1.1.3.3.3.cmml">4096</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S1.p5.3.m3.1b"><apply id="S1.p5.3.m3.1.1.cmml" xref="S1.p5.3.m3.1.1"><in id="S1.p5.3.m3.1.1.1.cmml" xref="S1.p5.3.m3.1.1.1"></in><ci id="S1.p5.3.m3.1.1.2.cmml" xref="S1.p5.3.m3.1.1.2">𝐵</ci><apply id="S1.p5.3.m3.1.1.3.cmml" xref="S1.p5.3.m3.1.1.3"><csymbol cd="ambiguous" id="S1.p5.3.m3.1.1.3.1.cmml" xref="S1.p5.3.m3.1.1.3">superscript</csymbol><ci id="S1.p5.3.m3.1.1.3.2.cmml" xref="S1.p5.3.m3.1.1.3.2">ℝ</ci><apply id="S1.p5.3.m3.1.1.3.3.cmml" xref="S1.p5.3.m3.1.1.3.3"><times id="S1.p5.3.m3.1.1.3.3.1.cmml" xref="S1.p5.3.m3.1.1.3.3.1"></times><cn type="integer" id="S1.p5.3.m3.1.1.3.3.2.cmml" xref="S1.p5.3.m3.1.1.3.3.2">8</cn><cn type="integer" id="S1.p5.3.m3.1.1.3.3.3.cmml" xref="S1.p5.3.m3.1.1.3.3.3">4096</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p5.3.m3.1c">B\in\mathbb{R}^{8\times 4096}</annotation></semantics></math>, with <math id="S1.p5.4.m4.1" class="ltx_Math" alttext="rank(\Delta W)\leq 8" display="inline"><semantics id="S1.p5.4.m4.1a"><mrow id="S1.p5.4.m4.1.1" xref="S1.p5.4.m4.1.1.cmml"><mrow id="S1.p5.4.m4.1.1.1" xref="S1.p5.4.m4.1.1.1.cmml"><mi id="S1.p5.4.m4.1.1.1.3" xref="S1.p5.4.m4.1.1.1.3.cmml">r</mi><mo lspace="0em" rspace="0em" id="S1.p5.4.m4.1.1.1.2" xref="S1.p5.4.m4.1.1.1.2.cmml">​</mo><mi id="S1.p5.4.m4.1.1.1.4" xref="S1.p5.4.m4.1.1.1.4.cmml">a</mi><mo lspace="0em" rspace="0em" id="S1.p5.4.m4.1.1.1.2a" xref="S1.p5.4.m4.1.1.1.2.cmml">​</mo><mi id="S1.p5.4.m4.1.1.1.5" xref="S1.p5.4.m4.1.1.1.5.cmml">n</mi><mo lspace="0em" rspace="0em" id="S1.p5.4.m4.1.1.1.2b" xref="S1.p5.4.m4.1.1.1.2.cmml">​</mo><mi id="S1.p5.4.m4.1.1.1.6" xref="S1.p5.4.m4.1.1.1.6.cmml">k</mi><mo lspace="0em" rspace="0em" id="S1.p5.4.m4.1.1.1.2c" xref="S1.p5.4.m4.1.1.1.2.cmml">​</mo><mrow id="S1.p5.4.m4.1.1.1.1.1" xref="S1.p5.4.m4.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S1.p5.4.m4.1.1.1.1.1.2" xref="S1.p5.4.m4.1.1.1.1.1.1.cmml">(</mo><mrow id="S1.p5.4.m4.1.1.1.1.1.1" xref="S1.p5.4.m4.1.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S1.p5.4.m4.1.1.1.1.1.1.2" xref="S1.p5.4.m4.1.1.1.1.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S1.p5.4.m4.1.1.1.1.1.1.1" xref="S1.p5.4.m4.1.1.1.1.1.1.1.cmml">​</mo><mi id="S1.p5.4.m4.1.1.1.1.1.1.3" xref="S1.p5.4.m4.1.1.1.1.1.1.3.cmml">W</mi></mrow><mo stretchy="false" id="S1.p5.4.m4.1.1.1.1.1.3" xref="S1.p5.4.m4.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S1.p5.4.m4.1.1.2" xref="S1.p5.4.m4.1.1.2.cmml">≤</mo><mn id="S1.p5.4.m4.1.1.3" xref="S1.p5.4.m4.1.1.3.cmml">8</mn></mrow><annotation-xml encoding="MathML-Content" id="S1.p5.4.m4.1b"><apply id="S1.p5.4.m4.1.1.cmml" xref="S1.p5.4.m4.1.1"><leq id="S1.p5.4.m4.1.1.2.cmml" xref="S1.p5.4.m4.1.1.2"></leq><apply id="S1.p5.4.m4.1.1.1.cmml" xref="S1.p5.4.m4.1.1.1"><times id="S1.p5.4.m4.1.1.1.2.cmml" xref="S1.p5.4.m4.1.1.1.2"></times><ci id="S1.p5.4.m4.1.1.1.3.cmml" xref="S1.p5.4.m4.1.1.1.3">𝑟</ci><ci id="S1.p5.4.m4.1.1.1.4.cmml" xref="S1.p5.4.m4.1.1.1.4">𝑎</ci><ci id="S1.p5.4.m4.1.1.1.5.cmml" xref="S1.p5.4.m4.1.1.1.5">𝑛</ci><ci id="S1.p5.4.m4.1.1.1.6.cmml" xref="S1.p5.4.m4.1.1.1.6">𝑘</ci><apply id="S1.p5.4.m4.1.1.1.1.1.1.cmml" xref="S1.p5.4.m4.1.1.1.1.1"><times id="S1.p5.4.m4.1.1.1.1.1.1.1.cmml" xref="S1.p5.4.m4.1.1.1.1.1.1.1"></times><ci id="S1.p5.4.m4.1.1.1.1.1.1.2.cmml" xref="S1.p5.4.m4.1.1.1.1.1.1.2">Δ</ci><ci id="S1.p5.4.m4.1.1.1.1.1.1.3.cmml" xref="S1.p5.4.m4.1.1.1.1.1.1.3">𝑊</ci></apply></apply><cn type="integer" id="S1.p5.4.m4.1.1.3.cmml" xref="S1.p5.4.m4.1.1.3">8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p5.4.m4.1c">rank(\Delta W)\leq 8</annotation></semantics></math>. Under same number of parameters, our method uses a square matrix <math id="S1.p5.5.m5.1" class="ltx_Math" alttext="M\in\mathbb{R}^{256\times 256}" display="inline"><semantics id="S1.p5.5.m5.1a"><mrow id="S1.p5.5.m5.1.1" xref="S1.p5.5.m5.1.1.cmml"><mi id="S1.p5.5.m5.1.1.2" xref="S1.p5.5.m5.1.1.2.cmml">M</mi><mo id="S1.p5.5.m5.1.1.1" xref="S1.p5.5.m5.1.1.1.cmml">∈</mo><msup id="S1.p5.5.m5.1.1.3" xref="S1.p5.5.m5.1.1.3.cmml"><mi id="S1.p5.5.m5.1.1.3.2" xref="S1.p5.5.m5.1.1.3.2.cmml">ℝ</mi><mrow id="S1.p5.5.m5.1.1.3.3" xref="S1.p5.5.m5.1.1.3.3.cmml"><mn id="S1.p5.5.m5.1.1.3.3.2" xref="S1.p5.5.m5.1.1.3.3.2.cmml">256</mn><mo lspace="0.222em" rspace="0.222em" id="S1.p5.5.m5.1.1.3.3.1" xref="S1.p5.5.m5.1.1.3.3.1.cmml">×</mo><mn id="S1.p5.5.m5.1.1.3.3.3" xref="S1.p5.5.m5.1.1.3.3.3.cmml">256</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S1.p5.5.m5.1b"><apply id="S1.p5.5.m5.1.1.cmml" xref="S1.p5.5.m5.1.1"><in id="S1.p5.5.m5.1.1.1.cmml" xref="S1.p5.5.m5.1.1.1"></in><ci id="S1.p5.5.m5.1.1.2.cmml" xref="S1.p5.5.m5.1.1.2">𝑀</ci><apply id="S1.p5.5.m5.1.1.3.cmml" xref="S1.p5.5.m5.1.1.3"><csymbol cd="ambiguous" id="S1.p5.5.m5.1.1.3.1.cmml" xref="S1.p5.5.m5.1.1.3">superscript</csymbol><ci id="S1.p5.5.m5.1.1.3.2.cmml" xref="S1.p5.5.m5.1.1.3.2">ℝ</ci><apply id="S1.p5.5.m5.1.1.3.3.cmml" xref="S1.p5.5.m5.1.1.3.3"><times id="S1.p5.5.m5.1.1.3.3.1.cmml" xref="S1.p5.5.m5.1.1.3.3.1"></times><cn type="integer" id="S1.p5.5.m5.1.1.3.3.2.cmml" xref="S1.p5.5.m5.1.1.3.3.2">256</cn><cn type="integer" id="S1.p5.5.m5.1.1.3.3.3.cmml" xref="S1.p5.5.m5.1.1.3.3.3">256</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p5.5.m5.1c">M\in\mathbb{R}^{256\times 256}</annotation></semantics></math>, with <math id="S1.p5.6.m6.1" class="ltx_Math" alttext="rank(\Delta W)\leq 256" display="inline"><semantics id="S1.p5.6.m6.1a"><mrow id="S1.p5.6.m6.1.1" xref="S1.p5.6.m6.1.1.cmml"><mrow id="S1.p5.6.m6.1.1.1" xref="S1.p5.6.m6.1.1.1.cmml"><mi id="S1.p5.6.m6.1.1.1.3" xref="S1.p5.6.m6.1.1.1.3.cmml">r</mi><mo lspace="0em" rspace="0em" id="S1.p5.6.m6.1.1.1.2" xref="S1.p5.6.m6.1.1.1.2.cmml">​</mo><mi id="S1.p5.6.m6.1.1.1.4" xref="S1.p5.6.m6.1.1.1.4.cmml">a</mi><mo lspace="0em" rspace="0em" id="S1.p5.6.m6.1.1.1.2a" xref="S1.p5.6.m6.1.1.1.2.cmml">​</mo><mi id="S1.p5.6.m6.1.1.1.5" xref="S1.p5.6.m6.1.1.1.5.cmml">n</mi><mo lspace="0em" rspace="0em" id="S1.p5.6.m6.1.1.1.2b" xref="S1.p5.6.m6.1.1.1.2.cmml">​</mo><mi id="S1.p5.6.m6.1.1.1.6" xref="S1.p5.6.m6.1.1.1.6.cmml">k</mi><mo lspace="0em" rspace="0em" id="S1.p5.6.m6.1.1.1.2c" xref="S1.p5.6.m6.1.1.1.2.cmml">​</mo><mrow id="S1.p5.6.m6.1.1.1.1.1" xref="S1.p5.6.m6.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S1.p5.6.m6.1.1.1.1.1.2" xref="S1.p5.6.m6.1.1.1.1.1.1.cmml">(</mo><mrow id="S1.p5.6.m6.1.1.1.1.1.1" xref="S1.p5.6.m6.1.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S1.p5.6.m6.1.1.1.1.1.1.2" xref="S1.p5.6.m6.1.1.1.1.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S1.p5.6.m6.1.1.1.1.1.1.1" xref="S1.p5.6.m6.1.1.1.1.1.1.1.cmml">​</mo><mi id="S1.p5.6.m6.1.1.1.1.1.1.3" xref="S1.p5.6.m6.1.1.1.1.1.1.3.cmml">W</mi></mrow><mo stretchy="false" id="S1.p5.6.m6.1.1.1.1.1.3" xref="S1.p5.6.m6.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S1.p5.6.m6.1.1.2" xref="S1.p5.6.m6.1.1.2.cmml">≤</mo><mn id="S1.p5.6.m6.1.1.3" xref="S1.p5.6.m6.1.1.3.cmml">256</mn></mrow><annotation-xml encoding="MathML-Content" id="S1.p5.6.m6.1b"><apply id="S1.p5.6.m6.1.1.cmml" xref="S1.p5.6.m6.1.1"><leq id="S1.p5.6.m6.1.1.2.cmml" xref="S1.p5.6.m6.1.1.2"></leq><apply id="S1.p5.6.m6.1.1.1.cmml" xref="S1.p5.6.m6.1.1.1"><times id="S1.p5.6.m6.1.1.1.2.cmml" xref="S1.p5.6.m6.1.1.1.2"></times><ci id="S1.p5.6.m6.1.1.1.3.cmml" xref="S1.p5.6.m6.1.1.1.3">𝑟</ci><ci id="S1.p5.6.m6.1.1.1.4.cmml" xref="S1.p5.6.m6.1.1.1.4">𝑎</ci><ci id="S1.p5.6.m6.1.1.1.5.cmml" xref="S1.p5.6.m6.1.1.1.5">𝑛</ci><ci id="S1.p5.6.m6.1.1.1.6.cmml" xref="S1.p5.6.m6.1.1.1.6">𝑘</ci><apply id="S1.p5.6.m6.1.1.1.1.1.1.cmml" xref="S1.p5.6.m6.1.1.1.1.1"><times id="S1.p5.6.m6.1.1.1.1.1.1.1.cmml" xref="S1.p5.6.m6.1.1.1.1.1.1.1"></times><ci id="S1.p5.6.m6.1.1.1.1.1.1.2.cmml" xref="S1.p5.6.m6.1.1.1.1.1.1.2">Δ</ci><ci id="S1.p5.6.m6.1.1.1.1.1.1.3.cmml" xref="S1.p5.6.m6.1.1.1.1.1.1.3">𝑊</ci></apply></apply><cn type="integer" id="S1.p5.6.m6.1.1.3.cmml" xref="S1.p5.6.m6.1.1.3">256</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p5.6.m6.1c">rank(\Delta W)\leq 256</annotation></semantics></math>, as depicted in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.
Notably, our method exhibits a greater capacity than LoRA with a large rank.
To decrease the input dimension and increase the output dimension for <math id="S1.p5.7.m7.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S1.p5.7.m7.1a"><mi id="S1.p5.7.m7.1.1" xref="S1.p5.7.m7.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S1.p5.7.m7.1b"><ci id="S1.p5.7.m7.1.1.cmml" xref="S1.p5.7.m7.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p5.7.m7.1c">M</annotation></semantics></math>, we develop corresponding non-parameter operators.
Furthermore, these operators and <math id="S1.p5.8.m8.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S1.p5.8.m8.1a"><mi id="S1.p5.8.m8.1.1" xref="S1.p5.8.m8.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S1.p5.8.m8.1b"><ci id="S1.p5.8.m8.1.1.cmml" xref="S1.p5.8.m8.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.p5.8.m8.1c">M</annotation></semantics></math> can be substituted by a <math id="S1.p5.9.m9.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S1.p5.9.m9.1a"><mrow id="S1.p5.9.m9.1.1" xref="S1.p5.9.m9.1.1.cmml"><mi mathvariant="normal" id="S1.p5.9.m9.1.1.2" xref="S1.p5.9.m9.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S1.p5.9.m9.1.1.1" xref="S1.p5.9.m9.1.1.1.cmml">​</mo><mi id="S1.p5.9.m9.1.1.3" xref="S1.p5.9.m9.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S1.p5.9.m9.1b"><apply id="S1.p5.9.m9.1.1.cmml" xref="S1.p5.9.m9.1.1"><times id="S1.p5.9.m9.1.1.1.cmml" xref="S1.p5.9.m9.1.1.1"></times><ci id="S1.p5.9.m9.1.1.2.cmml" xref="S1.p5.9.m9.1.1.2">Δ</ci><ci id="S1.p5.9.m9.1.1.3.cmml" xref="S1.p5.9.m9.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p5.9.m9.1c">\Delta W</annotation></semantics></math>, ensuring our method can be merged back into LLM like LoRA.</p>
</div>
<div id="S1.p6" class="ltx_para">
<p id="S1.p6.1" class="ltx_p">Our contributions are as follows:</p>
</div>
<div id="S1.p7" class="ltx_para">
<ol id="S1.I1" class="ltx_enumerate">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p">We introduce MoRA, a novel method that employs a square matrix instead of low-rank matrices in LoRA to achieve high-rank updating, while maintaining the same number of trainable parameters.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p">We discuss four kinds of non-parameter operators of MoRA to reduce the input dimension and increase the output dimension for the square matrix, while ensures that the weight can be merged back into LLMs.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.1" class="ltx_p">We evaluate MoRA across five tasks: memory, instruction tuning, mathematical reasoning, continual pretraining, and pretraining. Our method outperforms LoRA on memory-intensive tasks and achieves comparable performance on other tasks, which demonstrates the effectiveness of high-rank updating.</p>
</div>
</li>
</ol>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>LoRA</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.2" class="ltx_p">LoRA is one of the most popular PEFT methods for fine-tuning LLM, owing to its broad applicability and robust performance in comparison to other methods.
To approximate the updated weight <math id="S2.SS1.p1.1.m1.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S2.SS1.p1.1.m1.1a"><mrow id="S2.SS1.p1.1.m1.1.1" xref="S2.SS1.p1.1.m1.1.1.cmml"><mi mathvariant="normal" id="S2.SS1.p1.1.m1.1.1.2" xref="S2.SS1.p1.1.m1.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S2.SS1.p1.1.m1.1.1.1" xref="S2.SS1.p1.1.m1.1.1.1.cmml">​</mo><mi id="S2.SS1.p1.1.m1.1.1.3" xref="S2.SS1.p1.1.m1.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.1.m1.1b"><apply id="S2.SS1.p1.1.m1.1.1.cmml" xref="S2.SS1.p1.1.m1.1.1"><times id="S2.SS1.p1.1.m1.1.1.1.cmml" xref="S2.SS1.p1.1.m1.1.1.1"></times><ci id="S2.SS1.p1.1.m1.1.1.2.cmml" xref="S2.SS1.p1.1.m1.1.1.2">Δ</ci><ci id="S2.SS1.p1.1.m1.1.1.3.cmml" xref="S2.SS1.p1.1.m1.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.1.m1.1c">\Delta W</annotation></semantics></math> in FFT, LoRA employs two low-rank matrices for its decomposition. By adjusting the rank of these two matrices, LoRA can accordingly modify the trainable parameters. Benefit from it, LoRA can merge these matrices after fine-tuning without the inference latency compared to FFT.
There are many methods to further improve LoRA, particularly for the application in LLMs. DoRA<cite class="ltx_cite ltx_citemacro_cite">Liu et al. (<a href="#bib.bib24" title="" class="ltx_ref">2024</a>)</cite> further decomposes the original weight into magnitude and direction components and uses LoRA to update the direction component. LoRA+<cite class="ltx_cite ltx_citemacro_cite">Hayou et al. (<a href="#bib.bib11" title="" class="ltx_ref">2024</a>)</cite> employs different learning rates for the two low-rank matrices to improve learning efficiency. ReLoRA<cite class="ltx_cite ltx_citemacro_cite">Lialin et al. (<a href="#bib.bib22" title="" class="ltx_ref">2023</a>)</cite> integrates LoRA into the LLM during training to increase the rank of the final <math id="S2.SS1.p1.2.m2.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S2.SS1.p1.2.m2.1a"><mrow id="S2.SS1.p1.2.m2.1.1" xref="S2.SS1.p1.2.m2.1.1.cmml"><mi mathvariant="normal" id="S2.SS1.p1.2.m2.1.1.2" xref="S2.SS1.p1.2.m2.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S2.SS1.p1.2.m2.1.1.1" xref="S2.SS1.p1.2.m2.1.1.1.cmml">​</mo><mi id="S2.SS1.p1.2.m2.1.1.3" xref="S2.SS1.p1.2.m2.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.2.m2.1b"><apply id="S2.SS1.p1.2.m2.1.1.cmml" xref="S2.SS1.p1.2.m2.1.1"><times id="S2.SS1.p1.2.m2.1.1.1.cmml" xref="S2.SS1.p1.2.m2.1.1.1"></times><ci id="S2.SS1.p1.2.m2.1.1.2.cmml" xref="S2.SS1.p1.2.m2.1.1.2">Δ</ci><ci id="S2.SS1.p1.2.m2.1.1.3.cmml" xref="S2.SS1.p1.2.m2.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.2.m2.1c">\Delta W</annotation></semantics></math>.</p>
</div>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Fine-Tuning with LLMs</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.1" class="ltx_p">Despite the impressive performance of LLMs with in-context learning, certain scenarios still necessitate fine-tuning, which can be broadly categorized into three types.
The first type, instruction tuning, aims to better align LLMs with end tasks and user preferences, without significantly enhancing the knowledge and capabilities of LLMs <cite class="ltx_cite ltx_citemacro_cite">Zhou et al. (<a href="#bib.bib40" title="" class="ltx_ref">2024</a>)</cite>. This approach simplifies the process of dealing with varied tasks and understanding complex instructions.
The second type involves complex reasoning tasks such as mathematical problem-solving <cite class="ltx_cite ltx_citemacro_cite">Collins et al. (<a href="#bib.bib5" title="" class="ltx_ref">2023</a>); Imani et al. (<a href="#bib.bib16" title="" class="ltx_ref">2023</a>); Yu et al. (<a href="#bib.bib38" title="" class="ltx_ref">2023</a>)</cite>, where general instruction tuning often falls short in handling complex, symbolic, multi-step reasoning tasks. To improve the reasoning abilities of LLMs, the majority of research focuses on creating corresponding training datasets, either by leveraging larger teacher models like GPT-4 <cite class="ltx_cite ltx_citemacro_cite">Fu et al. (<a href="#bib.bib8" title="" class="ltx_ref">2023</a>)</cite>, or by rephrasing questions along a reasoning path <cite class="ltx_cite ltx_citemacro_cite">Yu et al. (<a href="#bib.bib38" title="" class="ltx_ref">2023</a>)</cite>.
The third type, continual pretraining <cite class="ltx_cite ltx_citemacro_cite">Cheng et al. (<a href="#bib.bib3" title="" class="ltx_ref">2023</a>); Chen et al. (<a href="#bib.bib1" title="" class="ltx_ref">2023</a>); Han et al. (<a href="#bib.bib10" title="" class="ltx_ref">2023</a>); Liu et al. (<a href="#bib.bib23" title="" class="ltx_ref">2023</a>)</cite>, aims to enhance the domain-specific capabilities of LLMs. Unlike instruction tuning, it necessitates fine-tuning to augment the corresponding domain-specific knowledge and capabilities.</p>
</div>
<div id="S2.SS2.p2" class="ltx_para">
<p id="S2.SS2.p2.1" class="ltx_p">However, most variants of LoRA <cite class="ltx_cite ltx_citemacro_cite">Kopiczko et al. (<a href="#bib.bib20" title="" class="ltx_ref">2023</a>); Lialin et al. (<a href="#bib.bib22" title="" class="ltx_ref">2023</a>); Dettmers et al. (<a href="#bib.bib7" title="" class="ltx_ref">2024</a>); Zhu et al. (<a href="#bib.bib41" title="" class="ltx_ref">2024</a>)</cite> predominantly employ instruction tuning or text classification tasks from GLUE <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib35" title="" class="ltx_ref">2018</a>)</cite> to validate their efficacy on LLMs. Given that instruction tuning requires the least capacity for fine-tuning compared to other types, it may not accurately reflect the effectiveness of LoRA variants.
To better evaluate their methods, recent works <cite class="ltx_cite ltx_citemacro_cite">Meng et al. (<a href="#bib.bib27" title="" class="ltx_ref">2024</a>); Liu et al. (<a href="#bib.bib24" title="" class="ltx_ref">2024</a>); Shi et al. (<a href="#bib.bib32" title="" class="ltx_ref">2024</a>); Renduchintala et al. (<a href="#bib.bib29" title="" class="ltx_ref">2023</a>)</cite> have employed reasoning tasks to test their methods.
But the training sets used are often too small for LLMs to effectively learn reasoning. For instance, some methods <cite class="ltx_cite ltx_citemacro_cite">Meng et al. (<a href="#bib.bib27" title="" class="ltx_ref">2024</a>); Renduchintala et al. (<a href="#bib.bib29" title="" class="ltx_ref">2023</a>)</cite> utilize the GSM8K <cite class="ltx_cite ltx_citemacro_cite">Cobbe et al. (<a href="#bib.bib4" title="" class="ltx_ref">2021</a>)</cite> with only 7.5K training samples. Compare to the SOTA method with 395K training samples <cite class="ltx_cite ltx_citemacro_cite">Yu et al. (<a href="#bib.bib38" title="" class="ltx_ref">2023</a>)</cite>, this small training set achieves worse performance on reasoning and makes it hard to evaluate the effectiveness of these methods.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Analysis the Influence of Low-rank Updating</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.2" class="ltx_p">The key idea of LoRA <cite class="ltx_cite ltx_citemacro_cite">Hu et al. (<a href="#bib.bib15" title="" class="ltx_ref">2021</a>)</cite> involves the use of low-rank updates to estimate full-rank updates in FFT. Formally, given a pretrained parameter matrix <math id="S3.p1.1.m1.1" class="ltx_Math" alttext="W_{0}\in\mathbb{R}^{d\times k}" display="inline"><semantics id="S3.p1.1.m1.1a"><mrow id="S3.p1.1.m1.1.1" xref="S3.p1.1.m1.1.1.cmml"><msub id="S3.p1.1.m1.1.1.2" xref="S3.p1.1.m1.1.1.2.cmml"><mi id="S3.p1.1.m1.1.1.2.2" xref="S3.p1.1.m1.1.1.2.2.cmml">W</mi><mn id="S3.p1.1.m1.1.1.2.3" xref="S3.p1.1.m1.1.1.2.3.cmml">0</mn></msub><mo id="S3.p1.1.m1.1.1.1" xref="S3.p1.1.m1.1.1.1.cmml">∈</mo><msup id="S3.p1.1.m1.1.1.3" xref="S3.p1.1.m1.1.1.3.cmml"><mi id="S3.p1.1.m1.1.1.3.2" xref="S3.p1.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="S3.p1.1.m1.1.1.3.3" xref="S3.p1.1.m1.1.1.3.3.cmml"><mi id="S3.p1.1.m1.1.1.3.3.2" xref="S3.p1.1.m1.1.1.3.3.2.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="S3.p1.1.m1.1.1.3.3.1" xref="S3.p1.1.m1.1.1.3.3.1.cmml">×</mo><mi id="S3.p1.1.m1.1.1.3.3.3" xref="S3.p1.1.m1.1.1.3.3.3.cmml">k</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.1.m1.1b"><apply id="S3.p1.1.m1.1.1.cmml" xref="S3.p1.1.m1.1.1"><in id="S3.p1.1.m1.1.1.1.cmml" xref="S3.p1.1.m1.1.1.1"></in><apply id="S3.p1.1.m1.1.1.2.cmml" xref="S3.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.p1.1.m1.1.1.2.1.cmml" xref="S3.p1.1.m1.1.1.2">subscript</csymbol><ci id="S3.p1.1.m1.1.1.2.2.cmml" xref="S3.p1.1.m1.1.1.2.2">𝑊</ci><cn type="integer" id="S3.p1.1.m1.1.1.2.3.cmml" xref="S3.p1.1.m1.1.1.2.3">0</cn></apply><apply id="S3.p1.1.m1.1.1.3.cmml" xref="S3.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.p1.1.m1.1.1.3.1.cmml" xref="S3.p1.1.m1.1.1.3">superscript</csymbol><ci id="S3.p1.1.m1.1.1.3.2.cmml" xref="S3.p1.1.m1.1.1.3.2">ℝ</ci><apply id="S3.p1.1.m1.1.1.3.3.cmml" xref="S3.p1.1.m1.1.1.3.3"><times id="S3.p1.1.m1.1.1.3.3.1.cmml" xref="S3.p1.1.m1.1.1.3.3.1"></times><ci id="S3.p1.1.m1.1.1.3.3.2.cmml" xref="S3.p1.1.m1.1.1.3.3.2">𝑑</ci><ci id="S3.p1.1.m1.1.1.3.3.3.cmml" xref="S3.p1.1.m1.1.1.3.3.3">𝑘</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.1.m1.1c">W_{0}\in\mathbb{R}^{d\times k}</annotation></semantics></math>, LoRA employs two low-rank matrices to calculate the weight update <math id="S3.p1.2.m2.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S3.p1.2.m2.1a"><mrow id="S3.p1.2.m2.1.1" xref="S3.p1.2.m2.1.1.cmml"><mi mathvariant="normal" id="S3.p1.2.m2.1.1.2" xref="S3.p1.2.m2.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S3.p1.2.m2.1.1.1" xref="S3.p1.2.m2.1.1.1.cmml">​</mo><mi id="S3.p1.2.m2.1.1.3" xref="S3.p1.2.m2.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.2.m2.1b"><apply id="S3.p1.2.m2.1.1.cmml" xref="S3.p1.2.m2.1.1"><times id="S3.p1.2.m2.1.1.1.cmml" xref="S3.p1.2.m2.1.1.1"></times><ci id="S3.p1.2.m2.1.1.2.cmml" xref="S3.p1.2.m2.1.1.2">Δ</ci><ci id="S3.p1.2.m2.1.1.3.cmml" xref="S3.p1.2.m2.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.2.m2.1c">\Delta W</annotation></semantics></math>:</p>
<table id="S3.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E1.m1.1" class="ltx_Math" alttext="h=W_{0}x+\Delta Wx=W_{0}x+BAx" display="block"><semantics id="S3.E1.m1.1a"><mrow id="S3.E1.m1.1.1" xref="S3.E1.m1.1.1.cmml"><mi id="S3.E1.m1.1.1.2" xref="S3.E1.m1.1.1.2.cmml">h</mi><mo id="S3.E1.m1.1.1.3" xref="S3.E1.m1.1.1.3.cmml">=</mo><mrow id="S3.E1.m1.1.1.4" xref="S3.E1.m1.1.1.4.cmml"><mrow id="S3.E1.m1.1.1.4.2" xref="S3.E1.m1.1.1.4.2.cmml"><msub id="S3.E1.m1.1.1.4.2.2" xref="S3.E1.m1.1.1.4.2.2.cmml"><mi id="S3.E1.m1.1.1.4.2.2.2" xref="S3.E1.m1.1.1.4.2.2.2.cmml">W</mi><mn id="S3.E1.m1.1.1.4.2.2.3" xref="S3.E1.m1.1.1.4.2.2.3.cmml">0</mn></msub><mo lspace="0em" rspace="0em" id="S3.E1.m1.1.1.4.2.1" xref="S3.E1.m1.1.1.4.2.1.cmml">​</mo><mi id="S3.E1.m1.1.1.4.2.3" xref="S3.E1.m1.1.1.4.2.3.cmml">x</mi></mrow><mo id="S3.E1.m1.1.1.4.1" xref="S3.E1.m1.1.1.4.1.cmml">+</mo><mrow id="S3.E1.m1.1.1.4.3" xref="S3.E1.m1.1.1.4.3.cmml"><mi mathvariant="normal" id="S3.E1.m1.1.1.4.3.2" xref="S3.E1.m1.1.1.4.3.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S3.E1.m1.1.1.4.3.1" xref="S3.E1.m1.1.1.4.3.1.cmml">​</mo><mi id="S3.E1.m1.1.1.4.3.3" xref="S3.E1.m1.1.1.4.3.3.cmml">W</mi><mo lspace="0em" rspace="0em" id="S3.E1.m1.1.1.4.3.1a" xref="S3.E1.m1.1.1.4.3.1.cmml">​</mo><mi id="S3.E1.m1.1.1.4.3.4" xref="S3.E1.m1.1.1.4.3.4.cmml">x</mi></mrow></mrow><mo id="S3.E1.m1.1.1.5" xref="S3.E1.m1.1.1.5.cmml">=</mo><mrow id="S3.E1.m1.1.1.6" xref="S3.E1.m1.1.1.6.cmml"><mrow id="S3.E1.m1.1.1.6.2" xref="S3.E1.m1.1.1.6.2.cmml"><msub id="S3.E1.m1.1.1.6.2.2" xref="S3.E1.m1.1.1.6.2.2.cmml"><mi id="S3.E1.m1.1.1.6.2.2.2" xref="S3.E1.m1.1.1.6.2.2.2.cmml">W</mi><mn id="S3.E1.m1.1.1.6.2.2.3" xref="S3.E1.m1.1.1.6.2.2.3.cmml">0</mn></msub><mo lspace="0em" rspace="0em" id="S3.E1.m1.1.1.6.2.1" xref="S3.E1.m1.1.1.6.2.1.cmml">​</mo><mi id="S3.E1.m1.1.1.6.2.3" xref="S3.E1.m1.1.1.6.2.3.cmml">x</mi></mrow><mo id="S3.E1.m1.1.1.6.1" xref="S3.E1.m1.1.1.6.1.cmml">+</mo><mrow id="S3.E1.m1.1.1.6.3" xref="S3.E1.m1.1.1.6.3.cmml"><mi id="S3.E1.m1.1.1.6.3.2" xref="S3.E1.m1.1.1.6.3.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S3.E1.m1.1.1.6.3.1" xref="S3.E1.m1.1.1.6.3.1.cmml">​</mo><mi id="S3.E1.m1.1.1.6.3.3" xref="S3.E1.m1.1.1.6.3.3.cmml">A</mi><mo lspace="0em" rspace="0em" id="S3.E1.m1.1.1.6.3.1a" xref="S3.E1.m1.1.1.6.3.1.cmml">​</mo><mi id="S3.E1.m1.1.1.6.3.4" xref="S3.E1.m1.1.1.6.3.4.cmml">x</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E1.m1.1b"><apply id="S3.E1.m1.1.1.cmml" xref="S3.E1.m1.1.1"><and id="S3.E1.m1.1.1a.cmml" xref="S3.E1.m1.1.1"></and><apply id="S3.E1.m1.1.1b.cmml" xref="S3.E1.m1.1.1"><eq id="S3.E1.m1.1.1.3.cmml" xref="S3.E1.m1.1.1.3"></eq><ci id="S3.E1.m1.1.1.2.cmml" xref="S3.E1.m1.1.1.2">ℎ</ci><apply id="S3.E1.m1.1.1.4.cmml" xref="S3.E1.m1.1.1.4"><plus id="S3.E1.m1.1.1.4.1.cmml" xref="S3.E1.m1.1.1.4.1"></plus><apply id="S3.E1.m1.1.1.4.2.cmml" xref="S3.E1.m1.1.1.4.2"><times id="S3.E1.m1.1.1.4.2.1.cmml" xref="S3.E1.m1.1.1.4.2.1"></times><apply id="S3.E1.m1.1.1.4.2.2.cmml" xref="S3.E1.m1.1.1.4.2.2"><csymbol cd="ambiguous" id="S3.E1.m1.1.1.4.2.2.1.cmml" xref="S3.E1.m1.1.1.4.2.2">subscript</csymbol><ci id="S3.E1.m1.1.1.4.2.2.2.cmml" xref="S3.E1.m1.1.1.4.2.2.2">𝑊</ci><cn type="integer" id="S3.E1.m1.1.1.4.2.2.3.cmml" xref="S3.E1.m1.1.1.4.2.2.3">0</cn></apply><ci id="S3.E1.m1.1.1.4.2.3.cmml" xref="S3.E1.m1.1.1.4.2.3">𝑥</ci></apply><apply id="S3.E1.m1.1.1.4.3.cmml" xref="S3.E1.m1.1.1.4.3"><times id="S3.E1.m1.1.1.4.3.1.cmml" xref="S3.E1.m1.1.1.4.3.1"></times><ci id="S3.E1.m1.1.1.4.3.2.cmml" xref="S3.E1.m1.1.1.4.3.2">Δ</ci><ci id="S3.E1.m1.1.1.4.3.3.cmml" xref="S3.E1.m1.1.1.4.3.3">𝑊</ci><ci id="S3.E1.m1.1.1.4.3.4.cmml" xref="S3.E1.m1.1.1.4.3.4">𝑥</ci></apply></apply></apply><apply id="S3.E1.m1.1.1c.cmml" xref="S3.E1.m1.1.1"><eq id="S3.E1.m1.1.1.5.cmml" xref="S3.E1.m1.1.1.5"></eq><share href="#S3.E1.m1.1.1.4.cmml" id="S3.E1.m1.1.1d.cmml" xref="S3.E1.m1.1.1"></share><apply id="S3.E1.m1.1.1.6.cmml" xref="S3.E1.m1.1.1.6"><plus id="S3.E1.m1.1.1.6.1.cmml" xref="S3.E1.m1.1.1.6.1"></plus><apply id="S3.E1.m1.1.1.6.2.cmml" xref="S3.E1.m1.1.1.6.2"><times id="S3.E1.m1.1.1.6.2.1.cmml" xref="S3.E1.m1.1.1.6.2.1"></times><apply id="S3.E1.m1.1.1.6.2.2.cmml" xref="S3.E1.m1.1.1.6.2.2"><csymbol cd="ambiguous" id="S3.E1.m1.1.1.6.2.2.1.cmml" xref="S3.E1.m1.1.1.6.2.2">subscript</csymbol><ci id="S3.E1.m1.1.1.6.2.2.2.cmml" xref="S3.E1.m1.1.1.6.2.2.2">𝑊</ci><cn type="integer" id="S3.E1.m1.1.1.6.2.2.3.cmml" xref="S3.E1.m1.1.1.6.2.2.3">0</cn></apply><ci id="S3.E1.m1.1.1.6.2.3.cmml" xref="S3.E1.m1.1.1.6.2.3">𝑥</ci></apply><apply id="S3.E1.m1.1.1.6.3.cmml" xref="S3.E1.m1.1.1.6.3"><times id="S3.E1.m1.1.1.6.3.1.cmml" xref="S3.E1.m1.1.1.6.3.1"></times><ci id="S3.E1.m1.1.1.6.3.2.cmml" xref="S3.E1.m1.1.1.6.3.2">𝐵</ci><ci id="S3.E1.m1.1.1.6.3.3.cmml" xref="S3.E1.m1.1.1.6.3.3">𝐴</ci><ci id="S3.E1.m1.1.1.6.3.4.cmml" xref="S3.E1.m1.1.1.6.3.4">𝑥</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E1.m1.1c">h=W_{0}x+\Delta Wx=W_{0}x+BAx</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<p id="S3.p1.11" class="ltx_p">where <math id="S3.p1.3.m1.1" class="ltx_Math" alttext="A\in\mathbb{R}^{r\times k}" display="inline"><semantics id="S3.p1.3.m1.1a"><mrow id="S3.p1.3.m1.1.1" xref="S3.p1.3.m1.1.1.cmml"><mi id="S3.p1.3.m1.1.1.2" xref="S3.p1.3.m1.1.1.2.cmml">A</mi><mo id="S3.p1.3.m1.1.1.1" xref="S3.p1.3.m1.1.1.1.cmml">∈</mo><msup id="S3.p1.3.m1.1.1.3" xref="S3.p1.3.m1.1.1.3.cmml"><mi id="S3.p1.3.m1.1.1.3.2" xref="S3.p1.3.m1.1.1.3.2.cmml">ℝ</mi><mrow id="S3.p1.3.m1.1.1.3.3" xref="S3.p1.3.m1.1.1.3.3.cmml"><mi id="S3.p1.3.m1.1.1.3.3.2" xref="S3.p1.3.m1.1.1.3.3.2.cmml">r</mi><mo lspace="0.222em" rspace="0.222em" id="S3.p1.3.m1.1.1.3.3.1" xref="S3.p1.3.m1.1.1.3.3.1.cmml">×</mo><mi id="S3.p1.3.m1.1.1.3.3.3" xref="S3.p1.3.m1.1.1.3.3.3.cmml">k</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.3.m1.1b"><apply id="S3.p1.3.m1.1.1.cmml" xref="S3.p1.3.m1.1.1"><in id="S3.p1.3.m1.1.1.1.cmml" xref="S3.p1.3.m1.1.1.1"></in><ci id="S3.p1.3.m1.1.1.2.cmml" xref="S3.p1.3.m1.1.1.2">𝐴</ci><apply id="S3.p1.3.m1.1.1.3.cmml" xref="S3.p1.3.m1.1.1.3"><csymbol cd="ambiguous" id="S3.p1.3.m1.1.1.3.1.cmml" xref="S3.p1.3.m1.1.1.3">superscript</csymbol><ci id="S3.p1.3.m1.1.1.3.2.cmml" xref="S3.p1.3.m1.1.1.3.2">ℝ</ci><apply id="S3.p1.3.m1.1.1.3.3.cmml" xref="S3.p1.3.m1.1.1.3.3"><times id="S3.p1.3.m1.1.1.3.3.1.cmml" xref="S3.p1.3.m1.1.1.3.3.1"></times><ci id="S3.p1.3.m1.1.1.3.3.2.cmml" xref="S3.p1.3.m1.1.1.3.3.2">𝑟</ci><ci id="S3.p1.3.m1.1.1.3.3.3.cmml" xref="S3.p1.3.m1.1.1.3.3.3">𝑘</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.3.m1.1c">A\in\mathbb{R}^{r\times k}</annotation></semantics></math> and <math id="S3.p1.4.m2.1" class="ltx_Math" alttext="B\in\mathbb{R}^{d\times r}" display="inline"><semantics id="S3.p1.4.m2.1a"><mrow id="S3.p1.4.m2.1.1" xref="S3.p1.4.m2.1.1.cmml"><mi id="S3.p1.4.m2.1.1.2" xref="S3.p1.4.m2.1.1.2.cmml">B</mi><mo id="S3.p1.4.m2.1.1.1" xref="S3.p1.4.m2.1.1.1.cmml">∈</mo><msup id="S3.p1.4.m2.1.1.3" xref="S3.p1.4.m2.1.1.3.cmml"><mi id="S3.p1.4.m2.1.1.3.2" xref="S3.p1.4.m2.1.1.3.2.cmml">ℝ</mi><mrow id="S3.p1.4.m2.1.1.3.3" xref="S3.p1.4.m2.1.1.3.3.cmml"><mi id="S3.p1.4.m2.1.1.3.3.2" xref="S3.p1.4.m2.1.1.3.3.2.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="S3.p1.4.m2.1.1.3.3.1" xref="S3.p1.4.m2.1.1.3.3.1.cmml">×</mo><mi id="S3.p1.4.m2.1.1.3.3.3" xref="S3.p1.4.m2.1.1.3.3.3.cmml">r</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.4.m2.1b"><apply id="S3.p1.4.m2.1.1.cmml" xref="S3.p1.4.m2.1.1"><in id="S3.p1.4.m2.1.1.1.cmml" xref="S3.p1.4.m2.1.1.1"></in><ci id="S3.p1.4.m2.1.1.2.cmml" xref="S3.p1.4.m2.1.1.2">𝐵</ci><apply id="S3.p1.4.m2.1.1.3.cmml" xref="S3.p1.4.m2.1.1.3"><csymbol cd="ambiguous" id="S3.p1.4.m2.1.1.3.1.cmml" xref="S3.p1.4.m2.1.1.3">superscript</csymbol><ci id="S3.p1.4.m2.1.1.3.2.cmml" xref="S3.p1.4.m2.1.1.3.2">ℝ</ci><apply id="S3.p1.4.m2.1.1.3.3.cmml" xref="S3.p1.4.m2.1.1.3.3"><times id="S3.p1.4.m2.1.1.3.3.1.cmml" xref="S3.p1.4.m2.1.1.3.3.1"></times><ci id="S3.p1.4.m2.1.1.3.3.2.cmml" xref="S3.p1.4.m2.1.1.3.3.2">𝑑</ci><ci id="S3.p1.4.m2.1.1.3.3.3.cmml" xref="S3.p1.4.m2.1.1.3.3.3">𝑟</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.4.m2.1c">B\in\mathbb{R}^{d\times r}</annotation></semantics></math> represent the low-rank matrices in LoRA. To ensure that <math id="S3.p1.5.m3.1" class="ltx_Math" alttext="\Delta W=0" display="inline"><semantics id="S3.p1.5.m3.1a"><mrow id="S3.p1.5.m3.1.1" xref="S3.p1.5.m3.1.1.cmml"><mrow id="S3.p1.5.m3.1.1.2" xref="S3.p1.5.m3.1.1.2.cmml"><mi mathvariant="normal" id="S3.p1.5.m3.1.1.2.2" xref="S3.p1.5.m3.1.1.2.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S3.p1.5.m3.1.1.2.1" xref="S3.p1.5.m3.1.1.2.1.cmml">​</mo><mi id="S3.p1.5.m3.1.1.2.3" xref="S3.p1.5.m3.1.1.2.3.cmml">W</mi></mrow><mo id="S3.p1.5.m3.1.1.1" xref="S3.p1.5.m3.1.1.1.cmml">=</mo><mn id="S3.p1.5.m3.1.1.3" xref="S3.p1.5.m3.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.5.m3.1b"><apply id="S3.p1.5.m3.1.1.cmml" xref="S3.p1.5.m3.1.1"><eq id="S3.p1.5.m3.1.1.1.cmml" xref="S3.p1.5.m3.1.1.1"></eq><apply id="S3.p1.5.m3.1.1.2.cmml" xref="S3.p1.5.m3.1.1.2"><times id="S3.p1.5.m3.1.1.2.1.cmml" xref="S3.p1.5.m3.1.1.2.1"></times><ci id="S3.p1.5.m3.1.1.2.2.cmml" xref="S3.p1.5.m3.1.1.2.2">Δ</ci><ci id="S3.p1.5.m3.1.1.2.3.cmml" xref="S3.p1.5.m3.1.1.2.3">𝑊</ci></apply><cn type="integer" id="S3.p1.5.m3.1.1.3.cmml" xref="S3.p1.5.m3.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.5.m3.1c">\Delta W=0</annotation></semantics></math> at the beginning of training, LoRA initializes <math id="S3.p1.6.m4.1" class="ltx_Math" alttext="A" display="inline"><semantics id="S3.p1.6.m4.1a"><mi id="S3.p1.6.m4.1.1" xref="S3.p1.6.m4.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="S3.p1.6.m4.1b"><ci id="S3.p1.6.m4.1.1.cmml" xref="S3.p1.6.m4.1.1">𝐴</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.6.m4.1c">A</annotation></semantics></math> with a Gaussian distribution and <math id="S3.p1.7.m5.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S3.p1.7.m5.1a"><mi id="S3.p1.7.m5.1.1" xref="S3.p1.7.m5.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S3.p1.7.m5.1b"><ci id="S3.p1.7.m5.1.1.cmml" xref="S3.p1.7.m5.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.7.m5.1c">B</annotation></semantics></math> with zero. Due to the low-rank decomposition of <math id="S3.p1.8.m6.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S3.p1.8.m6.1a"><mrow id="S3.p1.8.m6.1.1" xref="S3.p1.8.m6.1.1.cmml"><mi mathvariant="normal" id="S3.p1.8.m6.1.1.2" xref="S3.p1.8.m6.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S3.p1.8.m6.1.1.1" xref="S3.p1.8.m6.1.1.1.cmml">​</mo><mi id="S3.p1.8.m6.1.1.3" xref="S3.p1.8.m6.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.8.m6.1b"><apply id="S3.p1.8.m6.1.1.cmml" xref="S3.p1.8.m6.1.1"><times id="S3.p1.8.m6.1.1.1.cmml" xref="S3.p1.8.m6.1.1.1"></times><ci id="S3.p1.8.m6.1.1.2.cmml" xref="S3.p1.8.m6.1.1.2">Δ</ci><ci id="S3.p1.8.m6.1.1.3.cmml" xref="S3.p1.8.m6.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.8.m6.1c">\Delta W</annotation></semantics></math> into <math id="S3.p1.9.m7.1" class="ltx_Math" alttext="BA" display="inline"><semantics id="S3.p1.9.m7.1a"><mrow id="S3.p1.9.m7.1.1" xref="S3.p1.9.m7.1.1.cmml"><mi id="S3.p1.9.m7.1.1.2" xref="S3.p1.9.m7.1.1.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S3.p1.9.m7.1.1.1" xref="S3.p1.9.m7.1.1.1.cmml">​</mo><mi id="S3.p1.9.m7.1.1.3" xref="S3.p1.9.m7.1.1.3.cmml">A</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.9.m7.1b"><apply id="S3.p1.9.m7.1.1.cmml" xref="S3.p1.9.m7.1.1"><times id="S3.p1.9.m7.1.1.1.cmml" xref="S3.p1.9.m7.1.1.1"></times><ci id="S3.p1.9.m7.1.1.2.cmml" xref="S3.p1.9.m7.1.1.2">𝐵</ci><ci id="S3.p1.9.m7.1.1.3.cmml" xref="S3.p1.9.m7.1.1.3">𝐴</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.9.m7.1c">BA</annotation></semantics></math>, the <math id="S3.p1.10.m8.1" class="ltx_Math" alttext="rank(\Delta W)\leq r" display="inline"><semantics id="S3.p1.10.m8.1a"><mrow id="S3.p1.10.m8.1.1" xref="S3.p1.10.m8.1.1.cmml"><mrow id="S3.p1.10.m8.1.1.1" xref="S3.p1.10.m8.1.1.1.cmml"><mi id="S3.p1.10.m8.1.1.1.3" xref="S3.p1.10.m8.1.1.1.3.cmml">r</mi><mo lspace="0em" rspace="0em" id="S3.p1.10.m8.1.1.1.2" xref="S3.p1.10.m8.1.1.1.2.cmml">​</mo><mi id="S3.p1.10.m8.1.1.1.4" xref="S3.p1.10.m8.1.1.1.4.cmml">a</mi><mo lspace="0em" rspace="0em" id="S3.p1.10.m8.1.1.1.2a" xref="S3.p1.10.m8.1.1.1.2.cmml">​</mo><mi id="S3.p1.10.m8.1.1.1.5" xref="S3.p1.10.m8.1.1.1.5.cmml">n</mi><mo lspace="0em" rspace="0em" id="S3.p1.10.m8.1.1.1.2b" xref="S3.p1.10.m8.1.1.1.2.cmml">​</mo><mi id="S3.p1.10.m8.1.1.1.6" xref="S3.p1.10.m8.1.1.1.6.cmml">k</mi><mo lspace="0em" rspace="0em" id="S3.p1.10.m8.1.1.1.2c" xref="S3.p1.10.m8.1.1.1.2.cmml">​</mo><mrow id="S3.p1.10.m8.1.1.1.1.1" xref="S3.p1.10.m8.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.p1.10.m8.1.1.1.1.1.2" xref="S3.p1.10.m8.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.p1.10.m8.1.1.1.1.1.1" xref="S3.p1.10.m8.1.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S3.p1.10.m8.1.1.1.1.1.1.2" xref="S3.p1.10.m8.1.1.1.1.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S3.p1.10.m8.1.1.1.1.1.1.1" xref="S3.p1.10.m8.1.1.1.1.1.1.1.cmml">​</mo><mi id="S3.p1.10.m8.1.1.1.1.1.1.3" xref="S3.p1.10.m8.1.1.1.1.1.1.3.cmml">W</mi></mrow><mo stretchy="false" id="S3.p1.10.m8.1.1.1.1.1.3" xref="S3.p1.10.m8.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.p1.10.m8.1.1.2" xref="S3.p1.10.m8.1.1.2.cmml">≤</mo><mi id="S3.p1.10.m8.1.1.3" xref="S3.p1.10.m8.1.1.3.cmml">r</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.10.m8.1b"><apply id="S3.p1.10.m8.1.1.cmml" xref="S3.p1.10.m8.1.1"><leq id="S3.p1.10.m8.1.1.2.cmml" xref="S3.p1.10.m8.1.1.2"></leq><apply id="S3.p1.10.m8.1.1.1.cmml" xref="S3.p1.10.m8.1.1.1"><times id="S3.p1.10.m8.1.1.1.2.cmml" xref="S3.p1.10.m8.1.1.1.2"></times><ci id="S3.p1.10.m8.1.1.1.3.cmml" xref="S3.p1.10.m8.1.1.1.3">𝑟</ci><ci id="S3.p1.10.m8.1.1.1.4.cmml" xref="S3.p1.10.m8.1.1.1.4">𝑎</ci><ci id="S3.p1.10.m8.1.1.1.5.cmml" xref="S3.p1.10.m8.1.1.1.5">𝑛</ci><ci id="S3.p1.10.m8.1.1.1.6.cmml" xref="S3.p1.10.m8.1.1.1.6">𝑘</ci><apply id="S3.p1.10.m8.1.1.1.1.1.1.cmml" xref="S3.p1.10.m8.1.1.1.1.1"><times id="S3.p1.10.m8.1.1.1.1.1.1.1.cmml" xref="S3.p1.10.m8.1.1.1.1.1.1.1"></times><ci id="S3.p1.10.m8.1.1.1.1.1.1.2.cmml" xref="S3.p1.10.m8.1.1.1.1.1.1.2">Δ</ci><ci id="S3.p1.10.m8.1.1.1.1.1.1.3.cmml" xref="S3.p1.10.m8.1.1.1.1.1.1.3">𝑊</ci></apply></apply><ci id="S3.p1.10.m8.1.1.3.cmml" xref="S3.p1.10.m8.1.1.3">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.10.m8.1c">rank(\Delta W)\leq r</annotation></semantics></math>.
The weight update in LoRA exhibits a markedly low rank, <math id="S3.p1.11.m9.3" class="ltx_Math" alttext="r\ll\min(d,k)" display="inline"><semantics id="S3.p1.11.m9.3a"><mrow id="S3.p1.11.m9.3.4" xref="S3.p1.11.m9.3.4.cmml"><mi id="S3.p1.11.m9.3.4.2" xref="S3.p1.11.m9.3.4.2.cmml">r</mi><mo id="S3.p1.11.m9.3.4.1" xref="S3.p1.11.m9.3.4.1.cmml">≪</mo><mrow id="S3.p1.11.m9.3.4.3.2" xref="S3.p1.11.m9.3.4.3.1.cmml"><mi id="S3.p1.11.m9.1.1" xref="S3.p1.11.m9.1.1.cmml">min</mi><mo id="S3.p1.11.m9.3.4.3.2a" xref="S3.p1.11.m9.3.4.3.1.cmml">⁡</mo><mrow id="S3.p1.11.m9.3.4.3.2.1" xref="S3.p1.11.m9.3.4.3.1.cmml"><mo stretchy="false" id="S3.p1.11.m9.3.4.3.2.1.1" xref="S3.p1.11.m9.3.4.3.1.cmml">(</mo><mi id="S3.p1.11.m9.2.2" xref="S3.p1.11.m9.2.2.cmml">d</mi><mo id="S3.p1.11.m9.3.4.3.2.1.2" xref="S3.p1.11.m9.3.4.3.1.cmml">,</mo><mi id="S3.p1.11.m9.3.3" xref="S3.p1.11.m9.3.3.cmml">k</mi><mo stretchy="false" id="S3.p1.11.m9.3.4.3.2.1.3" xref="S3.p1.11.m9.3.4.3.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.11.m9.3b"><apply id="S3.p1.11.m9.3.4.cmml" xref="S3.p1.11.m9.3.4"><csymbol cd="latexml" id="S3.p1.11.m9.3.4.1.cmml" xref="S3.p1.11.m9.3.4.1">much-less-than</csymbol><ci id="S3.p1.11.m9.3.4.2.cmml" xref="S3.p1.11.m9.3.4.2">𝑟</ci><apply id="S3.p1.11.m9.3.4.3.1.cmml" xref="S3.p1.11.m9.3.4.3.2"><min id="S3.p1.11.m9.1.1.cmml" xref="S3.p1.11.m9.1.1"></min><ci id="S3.p1.11.m9.2.2.cmml" xref="S3.p1.11.m9.2.2">𝑑</ci><ci id="S3.p1.11.m9.3.3.cmml" xref="S3.p1.11.m9.3.3">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.11.m9.3c">r\ll\min(d,k)</annotation></semantics></math>, in comparison to the full-rank updating in FFT.
Low-rank updating by LoRA shows on-par performance with full-rank updating in some tasks such as text classification or instruction tuning <cite class="ltx_cite ltx_citemacro_cite">Liu et al. (<a href="#bib.bib24" title="" class="ltx_ref">2024</a>); Meng et al. (<a href="#bib.bib27" title="" class="ltx_ref">2024</a>)</cite>.
However, for tasks like complex reasoning or continual pretraining, LoRA tends to show worse performance <cite class="ltx_cite ltx_citemacro_cite">Liu et al. (<a href="#bib.bib23" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
<div id="S3.p2" class="ltx_para">
<p id="S3.p2.1" class="ltx_p">Based on these observations, we propose a hypothesis that low-rank updating is easy to leverage original knowledge and capabilities of LLM to solve task, but it is struggle to handle tasks that require enhancing knowledge and capabilities of LLM.</p>
</div>
<figure id="S3.F2" class="ltx_figure"><img src="/html/2405.12130/assets/fft-memory-lora.png" id="S3.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="592" height="439" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>
Performance of memorizing UUID pairs through fine-tuning with FFT and LoRA.
</figcaption>
</figure>
<div id="S3.p3" class="ltx_para">
<p id="S3.p3.1" class="ltx_p">To substantiate this hypothesis, we examine the differences between LoRA and FFT in terms of memorizing new knowledge through fine-tuning. In order to circumvent leveraging the original knowledge of the LLM, we randomly generate 10K pairs of Universally Unique Identifiers (UUIDs), each pair comprising two UUIDs with 32 hexadecimal values. The task requires the LLM to generate the corresponding UUID based on the input UUID. For instance, given a UUID such as “205f3777-52b6-4270-9f67-c5125867d358”, the model should generate the corresponding UUID based on 10K training pairs.
This task can also be viewed as a question-answering task, while the knowledge indispensable for accomplishing it is exclusively from the training datasets rather than the LLM itself.</p>
</div>
<div id="S3.p4" class="ltx_para">
<p id="S3.p4.5" class="ltx_p">For the training settings, we employ LLaMA-2 7B as base model, utilizing 1,000 pairs per batch and conducting 100 epochs. For the LoRA, we apply low-rank matrices to all linear layers and search learning rate from <math id="S3.p4.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="S3.p4.1.m1.1a"><mo stretchy="false" id="S3.p4.1.m1.1.1" xref="S3.p4.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="S3.p4.1.m1.1b"><ci id="S3.p4.1.m1.1.1.cmml" xref="S3.p4.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p4.1.m1.1c">\{</annotation></semantics></math>1e-4<span id="S3.p4.5.1" class="ltx_text ltx_markedasmath">,</span>2e-4<span id="S3.p4.5.2" class="ltx_text ltx_markedasmath">,</span>3e-4<math id="S3.p4.4.m4.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="S3.p4.4.m4.1a"><mo stretchy="false" id="S3.p4.4.m4.1.1" xref="S3.p4.4.m4.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="S3.p4.4.m4.1b"><ci id="S3.p4.4.m4.1.1.cmml" xref="S3.p4.4.m4.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p4.4.m4.1c">\}</annotation></semantics></math> to enhance performances.
We conduct the experiment on LoRA using various ranks <math id="S3.p4.5.m5.6" class="ltx_Math" alttext="r\in\{8,16,32,64,128,256\}" display="inline"><semantics id="S3.p4.5.m5.6a"><mrow id="S3.p4.5.m5.6.7" xref="S3.p4.5.m5.6.7.cmml"><mi id="S3.p4.5.m5.6.7.2" xref="S3.p4.5.m5.6.7.2.cmml">r</mi><mo id="S3.p4.5.m5.6.7.1" xref="S3.p4.5.m5.6.7.1.cmml">∈</mo><mrow id="S3.p4.5.m5.6.7.3.2" xref="S3.p4.5.m5.6.7.3.1.cmml"><mo stretchy="false" id="S3.p4.5.m5.6.7.3.2.1" xref="S3.p4.5.m5.6.7.3.1.cmml">{</mo><mn id="S3.p4.5.m5.1.1" xref="S3.p4.5.m5.1.1.cmml">8</mn><mo id="S3.p4.5.m5.6.7.3.2.2" xref="S3.p4.5.m5.6.7.3.1.cmml">,</mo><mn id="S3.p4.5.m5.2.2" xref="S3.p4.5.m5.2.2.cmml">16</mn><mo id="S3.p4.5.m5.6.7.3.2.3" xref="S3.p4.5.m5.6.7.3.1.cmml">,</mo><mn id="S3.p4.5.m5.3.3" xref="S3.p4.5.m5.3.3.cmml">32</mn><mo id="S3.p4.5.m5.6.7.3.2.4" xref="S3.p4.5.m5.6.7.3.1.cmml">,</mo><mn id="S3.p4.5.m5.4.4" xref="S3.p4.5.m5.4.4.cmml">64</mn><mo id="S3.p4.5.m5.6.7.3.2.5" xref="S3.p4.5.m5.6.7.3.1.cmml">,</mo><mn id="S3.p4.5.m5.5.5" xref="S3.p4.5.m5.5.5.cmml">128</mn><mo id="S3.p4.5.m5.6.7.3.2.6" xref="S3.p4.5.m5.6.7.3.1.cmml">,</mo><mn id="S3.p4.5.m5.6.6" xref="S3.p4.5.m5.6.6.cmml">256</mn><mo stretchy="false" id="S3.p4.5.m5.6.7.3.2.7" xref="S3.p4.5.m5.6.7.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.p4.5.m5.6b"><apply id="S3.p4.5.m5.6.7.cmml" xref="S3.p4.5.m5.6.7"><in id="S3.p4.5.m5.6.7.1.cmml" xref="S3.p4.5.m5.6.7.1"></in><ci id="S3.p4.5.m5.6.7.2.cmml" xref="S3.p4.5.m5.6.7.2">𝑟</ci><set id="S3.p4.5.m5.6.7.3.1.cmml" xref="S3.p4.5.m5.6.7.3.2"><cn type="integer" id="S3.p4.5.m5.1.1.cmml" xref="S3.p4.5.m5.1.1">8</cn><cn type="integer" id="S3.p4.5.m5.2.2.cmml" xref="S3.p4.5.m5.2.2">16</cn><cn type="integer" id="S3.p4.5.m5.3.3.cmml" xref="S3.p4.5.m5.3.3">32</cn><cn type="integer" id="S3.p4.5.m5.4.4.cmml" xref="S3.p4.5.m5.4.4">64</cn><cn type="integer" id="S3.p4.5.m5.5.5.cmml" xref="S3.p4.5.m5.5.5">128</cn><cn type="integer" id="S3.p4.5.m5.6.6.cmml" xref="S3.p4.5.m5.6.6">256</cn></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p4.5.m5.6c">r\in\{8,16,32,64,128,256\}</annotation></semantics></math>.
For the FFT, we directly use a learning rate of 3e-5.
Based on Figure <a href="#S3.F2" title="Figure 2 ‣ 3 Analysis the Influence of Low-rank Updating ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, we observe low-rank updating are hard to memorizing new knowledge compared to FFT.
Although constantly increasing the rank of LoRA can alleviate this problem, the gap still exists.</p>
</div>
<div id="S3.p5" class="ltx_para">
<p id="S3.p5.1" class="ltx_p">In contrast to the memory task, we also evaluate the performance gap between LoRA and FFT on instruction tuning, which merely introduces new knowledge.
Similar to previous results <cite class="ltx_cite ltx_citemacro_cite">Meng et al. (<a href="#bib.bib27" title="" class="ltx_ref">2024</a>); Zhu et al. (<a href="#bib.bib41" title="" class="ltx_ref">2024</a>)</cite>, we also find that LoRA matches the performance of FFT with small rank <math id="S3.p5.1.m1.1" class="ltx_Math" alttext="r=8" display="inline"><semantics id="S3.p5.1.m1.1a"><mrow id="S3.p5.1.m1.1.1" xref="S3.p5.1.m1.1.1.cmml"><mi id="S3.p5.1.m1.1.1.2" xref="S3.p5.1.m1.1.1.2.cmml">r</mi><mo id="S3.p5.1.m1.1.1.1" xref="S3.p5.1.m1.1.1.1.cmml">=</mo><mn id="S3.p5.1.m1.1.1.3" xref="S3.p5.1.m1.1.1.3.cmml">8</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.p5.1.m1.1b"><apply id="S3.p5.1.m1.1.1.cmml" xref="S3.p5.1.m1.1.1"><eq id="S3.p5.1.m1.1.1.1.cmml" xref="S3.p5.1.m1.1.1.1"></eq><ci id="S3.p5.1.m1.1.1.2.cmml" xref="S3.p5.1.m1.1.1.2">𝑟</ci><cn type="integer" id="S3.p5.1.m1.1.1.3.cmml" xref="S3.p5.1.m1.1.1.3">8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p5.1.m1.1c">r=8</annotation></semantics></math> in Table <a href="#S5.T1" title="Table 1 ‣ 5 Experiment ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.
This indicates that LoRA can easily leverage the original knowledge of LLMs by fine-tuning like FFT.</p>
</div>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Method</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.9" class="ltx_p">Based on the above analysis, we propose a new method to alleviate the negative effects of low-rank updating. The main idea of our method is to utilize the same trainable parameters as much as possible to achieve a higher rank in <math id="S4.p1.1.m1.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S4.p1.1.m1.1a"><mrow id="S4.p1.1.m1.1.1" xref="S4.p1.1.m1.1.1.cmml"><mi mathvariant="normal" id="S4.p1.1.m1.1.1.2" xref="S4.p1.1.m1.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.p1.1.m1.1.1.1" xref="S4.p1.1.m1.1.1.1.cmml">​</mo><mi id="S4.p1.1.m1.1.1.3" xref="S4.p1.1.m1.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.1.m1.1b"><apply id="S4.p1.1.m1.1.1.cmml" xref="S4.p1.1.m1.1.1"><times id="S4.p1.1.m1.1.1.1.cmml" xref="S4.p1.1.m1.1.1.1"></times><ci id="S4.p1.1.m1.1.1.2.cmml" xref="S4.p1.1.m1.1.1.2">Δ</ci><ci id="S4.p1.1.m1.1.1.3.cmml" xref="S4.p1.1.m1.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.1.m1.1c">\Delta W</annotation></semantics></math>.
Consider to the pretrained weight <math id="S4.p1.2.m2.1" class="ltx_Math" alttext="W_{0}\in\mathbb{R}^{d\times k}" display="inline"><semantics id="S4.p1.2.m2.1a"><mrow id="S4.p1.2.m2.1.1" xref="S4.p1.2.m2.1.1.cmml"><msub id="S4.p1.2.m2.1.1.2" xref="S4.p1.2.m2.1.1.2.cmml"><mi id="S4.p1.2.m2.1.1.2.2" xref="S4.p1.2.m2.1.1.2.2.cmml">W</mi><mn id="S4.p1.2.m2.1.1.2.3" xref="S4.p1.2.m2.1.1.2.3.cmml">0</mn></msub><mo id="S4.p1.2.m2.1.1.1" xref="S4.p1.2.m2.1.1.1.cmml">∈</mo><msup id="S4.p1.2.m2.1.1.3" xref="S4.p1.2.m2.1.1.3.cmml"><mi id="S4.p1.2.m2.1.1.3.2" xref="S4.p1.2.m2.1.1.3.2.cmml">ℝ</mi><mrow id="S4.p1.2.m2.1.1.3.3" xref="S4.p1.2.m2.1.1.3.3.cmml"><mi id="S4.p1.2.m2.1.1.3.3.2" xref="S4.p1.2.m2.1.1.3.3.2.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="S4.p1.2.m2.1.1.3.3.1" xref="S4.p1.2.m2.1.1.3.3.1.cmml">×</mo><mi id="S4.p1.2.m2.1.1.3.3.3" xref="S4.p1.2.m2.1.1.3.3.3.cmml">k</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.2.m2.1b"><apply id="S4.p1.2.m2.1.1.cmml" xref="S4.p1.2.m2.1.1"><in id="S4.p1.2.m2.1.1.1.cmml" xref="S4.p1.2.m2.1.1.1"></in><apply id="S4.p1.2.m2.1.1.2.cmml" xref="S4.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="S4.p1.2.m2.1.1.2.1.cmml" xref="S4.p1.2.m2.1.1.2">subscript</csymbol><ci id="S4.p1.2.m2.1.1.2.2.cmml" xref="S4.p1.2.m2.1.1.2.2">𝑊</ci><cn type="integer" id="S4.p1.2.m2.1.1.2.3.cmml" xref="S4.p1.2.m2.1.1.2.3">0</cn></apply><apply id="S4.p1.2.m2.1.1.3.cmml" xref="S4.p1.2.m2.1.1.3"><csymbol cd="ambiguous" id="S4.p1.2.m2.1.1.3.1.cmml" xref="S4.p1.2.m2.1.1.3">superscript</csymbol><ci id="S4.p1.2.m2.1.1.3.2.cmml" xref="S4.p1.2.m2.1.1.3.2">ℝ</ci><apply id="S4.p1.2.m2.1.1.3.3.cmml" xref="S4.p1.2.m2.1.1.3.3"><times id="S4.p1.2.m2.1.1.3.3.1.cmml" xref="S4.p1.2.m2.1.1.3.3.1"></times><ci id="S4.p1.2.m2.1.1.3.3.2.cmml" xref="S4.p1.2.m2.1.1.3.3.2">𝑑</ci><ci id="S4.p1.2.m2.1.1.3.3.3.cmml" xref="S4.p1.2.m2.1.1.3.3.3">𝑘</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.2.m2.1c">W_{0}\in\mathbb{R}^{d\times k}</annotation></semantics></math>, LoRA uses two low-rank matrices <math id="S4.p1.3.m3.1" class="ltx_Math" alttext="A" display="inline"><semantics id="S4.p1.3.m3.1a"><mi id="S4.p1.3.m3.1.1" xref="S4.p1.3.m3.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="S4.p1.3.m3.1b"><ci id="S4.p1.3.m3.1.1.cmml" xref="S4.p1.3.m3.1.1">𝐴</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.3.m3.1c">A</annotation></semantics></math> and <math id="S4.p1.4.m4.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S4.p1.4.m4.1a"><mi id="S4.p1.4.m4.1.1" xref="S4.p1.4.m4.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S4.p1.4.m4.1b"><ci id="S4.p1.4.m4.1.1.cmml" xref="S4.p1.4.m4.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.4.m4.1c">B</annotation></semantics></math> with <math id="S4.p1.5.m5.1" class="ltx_Math" alttext="(d+k)r" display="inline"><semantics id="S4.p1.5.m5.1a"><mrow id="S4.p1.5.m5.1.1" xref="S4.p1.5.m5.1.1.cmml"><mrow id="S4.p1.5.m5.1.1.1.1" xref="S4.p1.5.m5.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.p1.5.m5.1.1.1.1.2" xref="S4.p1.5.m5.1.1.1.1.1.cmml">(</mo><mrow id="S4.p1.5.m5.1.1.1.1.1" xref="S4.p1.5.m5.1.1.1.1.1.cmml"><mi id="S4.p1.5.m5.1.1.1.1.1.2" xref="S4.p1.5.m5.1.1.1.1.1.2.cmml">d</mi><mo id="S4.p1.5.m5.1.1.1.1.1.1" xref="S4.p1.5.m5.1.1.1.1.1.1.cmml">+</mo><mi id="S4.p1.5.m5.1.1.1.1.1.3" xref="S4.p1.5.m5.1.1.1.1.1.3.cmml">k</mi></mrow><mo stretchy="false" id="S4.p1.5.m5.1.1.1.1.3" xref="S4.p1.5.m5.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S4.p1.5.m5.1.1.2" xref="S4.p1.5.m5.1.1.2.cmml">​</mo><mi id="S4.p1.5.m5.1.1.3" xref="S4.p1.5.m5.1.1.3.cmml">r</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.5.m5.1b"><apply id="S4.p1.5.m5.1.1.cmml" xref="S4.p1.5.m5.1.1"><times id="S4.p1.5.m5.1.1.2.cmml" xref="S4.p1.5.m5.1.1.2"></times><apply id="S4.p1.5.m5.1.1.1.1.1.cmml" xref="S4.p1.5.m5.1.1.1.1"><plus id="S4.p1.5.m5.1.1.1.1.1.1.cmml" xref="S4.p1.5.m5.1.1.1.1.1.1"></plus><ci id="S4.p1.5.m5.1.1.1.1.1.2.cmml" xref="S4.p1.5.m5.1.1.1.1.1.2">𝑑</ci><ci id="S4.p1.5.m5.1.1.1.1.1.3.cmml" xref="S4.p1.5.m5.1.1.1.1.1.3">𝑘</ci></apply><ci id="S4.p1.5.m5.1.1.3.cmml" xref="S4.p1.5.m5.1.1.3">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.5.m5.1c">(d+k)r</annotation></semantics></math> total trainable parameters for rank <math id="S4.p1.6.m6.1" class="ltx_Math" alttext="r" display="inline"><semantics id="S4.p1.6.m6.1a"><mi id="S4.p1.6.m6.1.1" xref="S4.p1.6.m6.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="S4.p1.6.m6.1b"><ci id="S4.p1.6.m6.1.1.cmml" xref="S4.p1.6.m6.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.6.m6.1c">r</annotation></semantics></math>.
Under same trainable parameters, a square matrix <math id="S4.p1.7.m7.1" class="ltx_Math" alttext="M\in\mathbb{R}^{\hat{r}\times\hat{r}}" display="inline"><semantics id="S4.p1.7.m7.1a"><mrow id="S4.p1.7.m7.1.1" xref="S4.p1.7.m7.1.1.cmml"><mi id="S4.p1.7.m7.1.1.2" xref="S4.p1.7.m7.1.1.2.cmml">M</mi><mo id="S4.p1.7.m7.1.1.1" xref="S4.p1.7.m7.1.1.1.cmml">∈</mo><msup id="S4.p1.7.m7.1.1.3" xref="S4.p1.7.m7.1.1.3.cmml"><mi id="S4.p1.7.m7.1.1.3.2" xref="S4.p1.7.m7.1.1.3.2.cmml">ℝ</mi><mrow id="S4.p1.7.m7.1.1.3.3" xref="S4.p1.7.m7.1.1.3.3.cmml"><mover accent="true" id="S4.p1.7.m7.1.1.3.3.2" xref="S4.p1.7.m7.1.1.3.3.2.cmml"><mi id="S4.p1.7.m7.1.1.3.3.2.2" xref="S4.p1.7.m7.1.1.3.3.2.2.cmml">r</mi><mo id="S4.p1.7.m7.1.1.3.3.2.1" xref="S4.p1.7.m7.1.1.3.3.2.1.cmml">^</mo></mover><mo lspace="0.222em" rspace="0.222em" id="S4.p1.7.m7.1.1.3.3.1" xref="S4.p1.7.m7.1.1.3.3.1.cmml">×</mo><mover accent="true" id="S4.p1.7.m7.1.1.3.3.3" xref="S4.p1.7.m7.1.1.3.3.3.cmml"><mi id="S4.p1.7.m7.1.1.3.3.3.2" xref="S4.p1.7.m7.1.1.3.3.3.2.cmml">r</mi><mo id="S4.p1.7.m7.1.1.3.3.3.1" xref="S4.p1.7.m7.1.1.3.3.3.1.cmml">^</mo></mover></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.7.m7.1b"><apply id="S4.p1.7.m7.1.1.cmml" xref="S4.p1.7.m7.1.1"><in id="S4.p1.7.m7.1.1.1.cmml" xref="S4.p1.7.m7.1.1.1"></in><ci id="S4.p1.7.m7.1.1.2.cmml" xref="S4.p1.7.m7.1.1.2">𝑀</ci><apply id="S4.p1.7.m7.1.1.3.cmml" xref="S4.p1.7.m7.1.1.3"><csymbol cd="ambiguous" id="S4.p1.7.m7.1.1.3.1.cmml" xref="S4.p1.7.m7.1.1.3">superscript</csymbol><ci id="S4.p1.7.m7.1.1.3.2.cmml" xref="S4.p1.7.m7.1.1.3.2">ℝ</ci><apply id="S4.p1.7.m7.1.1.3.3.cmml" xref="S4.p1.7.m7.1.1.3.3"><times id="S4.p1.7.m7.1.1.3.3.1.cmml" xref="S4.p1.7.m7.1.1.3.3.1"></times><apply id="S4.p1.7.m7.1.1.3.3.2.cmml" xref="S4.p1.7.m7.1.1.3.3.2"><ci id="S4.p1.7.m7.1.1.3.3.2.1.cmml" xref="S4.p1.7.m7.1.1.3.3.2.1">^</ci><ci id="S4.p1.7.m7.1.1.3.3.2.2.cmml" xref="S4.p1.7.m7.1.1.3.3.2.2">𝑟</ci></apply><apply id="S4.p1.7.m7.1.1.3.3.3.cmml" xref="S4.p1.7.m7.1.1.3.3.3"><ci id="S4.p1.7.m7.1.1.3.3.3.1.cmml" xref="S4.p1.7.m7.1.1.3.3.3.1">^</ci><ci id="S4.p1.7.m7.1.1.3.3.3.2.cmml" xref="S4.p1.7.m7.1.1.3.3.3.2">𝑟</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.7.m7.1c">M\in\mathbb{R}^{\hat{r}\times\hat{r}}</annotation></semantics></math> where <math id="S4.p1.8.m8.1" class="ltx_Math" alttext="\hat{r}=\lfloor\sqrt{(d+k)r}\rfloor" display="inline"><semantics id="S4.p1.8.m8.1a"><mrow id="S4.p1.8.m8.1.2" xref="S4.p1.8.m8.1.2.cmml"><mover accent="true" id="S4.p1.8.m8.1.2.2" xref="S4.p1.8.m8.1.2.2.cmml"><mi id="S4.p1.8.m8.1.2.2.2" xref="S4.p1.8.m8.1.2.2.2.cmml">r</mi><mo id="S4.p1.8.m8.1.2.2.1" xref="S4.p1.8.m8.1.2.2.1.cmml">^</mo></mover><mo id="S4.p1.8.m8.1.2.1" xref="S4.p1.8.m8.1.2.1.cmml">=</mo><mrow id="S4.p1.8.m8.1.2.3.2" xref="S4.p1.8.m8.1.2.3.1.cmml"><mo stretchy="false" id="S4.p1.8.m8.1.2.3.2.1" xref="S4.p1.8.m8.1.2.3.1.1.cmml">⌊</mo><msqrt id="S4.p1.8.m8.1.1" xref="S4.p1.8.m8.1.1.cmml"><mrow id="S4.p1.8.m8.1.1.1" xref="S4.p1.8.m8.1.1.1.cmml"><mrow id="S4.p1.8.m8.1.1.1.1.1" xref="S4.p1.8.m8.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.p1.8.m8.1.1.1.1.1.2" xref="S4.p1.8.m8.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.p1.8.m8.1.1.1.1.1.1" xref="S4.p1.8.m8.1.1.1.1.1.1.cmml"><mi id="S4.p1.8.m8.1.1.1.1.1.1.2" xref="S4.p1.8.m8.1.1.1.1.1.1.2.cmml">d</mi><mo id="S4.p1.8.m8.1.1.1.1.1.1.1" xref="S4.p1.8.m8.1.1.1.1.1.1.1.cmml">+</mo><mi id="S4.p1.8.m8.1.1.1.1.1.1.3" xref="S4.p1.8.m8.1.1.1.1.1.1.3.cmml">k</mi></mrow><mo stretchy="false" id="S4.p1.8.m8.1.1.1.1.1.3" xref="S4.p1.8.m8.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S4.p1.8.m8.1.1.1.2" xref="S4.p1.8.m8.1.1.1.2.cmml">​</mo><mi id="S4.p1.8.m8.1.1.1.3" xref="S4.p1.8.m8.1.1.1.3.cmml">r</mi></mrow></msqrt><mo stretchy="false" id="S4.p1.8.m8.1.2.3.2.2" xref="S4.p1.8.m8.1.2.3.1.1.cmml">⌋</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.8.m8.1b"><apply id="S4.p1.8.m8.1.2.cmml" xref="S4.p1.8.m8.1.2"><eq id="S4.p1.8.m8.1.2.1.cmml" xref="S4.p1.8.m8.1.2.1"></eq><apply id="S4.p1.8.m8.1.2.2.cmml" xref="S4.p1.8.m8.1.2.2"><ci id="S4.p1.8.m8.1.2.2.1.cmml" xref="S4.p1.8.m8.1.2.2.1">^</ci><ci id="S4.p1.8.m8.1.2.2.2.cmml" xref="S4.p1.8.m8.1.2.2.2">𝑟</ci></apply><apply id="S4.p1.8.m8.1.2.3.1.cmml" xref="S4.p1.8.m8.1.2.3.2"><floor id="S4.p1.8.m8.1.2.3.1.1.cmml" xref="S4.p1.8.m8.1.2.3.2.1"></floor><apply id="S4.p1.8.m8.1.1.cmml" xref="S4.p1.8.m8.1.1"><root id="S4.p1.8.m8.1.1a.cmml" xref="S4.p1.8.m8.1.1"></root><apply id="S4.p1.8.m8.1.1.1.cmml" xref="S4.p1.8.m8.1.1.1"><times id="S4.p1.8.m8.1.1.1.2.cmml" xref="S4.p1.8.m8.1.1.1.2"></times><apply id="S4.p1.8.m8.1.1.1.1.1.1.cmml" xref="S4.p1.8.m8.1.1.1.1.1"><plus id="S4.p1.8.m8.1.1.1.1.1.1.1.cmml" xref="S4.p1.8.m8.1.1.1.1.1.1.1"></plus><ci id="S4.p1.8.m8.1.1.1.1.1.1.2.cmml" xref="S4.p1.8.m8.1.1.1.1.1.1.2">𝑑</ci><ci id="S4.p1.8.m8.1.1.1.1.1.1.3.cmml" xref="S4.p1.8.m8.1.1.1.1.1.1.3">𝑘</ci></apply><ci id="S4.p1.8.m8.1.1.1.3.cmml" xref="S4.p1.8.m8.1.1.1.3">𝑟</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.8.m8.1c">\hat{r}=\lfloor\sqrt{(d+k)r}\rfloor</annotation></semantics></math> can achieve the highest rank due to <math id="S4.p1.9.m9.3" class="ltx_Math" alttext="r\ll\min(d,k)" display="inline"><semantics id="S4.p1.9.m9.3a"><mrow id="S4.p1.9.m9.3.4" xref="S4.p1.9.m9.3.4.cmml"><mi id="S4.p1.9.m9.3.4.2" xref="S4.p1.9.m9.3.4.2.cmml">r</mi><mo id="S4.p1.9.m9.3.4.1" xref="S4.p1.9.m9.3.4.1.cmml">≪</mo><mrow id="S4.p1.9.m9.3.4.3.2" xref="S4.p1.9.m9.3.4.3.1.cmml"><mi id="S4.p1.9.m9.1.1" xref="S4.p1.9.m9.1.1.cmml">min</mi><mo id="S4.p1.9.m9.3.4.3.2a" xref="S4.p1.9.m9.3.4.3.1.cmml">⁡</mo><mrow id="S4.p1.9.m9.3.4.3.2.1" xref="S4.p1.9.m9.3.4.3.1.cmml"><mo stretchy="false" id="S4.p1.9.m9.3.4.3.2.1.1" xref="S4.p1.9.m9.3.4.3.1.cmml">(</mo><mi id="S4.p1.9.m9.2.2" xref="S4.p1.9.m9.2.2.cmml">d</mi><mo id="S4.p1.9.m9.3.4.3.2.1.2" xref="S4.p1.9.m9.3.4.3.1.cmml">,</mo><mi id="S4.p1.9.m9.3.3" xref="S4.p1.9.m9.3.3.cmml">k</mi><mo stretchy="false" id="S4.p1.9.m9.3.4.3.2.1.3" xref="S4.p1.9.m9.3.4.3.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.9.m9.3b"><apply id="S4.p1.9.m9.3.4.cmml" xref="S4.p1.9.m9.3.4"><csymbol cd="latexml" id="S4.p1.9.m9.3.4.1.cmml" xref="S4.p1.9.m9.3.4.1">much-less-than</csymbol><ci id="S4.p1.9.m9.3.4.2.cmml" xref="S4.p1.9.m9.3.4.2">𝑟</ci><apply id="S4.p1.9.m9.3.4.3.1.cmml" xref="S4.p1.9.m9.3.4.3.2"><min id="S4.p1.9.m9.1.1.cmml" xref="S4.p1.9.m9.1.1"></min><ci id="S4.p1.9.m9.2.2.cmml" xref="S4.p1.9.m9.2.2">𝑑</ci><ci id="S4.p1.9.m9.3.3.cmml" xref="S4.p1.9.m9.3.3">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.9.m9.3c">r\ll\min(d,k)</annotation></semantics></math>.</p>
</div>
<div id="S4.p2" class="ltx_para">
<p id="S4.p2.1" class="ltx_p">To accomplish this, we need to reduce the input dimension and increase the output dimension for <math id="S4.p2.1.m1.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S4.p2.1.m1.1a"><mi id="S4.p2.1.m1.1.1" xref="S4.p2.1.m1.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S4.p2.1.m1.1b"><ci id="S4.p2.1.m1.1.1.cmml" xref="S4.p2.1.m1.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.1.m1.1c">M</annotation></semantics></math>. Formally,</p>
<table id="S4.E2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E2.m1.18" class="ltx_Math" alttext="\begin{split}h&amp;=W_{0}x+f_{\text{decomp}}\left(Mf_{\text{comp}}\left(x\right)\right)\end{split}" display="block"><semantics id="S4.E2.m1.18a"><mtable columnspacing="0pt" displaystyle="true" id="S4.E2.m1.18.18.2" xref="S4.E2.m1.17.17.1.cmml"><mtr id="S4.E2.m1.18.18.2a" xref="S4.E2.m1.17.17.1.cmml"><mtd class="ltx_align_right" columnalign="right" id="S4.E2.m1.18.18.2b" xref="S4.E2.m1.17.17.1.cmml"><mi id="S4.E2.m1.1.1.1.1.1.1" xref="S4.E2.m1.1.1.1.1.1.1.cmml">h</mi></mtd><mtd class="ltx_align_left" columnalign="left" id="S4.E2.m1.18.18.2c" xref="S4.E2.m1.17.17.1.cmml"><mrow id="S4.E2.m1.18.18.2.17.17.16" xref="S4.E2.m1.17.17.1.cmml"><mi id="S4.E2.m1.18.18.2.17.17.16.17" xref="S4.E2.m1.17.17.1.cmml"></mi><mo id="S4.E2.m1.2.2.2.2.1.1" xref="S4.E2.m1.2.2.2.2.1.1.cmml">=</mo><mrow id="S4.E2.m1.18.18.2.17.17.16.16" xref="S4.E2.m1.17.17.1.cmml"><mrow id="S4.E2.m1.18.18.2.17.17.16.16.2" xref="S4.E2.m1.17.17.1.cmml"><msub id="S4.E2.m1.18.18.2.17.17.16.16.2.2" xref="S4.E2.m1.17.17.1.cmml"><mi id="S4.E2.m1.3.3.3.3.2.2" xref="S4.E2.m1.3.3.3.3.2.2.cmml">W</mi><mn id="S4.E2.m1.4.4.4.4.3.3.1" xref="S4.E2.m1.4.4.4.4.3.3.1.cmml">0</mn></msub><mo lspace="0em" rspace="0em" id="S4.E2.m1.18.18.2.17.17.16.16.2.1" xref="S4.E2.m1.17.17.1.cmml">​</mo><mi id="S4.E2.m1.5.5.5.5.4.4" xref="S4.E2.m1.5.5.5.5.4.4.cmml">x</mi></mrow><mo id="S4.E2.m1.6.6.6.6.5.5" xref="S4.E2.m1.6.6.6.6.5.5.cmml">+</mo><mrow id="S4.E2.m1.18.18.2.17.17.16.16.1" xref="S4.E2.m1.17.17.1.cmml"><msub id="S4.E2.m1.18.18.2.17.17.16.16.1.3" xref="S4.E2.m1.17.17.1.cmml"><mi id="S4.E2.m1.7.7.7.7.6.6" xref="S4.E2.m1.7.7.7.7.6.6.cmml">f</mi><mtext id="S4.E2.m1.8.8.8.8.7.7.1" xref="S4.E2.m1.8.8.8.8.7.7.1a.cmml">decomp</mtext></msub><mo lspace="0em" rspace="0em" id="S4.E2.m1.18.18.2.17.17.16.16.1.2" xref="S4.E2.m1.17.17.1.cmml">​</mo><mrow id="S4.E2.m1.18.18.2.17.17.16.16.1.1.1" xref="S4.E2.m1.17.17.1.cmml"><mo id="S4.E2.m1.9.9.9.9.8.8" xref="S4.E2.m1.17.17.1.cmml">(</mo><mrow id="S4.E2.m1.18.18.2.17.17.16.16.1.1.1.1" xref="S4.E2.m1.17.17.1.cmml"><mi id="S4.E2.m1.10.10.10.10.9.9" xref="S4.E2.m1.10.10.10.10.9.9.cmml">M</mi><mo lspace="0em" rspace="0em" id="S4.E2.m1.18.18.2.17.17.16.16.1.1.1.1.1" xref="S4.E2.m1.17.17.1.cmml">​</mo><msub id="S4.E2.m1.18.18.2.17.17.16.16.1.1.1.1.2" xref="S4.E2.m1.17.17.1.cmml"><mi id="S4.E2.m1.11.11.11.11.10.10" xref="S4.E2.m1.11.11.11.11.10.10.cmml">f</mi><mtext id="S4.E2.m1.12.12.12.12.11.11.1" xref="S4.E2.m1.12.12.12.12.11.11.1a.cmml">comp</mtext></msub><mo lspace="0em" rspace="0em" id="S4.E2.m1.18.18.2.17.17.16.16.1.1.1.1.1a" xref="S4.E2.m1.17.17.1.cmml">​</mo><mrow id="S4.E2.m1.18.18.2.17.17.16.16.1.1.1.1.3" xref="S4.E2.m1.17.17.1.cmml"><mo id="S4.E2.m1.13.13.13.13.12.12" xref="S4.E2.m1.17.17.1.cmml">(</mo><mi id="S4.E2.m1.14.14.14.14.13.13" xref="S4.E2.m1.14.14.14.14.13.13.cmml">x</mi><mo id="S4.E2.m1.15.15.15.15.14.14" xref="S4.E2.m1.17.17.1.cmml">)</mo></mrow></mrow><mo id="S4.E2.m1.16.16.16.16.15.15" xref="S4.E2.m1.17.17.1.cmml">)</mo></mrow></mrow></mrow></mrow></mtd></mtr></mtable><annotation-xml encoding="MathML-Content" id="S4.E2.m1.18b"><apply id="S4.E2.m1.17.17.1.cmml" xref="S4.E2.m1.18.18.2"><eq id="S4.E2.m1.2.2.2.2.1.1.cmml" xref="S4.E2.m1.2.2.2.2.1.1"></eq><ci id="S4.E2.m1.1.1.1.1.1.1.cmml" xref="S4.E2.m1.1.1.1.1.1.1">ℎ</ci><apply id="S4.E2.m1.17.17.1.1.cmml" xref="S4.E2.m1.18.18.2"><plus id="S4.E2.m1.6.6.6.6.5.5.cmml" xref="S4.E2.m1.6.6.6.6.5.5"></plus><apply id="S4.E2.m1.17.17.1.1.3.cmml" xref="S4.E2.m1.18.18.2"><times id="S4.E2.m1.17.17.1.1.3.1.cmml" xref="S4.E2.m1.18.18.2"></times><apply id="S4.E2.m1.17.17.1.1.3.2.cmml" xref="S4.E2.m1.18.18.2"><csymbol cd="ambiguous" id="S4.E2.m1.17.17.1.1.3.2.1.cmml" xref="S4.E2.m1.18.18.2">subscript</csymbol><ci id="S4.E2.m1.3.3.3.3.2.2.cmml" xref="S4.E2.m1.3.3.3.3.2.2">𝑊</ci><cn type="integer" id="S4.E2.m1.4.4.4.4.3.3.1.cmml" xref="S4.E2.m1.4.4.4.4.3.3.1">0</cn></apply><ci id="S4.E2.m1.5.5.5.5.4.4.cmml" xref="S4.E2.m1.5.5.5.5.4.4">𝑥</ci></apply><apply id="S4.E2.m1.17.17.1.1.1.cmml" xref="S4.E2.m1.18.18.2"><times id="S4.E2.m1.17.17.1.1.1.2.cmml" xref="S4.E2.m1.18.18.2"></times><apply id="S4.E2.m1.17.17.1.1.1.3.cmml" xref="S4.E2.m1.18.18.2"><csymbol cd="ambiguous" id="S4.E2.m1.17.17.1.1.1.3.1.cmml" xref="S4.E2.m1.18.18.2">subscript</csymbol><ci id="S4.E2.m1.7.7.7.7.6.6.cmml" xref="S4.E2.m1.7.7.7.7.6.6">𝑓</ci><ci id="S4.E2.m1.8.8.8.8.7.7.1a.cmml" xref="S4.E2.m1.8.8.8.8.7.7.1"><mtext mathsize="70%" id="S4.E2.m1.8.8.8.8.7.7.1.cmml" xref="S4.E2.m1.8.8.8.8.7.7.1">decomp</mtext></ci></apply><apply id="S4.E2.m1.17.17.1.1.1.1.1.1.cmml" xref="S4.E2.m1.18.18.2"><times id="S4.E2.m1.17.17.1.1.1.1.1.1.1.cmml" xref="S4.E2.m1.18.18.2"></times><ci id="S4.E2.m1.10.10.10.10.9.9.cmml" xref="S4.E2.m1.10.10.10.10.9.9">𝑀</ci><apply id="S4.E2.m1.17.17.1.1.1.1.1.1.3.cmml" xref="S4.E2.m1.18.18.2"><csymbol cd="ambiguous" id="S4.E2.m1.17.17.1.1.1.1.1.1.3.1.cmml" xref="S4.E2.m1.18.18.2">subscript</csymbol><ci id="S4.E2.m1.11.11.11.11.10.10.cmml" xref="S4.E2.m1.11.11.11.11.10.10">𝑓</ci><ci id="S4.E2.m1.12.12.12.12.11.11.1a.cmml" xref="S4.E2.m1.12.12.12.12.11.11.1"><mtext mathsize="70%" id="S4.E2.m1.12.12.12.12.11.11.1.cmml" xref="S4.E2.m1.12.12.12.12.11.11.1">comp</mtext></ci></apply><ci id="S4.E2.m1.14.14.14.14.13.13.cmml" xref="S4.E2.m1.14.14.14.14.13.13">𝑥</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E2.m1.18c">\begin{split}h&amp;=W_{0}x+f_{\text{decomp}}\left(Mf_{\text{comp}}\left(x\right)\right)\end{split}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
<p id="S4.p2.13" class="ltx_p">where <math id="S4.p2.2.m1.1" class="ltx_Math" alttext="f_{\text{comp}}:\mathbb{R}^{k}\rightarrow\mathbb{R}^{\hat{r}}" display="inline"><semantics id="S4.p2.2.m1.1a"><mrow id="S4.p2.2.m1.1.1" xref="S4.p2.2.m1.1.1.cmml"><msub id="S4.p2.2.m1.1.1.2" xref="S4.p2.2.m1.1.1.2.cmml"><mi id="S4.p2.2.m1.1.1.2.2" xref="S4.p2.2.m1.1.1.2.2.cmml">f</mi><mtext id="S4.p2.2.m1.1.1.2.3" xref="S4.p2.2.m1.1.1.2.3a.cmml">comp</mtext></msub><mo lspace="0.278em" rspace="0.278em" id="S4.p2.2.m1.1.1.1" xref="S4.p2.2.m1.1.1.1.cmml">:</mo><mrow id="S4.p2.2.m1.1.1.3" xref="S4.p2.2.m1.1.1.3.cmml"><msup id="S4.p2.2.m1.1.1.3.2" xref="S4.p2.2.m1.1.1.3.2.cmml"><mi id="S4.p2.2.m1.1.1.3.2.2" xref="S4.p2.2.m1.1.1.3.2.2.cmml">ℝ</mi><mi id="S4.p2.2.m1.1.1.3.2.3" xref="S4.p2.2.m1.1.1.3.2.3.cmml">k</mi></msup><mo stretchy="false" id="S4.p2.2.m1.1.1.3.1" xref="S4.p2.2.m1.1.1.3.1.cmml">→</mo><msup id="S4.p2.2.m1.1.1.3.3" xref="S4.p2.2.m1.1.1.3.3.cmml"><mi id="S4.p2.2.m1.1.1.3.3.2" xref="S4.p2.2.m1.1.1.3.3.2.cmml">ℝ</mi><mover accent="true" id="S4.p2.2.m1.1.1.3.3.3" xref="S4.p2.2.m1.1.1.3.3.3.cmml"><mi id="S4.p2.2.m1.1.1.3.3.3.2" xref="S4.p2.2.m1.1.1.3.3.3.2.cmml">r</mi><mo id="S4.p2.2.m1.1.1.3.3.3.1" xref="S4.p2.2.m1.1.1.3.3.3.1.cmml">^</mo></mover></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.p2.2.m1.1b"><apply id="S4.p2.2.m1.1.1.cmml" xref="S4.p2.2.m1.1.1"><ci id="S4.p2.2.m1.1.1.1.cmml" xref="S4.p2.2.m1.1.1.1">:</ci><apply id="S4.p2.2.m1.1.1.2.cmml" xref="S4.p2.2.m1.1.1.2"><csymbol cd="ambiguous" id="S4.p2.2.m1.1.1.2.1.cmml" xref="S4.p2.2.m1.1.1.2">subscript</csymbol><ci id="S4.p2.2.m1.1.1.2.2.cmml" xref="S4.p2.2.m1.1.1.2.2">𝑓</ci><ci id="S4.p2.2.m1.1.1.2.3a.cmml" xref="S4.p2.2.m1.1.1.2.3"><mtext mathsize="70%" id="S4.p2.2.m1.1.1.2.3.cmml" xref="S4.p2.2.m1.1.1.2.3">comp</mtext></ci></apply><apply id="S4.p2.2.m1.1.1.3.cmml" xref="S4.p2.2.m1.1.1.3"><ci id="S4.p2.2.m1.1.1.3.1.cmml" xref="S4.p2.2.m1.1.1.3.1">→</ci><apply id="S4.p2.2.m1.1.1.3.2.cmml" xref="S4.p2.2.m1.1.1.3.2"><csymbol cd="ambiguous" id="S4.p2.2.m1.1.1.3.2.1.cmml" xref="S4.p2.2.m1.1.1.3.2">superscript</csymbol><ci id="S4.p2.2.m1.1.1.3.2.2.cmml" xref="S4.p2.2.m1.1.1.3.2.2">ℝ</ci><ci id="S4.p2.2.m1.1.1.3.2.3.cmml" xref="S4.p2.2.m1.1.1.3.2.3">𝑘</ci></apply><apply id="S4.p2.2.m1.1.1.3.3.cmml" xref="S4.p2.2.m1.1.1.3.3"><csymbol cd="ambiguous" id="S4.p2.2.m1.1.1.3.3.1.cmml" xref="S4.p2.2.m1.1.1.3.3">superscript</csymbol><ci id="S4.p2.2.m1.1.1.3.3.2.cmml" xref="S4.p2.2.m1.1.1.3.3.2">ℝ</ci><apply id="S4.p2.2.m1.1.1.3.3.3.cmml" xref="S4.p2.2.m1.1.1.3.3.3"><ci id="S4.p2.2.m1.1.1.3.3.3.1.cmml" xref="S4.p2.2.m1.1.1.3.3.3.1">^</ci><ci id="S4.p2.2.m1.1.1.3.3.3.2.cmml" xref="S4.p2.2.m1.1.1.3.3.3.2">𝑟</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.2.m1.1c">f_{\text{comp}}:\mathbb{R}^{k}\rightarrow\mathbb{R}^{\hat{r}}</annotation></semantics></math> denotes the function that decreases the input dimension of <math id="S4.p2.3.m2.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S4.p2.3.m2.1a"><mi id="S4.p2.3.m2.1.1" xref="S4.p2.3.m2.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S4.p2.3.m2.1b"><ci id="S4.p2.3.m2.1.1.cmml" xref="S4.p2.3.m2.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.3.m2.1c">x</annotation></semantics></math> from <math id="S4.p2.4.m3.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S4.p2.4.m3.1a"><mi id="S4.p2.4.m3.1.1" xref="S4.p2.4.m3.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S4.p2.4.m3.1b"><ci id="S4.p2.4.m3.1.1.cmml" xref="S4.p2.4.m3.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.4.m3.1c">k</annotation></semantics></math> to <math id="S4.p2.5.m4.1" class="ltx_Math" alttext="\hat{r}" display="inline"><semantics id="S4.p2.5.m4.1a"><mover accent="true" id="S4.p2.5.m4.1.1" xref="S4.p2.5.m4.1.1.cmml"><mi id="S4.p2.5.m4.1.1.2" xref="S4.p2.5.m4.1.1.2.cmml">r</mi><mo id="S4.p2.5.m4.1.1.1" xref="S4.p2.5.m4.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="S4.p2.5.m4.1b"><apply id="S4.p2.5.m4.1.1.cmml" xref="S4.p2.5.m4.1.1"><ci id="S4.p2.5.m4.1.1.1.cmml" xref="S4.p2.5.m4.1.1.1">^</ci><ci id="S4.p2.5.m4.1.1.2.cmml" xref="S4.p2.5.m4.1.1.2">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.5.m4.1c">\hat{r}</annotation></semantics></math>, and <math id="S4.p2.6.m5.1" class="ltx_Math" alttext="f_{\text{decomp}}:\mathbb{R}^{\hat{r}}\rightarrow\mathbb{R}^{d}" display="inline"><semantics id="S4.p2.6.m5.1a"><mrow id="S4.p2.6.m5.1.1" xref="S4.p2.6.m5.1.1.cmml"><msub id="S4.p2.6.m5.1.1.2" xref="S4.p2.6.m5.1.1.2.cmml"><mi id="S4.p2.6.m5.1.1.2.2" xref="S4.p2.6.m5.1.1.2.2.cmml">f</mi><mtext id="S4.p2.6.m5.1.1.2.3" xref="S4.p2.6.m5.1.1.2.3a.cmml">decomp</mtext></msub><mo lspace="0.278em" rspace="0.278em" id="S4.p2.6.m5.1.1.1" xref="S4.p2.6.m5.1.1.1.cmml">:</mo><mrow id="S4.p2.6.m5.1.1.3" xref="S4.p2.6.m5.1.1.3.cmml"><msup id="S4.p2.6.m5.1.1.3.2" xref="S4.p2.6.m5.1.1.3.2.cmml"><mi id="S4.p2.6.m5.1.1.3.2.2" xref="S4.p2.6.m5.1.1.3.2.2.cmml">ℝ</mi><mover accent="true" id="S4.p2.6.m5.1.1.3.2.3" xref="S4.p2.6.m5.1.1.3.2.3.cmml"><mi id="S4.p2.6.m5.1.1.3.2.3.2" xref="S4.p2.6.m5.1.1.3.2.3.2.cmml">r</mi><mo id="S4.p2.6.m5.1.1.3.2.3.1" xref="S4.p2.6.m5.1.1.3.2.3.1.cmml">^</mo></mover></msup><mo stretchy="false" id="S4.p2.6.m5.1.1.3.1" xref="S4.p2.6.m5.1.1.3.1.cmml">→</mo><msup id="S4.p2.6.m5.1.1.3.3" xref="S4.p2.6.m5.1.1.3.3.cmml"><mi id="S4.p2.6.m5.1.1.3.3.2" xref="S4.p2.6.m5.1.1.3.3.2.cmml">ℝ</mi><mi id="S4.p2.6.m5.1.1.3.3.3" xref="S4.p2.6.m5.1.1.3.3.3.cmml">d</mi></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.p2.6.m5.1b"><apply id="S4.p2.6.m5.1.1.cmml" xref="S4.p2.6.m5.1.1"><ci id="S4.p2.6.m5.1.1.1.cmml" xref="S4.p2.6.m5.1.1.1">:</ci><apply id="S4.p2.6.m5.1.1.2.cmml" xref="S4.p2.6.m5.1.1.2"><csymbol cd="ambiguous" id="S4.p2.6.m5.1.1.2.1.cmml" xref="S4.p2.6.m5.1.1.2">subscript</csymbol><ci id="S4.p2.6.m5.1.1.2.2.cmml" xref="S4.p2.6.m5.1.1.2.2">𝑓</ci><ci id="S4.p2.6.m5.1.1.2.3a.cmml" xref="S4.p2.6.m5.1.1.2.3"><mtext mathsize="70%" id="S4.p2.6.m5.1.1.2.3.cmml" xref="S4.p2.6.m5.1.1.2.3">decomp</mtext></ci></apply><apply id="S4.p2.6.m5.1.1.3.cmml" xref="S4.p2.6.m5.1.1.3"><ci id="S4.p2.6.m5.1.1.3.1.cmml" xref="S4.p2.6.m5.1.1.3.1">→</ci><apply id="S4.p2.6.m5.1.1.3.2.cmml" xref="S4.p2.6.m5.1.1.3.2"><csymbol cd="ambiguous" id="S4.p2.6.m5.1.1.3.2.1.cmml" xref="S4.p2.6.m5.1.1.3.2">superscript</csymbol><ci id="S4.p2.6.m5.1.1.3.2.2.cmml" xref="S4.p2.6.m5.1.1.3.2.2">ℝ</ci><apply id="S4.p2.6.m5.1.1.3.2.3.cmml" xref="S4.p2.6.m5.1.1.3.2.3"><ci id="S4.p2.6.m5.1.1.3.2.3.1.cmml" xref="S4.p2.6.m5.1.1.3.2.3.1">^</ci><ci id="S4.p2.6.m5.1.1.3.2.3.2.cmml" xref="S4.p2.6.m5.1.1.3.2.3.2">𝑟</ci></apply></apply><apply id="S4.p2.6.m5.1.1.3.3.cmml" xref="S4.p2.6.m5.1.1.3.3"><csymbol cd="ambiguous" id="S4.p2.6.m5.1.1.3.3.1.cmml" xref="S4.p2.6.m5.1.1.3.3">superscript</csymbol><ci id="S4.p2.6.m5.1.1.3.3.2.cmml" xref="S4.p2.6.m5.1.1.3.3.2">ℝ</ci><ci id="S4.p2.6.m5.1.1.3.3.3.cmml" xref="S4.p2.6.m5.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.6.m5.1c">f_{\text{decomp}}:\mathbb{R}^{\hat{r}}\rightarrow\mathbb{R}^{d}</annotation></semantics></math> represents the function that enhances the output dimension from <math id="S4.p2.7.m6.1" class="ltx_Math" alttext="\hat{r}" display="inline"><semantics id="S4.p2.7.m6.1a"><mover accent="true" id="S4.p2.7.m6.1.1" xref="S4.p2.7.m6.1.1.cmml"><mi id="S4.p2.7.m6.1.1.2" xref="S4.p2.7.m6.1.1.2.cmml">r</mi><mo id="S4.p2.7.m6.1.1.1" xref="S4.p2.7.m6.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="S4.p2.7.m6.1b"><apply id="S4.p2.7.m6.1.1.cmml" xref="S4.p2.7.m6.1.1"><ci id="S4.p2.7.m6.1.1.1.cmml" xref="S4.p2.7.m6.1.1.1">^</ci><ci id="S4.p2.7.m6.1.1.2.cmml" xref="S4.p2.7.m6.1.1.2">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.7.m6.1c">\hat{r}</annotation></semantics></math> to <math id="S4.p2.8.m7.1" class="ltx_Math" alttext="d" display="inline"><semantics id="S4.p2.8.m7.1a"><mi id="S4.p2.8.m7.1.1" xref="S4.p2.8.m7.1.1.cmml">d</mi><annotation-xml encoding="MathML-Content" id="S4.p2.8.m7.1b"><ci id="S4.p2.8.m7.1.1.cmml" xref="S4.p2.8.m7.1.1">𝑑</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.8.m7.1c">d</annotation></semantics></math>.
Furthermore, these two functions ought to be non-parameterized operators and expected to execute in linear time corresponding to the dimension. They should also have corresponding function, <math id="S4.p2.9.m8.1" class="ltx_Math" alttext="f_{\overline{\text{comp}}}:\mathbb{R}^{\hat{r}\times\hat{r}}\rightarrow\mathbb{R}^{\hat{r}\times k}" display="inline"><semantics id="S4.p2.9.m8.1a"><mrow id="S4.p2.9.m8.1.1" xref="S4.p2.9.m8.1.1.cmml"><msub id="S4.p2.9.m8.1.1.2" xref="S4.p2.9.m8.1.1.2.cmml"><mi id="S4.p2.9.m8.1.1.2.2" xref="S4.p2.9.m8.1.1.2.2.cmml">f</mi><mover accent="true" id="S4.p2.9.m8.1.1.2.3" xref="S4.p2.9.m8.1.1.2.3.cmml"><mtext id="S4.p2.9.m8.1.1.2.3.2" xref="S4.p2.9.m8.1.1.2.3.2a.cmml">comp</mtext><mo id="S4.p2.9.m8.1.1.2.3.1" xref="S4.p2.9.m8.1.1.2.3.1.cmml">¯</mo></mover></msub><mo lspace="0.278em" rspace="0.278em" id="S4.p2.9.m8.1.1.1" xref="S4.p2.9.m8.1.1.1.cmml">:</mo><mrow id="S4.p2.9.m8.1.1.3" xref="S4.p2.9.m8.1.1.3.cmml"><msup id="S4.p2.9.m8.1.1.3.2" xref="S4.p2.9.m8.1.1.3.2.cmml"><mi id="S4.p2.9.m8.1.1.3.2.2" xref="S4.p2.9.m8.1.1.3.2.2.cmml">ℝ</mi><mrow id="S4.p2.9.m8.1.1.3.2.3" xref="S4.p2.9.m8.1.1.3.2.3.cmml"><mover accent="true" id="S4.p2.9.m8.1.1.3.2.3.2" xref="S4.p2.9.m8.1.1.3.2.3.2.cmml"><mi id="S4.p2.9.m8.1.1.3.2.3.2.2" xref="S4.p2.9.m8.1.1.3.2.3.2.2.cmml">r</mi><mo id="S4.p2.9.m8.1.1.3.2.3.2.1" xref="S4.p2.9.m8.1.1.3.2.3.2.1.cmml">^</mo></mover><mo lspace="0.222em" rspace="0.222em" id="S4.p2.9.m8.1.1.3.2.3.1" xref="S4.p2.9.m8.1.1.3.2.3.1.cmml">×</mo><mover accent="true" id="S4.p2.9.m8.1.1.3.2.3.3" xref="S4.p2.9.m8.1.1.3.2.3.3.cmml"><mi id="S4.p2.9.m8.1.1.3.2.3.3.2" xref="S4.p2.9.m8.1.1.3.2.3.3.2.cmml">r</mi><mo id="S4.p2.9.m8.1.1.3.2.3.3.1" xref="S4.p2.9.m8.1.1.3.2.3.3.1.cmml">^</mo></mover></mrow></msup><mo stretchy="false" id="S4.p2.9.m8.1.1.3.1" xref="S4.p2.9.m8.1.1.3.1.cmml">→</mo><msup id="S4.p2.9.m8.1.1.3.3" xref="S4.p2.9.m8.1.1.3.3.cmml"><mi id="S4.p2.9.m8.1.1.3.3.2" xref="S4.p2.9.m8.1.1.3.3.2.cmml">ℝ</mi><mrow id="S4.p2.9.m8.1.1.3.3.3" xref="S4.p2.9.m8.1.1.3.3.3.cmml"><mover accent="true" id="S4.p2.9.m8.1.1.3.3.3.2" xref="S4.p2.9.m8.1.1.3.3.3.2.cmml"><mi id="S4.p2.9.m8.1.1.3.3.3.2.2" xref="S4.p2.9.m8.1.1.3.3.3.2.2.cmml">r</mi><mo id="S4.p2.9.m8.1.1.3.3.3.2.1" xref="S4.p2.9.m8.1.1.3.3.3.2.1.cmml">^</mo></mover><mo lspace="0.222em" rspace="0.222em" id="S4.p2.9.m8.1.1.3.3.3.1" xref="S4.p2.9.m8.1.1.3.3.3.1.cmml">×</mo><mi id="S4.p2.9.m8.1.1.3.3.3.3" xref="S4.p2.9.m8.1.1.3.3.3.3.cmml">k</mi></mrow></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.p2.9.m8.1b"><apply id="S4.p2.9.m8.1.1.cmml" xref="S4.p2.9.m8.1.1"><ci id="S4.p2.9.m8.1.1.1.cmml" xref="S4.p2.9.m8.1.1.1">:</ci><apply id="S4.p2.9.m8.1.1.2.cmml" xref="S4.p2.9.m8.1.1.2"><csymbol cd="ambiguous" id="S4.p2.9.m8.1.1.2.1.cmml" xref="S4.p2.9.m8.1.1.2">subscript</csymbol><ci id="S4.p2.9.m8.1.1.2.2.cmml" xref="S4.p2.9.m8.1.1.2.2">𝑓</ci><apply id="S4.p2.9.m8.1.1.2.3.cmml" xref="S4.p2.9.m8.1.1.2.3"><ci id="S4.p2.9.m8.1.1.2.3.1.cmml" xref="S4.p2.9.m8.1.1.2.3.1">¯</ci><ci id="S4.p2.9.m8.1.1.2.3.2a.cmml" xref="S4.p2.9.m8.1.1.2.3.2"><mtext mathsize="70%" id="S4.p2.9.m8.1.1.2.3.2.cmml" xref="S4.p2.9.m8.1.1.2.3.2">comp</mtext></ci></apply></apply><apply id="S4.p2.9.m8.1.1.3.cmml" xref="S4.p2.9.m8.1.1.3"><ci id="S4.p2.9.m8.1.1.3.1.cmml" xref="S4.p2.9.m8.1.1.3.1">→</ci><apply id="S4.p2.9.m8.1.1.3.2.cmml" xref="S4.p2.9.m8.1.1.3.2"><csymbol cd="ambiguous" id="S4.p2.9.m8.1.1.3.2.1.cmml" xref="S4.p2.9.m8.1.1.3.2">superscript</csymbol><ci id="S4.p2.9.m8.1.1.3.2.2.cmml" xref="S4.p2.9.m8.1.1.3.2.2">ℝ</ci><apply id="S4.p2.9.m8.1.1.3.2.3.cmml" xref="S4.p2.9.m8.1.1.3.2.3"><times id="S4.p2.9.m8.1.1.3.2.3.1.cmml" xref="S4.p2.9.m8.1.1.3.2.3.1"></times><apply id="S4.p2.9.m8.1.1.3.2.3.2.cmml" xref="S4.p2.9.m8.1.1.3.2.3.2"><ci id="S4.p2.9.m8.1.1.3.2.3.2.1.cmml" xref="S4.p2.9.m8.1.1.3.2.3.2.1">^</ci><ci id="S4.p2.9.m8.1.1.3.2.3.2.2.cmml" xref="S4.p2.9.m8.1.1.3.2.3.2.2">𝑟</ci></apply><apply id="S4.p2.9.m8.1.1.3.2.3.3.cmml" xref="S4.p2.9.m8.1.1.3.2.3.3"><ci id="S4.p2.9.m8.1.1.3.2.3.3.1.cmml" xref="S4.p2.9.m8.1.1.3.2.3.3.1">^</ci><ci id="S4.p2.9.m8.1.1.3.2.3.3.2.cmml" xref="S4.p2.9.m8.1.1.3.2.3.3.2">𝑟</ci></apply></apply></apply><apply id="S4.p2.9.m8.1.1.3.3.cmml" xref="S4.p2.9.m8.1.1.3.3"><csymbol cd="ambiguous" id="S4.p2.9.m8.1.1.3.3.1.cmml" xref="S4.p2.9.m8.1.1.3.3">superscript</csymbol><ci id="S4.p2.9.m8.1.1.3.3.2.cmml" xref="S4.p2.9.m8.1.1.3.3.2">ℝ</ci><apply id="S4.p2.9.m8.1.1.3.3.3.cmml" xref="S4.p2.9.m8.1.1.3.3.3"><times id="S4.p2.9.m8.1.1.3.3.3.1.cmml" xref="S4.p2.9.m8.1.1.3.3.3.1"></times><apply id="S4.p2.9.m8.1.1.3.3.3.2.cmml" xref="S4.p2.9.m8.1.1.3.3.3.2"><ci id="S4.p2.9.m8.1.1.3.3.3.2.1.cmml" xref="S4.p2.9.m8.1.1.3.3.3.2.1">^</ci><ci id="S4.p2.9.m8.1.1.3.3.3.2.2.cmml" xref="S4.p2.9.m8.1.1.3.3.3.2.2">𝑟</ci></apply><ci id="S4.p2.9.m8.1.1.3.3.3.3.cmml" xref="S4.p2.9.m8.1.1.3.3.3.3">𝑘</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.9.m8.1c">f_{\overline{\text{comp}}}:\mathbb{R}^{\hat{r}\times\hat{r}}\rightarrow\mathbb{R}^{\hat{r}\times k}</annotation></semantics></math> and <math id="S4.p2.10.m9.1" class="ltx_Math" alttext="f_{\overline{\text{decomp}}}:\mathbb{R}^{\hat{r}\times k}\rightarrow\mathbb{R}^{d\times k}" display="inline"><semantics id="S4.p2.10.m9.1a"><mrow id="S4.p2.10.m9.1.1" xref="S4.p2.10.m9.1.1.cmml"><msub id="S4.p2.10.m9.1.1.2" xref="S4.p2.10.m9.1.1.2.cmml"><mi id="S4.p2.10.m9.1.1.2.2" xref="S4.p2.10.m9.1.1.2.2.cmml">f</mi><mover accent="true" id="S4.p2.10.m9.1.1.2.3" xref="S4.p2.10.m9.1.1.2.3.cmml"><mtext id="S4.p2.10.m9.1.1.2.3.2" xref="S4.p2.10.m9.1.1.2.3.2a.cmml">decomp</mtext><mo id="S4.p2.10.m9.1.1.2.3.1" xref="S4.p2.10.m9.1.1.2.3.1.cmml">¯</mo></mover></msub><mo lspace="0.278em" rspace="0.278em" id="S4.p2.10.m9.1.1.1" xref="S4.p2.10.m9.1.1.1.cmml">:</mo><mrow id="S4.p2.10.m9.1.1.3" xref="S4.p2.10.m9.1.1.3.cmml"><msup id="S4.p2.10.m9.1.1.3.2" xref="S4.p2.10.m9.1.1.3.2.cmml"><mi id="S4.p2.10.m9.1.1.3.2.2" xref="S4.p2.10.m9.1.1.3.2.2.cmml">ℝ</mi><mrow id="S4.p2.10.m9.1.1.3.2.3" xref="S4.p2.10.m9.1.1.3.2.3.cmml"><mover accent="true" id="S4.p2.10.m9.1.1.3.2.3.2" xref="S4.p2.10.m9.1.1.3.2.3.2.cmml"><mi id="S4.p2.10.m9.1.1.3.2.3.2.2" xref="S4.p2.10.m9.1.1.3.2.3.2.2.cmml">r</mi><mo id="S4.p2.10.m9.1.1.3.2.3.2.1" xref="S4.p2.10.m9.1.1.3.2.3.2.1.cmml">^</mo></mover><mo lspace="0.222em" rspace="0.222em" id="S4.p2.10.m9.1.1.3.2.3.1" xref="S4.p2.10.m9.1.1.3.2.3.1.cmml">×</mo><mi id="S4.p2.10.m9.1.1.3.2.3.3" xref="S4.p2.10.m9.1.1.3.2.3.3.cmml">k</mi></mrow></msup><mo stretchy="false" id="S4.p2.10.m9.1.1.3.1" xref="S4.p2.10.m9.1.1.3.1.cmml">→</mo><msup id="S4.p2.10.m9.1.1.3.3" xref="S4.p2.10.m9.1.1.3.3.cmml"><mi id="S4.p2.10.m9.1.1.3.3.2" xref="S4.p2.10.m9.1.1.3.3.2.cmml">ℝ</mi><mrow id="S4.p2.10.m9.1.1.3.3.3" xref="S4.p2.10.m9.1.1.3.3.3.cmml"><mi id="S4.p2.10.m9.1.1.3.3.3.2" xref="S4.p2.10.m9.1.1.3.3.3.2.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="S4.p2.10.m9.1.1.3.3.3.1" xref="S4.p2.10.m9.1.1.3.3.3.1.cmml">×</mo><mi id="S4.p2.10.m9.1.1.3.3.3.3" xref="S4.p2.10.m9.1.1.3.3.3.3.cmml">k</mi></mrow></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.p2.10.m9.1b"><apply id="S4.p2.10.m9.1.1.cmml" xref="S4.p2.10.m9.1.1"><ci id="S4.p2.10.m9.1.1.1.cmml" xref="S4.p2.10.m9.1.1.1">:</ci><apply id="S4.p2.10.m9.1.1.2.cmml" xref="S4.p2.10.m9.1.1.2"><csymbol cd="ambiguous" id="S4.p2.10.m9.1.1.2.1.cmml" xref="S4.p2.10.m9.1.1.2">subscript</csymbol><ci id="S4.p2.10.m9.1.1.2.2.cmml" xref="S4.p2.10.m9.1.1.2.2">𝑓</ci><apply id="S4.p2.10.m9.1.1.2.3.cmml" xref="S4.p2.10.m9.1.1.2.3"><ci id="S4.p2.10.m9.1.1.2.3.1.cmml" xref="S4.p2.10.m9.1.1.2.3.1">¯</ci><ci id="S4.p2.10.m9.1.1.2.3.2a.cmml" xref="S4.p2.10.m9.1.1.2.3.2"><mtext mathsize="70%" id="S4.p2.10.m9.1.1.2.3.2.cmml" xref="S4.p2.10.m9.1.1.2.3.2">decomp</mtext></ci></apply></apply><apply id="S4.p2.10.m9.1.1.3.cmml" xref="S4.p2.10.m9.1.1.3"><ci id="S4.p2.10.m9.1.1.3.1.cmml" xref="S4.p2.10.m9.1.1.3.1">→</ci><apply id="S4.p2.10.m9.1.1.3.2.cmml" xref="S4.p2.10.m9.1.1.3.2"><csymbol cd="ambiguous" id="S4.p2.10.m9.1.1.3.2.1.cmml" xref="S4.p2.10.m9.1.1.3.2">superscript</csymbol><ci id="S4.p2.10.m9.1.1.3.2.2.cmml" xref="S4.p2.10.m9.1.1.3.2.2">ℝ</ci><apply id="S4.p2.10.m9.1.1.3.2.3.cmml" xref="S4.p2.10.m9.1.1.3.2.3"><times id="S4.p2.10.m9.1.1.3.2.3.1.cmml" xref="S4.p2.10.m9.1.1.3.2.3.1"></times><apply id="S4.p2.10.m9.1.1.3.2.3.2.cmml" xref="S4.p2.10.m9.1.1.3.2.3.2"><ci id="S4.p2.10.m9.1.1.3.2.3.2.1.cmml" xref="S4.p2.10.m9.1.1.3.2.3.2.1">^</ci><ci id="S4.p2.10.m9.1.1.3.2.3.2.2.cmml" xref="S4.p2.10.m9.1.1.3.2.3.2.2">𝑟</ci></apply><ci id="S4.p2.10.m9.1.1.3.2.3.3.cmml" xref="S4.p2.10.m9.1.1.3.2.3.3">𝑘</ci></apply></apply><apply id="S4.p2.10.m9.1.1.3.3.cmml" xref="S4.p2.10.m9.1.1.3.3"><csymbol cd="ambiguous" id="S4.p2.10.m9.1.1.3.3.1.cmml" xref="S4.p2.10.m9.1.1.3.3">superscript</csymbol><ci id="S4.p2.10.m9.1.1.3.3.2.cmml" xref="S4.p2.10.m9.1.1.3.3.2">ℝ</ci><apply id="S4.p2.10.m9.1.1.3.3.3.cmml" xref="S4.p2.10.m9.1.1.3.3.3"><times id="S4.p2.10.m9.1.1.3.3.3.1.cmml" xref="S4.p2.10.m9.1.1.3.3.3.1"></times><ci id="S4.p2.10.m9.1.1.3.3.3.2.cmml" xref="S4.p2.10.m9.1.1.3.3.3.2">𝑑</ci><ci id="S4.p2.10.m9.1.1.3.3.3.3.cmml" xref="S4.p2.10.m9.1.1.3.3.3.3">𝑘</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.10.m9.1c">f_{\overline{\text{decomp}}}:\mathbb{R}^{\hat{r}\times k}\rightarrow\mathbb{R}^{d\times k}</annotation></semantics></math>, to transform <math id="S4.p2.11.m10.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S4.p2.11.m10.1a"><mi id="S4.p2.11.m10.1.1" xref="S4.p2.11.m10.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S4.p2.11.m10.1b"><ci id="S4.p2.11.m10.1.1.cmml" xref="S4.p2.11.m10.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.11.m10.1c">M</annotation></semantics></math> into <math id="S4.p2.12.m11.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S4.p2.12.m11.1a"><mrow id="S4.p2.12.m11.1.1" xref="S4.p2.12.m11.1.1.cmml"><mi mathvariant="normal" id="S4.p2.12.m11.1.1.2" xref="S4.p2.12.m11.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.p2.12.m11.1.1.1" xref="S4.p2.12.m11.1.1.1.cmml">​</mo><mi id="S4.p2.12.m11.1.1.3" xref="S4.p2.12.m11.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p2.12.m11.1b"><apply id="S4.p2.12.m11.1.1.cmml" xref="S4.p2.12.m11.1.1"><times id="S4.p2.12.m11.1.1.1.cmml" xref="S4.p2.12.m11.1.1.1"></times><ci id="S4.p2.12.m11.1.1.2.cmml" xref="S4.p2.12.m11.1.1.2">Δ</ci><ci id="S4.p2.12.m11.1.1.3.cmml" xref="S4.p2.12.m11.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.12.m11.1c">\Delta W</annotation></semantics></math>. For any <math id="S4.p2.13.m12.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S4.p2.13.m12.1a"><mi id="S4.p2.13.m12.1.1" xref="S4.p2.13.m12.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S4.p2.13.m12.1b"><ci id="S4.p2.13.m12.1.1.cmml" xref="S4.p2.13.m12.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.13.m12.1c">x</annotation></semantics></math>, the following should hold:</p>
<table id="S4.E3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E3.m1.3" class="ltx_Math" alttext="f_{\text{decomp}}\left(Mf_{\text{comp}}\left(x\right)\right)=\Delta Wx,\forall x\in\mathbb{R}^{k}\\
" display="block"><semantics id="S4.E3.m1.3a"><mrow id="S4.E3.m1.3.3.2" xref="S4.E3.m1.3.3.3.cmml"><mrow id="S4.E3.m1.2.2.1.1" xref="S4.E3.m1.2.2.1.1.cmml"><mrow id="S4.E3.m1.2.2.1.1.1" xref="S4.E3.m1.2.2.1.1.1.cmml"><msub id="S4.E3.m1.2.2.1.1.1.3" xref="S4.E3.m1.2.2.1.1.1.3.cmml"><mi id="S4.E3.m1.2.2.1.1.1.3.2" xref="S4.E3.m1.2.2.1.1.1.3.2.cmml">f</mi><mtext id="S4.E3.m1.2.2.1.1.1.3.3" xref="S4.E3.m1.2.2.1.1.1.3.3a.cmml">decomp</mtext></msub><mo lspace="0em" rspace="0em" id="S4.E3.m1.2.2.1.1.1.2" xref="S4.E3.m1.2.2.1.1.1.2.cmml">​</mo><mrow id="S4.E3.m1.2.2.1.1.1.1.1" xref="S4.E3.m1.2.2.1.1.1.1.1.1.cmml"><mo id="S4.E3.m1.2.2.1.1.1.1.1.2" xref="S4.E3.m1.2.2.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.E3.m1.2.2.1.1.1.1.1.1" xref="S4.E3.m1.2.2.1.1.1.1.1.1.cmml"><mi id="S4.E3.m1.2.2.1.1.1.1.1.1.2" xref="S4.E3.m1.2.2.1.1.1.1.1.1.2.cmml">M</mi><mo lspace="0em" rspace="0em" id="S4.E3.m1.2.2.1.1.1.1.1.1.1" xref="S4.E3.m1.2.2.1.1.1.1.1.1.1.cmml">​</mo><msub id="S4.E3.m1.2.2.1.1.1.1.1.1.3" xref="S4.E3.m1.2.2.1.1.1.1.1.1.3.cmml"><mi id="S4.E3.m1.2.2.1.1.1.1.1.1.3.2" xref="S4.E3.m1.2.2.1.1.1.1.1.1.3.2.cmml">f</mi><mtext id="S4.E3.m1.2.2.1.1.1.1.1.1.3.3" xref="S4.E3.m1.2.2.1.1.1.1.1.1.3.3a.cmml">comp</mtext></msub><mo lspace="0em" rspace="0em" id="S4.E3.m1.2.2.1.1.1.1.1.1.1a" xref="S4.E3.m1.2.2.1.1.1.1.1.1.1.cmml">​</mo><mrow id="S4.E3.m1.2.2.1.1.1.1.1.1.4.2" xref="S4.E3.m1.2.2.1.1.1.1.1.1.cmml"><mo id="S4.E3.m1.2.2.1.1.1.1.1.1.4.2.1" xref="S4.E3.m1.2.2.1.1.1.1.1.1.cmml">(</mo><mi id="S4.E3.m1.1.1" xref="S4.E3.m1.1.1.cmml">x</mi><mo id="S4.E3.m1.2.2.1.1.1.1.1.1.4.2.2" xref="S4.E3.m1.2.2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S4.E3.m1.2.2.1.1.1.1.1.3" xref="S4.E3.m1.2.2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S4.E3.m1.2.2.1.1.2" xref="S4.E3.m1.2.2.1.1.2.cmml">=</mo><mrow id="S4.E3.m1.2.2.1.1.3" xref="S4.E3.m1.2.2.1.1.3.cmml"><mi mathvariant="normal" id="S4.E3.m1.2.2.1.1.3.2" xref="S4.E3.m1.2.2.1.1.3.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.E3.m1.2.2.1.1.3.1" xref="S4.E3.m1.2.2.1.1.3.1.cmml">​</mo><mi id="S4.E3.m1.2.2.1.1.3.3" xref="S4.E3.m1.2.2.1.1.3.3.cmml">W</mi><mo lspace="0em" rspace="0em" id="S4.E3.m1.2.2.1.1.3.1a" xref="S4.E3.m1.2.2.1.1.3.1.cmml">​</mo><mi id="S4.E3.m1.2.2.1.1.3.4" xref="S4.E3.m1.2.2.1.1.3.4.cmml">x</mi></mrow></mrow><mo id="S4.E3.m1.3.3.2.3" xref="S4.E3.m1.3.3.3a.cmml">,</mo><mrow id="S4.E3.m1.3.3.2.2" xref="S4.E3.m1.3.3.2.2.cmml"><mrow id="S4.E3.m1.3.3.2.2.2" xref="S4.E3.m1.3.3.2.2.2.cmml"><mo rspace="0.167em" id="S4.E3.m1.3.3.2.2.2.1" xref="S4.E3.m1.3.3.2.2.2.1.cmml">∀</mo><mi id="S4.E3.m1.3.3.2.2.2.2" xref="S4.E3.m1.3.3.2.2.2.2.cmml">x</mi></mrow><mo id="S4.E3.m1.3.3.2.2.1" xref="S4.E3.m1.3.3.2.2.1.cmml">∈</mo><msup id="S4.E3.m1.3.3.2.2.3" xref="S4.E3.m1.3.3.2.2.3.cmml"><mi id="S4.E3.m1.3.3.2.2.3.2" xref="S4.E3.m1.3.3.2.2.3.2.cmml">ℝ</mi><mi id="S4.E3.m1.3.3.2.2.3.3" xref="S4.E3.m1.3.3.2.2.3.3.cmml">k</mi></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.E3.m1.3b"><apply id="S4.E3.m1.3.3.3.cmml" xref="S4.E3.m1.3.3.2"><csymbol cd="ambiguous" id="S4.E3.m1.3.3.3a.cmml" xref="S4.E3.m1.3.3.2.3">formulae-sequence</csymbol><apply id="S4.E3.m1.2.2.1.1.cmml" xref="S4.E3.m1.2.2.1.1"><eq id="S4.E3.m1.2.2.1.1.2.cmml" xref="S4.E3.m1.2.2.1.1.2"></eq><apply id="S4.E3.m1.2.2.1.1.1.cmml" xref="S4.E3.m1.2.2.1.1.1"><times id="S4.E3.m1.2.2.1.1.1.2.cmml" xref="S4.E3.m1.2.2.1.1.1.2"></times><apply id="S4.E3.m1.2.2.1.1.1.3.cmml" xref="S4.E3.m1.2.2.1.1.1.3"><csymbol cd="ambiguous" id="S4.E3.m1.2.2.1.1.1.3.1.cmml" xref="S4.E3.m1.2.2.1.1.1.3">subscript</csymbol><ci id="S4.E3.m1.2.2.1.1.1.3.2.cmml" xref="S4.E3.m1.2.2.1.1.1.3.2">𝑓</ci><ci id="S4.E3.m1.2.2.1.1.1.3.3a.cmml" xref="S4.E3.m1.2.2.1.1.1.3.3"><mtext mathsize="70%" id="S4.E3.m1.2.2.1.1.1.3.3.cmml" xref="S4.E3.m1.2.2.1.1.1.3.3">decomp</mtext></ci></apply><apply id="S4.E3.m1.2.2.1.1.1.1.1.1.cmml" xref="S4.E3.m1.2.2.1.1.1.1.1"><times id="S4.E3.m1.2.2.1.1.1.1.1.1.1.cmml" xref="S4.E3.m1.2.2.1.1.1.1.1.1.1"></times><ci id="S4.E3.m1.2.2.1.1.1.1.1.1.2.cmml" xref="S4.E3.m1.2.2.1.1.1.1.1.1.2">𝑀</ci><apply id="S4.E3.m1.2.2.1.1.1.1.1.1.3.cmml" xref="S4.E3.m1.2.2.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.E3.m1.2.2.1.1.1.1.1.1.3.1.cmml" xref="S4.E3.m1.2.2.1.1.1.1.1.1.3">subscript</csymbol><ci id="S4.E3.m1.2.2.1.1.1.1.1.1.3.2.cmml" xref="S4.E3.m1.2.2.1.1.1.1.1.1.3.2">𝑓</ci><ci id="S4.E3.m1.2.2.1.1.1.1.1.1.3.3a.cmml" xref="S4.E3.m1.2.2.1.1.1.1.1.1.3.3"><mtext mathsize="70%" id="S4.E3.m1.2.2.1.1.1.1.1.1.3.3.cmml" xref="S4.E3.m1.2.2.1.1.1.1.1.1.3.3">comp</mtext></ci></apply><ci id="S4.E3.m1.1.1.cmml" xref="S4.E3.m1.1.1">𝑥</ci></apply></apply><apply id="S4.E3.m1.2.2.1.1.3.cmml" xref="S4.E3.m1.2.2.1.1.3"><times id="S4.E3.m1.2.2.1.1.3.1.cmml" xref="S4.E3.m1.2.2.1.1.3.1"></times><ci id="S4.E3.m1.2.2.1.1.3.2.cmml" xref="S4.E3.m1.2.2.1.1.3.2">Δ</ci><ci id="S4.E3.m1.2.2.1.1.3.3.cmml" xref="S4.E3.m1.2.2.1.1.3.3">𝑊</ci><ci id="S4.E3.m1.2.2.1.1.3.4.cmml" xref="S4.E3.m1.2.2.1.1.3.4">𝑥</ci></apply></apply><apply id="S4.E3.m1.3.3.2.2.cmml" xref="S4.E3.m1.3.3.2.2"><in id="S4.E3.m1.3.3.2.2.1.cmml" xref="S4.E3.m1.3.3.2.2.1"></in><apply id="S4.E3.m1.3.3.2.2.2.cmml" xref="S4.E3.m1.3.3.2.2.2"><csymbol cd="latexml" id="S4.E3.m1.3.3.2.2.2.1.cmml" xref="S4.E3.m1.3.3.2.2.2.1">for-all</csymbol><ci id="S4.E3.m1.3.3.2.2.2.2.cmml" xref="S4.E3.m1.3.3.2.2.2.2">𝑥</ci></apply><apply id="S4.E3.m1.3.3.2.2.3.cmml" xref="S4.E3.m1.3.3.2.2.3"><csymbol cd="ambiguous" id="S4.E3.m1.3.3.2.2.3.1.cmml" xref="S4.E3.m1.3.3.2.2.3">superscript</csymbol><ci id="S4.E3.m1.3.3.2.2.3.2.cmml" xref="S4.E3.m1.3.3.2.2.3.2">ℝ</ci><ci id="S4.E3.m1.3.3.2.2.3.3.cmml" xref="S4.E3.m1.3.3.2.2.3.3">𝑘</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E3.m1.3c">f_{\text{decomp}}\left(Mf_{\text{comp}}\left(x\right)\right)=\Delta Wx,\forall x\in\mathbb{R}^{k}\\
</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
<p id="S4.p2.18" class="ltx_p">where <math id="S4.p2.14.m1.2" class="ltx_Math" alttext="\Delta W=f_{\overline{\text{decomp}}}\left(f_{\overline{\text{comp}}}\left(M\right)\right)" display="inline"><semantics id="S4.p2.14.m1.2a"><mrow id="S4.p2.14.m1.2.2" xref="S4.p2.14.m1.2.2.cmml"><mrow id="S4.p2.14.m1.2.2.3" xref="S4.p2.14.m1.2.2.3.cmml"><mi mathvariant="normal" id="S4.p2.14.m1.2.2.3.2" xref="S4.p2.14.m1.2.2.3.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.p2.14.m1.2.2.3.1" xref="S4.p2.14.m1.2.2.3.1.cmml">​</mo><mi id="S4.p2.14.m1.2.2.3.3" xref="S4.p2.14.m1.2.2.3.3.cmml">W</mi></mrow><mo id="S4.p2.14.m1.2.2.2" xref="S4.p2.14.m1.2.2.2.cmml">=</mo><mrow id="S4.p2.14.m1.2.2.1" xref="S4.p2.14.m1.2.2.1.cmml"><msub id="S4.p2.14.m1.2.2.1.3" xref="S4.p2.14.m1.2.2.1.3.cmml"><mi id="S4.p2.14.m1.2.2.1.3.2" xref="S4.p2.14.m1.2.2.1.3.2.cmml">f</mi><mover accent="true" id="S4.p2.14.m1.2.2.1.3.3" xref="S4.p2.14.m1.2.2.1.3.3.cmml"><mtext id="S4.p2.14.m1.2.2.1.3.3.2" xref="S4.p2.14.m1.2.2.1.3.3.2a.cmml">decomp</mtext><mo id="S4.p2.14.m1.2.2.1.3.3.1" xref="S4.p2.14.m1.2.2.1.3.3.1.cmml">¯</mo></mover></msub><mo lspace="0em" rspace="0em" id="S4.p2.14.m1.2.2.1.2" xref="S4.p2.14.m1.2.2.1.2.cmml">​</mo><mrow id="S4.p2.14.m1.2.2.1.1.1" xref="S4.p2.14.m1.2.2.1.1.1.1.cmml"><mo id="S4.p2.14.m1.2.2.1.1.1.2" xref="S4.p2.14.m1.2.2.1.1.1.1.cmml">(</mo><mrow id="S4.p2.14.m1.2.2.1.1.1.1" xref="S4.p2.14.m1.2.2.1.1.1.1.cmml"><msub id="S4.p2.14.m1.2.2.1.1.1.1.2" xref="S4.p2.14.m1.2.2.1.1.1.1.2.cmml"><mi id="S4.p2.14.m1.2.2.1.1.1.1.2.2" xref="S4.p2.14.m1.2.2.1.1.1.1.2.2.cmml">f</mi><mover accent="true" id="S4.p2.14.m1.2.2.1.1.1.1.2.3" xref="S4.p2.14.m1.2.2.1.1.1.1.2.3.cmml"><mtext id="S4.p2.14.m1.2.2.1.1.1.1.2.3.2" xref="S4.p2.14.m1.2.2.1.1.1.1.2.3.2a.cmml">comp</mtext><mo id="S4.p2.14.m1.2.2.1.1.1.1.2.3.1" xref="S4.p2.14.m1.2.2.1.1.1.1.2.3.1.cmml">¯</mo></mover></msub><mo lspace="0em" rspace="0em" id="S4.p2.14.m1.2.2.1.1.1.1.1" xref="S4.p2.14.m1.2.2.1.1.1.1.1.cmml">​</mo><mrow id="S4.p2.14.m1.2.2.1.1.1.1.3.2" xref="S4.p2.14.m1.2.2.1.1.1.1.cmml"><mo id="S4.p2.14.m1.2.2.1.1.1.1.3.2.1" xref="S4.p2.14.m1.2.2.1.1.1.1.cmml">(</mo><mi id="S4.p2.14.m1.1.1" xref="S4.p2.14.m1.1.1.cmml">M</mi><mo id="S4.p2.14.m1.2.2.1.1.1.1.3.2.2" xref="S4.p2.14.m1.2.2.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S4.p2.14.m1.2.2.1.1.1.3" xref="S4.p2.14.m1.2.2.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.p2.14.m1.2b"><apply id="S4.p2.14.m1.2.2.cmml" xref="S4.p2.14.m1.2.2"><eq id="S4.p2.14.m1.2.2.2.cmml" xref="S4.p2.14.m1.2.2.2"></eq><apply id="S4.p2.14.m1.2.2.3.cmml" xref="S4.p2.14.m1.2.2.3"><times id="S4.p2.14.m1.2.2.3.1.cmml" xref="S4.p2.14.m1.2.2.3.1"></times><ci id="S4.p2.14.m1.2.2.3.2.cmml" xref="S4.p2.14.m1.2.2.3.2">Δ</ci><ci id="S4.p2.14.m1.2.2.3.3.cmml" xref="S4.p2.14.m1.2.2.3.3">𝑊</ci></apply><apply id="S4.p2.14.m1.2.2.1.cmml" xref="S4.p2.14.m1.2.2.1"><times id="S4.p2.14.m1.2.2.1.2.cmml" xref="S4.p2.14.m1.2.2.1.2"></times><apply id="S4.p2.14.m1.2.2.1.3.cmml" xref="S4.p2.14.m1.2.2.1.3"><csymbol cd="ambiguous" id="S4.p2.14.m1.2.2.1.3.1.cmml" xref="S4.p2.14.m1.2.2.1.3">subscript</csymbol><ci id="S4.p2.14.m1.2.2.1.3.2.cmml" xref="S4.p2.14.m1.2.2.1.3.2">𝑓</ci><apply id="S4.p2.14.m1.2.2.1.3.3.cmml" xref="S4.p2.14.m1.2.2.1.3.3"><ci id="S4.p2.14.m1.2.2.1.3.3.1.cmml" xref="S4.p2.14.m1.2.2.1.3.3.1">¯</ci><ci id="S4.p2.14.m1.2.2.1.3.3.2a.cmml" xref="S4.p2.14.m1.2.2.1.3.3.2"><mtext mathsize="70%" id="S4.p2.14.m1.2.2.1.3.3.2.cmml" xref="S4.p2.14.m1.2.2.1.3.3.2">decomp</mtext></ci></apply></apply><apply id="S4.p2.14.m1.2.2.1.1.1.1.cmml" xref="S4.p2.14.m1.2.2.1.1.1"><times id="S4.p2.14.m1.2.2.1.1.1.1.1.cmml" xref="S4.p2.14.m1.2.2.1.1.1.1.1"></times><apply id="S4.p2.14.m1.2.2.1.1.1.1.2.cmml" xref="S4.p2.14.m1.2.2.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.p2.14.m1.2.2.1.1.1.1.2.1.cmml" xref="S4.p2.14.m1.2.2.1.1.1.1.2">subscript</csymbol><ci id="S4.p2.14.m1.2.2.1.1.1.1.2.2.cmml" xref="S4.p2.14.m1.2.2.1.1.1.1.2.2">𝑓</ci><apply id="S4.p2.14.m1.2.2.1.1.1.1.2.3.cmml" xref="S4.p2.14.m1.2.2.1.1.1.1.2.3"><ci id="S4.p2.14.m1.2.2.1.1.1.1.2.3.1.cmml" xref="S4.p2.14.m1.2.2.1.1.1.1.2.3.1">¯</ci><ci id="S4.p2.14.m1.2.2.1.1.1.1.2.3.2a.cmml" xref="S4.p2.14.m1.2.2.1.1.1.1.2.3.2"><mtext mathsize="70%" id="S4.p2.14.m1.2.2.1.1.1.1.2.3.2.cmml" xref="S4.p2.14.m1.2.2.1.1.1.1.2.3.2">comp</mtext></ci></apply></apply><ci id="S4.p2.14.m1.1.1.cmml" xref="S4.p2.14.m1.1.1">𝑀</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.14.m1.2c">\Delta W=f_{\overline{\text{decomp}}}\left(f_{\overline{\text{comp}}}\left(M\right)\right)</annotation></semantics></math>.
If Eq. <a href="#S4.E3" title="In 4 Method ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> holds, <math id="S4.p2.15.m2.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S4.p2.15.m2.1a"><mi id="S4.p2.15.m2.1.1" xref="S4.p2.15.m2.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S4.p2.15.m2.1b"><ci id="S4.p2.15.m2.1.1.cmml" xref="S4.p2.15.m2.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.15.m2.1c">M</annotation></semantics></math> can be losslessly expanded to <math id="S4.p2.16.m3.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S4.p2.16.m3.1a"><mrow id="S4.p2.16.m3.1.1" xref="S4.p2.16.m3.1.1.cmml"><mi mathvariant="normal" id="S4.p2.16.m3.1.1.2" xref="S4.p2.16.m3.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.p2.16.m3.1.1.1" xref="S4.p2.16.m3.1.1.1.cmml">​</mo><mi id="S4.p2.16.m3.1.1.3" xref="S4.p2.16.m3.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p2.16.m3.1b"><apply id="S4.p2.16.m3.1.1.cmml" xref="S4.p2.16.m3.1.1"><times id="S4.p2.16.m3.1.1.1.cmml" xref="S4.p2.16.m3.1.1.1"></times><ci id="S4.p2.16.m3.1.1.2.cmml" xref="S4.p2.16.m3.1.1.2">Δ</ci><ci id="S4.p2.16.m3.1.1.3.cmml" xref="S4.p2.16.m3.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.16.m3.1c">\Delta W</annotation></semantics></math> based on <math id="S4.p2.17.m4.1" class="ltx_Math" alttext="f_{\text{comp}}" display="inline"><semantics id="S4.p2.17.m4.1a"><msub id="S4.p2.17.m4.1.1" xref="S4.p2.17.m4.1.1.cmml"><mi id="S4.p2.17.m4.1.1.2" xref="S4.p2.17.m4.1.1.2.cmml">f</mi><mtext id="S4.p2.17.m4.1.1.3" xref="S4.p2.17.m4.1.1.3a.cmml">comp</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.p2.17.m4.1b"><apply id="S4.p2.17.m4.1.1.cmml" xref="S4.p2.17.m4.1.1"><csymbol cd="ambiguous" id="S4.p2.17.m4.1.1.1.cmml" xref="S4.p2.17.m4.1.1">subscript</csymbol><ci id="S4.p2.17.m4.1.1.2.cmml" xref="S4.p2.17.m4.1.1.2">𝑓</ci><ci id="S4.p2.17.m4.1.1.3a.cmml" xref="S4.p2.17.m4.1.1.3"><mtext mathsize="70%" id="S4.p2.17.m4.1.1.3.cmml" xref="S4.p2.17.m4.1.1.3">comp</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.17.m4.1c">f_{\text{comp}}</annotation></semantics></math> and <math id="S4.p2.18.m5.1" class="ltx_Math" alttext="f_{\text{decomp}}" display="inline"><semantics id="S4.p2.18.m5.1a"><msub id="S4.p2.18.m5.1.1" xref="S4.p2.18.m5.1.1.cmml"><mi id="S4.p2.18.m5.1.1.2" xref="S4.p2.18.m5.1.1.2.cmml">f</mi><mtext id="S4.p2.18.m5.1.1.3" xref="S4.p2.18.m5.1.1.3a.cmml">decomp</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.p2.18.m5.1b"><apply id="S4.p2.18.m5.1.1.cmml" xref="S4.p2.18.m5.1.1"><csymbol cd="ambiguous" id="S4.p2.18.m5.1.1.1.cmml" xref="S4.p2.18.m5.1.1">subscript</csymbol><ci id="S4.p2.18.m5.1.1.2.cmml" xref="S4.p2.18.m5.1.1.2">𝑓</ci><ci id="S4.p2.18.m5.1.1.3a.cmml" xref="S4.p2.18.m5.1.1.3"><mtext mathsize="70%" id="S4.p2.18.m5.1.1.3.cmml" xref="S4.p2.18.m5.1.1.3">decomp</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.18.m5.1c">f_{\text{decomp}}</annotation></semantics></math>. This allows our method to merge back into the LLM like LoRA.</p>
</div>
<div id="S4.p3" class="ltx_para">
<p id="S4.p3.2" class="ltx_p">For the design of <math id="S4.p3.1.m1.1" class="ltx_Math" alttext="f_{\text{comp}}" display="inline"><semantics id="S4.p3.1.m1.1a"><msub id="S4.p3.1.m1.1.1" xref="S4.p3.1.m1.1.1.cmml"><mi id="S4.p3.1.m1.1.1.2" xref="S4.p3.1.m1.1.1.2.cmml">f</mi><mtext id="S4.p3.1.m1.1.1.3" xref="S4.p3.1.m1.1.1.3a.cmml">comp</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.p3.1.m1.1b"><apply id="S4.p3.1.m1.1.1.cmml" xref="S4.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S4.p3.1.m1.1.1.1.cmml" xref="S4.p3.1.m1.1.1">subscript</csymbol><ci id="S4.p3.1.m1.1.1.2.cmml" xref="S4.p3.1.m1.1.1.2">𝑓</ci><ci id="S4.p3.1.m1.1.1.3a.cmml" xref="S4.p3.1.m1.1.1.3"><mtext mathsize="70%" id="S4.p3.1.m1.1.1.3.cmml" xref="S4.p3.1.m1.1.1.3">comp</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.1.m1.1c">f_{\text{comp}}</annotation></semantics></math> and <math id="S4.p3.2.m2.1" class="ltx_Math" alttext="f_{\text{comp}}" display="inline"><semantics id="S4.p3.2.m2.1a"><msub id="S4.p3.2.m2.1.1" xref="S4.p3.2.m2.1.1.cmml"><mi id="S4.p3.2.m2.1.1.2" xref="S4.p3.2.m2.1.1.2.cmml">f</mi><mtext id="S4.p3.2.m2.1.1.3" xref="S4.p3.2.m2.1.1.3a.cmml">comp</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.p3.2.m2.1b"><apply id="S4.p3.2.m2.1.1.cmml" xref="S4.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S4.p3.2.m2.1.1.1.cmml" xref="S4.p3.2.m2.1.1">subscript</csymbol><ci id="S4.p3.2.m2.1.1.2.cmml" xref="S4.p3.2.m2.1.1.2">𝑓</ci><ci id="S4.p3.2.m2.1.1.3a.cmml" xref="S4.p3.2.m2.1.1.3"><mtext mathsize="70%" id="S4.p3.2.m2.1.1.3.cmml" xref="S4.p3.2.m2.1.1.3">comp</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.2.m2.1c">f_{\text{comp}}</annotation></semantics></math>, we explore several methods to implement these functions.
One straightforward method is truncating the dimension and subsequently add it in corresponding dimension.
Formally, this can be represented as:</p>
<table id="S4.E4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E4.m1.15" class="ltx_Math" alttext="\begin{split}f_{\text{comp}}\left(x\right)&amp;=x_{1:\hat{r}}\\
f_{\text{decomp}}\left(x\right)&amp;=\begin{bmatrix}x\\
\mathbf{0}\end{bmatrix}\end{split}" display="block"><semantics id="S4.E4.m1.15a"><mtable columnspacing="0pt" displaystyle="true" rowspacing="0pt" id="S4.E4.m1.15.15" xref="S4.E4.m1.15.16.1.cmml"><mtr id="S4.E4.m1.15.15a" xref="S4.E4.m1.15.16.1.cmml"><mtd class="ltx_align_right" columnalign="right" id="S4.E4.m1.15.15b" xref="S4.E4.m1.15.16.1.cmml"><mrow id="S4.E4.m1.6.6.6.5.5" xref="S4.E4.m1.15.16.1.cmml"><msub id="S4.E4.m1.6.6.6.5.5.7" xref="S4.E4.m1.15.16.1.cmml"><mi id="S4.E4.m1.2.2.2.1.1.1" xref="S4.E4.m1.2.2.2.1.1.1.cmml">f</mi><mtext id="S4.E4.m1.3.3.3.2.2.2.1" xref="S4.E4.m1.3.3.3.2.2.2.1a.cmml">comp</mtext></msub><mo lspace="0em" rspace="0em" id="S4.E4.m1.6.6.6.5.5.6" xref="S4.E4.m1.15.16.1a.cmml">​</mo><mrow id="S4.E4.m1.6.6.6.5.5.8" xref="S4.E4.m1.15.16.1.cmml"><mo id="S4.E4.m1.4.4.4.3.3.3" xref="S4.E4.m1.15.16.1a.cmml">(</mo><mi id="S4.E4.m1.5.5.5.4.4.4" xref="S4.E4.m1.5.5.5.4.4.4.cmml">x</mi><mo id="S4.E4.m1.6.6.6.5.5.5" xref="S4.E4.m1.15.16.1a.cmml">)</mo></mrow></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S4.E4.m1.15.15c" xref="S4.E4.m1.15.16.1.cmml"><mrow id="S4.E4.m1.9.9.9.8.3" xref="S4.E4.m1.15.16.1.cmml"><mi id="S4.E4.m1.9.9.9.8.3.4" xref="S4.E4.m1.15.16.1a.cmml"></mi><mo id="S4.E4.m1.7.7.7.6.1.1" xref="S4.E4.m1.7.7.7.6.1.1.cmml">=</mo><msub id="S4.E4.m1.9.9.9.8.3.5" xref="S4.E4.m1.15.16.1.cmml"><mi id="S4.E4.m1.8.8.8.7.2.2" xref="S4.E4.m1.8.8.8.7.2.2.cmml">x</mi><mrow id="S4.E4.m1.9.9.9.8.3.3.1" xref="S4.E4.m1.9.9.9.8.3.3.1.cmml"><mn id="S4.E4.m1.9.9.9.8.3.3.1.2" xref="S4.E4.m1.9.9.9.8.3.3.1.2.cmml">1</mn><mo lspace="0.278em" rspace="0.278em" id="S4.E4.m1.9.9.9.8.3.3.1.1" xref="S4.E4.m1.9.9.9.8.3.3.1.1.cmml">:</mo><mover accent="true" id="S4.E4.m1.9.9.9.8.3.3.1.3" xref="S4.E4.m1.9.9.9.8.3.3.1.3.cmml"><mi id="S4.E4.m1.9.9.9.8.3.3.1.3.2" xref="S4.E4.m1.9.9.9.8.3.3.1.3.2.cmml">r</mi><mo id="S4.E4.m1.9.9.9.8.3.3.1.3.1" xref="S4.E4.m1.9.9.9.8.3.3.1.3.1.cmml">^</mo></mover></mrow></msub></mrow></mtd></mtr><mtr id="S4.E4.m1.15.15d" xref="S4.E4.m1.15.16.1.cmml"><mtd class="ltx_align_right" columnalign="right" id="S4.E4.m1.15.15e" xref="S4.E4.m1.15.16.1.cmml"><mrow id="S4.E4.m1.14.14.14.6.5" xref="S4.E4.m1.15.16.1.cmml"><msub id="S4.E4.m1.14.14.14.6.5.7" xref="S4.E4.m1.15.16.1.cmml"><mi id="S4.E4.m1.10.10.10.2.1.1" xref="S4.E4.m1.10.10.10.2.1.1.cmml">f</mi><mtext id="S4.E4.m1.11.11.11.3.2.2.1" xref="S4.E4.m1.11.11.11.3.2.2.1a.cmml">decomp</mtext></msub><mo lspace="0em" rspace="0em" id="S4.E4.m1.14.14.14.6.5.6" xref="S4.E4.m1.15.16.1a.cmml">​</mo><mrow id="S4.E4.m1.14.14.14.6.5.8" xref="S4.E4.m1.15.16.1.cmml"><mo id="S4.E4.m1.12.12.12.4.3.3" xref="S4.E4.m1.15.16.1a.cmml">(</mo><mi id="S4.E4.m1.13.13.13.5.4.4" xref="S4.E4.m1.13.13.13.5.4.4.cmml">x</mi><mo id="S4.E4.m1.14.14.14.6.5.5" xref="S4.E4.m1.15.16.1a.cmml">)</mo></mrow></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S4.E4.m1.15.15f" xref="S4.E4.m1.15.16.1.cmml"><mrow id="S4.E4.m1.15.15.15.7.2" xref="S4.E4.m1.15.16.1.cmml"><mi id="S4.E4.m1.15.15.15.7.2.3" xref="S4.E4.m1.15.16.1a.cmml"></mi><mo id="S4.E4.m1.15.15.15.7.2.2" xref="S4.E4.m1.15.15.15.7.2.2.cmml">=</mo><mrow id="S4.E4.m1.1.1.1.1.1.1.3" xref="S4.E4.m1.1.1.1.1.1.1.2.cmml"><mo id="S4.E4.m1.1.1.1.1.1.1.3.1" xref="S4.E4.m1.1.1.1.1.1.1.2.1.cmml">[</mo><mtable displaystyle="true" rowspacing="0pt" id="S4.E4.m1.1.1.1.1.1.1.1.1" xref="S4.E4.m1.1.1.1.1.1.1.1.1.cmml"><mtr id="S4.E4.m1.1.1.1.1.1.1.1.1a" xref="S4.E4.m1.1.1.1.1.1.1.1.1.cmml"><mtd id="S4.E4.m1.1.1.1.1.1.1.1.1b" xref="S4.E4.m1.1.1.1.1.1.1.1.1.cmml"><mi id="S4.E4.m1.1.1.1.1.1.1.1.1.1.1.1" xref="S4.E4.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">x</mi></mtd></mtr><mtr id="S4.E4.m1.1.1.1.1.1.1.1.1c" xref="S4.E4.m1.1.1.1.1.1.1.1.1.cmml"><mtd id="S4.E4.m1.1.1.1.1.1.1.1.1d" xref="S4.E4.m1.1.1.1.1.1.1.1.1.cmml"><mn id="S4.E4.m1.1.1.1.1.1.1.1.1.2.1.1" xref="S4.E4.m1.1.1.1.1.1.1.1.1.2.1.1.cmml">𝟎</mn></mtd></mtr></mtable><mo id="S4.E4.m1.1.1.1.1.1.1.3.2" xref="S4.E4.m1.1.1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow></mtd></mtr></mtable><annotation-xml encoding="MathML-Content" id="S4.E4.m1.15b"><apply id="S4.E4.m1.15.16.1.cmml" xref="S4.E4.m1.15.15"><and id="S4.E4.m1.15.16.1a.cmml" xref="S4.E4.m1.6.6.6.5.5.6"></and><apply id="S4.E4.m1.15.16.1b.cmml" xref="S4.E4.m1.15.15"><eq id="S4.E4.m1.7.7.7.6.1.1.cmml" xref="S4.E4.m1.7.7.7.6.1.1"></eq><apply id="S4.E4.m1.15.16.1.2.cmml" xref="S4.E4.m1.15.15"><times id="S4.E4.m1.15.16.1.2.1.cmml" xref="S4.E4.m1.6.6.6.5.5.6"></times><apply id="S4.E4.m1.15.16.1.2.2.cmml" xref="S4.E4.m1.15.15"><csymbol cd="ambiguous" id="S4.E4.m1.15.16.1.2.2.1.cmml" xref="S4.E4.m1.6.6.6.5.5.6">subscript</csymbol><ci id="S4.E4.m1.2.2.2.1.1.1.cmml" xref="S4.E4.m1.2.2.2.1.1.1">𝑓</ci><ci id="S4.E4.m1.3.3.3.2.2.2.1a.cmml" xref="S4.E4.m1.3.3.3.2.2.2.1"><mtext mathsize="70%" id="S4.E4.m1.3.3.3.2.2.2.1.cmml" xref="S4.E4.m1.3.3.3.2.2.2.1">comp</mtext></ci></apply><ci id="S4.E4.m1.5.5.5.4.4.4.cmml" xref="S4.E4.m1.5.5.5.4.4.4">𝑥</ci></apply><apply id="S4.E4.m1.15.16.1.4.cmml" xref="S4.E4.m1.15.15"><times id="S4.E4.m1.15.16.1.4.1.cmml" xref="S4.E4.m1.6.6.6.5.5.6"></times><apply id="S4.E4.m1.15.16.1.4.2.cmml" xref="S4.E4.m1.15.15"><csymbol cd="ambiguous" id="S4.E4.m1.15.16.1.4.2.1.cmml" xref="S4.E4.m1.6.6.6.5.5.6">subscript</csymbol><ci id="S4.E4.m1.8.8.8.7.2.2.cmml" xref="S4.E4.m1.8.8.8.7.2.2">𝑥</ci><apply id="S4.E4.m1.9.9.9.8.3.3.1.cmml" xref="S4.E4.m1.9.9.9.8.3.3.1"><ci id="S4.E4.m1.9.9.9.8.3.3.1.1.cmml" xref="S4.E4.m1.9.9.9.8.3.3.1.1">:</ci><cn type="integer" id="S4.E4.m1.9.9.9.8.3.3.1.2.cmml" xref="S4.E4.m1.9.9.9.8.3.3.1.2">1</cn><apply id="S4.E4.m1.9.9.9.8.3.3.1.3.cmml" xref="S4.E4.m1.9.9.9.8.3.3.1.3"><ci id="S4.E4.m1.9.9.9.8.3.3.1.3.1.cmml" xref="S4.E4.m1.9.9.9.8.3.3.1.3.1">^</ci><ci id="S4.E4.m1.9.9.9.8.3.3.1.3.2.cmml" xref="S4.E4.m1.9.9.9.8.3.3.1.3.2">𝑟</ci></apply></apply></apply><apply id="S4.E4.m1.15.16.1.4.3.cmml" xref="S4.E4.m1.15.15"><csymbol cd="ambiguous" id="S4.E4.m1.15.16.1.4.3.1.cmml" xref="S4.E4.m1.6.6.6.5.5.6">subscript</csymbol><ci id="S4.E4.m1.10.10.10.2.1.1.cmml" xref="S4.E4.m1.10.10.10.2.1.1">𝑓</ci><ci id="S4.E4.m1.11.11.11.3.2.2.1a.cmml" xref="S4.E4.m1.11.11.11.3.2.2.1"><mtext mathsize="70%" id="S4.E4.m1.11.11.11.3.2.2.1.cmml" xref="S4.E4.m1.11.11.11.3.2.2.1">decomp</mtext></ci></apply><ci id="S4.E4.m1.13.13.13.5.4.4.cmml" xref="S4.E4.m1.13.13.13.5.4.4">𝑥</ci></apply></apply><apply id="S4.E4.m1.15.16.1c.cmml" xref="S4.E4.m1.15.15"><eq id="S4.E4.m1.15.15.15.7.2.2.cmml" xref="S4.E4.m1.15.15.15.7.2.2"></eq><share href="#S4.E4.m1.15.16.1.4.cmml" id="S4.E4.m1.15.16.1d.cmml" xref="S4.E4.m1.6.6.6.5.5.6"></share><apply id="S4.E4.m1.1.1.1.1.1.1.2.cmml" xref="S4.E4.m1.1.1.1.1.1.1.3"><csymbol cd="latexml" id="S4.E4.m1.1.1.1.1.1.1.2.1.cmml" xref="S4.E4.m1.1.1.1.1.1.1.3.1">matrix</csymbol><matrix id="S4.E4.m1.1.1.1.1.1.1.1.1.cmml" xref="S4.E4.m1.1.1.1.1.1.1.1.1"><matrixrow id="S4.E4.m1.1.1.1.1.1.1.1.1a.cmml" xref="S4.E4.m1.1.1.1.1.1.1.1.1"><ci id="S4.E4.m1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E4.m1.1.1.1.1.1.1.1.1.1.1.1">𝑥</ci></matrixrow><matrixrow id="S4.E4.m1.1.1.1.1.1.1.1.1b.cmml" xref="S4.E4.m1.1.1.1.1.1.1.1.1"><cn type="integer" id="S4.E4.m1.1.1.1.1.1.1.1.1.2.1.1.cmml" xref="S4.E4.m1.1.1.1.1.1.1.1.1.2.1.1">0</cn></matrixrow></matrix></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E4.m1.15c">\begin{split}f_{\text{comp}}\left(x\right)&amp;=x_{1:\hat{r}}\\
f_{\text{decomp}}\left(x\right)&amp;=\begin{bmatrix}x\\
\mathbf{0}\end{bmatrix}\end{split}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
<p id="S4.p3.3" class="ltx_p">and the corresponding <math id="S4.p3.3.m1.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S4.p3.3.m1.1a"><mrow id="S4.p3.3.m1.1.1" xref="S4.p3.3.m1.1.1.cmml"><mi mathvariant="normal" id="S4.p3.3.m1.1.1.2" xref="S4.p3.3.m1.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.p3.3.m1.1.1.1" xref="S4.p3.3.m1.1.1.1.cmml">​</mo><mi id="S4.p3.3.m1.1.1.3" xref="S4.p3.3.m1.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p3.3.m1.1b"><apply id="S4.p3.3.m1.1.1.cmml" xref="S4.p3.3.m1.1.1"><times id="S4.p3.3.m1.1.1.1.cmml" xref="S4.p3.3.m1.1.1.1"></times><ci id="S4.p3.3.m1.1.1.2.cmml" xref="S4.p3.3.m1.1.1.2">Δ</ci><ci id="S4.p3.3.m1.1.1.3.cmml" xref="S4.p3.3.m1.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.3.m1.1c">\Delta W</annotation></semantics></math> is:</p>
<table id="S4.E5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E5.m1.1" class="ltx_Math" alttext="\Delta W=\begin{bmatrix}M&amp;\mathbf{0}\\
\mathbf{0}&amp;\mathbf{0}\end{bmatrix}" display="block"><semantics id="S4.E5.m1.1a"><mrow id="S4.E5.m1.1.2" xref="S4.E5.m1.1.2.cmml"><mrow id="S4.E5.m1.1.2.2" xref="S4.E5.m1.1.2.2.cmml"><mi mathvariant="normal" id="S4.E5.m1.1.2.2.2" xref="S4.E5.m1.1.2.2.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.E5.m1.1.2.2.1" xref="S4.E5.m1.1.2.2.1.cmml">​</mo><mi id="S4.E5.m1.1.2.2.3" xref="S4.E5.m1.1.2.2.3.cmml">W</mi></mrow><mo id="S4.E5.m1.1.2.1" xref="S4.E5.m1.1.2.1.cmml">=</mo><mrow id="S4.E5.m1.1.1.3" xref="S4.E5.m1.1.1.2.cmml"><mo id="S4.E5.m1.1.1.3.1" xref="S4.E5.m1.1.1.2.1.cmml">[</mo><mtable columnspacing="5pt" displaystyle="true" rowspacing="0pt" id="S4.E5.m1.1.1.1.1" xref="S4.E5.m1.1.1.1.1.cmml"><mtr id="S4.E5.m1.1.1.1.1a" xref="S4.E5.m1.1.1.1.1.cmml"><mtd id="S4.E5.m1.1.1.1.1b" xref="S4.E5.m1.1.1.1.1.cmml"><mi id="S4.E5.m1.1.1.1.1.1.1.1" xref="S4.E5.m1.1.1.1.1.1.1.1.cmml">M</mi></mtd><mtd id="S4.E5.m1.1.1.1.1c" xref="S4.E5.m1.1.1.1.1.cmml"><mn id="S4.E5.m1.1.1.1.1.1.2.1" xref="S4.E5.m1.1.1.1.1.1.2.1.cmml">𝟎</mn></mtd></mtr><mtr id="S4.E5.m1.1.1.1.1d" xref="S4.E5.m1.1.1.1.1.cmml"><mtd id="S4.E5.m1.1.1.1.1e" xref="S4.E5.m1.1.1.1.1.cmml"><mn id="S4.E5.m1.1.1.1.1.2.1.1" xref="S4.E5.m1.1.1.1.1.2.1.1.cmml">𝟎</mn></mtd><mtd id="S4.E5.m1.1.1.1.1f" xref="S4.E5.m1.1.1.1.1.cmml"><mn id="S4.E5.m1.1.1.1.1.2.2.1" xref="S4.E5.m1.1.1.1.1.2.2.1.cmml">𝟎</mn></mtd></mtr></mtable><mo id="S4.E5.m1.1.1.3.2" xref="S4.E5.m1.1.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.E5.m1.1b"><apply id="S4.E5.m1.1.2.cmml" xref="S4.E5.m1.1.2"><eq id="S4.E5.m1.1.2.1.cmml" xref="S4.E5.m1.1.2.1"></eq><apply id="S4.E5.m1.1.2.2.cmml" xref="S4.E5.m1.1.2.2"><times id="S4.E5.m1.1.2.2.1.cmml" xref="S4.E5.m1.1.2.2.1"></times><ci id="S4.E5.m1.1.2.2.2.cmml" xref="S4.E5.m1.1.2.2.2">Δ</ci><ci id="S4.E5.m1.1.2.2.3.cmml" xref="S4.E5.m1.1.2.2.3">𝑊</ci></apply><apply id="S4.E5.m1.1.1.2.cmml" xref="S4.E5.m1.1.1.3"><csymbol cd="latexml" id="S4.E5.m1.1.1.2.1.cmml" xref="S4.E5.m1.1.1.3.1">matrix</csymbol><matrix id="S4.E5.m1.1.1.1.1.cmml" xref="S4.E5.m1.1.1.1.1"><matrixrow id="S4.E5.m1.1.1.1.1a.cmml" xref="S4.E5.m1.1.1.1.1"><ci id="S4.E5.m1.1.1.1.1.1.1.1.cmml" xref="S4.E5.m1.1.1.1.1.1.1.1">𝑀</ci><cn type="integer" id="S4.E5.m1.1.1.1.1.1.2.1.cmml" xref="S4.E5.m1.1.1.1.1.1.2.1">0</cn></matrixrow><matrixrow id="S4.E5.m1.1.1.1.1b.cmml" xref="S4.E5.m1.1.1.1.1"><cn type="integer" id="S4.E5.m1.1.1.1.1.2.1.1.cmml" xref="S4.E5.m1.1.1.1.1.2.1.1">0</cn><cn type="integer" id="S4.E5.m1.1.1.1.1.2.2.1.cmml" xref="S4.E5.m1.1.1.1.1.2.2.1">0</cn></matrixrow></matrix></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E5.m1.1c">\Delta W=\begin{bmatrix}M&amp;\mathbf{0}\\
\mathbf{0}&amp;\mathbf{0}\end{bmatrix}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(5)</span></td>
</tr></tbody>
</table>
<p id="S4.p3.4" class="ltx_p">However, this method leads to a significant loss of information during compression and only modifies a segment of the output by appending a zero vector during decompression.
To improve it, we can share the rows and columns of <math id="S4.p3.4.m1.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S4.p3.4.m1.1a"><mi id="S4.p3.4.m1.1.1" xref="S4.p3.4.m1.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S4.p3.4.m1.1b"><ci id="S4.p3.4.m1.1.1.cmml" xref="S4.p3.4.m1.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.4.m1.1c">M</annotation></semantics></math> to achieve a more efficient compression and decompression. Formally, this can be represented as:</p>
<table id="S4.E6" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E6.m1.18" class="ltx_Math" alttext="\begin{split}f_{\text{comp}}\left(x\right)&amp;=\begin{bmatrix}\sum_{j\in g_{i}}x_{j}\end{bmatrix}_{i=1}^{r}\\
f_{\text{decomp}}\left(x\right)&amp;=\begin{bmatrix}x_{\widetilde{g}^{\prime}_{i}}\end{bmatrix}_{i=1}^{d}\end{split}" display="block"><semantics id="S4.E6.m1.18a"><mtable columnspacing="0pt" displaystyle="true" rowspacing="0pt" id="S4.E6.m1.18.18" xref="S4.E6.m1.18.19.1.cmml"><mtr id="S4.E6.m1.18.18a" xref="S4.E6.m1.18.19.1.cmml"><mtd class="ltx_align_right" columnalign="right" id="S4.E6.m1.18.18b" xref="S4.E6.m1.18.19.1.cmml"><mrow id="S4.E6.m1.7.7.7.6.5" xref="S4.E6.m1.18.19.1.cmml"><msub id="S4.E6.m1.7.7.7.6.5.7" xref="S4.E6.m1.18.19.1.cmml"><mi id="S4.E6.m1.3.3.3.2.1.1" xref="S4.E6.m1.3.3.3.2.1.1.cmml">f</mi><mtext id="S4.E6.m1.4.4.4.3.2.2.1" xref="S4.E6.m1.4.4.4.3.2.2.1a.cmml">comp</mtext></msub><mo lspace="0em" rspace="0em" id="S4.E6.m1.7.7.7.6.5.6" xref="S4.E6.m1.18.19.1a.cmml">​</mo><mrow id="S4.E6.m1.7.7.7.6.5.8" xref="S4.E6.m1.18.19.1.cmml"><mo id="S4.E6.m1.5.5.5.4.3.3" xref="S4.E6.m1.18.19.1a.cmml">(</mo><mi id="S4.E6.m1.6.6.6.5.4.4" xref="S4.E6.m1.6.6.6.5.4.4.cmml">x</mi><mo id="S4.E6.m1.7.7.7.6.5.5" xref="S4.E6.m1.18.19.1a.cmml">)</mo></mrow></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S4.E6.m1.18.18c" xref="S4.E6.m1.18.19.1.cmml"><mrow id="S4.E6.m1.10.10.10.9.4" xref="S4.E6.m1.18.19.1.cmml"><mi id="S4.E6.m1.10.10.10.9.4.5" xref="S4.E6.m1.18.19.1a.cmml"></mi><mo id="S4.E6.m1.8.8.8.7.2.2" xref="S4.E6.m1.8.8.8.7.2.2.cmml">=</mo><msubsup id="S4.E6.m1.10.10.10.9.4.6" xref="S4.E6.m1.18.19.1.cmml"><mrow id="S4.E6.m1.1.1.1.1.1.1.3" xref="S4.E6.m1.1.1.1.1.1.1.2.cmml"><mo id="S4.E6.m1.1.1.1.1.1.1.3.1" xref="S4.E6.m1.1.1.1.1.1.1.2.1.cmml">[</mo><mtable displaystyle="true" id="S4.E6.m1.1.1.1.1.1.1.1.1" xref="S4.E6.m1.1.1.1.1.1.1.1.1.cmml"><mtr id="S4.E6.m1.1.1.1.1.1.1.1.1a" xref="S4.E6.m1.1.1.1.1.1.1.1.1.cmml"><mtd id="S4.E6.m1.1.1.1.1.1.1.1.1b" xref="S4.E6.m1.1.1.1.1.1.1.1.1.cmml"><mrow id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mstyle displaystyle="false" id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><msub id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1a" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">∑</mo><mrow id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.2" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml">j</mi><mo id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.1" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml">∈</mo><msub id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.3" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml"><mi id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.2" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.2.cmml">g</mi><mi id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.3" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.3.cmml">i</mi></msub></mrow></msub></mstyle><msub id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.2" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.3" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml">j</mi></msub></mrow></mtd></mtr></mtable><mo id="S4.E6.m1.1.1.1.1.1.1.3.2" xref="S4.E6.m1.1.1.1.1.1.1.2.1.cmml">]</mo></mrow><mrow id="S4.E6.m1.9.9.9.8.3.3.1" xref="S4.E6.m1.9.9.9.8.3.3.1.cmml"><mi id="S4.E6.m1.9.9.9.8.3.3.1.2" xref="S4.E6.m1.9.9.9.8.3.3.1.2.cmml">i</mi><mo id="S4.E6.m1.9.9.9.8.3.3.1.1" xref="S4.E6.m1.9.9.9.8.3.3.1.1.cmml">=</mo><mn id="S4.E6.m1.9.9.9.8.3.3.1.3" xref="S4.E6.m1.9.9.9.8.3.3.1.3.cmml">1</mn></mrow><mi id="S4.E6.m1.10.10.10.9.4.4.1" xref="S4.E6.m1.10.10.10.9.4.4.1.cmml">r</mi></msubsup></mrow></mtd></mtr><mtr id="S4.E6.m1.18.18d" xref="S4.E6.m1.18.19.1.cmml"><mtd class="ltx_align_right" columnalign="right" id="S4.E6.m1.18.18e" xref="S4.E6.m1.18.19.1.cmml"><mrow id="S4.E6.m1.15.15.15.6.5" xref="S4.E6.m1.18.19.1.cmml"><msub id="S4.E6.m1.15.15.15.6.5.7" xref="S4.E6.m1.18.19.1.cmml"><mi id="S4.E6.m1.11.11.11.2.1.1" xref="S4.E6.m1.11.11.11.2.1.1.cmml">f</mi><mtext id="S4.E6.m1.12.12.12.3.2.2.1" xref="S4.E6.m1.12.12.12.3.2.2.1a.cmml">decomp</mtext></msub><mo lspace="0em" rspace="0em" id="S4.E6.m1.15.15.15.6.5.6" xref="S4.E6.m1.18.19.1a.cmml">​</mo><mrow id="S4.E6.m1.15.15.15.6.5.8" xref="S4.E6.m1.18.19.1.cmml"><mo id="S4.E6.m1.13.13.13.4.3.3" xref="S4.E6.m1.18.19.1a.cmml">(</mo><mi id="S4.E6.m1.14.14.14.5.4.4" xref="S4.E6.m1.14.14.14.5.4.4.cmml">x</mi><mo id="S4.E6.m1.15.15.15.6.5.5" xref="S4.E6.m1.18.19.1a.cmml">)</mo></mrow></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S4.E6.m1.18.18f" xref="S4.E6.m1.18.19.1.cmml"><mrow id="S4.E6.m1.18.18.18.9.4" xref="S4.E6.m1.18.19.1.cmml"><mi id="S4.E6.m1.18.18.18.9.4.5" xref="S4.E6.m1.18.19.1a.cmml"></mi><mo id="S4.E6.m1.16.16.16.7.2.2" xref="S4.E6.m1.16.16.16.7.2.2.cmml">=</mo><msubsup id="S4.E6.m1.18.18.18.9.4.6" xref="S4.E6.m1.18.19.1.cmml"><mrow id="S4.E6.m1.2.2.2.1.1.1.3" xref="S4.E6.m1.2.2.2.1.1.1.2.cmml"><mo id="S4.E6.m1.2.2.2.1.1.1.3.1" xref="S4.E6.m1.2.2.2.1.1.1.2.1.cmml">[</mo><mtable displaystyle="true" id="S4.E6.m1.2.2.2.1.1.1.1.1" xref="S4.E6.m1.2.2.2.1.1.1.1.1.cmml"><mtr id="S4.E6.m1.2.2.2.1.1.1.1.1a" xref="S4.E6.m1.2.2.2.1.1.1.1.1.cmml"><mtd id="S4.E6.m1.2.2.2.1.1.1.1.1b" xref="S4.E6.m1.2.2.2.1.1.1.1.1.cmml"><msub id="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1" xref="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.cmml"><mi id="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.2" xref="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.2.cmml">x</mi><msubsup id="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3" xref="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.cmml"><mover accent="true" id="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.2" xref="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.2.cmml"><mi id="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.2.2" xref="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.2.2.cmml">g</mi><mo id="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.2.1" xref="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.2.1.cmml">~</mo></mover><mi id="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.3" xref="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.3.cmml">i</mi><mo id="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.3" xref="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.3.cmml">′</mo></msubsup></msub></mtd></mtr></mtable><mo id="S4.E6.m1.2.2.2.1.1.1.3.2" xref="S4.E6.m1.2.2.2.1.1.1.2.1.cmml">]</mo></mrow><mrow id="S4.E6.m1.17.17.17.8.3.3.1" xref="S4.E6.m1.17.17.17.8.3.3.1.cmml"><mi id="S4.E6.m1.17.17.17.8.3.3.1.2" xref="S4.E6.m1.17.17.17.8.3.3.1.2.cmml">i</mi><mo id="S4.E6.m1.17.17.17.8.3.3.1.1" xref="S4.E6.m1.17.17.17.8.3.3.1.1.cmml">=</mo><mn id="S4.E6.m1.17.17.17.8.3.3.1.3" xref="S4.E6.m1.17.17.17.8.3.3.1.3.cmml">1</mn></mrow><mi id="S4.E6.m1.18.18.18.9.4.4.1" xref="S4.E6.m1.18.18.18.9.4.4.1.cmml">d</mi></msubsup></mrow></mtd></mtr></mtable><annotation-xml encoding="MathML-Content" id="S4.E6.m1.18b"><apply id="S4.E6.m1.18.19.1.cmml" xref="S4.E6.m1.18.18"><and id="S4.E6.m1.18.19.1a.cmml" xref="S4.E6.m1.7.7.7.6.5.6"></and><apply id="S4.E6.m1.18.19.1b.cmml" xref="S4.E6.m1.18.18"><eq id="S4.E6.m1.8.8.8.7.2.2.cmml" xref="S4.E6.m1.8.8.8.7.2.2"></eq><apply id="S4.E6.m1.18.19.1.2.cmml" xref="S4.E6.m1.18.18"><times id="S4.E6.m1.18.19.1.2.1.cmml" xref="S4.E6.m1.7.7.7.6.5.6"></times><apply id="S4.E6.m1.18.19.1.2.2.cmml" xref="S4.E6.m1.18.18"><csymbol cd="ambiguous" id="S4.E6.m1.18.19.1.2.2.1.cmml" xref="S4.E6.m1.7.7.7.6.5.6">subscript</csymbol><ci id="S4.E6.m1.3.3.3.2.1.1.cmml" xref="S4.E6.m1.3.3.3.2.1.1">𝑓</ci><ci id="S4.E6.m1.4.4.4.3.2.2.1a.cmml" xref="S4.E6.m1.4.4.4.3.2.2.1"><mtext mathsize="70%" id="S4.E6.m1.4.4.4.3.2.2.1.cmml" xref="S4.E6.m1.4.4.4.3.2.2.1">comp</mtext></ci></apply><ci id="S4.E6.m1.6.6.6.5.4.4.cmml" xref="S4.E6.m1.6.6.6.5.4.4">𝑥</ci></apply><apply id="S4.E6.m1.18.19.1.4.cmml" xref="S4.E6.m1.18.18"><times id="S4.E6.m1.18.19.1.4.1.cmml" xref="S4.E6.m1.7.7.7.6.5.6"></times><apply id="S4.E6.m1.18.19.1.4.2.cmml" xref="S4.E6.m1.18.18"><csymbol cd="ambiguous" id="S4.E6.m1.18.19.1.4.2.1.cmml" xref="S4.E6.m1.7.7.7.6.5.6">superscript</csymbol><apply id="S4.E6.m1.18.19.1.4.2.2.cmml" xref="S4.E6.m1.18.18"><csymbol cd="ambiguous" id="S4.E6.m1.18.19.1.4.2.2.1.cmml" xref="S4.E6.m1.7.7.7.6.5.6">subscript</csymbol><apply id="S4.E6.m1.1.1.1.1.1.1.2.cmml" xref="S4.E6.m1.1.1.1.1.1.1.3"><csymbol cd="latexml" id="S4.E6.m1.1.1.1.1.1.1.2.1.cmml" xref="S4.E6.m1.1.1.1.1.1.1.3.1">matrix</csymbol><matrix id="S4.E6.m1.1.1.1.1.1.1.1.1.cmml" xref="S4.E6.m1.1.1.1.1.1.1.1.1"><matrixrow id="S4.E6.m1.1.1.1.1.1.1.1.1a.cmml" xref="S4.E6.m1.1.1.1.1.1.1.1.1"><apply id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1"><apply id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><sum id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.2"></sum><apply id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3"><in id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.1"></in><ci id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.2">𝑗</ci><apply id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.1.cmml" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.3">subscript</csymbol><ci id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.2.cmml" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.2">𝑔</ci><ci id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.3.cmml" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.3">𝑖</ci></apply></apply></apply><apply id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S4.E6.m1.1.1.1.1.1.1.1.1.1.1.1.2.3">𝑗</ci></apply></apply></matrixrow></matrix></apply><apply id="S4.E6.m1.9.9.9.8.3.3.1.cmml" xref="S4.E6.m1.9.9.9.8.3.3.1"><eq id="S4.E6.m1.9.9.9.8.3.3.1.1.cmml" xref="S4.E6.m1.9.9.9.8.3.3.1.1"></eq><ci id="S4.E6.m1.9.9.9.8.3.3.1.2.cmml" xref="S4.E6.m1.9.9.9.8.3.3.1.2">𝑖</ci><cn type="integer" id="S4.E6.m1.9.9.9.8.3.3.1.3.cmml" xref="S4.E6.m1.9.9.9.8.3.3.1.3">1</cn></apply></apply><ci id="S4.E6.m1.10.10.10.9.4.4.1.cmml" xref="S4.E6.m1.10.10.10.9.4.4.1">𝑟</ci></apply><apply id="S4.E6.m1.18.19.1.4.3.cmml" xref="S4.E6.m1.18.18"><csymbol cd="ambiguous" id="S4.E6.m1.18.19.1.4.3.1.cmml" xref="S4.E6.m1.7.7.7.6.5.6">subscript</csymbol><ci id="S4.E6.m1.11.11.11.2.1.1.cmml" xref="S4.E6.m1.11.11.11.2.1.1">𝑓</ci><ci id="S4.E6.m1.12.12.12.3.2.2.1a.cmml" xref="S4.E6.m1.12.12.12.3.2.2.1"><mtext mathsize="70%" id="S4.E6.m1.12.12.12.3.2.2.1.cmml" xref="S4.E6.m1.12.12.12.3.2.2.1">decomp</mtext></ci></apply><ci id="S4.E6.m1.14.14.14.5.4.4.cmml" xref="S4.E6.m1.14.14.14.5.4.4">𝑥</ci></apply></apply><apply id="S4.E6.m1.18.19.1c.cmml" xref="S4.E6.m1.18.18"><eq id="S4.E6.m1.16.16.16.7.2.2.cmml" xref="S4.E6.m1.16.16.16.7.2.2"></eq><share href="#S4.E6.m1.18.19.1.4.cmml" id="S4.E6.m1.18.19.1d.cmml" xref="S4.E6.m1.7.7.7.6.5.6"></share><apply id="S4.E6.m1.18.19.1.6.cmml" xref="S4.E6.m1.18.18"><csymbol cd="ambiguous" id="S4.E6.m1.18.19.1.6.1.cmml" xref="S4.E6.m1.7.7.7.6.5.6">superscript</csymbol><apply id="S4.E6.m1.18.19.1.6.2.cmml" xref="S4.E6.m1.18.18"><csymbol cd="ambiguous" id="S4.E6.m1.18.19.1.6.2.1.cmml" xref="S4.E6.m1.7.7.7.6.5.6">subscript</csymbol><apply id="S4.E6.m1.2.2.2.1.1.1.2.cmml" xref="S4.E6.m1.2.2.2.1.1.1.3"><csymbol cd="latexml" id="S4.E6.m1.2.2.2.1.1.1.2.1.cmml" xref="S4.E6.m1.2.2.2.1.1.1.3.1">matrix</csymbol><matrix id="S4.E6.m1.2.2.2.1.1.1.1.1.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.1"><matrixrow id="S4.E6.m1.2.2.2.1.1.1.1.1a.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.1"><apply id="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.2.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.2">𝑥</ci><apply id="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.1.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3">subscript</csymbol><apply id="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.1.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3">superscript</csymbol><apply id="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.2.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.2"><ci id="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.2.1.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.2.1">~</ci><ci id="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.2.2.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.2.2">𝑔</ci></apply><ci id="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.3.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.2.3">′</ci></apply><ci id="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.3.cmml" xref="S4.E6.m1.2.2.2.1.1.1.1.1.1.1.1.3.3">𝑖</ci></apply></apply></matrixrow></matrix></apply><apply id="S4.E6.m1.17.17.17.8.3.3.1.cmml" xref="S4.E6.m1.17.17.17.8.3.3.1"><eq id="S4.E6.m1.17.17.17.8.3.3.1.1.cmml" xref="S4.E6.m1.17.17.17.8.3.3.1.1"></eq><ci id="S4.E6.m1.17.17.17.8.3.3.1.2.cmml" xref="S4.E6.m1.17.17.17.8.3.3.1.2">𝑖</ci><cn type="integer" id="S4.E6.m1.17.17.17.8.3.3.1.3.cmml" xref="S4.E6.m1.17.17.17.8.3.3.1.3">1</cn></apply></apply><ci id="S4.E6.m1.18.18.18.9.4.4.1.cmml" xref="S4.E6.m1.18.18.18.9.4.4.1">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E6.m1.18c">\begin{split}f_{\text{comp}}\left(x\right)&amp;=\begin{bmatrix}\sum_{j\in g_{i}}x_{j}\end{bmatrix}_{i=1}^{r}\\
f_{\text{decomp}}\left(x\right)&amp;=\begin{bmatrix}x_{\widetilde{g}^{\prime}_{i}}\end{bmatrix}_{i=1}^{d}\end{split}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(6)</span></td>
</tr></tbody>
</table>
<p id="S4.p3.17" class="ltx_p">Here, <math id="S4.p3.5.m1.1" class="ltx_Math" alttext="g" display="inline"><semantics id="S4.p3.5.m1.1a"><mi id="S4.p3.5.m1.1.1" xref="S4.p3.5.m1.1.1.cmml">g</mi><annotation-xml encoding="MathML-Content" id="S4.p3.5.m1.1b"><ci id="S4.p3.5.m1.1.1.cmml" xref="S4.p3.5.m1.1.1">𝑔</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.5.m1.1c">g</annotation></semantics></math> and <math id="S4.p3.6.m2.1" class="ltx_Math" alttext="g^{\prime}" display="inline"><semantics id="S4.p3.6.m2.1a"><msup id="S4.p3.6.m2.1.1" xref="S4.p3.6.m2.1.1.cmml"><mi id="S4.p3.6.m2.1.1.2" xref="S4.p3.6.m2.1.1.2.cmml">g</mi><mo id="S4.p3.6.m2.1.1.3" xref="S4.p3.6.m2.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.p3.6.m2.1b"><apply id="S4.p3.6.m2.1.1.cmml" xref="S4.p3.6.m2.1.1"><csymbol cd="ambiguous" id="S4.p3.6.m2.1.1.1.cmml" xref="S4.p3.6.m2.1.1">superscript</csymbol><ci id="S4.p3.6.m2.1.1.2.cmml" xref="S4.p3.6.m2.1.1.2">𝑔</ci><ci id="S4.p3.6.m2.1.1.3.cmml" xref="S4.p3.6.m2.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.6.m2.1c">g^{\prime}</annotation></semantics></math> represent predefined groups that share the same row and column in <math id="S4.p3.7.m3.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S4.p3.7.m3.1a"><mi id="S4.p3.7.m3.1.1" xref="S4.p3.7.m3.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S4.p3.7.m3.1b"><ci id="S4.p3.7.m3.1.1.cmml" xref="S4.p3.7.m3.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.7.m3.1c">M</annotation></semantics></math>, respectively. The <math id="S4.p3.8.m4.1" class="ltx_Math" alttext="j\in g_{i}" display="inline"><semantics id="S4.p3.8.m4.1a"><mrow id="S4.p3.8.m4.1.1" xref="S4.p3.8.m4.1.1.cmml"><mi id="S4.p3.8.m4.1.1.2" xref="S4.p3.8.m4.1.1.2.cmml">j</mi><mo id="S4.p3.8.m4.1.1.1" xref="S4.p3.8.m4.1.1.1.cmml">∈</mo><msub id="S4.p3.8.m4.1.1.3" xref="S4.p3.8.m4.1.1.3.cmml"><mi id="S4.p3.8.m4.1.1.3.2" xref="S4.p3.8.m4.1.1.3.2.cmml">g</mi><mi id="S4.p3.8.m4.1.1.3.3" xref="S4.p3.8.m4.1.1.3.3.cmml">i</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S4.p3.8.m4.1b"><apply id="S4.p3.8.m4.1.1.cmml" xref="S4.p3.8.m4.1.1"><in id="S4.p3.8.m4.1.1.1.cmml" xref="S4.p3.8.m4.1.1.1"></in><ci id="S4.p3.8.m4.1.1.2.cmml" xref="S4.p3.8.m4.1.1.2">𝑗</ci><apply id="S4.p3.8.m4.1.1.3.cmml" xref="S4.p3.8.m4.1.1.3"><csymbol cd="ambiguous" id="S4.p3.8.m4.1.1.3.1.cmml" xref="S4.p3.8.m4.1.1.3">subscript</csymbol><ci id="S4.p3.8.m4.1.1.3.2.cmml" xref="S4.p3.8.m4.1.1.3.2">𝑔</ci><ci id="S4.p3.8.m4.1.1.3.3.cmml" xref="S4.p3.8.m4.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.8.m4.1c">j\in g_{i}</annotation></semantics></math> indicates that the <math id="S4.p3.9.m5.1" class="ltx_Math" alttext="j" display="inline"><semantics id="S4.p3.9.m5.1a"><mi id="S4.p3.9.m5.1.1" xref="S4.p3.9.m5.1.1.cmml">j</mi><annotation-xml encoding="MathML-Content" id="S4.p3.9.m5.1b"><ci id="S4.p3.9.m5.1.1.cmml" xref="S4.p3.9.m5.1.1">𝑗</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.9.m5.1c">j</annotation></semantics></math>-th dimension belongs to the <math id="S4.p3.10.m6.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S4.p3.10.m6.1a"><mi id="S4.p3.10.m6.1.1" xref="S4.p3.10.m6.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S4.p3.10.m6.1b"><ci id="S4.p3.10.m6.1.1.cmml" xref="S4.p3.10.m6.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.10.m6.1c">i</annotation></semantics></math>-th group in <math id="S4.p3.11.m7.1" class="ltx_Math" alttext="g" display="inline"><semantics id="S4.p3.11.m7.1a"><mi id="S4.p3.11.m7.1.1" xref="S4.p3.11.m7.1.1.cmml">g</mi><annotation-xml encoding="MathML-Content" id="S4.p3.11.m7.1b"><ci id="S4.p3.11.m7.1.1.cmml" xref="S4.p3.11.m7.1.1">𝑔</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.11.m7.1c">g</annotation></semantics></math>. The term <math id="S4.p3.12.m8.1" class="ltx_Math" alttext="\widetilde{g}^{\prime}_{i}" display="inline"><semantics id="S4.p3.12.m8.1a"><msubsup id="S4.p3.12.m8.1.1" xref="S4.p3.12.m8.1.1.cmml"><mover accent="true" id="S4.p3.12.m8.1.1.2.2" xref="S4.p3.12.m8.1.1.2.2.cmml"><mi id="S4.p3.12.m8.1.1.2.2.2" xref="S4.p3.12.m8.1.1.2.2.2.cmml">g</mi><mo id="S4.p3.12.m8.1.1.2.2.1" xref="S4.p3.12.m8.1.1.2.2.1.cmml">~</mo></mover><mi id="S4.p3.12.m8.1.1.3" xref="S4.p3.12.m8.1.1.3.cmml">i</mi><mo id="S4.p3.12.m8.1.1.2.3" xref="S4.p3.12.m8.1.1.2.3.cmml">′</mo></msubsup><annotation-xml encoding="MathML-Content" id="S4.p3.12.m8.1b"><apply id="S4.p3.12.m8.1.1.cmml" xref="S4.p3.12.m8.1.1"><csymbol cd="ambiguous" id="S4.p3.12.m8.1.1.1.cmml" xref="S4.p3.12.m8.1.1">subscript</csymbol><apply id="S4.p3.12.m8.1.1.2.cmml" xref="S4.p3.12.m8.1.1"><csymbol cd="ambiguous" id="S4.p3.12.m8.1.1.2.1.cmml" xref="S4.p3.12.m8.1.1">superscript</csymbol><apply id="S4.p3.12.m8.1.1.2.2.cmml" xref="S4.p3.12.m8.1.1.2.2"><ci id="S4.p3.12.m8.1.1.2.2.1.cmml" xref="S4.p3.12.m8.1.1.2.2.1">~</ci><ci id="S4.p3.12.m8.1.1.2.2.2.cmml" xref="S4.p3.12.m8.1.1.2.2.2">𝑔</ci></apply><ci id="S4.p3.12.m8.1.1.2.3.cmml" xref="S4.p3.12.m8.1.1.2.3">′</ci></apply><ci id="S4.p3.12.m8.1.1.3.cmml" xref="S4.p3.12.m8.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.12.m8.1c">\widetilde{g}^{\prime}_{i}</annotation></semantics></math> is the reverse of <math id="S4.p3.13.m9.1" class="ltx_Math" alttext="g^{\prime}_{i}" display="inline"><semantics id="S4.p3.13.m9.1a"><msubsup id="S4.p3.13.m9.1.1" xref="S4.p3.13.m9.1.1.cmml"><mi id="S4.p3.13.m9.1.1.2.2" xref="S4.p3.13.m9.1.1.2.2.cmml">g</mi><mi id="S4.p3.13.m9.1.1.3" xref="S4.p3.13.m9.1.1.3.cmml">i</mi><mo id="S4.p3.13.m9.1.1.2.3" xref="S4.p3.13.m9.1.1.2.3.cmml">′</mo></msubsup><annotation-xml encoding="MathML-Content" id="S4.p3.13.m9.1b"><apply id="S4.p3.13.m9.1.1.cmml" xref="S4.p3.13.m9.1.1"><csymbol cd="ambiguous" id="S4.p3.13.m9.1.1.1.cmml" xref="S4.p3.13.m9.1.1">subscript</csymbol><apply id="S4.p3.13.m9.1.1.2.cmml" xref="S4.p3.13.m9.1.1"><csymbol cd="ambiguous" id="S4.p3.13.m9.1.1.2.1.cmml" xref="S4.p3.13.m9.1.1">superscript</csymbol><ci id="S4.p3.13.m9.1.1.2.2.cmml" xref="S4.p3.13.m9.1.1.2.2">𝑔</ci><ci id="S4.p3.13.m9.1.1.2.3.cmml" xref="S4.p3.13.m9.1.1.2.3">′</ci></apply><ci id="S4.p3.13.m9.1.1.3.cmml" xref="S4.p3.13.m9.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.13.m9.1c">g^{\prime}_{i}</annotation></semantics></math>, referring to the <math id="S4.p3.14.m10.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S4.p3.14.m10.1a"><mi id="S4.p3.14.m10.1.1" xref="S4.p3.14.m10.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S4.p3.14.m10.1b"><ci id="S4.p3.14.m10.1.1.cmml" xref="S4.p3.14.m10.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.14.m10.1c">i</annotation></semantics></math>-th dimension associated with the <math id="S4.p3.15.m11.1" class="ltx_Math" alttext="\widetilde{g}^{\prime}_{i}" display="inline"><semantics id="S4.p3.15.m11.1a"><msubsup id="S4.p3.15.m11.1.1" xref="S4.p3.15.m11.1.1.cmml"><mover accent="true" id="S4.p3.15.m11.1.1.2.2" xref="S4.p3.15.m11.1.1.2.2.cmml"><mi id="S4.p3.15.m11.1.1.2.2.2" xref="S4.p3.15.m11.1.1.2.2.2.cmml">g</mi><mo id="S4.p3.15.m11.1.1.2.2.1" xref="S4.p3.15.m11.1.1.2.2.1.cmml">~</mo></mover><mi id="S4.p3.15.m11.1.1.3" xref="S4.p3.15.m11.1.1.3.cmml">i</mi><mo id="S4.p3.15.m11.1.1.2.3" xref="S4.p3.15.m11.1.1.2.3.cmml">′</mo></msubsup><annotation-xml encoding="MathML-Content" id="S4.p3.15.m11.1b"><apply id="S4.p3.15.m11.1.1.cmml" xref="S4.p3.15.m11.1.1"><csymbol cd="ambiguous" id="S4.p3.15.m11.1.1.1.cmml" xref="S4.p3.15.m11.1.1">subscript</csymbol><apply id="S4.p3.15.m11.1.1.2.cmml" xref="S4.p3.15.m11.1.1"><csymbol cd="ambiguous" id="S4.p3.15.m11.1.1.2.1.cmml" xref="S4.p3.15.m11.1.1">superscript</csymbol><apply id="S4.p3.15.m11.1.1.2.2.cmml" xref="S4.p3.15.m11.1.1.2.2"><ci id="S4.p3.15.m11.1.1.2.2.1.cmml" xref="S4.p3.15.m11.1.1.2.2.1">~</ci><ci id="S4.p3.15.m11.1.1.2.2.2.cmml" xref="S4.p3.15.m11.1.1.2.2.2">𝑔</ci></apply><ci id="S4.p3.15.m11.1.1.2.3.cmml" xref="S4.p3.15.m11.1.1.2.3">′</ci></apply><ci id="S4.p3.15.m11.1.1.3.cmml" xref="S4.p3.15.m11.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.15.m11.1c">\widetilde{g}^{\prime}_{i}</annotation></semantics></math>-th group in <math id="S4.p3.16.m12.1" class="ltx_Math" alttext="g^{\prime}" display="inline"><semantics id="S4.p3.16.m12.1a"><msup id="S4.p3.16.m12.1.1" xref="S4.p3.16.m12.1.1.cmml"><mi id="S4.p3.16.m12.1.1.2" xref="S4.p3.16.m12.1.1.2.cmml">g</mi><mo id="S4.p3.16.m12.1.1.3" xref="S4.p3.16.m12.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S4.p3.16.m12.1b"><apply id="S4.p3.16.m12.1.1.cmml" xref="S4.p3.16.m12.1.1"><csymbol cd="ambiguous" id="S4.p3.16.m12.1.1.1.cmml" xref="S4.p3.16.m12.1.1">superscript</csymbol><ci id="S4.p3.16.m12.1.1.2.cmml" xref="S4.p3.16.m12.1.1.2">𝑔</ci><ci id="S4.p3.16.m12.1.1.3.cmml" xref="S4.p3.16.m12.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.16.m12.1c">g^{\prime}</annotation></semantics></math>. The corresponding <math id="S4.p3.17.m13.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S4.p3.17.m13.1a"><mrow id="S4.p3.17.m13.1.1" xref="S4.p3.17.m13.1.1.cmml"><mi mathvariant="normal" id="S4.p3.17.m13.1.1.2" xref="S4.p3.17.m13.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.p3.17.m13.1.1.1" xref="S4.p3.17.m13.1.1.1.cmml">​</mo><mi id="S4.p3.17.m13.1.1.3" xref="S4.p3.17.m13.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p3.17.m13.1b"><apply id="S4.p3.17.m13.1.1.cmml" xref="S4.p3.17.m13.1.1"><times id="S4.p3.17.m13.1.1.1.cmml" xref="S4.p3.17.m13.1.1.1"></times><ci id="S4.p3.17.m13.1.1.2.cmml" xref="S4.p3.17.m13.1.1.2">Δ</ci><ci id="S4.p3.17.m13.1.1.3.cmml" xref="S4.p3.17.m13.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.17.m13.1c">\Delta W</annotation></semantics></math> is as follows:</p>
</div>
<div id="S4.p4" class="ltx_para">
<table id="S4.E7" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E7.m1.4" class="ltx_Math" alttext="\Delta W_{i,j}=M_{\widetilde{g}^{\prime}_{i},\widetilde{g}_{j}}" display="block"><semantics id="S4.E7.m1.4a"><mrow id="S4.E7.m1.4.5" xref="S4.E7.m1.4.5.cmml"><mrow id="S4.E7.m1.4.5.2" xref="S4.E7.m1.4.5.2.cmml"><mi mathvariant="normal" id="S4.E7.m1.4.5.2.2" xref="S4.E7.m1.4.5.2.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.E7.m1.4.5.2.1" xref="S4.E7.m1.4.5.2.1.cmml">​</mo><msub id="S4.E7.m1.4.5.2.3" xref="S4.E7.m1.4.5.2.3.cmml"><mi id="S4.E7.m1.4.5.2.3.2" xref="S4.E7.m1.4.5.2.3.2.cmml">W</mi><mrow id="S4.E7.m1.2.2.2.4" xref="S4.E7.m1.2.2.2.3.cmml"><mi id="S4.E7.m1.1.1.1.1" xref="S4.E7.m1.1.1.1.1.cmml">i</mi><mo id="S4.E7.m1.2.2.2.4.1" xref="S4.E7.m1.2.2.2.3.cmml">,</mo><mi id="S4.E7.m1.2.2.2.2" xref="S4.E7.m1.2.2.2.2.cmml">j</mi></mrow></msub></mrow><mo id="S4.E7.m1.4.5.1" xref="S4.E7.m1.4.5.1.cmml">=</mo><msub id="S4.E7.m1.4.5.3" xref="S4.E7.m1.4.5.3.cmml"><mi id="S4.E7.m1.4.5.3.2" xref="S4.E7.m1.4.5.3.2.cmml">M</mi><mrow id="S4.E7.m1.4.4.2.2" xref="S4.E7.m1.4.4.2.3.cmml"><msubsup id="S4.E7.m1.3.3.1.1.1" xref="S4.E7.m1.3.3.1.1.1.cmml"><mover accent="true" id="S4.E7.m1.3.3.1.1.1.2.2" xref="S4.E7.m1.3.3.1.1.1.2.2.cmml"><mi id="S4.E7.m1.3.3.1.1.1.2.2.2" xref="S4.E7.m1.3.3.1.1.1.2.2.2.cmml">g</mi><mo id="S4.E7.m1.3.3.1.1.1.2.2.1" xref="S4.E7.m1.3.3.1.1.1.2.2.1.cmml">~</mo></mover><mi id="S4.E7.m1.3.3.1.1.1.3" xref="S4.E7.m1.3.3.1.1.1.3.cmml">i</mi><mo id="S4.E7.m1.3.3.1.1.1.2.3" xref="S4.E7.m1.3.3.1.1.1.2.3.cmml">′</mo></msubsup><mo id="S4.E7.m1.4.4.2.2.3" xref="S4.E7.m1.4.4.2.3.cmml">,</mo><msub id="S4.E7.m1.4.4.2.2.2" xref="S4.E7.m1.4.4.2.2.2.cmml"><mover accent="true" id="S4.E7.m1.4.4.2.2.2.2" xref="S4.E7.m1.4.4.2.2.2.2.cmml"><mi id="S4.E7.m1.4.4.2.2.2.2.2" xref="S4.E7.m1.4.4.2.2.2.2.2.cmml">g</mi><mo id="S4.E7.m1.4.4.2.2.2.2.1" xref="S4.E7.m1.4.4.2.2.2.2.1.cmml">~</mo></mover><mi id="S4.E7.m1.4.4.2.2.2.3" xref="S4.E7.m1.4.4.2.2.2.3.cmml">j</mi></msub></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="S4.E7.m1.4b"><apply id="S4.E7.m1.4.5.cmml" xref="S4.E7.m1.4.5"><eq id="S4.E7.m1.4.5.1.cmml" xref="S4.E7.m1.4.5.1"></eq><apply id="S4.E7.m1.4.5.2.cmml" xref="S4.E7.m1.4.5.2"><times id="S4.E7.m1.4.5.2.1.cmml" xref="S4.E7.m1.4.5.2.1"></times><ci id="S4.E7.m1.4.5.2.2.cmml" xref="S4.E7.m1.4.5.2.2">Δ</ci><apply id="S4.E7.m1.4.5.2.3.cmml" xref="S4.E7.m1.4.5.2.3"><csymbol cd="ambiguous" id="S4.E7.m1.4.5.2.3.1.cmml" xref="S4.E7.m1.4.5.2.3">subscript</csymbol><ci id="S4.E7.m1.4.5.2.3.2.cmml" xref="S4.E7.m1.4.5.2.3.2">𝑊</ci><list id="S4.E7.m1.2.2.2.3.cmml" xref="S4.E7.m1.2.2.2.4"><ci id="S4.E7.m1.1.1.1.1.cmml" xref="S4.E7.m1.1.1.1.1">𝑖</ci><ci id="S4.E7.m1.2.2.2.2.cmml" xref="S4.E7.m1.2.2.2.2">𝑗</ci></list></apply></apply><apply id="S4.E7.m1.4.5.3.cmml" xref="S4.E7.m1.4.5.3"><csymbol cd="ambiguous" id="S4.E7.m1.4.5.3.1.cmml" xref="S4.E7.m1.4.5.3">subscript</csymbol><ci id="S4.E7.m1.4.5.3.2.cmml" xref="S4.E7.m1.4.5.3.2">𝑀</ci><list id="S4.E7.m1.4.4.2.3.cmml" xref="S4.E7.m1.4.4.2.2"><apply id="S4.E7.m1.3.3.1.1.1.cmml" xref="S4.E7.m1.3.3.1.1.1"><csymbol cd="ambiguous" id="S4.E7.m1.3.3.1.1.1.1.cmml" xref="S4.E7.m1.3.3.1.1.1">subscript</csymbol><apply id="S4.E7.m1.3.3.1.1.1.2.cmml" xref="S4.E7.m1.3.3.1.1.1"><csymbol cd="ambiguous" id="S4.E7.m1.3.3.1.1.1.2.1.cmml" xref="S4.E7.m1.3.3.1.1.1">superscript</csymbol><apply id="S4.E7.m1.3.3.1.1.1.2.2.cmml" xref="S4.E7.m1.3.3.1.1.1.2.2"><ci id="S4.E7.m1.3.3.1.1.1.2.2.1.cmml" xref="S4.E7.m1.3.3.1.1.1.2.2.1">~</ci><ci id="S4.E7.m1.3.3.1.1.1.2.2.2.cmml" xref="S4.E7.m1.3.3.1.1.1.2.2.2">𝑔</ci></apply><ci id="S4.E7.m1.3.3.1.1.1.2.3.cmml" xref="S4.E7.m1.3.3.1.1.1.2.3">′</ci></apply><ci id="S4.E7.m1.3.3.1.1.1.3.cmml" xref="S4.E7.m1.3.3.1.1.1.3">𝑖</ci></apply><apply id="S4.E7.m1.4.4.2.2.2.cmml" xref="S4.E7.m1.4.4.2.2.2"><csymbol cd="ambiguous" id="S4.E7.m1.4.4.2.2.2.1.cmml" xref="S4.E7.m1.4.4.2.2.2">subscript</csymbol><apply id="S4.E7.m1.4.4.2.2.2.2.cmml" xref="S4.E7.m1.4.4.2.2.2.2"><ci id="S4.E7.m1.4.4.2.2.2.2.1.cmml" xref="S4.E7.m1.4.4.2.2.2.2.1">~</ci><ci id="S4.E7.m1.4.4.2.2.2.2.2.cmml" xref="S4.E7.m1.4.4.2.2.2.2.2">𝑔</ci></apply><ci id="S4.E7.m1.4.4.2.2.2.3.cmml" xref="S4.E7.m1.4.4.2.2.2.3">𝑗</ci></apply></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E7.m1.4c">\Delta W_{i,j}=M_{\widetilde{g}^{\prime}_{i},\widetilde{g}_{j}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(7)</span></td>
</tr></tbody>
</table>
<p id="S4.p4.10" class="ltx_p">Sharing rows and columns can be efficient for larger ranks such as <math id="S4.p4.1.m1.1" class="ltx_Math" alttext="r=128" display="inline"><semantics id="S4.p4.1.m1.1a"><mrow id="S4.p4.1.m1.1.1" xref="S4.p4.1.m1.1.1.cmml"><mi id="S4.p4.1.m1.1.1.2" xref="S4.p4.1.m1.1.1.2.cmml">r</mi><mo id="S4.p4.1.m1.1.1.1" xref="S4.p4.1.m1.1.1.1.cmml">=</mo><mn id="S4.p4.1.m1.1.1.3" xref="S4.p4.1.m1.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.p4.1.m1.1b"><apply id="S4.p4.1.m1.1.1.cmml" xref="S4.p4.1.m1.1.1"><eq id="S4.p4.1.m1.1.1.1.cmml" xref="S4.p4.1.m1.1.1.1"></eq><ci id="S4.p4.1.m1.1.1.2.cmml" xref="S4.p4.1.m1.1.1.2">𝑟</ci><cn type="integer" id="S4.p4.1.m1.1.1.3.cmml" xref="S4.p4.1.m1.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p4.1.m1.1c">r=128</annotation></semantics></math> or <math id="S4.p4.2.m2.1" class="ltx_Math" alttext="r=256" display="inline"><semantics id="S4.p4.2.m2.1a"><mrow id="S4.p4.2.m2.1.1" xref="S4.p4.2.m2.1.1.cmml"><mi id="S4.p4.2.m2.1.1.2" xref="S4.p4.2.m2.1.1.2.cmml">r</mi><mo id="S4.p4.2.m2.1.1.1" xref="S4.p4.2.m2.1.1.1.cmml">=</mo><mn id="S4.p4.2.m2.1.1.3" xref="S4.p4.2.m2.1.1.3.cmml">256</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.p4.2.m2.1b"><apply id="S4.p4.2.m2.1.1.cmml" xref="S4.p4.2.m2.1.1"><eq id="S4.p4.2.m2.1.1.1.cmml" xref="S4.p4.2.m2.1.1.1"></eq><ci id="S4.p4.2.m2.1.1.2.cmml" xref="S4.p4.2.m2.1.1.2">𝑟</ci><cn type="integer" id="S4.p4.2.m2.1.1.3.cmml" xref="S4.p4.2.m2.1.1.3">256</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p4.2.m2.1c">r=256</annotation></semantics></math>, as only a few rows or columns in <math id="S4.p4.3.m3.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S4.p4.3.m3.1a"><mrow id="S4.p4.3.m3.1.1" xref="S4.p4.3.m3.1.1.cmml"><mi mathvariant="normal" id="S4.p4.3.m3.1.1.2" xref="S4.p4.3.m3.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.p4.3.m3.1.1.1" xref="S4.p4.3.m3.1.1.1.cmml">​</mo><mi id="S4.p4.3.m3.1.1.3" xref="S4.p4.3.m3.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p4.3.m3.1b"><apply id="S4.p4.3.m3.1.1.cmml" xref="S4.p4.3.m3.1.1"><times id="S4.p4.3.m3.1.1.1.cmml" xref="S4.p4.3.m3.1.1.1"></times><ci id="S4.p4.3.m3.1.1.2.cmml" xref="S4.p4.3.m3.1.1.2">Δ</ci><ci id="S4.p4.3.m3.1.1.3.cmml" xref="S4.p4.3.m3.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p4.3.m3.1c">\Delta W</annotation></semantics></math> share a common row or column.
For instance, considering to <math id="S4.p4.4.m4.1" class="ltx_Math" alttext="\Delta W\in\mathbb{R}^{4096\times 4096}" display="inline"><semantics id="S4.p4.4.m4.1a"><mrow id="S4.p4.4.m4.1.1" xref="S4.p4.4.m4.1.1.cmml"><mrow id="S4.p4.4.m4.1.1.2" xref="S4.p4.4.m4.1.1.2.cmml"><mi mathvariant="normal" id="S4.p4.4.m4.1.1.2.2" xref="S4.p4.4.m4.1.1.2.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.p4.4.m4.1.1.2.1" xref="S4.p4.4.m4.1.1.2.1.cmml">​</mo><mi id="S4.p4.4.m4.1.1.2.3" xref="S4.p4.4.m4.1.1.2.3.cmml">W</mi></mrow><mo id="S4.p4.4.m4.1.1.1" xref="S4.p4.4.m4.1.1.1.cmml">∈</mo><msup id="S4.p4.4.m4.1.1.3" xref="S4.p4.4.m4.1.1.3.cmml"><mi id="S4.p4.4.m4.1.1.3.2" xref="S4.p4.4.m4.1.1.3.2.cmml">ℝ</mi><mrow id="S4.p4.4.m4.1.1.3.3" xref="S4.p4.4.m4.1.1.3.3.cmml"><mn id="S4.p4.4.m4.1.1.3.3.2" xref="S4.p4.4.m4.1.1.3.3.2.cmml">4096</mn><mo lspace="0.222em" rspace="0.222em" id="S4.p4.4.m4.1.1.3.3.1" xref="S4.p4.4.m4.1.1.3.3.1.cmml">×</mo><mn id="S4.p4.4.m4.1.1.3.3.3" xref="S4.p4.4.m4.1.1.3.3.3.cmml">4096</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.p4.4.m4.1b"><apply id="S4.p4.4.m4.1.1.cmml" xref="S4.p4.4.m4.1.1"><in id="S4.p4.4.m4.1.1.1.cmml" xref="S4.p4.4.m4.1.1.1"></in><apply id="S4.p4.4.m4.1.1.2.cmml" xref="S4.p4.4.m4.1.1.2"><times id="S4.p4.4.m4.1.1.2.1.cmml" xref="S4.p4.4.m4.1.1.2.1"></times><ci id="S4.p4.4.m4.1.1.2.2.cmml" xref="S4.p4.4.m4.1.1.2.2">Δ</ci><ci id="S4.p4.4.m4.1.1.2.3.cmml" xref="S4.p4.4.m4.1.1.2.3">𝑊</ci></apply><apply id="S4.p4.4.m4.1.1.3.cmml" xref="S4.p4.4.m4.1.1.3"><csymbol cd="ambiguous" id="S4.p4.4.m4.1.1.3.1.cmml" xref="S4.p4.4.m4.1.1.3">superscript</csymbol><ci id="S4.p4.4.m4.1.1.3.2.cmml" xref="S4.p4.4.m4.1.1.3.2">ℝ</ci><apply id="S4.p4.4.m4.1.1.3.3.cmml" xref="S4.p4.4.m4.1.1.3.3"><times id="S4.p4.4.m4.1.1.3.3.1.cmml" xref="S4.p4.4.m4.1.1.3.3.1"></times><cn type="integer" id="S4.p4.4.m4.1.1.3.3.2.cmml" xref="S4.p4.4.m4.1.1.3.3.2">4096</cn><cn type="integer" id="S4.p4.4.m4.1.1.3.3.3.cmml" xref="S4.p4.4.m4.1.1.3.3.3">4096</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p4.4.m4.1c">\Delta W\in\mathbb{R}^{4096\times 4096}</annotation></semantics></math> for <math id="S4.p4.5.m5.1" class="ltx_Math" alttext="r=128" display="inline"><semantics id="S4.p4.5.m5.1a"><mrow id="S4.p4.5.m5.1.1" xref="S4.p4.5.m5.1.1.cmml"><mi id="S4.p4.5.m5.1.1.2" xref="S4.p4.5.m5.1.1.2.cmml">r</mi><mo id="S4.p4.5.m5.1.1.1" xref="S4.p4.5.m5.1.1.1.cmml">=</mo><mn id="S4.p4.5.m5.1.1.3" xref="S4.p4.5.m5.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.p4.5.m5.1b"><apply id="S4.p4.5.m5.1.1.cmml" xref="S4.p4.5.m5.1.1"><eq id="S4.p4.5.m5.1.1.1.cmml" xref="S4.p4.5.m5.1.1.1"></eq><ci id="S4.p4.5.m5.1.1.2.cmml" xref="S4.p4.5.m5.1.1.2">𝑟</ci><cn type="integer" id="S4.p4.5.m5.1.1.3.cmml" xref="S4.p4.5.m5.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p4.5.m5.1c">r=128</annotation></semantics></math>, which has <math id="S4.p4.6.m6.1" class="ltx_Math" alttext="\hat{r}=1024" display="inline"><semantics id="S4.p4.6.m6.1a"><mrow id="S4.p4.6.m6.1.1" xref="S4.p4.6.m6.1.1.cmml"><mover accent="true" id="S4.p4.6.m6.1.1.2" xref="S4.p4.6.m6.1.1.2.cmml"><mi id="S4.p4.6.m6.1.1.2.2" xref="S4.p4.6.m6.1.1.2.2.cmml">r</mi><mo id="S4.p4.6.m6.1.1.2.1" xref="S4.p4.6.m6.1.1.2.1.cmml">^</mo></mover><mo id="S4.p4.6.m6.1.1.1" xref="S4.p4.6.m6.1.1.1.cmml">=</mo><mn id="S4.p4.6.m6.1.1.3" xref="S4.p4.6.m6.1.1.3.cmml">1024</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.p4.6.m6.1b"><apply id="S4.p4.6.m6.1.1.cmml" xref="S4.p4.6.m6.1.1"><eq id="S4.p4.6.m6.1.1.1.cmml" xref="S4.p4.6.m6.1.1.1"></eq><apply id="S4.p4.6.m6.1.1.2.cmml" xref="S4.p4.6.m6.1.1.2"><ci id="S4.p4.6.m6.1.1.2.1.cmml" xref="S4.p4.6.m6.1.1.2.1">^</ci><ci id="S4.p4.6.m6.1.1.2.2.cmml" xref="S4.p4.6.m6.1.1.2.2">𝑟</ci></apply><cn type="integer" id="S4.p4.6.m6.1.1.3.cmml" xref="S4.p4.6.m6.1.1.3">1024</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p4.6.m6.1c">\hat{r}=1024</annotation></semantics></math> and <math id="S4.p4.7.m7.1" class="ltx_Math" alttext="M\in\mathbb{R}^{1024\times 1024}" display="inline"><semantics id="S4.p4.7.m7.1a"><mrow id="S4.p4.7.m7.1.1" xref="S4.p4.7.m7.1.1.cmml"><mi id="S4.p4.7.m7.1.1.2" xref="S4.p4.7.m7.1.1.2.cmml">M</mi><mo id="S4.p4.7.m7.1.1.1" xref="S4.p4.7.m7.1.1.1.cmml">∈</mo><msup id="S4.p4.7.m7.1.1.3" xref="S4.p4.7.m7.1.1.3.cmml"><mi id="S4.p4.7.m7.1.1.3.2" xref="S4.p4.7.m7.1.1.3.2.cmml">ℝ</mi><mrow id="S4.p4.7.m7.1.1.3.3" xref="S4.p4.7.m7.1.1.3.3.cmml"><mn id="S4.p4.7.m7.1.1.3.3.2" xref="S4.p4.7.m7.1.1.3.3.2.cmml">1024</mn><mo lspace="0.222em" rspace="0.222em" id="S4.p4.7.m7.1.1.3.3.1" xref="S4.p4.7.m7.1.1.3.3.1.cmml">×</mo><mn id="S4.p4.7.m7.1.1.3.3.3" xref="S4.p4.7.m7.1.1.3.3.3.cmml">1024</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.p4.7.m7.1b"><apply id="S4.p4.7.m7.1.1.cmml" xref="S4.p4.7.m7.1.1"><in id="S4.p4.7.m7.1.1.1.cmml" xref="S4.p4.7.m7.1.1.1"></in><ci id="S4.p4.7.m7.1.1.2.cmml" xref="S4.p4.7.m7.1.1.2">𝑀</ci><apply id="S4.p4.7.m7.1.1.3.cmml" xref="S4.p4.7.m7.1.1.3"><csymbol cd="ambiguous" id="S4.p4.7.m7.1.1.3.1.cmml" xref="S4.p4.7.m7.1.1.3">superscript</csymbol><ci id="S4.p4.7.m7.1.1.3.2.cmml" xref="S4.p4.7.m7.1.1.3.2">ℝ</ci><apply id="S4.p4.7.m7.1.1.3.3.cmml" xref="S4.p4.7.m7.1.1.3.3"><times id="S4.p4.7.m7.1.1.3.3.1.cmml" xref="S4.p4.7.m7.1.1.3.3.1"></times><cn type="integer" id="S4.p4.7.m7.1.1.3.3.2.cmml" xref="S4.p4.7.m7.1.1.3.3.2">1024</cn><cn type="integer" id="S4.p4.7.m7.1.1.3.3.3.cmml" xref="S4.p4.7.m7.1.1.3.3.3">1024</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p4.7.m7.1c">M\in\mathbb{R}^{1024\times 1024}</annotation></semantics></math>. In this situation, only 4 rows or columns share the same row or column.
Conversely, for smaller ranks such as <math id="S4.p4.8.m8.1" class="ltx_Math" alttext="r=8" display="inline"><semantics id="S4.p4.8.m8.1a"><mrow id="S4.p4.8.m8.1.1" xref="S4.p4.8.m8.1.1.cmml"><mi id="S4.p4.8.m8.1.1.2" xref="S4.p4.8.m8.1.1.2.cmml">r</mi><mo id="S4.p4.8.m8.1.1.1" xref="S4.p4.8.m8.1.1.1.cmml">=</mo><mn id="S4.p4.8.m8.1.1.3" xref="S4.p4.8.m8.1.1.3.cmml">8</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.p4.8.m8.1b"><apply id="S4.p4.8.m8.1.1.cmml" xref="S4.p4.8.m8.1.1"><eq id="S4.p4.8.m8.1.1.1.cmml" xref="S4.p4.8.m8.1.1.1"></eq><ci id="S4.p4.8.m8.1.1.2.cmml" xref="S4.p4.8.m8.1.1.2">𝑟</ci><cn type="integer" id="S4.p4.8.m8.1.1.3.cmml" xref="S4.p4.8.m8.1.1.3">8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p4.8.m8.1c">r=8</annotation></semantics></math>, where <math id="S4.p4.9.m9.1" class="ltx_Math" alttext="\hat{r}=256" display="inline"><semantics id="S4.p4.9.m9.1a"><mrow id="S4.p4.9.m9.1.1" xref="S4.p4.9.m9.1.1.cmml"><mover accent="true" id="S4.p4.9.m9.1.1.2" xref="S4.p4.9.m9.1.1.2.cmml"><mi id="S4.p4.9.m9.1.1.2.2" xref="S4.p4.9.m9.1.1.2.2.cmml">r</mi><mo id="S4.p4.9.m9.1.1.2.1" xref="S4.p4.9.m9.1.1.2.1.cmml">^</mo></mover><mo id="S4.p4.9.m9.1.1.1" xref="S4.p4.9.m9.1.1.1.cmml">=</mo><mn id="S4.p4.9.m9.1.1.3" xref="S4.p4.9.m9.1.1.3.cmml">256</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.p4.9.m9.1b"><apply id="S4.p4.9.m9.1.1.cmml" xref="S4.p4.9.m9.1.1"><eq id="S4.p4.9.m9.1.1.1.cmml" xref="S4.p4.9.m9.1.1.1"></eq><apply id="S4.p4.9.m9.1.1.2.cmml" xref="S4.p4.9.m9.1.1.2"><ci id="S4.p4.9.m9.1.1.2.1.cmml" xref="S4.p4.9.m9.1.1.2.1">^</ci><ci id="S4.p4.9.m9.1.1.2.2.cmml" xref="S4.p4.9.m9.1.1.2.2">𝑟</ci></apply><cn type="integer" id="S4.p4.9.m9.1.1.3.cmml" xref="S4.p4.9.m9.1.1.3">256</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p4.9.m9.1c">\hat{r}=256</annotation></semantics></math>, it requires average 16 rows or columns in a group to share the same row or column in <math id="S4.p4.10.m10.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S4.p4.10.m10.1a"><mi id="S4.p4.10.m10.1.1" xref="S4.p4.10.m10.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S4.p4.10.m10.1b"><ci id="S4.p4.10.m10.1.1.cmml" xref="S4.p4.10.m10.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p4.10.m10.1c">M</annotation></semantics></math>.
It can lead to inefficiencies due to the significant information loss during compression in Eq. <a href="#S4.E6" title="In 4 Method ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.</p>
</div>
<div id="S4.p5" class="ltx_para">
<p id="S4.p5.6" class="ltx_p">To enhance performance for smaller ranks, we reshape <math id="S4.p5.1.m1.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S4.p5.1.m1.1a"><mi id="S4.p5.1.m1.1.1" xref="S4.p5.1.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S4.p5.1.m1.1b"><ci id="S4.p5.1.m1.1.1.cmml" xref="S4.p5.1.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p5.1.m1.1c">x</annotation></semantics></math> instead of directly compressing it, to preserve the input information. In this context, <math id="S4.p5.2.m2.1" class="ltx_Math" alttext="f_{\text{comp}}\left(x\right):\mathbb{R}^{k}\rightarrow\mathbb{R}^{n\times\hat{r}}" display="inline"><semantics id="S4.p5.2.m2.1a"><mrow id="S4.p5.2.m2.1.2" xref="S4.p5.2.m2.1.2.cmml"><mrow id="S4.p5.2.m2.1.2.2" xref="S4.p5.2.m2.1.2.2.cmml"><msub id="S4.p5.2.m2.1.2.2.2" xref="S4.p5.2.m2.1.2.2.2.cmml"><mi id="S4.p5.2.m2.1.2.2.2.2" xref="S4.p5.2.m2.1.2.2.2.2.cmml">f</mi><mtext id="S4.p5.2.m2.1.2.2.2.3" xref="S4.p5.2.m2.1.2.2.2.3a.cmml">comp</mtext></msub><mo lspace="0em" rspace="0em" id="S4.p5.2.m2.1.2.2.1" xref="S4.p5.2.m2.1.2.2.1.cmml">​</mo><mrow id="S4.p5.2.m2.1.2.2.3.2" xref="S4.p5.2.m2.1.2.2.cmml"><mo id="S4.p5.2.m2.1.2.2.3.2.1" xref="S4.p5.2.m2.1.2.2.cmml">(</mo><mi id="S4.p5.2.m2.1.1" xref="S4.p5.2.m2.1.1.cmml">x</mi><mo rspace="0.278em" id="S4.p5.2.m2.1.2.2.3.2.2" xref="S4.p5.2.m2.1.2.2.cmml">)</mo></mrow></mrow><mo rspace="0.278em" id="S4.p5.2.m2.1.2.1" xref="S4.p5.2.m2.1.2.1.cmml">:</mo><mrow id="S4.p5.2.m2.1.2.3" xref="S4.p5.2.m2.1.2.3.cmml"><msup id="S4.p5.2.m2.1.2.3.2" xref="S4.p5.2.m2.1.2.3.2.cmml"><mi id="S4.p5.2.m2.1.2.3.2.2" xref="S4.p5.2.m2.1.2.3.2.2.cmml">ℝ</mi><mi id="S4.p5.2.m2.1.2.3.2.3" xref="S4.p5.2.m2.1.2.3.2.3.cmml">k</mi></msup><mo stretchy="false" id="S4.p5.2.m2.1.2.3.1" xref="S4.p5.2.m2.1.2.3.1.cmml">→</mo><msup id="S4.p5.2.m2.1.2.3.3" xref="S4.p5.2.m2.1.2.3.3.cmml"><mi id="S4.p5.2.m2.1.2.3.3.2" xref="S4.p5.2.m2.1.2.3.3.2.cmml">ℝ</mi><mrow id="S4.p5.2.m2.1.2.3.3.3" xref="S4.p5.2.m2.1.2.3.3.3.cmml"><mi id="S4.p5.2.m2.1.2.3.3.3.2" xref="S4.p5.2.m2.1.2.3.3.3.2.cmml">n</mi><mo lspace="0.222em" rspace="0.222em" id="S4.p5.2.m2.1.2.3.3.3.1" xref="S4.p5.2.m2.1.2.3.3.3.1.cmml">×</mo><mover accent="true" id="S4.p5.2.m2.1.2.3.3.3.3" xref="S4.p5.2.m2.1.2.3.3.3.3.cmml"><mi id="S4.p5.2.m2.1.2.3.3.3.3.2" xref="S4.p5.2.m2.1.2.3.3.3.3.2.cmml">r</mi><mo id="S4.p5.2.m2.1.2.3.3.3.3.1" xref="S4.p5.2.m2.1.2.3.3.3.3.1.cmml">^</mo></mover></mrow></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.p5.2.m2.1b"><apply id="S4.p5.2.m2.1.2.cmml" xref="S4.p5.2.m2.1.2"><ci id="S4.p5.2.m2.1.2.1.cmml" xref="S4.p5.2.m2.1.2.1">:</ci><apply id="S4.p5.2.m2.1.2.2.cmml" xref="S4.p5.2.m2.1.2.2"><times id="S4.p5.2.m2.1.2.2.1.cmml" xref="S4.p5.2.m2.1.2.2.1"></times><apply id="S4.p5.2.m2.1.2.2.2.cmml" xref="S4.p5.2.m2.1.2.2.2"><csymbol cd="ambiguous" id="S4.p5.2.m2.1.2.2.2.1.cmml" xref="S4.p5.2.m2.1.2.2.2">subscript</csymbol><ci id="S4.p5.2.m2.1.2.2.2.2.cmml" xref="S4.p5.2.m2.1.2.2.2.2">𝑓</ci><ci id="S4.p5.2.m2.1.2.2.2.3a.cmml" xref="S4.p5.2.m2.1.2.2.2.3"><mtext mathsize="70%" id="S4.p5.2.m2.1.2.2.2.3.cmml" xref="S4.p5.2.m2.1.2.2.2.3">comp</mtext></ci></apply><ci id="S4.p5.2.m2.1.1.cmml" xref="S4.p5.2.m2.1.1">𝑥</ci></apply><apply id="S4.p5.2.m2.1.2.3.cmml" xref="S4.p5.2.m2.1.2.3"><ci id="S4.p5.2.m2.1.2.3.1.cmml" xref="S4.p5.2.m2.1.2.3.1">→</ci><apply id="S4.p5.2.m2.1.2.3.2.cmml" xref="S4.p5.2.m2.1.2.3.2"><csymbol cd="ambiguous" id="S4.p5.2.m2.1.2.3.2.1.cmml" xref="S4.p5.2.m2.1.2.3.2">superscript</csymbol><ci id="S4.p5.2.m2.1.2.3.2.2.cmml" xref="S4.p5.2.m2.1.2.3.2.2">ℝ</ci><ci id="S4.p5.2.m2.1.2.3.2.3.cmml" xref="S4.p5.2.m2.1.2.3.2.3">𝑘</ci></apply><apply id="S4.p5.2.m2.1.2.3.3.cmml" xref="S4.p5.2.m2.1.2.3.3"><csymbol cd="ambiguous" id="S4.p5.2.m2.1.2.3.3.1.cmml" xref="S4.p5.2.m2.1.2.3.3">superscript</csymbol><ci id="S4.p5.2.m2.1.2.3.3.2.cmml" xref="S4.p5.2.m2.1.2.3.3.2">ℝ</ci><apply id="S4.p5.2.m2.1.2.3.3.3.cmml" xref="S4.p5.2.m2.1.2.3.3.3"><times id="S4.p5.2.m2.1.2.3.3.3.1.cmml" xref="S4.p5.2.m2.1.2.3.3.3.1"></times><ci id="S4.p5.2.m2.1.2.3.3.3.2.cmml" xref="S4.p5.2.m2.1.2.3.3.3.2">𝑛</ci><apply id="S4.p5.2.m2.1.2.3.3.3.3.cmml" xref="S4.p5.2.m2.1.2.3.3.3.3"><ci id="S4.p5.2.m2.1.2.3.3.3.3.1.cmml" xref="S4.p5.2.m2.1.2.3.3.3.3.1">^</ci><ci id="S4.p5.2.m2.1.2.3.3.3.3.2.cmml" xref="S4.p5.2.m2.1.2.3.3.3.3.2">𝑟</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p5.2.m2.1c">f_{\text{comp}}\left(x\right):\mathbb{R}^{k}\rightarrow\mathbb{R}^{n\times\hat{r}}</annotation></semantics></math> and <math id="S4.p5.3.m3.1" class="ltx_Math" alttext="f_{\text{decomp}}:\mathbb{R}^{n\times\hat{r}}\rightarrow\mathbb{R}^{d}" display="inline"><semantics id="S4.p5.3.m3.1a"><mrow id="S4.p5.3.m3.1.1" xref="S4.p5.3.m3.1.1.cmml"><msub id="S4.p5.3.m3.1.1.2" xref="S4.p5.3.m3.1.1.2.cmml"><mi id="S4.p5.3.m3.1.1.2.2" xref="S4.p5.3.m3.1.1.2.2.cmml">f</mi><mtext id="S4.p5.3.m3.1.1.2.3" xref="S4.p5.3.m3.1.1.2.3a.cmml">decomp</mtext></msub><mo lspace="0.278em" rspace="0.278em" id="S4.p5.3.m3.1.1.1" xref="S4.p5.3.m3.1.1.1.cmml">:</mo><mrow id="S4.p5.3.m3.1.1.3" xref="S4.p5.3.m3.1.1.3.cmml"><msup id="S4.p5.3.m3.1.1.3.2" xref="S4.p5.3.m3.1.1.3.2.cmml"><mi id="S4.p5.3.m3.1.1.3.2.2" xref="S4.p5.3.m3.1.1.3.2.2.cmml">ℝ</mi><mrow id="S4.p5.3.m3.1.1.3.2.3" xref="S4.p5.3.m3.1.1.3.2.3.cmml"><mi id="S4.p5.3.m3.1.1.3.2.3.2" xref="S4.p5.3.m3.1.1.3.2.3.2.cmml">n</mi><mo lspace="0.222em" rspace="0.222em" id="S4.p5.3.m3.1.1.3.2.3.1" xref="S4.p5.3.m3.1.1.3.2.3.1.cmml">×</mo><mover accent="true" id="S4.p5.3.m3.1.1.3.2.3.3" xref="S4.p5.3.m3.1.1.3.2.3.3.cmml"><mi id="S4.p5.3.m3.1.1.3.2.3.3.2" xref="S4.p5.3.m3.1.1.3.2.3.3.2.cmml">r</mi><mo id="S4.p5.3.m3.1.1.3.2.3.3.1" xref="S4.p5.3.m3.1.1.3.2.3.3.1.cmml">^</mo></mover></mrow></msup><mo stretchy="false" id="S4.p5.3.m3.1.1.3.1" xref="S4.p5.3.m3.1.1.3.1.cmml">→</mo><msup id="S4.p5.3.m3.1.1.3.3" xref="S4.p5.3.m3.1.1.3.3.cmml"><mi id="S4.p5.3.m3.1.1.3.3.2" xref="S4.p5.3.m3.1.1.3.3.2.cmml">ℝ</mi><mi id="S4.p5.3.m3.1.1.3.3.3" xref="S4.p5.3.m3.1.1.3.3.3.cmml">d</mi></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.p5.3.m3.1b"><apply id="S4.p5.3.m3.1.1.cmml" xref="S4.p5.3.m3.1.1"><ci id="S4.p5.3.m3.1.1.1.cmml" xref="S4.p5.3.m3.1.1.1">:</ci><apply id="S4.p5.3.m3.1.1.2.cmml" xref="S4.p5.3.m3.1.1.2"><csymbol cd="ambiguous" id="S4.p5.3.m3.1.1.2.1.cmml" xref="S4.p5.3.m3.1.1.2">subscript</csymbol><ci id="S4.p5.3.m3.1.1.2.2.cmml" xref="S4.p5.3.m3.1.1.2.2">𝑓</ci><ci id="S4.p5.3.m3.1.1.2.3a.cmml" xref="S4.p5.3.m3.1.1.2.3"><mtext mathsize="70%" id="S4.p5.3.m3.1.1.2.3.cmml" xref="S4.p5.3.m3.1.1.2.3">decomp</mtext></ci></apply><apply id="S4.p5.3.m3.1.1.3.cmml" xref="S4.p5.3.m3.1.1.3"><ci id="S4.p5.3.m3.1.1.3.1.cmml" xref="S4.p5.3.m3.1.1.3.1">→</ci><apply id="S4.p5.3.m3.1.1.3.2.cmml" xref="S4.p5.3.m3.1.1.3.2"><csymbol cd="ambiguous" id="S4.p5.3.m3.1.1.3.2.1.cmml" xref="S4.p5.3.m3.1.1.3.2">superscript</csymbol><ci id="S4.p5.3.m3.1.1.3.2.2.cmml" xref="S4.p5.3.m3.1.1.3.2.2">ℝ</ci><apply id="S4.p5.3.m3.1.1.3.2.3.cmml" xref="S4.p5.3.m3.1.1.3.2.3"><times id="S4.p5.3.m3.1.1.3.2.3.1.cmml" xref="S4.p5.3.m3.1.1.3.2.3.1"></times><ci id="S4.p5.3.m3.1.1.3.2.3.2.cmml" xref="S4.p5.3.m3.1.1.3.2.3.2">𝑛</ci><apply id="S4.p5.3.m3.1.1.3.2.3.3.cmml" xref="S4.p5.3.m3.1.1.3.2.3.3"><ci id="S4.p5.3.m3.1.1.3.2.3.3.1.cmml" xref="S4.p5.3.m3.1.1.3.2.3.3.1">^</ci><ci id="S4.p5.3.m3.1.1.3.2.3.3.2.cmml" xref="S4.p5.3.m3.1.1.3.2.3.3.2">𝑟</ci></apply></apply></apply><apply id="S4.p5.3.m3.1.1.3.3.cmml" xref="S4.p5.3.m3.1.1.3.3"><csymbol cd="ambiguous" id="S4.p5.3.m3.1.1.3.3.1.cmml" xref="S4.p5.3.m3.1.1.3.3">superscript</csymbol><ci id="S4.p5.3.m3.1.1.3.3.2.cmml" xref="S4.p5.3.m3.1.1.3.3.2">ℝ</ci><ci id="S4.p5.3.m3.1.1.3.3.3.cmml" xref="S4.p5.3.m3.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p5.3.m3.1c">f_{\text{decomp}}:\mathbb{R}^{n\times\hat{r}}\rightarrow\mathbb{R}^{d}</annotation></semantics></math>.
Corresponding <math id="S4.p5.4.m4.1" class="ltx_Math" alttext="f_{\text{comp}}" display="inline"><semantics id="S4.p5.4.m4.1a"><msub id="S4.p5.4.m4.1.1" xref="S4.p5.4.m4.1.1.cmml"><mi id="S4.p5.4.m4.1.1.2" xref="S4.p5.4.m4.1.1.2.cmml">f</mi><mtext id="S4.p5.4.m4.1.1.3" xref="S4.p5.4.m4.1.1.3a.cmml">comp</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.p5.4.m4.1b"><apply id="S4.p5.4.m4.1.1.cmml" xref="S4.p5.4.m4.1.1"><csymbol cd="ambiguous" id="S4.p5.4.m4.1.1.1.cmml" xref="S4.p5.4.m4.1.1">subscript</csymbol><ci id="S4.p5.4.m4.1.1.2.cmml" xref="S4.p5.4.m4.1.1.2">𝑓</ci><ci id="S4.p5.4.m4.1.1.3a.cmml" xref="S4.p5.4.m4.1.1.3"><mtext mathsize="70%" id="S4.p5.4.m4.1.1.3.cmml" xref="S4.p5.4.m4.1.1.3">comp</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p5.4.m4.1c">f_{\text{comp}}</annotation></semantics></math>, <math id="S4.p5.5.m5.1" class="ltx_Math" alttext="f_{\text{decomp}}" display="inline"><semantics id="S4.p5.5.m5.1a"><msub id="S4.p5.5.m5.1.1" xref="S4.p5.5.m5.1.1.cmml"><mi id="S4.p5.5.m5.1.1.2" xref="S4.p5.5.m5.1.1.2.cmml">f</mi><mtext id="S4.p5.5.m5.1.1.3" xref="S4.p5.5.m5.1.1.3a.cmml">decomp</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.p5.5.m5.1b"><apply id="S4.p5.5.m5.1.1.cmml" xref="S4.p5.5.m5.1.1"><csymbol cd="ambiguous" id="S4.p5.5.m5.1.1.1.cmml" xref="S4.p5.5.m5.1.1">subscript</csymbol><ci id="S4.p5.5.m5.1.1.2.cmml" xref="S4.p5.5.m5.1.1.2">𝑓</ci><ci id="S4.p5.5.m5.1.1.3a.cmml" xref="S4.p5.5.m5.1.1.3"><mtext mathsize="70%" id="S4.p5.5.m5.1.1.3.cmml" xref="S4.p5.5.m5.1.1.3">decomp</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p5.5.m5.1c">f_{\text{decomp}}</annotation></semantics></math> and <math id="S4.p5.6.m6.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S4.p5.6.m6.1a"><mrow id="S4.p5.6.m6.1.1" xref="S4.p5.6.m6.1.1.cmml"><mi mathvariant="normal" id="S4.p5.6.m6.1.1.2" xref="S4.p5.6.m6.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.p5.6.m6.1.1.1" xref="S4.p5.6.m6.1.1.1.cmml">​</mo><mi id="S4.p5.6.m6.1.1.3" xref="S4.p5.6.m6.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p5.6.m6.1b"><apply id="S4.p5.6.m6.1.1.cmml" xref="S4.p5.6.m6.1.1"><times id="S4.p5.6.m6.1.1.1.cmml" xref="S4.p5.6.m6.1.1.1"></times><ci id="S4.p5.6.m6.1.1.2.cmml" xref="S4.p5.6.m6.1.1.2">Δ</ci><ci id="S4.p5.6.m6.1.1.3.cmml" xref="S4.p5.6.m6.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p5.6.m6.1c">\Delta W</annotation></semantics></math> are as follows:</p>
<table id="S4.E8" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E8.m1.21" class="ltx_Math" alttext="\begin{split}f_{\text{comp}}\left(x\right)&amp;=\begin{bmatrix}x_{1:\hat{r}}&amp;x_{\hat{r}:2\hat{r}}&amp;\cdots&amp;x_{(n-1)\hat{r}:n\hat{r}}\end{bmatrix}\\
f_{\text{decomp}}\left(x\right)&amp;=\text{concat}(x)\\
\Delta W&amp;=\begin{bmatrix}M&amp;\mathbf{0}&amp;\cdots&amp;\mathbf{0}\\
\mathbf{0}&amp;M&amp;\cdots&amp;\mathbf{0}\\
\vdots&amp;\vdots&amp;\ddots&amp;\vdots\\
\mathbf{0}&amp;\mathbf{0}&amp;\cdots&amp;M\end{bmatrix}\end{split}" display="block"><semantics id="S4.E8.m1.21a"><mtable columnspacing="0pt" displaystyle="true" rowspacing="0pt" id="S4.E8.m1.21.21" xref="S4.E8.m1.21.22.1.cmml"><mtr id="S4.E8.m1.21.21a" xref="S4.E8.m1.21.22.1.cmml"><mtd class="ltx_align_right" columnalign="right" id="S4.E8.m1.21.21b" xref="S4.E8.m1.21.22.1.cmml"><mrow id="S4.E8.m1.7.7.7.6.5" xref="S4.E8.m1.21.22.1.cmml"><msub id="S4.E8.m1.7.7.7.6.5.7" xref="S4.E8.m1.21.22.1.cmml"><mi id="S4.E8.m1.3.3.3.2.1.1" xref="S4.E8.m1.3.3.3.2.1.1.cmml">f</mi><mtext id="S4.E8.m1.4.4.4.3.2.2.1" xref="S4.E8.m1.4.4.4.3.2.2.1a.cmml">comp</mtext></msub><mo lspace="0em" rspace="0em" id="S4.E8.m1.7.7.7.6.5.6" xref="S4.E8.m1.21.22.1a.cmml">​</mo><mrow id="S4.E8.m1.7.7.7.6.5.8" xref="S4.E8.m1.21.22.1.cmml"><mo id="S4.E8.m1.5.5.5.4.3.3" xref="S4.E8.m1.21.22.1a.cmml">(</mo><mi id="S4.E8.m1.6.6.6.5.4.4" xref="S4.E8.m1.6.6.6.5.4.4.cmml">x</mi><mo id="S4.E8.m1.7.7.7.6.5.5" xref="S4.E8.m1.21.22.1a.cmml">)</mo></mrow></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S4.E8.m1.21.21c" xref="S4.E8.m1.21.22.1.cmml"><mrow id="S4.E8.m1.8.8.8.7.2" xref="S4.E8.m1.21.22.1.cmml"><mi id="S4.E8.m1.8.8.8.7.2.3" xref="S4.E8.m1.21.22.1a.cmml"></mi><mo id="S4.E8.m1.8.8.8.7.2.2" xref="S4.E8.m1.8.8.8.7.2.2.cmml">=</mo><mrow id="S4.E8.m1.1.1.1.1.1.1.3" xref="S4.E8.m1.1.1.1.1.1.1.2.cmml"><mo id="S4.E8.m1.1.1.1.1.1.1.3.1" xref="S4.E8.m1.1.1.1.1.1.1.2.1.cmml">[</mo><mtable columnspacing="5pt" displaystyle="true" id="S4.E8.m1.1.1.1.1.1.1.1.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.cmml"><mtr id="S4.E8.m1.1.1.1.1.1.1.1.1a" xref="S4.E8.m1.1.1.1.1.1.1.1.1.cmml"><mtd id="S4.E8.m1.1.1.1.1.1.1.1.1b" xref="S4.E8.m1.1.1.1.1.1.1.1.1.cmml"><msub id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.cmml"><mi id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.2" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.2.cmml">x</mi><mrow id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.cmml"><mn id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.2" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.2.cmml">1</mn><mo lspace="0.278em" rspace="0.278em" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.1.cmml">:</mo><mover accent="true" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.3" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.3.cmml"><mi id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.3.2" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.3.2.cmml">r</mi><mo id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.3.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.3.1.cmml">^</mo></mover></mrow></msub></mtd><mtd id="S4.E8.m1.1.1.1.1.1.1.1.1c" xref="S4.E8.m1.1.1.1.1.1.1.1.1.cmml"><msub id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.cmml"><mi id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.2" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.2.cmml">x</mi><mrow id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.cmml"><mover accent="true" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.2" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.2.cmml"><mi id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.2.2" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.2.2.cmml">r</mi><mo id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.2.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.2.1.cmml">^</mo></mover><mo lspace="0.278em" rspace="0.278em" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.1.cmml">:</mo><mrow id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.cmml"><mn id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.2" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.1.cmml">​</mo><mover accent="true" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.3" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.3.cmml"><mi id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.3.2" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.3.2.cmml">r</mi><mo id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.3.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.3.1.cmml">^</mo></mover></mrow></mrow></msub></mtd><mtd id="S4.E8.m1.1.1.1.1.1.1.1.1d" xref="S4.E8.m1.1.1.1.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.4.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.4.1.cmml">⋯</mi></mtd><mtd id="S4.E8.m1.1.1.1.1.1.1.1.1e" xref="S4.E8.m1.1.1.1.1.1.1.1.1.cmml"><msub id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">x</mi><mrow id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">n</mi><mo id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">−</mo><mn id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">​</mo><mover accent="true" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml">r</mi><mo id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml">^</mo></mover></mrow><mo lspace="0.278em" rspace="0.278em" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">:</mo><mrow id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mover accent="true" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml"><mi id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.2" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.2.cmml">r</mi><mo id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.1" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.1.cmml">^</mo></mover></mrow></mrow></msub></mtd></mtr></mtable><mo id="S4.E8.m1.1.1.1.1.1.1.3.2" xref="S4.E8.m1.1.1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow></mtd></mtr><mtr id="S4.E8.m1.21.21d" xref="S4.E8.m1.21.22.1.cmml"><mtd class="ltx_align_right" columnalign="right" id="S4.E8.m1.21.21e" xref="S4.E8.m1.21.22.1.cmml"><mrow id="S4.E8.m1.13.13.13.5.5" xref="S4.E8.m1.21.22.1.cmml"><msub id="S4.E8.m1.13.13.13.5.5.7" xref="S4.E8.m1.21.22.1.cmml"><mi id="S4.E8.m1.9.9.9.1.1.1" xref="S4.E8.m1.9.9.9.1.1.1.cmml">f</mi><mtext id="S4.E8.m1.10.10.10.2.2.2.1" xref="S4.E8.m1.10.10.10.2.2.2.1a.cmml">decomp</mtext></msub><mo lspace="0em" rspace="0em" id="S4.E8.m1.13.13.13.5.5.6" xref="S4.E8.m1.21.22.1a.cmml">​</mo><mrow id="S4.E8.m1.13.13.13.5.5.8" xref="S4.E8.m1.21.22.1.cmml"><mo id="S4.E8.m1.11.11.11.3.3.3" xref="S4.E8.m1.21.22.1a.cmml">(</mo><mi id="S4.E8.m1.12.12.12.4.4.4" xref="S4.E8.m1.12.12.12.4.4.4.cmml">x</mi><mo id="S4.E8.m1.13.13.13.5.5.5" xref="S4.E8.m1.21.22.1a.cmml">)</mo></mrow></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S4.E8.m1.21.21f" xref="S4.E8.m1.21.22.1.cmml"><mrow id="S4.E8.m1.18.18.18.10.5" xref="S4.E8.m1.21.22.1.cmml"><mi id="S4.E8.m1.18.18.18.10.5.6" xref="S4.E8.m1.21.22.1a.cmml"></mi><mo id="S4.E8.m1.14.14.14.6.1.1" xref="S4.E8.m1.14.14.14.6.1.1.cmml">=</mo><mrow id="S4.E8.m1.18.18.18.10.5.7" xref="S4.E8.m1.21.22.1.cmml"><mtext id="S4.E8.m1.15.15.15.7.2.2" xref="S4.E8.m1.15.15.15.7.2.2a.cmml">concat</mtext><mo lspace="0em" rspace="0em" id="S4.E8.m1.18.18.18.10.5.7.1" xref="S4.E8.m1.21.22.1a.cmml">​</mo><mrow id="S4.E8.m1.18.18.18.10.5.7.2" xref="S4.E8.m1.21.22.1.cmml"><mo stretchy="false" id="S4.E8.m1.16.16.16.8.3.3" xref="S4.E8.m1.21.22.1a.cmml">(</mo><mi id="S4.E8.m1.17.17.17.9.4.4" xref="S4.E8.m1.17.17.17.9.4.4.cmml">x</mi><mo stretchy="false" id="S4.E8.m1.18.18.18.10.5.5" xref="S4.E8.m1.21.22.1a.cmml">)</mo></mrow></mrow></mrow></mtd></mtr><mtr id="S4.E8.m1.21.21g" xref="S4.E8.m1.21.22.1.cmml"><mtd class="ltx_align_right" columnalign="right" id="S4.E8.m1.21.21h" xref="S4.E8.m1.21.22.1.cmml"><mrow id="S4.E8.m1.20.20.20.3.2" xref="S4.E8.m1.21.22.1.cmml"><mi mathvariant="normal" id="S4.E8.m1.19.19.19.2.1.1" xref="S4.E8.m1.19.19.19.2.1.1.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.E8.m1.20.20.20.3.2.3" xref="S4.E8.m1.21.22.1a.cmml">​</mo><mi id="S4.E8.m1.20.20.20.3.2.2" xref="S4.E8.m1.20.20.20.3.2.2.cmml">W</mi></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S4.E8.m1.21.21i" xref="S4.E8.m1.21.22.1.cmml"><mrow id="S4.E8.m1.21.21.21.4.2" xref="S4.E8.m1.21.22.1.cmml"><mi id="S4.E8.m1.21.21.21.4.2.3" xref="S4.E8.m1.21.22.1a.cmml"></mi><mo id="S4.E8.m1.21.21.21.4.2.2" xref="S4.E8.m1.21.21.21.4.2.2.cmml">=</mo><mrow id="S4.E8.m1.2.2.2.1.1.1.3" xref="S4.E8.m1.2.2.2.1.1.1.2.cmml"><mo id="S4.E8.m1.2.2.2.1.1.1.3.1" xref="S4.E8.m1.2.2.2.1.1.1.2.1.cmml">[</mo><mtable columnspacing="5pt" displaystyle="true" rowspacing="0pt" id="S4.E8.m1.2.2.2.1.1.1.1.1" xref="S4.E8.m1.2.2.2.1.1.1.1.1.cmml"><mtr id="S4.E8.m1.2.2.2.1.1.1.1.1a" xref="S4.E8.m1.2.2.2.1.1.1.1.1.cmml"><mtd id="S4.E8.m1.2.2.2.1.1.1.1.1b" xref="S4.E8.m1.2.2.2.1.1.1.1.1.cmml"><mi id="S4.E8.m1.2.2.2.1.1.1.1.1.1.1.1" xref="S4.E8.m1.2.2.2.1.1.1.1.1.1.1.1.cmml">M</mi></mtd><mtd id="S4.E8.m1.2.2.2.1.1.1.1.1c" xref="S4.E8.m1.2.2.2.1.1.1.1.1.cmml"><mn id="S4.E8.m1.2.2.2.1.1.1.1.1.1.2.1" xref="S4.E8.m1.2.2.2.1.1.1.1.1.1.2.1.cmml">𝟎</mn></mtd><mtd id="S4.E8.m1.2.2.2.1.1.1.1.1d" xref="S4.E8.m1.2.2.2.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E8.m1.2.2.2.1.1.1.1.1.1.3.1" xref="S4.E8.m1.2.2.2.1.1.1.1.1.1.3.1.cmml">⋯</mi></mtd><mtd id="S4.E8.m1.2.2.2.1.1.1.1.1e" xref="S4.E8.m1.2.2.2.1.1.1.1.1.cmml"><mn id="S4.E8.m1.2.2.2.1.1.1.1.1.1.4.1" xref="S4.E8.m1.2.2.2.1.1.1.1.1.1.4.1.cmml">𝟎</mn></mtd></mtr><mtr id="S4.E8.m1.2.2.2.1.1.1.1.1f" xref="S4.E8.m1.2.2.2.1.1.1.1.1.cmml"><mtd id="S4.E8.m1.2.2.2.1.1.1.1.1g" xref="S4.E8.m1.2.2.2.1.1.1.1.1.cmml"><mn id="S4.E8.m1.2.2.2.1.1.1.1.1.2.1.1" xref="S4.E8.m1.2.2.2.1.1.1.1.1.2.1.1.cmml">𝟎</mn></mtd><mtd id="S4.E8.m1.2.2.2.1.1.1.1.1h" xref="S4.E8.m1.2.2.2.1.1.1.1.1.cmml"><mi id="S4.E8.m1.2.2.2.1.1.1.1.1.2.2.1" xref="S4.E8.m1.2.2.2.1.1.1.1.1.2.2.1.cmml">M</mi></mtd><mtd id="S4.E8.m1.2.2.2.1.1.1.1.1i" xref="S4.E8.m1.2.2.2.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E8.m1.2.2.2.1.1.1.1.1.2.3.1" xref="S4.E8.m1.2.2.2.1.1.1.1.1.2.3.1.cmml">⋯</mi></mtd><mtd id="S4.E8.m1.2.2.2.1.1.1.1.1j" xref="S4.E8.m1.2.2.2.1.1.1.1.1.cmml"><mn id="S4.E8.m1.2.2.2.1.1.1.1.1.2.4.1" xref="S4.E8.m1.2.2.2.1.1.1.1.1.2.4.1.cmml">𝟎</mn></mtd></mtr><mtr id="S4.E8.m1.2.2.2.1.1.1.1.1k" xref="S4.E8.m1.2.2.2.1.1.1.1.1.cmml"><mtd id="S4.E8.m1.2.2.2.1.1.1.1.1l" xref="S4.E8.m1.2.2.2.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E8.m1.2.2.2.1.1.1.1.1.3.1.1" xref="S4.E8.m1.2.2.2.1.1.1.1.1.3.1.1.cmml">⋮</mi></mtd><mtd id="S4.E8.m1.2.2.2.1.1.1.1.1m" xref="S4.E8.m1.2.2.2.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E8.m1.2.2.2.1.1.1.1.1.3.2.1" xref="S4.E8.m1.2.2.2.1.1.1.1.1.3.2.1.cmml">⋮</mi></mtd><mtd id="S4.E8.m1.2.2.2.1.1.1.1.1n" xref="S4.E8.m1.2.2.2.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E8.m1.2.2.2.1.1.1.1.1.3.3.1" xref="S4.E8.m1.2.2.2.1.1.1.1.1.3.3.1.cmml">⋱</mi></mtd><mtd id="S4.E8.m1.2.2.2.1.1.1.1.1o" xref="S4.E8.m1.2.2.2.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E8.m1.2.2.2.1.1.1.1.1.3.4.1" xref="S4.E8.m1.2.2.2.1.1.1.1.1.3.4.1.cmml">⋮</mi></mtd></mtr><mtr id="S4.E8.m1.2.2.2.1.1.1.1.1p" xref="S4.E8.m1.2.2.2.1.1.1.1.1.cmml"><mtd id="S4.E8.m1.2.2.2.1.1.1.1.1q" xref="S4.E8.m1.2.2.2.1.1.1.1.1.cmml"><mn id="S4.E8.m1.2.2.2.1.1.1.1.1.4.1.1" xref="S4.E8.m1.2.2.2.1.1.1.1.1.4.1.1.cmml">𝟎</mn></mtd><mtd id="S4.E8.m1.2.2.2.1.1.1.1.1r" xref="S4.E8.m1.2.2.2.1.1.1.1.1.cmml"><mn id="S4.E8.m1.2.2.2.1.1.1.1.1.4.2.1" xref="S4.E8.m1.2.2.2.1.1.1.1.1.4.2.1.cmml">𝟎</mn></mtd><mtd id="S4.E8.m1.2.2.2.1.1.1.1.1s" xref="S4.E8.m1.2.2.2.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E8.m1.2.2.2.1.1.1.1.1.4.3.1" xref="S4.E8.m1.2.2.2.1.1.1.1.1.4.3.1.cmml">⋯</mi></mtd><mtd id="S4.E8.m1.2.2.2.1.1.1.1.1t" xref="S4.E8.m1.2.2.2.1.1.1.1.1.cmml"><mi id="S4.E8.m1.2.2.2.1.1.1.1.1.4.4.1" xref="S4.E8.m1.2.2.2.1.1.1.1.1.4.4.1.cmml">M</mi></mtd></mtr></mtable><mo id="S4.E8.m1.2.2.2.1.1.1.3.2" xref="S4.E8.m1.2.2.2.1.1.1.2.1.cmml">]</mo></mrow></mrow></mtd></mtr></mtable><annotation-xml encoding="MathML-Content" id="S4.E8.m1.21b"><apply id="S4.E8.m1.21.22.1.cmml" xref="S4.E8.m1.21.21"><and id="S4.E8.m1.21.22.1a.cmml" xref="S4.E8.m1.7.7.7.6.5.6"></and><apply id="S4.E8.m1.21.22.1b.cmml" xref="S4.E8.m1.21.21"><eq id="S4.E8.m1.8.8.8.7.2.2.cmml" xref="S4.E8.m1.8.8.8.7.2.2"></eq><apply id="S4.E8.m1.21.22.1.2.cmml" xref="S4.E8.m1.21.21"><times id="S4.E8.m1.21.22.1.2.1.cmml" xref="S4.E8.m1.7.7.7.6.5.6"></times><apply id="S4.E8.m1.21.22.1.2.2.cmml" xref="S4.E8.m1.21.21"><csymbol cd="ambiguous" id="S4.E8.m1.21.22.1.2.2.1.cmml" xref="S4.E8.m1.7.7.7.6.5.6">subscript</csymbol><ci id="S4.E8.m1.3.3.3.2.1.1.cmml" xref="S4.E8.m1.3.3.3.2.1.1">𝑓</ci><ci id="S4.E8.m1.4.4.4.3.2.2.1a.cmml" xref="S4.E8.m1.4.4.4.3.2.2.1"><mtext mathsize="70%" id="S4.E8.m1.4.4.4.3.2.2.1.cmml" xref="S4.E8.m1.4.4.4.3.2.2.1">comp</mtext></ci></apply><ci id="S4.E8.m1.6.6.6.5.4.4.cmml" xref="S4.E8.m1.6.6.6.5.4.4">𝑥</ci></apply><apply id="S4.E8.m1.21.22.1.4.cmml" xref="S4.E8.m1.21.21"><times id="S4.E8.m1.21.22.1.4.1.cmml" xref="S4.E8.m1.7.7.7.6.5.6"></times><apply id="S4.E8.m1.1.1.1.1.1.1.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1.3"><csymbol cd="latexml" id="S4.E8.m1.1.1.1.1.1.1.2.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.3.1">matrix</csymbol><matrix id="S4.E8.m1.1.1.1.1.1.1.1.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1"><matrixrow id="S4.E8.m1.1.1.1.1.1.1.1.1a.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1"><apply id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1"><csymbol cd="ambiguous" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1">subscript</csymbol><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.2">𝑥</ci><apply id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3"><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.1">:</ci><cn type="integer" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.2">1</cn><apply id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.3.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.3"><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.3.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.3.1">^</ci><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.3.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.2.1.3.3.2">𝑟</ci></apply></apply></apply><apply id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1"><csymbol cd="ambiguous" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1">subscript</csymbol><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.2">𝑥</ci><apply id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3"><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.1">:</ci><apply id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.2"><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.2.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.2.1">^</ci><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.2.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.2.2">𝑟</ci></apply><apply id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3"><times id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.1"></times><cn type="integer" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.2">2</cn><apply id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.3.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.3"><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.3.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.3.1">^</ci><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.3.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.3.1.3.3.3.2">𝑟</ci></apply></apply></apply></apply><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.4.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.4.1">⋯</ci><apply id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.3">𝑥</ci><apply id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1"><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2">:</ci><apply id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1"><times id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2"></times><apply id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1"><minus id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1"></minus><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2">𝑛</ci><cn type="integer" id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3">1</cn></apply><apply id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3"><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1">^</ci><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2">𝑟</ci></apply></apply><apply id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3"><times id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.1"></times><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.2">𝑛</ci><apply id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3"><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.1.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.1">^</ci><ci id="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.2.cmml" xref="S4.E8.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.2">𝑟</ci></apply></apply></apply></apply></matrixrow></matrix></apply><apply id="S4.E8.m1.21.22.1.4.3.cmml" xref="S4.E8.m1.21.21"><csymbol cd="ambiguous" id="S4.E8.m1.21.22.1.4.3.1.cmml" xref="S4.E8.m1.7.7.7.6.5.6">subscript</csymbol><ci id="S4.E8.m1.9.9.9.1.1.1.cmml" xref="S4.E8.m1.9.9.9.1.1.1">𝑓</ci><ci id="S4.E8.m1.10.10.10.2.2.2.1a.cmml" xref="S4.E8.m1.10.10.10.2.2.2.1"><mtext mathsize="70%" id="S4.E8.m1.10.10.10.2.2.2.1.cmml" xref="S4.E8.m1.10.10.10.2.2.2.1">decomp</mtext></ci></apply><ci id="S4.E8.m1.12.12.12.4.4.4.cmml" xref="S4.E8.m1.12.12.12.4.4.4">𝑥</ci></apply></apply><apply id="S4.E8.m1.21.22.1c.cmml" xref="S4.E8.m1.21.21"><eq id="S4.E8.m1.14.14.14.6.1.1.cmml" xref="S4.E8.m1.14.14.14.6.1.1"></eq><share href="#S4.E8.m1.21.22.1.4.cmml" id="S4.E8.m1.21.22.1d.cmml" xref="S4.E8.m1.7.7.7.6.5.6"></share><apply id="S4.E8.m1.21.22.1.6.cmml" xref="S4.E8.m1.21.21"><times id="S4.E8.m1.21.22.1.6.1.cmml" xref="S4.E8.m1.7.7.7.6.5.6"></times><ci id="S4.E8.m1.15.15.15.7.2.2a.cmml" xref="S4.E8.m1.15.15.15.7.2.2"><mtext id="S4.E8.m1.15.15.15.7.2.2.cmml" xref="S4.E8.m1.15.15.15.7.2.2">concat</mtext></ci><ci id="S4.E8.m1.17.17.17.9.4.4.cmml" xref="S4.E8.m1.17.17.17.9.4.4">𝑥</ci><ci id="S4.E8.m1.19.19.19.2.1.1.cmml" xref="S4.E8.m1.19.19.19.2.1.1">Δ</ci><ci id="S4.E8.m1.20.20.20.3.2.2.cmml" xref="S4.E8.m1.20.20.20.3.2.2">𝑊</ci></apply></apply><apply id="S4.E8.m1.21.22.1e.cmml" xref="S4.E8.m1.21.21"><eq id="S4.E8.m1.21.21.21.4.2.2.cmml" xref="S4.E8.m1.21.21.21.4.2.2"></eq><share href="#S4.E8.m1.21.22.1.6.cmml" id="S4.E8.m1.21.22.1f.cmml" xref="S4.E8.m1.7.7.7.6.5.6"></share><apply id="S4.E8.m1.2.2.2.1.1.1.2.cmml" xref="S4.E8.m1.2.2.2.1.1.1.3"><csymbol cd="latexml" id="S4.E8.m1.2.2.2.1.1.1.2.1.cmml" xref="S4.E8.m1.2.2.2.1.1.1.3.1">matrix</csymbol><matrix id="S4.E8.m1.2.2.2.1.1.1.1.1.cmml" xref="S4.E8.m1.2.2.2.1.1.1.1.1"><matrixrow id="S4.E8.m1.2.2.2.1.1.1.1.1a.cmml" xref="S4.E8.m1.2.2.2.1.1.1.1.1"><ci id="S4.E8.m1.2.2.2.1.1.1.1.1.1.1.1.cmml" xref="S4.E8.m1.2.2.2.1.1.1.1.1.1.1.1">𝑀</ci><cn type="integer" id="S4.E8.m1.2.2.2.1.1.1.1.1.1.2.1.cmml" xref="S4.E8.m1.2.2.2.1.1.1.1.1.1.2.1">0</cn><ci id="S4.E8.m1.2.2.2.1.1.1.1.1.1.3.1.cmml" xref="S4.E8.m1.2.2.2.1.1.1.1.1.1.3.1">⋯</ci><cn type="integer" id="S4.E8.m1.2.2.2.1.1.1.1.1.1.4.1.cmml" xref="S4.E8.m1.2.2.2.1.1.1.1.1.1.4.1">0</cn></matrixrow><matrixrow id="S4.E8.m1.2.2.2.1.1.1.1.1b.cmml" xref="S4.E8.m1.2.2.2.1.1.1.1.1"><cn type="integer" id="S4.E8.m1.2.2.2.1.1.1.1.1.2.1.1.cmml" xref="S4.E8.m1.2.2.2.1.1.1.1.1.2.1.1">0</cn><ci id="S4.E8.m1.2.2.2.1.1.1.1.1.2.2.1.cmml" xref="S4.E8.m1.2.2.2.1.1.1.1.1.2.2.1">𝑀</ci><ci id="S4.E8.m1.2.2.2.1.1.1.1.1.2.3.1.cmml" xref="S4.E8.m1.2.2.2.1.1.1.1.1.2.3.1">⋯</ci><cn type="integer" id="S4.E8.m1.2.2.2.1.1.1.1.1.2.4.1.cmml" xref="S4.E8.m1.2.2.2.1.1.1.1.1.2.4.1">0</cn></matrixrow><matrixrow id="S4.E8.m1.2.2.2.1.1.1.1.1c.cmml" xref="S4.E8.m1.2.2.2.1.1.1.1.1"><ci id="S4.E8.m1.2.2.2.1.1.1.1.1.3.1.1.cmml" xref="S4.E8.m1.2.2.2.1.1.1.1.1.3.1.1">⋮</ci><ci id="S4.E8.m1.2.2.2.1.1.1.1.1.3.2.1.cmml" xref="S4.E8.m1.2.2.2.1.1.1.1.1.3.2.1">⋮</ci><ci id="S4.E8.m1.2.2.2.1.1.1.1.1.3.3.1.cmml" xref="S4.E8.m1.2.2.2.1.1.1.1.1.3.3.1">⋱</ci><ci id="S4.E8.m1.2.2.2.1.1.1.1.1.3.4.1.cmml" xref="S4.E8.m1.2.2.2.1.1.1.1.1.3.4.1">⋮</ci></matrixrow><matrixrow id="S4.E8.m1.2.2.2.1.1.1.1.1d.cmml" xref="S4.E8.m1.2.2.2.1.1.1.1.1"><cn type="integer" id="S4.E8.m1.2.2.2.1.1.1.1.1.4.1.1.cmml" xref="S4.E8.m1.2.2.2.1.1.1.1.1.4.1.1">0</cn><cn type="integer" id="S4.E8.m1.2.2.2.1.1.1.1.1.4.2.1.cmml" xref="S4.E8.m1.2.2.2.1.1.1.1.1.4.2.1">0</cn><ci id="S4.E8.m1.2.2.2.1.1.1.1.1.4.3.1.cmml" xref="S4.E8.m1.2.2.2.1.1.1.1.1.4.3.1">⋯</ci><ci id="S4.E8.m1.2.2.2.1.1.1.1.1.4.4.1.cmml" xref="S4.E8.m1.2.2.2.1.1.1.1.1.4.4.1">𝑀</ci></matrixrow></matrix></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E8.m1.21c">\begin{split}f_{\text{comp}}\left(x\right)&amp;=\begin{bmatrix}x_{1:\hat{r}}&amp;x_{\hat{r}:2\hat{r}}&amp;\cdots&amp;x_{(n-1)\hat{r}:n\hat{r}}\end{bmatrix}\\
f_{\text{decomp}}\left(x\right)&amp;=\text{concat}(x)\\
\Delta W&amp;=\begin{bmatrix}M&amp;\mathbf{0}&amp;\cdots&amp;\mathbf{0}\\
\mathbf{0}&amp;M&amp;\cdots&amp;\mathbf{0}\\
\vdots&amp;\vdots&amp;\ddots&amp;\vdots\\
\mathbf{0}&amp;\mathbf{0}&amp;\cdots&amp;M\end{bmatrix}\end{split}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(8)</span></td>
</tr></tbody>
</table>
<p id="S4.p5.15" class="ltx_p">where <math id="S4.p5.7.m1.1" class="ltx_Math" alttext="\text{concat}(x)" display="inline"><semantics id="S4.p5.7.m1.1a"><mrow id="S4.p5.7.m1.1.2" xref="S4.p5.7.m1.1.2.cmml"><mtext id="S4.p5.7.m1.1.2.2" xref="S4.p5.7.m1.1.2.2a.cmml">concat</mtext><mo lspace="0em" rspace="0em" id="S4.p5.7.m1.1.2.1" xref="S4.p5.7.m1.1.2.1.cmml">​</mo><mrow id="S4.p5.7.m1.1.2.3.2" xref="S4.p5.7.m1.1.2.cmml"><mo stretchy="false" id="S4.p5.7.m1.1.2.3.2.1" xref="S4.p5.7.m1.1.2.cmml">(</mo><mi id="S4.p5.7.m1.1.1" xref="S4.p5.7.m1.1.1.cmml">x</mi><mo stretchy="false" id="S4.p5.7.m1.1.2.3.2.2" xref="S4.p5.7.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.p5.7.m1.1b"><apply id="S4.p5.7.m1.1.2.cmml" xref="S4.p5.7.m1.1.2"><times id="S4.p5.7.m1.1.2.1.cmml" xref="S4.p5.7.m1.1.2.1"></times><ci id="S4.p5.7.m1.1.2.2a.cmml" xref="S4.p5.7.m1.1.2.2"><mtext id="S4.p5.7.m1.1.2.2.cmml" xref="S4.p5.7.m1.1.2.2">concat</mtext></ci><ci id="S4.p5.7.m1.1.1.cmml" xref="S4.p5.7.m1.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p5.7.m1.1c">\text{concat}(x)</annotation></semantics></math> refers to concatenate the rows of <math id="S4.p5.8.m2.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S4.p5.8.m2.1a"><mi id="S4.p5.8.m2.1.1" xref="S4.p5.8.m2.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S4.p5.8.m2.1b"><ci id="S4.p5.8.m2.1.1.cmml" xref="S4.p5.8.m2.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p5.8.m2.1c">x</annotation></semantics></math> into a vector.
For simplicity, we omit the padding and truncation operators in above functions and focus on the case where <math id="S4.p5.9.m3.1" class="ltx_Math" alttext="d=k" display="inline"><semantics id="S4.p5.9.m3.1a"><mrow id="S4.p5.9.m3.1.1" xref="S4.p5.9.m3.1.1.cmml"><mi id="S4.p5.9.m3.1.1.2" xref="S4.p5.9.m3.1.1.2.cmml">d</mi><mo id="S4.p5.9.m3.1.1.1" xref="S4.p5.9.m3.1.1.1.cmml">=</mo><mi id="S4.p5.9.m3.1.1.3" xref="S4.p5.9.m3.1.1.3.cmml">k</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.p5.9.m3.1b"><apply id="S4.p5.9.m3.1.1.cmml" xref="S4.p5.9.m3.1.1"><eq id="S4.p5.9.m3.1.1.1.cmml" xref="S4.p5.9.m3.1.1.1"></eq><ci id="S4.p5.9.m3.1.1.2.cmml" xref="S4.p5.9.m3.1.1.2">𝑑</ci><ci id="S4.p5.9.m3.1.1.3.cmml" xref="S4.p5.9.m3.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p5.9.m3.1c">d=k</annotation></semantics></math>. In comparison to sharing columns and rows, this method incurs additional computational overhead by reshaping <math id="S4.p5.10.m4.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S4.p5.10.m4.1a"><mi id="S4.p5.10.m4.1.1" xref="S4.p5.10.m4.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S4.p5.10.m4.1b"><ci id="S4.p5.10.m4.1.1.cmml" xref="S4.p5.10.m4.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p5.10.m4.1c">x</annotation></semantics></math> into <math id="S4.p5.11.m5.1" class="ltx_Math" alttext="\mathbb{R}^{n\times\hat{r}}" display="inline"><semantics id="S4.p5.11.m5.1a"><msup id="S4.p5.11.m5.1.1" xref="S4.p5.11.m5.1.1.cmml"><mi id="S4.p5.11.m5.1.1.2" xref="S4.p5.11.m5.1.1.2.cmml">ℝ</mi><mrow id="S4.p5.11.m5.1.1.3" xref="S4.p5.11.m5.1.1.3.cmml"><mi id="S4.p5.11.m5.1.1.3.2" xref="S4.p5.11.m5.1.1.3.2.cmml">n</mi><mo lspace="0.222em" rspace="0.222em" id="S4.p5.11.m5.1.1.3.1" xref="S4.p5.11.m5.1.1.3.1.cmml">×</mo><mover accent="true" id="S4.p5.11.m5.1.1.3.3" xref="S4.p5.11.m5.1.1.3.3.cmml"><mi id="S4.p5.11.m5.1.1.3.3.2" xref="S4.p5.11.m5.1.1.3.3.2.cmml">r</mi><mo id="S4.p5.11.m5.1.1.3.3.1" xref="S4.p5.11.m5.1.1.3.3.1.cmml">^</mo></mover></mrow></msup><annotation-xml encoding="MathML-Content" id="S4.p5.11.m5.1b"><apply id="S4.p5.11.m5.1.1.cmml" xref="S4.p5.11.m5.1.1"><csymbol cd="ambiguous" id="S4.p5.11.m5.1.1.1.cmml" xref="S4.p5.11.m5.1.1">superscript</csymbol><ci id="S4.p5.11.m5.1.1.2.cmml" xref="S4.p5.11.m5.1.1.2">ℝ</ci><apply id="S4.p5.11.m5.1.1.3.cmml" xref="S4.p5.11.m5.1.1.3"><times id="S4.p5.11.m5.1.1.3.1.cmml" xref="S4.p5.11.m5.1.1.3.1"></times><ci id="S4.p5.11.m5.1.1.3.2.cmml" xref="S4.p5.11.m5.1.1.3.2">𝑛</ci><apply id="S4.p5.11.m5.1.1.3.3.cmml" xref="S4.p5.11.m5.1.1.3.3"><ci id="S4.p5.11.m5.1.1.3.3.1.cmml" xref="S4.p5.11.m5.1.1.3.3.1">^</ci><ci id="S4.p5.11.m5.1.1.3.3.2.cmml" xref="S4.p5.11.m5.1.1.3.3.2">𝑟</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p5.11.m5.1c">\mathbb{R}^{n\times\hat{r}}</annotation></semantics></math> instead of <math id="S4.p5.12.m6.1" class="ltx_Math" alttext="\mathbb{R}^{\hat{r}}" display="inline"><semantics id="S4.p5.12.m6.1a"><msup id="S4.p5.12.m6.1.1" xref="S4.p5.12.m6.1.1.cmml"><mi id="S4.p5.12.m6.1.1.2" xref="S4.p5.12.m6.1.1.2.cmml">ℝ</mi><mover accent="true" id="S4.p5.12.m6.1.1.3" xref="S4.p5.12.m6.1.1.3.cmml"><mi id="S4.p5.12.m6.1.1.3.2" xref="S4.p5.12.m6.1.1.3.2.cmml">r</mi><mo id="S4.p5.12.m6.1.1.3.1" xref="S4.p5.12.m6.1.1.3.1.cmml">^</mo></mover></msup><annotation-xml encoding="MathML-Content" id="S4.p5.12.m6.1b"><apply id="S4.p5.12.m6.1.1.cmml" xref="S4.p5.12.m6.1.1"><csymbol cd="ambiguous" id="S4.p5.12.m6.1.1.1.cmml" xref="S4.p5.12.m6.1.1">superscript</csymbol><ci id="S4.p5.12.m6.1.1.2.cmml" xref="S4.p5.12.m6.1.1.2">ℝ</ci><apply id="S4.p5.12.m6.1.1.3.cmml" xref="S4.p5.12.m6.1.1.3"><ci id="S4.p5.12.m6.1.1.3.1.cmml" xref="S4.p5.12.m6.1.1.3.1">^</ci><ci id="S4.p5.12.m6.1.1.3.2.cmml" xref="S4.p5.12.m6.1.1.3.2">𝑟</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p5.12.m6.1c">\mathbb{R}^{\hat{r}}</annotation></semantics></math>. However, given that the size of <math id="S4.p5.13.m7.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S4.p5.13.m7.1a"><mi id="S4.p5.13.m7.1.1" xref="S4.p5.13.m7.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S4.p5.13.m7.1b"><ci id="S4.p5.13.m7.1.1.cmml" xref="S4.p5.13.m7.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p5.13.m7.1c">M</annotation></semantics></math> is significantly smaller than <math id="S4.p5.14.m8.1" class="ltx_Math" alttext="W_{0}" display="inline"><semantics id="S4.p5.14.m8.1a"><msub id="S4.p5.14.m8.1.1" xref="S4.p5.14.m8.1.1.cmml"><mi id="S4.p5.14.m8.1.1.2" xref="S4.p5.14.m8.1.1.2.cmml">W</mi><mn id="S4.p5.14.m8.1.1.3" xref="S4.p5.14.m8.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S4.p5.14.m8.1b"><apply id="S4.p5.14.m8.1.1.cmml" xref="S4.p5.14.m8.1.1"><csymbol cd="ambiguous" id="S4.p5.14.m8.1.1.1.cmml" xref="S4.p5.14.m8.1.1">subscript</csymbol><ci id="S4.p5.14.m8.1.1.2.cmml" xref="S4.p5.14.m8.1.1.2">𝑊</ci><cn type="integer" id="S4.p5.14.m8.1.1.3.cmml" xref="S4.p5.14.m8.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p5.14.m8.1c">W_{0}</annotation></semantics></math>, this additional computation is very small for rank like 8. For instance, when fine-tuning the 7B model with rank of 8 (<math id="S4.p5.15.m9.1" class="ltx_Math" alttext="\hat{r}=256" display="inline"><semantics id="S4.p5.15.m9.1a"><mrow id="S4.p5.15.m9.1.1" xref="S4.p5.15.m9.1.1.cmml"><mover accent="true" id="S4.p5.15.m9.1.1.2" xref="S4.p5.15.m9.1.1.2.cmml"><mi id="S4.p5.15.m9.1.1.2.2" xref="S4.p5.15.m9.1.1.2.2.cmml">r</mi><mo id="S4.p5.15.m9.1.1.2.1" xref="S4.p5.15.m9.1.1.2.1.cmml">^</mo></mover><mo id="S4.p5.15.m9.1.1.1" xref="S4.p5.15.m9.1.1.1.cmml">=</mo><mn id="S4.p5.15.m9.1.1.3" xref="S4.p5.15.m9.1.1.3.cmml">256</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.p5.15.m9.1b"><apply id="S4.p5.15.m9.1.1.cmml" xref="S4.p5.15.m9.1.1"><eq id="S4.p5.15.m9.1.1.1.cmml" xref="S4.p5.15.m9.1.1.1"></eq><apply id="S4.p5.15.m9.1.1.2.cmml" xref="S4.p5.15.m9.1.1.2"><ci id="S4.p5.15.m9.1.1.2.1.cmml" xref="S4.p5.15.m9.1.1.2.1">^</ci><ci id="S4.p5.15.m9.1.1.2.2.cmml" xref="S4.p5.15.m9.1.1.2.2">𝑟</ci></apply><cn type="integer" id="S4.p5.15.m9.1.1.3.cmml" xref="S4.p5.15.m9.1.1.3">256</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p5.15.m9.1c">\hat{r}=256</annotation></semantics></math>), this method is only 1.03 times slower than previous methods.</p>
</div>
<div id="S4.p6" class="ltx_para">
<p id="S4.p6.3" class="ltx_p">Inspired by RoPE <cite class="ltx_cite ltx_citemacro_cite">Su et al. (<a href="#bib.bib34" title="" class="ltx_ref">2024</a>)</cite>, we can further refine this method by incorporating rotation operators into <math id="S4.p6.1.m1.1" class="ltx_Math" alttext="f_{\text{comp}}" display="inline"><semantics id="S4.p6.1.m1.1a"><msub id="S4.p6.1.m1.1.1" xref="S4.p6.1.m1.1.1.cmml"><mi id="S4.p6.1.m1.1.1.2" xref="S4.p6.1.m1.1.1.2.cmml">f</mi><mtext id="S4.p6.1.m1.1.1.3" xref="S4.p6.1.m1.1.1.3a.cmml">comp</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.p6.1.m1.1b"><apply id="S4.p6.1.m1.1.1.cmml" xref="S4.p6.1.m1.1.1"><csymbol cd="ambiguous" id="S4.p6.1.m1.1.1.1.cmml" xref="S4.p6.1.m1.1.1">subscript</csymbol><ci id="S4.p6.1.m1.1.1.2.cmml" xref="S4.p6.1.m1.1.1.2">𝑓</ci><ci id="S4.p6.1.m1.1.1.3a.cmml" xref="S4.p6.1.m1.1.1.3"><mtext mathsize="70%" id="S4.p6.1.m1.1.1.3.cmml" xref="S4.p6.1.m1.1.1.3">comp</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p6.1.m1.1c">f_{\text{comp}}</annotation></semantics></math> to augment the expressiveness of <math id="S4.p6.2.m2.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S4.p6.2.m2.1a"><mi id="S4.p6.2.m2.1.1" xref="S4.p6.2.m2.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S4.p6.2.m2.1b"><ci id="S4.p6.2.m2.1.1.cmml" xref="S4.p6.2.m2.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p6.2.m2.1c">M</annotation></semantics></math> by enable it to differentiate between various <math id="S4.p6.3.m3.1" class="ltx_Math" alttext="x_{i\hat{r}:(i+1)\hat{r}}" display="inline"><semantics id="S4.p6.3.m3.1a"><msub id="S4.p6.3.m3.1.2" xref="S4.p6.3.m3.1.2.cmml"><mi id="S4.p6.3.m3.1.2.2" xref="S4.p6.3.m3.1.2.2.cmml">x</mi><mrow id="S4.p6.3.m3.1.1.1" xref="S4.p6.3.m3.1.1.1.cmml"><mrow id="S4.p6.3.m3.1.1.1.3" xref="S4.p6.3.m3.1.1.1.3.cmml"><mi id="S4.p6.3.m3.1.1.1.3.2" xref="S4.p6.3.m3.1.1.1.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S4.p6.3.m3.1.1.1.3.1" xref="S4.p6.3.m3.1.1.1.3.1.cmml">​</mo><mover accent="true" id="S4.p6.3.m3.1.1.1.3.3" xref="S4.p6.3.m3.1.1.1.3.3.cmml"><mi id="S4.p6.3.m3.1.1.1.3.3.2" xref="S4.p6.3.m3.1.1.1.3.3.2.cmml">r</mi><mo id="S4.p6.3.m3.1.1.1.3.3.1" xref="S4.p6.3.m3.1.1.1.3.3.1.cmml">^</mo></mover></mrow><mo lspace="0.278em" rspace="0.278em" id="S4.p6.3.m3.1.1.1.2" xref="S4.p6.3.m3.1.1.1.2.cmml">:</mo><mrow id="S4.p6.3.m3.1.1.1.1" xref="S4.p6.3.m3.1.1.1.1.cmml"><mrow id="S4.p6.3.m3.1.1.1.1.1.1" xref="S4.p6.3.m3.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.p6.3.m3.1.1.1.1.1.1.2" xref="S4.p6.3.m3.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.p6.3.m3.1.1.1.1.1.1.1" xref="S4.p6.3.m3.1.1.1.1.1.1.1.cmml"><mi id="S4.p6.3.m3.1.1.1.1.1.1.1.2" xref="S4.p6.3.m3.1.1.1.1.1.1.1.2.cmml">i</mi><mo id="S4.p6.3.m3.1.1.1.1.1.1.1.1" xref="S4.p6.3.m3.1.1.1.1.1.1.1.1.cmml">+</mo><mn id="S4.p6.3.m3.1.1.1.1.1.1.1.3" xref="S4.p6.3.m3.1.1.1.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="S4.p6.3.m3.1.1.1.1.1.1.3" xref="S4.p6.3.m3.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S4.p6.3.m3.1.1.1.1.2" xref="S4.p6.3.m3.1.1.1.1.2.cmml">​</mo><mover accent="true" id="S4.p6.3.m3.1.1.1.1.3" xref="S4.p6.3.m3.1.1.1.1.3.cmml"><mi id="S4.p6.3.m3.1.1.1.1.3.2" xref="S4.p6.3.m3.1.1.1.1.3.2.cmml">r</mi><mo id="S4.p6.3.m3.1.1.1.1.3.1" xref="S4.p6.3.m3.1.1.1.1.3.1.cmml">^</mo></mover></mrow></mrow></msub><annotation-xml encoding="MathML-Content" id="S4.p6.3.m3.1b"><apply id="S4.p6.3.m3.1.2.cmml" xref="S4.p6.3.m3.1.2"><csymbol cd="ambiguous" id="S4.p6.3.m3.1.2.1.cmml" xref="S4.p6.3.m3.1.2">subscript</csymbol><ci id="S4.p6.3.m3.1.2.2.cmml" xref="S4.p6.3.m3.1.2.2">𝑥</ci><apply id="S4.p6.3.m3.1.1.1.cmml" xref="S4.p6.3.m3.1.1.1"><ci id="S4.p6.3.m3.1.1.1.2.cmml" xref="S4.p6.3.m3.1.1.1.2">:</ci><apply id="S4.p6.3.m3.1.1.1.3.cmml" xref="S4.p6.3.m3.1.1.1.3"><times id="S4.p6.3.m3.1.1.1.3.1.cmml" xref="S4.p6.3.m3.1.1.1.3.1"></times><ci id="S4.p6.3.m3.1.1.1.3.2.cmml" xref="S4.p6.3.m3.1.1.1.3.2">𝑖</ci><apply id="S4.p6.3.m3.1.1.1.3.3.cmml" xref="S4.p6.3.m3.1.1.1.3.3"><ci id="S4.p6.3.m3.1.1.1.3.3.1.cmml" xref="S4.p6.3.m3.1.1.1.3.3.1">^</ci><ci id="S4.p6.3.m3.1.1.1.3.3.2.cmml" xref="S4.p6.3.m3.1.1.1.3.3.2">𝑟</ci></apply></apply><apply id="S4.p6.3.m3.1.1.1.1.cmml" xref="S4.p6.3.m3.1.1.1.1"><times id="S4.p6.3.m3.1.1.1.1.2.cmml" xref="S4.p6.3.m3.1.1.1.1.2"></times><apply id="S4.p6.3.m3.1.1.1.1.1.1.1.cmml" xref="S4.p6.3.m3.1.1.1.1.1.1"><plus id="S4.p6.3.m3.1.1.1.1.1.1.1.1.cmml" xref="S4.p6.3.m3.1.1.1.1.1.1.1.1"></plus><ci id="S4.p6.3.m3.1.1.1.1.1.1.1.2.cmml" xref="S4.p6.3.m3.1.1.1.1.1.1.1.2">𝑖</ci><cn type="integer" id="S4.p6.3.m3.1.1.1.1.1.1.1.3.cmml" xref="S4.p6.3.m3.1.1.1.1.1.1.1.3">1</cn></apply><apply id="S4.p6.3.m3.1.1.1.1.3.cmml" xref="S4.p6.3.m3.1.1.1.1.3"><ci id="S4.p6.3.m3.1.1.1.1.3.1.cmml" xref="S4.p6.3.m3.1.1.1.1.3.1">^</ci><ci id="S4.p6.3.m3.1.1.1.1.3.2.cmml" xref="S4.p6.3.m3.1.1.1.1.3.2">𝑟</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p6.3.m3.1c">x_{i\hat{r}:(i+1)\hat{r}}</annotation></semantics></math> by rotating them.
We can modify Eq. <a href="#S4.E8" title="In 4 Method ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> as follows:</p>
<table id="S4.E9" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E9.m1.11" class="ltx_Math" alttext="\begin{split}f_{\text{comp}}\left(x\right)&amp;=\begin{bmatrix}a^{1}&amp;a^{2}&amp;\cdots&amp;a^{n-1}\end{bmatrix}\\
\Delta W&amp;=\begin{bmatrix}P^{1}&amp;\mathbf{0}&amp;\cdots&amp;\mathbf{0}\\
\mathbf{0}&amp;P^{2}&amp;\cdots&amp;\mathbf{0}\\
\vdots&amp;\vdots&amp;\ddots&amp;\vdots\\
\mathbf{0}&amp;\mathbf{0}&amp;\cdots&amp;P^{n-1}\end{bmatrix}\end{split}" display="block"><semantics id="S4.E9.m1.11a"><mtable columnspacing="0pt" displaystyle="true" rowspacing="0pt" id="S4.E9.m1.11.11" xref="S4.E9.m1.11.12.1.cmml"><mtr id="S4.E9.m1.11.11a" xref="S4.E9.m1.11.12.1.cmml"><mtd class="ltx_align_right" columnalign="right" id="S4.E9.m1.11.11b" xref="S4.E9.m1.11.12.1.cmml"><mrow id="S4.E9.m1.7.7.7.6.5" xref="S4.E9.m1.11.12.1.cmml"><msub id="S4.E9.m1.7.7.7.6.5.7" xref="S4.E9.m1.11.12.1.cmml"><mi id="S4.E9.m1.3.3.3.2.1.1" xref="S4.E9.m1.3.3.3.2.1.1.cmml">f</mi><mtext id="S4.E9.m1.4.4.4.3.2.2.1" xref="S4.E9.m1.4.4.4.3.2.2.1a.cmml">comp</mtext></msub><mo lspace="0em" rspace="0em" id="S4.E9.m1.7.7.7.6.5.6" xref="S4.E9.m1.11.12.1a.cmml">​</mo><mrow id="S4.E9.m1.7.7.7.6.5.8" xref="S4.E9.m1.11.12.1.cmml"><mo id="S4.E9.m1.5.5.5.4.3.3" xref="S4.E9.m1.11.12.1a.cmml">(</mo><mi id="S4.E9.m1.6.6.6.5.4.4" xref="S4.E9.m1.6.6.6.5.4.4.cmml">x</mi><mo id="S4.E9.m1.7.7.7.6.5.5" xref="S4.E9.m1.11.12.1a.cmml">)</mo></mrow></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S4.E9.m1.11.11c" xref="S4.E9.m1.11.12.1.cmml"><mrow id="S4.E9.m1.8.8.8.7.2" xref="S4.E9.m1.11.12.1.cmml"><mi id="S4.E9.m1.8.8.8.7.2.3" xref="S4.E9.m1.11.12.1a.cmml"></mi><mo id="S4.E9.m1.8.8.8.7.2.2" xref="S4.E9.m1.8.8.8.7.2.2.cmml">=</mo><mrow id="S4.E9.m1.1.1.1.1.1.1.3" xref="S4.E9.m1.1.1.1.1.1.1.2.cmml"><mo id="S4.E9.m1.1.1.1.1.1.1.3.1" xref="S4.E9.m1.1.1.1.1.1.1.2.1.cmml">[</mo><mtable columnspacing="5pt" displaystyle="true" id="S4.E9.m1.1.1.1.1.1.1.1.1" xref="S4.E9.m1.1.1.1.1.1.1.1.1.cmml"><mtr id="S4.E9.m1.1.1.1.1.1.1.1.1a" xref="S4.E9.m1.1.1.1.1.1.1.1.1.cmml"><mtd id="S4.E9.m1.1.1.1.1.1.1.1.1b" xref="S4.E9.m1.1.1.1.1.1.1.1.1.cmml"><msup id="S4.E9.m1.1.1.1.1.1.1.1.1.1.1.1" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S4.E9.m1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">a</mi><mn id="S4.E9.m1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">1</mn></msup></mtd><mtd id="S4.E9.m1.1.1.1.1.1.1.1.1c" xref="S4.E9.m1.1.1.1.1.1.1.1.1.cmml"><msup id="S4.E9.m1.1.1.1.1.1.1.1.1.1.2.1" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.2.1.cmml"><mi id="S4.E9.m1.1.1.1.1.1.1.1.1.1.2.1.2" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.2.1.2.cmml">a</mi><mn id="S4.E9.m1.1.1.1.1.1.1.1.1.1.2.1.3" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.2.1.3.cmml">2</mn></msup></mtd><mtd id="S4.E9.m1.1.1.1.1.1.1.1.1d" xref="S4.E9.m1.1.1.1.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E9.m1.1.1.1.1.1.1.1.1.1.3.1" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.3.1.cmml">⋯</mi></mtd><mtd id="S4.E9.m1.1.1.1.1.1.1.1.1e" xref="S4.E9.m1.1.1.1.1.1.1.1.1.cmml"><msup id="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.cmml"><mi id="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.2" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.2.cmml">a</mi><mrow id="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.3" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.3.cmml"><mi id="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.3.2" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.3.2.cmml">n</mi><mo id="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.3.1" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.3.1.cmml">−</mo><mn id="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.3.3" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.3.3.cmml">1</mn></mrow></msup></mtd></mtr></mtable><mo id="S4.E9.m1.1.1.1.1.1.1.3.2" xref="S4.E9.m1.1.1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow></mtd></mtr><mtr id="S4.E9.m1.11.11d" xref="S4.E9.m1.11.12.1.cmml"><mtd class="ltx_align_right" columnalign="right" id="S4.E9.m1.11.11e" xref="S4.E9.m1.11.12.1.cmml"><mrow id="S4.E9.m1.10.10.10.3.2" xref="S4.E9.m1.11.12.1.cmml"><mi mathvariant="normal" id="S4.E9.m1.9.9.9.2.1.1" xref="S4.E9.m1.9.9.9.2.1.1.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S4.E9.m1.10.10.10.3.2.3" xref="S4.E9.m1.11.12.1a.cmml">​</mo><mi id="S4.E9.m1.10.10.10.3.2.2" xref="S4.E9.m1.10.10.10.3.2.2.cmml">W</mi></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S4.E9.m1.11.11f" xref="S4.E9.m1.11.12.1.cmml"><mrow id="S4.E9.m1.11.11.11.4.2" xref="S4.E9.m1.11.12.1.cmml"><mi id="S4.E9.m1.11.11.11.4.2.3" xref="S4.E9.m1.11.12.1a.cmml"></mi><mo id="S4.E9.m1.11.11.11.4.2.2" xref="S4.E9.m1.11.11.11.4.2.2.cmml">=</mo><mrow id="S4.E9.m1.2.2.2.1.1.1.3" xref="S4.E9.m1.2.2.2.1.1.1.2.cmml"><mo id="S4.E9.m1.2.2.2.1.1.1.3.1" xref="S4.E9.m1.2.2.2.1.1.1.2.1.cmml">[</mo><mtable columnspacing="5pt" displaystyle="true" rowspacing="0pt" id="S4.E9.m1.2.2.2.1.1.1.1.1" xref="S4.E9.m1.2.2.2.1.1.1.1.1.cmml"><mtr id="S4.E9.m1.2.2.2.1.1.1.1.1a" xref="S4.E9.m1.2.2.2.1.1.1.1.1.cmml"><mtd id="S4.E9.m1.2.2.2.1.1.1.1.1b" xref="S4.E9.m1.2.2.2.1.1.1.1.1.cmml"><msup id="S4.E9.m1.2.2.2.1.1.1.1.1.1.1.1" xref="S4.E9.m1.2.2.2.1.1.1.1.1.1.1.1.cmml"><mi id="S4.E9.m1.2.2.2.1.1.1.1.1.1.1.1.2" xref="S4.E9.m1.2.2.2.1.1.1.1.1.1.1.1.2.cmml">P</mi><mn id="S4.E9.m1.2.2.2.1.1.1.1.1.1.1.1.3" xref="S4.E9.m1.2.2.2.1.1.1.1.1.1.1.1.3.cmml">1</mn></msup></mtd><mtd id="S4.E9.m1.2.2.2.1.1.1.1.1c" xref="S4.E9.m1.2.2.2.1.1.1.1.1.cmml"><mn id="S4.E9.m1.2.2.2.1.1.1.1.1.1.2.1" xref="S4.E9.m1.2.2.2.1.1.1.1.1.1.2.1.cmml">𝟎</mn></mtd><mtd id="S4.E9.m1.2.2.2.1.1.1.1.1d" xref="S4.E9.m1.2.2.2.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E9.m1.2.2.2.1.1.1.1.1.1.3.1" xref="S4.E9.m1.2.2.2.1.1.1.1.1.1.3.1.cmml">⋯</mi></mtd><mtd id="S4.E9.m1.2.2.2.1.1.1.1.1e" xref="S4.E9.m1.2.2.2.1.1.1.1.1.cmml"><mn id="S4.E9.m1.2.2.2.1.1.1.1.1.1.4.1" xref="S4.E9.m1.2.2.2.1.1.1.1.1.1.4.1.cmml">𝟎</mn></mtd></mtr><mtr id="S4.E9.m1.2.2.2.1.1.1.1.1f" xref="S4.E9.m1.2.2.2.1.1.1.1.1.cmml"><mtd id="S4.E9.m1.2.2.2.1.1.1.1.1g" xref="S4.E9.m1.2.2.2.1.1.1.1.1.cmml"><mn id="S4.E9.m1.2.2.2.1.1.1.1.1.2.1.1" xref="S4.E9.m1.2.2.2.1.1.1.1.1.2.1.1.cmml">𝟎</mn></mtd><mtd id="S4.E9.m1.2.2.2.1.1.1.1.1h" xref="S4.E9.m1.2.2.2.1.1.1.1.1.cmml"><msup id="S4.E9.m1.2.2.2.1.1.1.1.1.2.2.1" xref="S4.E9.m1.2.2.2.1.1.1.1.1.2.2.1.cmml"><mi id="S4.E9.m1.2.2.2.1.1.1.1.1.2.2.1.2" xref="S4.E9.m1.2.2.2.1.1.1.1.1.2.2.1.2.cmml">P</mi><mn id="S4.E9.m1.2.2.2.1.1.1.1.1.2.2.1.3" xref="S4.E9.m1.2.2.2.1.1.1.1.1.2.2.1.3.cmml">2</mn></msup></mtd><mtd id="S4.E9.m1.2.2.2.1.1.1.1.1i" xref="S4.E9.m1.2.2.2.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E9.m1.2.2.2.1.1.1.1.1.2.3.1" xref="S4.E9.m1.2.2.2.1.1.1.1.1.2.3.1.cmml">⋯</mi></mtd><mtd id="S4.E9.m1.2.2.2.1.1.1.1.1j" xref="S4.E9.m1.2.2.2.1.1.1.1.1.cmml"><mn id="S4.E9.m1.2.2.2.1.1.1.1.1.2.4.1" xref="S4.E9.m1.2.2.2.1.1.1.1.1.2.4.1.cmml">𝟎</mn></mtd></mtr><mtr id="S4.E9.m1.2.2.2.1.1.1.1.1k" xref="S4.E9.m1.2.2.2.1.1.1.1.1.cmml"><mtd id="S4.E9.m1.2.2.2.1.1.1.1.1l" xref="S4.E9.m1.2.2.2.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E9.m1.2.2.2.1.1.1.1.1.3.1.1" xref="S4.E9.m1.2.2.2.1.1.1.1.1.3.1.1.cmml">⋮</mi></mtd><mtd id="S4.E9.m1.2.2.2.1.1.1.1.1m" xref="S4.E9.m1.2.2.2.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E9.m1.2.2.2.1.1.1.1.1.3.2.1" xref="S4.E9.m1.2.2.2.1.1.1.1.1.3.2.1.cmml">⋮</mi></mtd><mtd id="S4.E9.m1.2.2.2.1.1.1.1.1n" xref="S4.E9.m1.2.2.2.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E9.m1.2.2.2.1.1.1.1.1.3.3.1" xref="S4.E9.m1.2.2.2.1.1.1.1.1.3.3.1.cmml">⋱</mi></mtd><mtd id="S4.E9.m1.2.2.2.1.1.1.1.1o" xref="S4.E9.m1.2.2.2.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E9.m1.2.2.2.1.1.1.1.1.3.4.1" xref="S4.E9.m1.2.2.2.1.1.1.1.1.3.4.1.cmml">⋮</mi></mtd></mtr><mtr id="S4.E9.m1.2.2.2.1.1.1.1.1p" xref="S4.E9.m1.2.2.2.1.1.1.1.1.cmml"><mtd id="S4.E9.m1.2.2.2.1.1.1.1.1q" xref="S4.E9.m1.2.2.2.1.1.1.1.1.cmml"><mn id="S4.E9.m1.2.2.2.1.1.1.1.1.4.1.1" xref="S4.E9.m1.2.2.2.1.1.1.1.1.4.1.1.cmml">𝟎</mn></mtd><mtd id="S4.E9.m1.2.2.2.1.1.1.1.1r" xref="S4.E9.m1.2.2.2.1.1.1.1.1.cmml"><mn id="S4.E9.m1.2.2.2.1.1.1.1.1.4.2.1" xref="S4.E9.m1.2.2.2.1.1.1.1.1.4.2.1.cmml">𝟎</mn></mtd><mtd id="S4.E9.m1.2.2.2.1.1.1.1.1s" xref="S4.E9.m1.2.2.2.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E9.m1.2.2.2.1.1.1.1.1.4.3.1" xref="S4.E9.m1.2.2.2.1.1.1.1.1.4.3.1.cmml">⋯</mi></mtd><mtd id="S4.E9.m1.2.2.2.1.1.1.1.1t" xref="S4.E9.m1.2.2.2.1.1.1.1.1.cmml"><msup id="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1" xref="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.cmml"><mi id="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.2" xref="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.2.cmml">P</mi><mrow id="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.3" xref="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.3.cmml"><mi id="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.3.2" xref="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.3.2.cmml">n</mi><mo id="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.3.1" xref="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.3.1.cmml">−</mo><mn id="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.3.3" xref="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.3.3.cmml">1</mn></mrow></msup></mtd></mtr></mtable><mo id="S4.E9.m1.2.2.2.1.1.1.3.2" xref="S4.E9.m1.2.2.2.1.1.1.2.1.cmml">]</mo></mrow></mrow></mtd></mtr></mtable><annotation-xml encoding="MathML-Content" id="S4.E9.m1.11b"><apply id="S4.E9.m1.11.12.1.cmml" xref="S4.E9.m1.11.11"><and id="S4.E9.m1.11.12.1a.cmml" xref="S4.E9.m1.7.7.7.6.5.6"></and><apply id="S4.E9.m1.11.12.1b.cmml" xref="S4.E9.m1.11.11"><eq id="S4.E9.m1.8.8.8.7.2.2.cmml" xref="S4.E9.m1.8.8.8.7.2.2"></eq><apply id="S4.E9.m1.11.12.1.2.cmml" xref="S4.E9.m1.11.11"><times id="S4.E9.m1.11.12.1.2.1.cmml" xref="S4.E9.m1.7.7.7.6.5.6"></times><apply id="S4.E9.m1.11.12.1.2.2.cmml" xref="S4.E9.m1.11.11"><csymbol cd="ambiguous" id="S4.E9.m1.11.12.1.2.2.1.cmml" xref="S4.E9.m1.7.7.7.6.5.6">subscript</csymbol><ci id="S4.E9.m1.3.3.3.2.1.1.cmml" xref="S4.E9.m1.3.3.3.2.1.1">𝑓</ci><ci id="S4.E9.m1.4.4.4.3.2.2.1a.cmml" xref="S4.E9.m1.4.4.4.3.2.2.1"><mtext mathsize="70%" id="S4.E9.m1.4.4.4.3.2.2.1.cmml" xref="S4.E9.m1.4.4.4.3.2.2.1">comp</mtext></ci></apply><ci id="S4.E9.m1.6.6.6.5.4.4.cmml" xref="S4.E9.m1.6.6.6.5.4.4">𝑥</ci></apply><apply id="S4.E9.m1.11.12.1.4.cmml" xref="S4.E9.m1.11.11"><times id="S4.E9.m1.11.12.1.4.1.cmml" xref="S4.E9.m1.7.7.7.6.5.6"></times><apply id="S4.E9.m1.1.1.1.1.1.1.2.cmml" xref="S4.E9.m1.1.1.1.1.1.1.3"><csymbol cd="latexml" id="S4.E9.m1.1.1.1.1.1.1.2.1.cmml" xref="S4.E9.m1.1.1.1.1.1.1.3.1">matrix</csymbol><matrix id="S4.E9.m1.1.1.1.1.1.1.1.1.cmml" xref="S4.E9.m1.1.1.1.1.1.1.1.1"><matrixrow id="S4.E9.m1.1.1.1.1.1.1.1.1a.cmml" xref="S4.E9.m1.1.1.1.1.1.1.1.1"><apply id="S4.E9.m1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.E9.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.1.1">superscript</csymbol><ci id="S4.E9.m1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.1.1.2">𝑎</ci><cn type="integer" id="S4.E9.m1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.1.1.3">1</cn></apply><apply id="S4.E9.m1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.2.1"><csymbol cd="ambiguous" id="S4.E9.m1.1.1.1.1.1.1.1.1.1.2.1.1.cmml" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.2.1">superscript</csymbol><ci id="S4.E9.m1.1.1.1.1.1.1.1.1.1.2.1.2.cmml" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.2.1.2">𝑎</ci><cn type="integer" id="S4.E9.m1.1.1.1.1.1.1.1.1.1.2.1.3.cmml" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.2.1.3">2</cn></apply><ci id="S4.E9.m1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.3.1">⋯</ci><apply id="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.cmml" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1"><csymbol cd="ambiguous" id="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.1.cmml" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1">superscript</csymbol><ci id="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.2.cmml" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.2">𝑎</ci><apply id="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.3.cmml" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.3"><minus id="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.3.1.cmml" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.3.1"></minus><ci id="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.3.2.cmml" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.3.2">𝑛</ci><cn type="integer" id="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.3.3.cmml" xref="S4.E9.m1.1.1.1.1.1.1.1.1.1.4.1.3.3">1</cn></apply></apply></matrixrow></matrix></apply><ci id="S4.E9.m1.9.9.9.2.1.1.cmml" xref="S4.E9.m1.9.9.9.2.1.1">Δ</ci><ci id="S4.E9.m1.10.10.10.3.2.2.cmml" xref="S4.E9.m1.10.10.10.3.2.2">𝑊</ci></apply></apply><apply id="S4.E9.m1.11.12.1c.cmml" xref="S4.E9.m1.11.11"><eq id="S4.E9.m1.11.11.11.4.2.2.cmml" xref="S4.E9.m1.11.11.11.4.2.2"></eq><share href="#S4.E9.m1.11.12.1.4.cmml" id="S4.E9.m1.11.12.1d.cmml" xref="S4.E9.m1.7.7.7.6.5.6"></share><apply id="S4.E9.m1.2.2.2.1.1.1.2.cmml" xref="S4.E9.m1.2.2.2.1.1.1.3"><csymbol cd="latexml" id="S4.E9.m1.2.2.2.1.1.1.2.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.3.1">matrix</csymbol><matrix id="S4.E9.m1.2.2.2.1.1.1.1.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1"><matrixrow id="S4.E9.m1.2.2.2.1.1.1.1.1a.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1"><apply id="S4.E9.m1.2.2.2.1.1.1.1.1.1.1.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.E9.m1.2.2.2.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.1.1.1">superscript</csymbol><ci id="S4.E9.m1.2.2.2.1.1.1.1.1.1.1.1.2.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.1.1.1.2">𝑃</ci><cn type="integer" id="S4.E9.m1.2.2.2.1.1.1.1.1.1.1.1.3.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.1.1.1.3">1</cn></apply><cn type="integer" id="S4.E9.m1.2.2.2.1.1.1.1.1.1.2.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.1.2.1">0</cn><ci id="S4.E9.m1.2.2.2.1.1.1.1.1.1.3.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.1.3.1">⋯</ci><cn type="integer" id="S4.E9.m1.2.2.2.1.1.1.1.1.1.4.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.1.4.1">0</cn></matrixrow><matrixrow id="S4.E9.m1.2.2.2.1.1.1.1.1b.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1"><cn type="integer" id="S4.E9.m1.2.2.2.1.1.1.1.1.2.1.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.2.1.1">0</cn><apply id="S4.E9.m1.2.2.2.1.1.1.1.1.2.2.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.2.2.1"><csymbol cd="ambiguous" id="S4.E9.m1.2.2.2.1.1.1.1.1.2.2.1.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.2.2.1">superscript</csymbol><ci id="S4.E9.m1.2.2.2.1.1.1.1.1.2.2.1.2.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.2.2.1.2">𝑃</ci><cn type="integer" id="S4.E9.m1.2.2.2.1.1.1.1.1.2.2.1.3.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.2.2.1.3">2</cn></apply><ci id="S4.E9.m1.2.2.2.1.1.1.1.1.2.3.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.2.3.1">⋯</ci><cn type="integer" id="S4.E9.m1.2.2.2.1.1.1.1.1.2.4.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.2.4.1">0</cn></matrixrow><matrixrow id="S4.E9.m1.2.2.2.1.1.1.1.1c.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1"><ci id="S4.E9.m1.2.2.2.1.1.1.1.1.3.1.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.3.1.1">⋮</ci><ci id="S4.E9.m1.2.2.2.1.1.1.1.1.3.2.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.3.2.1">⋮</ci><ci id="S4.E9.m1.2.2.2.1.1.1.1.1.3.3.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.3.3.1">⋱</ci><ci id="S4.E9.m1.2.2.2.1.1.1.1.1.3.4.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.3.4.1">⋮</ci></matrixrow><matrixrow id="S4.E9.m1.2.2.2.1.1.1.1.1d.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1"><cn type="integer" id="S4.E9.m1.2.2.2.1.1.1.1.1.4.1.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.4.1.1">0</cn><cn type="integer" id="S4.E9.m1.2.2.2.1.1.1.1.1.4.2.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.4.2.1">0</cn><ci id="S4.E9.m1.2.2.2.1.1.1.1.1.4.3.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.4.3.1">⋯</ci><apply id="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1"><csymbol cd="ambiguous" id="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1">superscript</csymbol><ci id="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.2.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.2">𝑃</ci><apply id="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.3.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.3"><minus id="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.3.1.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.3.1"></minus><ci id="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.3.2.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.3.2">𝑛</ci><cn type="integer" id="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.3.3.cmml" xref="S4.E9.m1.2.2.2.1.1.1.1.1.4.4.1.3.3">1</cn></apply></apply></matrixrow></matrix></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E9.m1.11c">\begin{split}f_{\text{comp}}\left(x\right)&amp;=\begin{bmatrix}a^{1}&amp;a^{2}&amp;\cdots&amp;a^{n-1}\end{bmatrix}\\
\Delta W&amp;=\begin{bmatrix}P^{1}&amp;\mathbf{0}&amp;\cdots&amp;\mathbf{0}\\
\mathbf{0}&amp;P^{2}&amp;\cdots&amp;\mathbf{0}\\
\vdots&amp;\vdots&amp;\ddots&amp;\vdots\\
\mathbf{0}&amp;\mathbf{0}&amp;\cdots&amp;P^{n-1}\end{bmatrix}\end{split}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(9)</span></td>
</tr></tbody>
</table>
<p id="S4.p6.12" class="ltx_p">where <math id="S4.p6.4.m1.1" class="ltx_Math" alttext="a^{i}" display="inline"><semantics id="S4.p6.4.m1.1a"><msup id="S4.p6.4.m1.1.1" xref="S4.p6.4.m1.1.1.cmml"><mi id="S4.p6.4.m1.1.1.2" xref="S4.p6.4.m1.1.1.2.cmml">a</mi><mi id="S4.p6.4.m1.1.1.3" xref="S4.p6.4.m1.1.1.3.cmml">i</mi></msup><annotation-xml encoding="MathML-Content" id="S4.p6.4.m1.1b"><apply id="S4.p6.4.m1.1.1.cmml" xref="S4.p6.4.m1.1.1"><csymbol cd="ambiguous" id="S4.p6.4.m1.1.1.1.cmml" xref="S4.p6.4.m1.1.1">superscript</csymbol><ci id="S4.p6.4.m1.1.1.2.cmml" xref="S4.p6.4.m1.1.1.2">𝑎</ci><ci id="S4.p6.4.m1.1.1.3.cmml" xref="S4.p6.4.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p6.4.m1.1c">a^{i}</annotation></semantics></math> and <math id="S4.p6.5.m2.1" class="ltx_Math" alttext="P^{i}" display="inline"><semantics id="S4.p6.5.m2.1a"><msup id="S4.p6.5.m2.1.1" xref="S4.p6.5.m2.1.1.cmml"><mi id="S4.p6.5.m2.1.1.2" xref="S4.p6.5.m2.1.1.2.cmml">P</mi><mi id="S4.p6.5.m2.1.1.3" xref="S4.p6.5.m2.1.1.3.cmml">i</mi></msup><annotation-xml encoding="MathML-Content" id="S4.p6.5.m2.1b"><apply id="S4.p6.5.m2.1.1.cmml" xref="S4.p6.5.m2.1.1"><csymbol cd="ambiguous" id="S4.p6.5.m2.1.1.1.cmml" xref="S4.p6.5.m2.1.1">superscript</csymbol><ci id="S4.p6.5.m2.1.1.2.cmml" xref="S4.p6.5.m2.1.1.2">𝑃</ci><ci id="S4.p6.5.m2.1.1.3.cmml" xref="S4.p6.5.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p6.5.m2.1c">P^{i}</annotation></semantics></math> represent the corresponding values of <math id="S4.p6.6.m3.1" class="ltx_Math" alttext="x_{i\hat{r}:(i+1)\hat{r}}" display="inline"><semantics id="S4.p6.6.m3.1a"><msub id="S4.p6.6.m3.1.2" xref="S4.p6.6.m3.1.2.cmml"><mi id="S4.p6.6.m3.1.2.2" xref="S4.p6.6.m3.1.2.2.cmml">x</mi><mrow id="S4.p6.6.m3.1.1.1" xref="S4.p6.6.m3.1.1.1.cmml"><mrow id="S4.p6.6.m3.1.1.1.3" xref="S4.p6.6.m3.1.1.1.3.cmml"><mi id="S4.p6.6.m3.1.1.1.3.2" xref="S4.p6.6.m3.1.1.1.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S4.p6.6.m3.1.1.1.3.1" xref="S4.p6.6.m3.1.1.1.3.1.cmml">​</mo><mover accent="true" id="S4.p6.6.m3.1.1.1.3.3" xref="S4.p6.6.m3.1.1.1.3.3.cmml"><mi id="S4.p6.6.m3.1.1.1.3.3.2" xref="S4.p6.6.m3.1.1.1.3.3.2.cmml">r</mi><mo id="S4.p6.6.m3.1.1.1.3.3.1" xref="S4.p6.6.m3.1.1.1.3.3.1.cmml">^</mo></mover></mrow><mo lspace="0.278em" rspace="0.278em" id="S4.p6.6.m3.1.1.1.2" xref="S4.p6.6.m3.1.1.1.2.cmml">:</mo><mrow id="S4.p6.6.m3.1.1.1.1" xref="S4.p6.6.m3.1.1.1.1.cmml"><mrow id="S4.p6.6.m3.1.1.1.1.1.1" xref="S4.p6.6.m3.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.p6.6.m3.1.1.1.1.1.1.2" xref="S4.p6.6.m3.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.p6.6.m3.1.1.1.1.1.1.1" xref="S4.p6.6.m3.1.1.1.1.1.1.1.cmml"><mi id="S4.p6.6.m3.1.1.1.1.1.1.1.2" xref="S4.p6.6.m3.1.1.1.1.1.1.1.2.cmml">i</mi><mo id="S4.p6.6.m3.1.1.1.1.1.1.1.1" xref="S4.p6.6.m3.1.1.1.1.1.1.1.1.cmml">+</mo><mn id="S4.p6.6.m3.1.1.1.1.1.1.1.3" xref="S4.p6.6.m3.1.1.1.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="S4.p6.6.m3.1.1.1.1.1.1.3" xref="S4.p6.6.m3.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S4.p6.6.m3.1.1.1.1.2" xref="S4.p6.6.m3.1.1.1.1.2.cmml">​</mo><mover accent="true" id="S4.p6.6.m3.1.1.1.1.3" xref="S4.p6.6.m3.1.1.1.1.3.cmml"><mi id="S4.p6.6.m3.1.1.1.1.3.2" xref="S4.p6.6.m3.1.1.1.1.3.2.cmml">r</mi><mo id="S4.p6.6.m3.1.1.1.1.3.1" xref="S4.p6.6.m3.1.1.1.1.3.1.cmml">^</mo></mover></mrow></mrow></msub><annotation-xml encoding="MathML-Content" id="S4.p6.6.m3.1b"><apply id="S4.p6.6.m3.1.2.cmml" xref="S4.p6.6.m3.1.2"><csymbol cd="ambiguous" id="S4.p6.6.m3.1.2.1.cmml" xref="S4.p6.6.m3.1.2">subscript</csymbol><ci id="S4.p6.6.m3.1.2.2.cmml" xref="S4.p6.6.m3.1.2.2">𝑥</ci><apply id="S4.p6.6.m3.1.1.1.cmml" xref="S4.p6.6.m3.1.1.1"><ci id="S4.p6.6.m3.1.1.1.2.cmml" xref="S4.p6.6.m3.1.1.1.2">:</ci><apply id="S4.p6.6.m3.1.1.1.3.cmml" xref="S4.p6.6.m3.1.1.1.3"><times id="S4.p6.6.m3.1.1.1.3.1.cmml" xref="S4.p6.6.m3.1.1.1.3.1"></times><ci id="S4.p6.6.m3.1.1.1.3.2.cmml" xref="S4.p6.6.m3.1.1.1.3.2">𝑖</ci><apply id="S4.p6.6.m3.1.1.1.3.3.cmml" xref="S4.p6.6.m3.1.1.1.3.3"><ci id="S4.p6.6.m3.1.1.1.3.3.1.cmml" xref="S4.p6.6.m3.1.1.1.3.3.1">^</ci><ci id="S4.p6.6.m3.1.1.1.3.3.2.cmml" xref="S4.p6.6.m3.1.1.1.3.3.2">𝑟</ci></apply></apply><apply id="S4.p6.6.m3.1.1.1.1.cmml" xref="S4.p6.6.m3.1.1.1.1"><times id="S4.p6.6.m3.1.1.1.1.2.cmml" xref="S4.p6.6.m3.1.1.1.1.2"></times><apply id="S4.p6.6.m3.1.1.1.1.1.1.1.cmml" xref="S4.p6.6.m3.1.1.1.1.1.1"><plus id="S4.p6.6.m3.1.1.1.1.1.1.1.1.cmml" xref="S4.p6.6.m3.1.1.1.1.1.1.1.1"></plus><ci id="S4.p6.6.m3.1.1.1.1.1.1.1.2.cmml" xref="S4.p6.6.m3.1.1.1.1.1.1.1.2">𝑖</ci><cn type="integer" id="S4.p6.6.m3.1.1.1.1.1.1.1.3.cmml" xref="S4.p6.6.m3.1.1.1.1.1.1.1.3">1</cn></apply><apply id="S4.p6.6.m3.1.1.1.1.3.cmml" xref="S4.p6.6.m3.1.1.1.1.3"><ci id="S4.p6.6.m3.1.1.1.1.3.1.cmml" xref="S4.p6.6.m3.1.1.1.1.3.1">^</ci><ci id="S4.p6.6.m3.1.1.1.1.3.2.cmml" xref="S4.p6.6.m3.1.1.1.1.3.2">𝑟</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p6.6.m3.1c">x_{i\hat{r}:(i+1)\hat{r}}</annotation></semantics></math> and <math id="S4.p6.7.m4.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S4.p6.7.m4.1a"><mi id="S4.p6.7.m4.1.1" xref="S4.p6.7.m4.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S4.p6.7.m4.1b"><ci id="S4.p6.7.m4.1.1.cmml" xref="S4.p6.7.m4.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p6.7.m4.1c">M</annotation></semantics></math> post-rotation, respectively.
Following RoPE, we use a <math id="S4.p6.8.m5.1" class="ltx_Math" alttext="\hat{r}\times\hat{r}" display="inline"><semantics id="S4.p6.8.m5.1a"><mrow id="S4.p6.8.m5.1.1" xref="S4.p6.8.m5.1.1.cmml"><mover accent="true" id="S4.p6.8.m5.1.1.2" xref="S4.p6.8.m5.1.1.2.cmml"><mi id="S4.p6.8.m5.1.1.2.2" xref="S4.p6.8.m5.1.1.2.2.cmml">r</mi><mo id="S4.p6.8.m5.1.1.2.1" xref="S4.p6.8.m5.1.1.2.1.cmml">^</mo></mover><mo lspace="0.222em" rspace="0.222em" id="S4.p6.8.m5.1.1.1" xref="S4.p6.8.m5.1.1.1.cmml">×</mo><mover accent="true" id="S4.p6.8.m5.1.1.3" xref="S4.p6.8.m5.1.1.3.cmml"><mi id="S4.p6.8.m5.1.1.3.2" xref="S4.p6.8.m5.1.1.3.2.cmml">r</mi><mo id="S4.p6.8.m5.1.1.3.1" xref="S4.p6.8.m5.1.1.3.1.cmml">^</mo></mover></mrow><annotation-xml encoding="MathML-Content" id="S4.p6.8.m5.1b"><apply id="S4.p6.8.m5.1.1.cmml" xref="S4.p6.8.m5.1.1"><times id="S4.p6.8.m5.1.1.1.cmml" xref="S4.p6.8.m5.1.1.1"></times><apply id="S4.p6.8.m5.1.1.2.cmml" xref="S4.p6.8.m5.1.1.2"><ci id="S4.p6.8.m5.1.1.2.1.cmml" xref="S4.p6.8.m5.1.1.2.1">^</ci><ci id="S4.p6.8.m5.1.1.2.2.cmml" xref="S4.p6.8.m5.1.1.2.2">𝑟</ci></apply><apply id="S4.p6.8.m5.1.1.3.cmml" xref="S4.p6.8.m5.1.1.3"><ci id="S4.p6.8.m5.1.1.3.1.cmml" xref="S4.p6.8.m5.1.1.3.1">^</ci><ci id="S4.p6.8.m5.1.1.3.2.cmml" xref="S4.p6.8.m5.1.1.3.2">𝑟</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p6.8.m5.1c">\hat{r}\times\hat{r}</annotation></semantics></math> block diagonal matrix to achieve the rotation.
However, our method use rotation information to enable <math id="S4.p6.9.m6.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S4.p6.9.m6.1a"><mi id="S4.p6.9.m6.1.1" xref="S4.p6.9.m6.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S4.p6.9.m6.1b"><ci id="S4.p6.9.m6.1.1.cmml" xref="S4.p6.9.m6.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p6.9.m6.1c">M</annotation></semantics></math> distinguish the <math id="S4.p6.10.m7.1" class="ltx_Math" alttext="x_{i\hat{r}:(i+1)\hat{r}}" display="inline"><semantics id="S4.p6.10.m7.1a"><msub id="S4.p6.10.m7.1.2" xref="S4.p6.10.m7.1.2.cmml"><mi id="S4.p6.10.m7.1.2.2" xref="S4.p6.10.m7.1.2.2.cmml">x</mi><mrow id="S4.p6.10.m7.1.1.1" xref="S4.p6.10.m7.1.1.1.cmml"><mrow id="S4.p6.10.m7.1.1.1.3" xref="S4.p6.10.m7.1.1.1.3.cmml"><mi id="S4.p6.10.m7.1.1.1.3.2" xref="S4.p6.10.m7.1.1.1.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S4.p6.10.m7.1.1.1.3.1" xref="S4.p6.10.m7.1.1.1.3.1.cmml">​</mo><mover accent="true" id="S4.p6.10.m7.1.1.1.3.3" xref="S4.p6.10.m7.1.1.1.3.3.cmml"><mi id="S4.p6.10.m7.1.1.1.3.3.2" xref="S4.p6.10.m7.1.1.1.3.3.2.cmml">r</mi><mo id="S4.p6.10.m7.1.1.1.3.3.1" xref="S4.p6.10.m7.1.1.1.3.3.1.cmml">^</mo></mover></mrow><mo lspace="0.278em" rspace="0.278em" id="S4.p6.10.m7.1.1.1.2" xref="S4.p6.10.m7.1.1.1.2.cmml">:</mo><mrow id="S4.p6.10.m7.1.1.1.1" xref="S4.p6.10.m7.1.1.1.1.cmml"><mrow id="S4.p6.10.m7.1.1.1.1.1.1" xref="S4.p6.10.m7.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.p6.10.m7.1.1.1.1.1.1.2" xref="S4.p6.10.m7.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.p6.10.m7.1.1.1.1.1.1.1" xref="S4.p6.10.m7.1.1.1.1.1.1.1.cmml"><mi id="S4.p6.10.m7.1.1.1.1.1.1.1.2" xref="S4.p6.10.m7.1.1.1.1.1.1.1.2.cmml">i</mi><mo id="S4.p6.10.m7.1.1.1.1.1.1.1.1" xref="S4.p6.10.m7.1.1.1.1.1.1.1.1.cmml">+</mo><mn id="S4.p6.10.m7.1.1.1.1.1.1.1.3" xref="S4.p6.10.m7.1.1.1.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="S4.p6.10.m7.1.1.1.1.1.1.3" xref="S4.p6.10.m7.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S4.p6.10.m7.1.1.1.1.2" xref="S4.p6.10.m7.1.1.1.1.2.cmml">​</mo><mover accent="true" id="S4.p6.10.m7.1.1.1.1.3" xref="S4.p6.10.m7.1.1.1.1.3.cmml"><mi id="S4.p6.10.m7.1.1.1.1.3.2" xref="S4.p6.10.m7.1.1.1.1.3.2.cmml">r</mi><mo id="S4.p6.10.m7.1.1.1.1.3.1" xref="S4.p6.10.m7.1.1.1.1.3.1.cmml">^</mo></mover></mrow></mrow></msub><annotation-xml encoding="MathML-Content" id="S4.p6.10.m7.1b"><apply id="S4.p6.10.m7.1.2.cmml" xref="S4.p6.10.m7.1.2"><csymbol cd="ambiguous" id="S4.p6.10.m7.1.2.1.cmml" xref="S4.p6.10.m7.1.2">subscript</csymbol><ci id="S4.p6.10.m7.1.2.2.cmml" xref="S4.p6.10.m7.1.2.2">𝑥</ci><apply id="S4.p6.10.m7.1.1.1.cmml" xref="S4.p6.10.m7.1.1.1"><ci id="S4.p6.10.m7.1.1.1.2.cmml" xref="S4.p6.10.m7.1.1.1.2">:</ci><apply id="S4.p6.10.m7.1.1.1.3.cmml" xref="S4.p6.10.m7.1.1.1.3"><times id="S4.p6.10.m7.1.1.1.3.1.cmml" xref="S4.p6.10.m7.1.1.1.3.1"></times><ci id="S4.p6.10.m7.1.1.1.3.2.cmml" xref="S4.p6.10.m7.1.1.1.3.2">𝑖</ci><apply id="S4.p6.10.m7.1.1.1.3.3.cmml" xref="S4.p6.10.m7.1.1.1.3.3"><ci id="S4.p6.10.m7.1.1.1.3.3.1.cmml" xref="S4.p6.10.m7.1.1.1.3.3.1">^</ci><ci id="S4.p6.10.m7.1.1.1.3.3.2.cmml" xref="S4.p6.10.m7.1.1.1.3.3.2">𝑟</ci></apply></apply><apply id="S4.p6.10.m7.1.1.1.1.cmml" xref="S4.p6.10.m7.1.1.1.1"><times id="S4.p6.10.m7.1.1.1.1.2.cmml" xref="S4.p6.10.m7.1.1.1.1.2"></times><apply id="S4.p6.10.m7.1.1.1.1.1.1.1.cmml" xref="S4.p6.10.m7.1.1.1.1.1.1"><plus id="S4.p6.10.m7.1.1.1.1.1.1.1.1.cmml" xref="S4.p6.10.m7.1.1.1.1.1.1.1.1"></plus><ci id="S4.p6.10.m7.1.1.1.1.1.1.1.2.cmml" xref="S4.p6.10.m7.1.1.1.1.1.1.1.2">𝑖</ci><cn type="integer" id="S4.p6.10.m7.1.1.1.1.1.1.1.3.cmml" xref="S4.p6.10.m7.1.1.1.1.1.1.1.3">1</cn></apply><apply id="S4.p6.10.m7.1.1.1.1.3.cmml" xref="S4.p6.10.m7.1.1.1.1.3"><ci id="S4.p6.10.m7.1.1.1.1.3.1.cmml" xref="S4.p6.10.m7.1.1.1.1.3.1">^</ci><ci id="S4.p6.10.m7.1.1.1.1.3.2.cmml" xref="S4.p6.10.m7.1.1.1.1.3.2">𝑟</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p6.10.m7.1c">x_{i\hat{r}:(i+1)\hat{r}}</annotation></semantics></math> instead of token position in RoPE.
We can define <math id="S4.p6.11.m8.1" class="ltx_Math" alttext="a^{i}" display="inline"><semantics id="S4.p6.11.m8.1a"><msup id="S4.p6.11.m8.1.1" xref="S4.p6.11.m8.1.1.cmml"><mi id="S4.p6.11.m8.1.1.2" xref="S4.p6.11.m8.1.1.2.cmml">a</mi><mi id="S4.p6.11.m8.1.1.3" xref="S4.p6.11.m8.1.1.3.cmml">i</mi></msup><annotation-xml encoding="MathML-Content" id="S4.p6.11.m8.1b"><apply id="S4.p6.11.m8.1.1.cmml" xref="S4.p6.11.m8.1.1"><csymbol cd="ambiguous" id="S4.p6.11.m8.1.1.1.cmml" xref="S4.p6.11.m8.1.1">superscript</csymbol><ci id="S4.p6.11.m8.1.1.2.cmml" xref="S4.p6.11.m8.1.1.2">𝑎</ci><ci id="S4.p6.11.m8.1.1.3.cmml" xref="S4.p6.11.m8.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p6.11.m8.1c">a^{i}</annotation></semantics></math> and <math id="S4.p6.12.m9.1" class="ltx_Math" alttext="P^{i}" display="inline"><semantics id="S4.p6.12.m9.1a"><msup id="S4.p6.12.m9.1.1" xref="S4.p6.12.m9.1.1.cmml"><mi id="S4.p6.12.m9.1.1.2" xref="S4.p6.12.m9.1.1.2.cmml">P</mi><mi id="S4.p6.12.m9.1.1.3" xref="S4.p6.12.m9.1.1.3.cmml">i</mi></msup><annotation-xml encoding="MathML-Content" id="S4.p6.12.m9.1b"><apply id="S4.p6.12.m9.1.1.cmml" xref="S4.p6.12.m9.1.1"><csymbol cd="ambiguous" id="S4.p6.12.m9.1.1.1.cmml" xref="S4.p6.12.m9.1.1">superscript</csymbol><ci id="S4.p6.12.m9.1.1.2.cmml" xref="S4.p6.12.m9.1.1.2">𝑃</ci><ci id="S4.p6.12.m9.1.1.3.cmml" xref="S4.p6.12.m9.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p6.12.m9.1c">P^{i}</annotation></semantics></math> as follows:</p>
<table id="S4.E10" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E10.m1.11" class="ltx_Math" alttext="\begin{split}a^{i}&amp;=\begin{bmatrix}{R}_{\theta_{1},i}&amp;\mathbf{0}&amp;\cdots&amp;\mathbf{0}\\
\mathbf{0}&{R}_{\theta_{2},i}&amp;\cdots&amp;\mathbf{0}\\
\vdots&amp;\vdots&amp;\ddots&amp;\vdots\\
\mathbf{0}&amp;\mathbf{0}&amp;\cdots&{R}_{\theta_{\frac{\hat{r}}{2}},i}\end{bmatrix}x_{i\hat{r}:(i+1)\hat{r}}\\
P^{i}&amp;=M\begin{bmatrix}{R}_{\theta_{1},i}&amp;\mathbf{0}&amp;\cdots&amp;\mathbf{0}\\
\mathbf{0}&{R}_{\theta_{2},i}&amp;\cdots&amp;\mathbf{0}\\
\vdots&amp;\vdots&amp;\ddots&amp;\vdots\\
\mathbf{0}&amp;\mathbf{0}&amp;\cdots&{R}_{\theta_{\frac{\hat{r}}{2}},i}\end{bmatrix}\end{split}" display="block"><semantics id="S4.E10.m1.11a"><mtable columnspacing="0pt" displaystyle="true" rowspacing="0pt" id="S4.E10.m1.11.11" xref="S4.E10.m1.11.12.1.cmml"><mtr id="S4.E10.m1.11.11a" xref="S4.E10.m1.11.12.1.cmml"><mtd class="ltx_align_right" columnalign="right" id="S4.E10.m1.11.11b" xref="S4.E10.m1.11.12.1.cmml"><msup id="S4.E10.m1.4.4.4.3.2" xref="S4.E10.m1.11.12.1.cmml"><mi id="S4.E10.m1.3.3.3.2.1.1" xref="S4.E10.m1.3.3.3.2.1.1.cmml">a</mi><mi id="S4.E10.m1.4.4.4.3.2.2.1" xref="S4.E10.m1.4.4.4.3.2.2.1.cmml">i</mi></msup></mtd><mtd class="ltx_align_left" columnalign="left" id="S4.E10.m1.11.11c" xref="S4.E10.m1.11.12.1.cmml"><mrow id="S4.E10.m1.7.7.7.6.4" xref="S4.E10.m1.11.12.1.cmml"><mi id="S4.E10.m1.7.7.7.6.4.5" xref="S4.E10.m1.11.12.1a.cmml"></mi><mo id="S4.E10.m1.5.5.5.4.2.2" xref="S4.E10.m1.5.5.5.4.2.2.cmml">=</mo><mrow id="S4.E10.m1.7.7.7.6.4.6" xref="S4.E10.m1.11.12.1.cmml"><mrow id="S4.E10.m1.1.1.1.1.1.1.3" xref="S4.E10.m1.1.1.1.1.1.1.2.cmml"><mo id="S4.E10.m1.1.1.1.1.1.1.3.1" xref="S4.E10.m1.1.1.1.1.1.1.2.1.cmml">[</mo><mtable columnspacing="5pt" displaystyle="true" rowspacing="0pt" id="S4.E10.m1.1.1.1.1.1.1.1.1" xref="S4.E10.m1.1.1.1.1.1.1.1.1.cmml"><mtr id="S4.E10.m1.1.1.1.1.1.1.1.1a" xref="S4.E10.m1.1.1.1.1.1.1.1.1.cmml"><mtd id="S4.E10.m1.1.1.1.1.1.1.1.1b" xref="S4.E10.m1.1.1.1.1.1.1.1.1.cmml"><msub id="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2" xref="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.cmml"><mi id="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.4" xref="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.4.cmml">R</mi><mrow id="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.2" xref="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.3.cmml"><msub id="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.2.1" xref="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.2.1.cmml"><mi id="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.2.1.2" xref="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.2.1.2.cmml">θ</mi><mn id="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.2.1.3" xref="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.2.1.3.cmml">1</mn></msub><mo id="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.2.2" xref="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.3.cmml">,</mo><mi id="S4.E10.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S4.E10.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">i</mi></mrow></msub></mtd><mtd id="S4.E10.m1.1.1.1.1.1.1.1.1c" xref="S4.E10.m1.1.1.1.1.1.1.1.1.cmml"><mn id="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.3.1" xref="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.3.1.cmml">𝟎</mn></mtd><mtd id="S4.E10.m1.1.1.1.1.1.1.1.1d" xref="S4.E10.m1.1.1.1.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.4.1" xref="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.4.1.cmml">⋯</mi></mtd><mtd id="S4.E10.m1.1.1.1.1.1.1.1.1e" xref="S4.E10.m1.1.1.1.1.1.1.1.1.cmml"><mn id="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.5.1" xref="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.5.1.cmml">𝟎</mn></mtd></mtr><mtr id="S4.E10.m1.1.1.1.1.1.1.1.1f" xref="S4.E10.m1.1.1.1.1.1.1.1.1.cmml"><mtd id="S4.E10.m1.1.1.1.1.1.1.1.1g" xref="S4.E10.m1.1.1.1.1.1.1.1.1.cmml"><mn id="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.3.1" xref="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.3.1.cmml">𝟎</mn></mtd><mtd id="S4.E10.m1.1.1.1.1.1.1.1.1h" xref="S4.E10.m1.1.1.1.1.1.1.1.1.cmml"><msub id="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2" xref="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.cmml"><mi id="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.4" xref="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.4.cmml">R</mi><mrow id="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.2.2.2" xref="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.2.2.3.cmml"><msub id="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.2.2.2.1" xref="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.2.2.2.1.cmml"><mi id="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.2.2.2.1.2" xref="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.2.2.2.1.2.cmml">θ</mi><mn id="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.2.2.2.1.3" xref="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.2.2.2.1.3.cmml">2</mn></msub><mo id="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.2.2.2.2" xref="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.2.2.3.cmml">,</mo><mi id="S4.E10.m1.1.1.1.1.1.1.1.1.3.3.1.1.1.1.1" xref="S4.E10.m1.1.1.1.1.1.1.1.1.3.3.1.1.1.1.1.cmml">i</mi></mrow></msub></mtd><mtd id="S4.E10.m1.1.1.1.1.1.1.1.1i" xref="S4.E10.m1.1.1.1.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.4.1" xref="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.4.1.cmml">⋯</mi></mtd><mtd id="S4.E10.m1.1.1.1.1.1.1.1.1j" xref="S4.E10.m1.1.1.1.1.1.1.1.1.cmml"><mn id="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.5.1" xref="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.5.1.cmml">𝟎</mn></mtd></mtr><mtr id="S4.E10.m1.1.1.1.1.1.1.1.1k" xref="S4.E10.m1.1.1.1.1.1.1.1.1.cmml"><mtd id="S4.E10.m1.1.1.1.1.1.1.1.1l" xref="S4.E10.m1.1.1.1.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E10.m1.1.1.1.1.1.1.1.1.7.1.1" xref="S4.E10.m1.1.1.1.1.1.1.1.1.7.1.1.cmml">⋮</mi></mtd><mtd id="S4.E10.m1.1.1.1.1.1.1.1.1m" xref="S4.E10.m1.1.1.1.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E10.m1.1.1.1.1.1.1.1.1.7.2.1" xref="S4.E10.m1.1.1.1.1.1.1.1.1.7.2.1.cmml">⋮</mi></mtd><mtd id="S4.E10.m1.1.1.1.1.1.1.1.1n" xref="S4.E10.m1.1.1.1.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E10.m1.1.1.1.1.1.1.1.1.7.3.1" xref="S4.E10.m1.1.1.1.1.1.1.1.1.7.3.1.cmml">⋱</mi></mtd><mtd id="S4.E10.m1.1.1.1.1.1.1.1.1o" xref="S4.E10.m1.1.1.1.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E10.m1.1.1.1.1.1.1.1.1.7.4.1" xref="S4.E10.m1.1.1.1.1.1.1.1.1.7.4.1.cmml">⋮</mi></mtd></mtr><mtr id="S4.E10.m1.1.1.1.1.1.1.1.1p" xref="S4.E10.m1.1.1.1.1.1.1.1.1.cmml"><mtd id="S4.E10.m1.1.1.1.1.1.1.1.1q" xref="S4.E10.m1.1.1.1.1.1.1.1.1.cmml"><mn id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.3.1" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.3.1.cmml">𝟎</mn></mtd><mtd id="S4.E10.m1.1.1.1.1.1.1.1.1r" xref="S4.E10.m1.1.1.1.1.1.1.1.1.cmml"><mn id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.4.1" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.4.1.cmml">𝟎</mn></mtd><mtd id="S4.E10.m1.1.1.1.1.1.1.1.1s" xref="S4.E10.m1.1.1.1.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.5.1" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.5.1.cmml">⋯</mi></mtd><mtd id="S4.E10.m1.1.1.1.1.1.1.1.1t" xref="S4.E10.m1.1.1.1.1.1.1.1.1.cmml"><msub id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.cmml"><mi id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.4" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.4.cmml">R</mi><mrow id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.3.cmml"><msub id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.cmml"><mi id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.2" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.2.cmml">θ</mi><mfrac id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3.cmml"><mover accent="true" id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2.cmml"><mi id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2.2" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2.2.cmml">r</mi><mo id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2.1" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2.1.cmml">^</mo></mover><mn id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3.3" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3.3.cmml">2</mn></mfrac></msub><mo id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.2" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.3.cmml">,</mo><mi id="S4.E10.m1.1.1.1.1.1.1.1.1.5.5.1.1.1.1.1" xref="S4.E10.m1.1.1.1.1.1.1.1.1.5.5.1.1.1.1.1.cmml">i</mi></mrow></msub></mtd></mtr></mtable><mo id="S4.E10.m1.1.1.1.1.1.1.3.2" xref="S4.E10.m1.1.1.1.1.1.1.2.1.cmml">]</mo></mrow><mo lspace="0em" rspace="0em" id="S4.E10.m1.7.7.7.6.4.6.1" xref="S4.E10.m1.11.12.1a.cmml">​</mo><msub id="S4.E10.m1.7.7.7.6.4.6.2" xref="S4.E10.m1.11.12.1.cmml"><mi id="S4.E10.m1.6.6.6.5.3.3" xref="S4.E10.m1.6.6.6.5.3.3.cmml">x</mi><mrow id="S4.E10.m1.7.7.7.6.4.4.1" xref="S4.E10.m1.7.7.7.6.4.4.1.cmml"><mrow id="S4.E10.m1.7.7.7.6.4.4.1.3" xref="S4.E10.m1.7.7.7.6.4.4.1.3.cmml"><mi id="S4.E10.m1.7.7.7.6.4.4.1.3.2" xref="S4.E10.m1.7.7.7.6.4.4.1.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S4.E10.m1.7.7.7.6.4.4.1.3.1" xref="S4.E10.m1.7.7.7.6.4.4.1.3.1.cmml">​</mo><mover accent="true" id="S4.E10.m1.7.7.7.6.4.4.1.3.3" xref="S4.E10.m1.7.7.7.6.4.4.1.3.3.cmml"><mi id="S4.E10.m1.7.7.7.6.4.4.1.3.3.2" xref="S4.E10.m1.7.7.7.6.4.4.1.3.3.2.cmml">r</mi><mo id="S4.E10.m1.7.7.7.6.4.4.1.3.3.1" xref="S4.E10.m1.7.7.7.6.4.4.1.3.3.1.cmml">^</mo></mover></mrow><mo lspace="0.278em" rspace="0.278em" id="S4.E10.m1.7.7.7.6.4.4.1.2" xref="S4.E10.m1.7.7.7.6.4.4.1.2.cmml">:</mo><mrow id="S4.E10.m1.7.7.7.6.4.4.1.1" xref="S4.E10.m1.7.7.7.6.4.4.1.1.cmml"><mrow id="S4.E10.m1.7.7.7.6.4.4.1.1.1.1" xref="S4.E10.m1.7.7.7.6.4.4.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.E10.m1.7.7.7.6.4.4.1.1.1.1.2" xref="S4.E10.m1.7.7.7.6.4.4.1.1.1.1.1.cmml">(</mo><mrow id="S4.E10.m1.7.7.7.6.4.4.1.1.1.1.1" xref="S4.E10.m1.7.7.7.6.4.4.1.1.1.1.1.cmml"><mi id="S4.E10.m1.7.7.7.6.4.4.1.1.1.1.1.2" xref="S4.E10.m1.7.7.7.6.4.4.1.1.1.1.1.2.cmml">i</mi><mo id="S4.E10.m1.7.7.7.6.4.4.1.1.1.1.1.1" xref="S4.E10.m1.7.7.7.6.4.4.1.1.1.1.1.1.cmml">+</mo><mn id="S4.E10.m1.7.7.7.6.4.4.1.1.1.1.1.3" xref="S4.E10.m1.7.7.7.6.4.4.1.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="S4.E10.m1.7.7.7.6.4.4.1.1.1.1.3" xref="S4.E10.m1.7.7.7.6.4.4.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S4.E10.m1.7.7.7.6.4.4.1.1.2" xref="S4.E10.m1.7.7.7.6.4.4.1.1.2.cmml">​</mo><mover accent="true" id="S4.E10.m1.7.7.7.6.4.4.1.1.3" xref="S4.E10.m1.7.7.7.6.4.4.1.1.3.cmml"><mi id="S4.E10.m1.7.7.7.6.4.4.1.1.3.2" xref="S4.E10.m1.7.7.7.6.4.4.1.1.3.2.cmml">r</mi><mo id="S4.E10.m1.7.7.7.6.4.4.1.1.3.1" xref="S4.E10.m1.7.7.7.6.4.4.1.1.3.1.cmml">^</mo></mover></mrow></mrow></msub></mrow></mrow></mtd></mtr><mtr id="S4.E10.m1.11.11d" xref="S4.E10.m1.11.12.1.cmml"><mtd class="ltx_align_right" columnalign="right" id="S4.E10.m1.11.11e" xref="S4.E10.m1.11.12.1.cmml"><msup id="S4.E10.m1.9.9.9.3.2" xref="S4.E10.m1.11.12.1.cmml"><mi id="S4.E10.m1.8.8.8.2.1.1" xref="S4.E10.m1.8.8.8.2.1.1.cmml">P</mi><mi id="S4.E10.m1.9.9.9.3.2.2.1" xref="S4.E10.m1.9.9.9.3.2.2.1.cmml">i</mi></msup></mtd><mtd class="ltx_align_left" columnalign="left" id="S4.E10.m1.11.11f" xref="S4.E10.m1.11.12.1.cmml"><mrow id="S4.E10.m1.11.11.11.5.3" xref="S4.E10.m1.11.12.1.cmml"><mi id="S4.E10.m1.11.11.11.5.3.4" xref="S4.E10.m1.11.12.1a.cmml"></mi><mo id="S4.E10.m1.10.10.10.4.2.2" xref="S4.E10.m1.10.10.10.4.2.2.cmml">=</mo><mrow id="S4.E10.m1.11.11.11.5.3.5" xref="S4.E10.m1.11.12.1.cmml"><mi id="S4.E10.m1.11.11.11.5.3.3" xref="S4.E10.m1.11.11.11.5.3.3.cmml">M</mi><mo lspace="0em" rspace="0em" id="S4.E10.m1.11.11.11.5.3.5.1" xref="S4.E10.m1.11.12.1a.cmml">​</mo><mrow id="S4.E10.m1.2.2.2.1.1.1.3" xref="S4.E10.m1.2.2.2.1.1.1.2.cmml"><mo id="S4.E10.m1.2.2.2.1.1.1.3.1" xref="S4.E10.m1.2.2.2.1.1.1.2.1.cmml">[</mo><mtable columnspacing="5pt" displaystyle="true" rowspacing="0pt" id="S4.E10.m1.2.2.2.1.1.1.1.1" xref="S4.E10.m1.2.2.2.1.1.1.1.1.cmml"><mtr id="S4.E10.m1.2.2.2.1.1.1.1.1a" xref="S4.E10.m1.2.2.2.1.1.1.1.1.cmml"><mtd id="S4.E10.m1.2.2.2.1.1.1.1.1b" xref="S4.E10.m1.2.2.2.1.1.1.1.1.cmml"><msub id="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2" xref="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.cmml"><mi id="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.4" xref="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.4.cmml">R</mi><mrow id="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.2.2.2" xref="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.2.2.3.cmml"><msub id="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.2.2.2.1" xref="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.2.2.2.1.cmml"><mi id="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.2.2.2.1.2" xref="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.2.2.2.1.2.cmml">θ</mi><mn id="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.2.2.2.1.3" xref="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.2.2.2.1.3.cmml">1</mn></msub><mo id="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.2.2.2.2" xref="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.2.2.3.cmml">,</mo><mi id="S4.E10.m1.2.2.2.1.1.1.1.1.1.1.1.1.1.1.1" xref="S4.E10.m1.2.2.2.1.1.1.1.1.1.1.1.1.1.1.1.cmml">i</mi></mrow></msub></mtd><mtd id="S4.E10.m1.2.2.2.1.1.1.1.1c" xref="S4.E10.m1.2.2.2.1.1.1.1.1.cmml"><mn id="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.3.1" xref="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.3.1.cmml">𝟎</mn></mtd><mtd id="S4.E10.m1.2.2.2.1.1.1.1.1d" xref="S4.E10.m1.2.2.2.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.4.1" xref="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.4.1.cmml">⋯</mi></mtd><mtd id="S4.E10.m1.2.2.2.1.1.1.1.1e" xref="S4.E10.m1.2.2.2.1.1.1.1.1.cmml"><mn id="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.5.1" xref="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.5.1.cmml">𝟎</mn></mtd></mtr><mtr id="S4.E10.m1.2.2.2.1.1.1.1.1f" xref="S4.E10.m1.2.2.2.1.1.1.1.1.cmml"><mtd id="S4.E10.m1.2.2.2.1.1.1.1.1g" xref="S4.E10.m1.2.2.2.1.1.1.1.1.cmml"><mn id="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.3.1" xref="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.3.1.cmml">𝟎</mn></mtd><mtd id="S4.E10.m1.2.2.2.1.1.1.1.1h" xref="S4.E10.m1.2.2.2.1.1.1.1.1.cmml"><msub id="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2" xref="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.cmml"><mi id="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.4" xref="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.4.cmml">R</mi><mrow id="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.2.2.2" xref="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.2.2.3.cmml"><msub id="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.2.2.2.1" xref="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.2.2.2.1.cmml"><mi id="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.2.2.2.1.2" xref="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.2.2.2.1.2.cmml">θ</mi><mn id="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.2.2.2.1.3" xref="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.2.2.2.1.3.cmml">2</mn></msub><mo id="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.2.2.2.2" xref="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.2.2.3.cmml">,</mo><mi id="S4.E10.m1.2.2.2.1.1.1.1.1.3.3.1.1.1.1.1" xref="S4.E10.m1.2.2.2.1.1.1.1.1.3.3.1.1.1.1.1.cmml">i</mi></mrow></msub></mtd><mtd id="S4.E10.m1.2.2.2.1.1.1.1.1i" xref="S4.E10.m1.2.2.2.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.4.1" xref="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.4.1.cmml">⋯</mi></mtd><mtd id="S4.E10.m1.2.2.2.1.1.1.1.1j" xref="S4.E10.m1.2.2.2.1.1.1.1.1.cmml"><mn id="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.5.1" xref="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.5.1.cmml">𝟎</mn></mtd></mtr><mtr id="S4.E10.m1.2.2.2.1.1.1.1.1k" xref="S4.E10.m1.2.2.2.1.1.1.1.1.cmml"><mtd id="S4.E10.m1.2.2.2.1.1.1.1.1l" xref="S4.E10.m1.2.2.2.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E10.m1.2.2.2.1.1.1.1.1.7.1.1" xref="S4.E10.m1.2.2.2.1.1.1.1.1.7.1.1.cmml">⋮</mi></mtd><mtd id="S4.E10.m1.2.2.2.1.1.1.1.1m" xref="S4.E10.m1.2.2.2.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E10.m1.2.2.2.1.1.1.1.1.7.2.1" xref="S4.E10.m1.2.2.2.1.1.1.1.1.7.2.1.cmml">⋮</mi></mtd><mtd id="S4.E10.m1.2.2.2.1.1.1.1.1n" xref="S4.E10.m1.2.2.2.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E10.m1.2.2.2.1.1.1.1.1.7.3.1" xref="S4.E10.m1.2.2.2.1.1.1.1.1.7.3.1.cmml">⋱</mi></mtd><mtd id="S4.E10.m1.2.2.2.1.1.1.1.1o" xref="S4.E10.m1.2.2.2.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E10.m1.2.2.2.1.1.1.1.1.7.4.1" xref="S4.E10.m1.2.2.2.1.1.1.1.1.7.4.1.cmml">⋮</mi></mtd></mtr><mtr id="S4.E10.m1.2.2.2.1.1.1.1.1p" xref="S4.E10.m1.2.2.2.1.1.1.1.1.cmml"><mtd id="S4.E10.m1.2.2.2.1.1.1.1.1q" xref="S4.E10.m1.2.2.2.1.1.1.1.1.cmml"><mn id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.3.1" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.3.1.cmml">𝟎</mn></mtd><mtd id="S4.E10.m1.2.2.2.1.1.1.1.1r" xref="S4.E10.m1.2.2.2.1.1.1.1.1.cmml"><mn id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.4.1" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.4.1.cmml">𝟎</mn></mtd><mtd id="S4.E10.m1.2.2.2.1.1.1.1.1s" xref="S4.E10.m1.2.2.2.1.1.1.1.1.cmml"><mi mathvariant="normal" id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.5.1" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.5.1.cmml">⋯</mi></mtd><mtd id="S4.E10.m1.2.2.2.1.1.1.1.1t" xref="S4.E10.m1.2.2.2.1.1.1.1.1.cmml"><msub id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.cmml"><mi id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.4" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.4.cmml">R</mi><mrow id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.3.cmml"><msub id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.cmml"><mi id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.2" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.2.cmml">θ</mi><mfrac id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3.cmml"><mover accent="true" id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2.cmml"><mi id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2.2" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2.2.cmml">r</mi><mo id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2.1" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2.1.cmml">^</mo></mover><mn id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3.3" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3.3.cmml">2</mn></mfrac></msub><mo id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.2" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.3.cmml">,</mo><mi id="S4.E10.m1.2.2.2.1.1.1.1.1.5.5.1.1.1.1.1" xref="S4.E10.m1.2.2.2.1.1.1.1.1.5.5.1.1.1.1.1.cmml">i</mi></mrow></msub></mtd></mtr></mtable><mo id="S4.E10.m1.2.2.2.1.1.1.3.2" xref="S4.E10.m1.2.2.2.1.1.1.2.1.cmml">]</mo></mrow></mrow></mrow></mtd></mtr></mtable><annotation-xml encoding="MathML-Content" id="S4.E10.m1.11b"><apply id="S4.E10.m1.11.12.1.cmml" xref="S4.E10.m1.11.11"><and id="S4.E10.m1.11.12.1a.cmml" xref="S4.E10.m1.7.7.7.6.4.5"></and><apply id="S4.E10.m1.11.12.1b.cmml" xref="S4.E10.m1.11.11"><eq id="S4.E10.m1.5.5.5.4.2.2.cmml" xref="S4.E10.m1.5.5.5.4.2.2"></eq><apply id="S4.E10.m1.11.12.1.2.cmml" xref="S4.E10.m1.11.11"><csymbol cd="ambiguous" id="S4.E10.m1.11.12.1.2.1.cmml" xref="S4.E10.m1.7.7.7.6.4.5">superscript</csymbol><ci id="S4.E10.m1.3.3.3.2.1.1.cmml" xref="S4.E10.m1.3.3.3.2.1.1">𝑎</ci><ci id="S4.E10.m1.4.4.4.3.2.2.1.cmml" xref="S4.E10.m1.4.4.4.3.2.2.1">𝑖</ci></apply><apply id="S4.E10.m1.11.12.1.4.cmml" xref="S4.E10.m1.11.11"><times id="S4.E10.m1.11.12.1.4.1.cmml" xref="S4.E10.m1.7.7.7.6.4.5"></times><apply id="S4.E10.m1.1.1.1.1.1.1.2.cmml" xref="S4.E10.m1.1.1.1.1.1.1.3"><csymbol cd="latexml" id="S4.E10.m1.1.1.1.1.1.1.2.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.3.1">matrix</csymbol><matrix id="S4.E10.m1.1.1.1.1.1.1.1.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1"><matrixrow id="S4.E10.m1.1.1.1.1.1.1.1.1a.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1"><apply id="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.3.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2">subscript</csymbol><ci id="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.4.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.4">𝑅</ci><list id="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.3.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.2"><apply id="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.2.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.2.1"><csymbol cd="ambiguous" id="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.2.1.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.2.1">subscript</csymbol><ci id="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.2.1.2.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.2.1.2">𝜃</ci><cn type="integer" id="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.2.1.3.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.2.1.3">1</cn></apply><ci id="S4.E10.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1">𝑖</ci></list></apply><cn type="integer" id="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.3.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.3.1">0</cn><ci id="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.4.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.4.1">⋯</ci><cn type="integer" id="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.5.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.2.2.5.1">0</cn></matrixrow><matrixrow id="S4.E10.m1.1.1.1.1.1.1.1.1b.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1"><cn type="integer" id="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.3.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.3.1">0</cn><apply id="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2"><csymbol cd="ambiguous" id="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.3.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2">subscript</csymbol><ci id="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.4.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.4">𝑅</ci><list id="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.2.2.3.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.2.2.2"><apply id="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.2.2.2.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.2.2.2.1"><csymbol cd="ambiguous" id="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.2.2.2.1.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.2.2.2.1">subscript</csymbol><ci id="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.2.2.2.1.2.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.2.2.2.1.2">𝜃</ci><cn type="integer" id="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.2.2.2.1.3.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.2.2.2.2.2.1.3">2</cn></apply><ci id="S4.E10.m1.1.1.1.1.1.1.1.1.3.3.1.1.1.1.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.3.3.1.1.1.1.1">𝑖</ci></list></apply><ci id="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.4.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.4.1">⋯</ci><cn type="integer" id="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.5.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.4.4.5.1">0</cn></matrixrow><matrixrow id="S4.E10.m1.1.1.1.1.1.1.1.1c.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1"><ci id="S4.E10.m1.1.1.1.1.1.1.1.1.7.1.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.7.1.1">⋮</ci><ci id="S4.E10.m1.1.1.1.1.1.1.1.1.7.2.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.7.2.1">⋮</ci><ci id="S4.E10.m1.1.1.1.1.1.1.1.1.7.3.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.7.3.1">⋱</ci><ci id="S4.E10.m1.1.1.1.1.1.1.1.1.7.4.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.7.4.1">⋮</ci></matrixrow><matrixrow id="S4.E10.m1.1.1.1.1.1.1.1.1d.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1"><cn type="integer" id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.3.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.3.1">0</cn><cn type="integer" id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.4.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.4.1">0</cn><ci id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.5.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.5.1">⋯</ci><apply id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2"><csymbol cd="ambiguous" id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.3.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2">subscript</csymbol><ci id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.4.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.4">𝑅</ci><list id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.3.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2"><apply id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1"><csymbol cd="ambiguous" id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1">subscript</csymbol><ci id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.2.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.2">𝜃</ci><apply id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3"><divide id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3"></divide><apply id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2"><ci id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2.1">^</ci><ci id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2.2.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2.2">𝑟</ci></apply><cn type="integer" id="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3.3.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.6.6.2.2.2.2.2.1.3.3">2</cn></apply></apply><ci id="S4.E10.m1.1.1.1.1.1.1.1.1.5.5.1.1.1.1.1.cmml" xref="S4.E10.m1.1.1.1.1.1.1.1.1.5.5.1.1.1.1.1">𝑖</ci></list></apply></matrixrow></matrix></apply><apply id="S4.E10.m1.11.12.1.4.3.cmml" xref="S4.E10.m1.11.11"><csymbol cd="ambiguous" id="S4.E10.m1.11.12.1.4.3.1.cmml" xref="S4.E10.m1.7.7.7.6.4.5">subscript</csymbol><ci id="S4.E10.m1.6.6.6.5.3.3.cmml" xref="S4.E10.m1.6.6.6.5.3.3">𝑥</ci><apply id="S4.E10.m1.7.7.7.6.4.4.1.cmml" xref="S4.E10.m1.7.7.7.6.4.4.1"><ci id="S4.E10.m1.7.7.7.6.4.4.1.2.cmml" xref="S4.E10.m1.7.7.7.6.4.4.1.2">:</ci><apply id="S4.E10.m1.7.7.7.6.4.4.1.3.cmml" xref="S4.E10.m1.7.7.7.6.4.4.1.3"><times id="S4.E10.m1.7.7.7.6.4.4.1.3.1.cmml" xref="S4.E10.m1.7.7.7.6.4.4.1.3.1"></times><ci id="S4.E10.m1.7.7.7.6.4.4.1.3.2.cmml" xref="S4.E10.m1.7.7.7.6.4.4.1.3.2">𝑖</ci><apply id="S4.E10.m1.7.7.7.6.4.4.1.3.3.cmml" xref="S4.E10.m1.7.7.7.6.4.4.1.3.3"><ci id="S4.E10.m1.7.7.7.6.4.4.1.3.3.1.cmml" xref="S4.E10.m1.7.7.7.6.4.4.1.3.3.1">^</ci><ci id="S4.E10.m1.7.7.7.6.4.4.1.3.3.2.cmml" xref="S4.E10.m1.7.7.7.6.4.4.1.3.3.2">𝑟</ci></apply></apply><apply id="S4.E10.m1.7.7.7.6.4.4.1.1.cmml" xref="S4.E10.m1.7.7.7.6.4.4.1.1"><times id="S4.E10.m1.7.7.7.6.4.4.1.1.2.cmml" xref="S4.E10.m1.7.7.7.6.4.4.1.1.2"></times><apply id="S4.E10.m1.7.7.7.6.4.4.1.1.1.1.1.cmml" xref="S4.E10.m1.7.7.7.6.4.4.1.1.1.1"><plus id="S4.E10.m1.7.7.7.6.4.4.1.1.1.1.1.1.cmml" xref="S4.E10.m1.7.7.7.6.4.4.1.1.1.1.1.1"></plus><ci id="S4.E10.m1.7.7.7.6.4.4.1.1.1.1.1.2.cmml" xref="S4.E10.m1.7.7.7.6.4.4.1.1.1.1.1.2">𝑖</ci><cn type="integer" id="S4.E10.m1.7.7.7.6.4.4.1.1.1.1.1.3.cmml" xref="S4.E10.m1.7.7.7.6.4.4.1.1.1.1.1.3">1</cn></apply><apply id="S4.E10.m1.7.7.7.6.4.4.1.1.3.cmml" xref="S4.E10.m1.7.7.7.6.4.4.1.1.3"><ci id="S4.E10.m1.7.7.7.6.4.4.1.1.3.1.cmml" xref="S4.E10.m1.7.7.7.6.4.4.1.1.3.1">^</ci><ci id="S4.E10.m1.7.7.7.6.4.4.1.1.3.2.cmml" xref="S4.E10.m1.7.7.7.6.4.4.1.1.3.2">𝑟</ci></apply></apply></apply></apply><apply id="S4.E10.m1.11.12.1.4.4.cmml" xref="S4.E10.m1.11.11"><csymbol cd="ambiguous" id="S4.E10.m1.11.12.1.4.4.1.cmml" xref="S4.E10.m1.7.7.7.6.4.5">superscript</csymbol><ci id="S4.E10.m1.8.8.8.2.1.1.cmml" xref="S4.E10.m1.8.8.8.2.1.1">𝑃</ci><ci id="S4.E10.m1.9.9.9.3.2.2.1.cmml" xref="S4.E10.m1.9.9.9.3.2.2.1">𝑖</ci></apply></apply></apply><apply id="S4.E10.m1.11.12.1c.cmml" xref="S4.E10.m1.11.11"><eq id="S4.E10.m1.10.10.10.4.2.2.cmml" xref="S4.E10.m1.10.10.10.4.2.2"></eq><share href="#S4.E10.m1.11.12.1.4.cmml" id="S4.E10.m1.11.12.1d.cmml" xref="S4.E10.m1.7.7.7.6.4.5"></share><apply id="S4.E10.m1.11.12.1.6.cmml" xref="S4.E10.m1.11.11"><times id="S4.E10.m1.11.12.1.6.1.cmml" xref="S4.E10.m1.7.7.7.6.4.5"></times><ci id="S4.E10.m1.11.11.11.5.3.3.cmml" xref="S4.E10.m1.11.11.11.5.3.3">𝑀</ci><apply id="S4.E10.m1.2.2.2.1.1.1.2.cmml" xref="S4.E10.m1.2.2.2.1.1.1.3"><csymbol cd="latexml" id="S4.E10.m1.2.2.2.1.1.1.2.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.3.1">matrix</csymbol><matrix id="S4.E10.m1.2.2.2.1.1.1.1.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1"><matrixrow id="S4.E10.m1.2.2.2.1.1.1.1.1a.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1"><apply id="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.3.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2">subscript</csymbol><ci id="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.4.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.4">𝑅</ci><list id="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.2.2.3.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.2.2.2"><apply id="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.2.2.2.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.2.2.2.1"><csymbol cd="ambiguous" id="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.2.2.2.1.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.2.2.2.1">subscript</csymbol><ci id="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.2.2.2.1.2.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.2.2.2.1.2">𝜃</ci><cn type="integer" id="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.2.2.2.1.3.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.2.2.2.2.2.1.3">1</cn></apply><ci id="S4.E10.m1.2.2.2.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.1.1.1.1.1.1.1">𝑖</ci></list></apply><cn type="integer" id="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.3.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.3.1">0</cn><ci id="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.4.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.4.1">⋯</ci><cn type="integer" id="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.5.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.2.2.5.1">0</cn></matrixrow><matrixrow id="S4.E10.m1.2.2.2.1.1.1.1.1b.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1"><cn type="integer" id="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.3.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.3.1">0</cn><apply id="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2"><csymbol cd="ambiguous" id="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.3.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2">subscript</csymbol><ci id="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.4.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.4">𝑅</ci><list id="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.2.2.3.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.2.2.2"><apply id="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.2.2.2.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.2.2.2.1"><csymbol cd="ambiguous" id="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.2.2.2.1.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.2.2.2.1">subscript</csymbol><ci id="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.2.2.2.1.2.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.2.2.2.1.2">𝜃</ci><cn type="integer" id="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.2.2.2.1.3.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.2.2.2.2.2.1.3">2</cn></apply><ci id="S4.E10.m1.2.2.2.1.1.1.1.1.3.3.1.1.1.1.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.3.3.1.1.1.1.1">𝑖</ci></list></apply><ci id="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.4.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.4.1">⋯</ci><cn type="integer" id="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.5.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.4.4.5.1">0</cn></matrixrow><matrixrow id="S4.E10.m1.2.2.2.1.1.1.1.1c.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1"><ci id="S4.E10.m1.2.2.2.1.1.1.1.1.7.1.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.7.1.1">⋮</ci><ci id="S4.E10.m1.2.2.2.1.1.1.1.1.7.2.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.7.2.1">⋮</ci><ci id="S4.E10.m1.2.2.2.1.1.1.1.1.7.3.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.7.3.1">⋱</ci><ci id="S4.E10.m1.2.2.2.1.1.1.1.1.7.4.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.7.4.1">⋮</ci></matrixrow><matrixrow id="S4.E10.m1.2.2.2.1.1.1.1.1d.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1"><cn type="integer" id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.3.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.3.1">0</cn><cn type="integer" id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.4.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.4.1">0</cn><ci id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.5.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.5.1">⋯</ci><apply id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2"><csymbol cd="ambiguous" id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.3.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2">subscript</csymbol><ci id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.4.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.4">𝑅</ci><list id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.3.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2"><apply id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1"><csymbol cd="ambiguous" id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1">subscript</csymbol><ci id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.2.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.2">𝜃</ci><apply id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3"><divide id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3"></divide><apply id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2"><ci id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2.1">^</ci><ci id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2.2.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3.2.2">𝑟</ci></apply><cn type="integer" id="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3.3.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.6.6.2.2.2.2.2.1.3.3">2</cn></apply></apply><ci id="S4.E10.m1.2.2.2.1.1.1.1.1.5.5.1.1.1.1.1.cmml" xref="S4.E10.m1.2.2.2.1.1.1.1.1.5.5.1.1.1.1.1">𝑖</ci></list></apply></matrixrow></matrix></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E10.m1.11c">\begin{split}a^{i}&amp;=\begin{bmatrix}{R}_{\theta_{1},i}&amp;\mathbf{0}&amp;\cdots&amp;\mathbf{0}\\
\mathbf{0}&amp;{R}_{\theta_{2},i}&amp;\cdots&amp;\mathbf{0}\\
\vdots&amp;\vdots&amp;\ddots&amp;\vdots\\
\mathbf{0}&amp;\mathbf{0}&amp;\cdots&amp;{R}_{\theta_{\frac{\hat{r}}{2}},i}\end{bmatrix}x_{i\hat{r}:(i+1)\hat{r}}\\
P^{i}&amp;=M\begin{bmatrix}{R}_{\theta_{1},i}&amp;\mathbf{0}&amp;\cdots&amp;\mathbf{0}\\
\mathbf{0}&amp;{R}_{\theta_{2},i}&amp;\cdots&amp;\mathbf{0}\\
\vdots&amp;\vdots&amp;\ddots&amp;\vdots\\
\mathbf{0}&amp;\mathbf{0}&amp;\cdots&amp;{R}_{\theta_{\frac{\hat{r}}{2}},i}\end{bmatrix}\end{split}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(10)</span></td>
</tr></tbody>
</table>
<p id="S4.p6.14" class="ltx_p">where <math id="S4.p6.13.m1.1" class="ltx_Math" alttext="\theta_{j}=10000^{-2(j-1)/\hat{r}}" display="inline"><semantics id="S4.p6.13.m1.1a"><mrow id="S4.p6.13.m1.1.2" xref="S4.p6.13.m1.1.2.cmml"><msub id="S4.p6.13.m1.1.2.2" xref="S4.p6.13.m1.1.2.2.cmml"><mi id="S4.p6.13.m1.1.2.2.2" xref="S4.p6.13.m1.1.2.2.2.cmml">θ</mi><mi id="S4.p6.13.m1.1.2.2.3" xref="S4.p6.13.m1.1.2.2.3.cmml">j</mi></msub><mo id="S4.p6.13.m1.1.2.1" xref="S4.p6.13.m1.1.2.1.cmml">=</mo><msup id="S4.p6.13.m1.1.2.3" xref="S4.p6.13.m1.1.2.3.cmml"><mn id="S4.p6.13.m1.1.2.3.2" xref="S4.p6.13.m1.1.2.3.2.cmml">10000</mn><mrow id="S4.p6.13.m1.1.1.1" xref="S4.p6.13.m1.1.1.1.cmml"><mo id="S4.p6.13.m1.1.1.1a" xref="S4.p6.13.m1.1.1.1.cmml">−</mo><mrow id="S4.p6.13.m1.1.1.1.1" xref="S4.p6.13.m1.1.1.1.1.cmml"><mrow id="S4.p6.13.m1.1.1.1.1.1" xref="S4.p6.13.m1.1.1.1.1.1.cmml"><mn id="S4.p6.13.m1.1.1.1.1.1.3" xref="S4.p6.13.m1.1.1.1.1.1.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="S4.p6.13.m1.1.1.1.1.1.2" xref="S4.p6.13.m1.1.1.1.1.1.2.cmml">​</mo><mrow id="S4.p6.13.m1.1.1.1.1.1.1.1" xref="S4.p6.13.m1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.p6.13.m1.1.1.1.1.1.1.1.2" xref="S4.p6.13.m1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.p6.13.m1.1.1.1.1.1.1.1.1" xref="S4.p6.13.m1.1.1.1.1.1.1.1.1.cmml"><mi id="S4.p6.13.m1.1.1.1.1.1.1.1.1.2" xref="S4.p6.13.m1.1.1.1.1.1.1.1.1.2.cmml">j</mi><mo id="S4.p6.13.m1.1.1.1.1.1.1.1.1.1" xref="S4.p6.13.m1.1.1.1.1.1.1.1.1.1.cmml">−</mo><mn id="S4.p6.13.m1.1.1.1.1.1.1.1.1.3" xref="S4.p6.13.m1.1.1.1.1.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="S4.p6.13.m1.1.1.1.1.1.1.1.3" xref="S4.p6.13.m1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S4.p6.13.m1.1.1.1.1.2" xref="S4.p6.13.m1.1.1.1.1.2.cmml">/</mo><mover accent="true" id="S4.p6.13.m1.1.1.1.1.3" xref="S4.p6.13.m1.1.1.1.1.3.cmml"><mi id="S4.p6.13.m1.1.1.1.1.3.2" xref="S4.p6.13.m1.1.1.1.1.3.2.cmml">r</mi><mo id="S4.p6.13.m1.1.1.1.1.3.1" xref="S4.p6.13.m1.1.1.1.1.3.1.cmml">^</mo></mover></mrow></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.p6.13.m1.1b"><apply id="S4.p6.13.m1.1.2.cmml" xref="S4.p6.13.m1.1.2"><eq id="S4.p6.13.m1.1.2.1.cmml" xref="S4.p6.13.m1.1.2.1"></eq><apply id="S4.p6.13.m1.1.2.2.cmml" xref="S4.p6.13.m1.1.2.2"><csymbol cd="ambiguous" id="S4.p6.13.m1.1.2.2.1.cmml" xref="S4.p6.13.m1.1.2.2">subscript</csymbol><ci id="S4.p6.13.m1.1.2.2.2.cmml" xref="S4.p6.13.m1.1.2.2.2">𝜃</ci><ci id="S4.p6.13.m1.1.2.2.3.cmml" xref="S4.p6.13.m1.1.2.2.3">𝑗</ci></apply><apply id="S4.p6.13.m1.1.2.3.cmml" xref="S4.p6.13.m1.1.2.3"><csymbol cd="ambiguous" id="S4.p6.13.m1.1.2.3.1.cmml" xref="S4.p6.13.m1.1.2.3">superscript</csymbol><cn type="integer" id="S4.p6.13.m1.1.2.3.2.cmml" xref="S4.p6.13.m1.1.2.3.2">10000</cn><apply id="S4.p6.13.m1.1.1.1.cmml" xref="S4.p6.13.m1.1.1.1"><minus id="S4.p6.13.m1.1.1.1.2.cmml" xref="S4.p6.13.m1.1.1.1"></minus><apply id="S4.p6.13.m1.1.1.1.1.cmml" xref="S4.p6.13.m1.1.1.1.1"><divide id="S4.p6.13.m1.1.1.1.1.2.cmml" xref="S4.p6.13.m1.1.1.1.1.2"></divide><apply id="S4.p6.13.m1.1.1.1.1.1.cmml" xref="S4.p6.13.m1.1.1.1.1.1"><times id="S4.p6.13.m1.1.1.1.1.1.2.cmml" xref="S4.p6.13.m1.1.1.1.1.1.2"></times><cn type="integer" id="S4.p6.13.m1.1.1.1.1.1.3.cmml" xref="S4.p6.13.m1.1.1.1.1.1.3">2</cn><apply id="S4.p6.13.m1.1.1.1.1.1.1.1.1.cmml" xref="S4.p6.13.m1.1.1.1.1.1.1.1"><minus id="S4.p6.13.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.p6.13.m1.1.1.1.1.1.1.1.1.1"></minus><ci id="S4.p6.13.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.p6.13.m1.1.1.1.1.1.1.1.1.2">𝑗</ci><cn type="integer" id="S4.p6.13.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S4.p6.13.m1.1.1.1.1.1.1.1.1.3">1</cn></apply></apply><apply id="S4.p6.13.m1.1.1.1.1.3.cmml" xref="S4.p6.13.m1.1.1.1.1.3"><ci id="S4.p6.13.m1.1.1.1.1.3.1.cmml" xref="S4.p6.13.m1.1.1.1.1.3.1">^</ci><ci id="S4.p6.13.m1.1.1.1.1.3.2.cmml" xref="S4.p6.13.m1.1.1.1.1.3.2">𝑟</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p6.13.m1.1c">\theta_{j}=10000^{-2(j-1)/\hat{r}}</annotation></semantics></math> and <math id="S4.p6.14.m2.2" class="ltx_Math" alttext="{R}_{\theta_{j},i}\in\mathbb{R}^{2\times 2}" display="inline"><semantics id="S4.p6.14.m2.2a"><mrow id="S4.p6.14.m2.2.3" xref="S4.p6.14.m2.2.3.cmml"><msub id="S4.p6.14.m2.2.3.2" xref="S4.p6.14.m2.2.3.2.cmml"><mi id="S4.p6.14.m2.2.3.2.2" xref="S4.p6.14.m2.2.3.2.2.cmml">R</mi><mrow id="S4.p6.14.m2.2.2.2.2" xref="S4.p6.14.m2.2.2.2.3.cmml"><msub id="S4.p6.14.m2.2.2.2.2.1" xref="S4.p6.14.m2.2.2.2.2.1.cmml"><mi id="S4.p6.14.m2.2.2.2.2.1.2" xref="S4.p6.14.m2.2.2.2.2.1.2.cmml">θ</mi><mi id="S4.p6.14.m2.2.2.2.2.1.3" xref="S4.p6.14.m2.2.2.2.2.1.3.cmml">j</mi></msub><mo id="S4.p6.14.m2.2.2.2.2.2" xref="S4.p6.14.m2.2.2.2.3.cmml">,</mo><mi id="S4.p6.14.m2.1.1.1.1" xref="S4.p6.14.m2.1.1.1.1.cmml">i</mi></mrow></msub><mo id="S4.p6.14.m2.2.3.1" xref="S4.p6.14.m2.2.3.1.cmml">∈</mo><msup id="S4.p6.14.m2.2.3.3" xref="S4.p6.14.m2.2.3.3.cmml"><mi id="S4.p6.14.m2.2.3.3.2" xref="S4.p6.14.m2.2.3.3.2.cmml">ℝ</mi><mrow id="S4.p6.14.m2.2.3.3.3" xref="S4.p6.14.m2.2.3.3.3.cmml"><mn id="S4.p6.14.m2.2.3.3.3.2" xref="S4.p6.14.m2.2.3.3.3.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="S4.p6.14.m2.2.3.3.3.1" xref="S4.p6.14.m2.2.3.3.3.1.cmml">×</mo><mn id="S4.p6.14.m2.2.3.3.3.3" xref="S4.p6.14.m2.2.3.3.3.3.cmml">2</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.p6.14.m2.2b"><apply id="S4.p6.14.m2.2.3.cmml" xref="S4.p6.14.m2.2.3"><in id="S4.p6.14.m2.2.3.1.cmml" xref="S4.p6.14.m2.2.3.1"></in><apply id="S4.p6.14.m2.2.3.2.cmml" xref="S4.p6.14.m2.2.3.2"><csymbol cd="ambiguous" id="S4.p6.14.m2.2.3.2.1.cmml" xref="S4.p6.14.m2.2.3.2">subscript</csymbol><ci id="S4.p6.14.m2.2.3.2.2.cmml" xref="S4.p6.14.m2.2.3.2.2">𝑅</ci><list id="S4.p6.14.m2.2.2.2.3.cmml" xref="S4.p6.14.m2.2.2.2.2"><apply id="S4.p6.14.m2.2.2.2.2.1.cmml" xref="S4.p6.14.m2.2.2.2.2.1"><csymbol cd="ambiguous" id="S4.p6.14.m2.2.2.2.2.1.1.cmml" xref="S4.p6.14.m2.2.2.2.2.1">subscript</csymbol><ci id="S4.p6.14.m2.2.2.2.2.1.2.cmml" xref="S4.p6.14.m2.2.2.2.2.1.2">𝜃</ci><ci id="S4.p6.14.m2.2.2.2.2.1.3.cmml" xref="S4.p6.14.m2.2.2.2.2.1.3">𝑗</ci></apply><ci id="S4.p6.14.m2.1.1.1.1.cmml" xref="S4.p6.14.m2.1.1.1.1">𝑖</ci></list></apply><apply id="S4.p6.14.m2.2.3.3.cmml" xref="S4.p6.14.m2.2.3.3"><csymbol cd="ambiguous" id="S4.p6.14.m2.2.3.3.1.cmml" xref="S4.p6.14.m2.2.3.3">superscript</csymbol><ci id="S4.p6.14.m2.2.3.3.2.cmml" xref="S4.p6.14.m2.2.3.3.2">ℝ</ci><apply id="S4.p6.14.m2.2.3.3.3.cmml" xref="S4.p6.14.m2.2.3.3.3"><times id="S4.p6.14.m2.2.3.3.3.1.cmml" xref="S4.p6.14.m2.2.3.3.3.1"></times><cn type="integer" id="S4.p6.14.m2.2.3.3.3.2.cmml" xref="S4.p6.14.m2.2.3.3.3.2">2</cn><cn type="integer" id="S4.p6.14.m2.2.3.3.3.3.cmml" xref="S4.p6.14.m2.2.3.3.3.3">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p6.14.m2.2c">{R}_{\theta_{j},i}\in\mathbb{R}^{2\times 2}</annotation></semantics></math> is a rotation matrix:</p>
<table id="S4.E11" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E11.m1.3" class="ltx_Math" alttext="{R}_{\theta_{j},i}=\begin{bmatrix}\cos i\theta_{j}&amp;-\sin i\theta_{j}\\
\sin i\theta_{j}&amp;\cos i\theta_{j}\end{bmatrix}\\
" display="block"><semantics id="S4.E11.m1.3a"><mrow id="S4.E11.m1.3.4" xref="S4.E11.m1.3.4.cmml"><msub id="S4.E11.m1.3.4.2" xref="S4.E11.m1.3.4.2.cmml"><mi id="S4.E11.m1.3.4.2.2" xref="S4.E11.m1.3.4.2.2.cmml">R</mi><mrow id="S4.E11.m1.3.3.2.2" xref="S4.E11.m1.3.3.2.3.cmml"><msub id="S4.E11.m1.3.3.2.2.1" xref="S4.E11.m1.3.3.2.2.1.cmml"><mi id="S4.E11.m1.3.3.2.2.1.2" xref="S4.E11.m1.3.3.2.2.1.2.cmml">θ</mi><mi id="S4.E11.m1.3.3.2.2.1.3" xref="S4.E11.m1.3.3.2.2.1.3.cmml">j</mi></msub><mo id="S4.E11.m1.3.3.2.2.2" xref="S4.E11.m1.3.3.2.3.cmml">,</mo><mi id="S4.E11.m1.2.2.1.1" xref="S4.E11.m1.2.2.1.1.cmml">i</mi></mrow></msub><mo id="S4.E11.m1.3.4.1" xref="S4.E11.m1.3.4.1.cmml">=</mo><mrow id="S4.E11.m1.1.1.3" xref="S4.E11.m1.1.1.2.cmml"><mo id="S4.E11.m1.1.1.3.1" xref="S4.E11.m1.1.1.2.1.cmml">[</mo><mtable columnspacing="5pt" displaystyle="true" rowspacing="0pt" id="S4.E11.m1.1.1.1.1" xref="S4.E11.m1.1.1.1.1.cmml"><mtr id="S4.E11.m1.1.1.1.1a" xref="S4.E11.m1.1.1.1.1.cmml"><mtd id="S4.E11.m1.1.1.1.1b" xref="S4.E11.m1.1.1.1.1.cmml"><mrow id="S4.E11.m1.1.1.1.1.1.1.1" xref="S4.E11.m1.1.1.1.1.1.1.1.cmml"><mi id="S4.E11.m1.1.1.1.1.1.1.1.1" xref="S4.E11.m1.1.1.1.1.1.1.1.1.cmml">cos</mi><mo lspace="0.167em" id="S4.E11.m1.1.1.1.1.1.1.1a" xref="S4.E11.m1.1.1.1.1.1.1.1.cmml">⁡</mo><mrow id="S4.E11.m1.1.1.1.1.1.1.1.2" xref="S4.E11.m1.1.1.1.1.1.1.1.2.cmml"><mi id="S4.E11.m1.1.1.1.1.1.1.1.2.2" xref="S4.E11.m1.1.1.1.1.1.1.1.2.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S4.E11.m1.1.1.1.1.1.1.1.2.1" xref="S4.E11.m1.1.1.1.1.1.1.1.2.1.cmml">​</mo><msub id="S4.E11.m1.1.1.1.1.1.1.1.2.3" xref="S4.E11.m1.1.1.1.1.1.1.1.2.3.cmml"><mi id="S4.E11.m1.1.1.1.1.1.1.1.2.3.2" xref="S4.E11.m1.1.1.1.1.1.1.1.2.3.2.cmml">θ</mi><mi id="S4.E11.m1.1.1.1.1.1.1.1.2.3.3" xref="S4.E11.m1.1.1.1.1.1.1.1.2.3.3.cmml">j</mi></msub></mrow></mrow></mtd><mtd id="S4.E11.m1.1.1.1.1c" xref="S4.E11.m1.1.1.1.1.cmml"><mrow id="S4.E11.m1.1.1.1.1.1.2.1" xref="S4.E11.m1.1.1.1.1.1.2.1.cmml"><mo rspace="0.167em" id="S4.E11.m1.1.1.1.1.1.2.1a" xref="S4.E11.m1.1.1.1.1.1.2.1.cmml">−</mo><mrow id="S4.E11.m1.1.1.1.1.1.2.1.2" xref="S4.E11.m1.1.1.1.1.1.2.1.2.cmml"><mi id="S4.E11.m1.1.1.1.1.1.2.1.2.1" xref="S4.E11.m1.1.1.1.1.1.2.1.2.1.cmml">sin</mi><mo lspace="0.167em" id="S4.E11.m1.1.1.1.1.1.2.1.2a" xref="S4.E11.m1.1.1.1.1.1.2.1.2.cmml">⁡</mo><mrow id="S4.E11.m1.1.1.1.1.1.2.1.2.2" xref="S4.E11.m1.1.1.1.1.1.2.1.2.2.cmml"><mi id="S4.E11.m1.1.1.1.1.1.2.1.2.2.2" xref="S4.E11.m1.1.1.1.1.1.2.1.2.2.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S4.E11.m1.1.1.1.1.1.2.1.2.2.1" xref="S4.E11.m1.1.1.1.1.1.2.1.2.2.1.cmml">​</mo><msub id="S4.E11.m1.1.1.1.1.1.2.1.2.2.3" xref="S4.E11.m1.1.1.1.1.1.2.1.2.2.3.cmml"><mi id="S4.E11.m1.1.1.1.1.1.2.1.2.2.3.2" xref="S4.E11.m1.1.1.1.1.1.2.1.2.2.3.2.cmml">θ</mi><mi id="S4.E11.m1.1.1.1.1.1.2.1.2.2.3.3" xref="S4.E11.m1.1.1.1.1.1.2.1.2.2.3.3.cmml">j</mi></msub></mrow></mrow></mrow></mtd></mtr><mtr id="S4.E11.m1.1.1.1.1d" xref="S4.E11.m1.1.1.1.1.cmml"><mtd id="S4.E11.m1.1.1.1.1e" xref="S4.E11.m1.1.1.1.1.cmml"><mrow id="S4.E11.m1.1.1.1.1.2.1.1" xref="S4.E11.m1.1.1.1.1.2.1.1.cmml"><mi id="S4.E11.m1.1.1.1.1.2.1.1.1" xref="S4.E11.m1.1.1.1.1.2.1.1.1.cmml">sin</mi><mo lspace="0.167em" id="S4.E11.m1.1.1.1.1.2.1.1a" xref="S4.E11.m1.1.1.1.1.2.1.1.cmml">⁡</mo><mrow id="S4.E11.m1.1.1.1.1.2.1.1.2" xref="S4.E11.m1.1.1.1.1.2.1.1.2.cmml"><mi id="S4.E11.m1.1.1.1.1.2.1.1.2.2" xref="S4.E11.m1.1.1.1.1.2.1.1.2.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S4.E11.m1.1.1.1.1.2.1.1.2.1" xref="S4.E11.m1.1.1.1.1.2.1.1.2.1.cmml">​</mo><msub id="S4.E11.m1.1.1.1.1.2.1.1.2.3" xref="S4.E11.m1.1.1.1.1.2.1.1.2.3.cmml"><mi id="S4.E11.m1.1.1.1.1.2.1.1.2.3.2" xref="S4.E11.m1.1.1.1.1.2.1.1.2.3.2.cmml">θ</mi><mi id="S4.E11.m1.1.1.1.1.2.1.1.2.3.3" xref="S4.E11.m1.1.1.1.1.2.1.1.2.3.3.cmml">j</mi></msub></mrow></mrow></mtd><mtd id="S4.E11.m1.1.1.1.1f" xref="S4.E11.m1.1.1.1.1.cmml"><mrow id="S4.E11.m1.1.1.1.1.2.2.1" xref="S4.E11.m1.1.1.1.1.2.2.1.cmml"><mi id="S4.E11.m1.1.1.1.1.2.2.1.1" xref="S4.E11.m1.1.1.1.1.2.2.1.1.cmml">cos</mi><mo lspace="0.167em" id="S4.E11.m1.1.1.1.1.2.2.1a" xref="S4.E11.m1.1.1.1.1.2.2.1.cmml">⁡</mo><mrow id="S4.E11.m1.1.1.1.1.2.2.1.2" xref="S4.E11.m1.1.1.1.1.2.2.1.2.cmml"><mi id="S4.E11.m1.1.1.1.1.2.2.1.2.2" xref="S4.E11.m1.1.1.1.1.2.2.1.2.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S4.E11.m1.1.1.1.1.2.2.1.2.1" xref="S4.E11.m1.1.1.1.1.2.2.1.2.1.cmml">​</mo><msub id="S4.E11.m1.1.1.1.1.2.2.1.2.3" xref="S4.E11.m1.1.1.1.1.2.2.1.2.3.cmml"><mi id="S4.E11.m1.1.1.1.1.2.2.1.2.3.2" xref="S4.E11.m1.1.1.1.1.2.2.1.2.3.2.cmml">θ</mi><mi id="S4.E11.m1.1.1.1.1.2.2.1.2.3.3" xref="S4.E11.m1.1.1.1.1.2.2.1.2.3.3.cmml">j</mi></msub></mrow></mrow></mtd></mtr></mtable><mo id="S4.E11.m1.1.1.3.2" xref="S4.E11.m1.1.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.E11.m1.3b"><apply id="S4.E11.m1.3.4.cmml" xref="S4.E11.m1.3.4"><eq id="S4.E11.m1.3.4.1.cmml" xref="S4.E11.m1.3.4.1"></eq><apply id="S4.E11.m1.3.4.2.cmml" xref="S4.E11.m1.3.4.2"><csymbol cd="ambiguous" id="S4.E11.m1.3.4.2.1.cmml" xref="S4.E11.m1.3.4.2">subscript</csymbol><ci id="S4.E11.m1.3.4.2.2.cmml" xref="S4.E11.m1.3.4.2.2">𝑅</ci><list id="S4.E11.m1.3.3.2.3.cmml" xref="S4.E11.m1.3.3.2.2"><apply id="S4.E11.m1.3.3.2.2.1.cmml" xref="S4.E11.m1.3.3.2.2.1"><csymbol cd="ambiguous" id="S4.E11.m1.3.3.2.2.1.1.cmml" xref="S4.E11.m1.3.3.2.2.1">subscript</csymbol><ci id="S4.E11.m1.3.3.2.2.1.2.cmml" xref="S4.E11.m1.3.3.2.2.1.2">𝜃</ci><ci id="S4.E11.m1.3.3.2.2.1.3.cmml" xref="S4.E11.m1.3.3.2.2.1.3">𝑗</ci></apply><ci id="S4.E11.m1.2.2.1.1.cmml" xref="S4.E11.m1.2.2.1.1">𝑖</ci></list></apply><apply id="S4.E11.m1.1.1.2.cmml" xref="S4.E11.m1.1.1.3"><csymbol cd="latexml" id="S4.E11.m1.1.1.2.1.cmml" xref="S4.E11.m1.1.1.3.1">matrix</csymbol><matrix id="S4.E11.m1.1.1.1.1.cmml" xref="S4.E11.m1.1.1.1.1"><matrixrow id="S4.E11.m1.1.1.1.1a.cmml" xref="S4.E11.m1.1.1.1.1"><apply id="S4.E11.m1.1.1.1.1.1.1.1.cmml" xref="S4.E11.m1.1.1.1.1.1.1.1"><cos id="S4.E11.m1.1.1.1.1.1.1.1.1.cmml" xref="S4.E11.m1.1.1.1.1.1.1.1.1"></cos><apply id="S4.E11.m1.1.1.1.1.1.1.1.2.cmml" xref="S4.E11.m1.1.1.1.1.1.1.1.2"><times id="S4.E11.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S4.E11.m1.1.1.1.1.1.1.1.2.1"></times><ci id="S4.E11.m1.1.1.1.1.1.1.1.2.2.cmml" xref="S4.E11.m1.1.1.1.1.1.1.1.2.2">𝑖</ci><apply id="S4.E11.m1.1.1.1.1.1.1.1.2.3.cmml" xref="S4.E11.m1.1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S4.E11.m1.1.1.1.1.1.1.1.2.3.1.cmml" xref="S4.E11.m1.1.1.1.1.1.1.1.2.3">subscript</csymbol><ci id="S4.E11.m1.1.1.1.1.1.1.1.2.3.2.cmml" xref="S4.E11.m1.1.1.1.1.1.1.1.2.3.2">𝜃</ci><ci id="S4.E11.m1.1.1.1.1.1.1.1.2.3.3.cmml" xref="S4.E11.m1.1.1.1.1.1.1.1.2.3.3">𝑗</ci></apply></apply></apply><apply id="S4.E11.m1.1.1.1.1.1.2.1.cmml" xref="S4.E11.m1.1.1.1.1.1.2.1"><minus id="S4.E11.m1.1.1.1.1.1.2.1.1.cmml" xref="S4.E11.m1.1.1.1.1.1.2.1"></minus><apply id="S4.E11.m1.1.1.1.1.1.2.1.2.cmml" xref="S4.E11.m1.1.1.1.1.1.2.1.2"><sin id="S4.E11.m1.1.1.1.1.1.2.1.2.1.cmml" xref="S4.E11.m1.1.1.1.1.1.2.1.2.1"></sin><apply id="S4.E11.m1.1.1.1.1.1.2.1.2.2.cmml" xref="S4.E11.m1.1.1.1.1.1.2.1.2.2"><times id="S4.E11.m1.1.1.1.1.1.2.1.2.2.1.cmml" xref="S4.E11.m1.1.1.1.1.1.2.1.2.2.1"></times><ci id="S4.E11.m1.1.1.1.1.1.2.1.2.2.2.cmml" xref="S4.E11.m1.1.1.1.1.1.2.1.2.2.2">𝑖</ci><apply id="S4.E11.m1.1.1.1.1.1.2.1.2.2.3.cmml" xref="S4.E11.m1.1.1.1.1.1.2.1.2.2.3"><csymbol cd="ambiguous" id="S4.E11.m1.1.1.1.1.1.2.1.2.2.3.1.cmml" xref="S4.E11.m1.1.1.1.1.1.2.1.2.2.3">subscript</csymbol><ci id="S4.E11.m1.1.1.1.1.1.2.1.2.2.3.2.cmml" xref="S4.E11.m1.1.1.1.1.1.2.1.2.2.3.2">𝜃</ci><ci id="S4.E11.m1.1.1.1.1.1.2.1.2.2.3.3.cmml" xref="S4.E11.m1.1.1.1.1.1.2.1.2.2.3.3">𝑗</ci></apply></apply></apply></apply></matrixrow><matrixrow id="S4.E11.m1.1.1.1.1b.cmml" xref="S4.E11.m1.1.1.1.1"><apply id="S4.E11.m1.1.1.1.1.2.1.1.cmml" xref="S4.E11.m1.1.1.1.1.2.1.1"><sin id="S4.E11.m1.1.1.1.1.2.1.1.1.cmml" xref="S4.E11.m1.1.1.1.1.2.1.1.1"></sin><apply id="S4.E11.m1.1.1.1.1.2.1.1.2.cmml" xref="S4.E11.m1.1.1.1.1.2.1.1.2"><times id="S4.E11.m1.1.1.1.1.2.1.1.2.1.cmml" xref="S4.E11.m1.1.1.1.1.2.1.1.2.1"></times><ci id="S4.E11.m1.1.1.1.1.2.1.1.2.2.cmml" xref="S4.E11.m1.1.1.1.1.2.1.1.2.2">𝑖</ci><apply id="S4.E11.m1.1.1.1.1.2.1.1.2.3.cmml" xref="S4.E11.m1.1.1.1.1.2.1.1.2.3"><csymbol cd="ambiguous" id="S4.E11.m1.1.1.1.1.2.1.1.2.3.1.cmml" xref="S4.E11.m1.1.1.1.1.2.1.1.2.3">subscript</csymbol><ci id="S4.E11.m1.1.1.1.1.2.1.1.2.3.2.cmml" xref="S4.E11.m1.1.1.1.1.2.1.1.2.3.2">𝜃</ci><ci id="S4.E11.m1.1.1.1.1.2.1.1.2.3.3.cmml" xref="S4.E11.m1.1.1.1.1.2.1.1.2.3.3">𝑗</ci></apply></apply></apply><apply id="S4.E11.m1.1.1.1.1.2.2.1.cmml" xref="S4.E11.m1.1.1.1.1.2.2.1"><cos id="S4.E11.m1.1.1.1.1.2.2.1.1.cmml" xref="S4.E11.m1.1.1.1.1.2.2.1.1"></cos><apply id="S4.E11.m1.1.1.1.1.2.2.1.2.cmml" xref="S4.E11.m1.1.1.1.1.2.2.1.2"><times id="S4.E11.m1.1.1.1.1.2.2.1.2.1.cmml" xref="S4.E11.m1.1.1.1.1.2.2.1.2.1"></times><ci id="S4.E11.m1.1.1.1.1.2.2.1.2.2.cmml" xref="S4.E11.m1.1.1.1.1.2.2.1.2.2">𝑖</ci><apply id="S4.E11.m1.1.1.1.1.2.2.1.2.3.cmml" xref="S4.E11.m1.1.1.1.1.2.2.1.2.3"><csymbol cd="ambiguous" id="S4.E11.m1.1.1.1.1.2.2.1.2.3.1.cmml" xref="S4.E11.m1.1.1.1.1.2.2.1.2.3">subscript</csymbol><ci id="S4.E11.m1.1.1.1.1.2.2.1.2.3.2.cmml" xref="S4.E11.m1.1.1.1.1.2.2.1.2.3.2">𝜃</ci><ci id="S4.E11.m1.1.1.1.1.2.2.1.2.3.3.cmml" xref="S4.E11.m1.1.1.1.1.2.2.1.2.3.3">𝑗</ci></apply></apply></apply></matrixrow></matrix></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E11.m1.3c">{R}_{\theta_{j},i}=\begin{bmatrix}\cos i\theta_{j}&amp;-\sin i\theta_{j}\\
\sin i\theta_{j}&amp;\cos i\theta_{j}\end{bmatrix}\\
</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(11)</span></td>
</tr></tbody>
</table>
</div>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Experiment</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">We evaluate our method on various tasks to understand the influence of high-rank updating.
In Section <a href="#S5.SS1" title="5.1 Memorizing UUID Pairs ‣ 5 Experiment ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.1</span></a>, we evaluate our method with LoRA and our method on memorizing UUID pairs to show the benefit of high-rank updating on memorizing.
In Section <a href="#S5.SS2" title="5.2 Fine-tuning Tasks ‣ 5 Experiment ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.2</span></a>, we reproduce LoRA, LoRA variants and FFT on three fine-tuning tasks: instruction tuning, mathematical reasoning and continual pretraining.
In Section <a href="#S5.SS3" title="5.3 Pretraining ‣ 5 Experiment ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.3</span></a>, we compare our method with LoRA and ReLoRA on pretraining by training transformer from scratch.</p>
</div>
<figure id="S5.F3" class="ltx_figure"><img src="/html/2405.12130/assets/moravslora-memory-lora.png" id="S5.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="592" height="439" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>
Performance of memorizing UUID pairs with LoRA and our method on rank 8 and 256.
</figcaption>
</figure>
<figure id="S5.T1" class="ltx_table">
<table id="S5.T1.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T1.1.1.1" class="ltx_tr">
<th id="S5.T1.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="S5.T1.1.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_tt"></th>
<th id="S5.T1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2">Instruction Tuning</th>
<th id="S5.T1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2">Mathematical Reasoning</th>
<th id="S5.T1.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2">Continual Pretraining</th>
</tr>
<tr id="S5.T1.1.2.2" class="ltx_tr">
<th id="S5.T1.1.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row"><span id="S5.T1.1.2.2.1.1" class="ltx_text ltx_font_bold">Method</span></th>
<th id="S5.T1.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row"><span id="S5.T1.1.2.2.2.1" class="ltx_text ltx_font_bold">Rank</span></th>
<th id="S5.T1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t"><span id="S5.T1.1.2.2.3.1" class="ltx_text ltx_font_bold">MMLU 0</span></th>
<th id="S5.T1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t"><span id="S5.T1.1.2.2.4.1" class="ltx_text ltx_font_bold">MMLU 5</span></th>
<th id="S5.T1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t"><span id="S5.T1.1.2.2.5.1" class="ltx_text ltx_font_bold">GSM8K</span></th>
<th id="S5.T1.1.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t"><span id="S5.T1.1.2.2.6.1" class="ltx_text ltx_font_bold">MATH</span></th>
<th id="S5.T1.1.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t"><span id="S5.T1.1.2.2.7.1" class="ltx_text ltx_font_bold">BioMed.</span></th>
<th id="S5.T1.1.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t"><span id="S5.T1.1.2.2.8.1" class="ltx_text ltx_font_bold">Finance</span></th>
</tr>
<tr id="S5.T1.1.3.3" class="ltx_tr">
<th id="S5.T1.1.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_t">FFT</th>
<th id="S5.T1.1.3.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_t">-</th>
<th id="S5.T1.1.3.3.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">50.6</th>
<th id="S5.T1.1.3.3.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">51.3</th>
<th id="S5.T1.1.3.3.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">66.6</th>
<th id="S5.T1.1.3.3.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">20.1</th>
<th id="S5.T1.1.3.3.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">56.4</th>
<th id="S5.T1.1.3.3.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">69.6</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T1.1.4.1" class="ltx_tr">
<th id="S5.T1.1.4.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">LoRA</th>
<th id="S5.T1.1.4.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t">8</th>
<td id="S5.T1.1.4.1.3" class="ltx_td ltx_align_center ltx_border_t">50.2</td>
<td id="S5.T1.1.4.1.4" class="ltx_td ltx_align_center ltx_border_t">51.5</td>
<td id="S5.T1.1.4.1.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S5.T1.1.4.1.5.1" class="ltx_text ltx_font_bold">64.6</span></td>
<td id="S5.T1.1.4.1.6" class="ltx_td ltx_align_center ltx_border_t">15.1</td>
<td id="S5.T1.1.4.1.7" class="ltx_td ltx_align_center ltx_border_t">52.3</td>
<td id="S5.T1.1.4.1.8" class="ltx_td ltx_align_center ltx_border_t">64.0</td>
</tr>
<tr id="S5.T1.1.5.2" class="ltx_tr">
<th id="S5.T1.1.5.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">LoRA+</th>
<th id="S5.T1.1.5.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">8</th>
<td id="S5.T1.1.5.2.3" class="ltx_td ltx_align_center">49.2</td>
<td id="S5.T1.1.5.2.4" class="ltx_td ltx_align_center">51.1</td>
<td id="S5.T1.1.5.2.5" class="ltx_td ltx_align_center">64.1</td>
<td id="S5.T1.1.5.2.6" class="ltx_td ltx_align_center"><span id="S5.T1.1.5.2.6.1" class="ltx_text ltx_font_bold">15.8</span></td>
<td id="S5.T1.1.5.2.7" class="ltx_td ltx_align_center">52.2</td>
<td id="S5.T1.1.5.2.8" class="ltx_td ltx_align_center">64.9</td>
</tr>
<tr id="S5.T1.1.6.3" class="ltx_tr">
<th id="S5.T1.1.6.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">ReLoRA</th>
<th id="S5.T1.1.6.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">8</th>
<td id="S5.T1.1.6.3.3" class="ltx_td ltx_align_center">49.3</td>
<td id="S5.T1.1.6.3.4" class="ltx_td ltx_align_center">50.2</td>
<td id="S5.T1.1.6.3.5" class="ltx_td ltx_align_center">61.5</td>
<td id="S5.T1.1.6.3.6" class="ltx_td ltx_align_center">14.5</td>
<td id="S5.T1.1.6.3.7" class="ltx_td ltx_align_center">46.3</td>
<td id="S5.T1.1.6.3.8" class="ltx_td ltx_align_center">61.0</td>
</tr>
<tr id="S5.T1.1.7.4" class="ltx_tr">
<th id="S5.T1.1.7.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">AsyLoRA</th>
<th id="S5.T1.1.7.4.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">8</th>
<td id="S5.T1.1.7.4.3" class="ltx_td ltx_align_center"><span id="S5.T1.1.7.4.3.1" class="ltx_text ltx_font_bold">50.3</span></td>
<td id="S5.T1.1.7.4.4" class="ltx_td ltx_align_center"><span id="S5.T1.1.7.4.4.1" class="ltx_text ltx_font_bold">52.2</span></td>
<td id="S5.T1.1.7.4.5" class="ltx_td ltx_align_center">64.5</td>
<td id="S5.T1.1.7.4.6" class="ltx_td ltx_align_center">15.0</td>
<td id="S5.T1.1.7.4.7" class="ltx_td ltx_align_center">52.5</td>
<td id="S5.T1.1.7.4.8" class="ltx_td ltx_align_center">63.5</td>
</tr>
<tr id="S5.T1.1.8.5" class="ltx_tr">
<th id="S5.T1.1.8.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">DoRA</th>
<th id="S5.T1.1.8.5.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">8</th>
<td id="S5.T1.1.8.5.3" class="ltx_td ltx_align_center">50.2</td>
<td id="S5.T1.1.8.5.4" class="ltx_td ltx_align_center">51.5</td>
<td id="S5.T1.1.8.5.5" class="ltx_td ltx_align_center">64.5</td>
<td id="S5.T1.1.8.5.6" class="ltx_td ltx_align_center">14.6</td>
<td id="S5.T1.1.8.5.7" class="ltx_td ltx_align_center">52.5</td>
<td id="S5.T1.1.8.5.8" class="ltx_td ltx_align_center">63.9</td>
</tr>
<tr id="S5.T1.1.9.6" class="ltx_tr">
<th id="S5.T1.1.9.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MoRA (Ours)</th>
<th id="S5.T1.1.9.6.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">8</th>
<td id="S5.T1.1.9.6.3" class="ltx_td ltx_align_center">49.7</td>
<td id="S5.T1.1.9.6.4" class="ltx_td ltx_align_center">51.5</td>
<td id="S5.T1.1.9.6.5" class="ltx_td ltx_align_center">64.2</td>
<td id="S5.T1.1.9.6.6" class="ltx_td ltx_align_center">15.4</td>
<td id="S5.T1.1.9.6.7" class="ltx_td ltx_align_center"><span id="S5.T1.1.9.6.7.1" class="ltx_text ltx_font_bold">53.3</span></td>
<td id="S5.T1.1.9.6.8" class="ltx_td ltx_align_center"><span id="S5.T1.1.9.6.8.1" class="ltx_text ltx_font_bold">67.1</span></td>
</tr>
<tr id="S5.T1.1.10.7" class="ltx_tr">
<th id="S5.T1.1.10.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">LoRA</th>
<th id="S5.T1.1.10.7.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t">256</th>
<td id="S5.T1.1.10.7.3" class="ltx_td ltx_align_center ltx_border_t">49.7</td>
<td id="S5.T1.1.10.7.4" class="ltx_td ltx_align_center ltx_border_t">50.8</td>
<td id="S5.T1.1.10.7.5" class="ltx_td ltx_align_center ltx_border_t">67.9</td>
<td id="S5.T1.1.10.7.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S5.T1.1.10.7.6.1" class="ltx_text ltx_font_bold">19.9</span></td>
<td id="S5.T1.1.10.7.7" class="ltx_td ltx_align_center ltx_border_t">54.1</td>
<td id="S5.T1.1.10.7.8" class="ltx_td ltx_align_center ltx_border_t">67.3</td>
</tr>
<tr id="S5.T1.1.11.8" class="ltx_tr">
<th id="S5.T1.1.11.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">LoRA+</th>
<th id="S5.T1.1.11.8.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">256</th>
<td id="S5.T1.1.11.8.3" class="ltx_td ltx_align_center">49.2</td>
<td id="S5.T1.1.11.8.4" class="ltx_td ltx_align_center">51.3</td>
<td id="S5.T1.1.11.8.5" class="ltx_td ltx_align_center"><span id="S5.T1.1.11.8.5.1" class="ltx_text ltx_font_bold">68.2</span></td>
<td id="S5.T1.1.11.8.6" class="ltx_td ltx_align_center">17.1</td>
<td id="S5.T1.1.11.8.7" class="ltx_td ltx_align_center">54.2</td>
<td id="S5.T1.1.11.8.8" class="ltx_td ltx_align_center">66.7</td>
</tr>
<tr id="S5.T1.1.12.9" class="ltx_tr">
<th id="S5.T1.1.12.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">ReLoRA</th>
<th id="S5.T1.1.12.9.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">256</th>
<td id="S5.T1.1.12.9.3" class="ltx_td ltx_align_center">-</td>
<td id="S5.T1.1.12.9.4" class="ltx_td ltx_align_center">-</td>
<td id="S5.T1.1.12.9.5" class="ltx_td ltx_align_center">64.0</td>
<td id="S5.T1.1.12.9.6" class="ltx_td ltx_align_center">18.1</td>
<td id="S5.T1.1.12.9.7" class="ltx_td ltx_align_center">52.9</td>
<td id="S5.T1.1.12.9.8" class="ltx_td ltx_align_center">57.9</td>
</tr>
<tr id="S5.T1.1.13.10" class="ltx_tr">
<th id="S5.T1.1.13.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">AsyLoRA</th>
<th id="S5.T1.1.13.10.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">256</th>
<td id="S5.T1.1.13.10.3" class="ltx_td ltx_align_center"><span id="S5.T1.1.13.10.3.1" class="ltx_text ltx_font_bold">50.1</span></td>
<td id="S5.T1.1.13.10.4" class="ltx_td ltx_align_center"><span id="S5.T1.1.13.10.4.1" class="ltx_text ltx_font_bold">52.0</span></td>
<td id="S5.T1.1.13.10.5" class="ltx_td ltx_align_center">66.9</td>
<td id="S5.T1.1.13.10.6" class="ltx_td ltx_align_center">19.3</td>
<td id="S5.T1.1.13.10.7" class="ltx_td ltx_align_center">54.1</td>
<td id="S5.T1.1.13.10.8" class="ltx_td ltx_align_center">66.9</td>
</tr>
<tr id="S5.T1.1.14.11" class="ltx_tr">
<th id="S5.T1.1.14.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">DoRA</th>
<th id="S5.T1.1.14.11.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">256</th>
<td id="S5.T1.1.14.11.3" class="ltx_td ltx_align_center">49.6</td>
<td id="S5.T1.1.14.11.4" class="ltx_td ltx_align_center">51.1</td>
<td id="S5.T1.1.14.11.5" class="ltx_td ltx_align_center">67.4</td>
<td id="S5.T1.1.14.11.6" class="ltx_td ltx_align_center">19.5</td>
<td id="S5.T1.1.14.11.7" class="ltx_td ltx_align_center">54.2</td>
<td id="S5.T1.1.14.11.8" class="ltx_td ltx_align_center">66.0</td>
</tr>
<tr id="S5.T1.1.15.12" class="ltx_tr">
<th id="S5.T1.1.15.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">MoRA (Ours)</th>
<th id="S5.T1.1.15.12.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb">256</th>
<td id="S5.T1.1.15.12.3" class="ltx_td ltx_align_center ltx_border_bb">49.9</td>
<td id="S5.T1.1.15.12.4" class="ltx_td ltx_align_center ltx_border_bb">51.4</td>
<td id="S5.T1.1.15.12.5" class="ltx_td ltx_align_center ltx_border_bb">67.9</td>
<td id="S5.T1.1.15.12.6" class="ltx_td ltx_align_center ltx_border_bb">19.2</td>
<td id="S5.T1.1.15.12.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T1.1.15.12.7.1" class="ltx_text ltx_font_bold">55.4</span></td>
<td id="S5.T1.1.15.12.8" class="ltx_td ltx_align_center ltx_border_bb"><span id="S5.T1.1.15.12.8.1" class="ltx_text ltx_font_bold">68.7</span></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>Performance of FFT, LoRA, LoRA variants and our method on instruction tuning, mathematical reasoning and continual pretraining tasks.</figcaption>
</figure>
<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>Memorizing UUID Pairs</h3>

<div id="S5.SS1.p1" class="ltx_para">
<p id="S5.SS1.p1.12" class="ltx_p">We first compare our method with LoRA and FFT on memorizing UUID pairs to demonstrate improvements through high-rank updating.
Following the training settings in Section <a href="#S3" title="3 Analysis the Influence of Low-rank Updating ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, we search learning rate from <math id="S5.SS1.p1.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="S5.SS1.p1.1.m1.1a"><mo stretchy="false" id="S5.SS1.p1.1.m1.1.1" xref="S5.SS1.p1.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.1.m1.1b"><ci id="S5.SS1.p1.1.m1.1.1.cmml" xref="S5.SS1.p1.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.1.m1.1c">\{</annotation></semantics></math>5e-5<span id="S5.SS1.p1.12.1" class="ltx_text ltx_markedasmath">,</span>1e-4<span id="S5.SS1.p1.12.2" class="ltx_text ltx_markedasmath">,</span>2e-4<math id="S5.SS1.p1.4.m4.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="S5.SS1.p1.4.m4.1a"><mo stretchy="false" id="S5.SS1.p1.4.m4.1.1" xref="S5.SS1.p1.4.m4.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.4.m4.1b"><ci id="S5.SS1.p1.4.m4.1.1.cmml" xref="S5.SS1.p1.4.m4.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.4.m4.1c">\}</annotation></semantics></math> and use decompress and compress functions in Eq. <a href="#S4.E8" title="In 4 Method ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>, sharing rows and columns in <math id="S5.SS1.p1.5.m5.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S5.SS1.p1.5.m5.1a"><mi id="S5.SS1.p1.5.m5.1.1" xref="S5.SS1.p1.5.m5.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.5.m5.1b"><ci id="S5.SS1.p1.5.m5.1.1.cmml" xref="S5.SS1.p1.5.m5.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.5.m5.1c">M</annotation></semantics></math>.
Due to use one matrix <math id="S5.SS1.p1.6.m6.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S5.SS1.p1.6.m6.1a"><mi id="S5.SS1.p1.6.m6.1.1" xref="S5.SS1.p1.6.m6.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.6.m6.1b"><ci id="S5.SS1.p1.6.m6.1.1.cmml" xref="S5.SS1.p1.6.m6.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.6.m6.1c">M</annotation></semantics></math> instead of two matrices <math id="S5.SS1.p1.7.m7.1" class="ltx_Math" alttext="A" display="inline"><semantics id="S5.SS1.p1.7.m7.1a"><mi id="S5.SS1.p1.7.m7.1.1" xref="S5.SS1.p1.7.m7.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.7.m7.1b"><ci id="S5.SS1.p1.7.m7.1.1.cmml" xref="S5.SS1.p1.7.m7.1.1">𝐴</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.7.m7.1c">A</annotation></semantics></math> and <math id="S5.SS1.p1.8.m8.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S5.SS1.p1.8.m8.1a"><mi id="S5.SS1.p1.8.m8.1.1" xref="S5.SS1.p1.8.m8.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.8.m8.1b"><ci id="S5.SS1.p1.8.m8.1.1.cmml" xref="S5.SS1.p1.8.m8.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.8.m8.1c">B</annotation></semantics></math>, we can directly initialize <math id="S5.SS1.p1.9.m9.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S5.SS1.p1.9.m9.1a"><mi id="S5.SS1.p1.9.m9.1.1" xref="S5.SS1.p1.9.m9.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.9.m9.1b"><ci id="S5.SS1.p1.9.m9.1.1.cmml" xref="S5.SS1.p1.9.m9.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.9.m9.1c">M</annotation></semantics></math> with zeros.
For the predefined groups <math id="S5.SS1.p1.10.m10.1" class="ltx_Math" alttext="g" display="inline"><semantics id="S5.SS1.p1.10.m10.1a"><mi id="S5.SS1.p1.10.m10.1.1" xref="S5.SS1.p1.10.m10.1.1.cmml">g</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.10.m10.1b"><ci id="S5.SS1.p1.10.m10.1.1.cmml" xref="S5.SS1.p1.10.m10.1.1">𝑔</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.10.m10.1c">g</annotation></semantics></math> and <math id="S5.SS1.p1.11.m11.1" class="ltx_Math" alttext="g^{\prime}" display="inline"><semantics id="S5.SS1.p1.11.m11.1a"><msup id="S5.SS1.p1.11.m11.1.1" xref="S5.SS1.p1.11.m11.1.1.cmml"><mi id="S5.SS1.p1.11.m11.1.1.2" xref="S5.SS1.p1.11.m11.1.1.2.cmml">g</mi><mo id="S5.SS1.p1.11.m11.1.1.3" xref="S5.SS1.p1.11.m11.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.11.m11.1b"><apply id="S5.SS1.p1.11.m11.1.1.cmml" xref="S5.SS1.p1.11.m11.1.1"><csymbol cd="ambiguous" id="S5.SS1.p1.11.m11.1.1.1.cmml" xref="S5.SS1.p1.11.m11.1.1">superscript</csymbol><ci id="S5.SS1.p1.11.m11.1.1.2.cmml" xref="S5.SS1.p1.11.m11.1.1.2">𝑔</ci><ci id="S5.SS1.p1.11.m11.1.1.3.cmml" xref="S5.SS1.p1.11.m11.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.11.m11.1c">g^{\prime}</annotation></semantics></math>, we group every adjacent <math id="S5.SS1.p1.12.m12.1" class="ltx_Math" alttext="\hat{r}" display="inline"><semantics id="S5.SS1.p1.12.m12.1a"><mover accent="true" id="S5.SS1.p1.12.m12.1.1" xref="S5.SS1.p1.12.m12.1.1.cmml"><mi id="S5.SS1.p1.12.m12.1.1.2" xref="S5.SS1.p1.12.m12.1.1.2.cmml">r</mi><mo id="S5.SS1.p1.12.m12.1.1.1" xref="S5.SS1.p1.12.m12.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.12.m12.1b"><apply id="S5.SS1.p1.12.m12.1.1.cmml" xref="S5.SS1.p1.12.m12.1.1"><ci id="S5.SS1.p1.12.m12.1.1.1.cmml" xref="S5.SS1.p1.12.m12.1.1.1">^</ci><ci id="S5.SS1.p1.12.m12.1.1.2.cmml" xref="S5.SS1.p1.12.m12.1.1.2">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.12.m12.1c">\hat{r}</annotation></semantics></math> rows or columns together.
The training loss is presented in Figure<a href="#S5.F3" title="Figure 3 ‣ 5 Experiment ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. Our method shows significant improvements over LoRA with the same number of trainable parameters, benefiting from high-rank updating. We also report character-level accuracy at various training steps in Table <a href="#S5.T2" title="Table 2 ‣ 5.1 Memorizing UUID Pairs ‣ 5 Experiment ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>. MoRA requires fewer training steps to memorize these UUID pairs compared to LoRA.
Compared to FFT, MoRA with 256 rank can achieve similar performance and both method can memorize all UUID pairs in 500 steps.</p>
</div>
<figure id="S5.T2" class="ltx_table">
<table id="S5.T2.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T2.1.1.1" class="ltx_tr">
<th id="S5.T2.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="S5.T2.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt">Rank</th>
<th id="S5.T2.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">300</th>
<th id="S5.T2.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">500</th>
<th id="S5.T2.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">700</th>
<th id="S5.T2.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">900</th>
</tr>
<tr id="S5.T2.1.2.2" class="ltx_tr">
<th id="S5.T2.1.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_t">FFT</th>
<th id="S5.T2.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_t">-</th>
<th id="S5.T2.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">42.5</th>
<th id="S5.T2.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="background-color:#E0FFFF;"><span id="S5.T2.1.2.2.4.1" class="ltx_text" style="background-color:#E0FFFF;">100</span></th>
<th id="S5.T2.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="background-color:#E0FFFF;"><span id="S5.T2.1.2.2.5.1" class="ltx_text" style="background-color:#E0FFFF;">100</span></th>
<th id="S5.T2.1.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t" style="background-color:#E0FFFF;"><span id="S5.T2.1.2.2.6.1" class="ltx_text" style="background-color:#E0FFFF;">100</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T2.1.3.1" class="ltx_tr">
<th id="S5.T2.1.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">LoRA</th>
<th id="S5.T2.1.3.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">8</th>
<td id="S5.T2.1.3.1.3" class="ltx_td ltx_align_center">9.9</td>
<td id="S5.T2.1.3.1.4" class="ltx_td ltx_align_center">10.0</td>
<td id="S5.T2.1.3.1.5" class="ltx_td ltx_align_center">10.7</td>
<td id="S5.T2.1.3.1.6" class="ltx_td ltx_align_center">54.2</td>
</tr>
<tr id="S5.T2.1.4.2" class="ltx_tr">
<th id="S5.T2.1.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MoRA</th>
<th id="S5.T2.1.4.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">8</th>
<td id="S5.T2.1.4.2.3" class="ltx_td ltx_align_center">10.1</td>
<td id="S5.T2.1.4.2.4" class="ltx_td ltx_align_center">15.7</td>
<td id="S5.T2.1.4.2.5" class="ltx_td ltx_align_center">87.4</td>
<td id="S5.T2.1.4.2.6" class="ltx_td ltx_align_center" style="background-color:#E0FFFF;"><span id="S5.T2.1.4.2.6.1" class="ltx_text" style="background-color:#E0FFFF;">100</span></td>
</tr>
<tr id="S5.T2.1.5.3" class="ltx_tr">
<th id="S5.T2.1.5.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">LoRA</th>
<th id="S5.T2.1.5.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">256</th>
<td id="S5.T2.1.5.3.3" class="ltx_td ltx_align_center">9.9</td>
<td id="S5.T2.1.5.3.4" class="ltx_td ltx_align_center">70.6</td>
<td id="S5.T2.1.5.3.5" class="ltx_td ltx_align_center" style="background-color:#E0FFFF;"><span id="S5.T2.1.5.3.5.1" class="ltx_text" style="background-color:#E0FFFF;">100</span></td>
<td id="S5.T2.1.5.3.6" class="ltx_td ltx_align_center" style="background-color:#E0FFFF;"><span id="S5.T2.1.5.3.6.1" class="ltx_text" style="background-color:#E0FFFF;">100</span></td>
</tr>
<tr id="S5.T2.1.6.4" class="ltx_tr">
<th id="S5.T2.1.6.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">MoRA</th>
<th id="S5.T2.1.6.4.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb">256</th>
<td id="S5.T2.1.6.4.3" class="ltx_td ltx_align_center ltx_border_bb">41.6</td>
<td id="S5.T2.1.6.4.4" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E0FFFF;"><span id="S5.T2.1.6.4.4.1" class="ltx_text" style="background-color:#E0FFFF;">100</span></td>
<td id="S5.T2.1.6.4.5" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E0FFFF;"><span id="S5.T2.1.6.4.5.1" class="ltx_text" style="background-color:#E0FFFF;">100</span></td>
<td id="S5.T2.1.6.4.6" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E0FFFF;"><span id="S5.T2.1.6.4.6.1" class="ltx_text" style="background-color:#E0FFFF;">100</span></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>Character-level accuracy of memorizing UUID pairs by generating the value of corresponding key in 300, 500, 700 and 900 training steps.</figcaption>
</figure>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>Fine-tuning Tasks</h3>

<section id="S5.SS2.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">5.2.1 </span>Setup</h4>

<div id="S5.SS2.SSS1.p1" class="ltx_para">
<p id="S5.SS2.SSS1.p1.1" class="ltx_p">We evaluate our method across three fine-tuning tasks for large language models (LLMs): instruction tuning, mathematical reasoning, and continual pretraining.
For these tasks, we select high-quality corresponding datasets to test both LoRA and our method.
In instruction tuning, we utilize Tülu v2 <cite class="ltx_cite ltx_citemacro_cite">Ivison et al. (<a href="#bib.bib17" title="" class="ltx_ref">2023</a>)</cite>, a blend of several high-quality instruction datasets, containing 326k filtered samples. We assess instruction performance using the MMLU <cite class="ltx_cite ltx_citemacro_cite">Hendrycks et al. (<a href="#bib.bib12" title="" class="ltx_ref">2020</a>)</cite> in both zero-shot and five-shot settings. For mathematical reasoning, we employ the MetaMath <cite class="ltx_cite ltx_citemacro_cite">Yu et al. (<a href="#bib.bib38" title="" class="ltx_ref">2023</a>)</cite> with its 395k samples to enhance mathematical reasoning capabilities and also use GSM8K <cite class="ltx_cite ltx_citemacro_cite">Cobbe et al. (<a href="#bib.bib4" title="" class="ltx_ref">2021</a>)</cite> and MATH <cite class="ltx_cite ltx_citemacro_cite">Hendrycks et al. (<a href="#bib.bib13" title="" class="ltx_ref">2021</a>)</cite> for further evaluation. In continual pretraining, we adapt an LLM to the biomedicine and finance using PubMed abstracts from the Pile <cite class="ltx_cite ltx_citemacro_cite">Gao et al. (<a href="#bib.bib9" title="" class="ltx_ref">2020</a>)</cite> and finicial news, complemented by data preprocessing methods from AdaptLLM <cite class="ltx_cite ltx_citemacro_cite">Cheng et al. (<a href="#bib.bib3" title="" class="ltx_ref">2023</a>)</cite> to boost performance. We report the average performance of corresponding tasks for continual pretraining. More details can be found in Appendix <a href="#A3" title="Appendix C Downstream Tasks of Continual Pretraining ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>.</p>
</div>
<figure id="S5.F4" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S5.F4.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2405.12130/assets/250m-loss.png" id="S5.F4.sf1.g1" class="ltx_graphics ltx_img_landscape" width="598" height="419" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span>Pretraining loss at 250M models.</figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S5.F4.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2405.12130/assets/1b-loss.png" id="S5.F4.sf2.g1" class="ltx_graphics ltx_img_landscape" width="598" height="419" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span>Pretraining loss at 1.3B models.</figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>
Pretraining loss with LoRA and MoRA on 250M and 1B models from scratch.
Both LoRA and MoRA use same amount of trainable parameters with <math id="S5.F4.3.m1.1" class="ltx_Math" alttext="r=128" display="inline"><semantics id="S5.F4.3.m1.1b"><mrow id="S5.F4.3.m1.1.1" xref="S5.F4.3.m1.1.1.cmml"><mi id="S5.F4.3.m1.1.1.2" xref="S5.F4.3.m1.1.1.2.cmml">r</mi><mo id="S5.F4.3.m1.1.1.1" xref="S5.F4.3.m1.1.1.1.cmml">=</mo><mn id="S5.F4.3.m1.1.1.3" xref="S5.F4.3.m1.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.F4.3.m1.1c"><apply id="S5.F4.3.m1.1.1.cmml" xref="S5.F4.3.m1.1.1"><eq id="S5.F4.3.m1.1.1.1.cmml" xref="S5.F4.3.m1.1.1.1"></eq><ci id="S5.F4.3.m1.1.1.2.cmml" xref="S5.F4.3.m1.1.1.2">𝑟</ci><cn type="integer" id="S5.F4.3.m1.1.1.3.cmml" xref="S5.F4.3.m1.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.F4.3.m1.1d">r=128</annotation></semantics></math>.
ReMoRA and ReLoRA refer to merge MoRA or LoRA back to the model during training to increase the rank of <math id="S5.F4.4.m2.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S5.F4.4.m2.1b"><mrow id="S5.F4.4.m2.1.1" xref="S5.F4.4.m2.1.1.cmml"><mi mathvariant="normal" id="S5.F4.4.m2.1.1.2" xref="S5.F4.4.m2.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S5.F4.4.m2.1.1.1" xref="S5.F4.4.m2.1.1.1.cmml">​</mo><mi id="S5.F4.4.m2.1.1.3" xref="S5.F4.4.m2.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.F4.4.m2.1c"><apply id="S5.F4.4.m2.1.1.cmml" xref="S5.F4.4.m2.1.1"><times id="S5.F4.4.m2.1.1.1.cmml" xref="S5.F4.4.m2.1.1.1"></times><ci id="S5.F4.4.m2.1.1.2.cmml" xref="S5.F4.4.m2.1.1.2">Δ</ci><ci id="S5.F4.4.m2.1.1.3.cmml" xref="S5.F4.4.m2.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.F4.4.m2.1d">\Delta W</annotation></semantics></math>.
</figcaption>
</figure>
</section>
<section id="S5.SS2.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">5.2.2 </span>Baselines and Implements</h4>

<div id="S5.SS2.SSS2.p1" class="ltx_para">
<p id="S5.SS2.SSS2.p1.15" class="ltx_p">For LoRA-like methods and MoRA, we conducted experiments at <math id="S5.SS2.SSS2.p1.1.m1.1" class="ltx_Math" alttext="r=8" display="inline"><semantics id="S5.SS2.SSS2.p1.1.m1.1a"><mrow id="S5.SS2.SSS2.p1.1.m1.1.1" xref="S5.SS2.SSS2.p1.1.m1.1.1.cmml"><mi id="S5.SS2.SSS2.p1.1.m1.1.1.2" xref="S5.SS2.SSS2.p1.1.m1.1.1.2.cmml">r</mi><mo id="S5.SS2.SSS2.p1.1.m1.1.1.1" xref="S5.SS2.SSS2.p1.1.m1.1.1.1.cmml">=</mo><mn id="S5.SS2.SSS2.p1.1.m1.1.1.3" xref="S5.SS2.SSS2.p1.1.m1.1.1.3.cmml">8</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS2.p1.1.m1.1b"><apply id="S5.SS2.SSS2.p1.1.m1.1.1.cmml" xref="S5.SS2.SSS2.p1.1.m1.1.1"><eq id="S5.SS2.SSS2.p1.1.m1.1.1.1.cmml" xref="S5.SS2.SSS2.p1.1.m1.1.1.1"></eq><ci id="S5.SS2.SSS2.p1.1.m1.1.1.2.cmml" xref="S5.SS2.SSS2.p1.1.m1.1.1.2">𝑟</ci><cn type="integer" id="S5.SS2.SSS2.p1.1.m1.1.1.3.cmml" xref="S5.SS2.SSS2.p1.1.m1.1.1.3">8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS2.p1.1.m1.1c">r=8</annotation></semantics></math> and <math id="S5.SS2.SSS2.p1.2.m2.1" class="ltx_Math" alttext="r=256" display="inline"><semantics id="S5.SS2.SSS2.p1.2.m2.1a"><mrow id="S5.SS2.SSS2.p1.2.m2.1.1" xref="S5.SS2.SSS2.p1.2.m2.1.1.cmml"><mi id="S5.SS2.SSS2.p1.2.m2.1.1.2" xref="S5.SS2.SSS2.p1.2.m2.1.1.2.cmml">r</mi><mo id="S5.SS2.SSS2.p1.2.m2.1.1.1" xref="S5.SS2.SSS2.p1.2.m2.1.1.1.cmml">=</mo><mn id="S5.SS2.SSS2.p1.2.m2.1.1.3" xref="S5.SS2.SSS2.p1.2.m2.1.1.3.cmml">256</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS2.p1.2.m2.1b"><apply id="S5.SS2.SSS2.p1.2.m2.1.1.cmml" xref="S5.SS2.SSS2.p1.2.m2.1.1"><eq id="S5.SS2.SSS2.p1.2.m2.1.1.1.cmml" xref="S5.SS2.SSS2.p1.2.m2.1.1.1"></eq><ci id="S5.SS2.SSS2.p1.2.m2.1.1.2.cmml" xref="S5.SS2.SSS2.p1.2.m2.1.1.2">𝑟</ci><cn type="integer" id="S5.SS2.SSS2.p1.2.m2.1.1.3.cmml" xref="S5.SS2.SSS2.p1.2.m2.1.1.3">256</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS2.p1.2.m2.1c">r=256</annotation></semantics></math>, and reproduce following methods across three tasks: FFT, LoRA, LoRA+ <cite class="ltx_cite ltx_citemacro_cite">Hayou et al. (<a href="#bib.bib11" title="" class="ltx_ref">2024</a>)</cite>, AsyLoRA <cite class="ltx_cite ltx_citemacro_cite">Zhu et al. (<a href="#bib.bib41" title="" class="ltx_ref">2024</a>)</cite>, ReLoRA <cite class="ltx_cite ltx_citemacro_cite">Lialin et al. (<a href="#bib.bib22" title="" class="ltx_ref">2023</a>)</cite> and DoRA <cite class="ltx_cite ltx_citemacro_cite">Liu et al. (<a href="#bib.bib24" title="" class="ltx_ref">2024</a>)</cite>.
LoRA+ enhances the learning rate of matrix <math id="S5.SS2.SSS2.p1.3.m3.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S5.SS2.SSS2.p1.3.m3.1a"><mi id="S5.SS2.SSS2.p1.3.m3.1.1" xref="S5.SS2.SSS2.p1.3.m3.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS2.p1.3.m3.1b"><ci id="S5.SS2.SSS2.p1.3.m3.1.1.cmml" xref="S5.SS2.SSS2.p1.3.m3.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS2.p1.3.m3.1c">B</annotation></semantics></math> in LoRA to facilitate efficient feature learning based on theoretical analysis.
We search the corresponding the hyperparameter <math id="S5.SS2.SSS2.p1.4.m4.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="S5.SS2.SSS2.p1.4.m4.1a"><mi id="S5.SS2.SSS2.p1.4.m4.1.1" xref="S5.SS2.SSS2.p1.4.m4.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS2.p1.4.m4.1b"><ci id="S5.SS2.SSS2.p1.4.m4.1.1.cmml" xref="S5.SS2.SSS2.p1.4.m4.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS2.p1.4.m4.1c">\lambda</annotation></semantics></math> from <math id="S5.SS2.SSS2.p1.5.m5.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="S5.SS2.SSS2.p1.5.m5.1a"><mo stretchy="false" id="S5.SS2.SSS2.p1.5.m5.1.1" xref="S5.SS2.SSS2.p1.5.m5.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS2.p1.5.m5.1b"><ci id="S5.SS2.SSS2.p1.5.m5.1.1.cmml" xref="S5.SS2.SSS2.p1.5.m5.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS2.p1.5.m5.1c">\{</annotation></semantics></math>2<span id="S5.SS2.SSS2.p1.15.1" class="ltx_text ltx_markedasmath">,</span>4<math id="S5.SS2.SSS2.p1.7.m7.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="S5.SS2.SSS2.p1.7.m7.1a"><mo stretchy="false" id="S5.SS2.SSS2.p1.7.m7.1.1" xref="S5.SS2.SSS2.p1.7.m7.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS2.p1.7.m7.1b"><ci id="S5.SS2.SSS2.p1.7.m7.1.1.cmml" xref="S5.SS2.SSS2.p1.7.m7.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS2.p1.7.m7.1c">\}</annotation></semantics></math>.
AsyLoRA also analyzes asymmetry in the <math id="S5.SS2.SSS2.p1.8.m8.1" class="ltx_Math" alttext="A" display="inline"><semantics id="S5.SS2.SSS2.p1.8.m8.1a"><mi id="S5.SS2.SSS2.p1.8.m8.1.1" xref="S5.SS2.SSS2.p1.8.m8.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS2.p1.8.m8.1b"><ci id="S5.SS2.SSS2.p1.8.m8.1.1.cmml" xref="S5.SS2.SSS2.p1.8.m8.1.1">𝐴</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS2.p1.8.m8.1c">A</annotation></semantics></math> and <math id="S5.SS2.SSS2.p1.9.m9.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S5.SS2.SSS2.p1.9.m9.1a"><mi id="S5.SS2.SSS2.p1.9.m9.1.1" xref="S5.SS2.SSS2.p1.9.m9.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS2.p1.9.m9.1b"><ci id="S5.SS2.SSS2.p1.9.m9.1.1.cmml" xref="S5.SS2.SSS2.p1.9.m9.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS2.p1.9.m9.1c">B</annotation></semantics></math> matrices, and we adopted their initialization strategy.
ReLoRA proposes a method to merge low-rank matrices into the model during training to increase the rank of <math id="S5.SS2.SSS2.p1.10.m10.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S5.SS2.SSS2.p1.10.m10.1a"><mrow id="S5.SS2.SSS2.p1.10.m10.1.1" xref="S5.SS2.SSS2.p1.10.m10.1.1.cmml"><mi mathvariant="normal" id="S5.SS2.SSS2.p1.10.m10.1.1.2" xref="S5.SS2.SSS2.p1.10.m10.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S5.SS2.SSS2.p1.10.m10.1.1.1" xref="S5.SS2.SSS2.p1.10.m10.1.1.1.cmml">​</mo><mi id="S5.SS2.SSS2.p1.10.m10.1.1.3" xref="S5.SS2.SSS2.p1.10.m10.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS2.p1.10.m10.1b"><apply id="S5.SS2.SSS2.p1.10.m10.1.1.cmml" xref="S5.SS2.SSS2.p1.10.m10.1.1"><times id="S5.SS2.SSS2.p1.10.m10.1.1.1.cmml" xref="S5.SS2.SSS2.p1.10.m10.1.1.1"></times><ci id="S5.SS2.SSS2.p1.10.m10.1.1.2.cmml" xref="S5.SS2.SSS2.p1.10.m10.1.1.2">Δ</ci><ci id="S5.SS2.SSS2.p1.10.m10.1.1.3.cmml" xref="S5.SS2.SSS2.p1.10.m10.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS2.p1.10.m10.1c">\Delta W</annotation></semantics></math>.
we search merge steps from {<math id="S5.SS2.SSS2.p1.11.m11.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S5.SS2.SSS2.p1.11.m11.1a"><mn id="S5.SS2.SSS2.p1.11.m11.1.1" xref="S5.SS2.SSS2.p1.11.m11.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS2.p1.11.m11.1b"><cn type="integer" id="S5.SS2.SSS2.p1.11.m11.1.1.cmml" xref="S5.SS2.SSS2.p1.11.m11.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS2.p1.11.m11.1c">1</annotation></semantics></math>k, <math id="S5.SS2.SSS2.p1.12.m12.1" class="ltx_Math" alttext="2" display="inline"><semantics id="S5.SS2.SSS2.p1.12.m12.1a"><mn id="S5.SS2.SSS2.p1.12.m12.1.1" xref="S5.SS2.SSS2.p1.12.m12.1.1.cmml">2</mn><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS2.p1.12.m12.1b"><cn type="integer" id="S5.SS2.SSS2.p1.12.m12.1.1.cmml" xref="S5.SS2.SSS2.p1.12.m12.1.1">2</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS2.p1.12.m12.1c">2</annotation></semantics></math>k} and use 50 steps restarts warmup.
DoRA leverages weight decomposition to enhance performance as a robust baseline.
For FFT, we follow the settings proposed by corresponding datasets.
For MoRA, we employed rotation operators as outlined in Eq. <a href="#S4.E9" title="In 4 Method ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a> to implement compression and decompression for <math id="S5.SS2.SSS2.p1.13.m13.1" class="ltx_Math" alttext="r=8" display="inline"><semantics id="S5.SS2.SSS2.p1.13.m13.1a"><mrow id="S5.SS2.SSS2.p1.13.m13.1.1" xref="S5.SS2.SSS2.p1.13.m13.1.1.cmml"><mi id="S5.SS2.SSS2.p1.13.m13.1.1.2" xref="S5.SS2.SSS2.p1.13.m13.1.1.2.cmml">r</mi><mo id="S5.SS2.SSS2.p1.13.m13.1.1.1" xref="S5.SS2.SSS2.p1.13.m13.1.1.1.cmml">=</mo><mn id="S5.SS2.SSS2.p1.13.m13.1.1.3" xref="S5.SS2.SSS2.p1.13.m13.1.1.3.cmml">8</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS2.p1.13.m13.1b"><apply id="S5.SS2.SSS2.p1.13.m13.1.1.cmml" xref="S5.SS2.SSS2.p1.13.m13.1.1"><eq id="S5.SS2.SSS2.p1.13.m13.1.1.1.cmml" xref="S5.SS2.SSS2.p1.13.m13.1.1.1"></eq><ci id="S5.SS2.SSS2.p1.13.m13.1.1.2.cmml" xref="S5.SS2.SSS2.p1.13.m13.1.1.2">𝑟</ci><cn type="integer" id="S5.SS2.SSS2.p1.13.m13.1.1.3.cmml" xref="S5.SS2.SSS2.p1.13.m13.1.1.3">8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS2.p1.13.m13.1c">r=8</annotation></semantics></math>, and for <math id="S5.SS2.SSS2.p1.14.m14.1" class="ltx_Math" alttext="r=256" display="inline"><semantics id="S5.SS2.SSS2.p1.14.m14.1a"><mrow id="S5.SS2.SSS2.p1.14.m14.1.1" xref="S5.SS2.SSS2.p1.14.m14.1.1.cmml"><mi id="S5.SS2.SSS2.p1.14.m14.1.1.2" xref="S5.SS2.SSS2.p1.14.m14.1.1.2.cmml">r</mi><mo id="S5.SS2.SSS2.p1.14.m14.1.1.1" xref="S5.SS2.SSS2.p1.14.m14.1.1.1.cmml">=</mo><mn id="S5.SS2.SSS2.p1.14.m14.1.1.3" xref="S5.SS2.SSS2.p1.14.m14.1.1.3.cmml">256</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS2.p1.14.m14.1b"><apply id="S5.SS2.SSS2.p1.14.m14.1.1.cmml" xref="S5.SS2.SSS2.p1.14.m14.1.1"><eq id="S5.SS2.SSS2.p1.14.m14.1.1.1.cmml" xref="S5.SS2.SSS2.p1.14.m14.1.1.1"></eq><ci id="S5.SS2.SSS2.p1.14.m14.1.1.2.cmml" xref="S5.SS2.SSS2.p1.14.m14.1.1.2">𝑟</ci><cn type="integer" id="S5.SS2.SSS2.p1.14.m14.1.1.3.cmml" xref="S5.SS2.SSS2.p1.14.m14.1.1.3">256</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS2.p1.14.m14.1c">r=256</annotation></semantics></math>, we utilized shared rows and columns as specified in Eq. <a href="#S4.E6" title="In 4 Method ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> and group every adjacent <math id="S5.SS2.SSS2.p1.15.m15.1" class="ltx_Math" alttext="\hat{r}" display="inline"><semantics id="S5.SS2.SSS2.p1.15.m15.1a"><mover accent="true" id="S5.SS2.SSS2.p1.15.m15.1.1" xref="S5.SS2.SSS2.p1.15.m15.1.1.cmml"><mi id="S5.SS2.SSS2.p1.15.m15.1.1.2" xref="S5.SS2.SSS2.p1.15.m15.1.1.2.cmml">r</mi><mo id="S5.SS2.SSS2.p1.15.m15.1.1.1" xref="S5.SS2.SSS2.p1.15.m15.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS2.p1.15.m15.1b"><apply id="S5.SS2.SSS2.p1.15.m15.1.1.cmml" xref="S5.SS2.SSS2.p1.15.m15.1.1"><ci id="S5.SS2.SSS2.p1.15.m15.1.1.1.cmml" xref="S5.SS2.SSS2.p1.15.m15.1.1.1">^</ci><ci id="S5.SS2.SSS2.p1.15.m15.1.1.2.cmml" xref="S5.SS2.SSS2.p1.15.m15.1.1.2">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS2.p1.15.m15.1c">\hat{r}</annotation></semantics></math> rows or columns together.
The details hyperparameters about fine-tuning can be found in Appendix <a href="#A1" title="Appendix A Hyperparameters ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>.</p>
</div>
</section>
<section id="S5.SS2.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">5.2.3 </span>Results and Analysis</h4>

<div id="S5.SS2.SSS3.p1" class="ltx_para">
<p id="S5.SS2.SSS3.p1.1" class="ltx_p">We present the results of fine-tuning tasks in Table <a href="#S5.T1" title="Table 1 ‣ 5 Experiment ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.
We report the results of MMLU with zero-shot and 5-shot settings for instruction tuning, GSM8K and MATH for mathematical reasoning, and average performance on biomedical tasks and financial tasks for continual pretraining.</p>
</div>
<div id="S5.SS2.SSS3.p2" class="ltx_para">
<p id="S5.SS2.SSS3.p2.1" class="ltx_p">MoRA shows on par performances with LoRA on instruction tuning and mathematical reasoning.
Benefit from high-rank updating to memorize new knowledge, MoRA outperforms LoRA on both biomedical and financial domains for continual pretraining.
</p>
</div>
<div id="S5.SS2.SSS3.p3" class="ltx_para">
<p id="S5.SS2.SSS3.p3.1" class="ltx_p">We also find that LoRA variants exhibit similar performances on these fine-tuning tasks as compared to LoRA. Although AsyLoRA achieves the best performance in instruction tuning, it demonstrates poor performance in mathematical reasoning. For ReLoRA, merging low-rank matrices during training can harm performance, particularly at the the high rank like 256.</p>
</div>
<div id="S5.SS2.SSS3.p4" class="ltx_para">
<p id="S5.SS2.SSS3.p4.1" class="ltx_p">Consider the difference between three tasks, they show different requirements for fine-tuning capabilities.
For instruction tuning, which does not learn new knowledge from fine-tuning, rank 8 is enough to achieve performance similar to FFT. For mathematical reasoning, rank 8 is unable to match FFT performance.
However, increasing the rank from 8 to 256 can eliminate the performance gap. For continual pretraining, LoRA with rank 256 still underperforms FFT.</p>
</div>
</section>
</section>
<section id="S5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.3 </span>Pretraining</h3>

<figure id="S5.T3" class="ltx_table">
<table id="S5.T3.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T3.1.1.1" class="ltx_tr">
<th id="S5.T3.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="S5.T3.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">250M</th>
<th id="S5.T3.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">1.3B</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T3.1.2.1" class="ltx_tr">
<th id="S5.T3.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">LoRA</th>
<td id="S5.T3.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">33.40</td>
<td id="S5.T3.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">28.56</td>
</tr>
<tr id="S5.T3.1.3.2" class="ltx_tr">
<th id="S5.T3.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MoRA (Ours)</th>
<td id="S5.T3.1.3.2.2" class="ltx_td ltx_align_center">28.54</td>
<td id="S5.T3.1.3.2.3" class="ltx_td ltx_align_center">25.25</td>
</tr>
<tr id="S5.T3.1.4.3" class="ltx_tr">
<th id="S5.T3.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">ReLoRA</th>
<td id="S5.T3.1.4.3.2" class="ltx_td ltx_align_center">32.19</td>
<td id="S5.T3.1.4.3.3" class="ltx_td ltx_align_center">27.80</td>
</tr>
<tr id="S5.T3.1.5.4" class="ltx_tr">
<th id="S5.T3.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">ReMoRA (Ours)</th>
<td id="S5.T3.1.5.4.2" class="ltx_td ltx_align_center ltx_border_bb">26.74</td>
<td id="S5.T3.1.5.4.3" class="ltx_td ltx_align_center ltx_border_bb">23.34</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>Perplexity on C4 validation dataset.
</figcaption>
</figure>
<div id="S5.SS3.p1" class="ltx_para">
<p id="S5.SS3.p1.1" class="ltx_p">To understand the influence of high-rank updating,
we train transformer from scratch on the C4 datasets <cite class="ltx_cite ltx_citemacro_cite">Raffel et al. (<a href="#bib.bib28" title="" class="ltx_ref">2020</a>)</cite>.
For the model architeture, we train LLaMA-based model with RMSNorm <cite class="ltx_cite ltx_citemacro_cite">Zhang and Sennrich (<a href="#bib.bib39" title="" class="ltx_ref">2019</a>)</cite>, SwiGLU <cite class="ltx_cite ltx_citemacro_cite">Shazeer (<a href="#bib.bib31" title="" class="ltx_ref">2020</a>)</cite> and RoPE <cite class="ltx_cite ltx_citemacro_cite">Su et al. (<a href="#bib.bib34" title="" class="ltx_ref">2024</a>)</cite> on 250M and 1.3B sizes.
For the hyperparameters, we use 10k steps, 1024 batch size, 512 sequence length and follow <cite class="ltx_cite ltx_citemacro_citeauthor"><a href="#bib.bib22" title="" class="ltx_ref">Lialin et al.</a></cite>, using rank <math id="S5.SS3.p1.1.m1.1" class="ltx_Math" alttext="r=128" display="inline"><semantics id="S5.SS3.p1.1.m1.1a"><mrow id="S5.SS3.p1.1.m1.1.1" xref="S5.SS3.p1.1.m1.1.1.cmml"><mi id="S5.SS3.p1.1.m1.1.1.2" xref="S5.SS3.p1.1.m1.1.1.2.cmml">r</mi><mo id="S5.SS3.p1.1.m1.1.1.1" xref="S5.SS3.p1.1.m1.1.1.1.cmml">=</mo><mn id="S5.SS3.p1.1.m1.1.1.3" xref="S5.SS3.p1.1.m1.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.SS3.p1.1.m1.1b"><apply id="S5.SS3.p1.1.m1.1.1.cmml" xref="S5.SS3.p1.1.m1.1.1"><eq id="S5.SS3.p1.1.m1.1.1.1.cmml" xref="S5.SS3.p1.1.m1.1.1.1"></eq><ci id="S5.SS3.p1.1.m1.1.1.2.cmml" xref="S5.SS3.p1.1.m1.1.1.2">𝑟</ci><cn type="integer" id="S5.SS3.p1.1.m1.1.1.3.cmml" xref="S5.SS3.p1.1.m1.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p1.1.m1.1c">r=128</annotation></semantics></math> for LoRA and our methods and also keep modules without applying LoRA-like layernorm or embeddings unfreezed.
We compare our method with LoRA and ReLoRA.
To better show the difference between high-rank and low-rank updating, we reproduce ReLoRA and other methods without full-rank training warmup.
For MoRA, we use compression and decompression functions in Eq. <a href="#S4.E6" title="In 4 Method ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> by sharing columns and rows.</p>
</div>
<div id="S5.SS3.p2" class="ltx_para">
<p id="S5.SS3.p2.9" class="ltx_p">We also combine merge-and-reint in ReLoRA with our method called ReMoRA by merging <math id="S5.SS3.p2.1.m1.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S5.SS3.p2.1.m1.1a"><mi id="S5.SS3.p2.1.m1.1.1" xref="S5.SS3.p2.1.m1.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S5.SS3.p2.1.m1.1b"><ci id="S5.SS3.p2.1.m1.1.1.cmml" xref="S5.SS3.p2.1.m1.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p2.1.m1.1c">M</annotation></semantics></math> back into the original parameters during training to increase the rank of <math id="S5.SS3.p2.2.m2.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S5.SS3.p2.2.m2.1a"><mrow id="S5.SS3.p2.2.m2.1.1" xref="S5.SS3.p2.2.m2.1.1.cmml"><mi mathvariant="normal" id="S5.SS3.p2.2.m2.1.1.2" xref="S5.SS3.p2.2.m2.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S5.SS3.p2.2.m2.1.1.1" xref="S5.SS3.p2.2.m2.1.1.1.cmml">​</mo><mi id="S5.SS3.p2.2.m2.1.1.3" xref="S5.SS3.p2.2.m2.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.SS3.p2.2.m2.1b"><apply id="S5.SS3.p2.2.m2.1.1.cmml" xref="S5.SS3.p2.2.m2.1.1"><times id="S5.SS3.p2.2.m2.1.1.1.cmml" xref="S5.SS3.p2.2.m2.1.1.1"></times><ci id="S5.SS3.p2.2.m2.1.1.2.cmml" xref="S5.SS3.p2.2.m2.1.1.2">Δ</ci><ci id="S5.SS3.p2.2.m2.1.1.3.cmml" xref="S5.SS3.p2.2.m2.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p2.2.m2.1c">\Delta W</annotation></semantics></math>.
However, if we directly merge <math id="S5.SS3.p2.3.m3.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S5.SS3.p2.3.m3.1a"><mi id="S5.SS3.p2.3.m3.1.1" xref="S5.SS3.p2.3.m3.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S5.SS3.p2.3.m3.1b"><ci id="S5.SS3.p2.3.m3.1.1.cmml" xref="S5.SS3.p2.3.m3.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p2.3.m3.1c">M</annotation></semantics></math> with <math id="S5.SS3.p2.4.m4.1" class="ltx_Math" alttext="g" display="inline"><semantics id="S5.SS3.p2.4.m4.1a"><mi id="S5.SS3.p2.4.m4.1.1" xref="S5.SS3.p2.4.m4.1.1.cmml">g</mi><annotation-xml encoding="MathML-Content" id="S5.SS3.p2.4.m4.1b"><ci id="S5.SS3.p2.4.m4.1.1.cmml" xref="S5.SS3.p2.4.m4.1.1">𝑔</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p2.4.m4.1c">g</annotation></semantics></math> and <math id="S5.SS3.p2.5.m5.1" class="ltx_Math" alttext="g^{\prime}" display="inline"><semantics id="S5.SS3.p2.5.m5.1a"><msup id="S5.SS3.p2.5.m5.1.1" xref="S5.SS3.p2.5.m5.1.1.cmml"><mi id="S5.SS3.p2.5.m5.1.1.2" xref="S5.SS3.p2.5.m5.1.1.2.cmml">g</mi><mo id="S5.SS3.p2.5.m5.1.1.3" xref="S5.SS3.p2.5.m5.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S5.SS3.p2.5.m5.1b"><apply id="S5.SS3.p2.5.m5.1.1.cmml" xref="S5.SS3.p2.5.m5.1.1"><csymbol cd="ambiguous" id="S5.SS3.p2.5.m5.1.1.1.cmml" xref="S5.SS3.p2.5.m5.1.1">superscript</csymbol><ci id="S5.SS3.p2.5.m5.1.1.2.cmml" xref="S5.SS3.p2.5.m5.1.1.2">𝑔</ci><ci id="S5.SS3.p2.5.m5.1.1.3.cmml" xref="S5.SS3.p2.5.m5.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p2.5.m5.1c">g^{\prime}</annotation></semantics></math> in Eq. <a href="#S4.E6" title="In 4 Method ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>, the final rank of <math id="S5.SS3.p2.6.m6.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S5.SS3.p2.6.m6.1a"><mrow id="S5.SS3.p2.6.m6.1.1" xref="S5.SS3.p2.6.m6.1.1.cmml"><mi mathvariant="normal" id="S5.SS3.p2.6.m6.1.1.2" xref="S5.SS3.p2.6.m6.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S5.SS3.p2.6.m6.1.1.1" xref="S5.SS3.p2.6.m6.1.1.1.cmml">​</mo><mi id="S5.SS3.p2.6.m6.1.1.3" xref="S5.SS3.p2.6.m6.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.SS3.p2.6.m6.1b"><apply id="S5.SS3.p2.6.m6.1.1.cmml" xref="S5.SS3.p2.6.m6.1.1"><times id="S5.SS3.p2.6.m6.1.1.1.cmml" xref="S5.SS3.p2.6.m6.1.1.1"></times><ci id="S5.SS3.p2.6.m6.1.1.2.cmml" xref="S5.SS3.p2.6.m6.1.1.2">Δ</ci><ci id="S5.SS3.p2.6.m6.1.1.3.cmml" xref="S5.SS3.p2.6.m6.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p2.6.m6.1c">\Delta W</annotation></semantics></math> is unchanged due to the same expand pattern.
To solve this problem, we can change <math id="S5.SS3.p2.7.m7.1" class="ltx_Math" alttext="g" display="inline"><semantics id="S5.SS3.p2.7.m7.1a"><mi id="S5.SS3.p2.7.m7.1.1" xref="S5.SS3.p2.7.m7.1.1.cmml">g</mi><annotation-xml encoding="MathML-Content" id="S5.SS3.p2.7.m7.1b"><ci id="S5.SS3.p2.7.m7.1.1.cmml" xref="S5.SS3.p2.7.m7.1.1">𝑔</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p2.7.m7.1c">g</annotation></semantics></math> and <math id="S5.SS3.p2.8.m8.1" class="ltx_Math" alttext="g^{\prime}" display="inline"><semantics id="S5.SS3.p2.8.m8.1a"><msup id="S5.SS3.p2.8.m8.1.1" xref="S5.SS3.p2.8.m8.1.1.cmml"><mi id="S5.SS3.p2.8.m8.1.1.2" xref="S5.SS3.p2.8.m8.1.1.2.cmml">g</mi><mo id="S5.SS3.p2.8.m8.1.1.3" xref="S5.SS3.p2.8.m8.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="S5.SS3.p2.8.m8.1b"><apply id="S5.SS3.p2.8.m8.1.1.cmml" xref="S5.SS3.p2.8.m8.1.1"><csymbol cd="ambiguous" id="S5.SS3.p2.8.m8.1.1.1.cmml" xref="S5.SS3.p2.8.m8.1.1">superscript</csymbol><ci id="S5.SS3.p2.8.m8.1.1.2.cmml" xref="S5.SS3.p2.8.m8.1.1.2">𝑔</ci><ci id="S5.SS3.p2.8.m8.1.1.3.cmml" xref="S5.SS3.p2.8.m8.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p2.8.m8.1c">g^{\prime}</annotation></semantics></math> after merging to ensure the rank of <math id="S5.SS3.p2.9.m9.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S5.SS3.p2.9.m9.1a"><mrow id="S5.SS3.p2.9.m9.1.1" xref="S5.SS3.p2.9.m9.1.1.cmml"><mi mathvariant="normal" id="S5.SS3.p2.9.m9.1.1.2" xref="S5.SS3.p2.9.m9.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S5.SS3.p2.9.m9.1.1.1" xref="S5.SS3.p2.9.m9.1.1.1.cmml">​</mo><mi id="S5.SS3.p2.9.m9.1.1.3" xref="S5.SS3.p2.9.m9.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.SS3.p2.9.m9.1b"><apply id="S5.SS3.p2.9.m9.1.1.cmml" xref="S5.SS3.p2.9.m9.1.1"><times id="S5.SS3.p2.9.m9.1.1.1.cmml" xref="S5.SS3.p2.9.m9.1.1.1"></times><ci id="S5.SS3.p2.9.m9.1.1.2.cmml" xref="S5.SS3.p2.9.m9.1.1.2">Δ</ci><ci id="S5.SS3.p2.9.m9.1.1.3.cmml" xref="S5.SS3.p2.9.m9.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p2.9.m9.1c">\Delta W</annotation></semantics></math> increasing.
More details about ReMoRA can be found in Appendix <a href="#A2" title="Appendix B Implementation of ReMoRA ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a>.
For the hyperparameters corresponding to ReLoRA and ReMoRA, we merge every 2k steps and use 50 steps restarts warmup with optimizer reseting and jagged scheduler.</p>
</div>
<div id="S5.SS3.p3" class="ltx_para">
<p id="S5.SS3.p3.1" class="ltx_p">We show pretraining loss in Figure <a href="#S5.F4" title="Figure 4 ‣ 5.2.1 Setup ‣ 5.2 Fine-tuning Tasks ‣ 5 Experiment ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> and corresponding perplexity on C4 validation dataset in Table <a href="#S5.T3" title="Table 3 ‣ 5.3 Pretraining ‣ 5 Experiment ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>.
Our method show better performance on pretraining compared to LoRA and ReLoRA with same amount of trainable parameters.
Benefiting from high-rank updating, ReMoRA also achieves more improvements on MoRA compared to ReLoRA, which
demonstrates the effectiveness of merge-and-reint strategy in ReMoRA.</p>
</div>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Analysis</h2>

<figure id="S6.F5" class="ltx_figure"><img src="/html/2405.12130/assets/svd.png" id="S6.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="441" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>
The number of singular values <math id="S6.F5.3.m1.1" class="ltx_Math" alttext="\textgreater 0.1" display="inline"><semantics id="S6.F5.3.m1.1b"><mrow id="S6.F5.3.m1.1.1" xref="S6.F5.3.m1.1.1.cmml"><mi id="S6.F5.3.m1.1.1.2" xref="S6.F5.3.m1.1.1.2.cmml"></mi><mo id="S6.F5.3.m1.1.1.1" xref="S6.F5.3.m1.1.1.1.cmml">&gt;</mo><mn id="S6.F5.3.m1.1.1.3" xref="S6.F5.3.m1.1.1.3.cmml">0.1</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.F5.3.m1.1c"><apply id="S6.F5.3.m1.1.1.cmml" xref="S6.F5.3.m1.1.1"><gt id="S6.F5.3.m1.1.1.1.cmml" xref="S6.F5.3.m1.1.1.1"></gt><csymbol cd="latexml" id="S6.F5.3.m1.1.1.2.cmml" xref="S6.F5.3.m1.1.1.2">absent</csymbol><cn type="float" id="S6.F5.3.m1.1.1.3.cmml" xref="S6.F5.3.m1.1.1.3">0.1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.F5.3.m1.1d">\textgreater 0.1</annotation></semantics></math> in <math id="S6.F5.4.m2.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S6.F5.4.m2.1b"><mrow id="S6.F5.4.m2.1.1" xref="S6.F5.4.m2.1.1.cmml"><mi mathvariant="normal" id="S6.F5.4.m2.1.1.2" xref="S6.F5.4.m2.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S6.F5.4.m2.1.1.1" xref="S6.F5.4.m2.1.1.1.cmml">​</mo><mi id="S6.F5.4.m2.1.1.3" xref="S6.F5.4.m2.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S6.F5.4.m2.1c"><apply id="S6.F5.4.m2.1.1.cmml" xref="S6.F5.4.m2.1.1"><times id="S6.F5.4.m2.1.1.1.cmml" xref="S6.F5.4.m2.1.1.1"></times><ci id="S6.F5.4.m2.1.1.2.cmml" xref="S6.F5.4.m2.1.1.2">Δ</ci><ci id="S6.F5.4.m2.1.1.3.cmml" xref="S6.F5.4.m2.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.F5.4.m2.1d">\Delta W</annotation></semantics></math> on the 250M pretraining model.
</figcaption>
</figure>
<section id="S6.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.1 </span>High-rank Updating</h3>

<div id="S6.SS1.p1" class="ltx_para">
<p id="S6.SS1.p1.10" class="ltx_p">To demonstrate the impact of high-rank updating on the rank of <math id="S6.SS1.p1.1.m1.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S6.SS1.p1.1.m1.1a"><mrow id="S6.SS1.p1.1.m1.1.1" xref="S6.SS1.p1.1.m1.1.1.cmml"><mi mathvariant="normal" id="S6.SS1.p1.1.m1.1.1.2" xref="S6.SS1.p1.1.m1.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S6.SS1.p1.1.m1.1.1.1" xref="S6.SS1.p1.1.m1.1.1.1.cmml">​</mo><mi id="S6.SS1.p1.1.m1.1.1.3" xref="S6.SS1.p1.1.m1.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.1.m1.1b"><apply id="S6.SS1.p1.1.m1.1.1.cmml" xref="S6.SS1.p1.1.m1.1.1"><times id="S6.SS1.p1.1.m1.1.1.1.cmml" xref="S6.SS1.p1.1.m1.1.1.1"></times><ci id="S6.SS1.p1.1.m1.1.1.2.cmml" xref="S6.SS1.p1.1.m1.1.1.2">Δ</ci><ci id="S6.SS1.p1.1.m1.1.1.3.cmml" xref="S6.SS1.p1.1.m1.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.1.m1.1c">\Delta W</annotation></semantics></math>, we analyzed the spectrum of singular values for the learned <math id="S6.SS1.p1.2.m2.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S6.SS1.p1.2.m2.1a"><mrow id="S6.SS1.p1.2.m2.1.1" xref="S6.SS1.p1.2.m2.1.1.cmml"><mi mathvariant="normal" id="S6.SS1.p1.2.m2.1.1.2" xref="S6.SS1.p1.2.m2.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S6.SS1.p1.2.m2.1.1.1" xref="S6.SS1.p1.2.m2.1.1.1.cmml">​</mo><mi id="S6.SS1.p1.2.m2.1.1.3" xref="S6.SS1.p1.2.m2.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.2.m2.1b"><apply id="S6.SS1.p1.2.m2.1.1.cmml" xref="S6.SS1.p1.2.m2.1.1"><times id="S6.SS1.p1.2.m2.1.1.1.cmml" xref="S6.SS1.p1.2.m2.1.1.1"></times><ci id="S6.SS1.p1.2.m2.1.1.2.cmml" xref="S6.SS1.p1.2.m2.1.1.2">Δ</ci><ci id="S6.SS1.p1.2.m2.1.1.3.cmml" xref="S6.SS1.p1.2.m2.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.2.m2.1c">\Delta W</annotation></semantics></math> on 250M pretraining 250M model.
We present the average count of singular values exceeding 0.1 across all layers for <math id="S6.SS1.p1.3.m3.1" class="ltx_Math" alttext="\Delta W_{q}" display="inline"><semantics id="S6.SS1.p1.3.m3.1a"><mrow id="S6.SS1.p1.3.m3.1.1" xref="S6.SS1.p1.3.m3.1.1.cmml"><mi mathvariant="normal" id="S6.SS1.p1.3.m3.1.1.2" xref="S6.SS1.p1.3.m3.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S6.SS1.p1.3.m3.1.1.1" xref="S6.SS1.p1.3.m3.1.1.1.cmml">​</mo><msub id="S6.SS1.p1.3.m3.1.1.3" xref="S6.SS1.p1.3.m3.1.1.3.cmml"><mi id="S6.SS1.p1.3.m3.1.1.3.2" xref="S6.SS1.p1.3.m3.1.1.3.2.cmml">W</mi><mi id="S6.SS1.p1.3.m3.1.1.3.3" xref="S6.SS1.p1.3.m3.1.1.3.3.cmml">q</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.3.m3.1b"><apply id="S6.SS1.p1.3.m3.1.1.cmml" xref="S6.SS1.p1.3.m3.1.1"><times id="S6.SS1.p1.3.m3.1.1.1.cmml" xref="S6.SS1.p1.3.m3.1.1.1"></times><ci id="S6.SS1.p1.3.m3.1.1.2.cmml" xref="S6.SS1.p1.3.m3.1.1.2">Δ</ci><apply id="S6.SS1.p1.3.m3.1.1.3.cmml" xref="S6.SS1.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="S6.SS1.p1.3.m3.1.1.3.1.cmml" xref="S6.SS1.p1.3.m3.1.1.3">subscript</csymbol><ci id="S6.SS1.p1.3.m3.1.1.3.2.cmml" xref="S6.SS1.p1.3.m3.1.1.3.2">𝑊</ci><ci id="S6.SS1.p1.3.m3.1.1.3.3.cmml" xref="S6.SS1.p1.3.m3.1.1.3.3">𝑞</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.3.m3.1c">\Delta W_{q}</annotation></semantics></math>, <math id="S6.SS1.p1.4.m4.1" class="ltx_Math" alttext="\Delta W_{k}" display="inline"><semantics id="S6.SS1.p1.4.m4.1a"><mrow id="S6.SS1.p1.4.m4.1.1" xref="S6.SS1.p1.4.m4.1.1.cmml"><mi mathvariant="normal" id="S6.SS1.p1.4.m4.1.1.2" xref="S6.SS1.p1.4.m4.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S6.SS1.p1.4.m4.1.1.1" xref="S6.SS1.p1.4.m4.1.1.1.cmml">​</mo><msub id="S6.SS1.p1.4.m4.1.1.3" xref="S6.SS1.p1.4.m4.1.1.3.cmml"><mi id="S6.SS1.p1.4.m4.1.1.3.2" xref="S6.SS1.p1.4.m4.1.1.3.2.cmml">W</mi><mi id="S6.SS1.p1.4.m4.1.1.3.3" xref="S6.SS1.p1.4.m4.1.1.3.3.cmml">k</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.4.m4.1b"><apply id="S6.SS1.p1.4.m4.1.1.cmml" xref="S6.SS1.p1.4.m4.1.1"><times id="S6.SS1.p1.4.m4.1.1.1.cmml" xref="S6.SS1.p1.4.m4.1.1.1"></times><ci id="S6.SS1.p1.4.m4.1.1.2.cmml" xref="S6.SS1.p1.4.m4.1.1.2">Δ</ci><apply id="S6.SS1.p1.4.m4.1.1.3.cmml" xref="S6.SS1.p1.4.m4.1.1.3"><csymbol cd="ambiguous" id="S6.SS1.p1.4.m4.1.1.3.1.cmml" xref="S6.SS1.p1.4.m4.1.1.3">subscript</csymbol><ci id="S6.SS1.p1.4.m4.1.1.3.2.cmml" xref="S6.SS1.p1.4.m4.1.1.3.2">𝑊</ci><ci id="S6.SS1.p1.4.m4.1.1.3.3.cmml" xref="S6.SS1.p1.4.m4.1.1.3.3">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.4.m4.1c">\Delta W_{k}</annotation></semantics></math>, <math id="S6.SS1.p1.5.m5.1" class="ltx_Math" alttext="\Delta W_{v}" display="inline"><semantics id="S6.SS1.p1.5.m5.1a"><mrow id="S6.SS1.p1.5.m5.1.1" xref="S6.SS1.p1.5.m5.1.1.cmml"><mi mathvariant="normal" id="S6.SS1.p1.5.m5.1.1.2" xref="S6.SS1.p1.5.m5.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S6.SS1.p1.5.m5.1.1.1" xref="S6.SS1.p1.5.m5.1.1.1.cmml">​</mo><msub id="S6.SS1.p1.5.m5.1.1.3" xref="S6.SS1.p1.5.m5.1.1.3.cmml"><mi id="S6.SS1.p1.5.m5.1.1.3.2" xref="S6.SS1.p1.5.m5.1.1.3.2.cmml">W</mi><mi id="S6.SS1.p1.5.m5.1.1.3.3" xref="S6.SS1.p1.5.m5.1.1.3.3.cmml">v</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.5.m5.1b"><apply id="S6.SS1.p1.5.m5.1.1.cmml" xref="S6.SS1.p1.5.m5.1.1"><times id="S6.SS1.p1.5.m5.1.1.1.cmml" xref="S6.SS1.p1.5.m5.1.1.1"></times><ci id="S6.SS1.p1.5.m5.1.1.2.cmml" xref="S6.SS1.p1.5.m5.1.1.2">Δ</ci><apply id="S6.SS1.p1.5.m5.1.1.3.cmml" xref="S6.SS1.p1.5.m5.1.1.3"><csymbol cd="ambiguous" id="S6.SS1.p1.5.m5.1.1.3.1.cmml" xref="S6.SS1.p1.5.m5.1.1.3">subscript</csymbol><ci id="S6.SS1.p1.5.m5.1.1.3.2.cmml" xref="S6.SS1.p1.5.m5.1.1.3.2">𝑊</ci><ci id="S6.SS1.p1.5.m5.1.1.3.3.cmml" xref="S6.SS1.p1.5.m5.1.1.3.3">𝑣</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.5.m5.1c">\Delta W_{v}</annotation></semantics></math>, <math id="S6.SS1.p1.6.m6.1" class="ltx_Math" alttext="\Delta W_{o}" display="inline"><semantics id="S6.SS1.p1.6.m6.1a"><mrow id="S6.SS1.p1.6.m6.1.1" xref="S6.SS1.p1.6.m6.1.1.cmml"><mi mathvariant="normal" id="S6.SS1.p1.6.m6.1.1.2" xref="S6.SS1.p1.6.m6.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S6.SS1.p1.6.m6.1.1.1" xref="S6.SS1.p1.6.m6.1.1.1.cmml">​</mo><msub id="S6.SS1.p1.6.m6.1.1.3" xref="S6.SS1.p1.6.m6.1.1.3.cmml"><mi id="S6.SS1.p1.6.m6.1.1.3.2" xref="S6.SS1.p1.6.m6.1.1.3.2.cmml">W</mi><mi id="S6.SS1.p1.6.m6.1.1.3.3" xref="S6.SS1.p1.6.m6.1.1.3.3.cmml">o</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.6.m6.1b"><apply id="S6.SS1.p1.6.m6.1.1.cmml" xref="S6.SS1.p1.6.m6.1.1"><times id="S6.SS1.p1.6.m6.1.1.1.cmml" xref="S6.SS1.p1.6.m6.1.1.1"></times><ci id="S6.SS1.p1.6.m6.1.1.2.cmml" xref="S6.SS1.p1.6.m6.1.1.2">Δ</ci><apply id="S6.SS1.p1.6.m6.1.1.3.cmml" xref="S6.SS1.p1.6.m6.1.1.3"><csymbol cd="ambiguous" id="S6.SS1.p1.6.m6.1.1.3.1.cmml" xref="S6.SS1.p1.6.m6.1.1.3">subscript</csymbol><ci id="S6.SS1.p1.6.m6.1.1.3.2.cmml" xref="S6.SS1.p1.6.m6.1.1.3.2">𝑊</ci><ci id="S6.SS1.p1.6.m6.1.1.3.3.cmml" xref="S6.SS1.p1.6.m6.1.1.3.3">𝑜</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.6.m6.1c">\Delta W_{o}</annotation></semantics></math>, <math id="S6.SS1.p1.7.m7.1" class="ltx_Math" alttext="\Delta W_{up}" display="inline"><semantics id="S6.SS1.p1.7.m7.1a"><mrow id="S6.SS1.p1.7.m7.1.1" xref="S6.SS1.p1.7.m7.1.1.cmml"><mi mathvariant="normal" id="S6.SS1.p1.7.m7.1.1.2" xref="S6.SS1.p1.7.m7.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S6.SS1.p1.7.m7.1.1.1" xref="S6.SS1.p1.7.m7.1.1.1.cmml">​</mo><msub id="S6.SS1.p1.7.m7.1.1.3" xref="S6.SS1.p1.7.m7.1.1.3.cmml"><mi id="S6.SS1.p1.7.m7.1.1.3.2" xref="S6.SS1.p1.7.m7.1.1.3.2.cmml">W</mi><mrow id="S6.SS1.p1.7.m7.1.1.3.3" xref="S6.SS1.p1.7.m7.1.1.3.3.cmml"><mi id="S6.SS1.p1.7.m7.1.1.3.3.2" xref="S6.SS1.p1.7.m7.1.1.3.3.2.cmml">u</mi><mo lspace="0em" rspace="0em" id="S6.SS1.p1.7.m7.1.1.3.3.1" xref="S6.SS1.p1.7.m7.1.1.3.3.1.cmml">​</mo><mi id="S6.SS1.p1.7.m7.1.1.3.3.3" xref="S6.SS1.p1.7.m7.1.1.3.3.3.cmml">p</mi></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.7.m7.1b"><apply id="S6.SS1.p1.7.m7.1.1.cmml" xref="S6.SS1.p1.7.m7.1.1"><times id="S6.SS1.p1.7.m7.1.1.1.cmml" xref="S6.SS1.p1.7.m7.1.1.1"></times><ci id="S6.SS1.p1.7.m7.1.1.2.cmml" xref="S6.SS1.p1.7.m7.1.1.2">Δ</ci><apply id="S6.SS1.p1.7.m7.1.1.3.cmml" xref="S6.SS1.p1.7.m7.1.1.3"><csymbol cd="ambiguous" id="S6.SS1.p1.7.m7.1.1.3.1.cmml" xref="S6.SS1.p1.7.m7.1.1.3">subscript</csymbol><ci id="S6.SS1.p1.7.m7.1.1.3.2.cmml" xref="S6.SS1.p1.7.m7.1.1.3.2">𝑊</ci><apply id="S6.SS1.p1.7.m7.1.1.3.3.cmml" xref="S6.SS1.p1.7.m7.1.1.3.3"><times id="S6.SS1.p1.7.m7.1.1.3.3.1.cmml" xref="S6.SS1.p1.7.m7.1.1.3.3.1"></times><ci id="S6.SS1.p1.7.m7.1.1.3.3.2.cmml" xref="S6.SS1.p1.7.m7.1.1.3.3.2">𝑢</ci><ci id="S6.SS1.p1.7.m7.1.1.3.3.3.cmml" xref="S6.SS1.p1.7.m7.1.1.3.3.3">𝑝</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.7.m7.1c">\Delta W_{up}</annotation></semantics></math>, <math id="S6.SS1.p1.8.m8.1" class="ltx_Math" alttext="\Delta W_{down}" display="inline"><semantics id="S6.SS1.p1.8.m8.1a"><mrow id="S6.SS1.p1.8.m8.1.1" xref="S6.SS1.p1.8.m8.1.1.cmml"><mi mathvariant="normal" id="S6.SS1.p1.8.m8.1.1.2" xref="S6.SS1.p1.8.m8.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S6.SS1.p1.8.m8.1.1.1" xref="S6.SS1.p1.8.m8.1.1.1.cmml">​</mo><msub id="S6.SS1.p1.8.m8.1.1.3" xref="S6.SS1.p1.8.m8.1.1.3.cmml"><mi id="S6.SS1.p1.8.m8.1.1.3.2" xref="S6.SS1.p1.8.m8.1.1.3.2.cmml">W</mi><mrow id="S6.SS1.p1.8.m8.1.1.3.3" xref="S6.SS1.p1.8.m8.1.1.3.3.cmml"><mi id="S6.SS1.p1.8.m8.1.1.3.3.2" xref="S6.SS1.p1.8.m8.1.1.3.3.2.cmml">d</mi><mo lspace="0em" rspace="0em" id="S6.SS1.p1.8.m8.1.1.3.3.1" xref="S6.SS1.p1.8.m8.1.1.3.3.1.cmml">​</mo><mi id="S6.SS1.p1.8.m8.1.1.3.3.3" xref="S6.SS1.p1.8.m8.1.1.3.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S6.SS1.p1.8.m8.1.1.3.3.1a" xref="S6.SS1.p1.8.m8.1.1.3.3.1.cmml">​</mo><mi id="S6.SS1.p1.8.m8.1.1.3.3.4" xref="S6.SS1.p1.8.m8.1.1.3.3.4.cmml">w</mi><mo lspace="0em" rspace="0em" id="S6.SS1.p1.8.m8.1.1.3.3.1b" xref="S6.SS1.p1.8.m8.1.1.3.3.1.cmml">​</mo><mi id="S6.SS1.p1.8.m8.1.1.3.3.5" xref="S6.SS1.p1.8.m8.1.1.3.3.5.cmml">n</mi></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.8.m8.1b"><apply id="S6.SS1.p1.8.m8.1.1.cmml" xref="S6.SS1.p1.8.m8.1.1"><times id="S6.SS1.p1.8.m8.1.1.1.cmml" xref="S6.SS1.p1.8.m8.1.1.1"></times><ci id="S6.SS1.p1.8.m8.1.1.2.cmml" xref="S6.SS1.p1.8.m8.1.1.2">Δ</ci><apply id="S6.SS1.p1.8.m8.1.1.3.cmml" xref="S6.SS1.p1.8.m8.1.1.3"><csymbol cd="ambiguous" id="S6.SS1.p1.8.m8.1.1.3.1.cmml" xref="S6.SS1.p1.8.m8.1.1.3">subscript</csymbol><ci id="S6.SS1.p1.8.m8.1.1.3.2.cmml" xref="S6.SS1.p1.8.m8.1.1.3.2">𝑊</ci><apply id="S6.SS1.p1.8.m8.1.1.3.3.cmml" xref="S6.SS1.p1.8.m8.1.1.3.3"><times id="S6.SS1.p1.8.m8.1.1.3.3.1.cmml" xref="S6.SS1.p1.8.m8.1.1.3.3.1"></times><ci id="S6.SS1.p1.8.m8.1.1.3.3.2.cmml" xref="S6.SS1.p1.8.m8.1.1.3.3.2">𝑑</ci><ci id="S6.SS1.p1.8.m8.1.1.3.3.3.cmml" xref="S6.SS1.p1.8.m8.1.1.3.3.3">𝑜</ci><ci id="S6.SS1.p1.8.m8.1.1.3.3.4.cmml" xref="S6.SS1.p1.8.m8.1.1.3.3.4">𝑤</ci><ci id="S6.SS1.p1.8.m8.1.1.3.3.5.cmml" xref="S6.SS1.p1.8.m8.1.1.3.3.5">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.8.m8.1c">\Delta W_{down}</annotation></semantics></math>, and <math id="S6.SS1.p1.9.m9.1" class="ltx_Math" alttext="\Delta W_{gate}" display="inline"><semantics id="S6.SS1.p1.9.m9.1a"><mrow id="S6.SS1.p1.9.m9.1.1" xref="S6.SS1.p1.9.m9.1.1.cmml"><mi mathvariant="normal" id="S6.SS1.p1.9.m9.1.1.2" xref="S6.SS1.p1.9.m9.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S6.SS1.p1.9.m9.1.1.1" xref="S6.SS1.p1.9.m9.1.1.1.cmml">​</mo><msub id="S6.SS1.p1.9.m9.1.1.3" xref="S6.SS1.p1.9.m9.1.1.3.cmml"><mi id="S6.SS1.p1.9.m9.1.1.3.2" xref="S6.SS1.p1.9.m9.1.1.3.2.cmml">W</mi><mrow id="S6.SS1.p1.9.m9.1.1.3.3" xref="S6.SS1.p1.9.m9.1.1.3.3.cmml"><mi id="S6.SS1.p1.9.m9.1.1.3.3.2" xref="S6.SS1.p1.9.m9.1.1.3.3.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="S6.SS1.p1.9.m9.1.1.3.3.1" xref="S6.SS1.p1.9.m9.1.1.3.3.1.cmml">​</mo><mi id="S6.SS1.p1.9.m9.1.1.3.3.3" xref="S6.SS1.p1.9.m9.1.1.3.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="S6.SS1.p1.9.m9.1.1.3.3.1a" xref="S6.SS1.p1.9.m9.1.1.3.3.1.cmml">​</mo><mi id="S6.SS1.p1.9.m9.1.1.3.3.4" xref="S6.SS1.p1.9.m9.1.1.3.3.4.cmml">t</mi><mo lspace="0em" rspace="0em" id="S6.SS1.p1.9.m9.1.1.3.3.1b" xref="S6.SS1.p1.9.m9.1.1.3.3.1.cmml">​</mo><mi id="S6.SS1.p1.9.m9.1.1.3.3.5" xref="S6.SS1.p1.9.m9.1.1.3.3.5.cmml">e</mi></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.9.m9.1b"><apply id="S6.SS1.p1.9.m9.1.1.cmml" xref="S6.SS1.p1.9.m9.1.1"><times id="S6.SS1.p1.9.m9.1.1.1.cmml" xref="S6.SS1.p1.9.m9.1.1.1"></times><ci id="S6.SS1.p1.9.m9.1.1.2.cmml" xref="S6.SS1.p1.9.m9.1.1.2">Δ</ci><apply id="S6.SS1.p1.9.m9.1.1.3.cmml" xref="S6.SS1.p1.9.m9.1.1.3"><csymbol cd="ambiguous" id="S6.SS1.p1.9.m9.1.1.3.1.cmml" xref="S6.SS1.p1.9.m9.1.1.3">subscript</csymbol><ci id="S6.SS1.p1.9.m9.1.1.3.2.cmml" xref="S6.SS1.p1.9.m9.1.1.3.2">𝑊</ci><apply id="S6.SS1.p1.9.m9.1.1.3.3.cmml" xref="S6.SS1.p1.9.m9.1.1.3.3"><times id="S6.SS1.p1.9.m9.1.1.3.3.1.cmml" xref="S6.SS1.p1.9.m9.1.1.3.3.1"></times><ci id="S6.SS1.p1.9.m9.1.1.3.3.2.cmml" xref="S6.SS1.p1.9.m9.1.1.3.3.2">𝑔</ci><ci id="S6.SS1.p1.9.m9.1.1.3.3.3.cmml" xref="S6.SS1.p1.9.m9.1.1.3.3.3">𝑎</ci><ci id="S6.SS1.p1.9.m9.1.1.3.3.4.cmml" xref="S6.SS1.p1.9.m9.1.1.3.3.4">𝑡</ci><ci id="S6.SS1.p1.9.m9.1.1.3.3.5.cmml" xref="S6.SS1.p1.9.m9.1.1.3.3.5">𝑒</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.9.m9.1c">\Delta W_{gate}</annotation></semantics></math> in Figure <a href="#S6.F5" title="Figure 5 ‣ 6 Analysis ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> following <cite class="ltx_cite ltx_citemacro_cite">Lialin et al. (<a href="#bib.bib22" title="" class="ltx_ref">2023</a>)</cite>.
MoRA and ReMoRA exhibit a substantially higher number of significant singular values compared to LoRA and ReLoRA, highlighting the effectiveness of our methods in increasing the rank of <math id="S6.SS1.p1.10.m10.1" class="ltx_Math" alttext="\Delta W" display="inline"><semantics id="S6.SS1.p1.10.m10.1a"><mrow id="S6.SS1.p1.10.m10.1.1" xref="S6.SS1.p1.10.m10.1.1.cmml"><mi mathvariant="normal" id="S6.SS1.p1.10.m10.1.1.2" xref="S6.SS1.p1.10.m10.1.1.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S6.SS1.p1.10.m10.1.1.1" xref="S6.SS1.p1.10.m10.1.1.1.cmml">​</mo><mi id="S6.SS1.p1.10.m10.1.1.3" xref="S6.SS1.p1.10.m10.1.1.3.cmml">W</mi></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.10.m10.1b"><apply id="S6.SS1.p1.10.m10.1.1.cmml" xref="S6.SS1.p1.10.m10.1.1"><times id="S6.SS1.p1.10.m10.1.1.1.cmml" xref="S6.SS1.p1.10.m10.1.1.1"></times><ci id="S6.SS1.p1.10.m10.1.1.2.cmml" xref="S6.SS1.p1.10.m10.1.1.2">Δ</ci><ci id="S6.SS1.p1.10.m10.1.1.3.cmml" xref="S6.SS1.p1.10.m10.1.1.3">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.10.m10.1c">\Delta W</annotation></semantics></math>. We find the quantity of singular values shown in Figure <a href="#S6.F5" title="Figure 5 ‣ 6 Analysis ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> can be correlated with the perplexity metrics listed in Table <a href="#S5.T3" title="Table 3 ‣ 5.3 Pretraining ‣ 5 Experiment ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. Moreover, MoRA, without merge-and-reint strategy in ReLoRA and ReMoRA, can achieve a lower perplexity than ReLoRA along with a higher significant singular values.</p>
</div>
</section>
<section id="S6.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.2 </span>Influence of Decompression and Compression</h3>

<div id="S6.SS2.p1" class="ltx_para">
<p id="S6.SS2.p1.1" class="ltx_p">To explore the impact of decompression and compression functions in MoRA, we report the performance on GSM8K using various methods: truncation, sharing, decoupling, and rotation in Table <a href="#S6.T4" title="Table 4 ‣ 6.2 Influence of Decompression and Compression ‣ 6 Analysis ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>.
Among these methods, truncation shows the worst performance due to the significant information loss during compression.
Sharing can achieve better performance than truncation by leveraging the shared rows or columns to preserve the input information.
But in the case of <math id="S6.SS2.p1.1.m1.1" class="ltx_Math" alttext="r=8" display="inline"><semantics id="S6.SS2.p1.1.m1.1a"><mrow id="S6.SS2.p1.1.m1.1.1" xref="S6.SS2.p1.1.m1.1.1.cmml"><mi id="S6.SS2.p1.1.m1.1.1.2" xref="S6.SS2.p1.1.m1.1.1.2.cmml">r</mi><mo id="S6.SS2.p1.1.m1.1.1.1" xref="S6.SS2.p1.1.m1.1.1.1.cmml">=</mo><mn id="S6.SS2.p1.1.m1.1.1.3" xref="S6.SS2.p1.1.m1.1.1.3.cmml">8</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.1.m1.1b"><apply id="S6.SS2.p1.1.m1.1.1.cmml" xref="S6.SS2.p1.1.m1.1.1"><eq id="S6.SS2.p1.1.m1.1.1.1.cmml" xref="S6.SS2.p1.1.m1.1.1.1"></eq><ci id="S6.SS2.p1.1.m1.1.1.2.cmml" xref="S6.SS2.p1.1.m1.1.1.2">𝑟</ci><cn type="integer" id="S6.SS2.p1.1.m1.1.1.3.cmml" xref="S6.SS2.p1.1.m1.1.1.3">8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.1.m1.1c">r=8</annotation></semantics></math>, sharing shows worse performance than decouple and rotation due to the large number of sharing rows or columns, as we discussed in Section <a href="#S4" title="4 Method ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>.
Rotation is more efficient than decouple, due to the rotation information can help the square matrix to distinguish the input information.</p>
</div>
<figure id="S6.T4" class="ltx_table">
<table id="S6.T4.4" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S6.T4.4.4" class="ltx_tr">
<td id="S6.T4.4.4.5" class="ltx_td ltx_border_tt"></td>
<th id="S6.T4.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">
<math id="S6.T4.1.1.1.m1.1" class="ltx_Math" alttext="f_{comp}" display="inline"><semantics id="S6.T4.1.1.1.m1.1a"><msub id="S6.T4.1.1.1.m1.1.1" xref="S6.T4.1.1.1.m1.1.1.cmml"><mi id="S6.T4.1.1.1.m1.1.1.2" xref="S6.T4.1.1.1.m1.1.1.2.cmml">f</mi><mrow id="S6.T4.1.1.1.m1.1.1.3" xref="S6.T4.1.1.1.m1.1.1.3.cmml"><mi id="S6.T4.1.1.1.m1.1.1.3.2" xref="S6.T4.1.1.1.m1.1.1.3.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="S6.T4.1.1.1.m1.1.1.3.1" xref="S6.T4.1.1.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.T4.1.1.1.m1.1.1.3.3" xref="S6.T4.1.1.1.m1.1.1.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S6.T4.1.1.1.m1.1.1.3.1a" xref="S6.T4.1.1.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.T4.1.1.1.m1.1.1.3.4" xref="S6.T4.1.1.1.m1.1.1.3.4.cmml">m</mi><mo lspace="0em" rspace="0em" id="S6.T4.1.1.1.m1.1.1.3.1b" xref="S6.T4.1.1.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.T4.1.1.1.m1.1.1.3.5" xref="S6.T4.1.1.1.m1.1.1.3.5.cmml">p</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S6.T4.1.1.1.m1.1b"><apply id="S6.T4.1.1.1.m1.1.1.cmml" xref="S6.T4.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="S6.T4.1.1.1.m1.1.1.1.cmml" xref="S6.T4.1.1.1.m1.1.1">subscript</csymbol><ci id="S6.T4.1.1.1.m1.1.1.2.cmml" xref="S6.T4.1.1.1.m1.1.1.2">𝑓</ci><apply id="S6.T4.1.1.1.m1.1.1.3.cmml" xref="S6.T4.1.1.1.m1.1.1.3"><times id="S6.T4.1.1.1.m1.1.1.3.1.cmml" xref="S6.T4.1.1.1.m1.1.1.3.1"></times><ci id="S6.T4.1.1.1.m1.1.1.3.2.cmml" xref="S6.T4.1.1.1.m1.1.1.3.2">𝑐</ci><ci id="S6.T4.1.1.1.m1.1.1.3.3.cmml" xref="S6.T4.1.1.1.m1.1.1.3.3">𝑜</ci><ci id="S6.T4.1.1.1.m1.1.1.3.4.cmml" xref="S6.T4.1.1.1.m1.1.1.3.4">𝑚</ci><ci id="S6.T4.1.1.1.m1.1.1.3.5.cmml" xref="S6.T4.1.1.1.m1.1.1.3.5">𝑝</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T4.1.1.1.m1.1c">f_{comp}</annotation></semantics></math>, <math id="S6.T4.2.2.2.m2.1" class="ltx_Math" alttext="f_{decomp}" display="inline"><semantics id="S6.T4.2.2.2.m2.1a"><msub id="S6.T4.2.2.2.m2.1.1" xref="S6.T4.2.2.2.m2.1.1.cmml"><mi id="S6.T4.2.2.2.m2.1.1.2" xref="S6.T4.2.2.2.m2.1.1.2.cmml">f</mi><mrow id="S6.T4.2.2.2.m2.1.1.3" xref="S6.T4.2.2.2.m2.1.1.3.cmml"><mi id="S6.T4.2.2.2.m2.1.1.3.2" xref="S6.T4.2.2.2.m2.1.1.3.2.cmml">d</mi><mo lspace="0em" rspace="0em" id="S6.T4.2.2.2.m2.1.1.3.1" xref="S6.T4.2.2.2.m2.1.1.3.1.cmml">​</mo><mi id="S6.T4.2.2.2.m2.1.1.3.3" xref="S6.T4.2.2.2.m2.1.1.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S6.T4.2.2.2.m2.1.1.3.1a" xref="S6.T4.2.2.2.m2.1.1.3.1.cmml">​</mo><mi id="S6.T4.2.2.2.m2.1.1.3.4" xref="S6.T4.2.2.2.m2.1.1.3.4.cmml">c</mi><mo lspace="0em" rspace="0em" id="S6.T4.2.2.2.m2.1.1.3.1b" xref="S6.T4.2.2.2.m2.1.1.3.1.cmml">​</mo><mi id="S6.T4.2.2.2.m2.1.1.3.5" xref="S6.T4.2.2.2.m2.1.1.3.5.cmml">o</mi><mo lspace="0em" rspace="0em" id="S6.T4.2.2.2.m2.1.1.3.1c" xref="S6.T4.2.2.2.m2.1.1.3.1.cmml">​</mo><mi id="S6.T4.2.2.2.m2.1.1.3.6" xref="S6.T4.2.2.2.m2.1.1.3.6.cmml">m</mi><mo lspace="0em" rspace="0em" id="S6.T4.2.2.2.m2.1.1.3.1d" xref="S6.T4.2.2.2.m2.1.1.3.1.cmml">​</mo><mi id="S6.T4.2.2.2.m2.1.1.3.7" xref="S6.T4.2.2.2.m2.1.1.3.7.cmml">p</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S6.T4.2.2.2.m2.1b"><apply id="S6.T4.2.2.2.m2.1.1.cmml" xref="S6.T4.2.2.2.m2.1.1"><csymbol cd="ambiguous" id="S6.T4.2.2.2.m2.1.1.1.cmml" xref="S6.T4.2.2.2.m2.1.1">subscript</csymbol><ci id="S6.T4.2.2.2.m2.1.1.2.cmml" xref="S6.T4.2.2.2.m2.1.1.2">𝑓</ci><apply id="S6.T4.2.2.2.m2.1.1.3.cmml" xref="S6.T4.2.2.2.m2.1.1.3"><times id="S6.T4.2.2.2.m2.1.1.3.1.cmml" xref="S6.T4.2.2.2.m2.1.1.3.1"></times><ci id="S6.T4.2.2.2.m2.1.1.3.2.cmml" xref="S6.T4.2.2.2.m2.1.1.3.2">𝑑</ci><ci id="S6.T4.2.2.2.m2.1.1.3.3.cmml" xref="S6.T4.2.2.2.m2.1.1.3.3">𝑒</ci><ci id="S6.T4.2.2.2.m2.1.1.3.4.cmml" xref="S6.T4.2.2.2.m2.1.1.3.4">𝑐</ci><ci id="S6.T4.2.2.2.m2.1.1.3.5.cmml" xref="S6.T4.2.2.2.m2.1.1.3.5">𝑜</ci><ci id="S6.T4.2.2.2.m2.1.1.3.6.cmml" xref="S6.T4.2.2.2.m2.1.1.3.6">𝑚</ci><ci id="S6.T4.2.2.2.m2.1.1.3.7.cmml" xref="S6.T4.2.2.2.m2.1.1.3.7">𝑝</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T4.2.2.2.m2.1c">f_{decomp}</annotation></semantics></math>
</th>
<th id="S6.T4.3.3.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><math id="S6.T4.3.3.3.m1.1" class="ltx_Math" alttext="r=8" display="inline"><semantics id="S6.T4.3.3.3.m1.1a"><mrow id="S6.T4.3.3.3.m1.1.1" xref="S6.T4.3.3.3.m1.1.1.cmml"><mi id="S6.T4.3.3.3.m1.1.1.2" xref="S6.T4.3.3.3.m1.1.1.2.cmml">r</mi><mo id="S6.T4.3.3.3.m1.1.1.1" xref="S6.T4.3.3.3.m1.1.1.1.cmml">=</mo><mn id="S6.T4.3.3.3.m1.1.1.3" xref="S6.T4.3.3.3.m1.1.1.3.cmml">8</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T4.3.3.3.m1.1b"><apply id="S6.T4.3.3.3.m1.1.1.cmml" xref="S6.T4.3.3.3.m1.1.1"><eq id="S6.T4.3.3.3.m1.1.1.1.cmml" xref="S6.T4.3.3.3.m1.1.1.1"></eq><ci id="S6.T4.3.3.3.m1.1.1.2.cmml" xref="S6.T4.3.3.3.m1.1.1.2">𝑟</ci><cn type="integer" id="S6.T4.3.3.3.m1.1.1.3.cmml" xref="S6.T4.3.3.3.m1.1.1.3">8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T4.3.3.3.m1.1c">r=8</annotation></semantics></math></th>
<th id="S6.T4.4.4.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><math id="S6.T4.4.4.4.m1.1" class="ltx_Math" alttext="r=256" display="inline"><semantics id="S6.T4.4.4.4.m1.1a"><mrow id="S6.T4.4.4.4.m1.1.1" xref="S6.T4.4.4.4.m1.1.1.cmml"><mi id="S6.T4.4.4.4.m1.1.1.2" xref="S6.T4.4.4.4.m1.1.1.2.cmml">r</mi><mo id="S6.T4.4.4.4.m1.1.1.1" xref="S6.T4.4.4.4.m1.1.1.1.cmml">=</mo><mn id="S6.T4.4.4.4.m1.1.1.3" xref="S6.T4.4.4.4.m1.1.1.3.cmml">256</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T4.4.4.4.m1.1b"><apply id="S6.T4.4.4.4.m1.1.1.cmml" xref="S6.T4.4.4.4.m1.1.1"><eq id="S6.T4.4.4.4.m1.1.1.1.cmml" xref="S6.T4.4.4.4.m1.1.1.1"></eq><ci id="S6.T4.4.4.4.m1.1.1.2.cmml" xref="S6.T4.4.4.4.m1.1.1.2">𝑟</ci><cn type="integer" id="S6.T4.4.4.4.m1.1.1.3.cmml" xref="S6.T4.4.4.4.m1.1.1.3">256</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T4.4.4.4.m1.1c">r=256</annotation></semantics></math></th>
</tr>
<tr id="S6.T4.4.5.1" class="ltx_tr">
<td id="S6.T4.4.5.1.1" class="ltx_td ltx_align_center ltx_border_t">Truncation</td>
<td id="S6.T4.4.5.1.2" class="ltx_td ltx_align_center ltx_border_t">Eq. <a href="#S4.E4" title="In 4 Method ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>
</td>
<td id="S6.T4.4.5.1.3" class="ltx_td ltx_align_center ltx_border_t">59.5</td>
<td id="S6.T4.4.5.1.4" class="ltx_td ltx_align_center ltx_border_t">66.6</td>
</tr>
<tr id="S6.T4.4.6.2" class="ltx_tr">
<td id="S6.T4.4.6.2.1" class="ltx_td ltx_align_center">Sharing</td>
<td id="S6.T4.4.6.2.2" class="ltx_td ltx_align_center">Eq. <a href="#S4.E6" title="In 4 Method ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>
</td>
<td id="S6.T4.4.6.2.3" class="ltx_td ltx_align_center">62.5</td>
<td id="S6.T4.4.6.2.4" class="ltx_td ltx_align_center">67.9</td>
</tr>
<tr id="S6.T4.4.7.3" class="ltx_tr">
<td id="S6.T4.4.7.3.1" class="ltx_td ltx_align_center">Decouple</td>
<td id="S6.T4.4.7.3.2" class="ltx_td ltx_align_center">Eq. <a href="#S4.E8" title="In 4 Method ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>
</td>
<td id="S6.T4.4.7.3.3" class="ltx_td ltx_align_center">63.6</td>
<td id="S6.T4.4.7.3.4" class="ltx_td ltx_align_center">67.8</td>
</tr>
<tr id="S6.T4.4.8.4" class="ltx_tr">
<td id="S6.T4.4.8.4.1" class="ltx_td ltx_align_center ltx_border_bb">Rotation</td>
<td id="S6.T4.4.8.4.2" class="ltx_td ltx_align_center ltx_border_bb">Eq. <a href="#S4.E9" title="In 4 Method ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>
</td>
<td id="S6.T4.4.8.4.3" class="ltx_td ltx_align_center ltx_border_bb">64.2</td>
<td id="S6.T4.4.8.4.4" class="ltx_td ltx_align_center ltx_border_bb">67.9</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>Influence of decompression and compression functions on <math id="S6.T4.7.m1.1" class="ltx_Math" alttext="r=8" display="inline"><semantics id="S6.T4.7.m1.1b"><mrow id="S6.T4.7.m1.1.1" xref="S6.T4.7.m1.1.1.cmml"><mi id="S6.T4.7.m1.1.1.2" xref="S6.T4.7.m1.1.1.2.cmml">r</mi><mo id="S6.T4.7.m1.1.1.1" xref="S6.T4.7.m1.1.1.1.cmml">=</mo><mn id="S6.T4.7.m1.1.1.3" xref="S6.T4.7.m1.1.1.3.cmml">8</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T4.7.m1.1c"><apply id="S6.T4.7.m1.1.1.cmml" xref="S6.T4.7.m1.1.1"><eq id="S6.T4.7.m1.1.1.1.cmml" xref="S6.T4.7.m1.1.1.1"></eq><ci id="S6.T4.7.m1.1.1.2.cmml" xref="S6.T4.7.m1.1.1.2">𝑟</ci><cn type="integer" id="S6.T4.7.m1.1.1.3.cmml" xref="S6.T4.7.m1.1.1.3">8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T4.7.m1.1d">r=8</annotation></semantics></math> and <math id="S6.T4.8.m2.1" class="ltx_Math" alttext="r=256" display="inline"><semantics id="S6.T4.8.m2.1b"><mrow id="S6.T4.8.m2.1.1" xref="S6.T4.8.m2.1.1.cmml"><mi id="S6.T4.8.m2.1.1.2" xref="S6.T4.8.m2.1.1.2.cmml">r</mi><mo id="S6.T4.8.m2.1.1.1" xref="S6.T4.8.m2.1.1.1.cmml">=</mo><mn id="S6.T4.8.m2.1.1.3" xref="S6.T4.8.m2.1.1.3.cmml">256</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T4.8.m2.1c"><apply id="S6.T4.8.m2.1.1.cmml" xref="S6.T4.8.m2.1.1"><eq id="S6.T4.8.m2.1.1.1.cmml" xref="S6.T4.8.m2.1.1.1"></eq><ci id="S6.T4.8.m2.1.1.2.cmml" xref="S6.T4.8.m2.1.1.2">𝑟</ci><cn type="integer" id="S6.T4.8.m2.1.1.3.cmml" xref="S6.T4.8.m2.1.1.3">256</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T4.8.m2.1d">r=256</annotation></semantics></math> on GSM8K.</figcaption>
</figure>
</section>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Conclusion</h2>

<div id="S7.p1" class="ltx_para">
<p id="S7.p1.1" class="ltx_p">In this paper, we analyze the impact of low-rank updating through LoRA and observe that such updating struggles for memory-intensive tasks, which also limits current LoRA variants. To overcome this limitation, we introduce MoRA, a method that utilizes non-parameterized operators for high-rank updating. Within the MoRA framework, we explore various methods to implement decompression and compression functions. Performance comparisons indicate that MoRA matches LoRA in instruction tuning and mathematical reasoning, and exhibits superior performance in continual pretraining and memory tasks. Additionally, we conduct pretraining experiments to further demonstrate the effectiveness of high-rank updating and show superior results compared to ReLoRA.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2023)</span>
<span class="ltx_bibblock">
Wei Chen, Qiushi Wang, Zefei Long, Xianyin Zhang, Zhongtian Lu, Bingxuan Li, Siyuan Wang, Jiarong Xu, Xiang Bai, Xuanjing Huang, et al. 2023.

</span>
<span class="ltx_bibblock">Disc-finllm: A chinese financial large language model based on multiple experts fine-tuning.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.15205</em>.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2022)</span>
<span class="ltx_bibblock">
Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022.

</span>
<span class="ltx_bibblock">Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2210.03849</em>.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cheng et al. (2023)</span>
<span class="ltx_bibblock">
Daixuan Cheng, Shaohan Huang, and Furu Wei. 2023.

</span>
<span class="ltx_bibblock">Adapting large language models via reading comprehension.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2309.09530</em>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cobbe et al. (2021)</span>
<span class="ltx_bibblock">
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021.

</span>
<span class="ltx_bibblock">Training verifiers to solve math word problems.

</span>
<span class="ltx_bibblock"><em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2110.14168</em>.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Collins et al. (2023)</span>
<span class="ltx_bibblock">
Katherine M Collins, Albert Q Jiang, Simon Frieder, Lionel Wong, Miri Zilka, Umang Bhatt, Thomas Lukasiewicz, Yuhuai Wu, Joshua B Tenenbaum, William Hart, et al. 2023.

</span>
<span class="ltx_bibblock">Evaluating language models for mathematics through interactions.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.01694</em>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dernoncourt and Lee (2017)</span>
<span class="ltx_bibblock">
Franck Dernoncourt and Ji Young Lee. 2017.

</span>
<span class="ltx_bibblock">Pubmed 200k rct: a dataset for sequential sentence classification in medical abstracts.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1710.06071</em>.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dettmers et al. (2024)</span>
<span class="ltx_bibblock">
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024.

</span>
<span class="ltx_bibblock">Qlora: Efficient finetuning of quantized llms.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fu et al. (2023)</span>
<span class="ltx_bibblock">
Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023.

</span>
<span class="ltx_bibblock">Specializing smaller language models towards multi-step reasoning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pages 10421–10430. PMLR.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. (2020)</span>
<span class="ltx_bibblock">
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020.

</span>
<span class="ltx_bibblock">The pile: An 800gb dataset of diverse text for language modeling.

</span>
<span class="ltx_bibblock"><em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2101.00027</em>.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Han et al. (2023)</span>
<span class="ltx_bibblock">
Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K Bressem. 2023.

</span>
<span class="ltx_bibblock">Medalpaca–an open-source collection of medical conversational ai models and training data.

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2304.08247</em>.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hayou et al. (2024)</span>
<span class="ltx_bibblock">
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://arxiv.org/abs/2402.12354" title="" class="ltx_ref ltx_href">LoRA+: Efficient Low Rank Adaptation of Large Models</a>.

</span>
<span class="ltx_bibblock">3.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. (2020)</span>
<span class="ltx_bibblock">
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.

</span>
<span class="ltx_bibblock">Measuring massive multitask language understanding.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2009.03300</em>.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. (2021)</span>
<span class="ltx_bibblock">
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021.

</span>
<span class="ltx_bibblock">Measuring mathematical problem solving with the math dataset.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2103.03874</em>.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Houlsby et al. (2019)</span>
<span class="ltx_bibblock">
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.

</span>
<span class="ltx_bibblock">Parameter-efficient transfer learning for nlp.

</span>
<span class="ltx_bibblock">In <em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">International conference on machine learning</em>, pages 2790–2799. PMLR.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hu et al. (2021)</span>
<span class="ltx_bibblock">
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.

</span>
<span class="ltx_bibblock">Lora: Low-rank adaptation of large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2106.09685</em>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Imani et al. (2023)</span>
<span class="ltx_bibblock">
Shima Imani, Liang Du, and Harsh Shrivastava. 2023.

</span>
<span class="ltx_bibblock">Mathprompter: Mathematical reasoning using large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2303.05398</em>.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ivison et al. (2023)</span>
<span class="ltx_bibblock">
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. 2023.

</span>
<span class="ltx_bibblock">Camels in a changing climate: Enhancing lm adaptation with tulu 2.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2311.10702</em>.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jin et al. (2021)</span>
<span class="ltx_bibblock">
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021.

</span>
<span class="ltx_bibblock">What disease does this patient have? a large-scale open domain question answering dataset from medical exams.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">Applied Sciences</em>, 11(14):6421.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jin et al. (2019)</span>
<span class="ltx_bibblock">
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019.

</span>
<span class="ltx_bibblock">Pubmedqa: A dataset for biomedical research question answering.

</span>
<span class="ltx_bibblock">In <em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)</em>, pages 2567–2577.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kopiczko et al. (2023)</span>
<span class="ltx_bibblock">
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. 2023.

</span>
<span class="ltx_bibblock">Vera: Vector-based random matrix adaptation.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.11454</em>.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lester et al. (2021)</span>
<span class="ltx_bibblock">
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.

</span>
<span class="ltx_bibblock">The power of scale for parameter-efficient prompt tuning.

</span>
<span class="ltx_bibblock"><em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2104.08691</em>.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lialin et al. (2023)</span>
<span class="ltx_bibblock">
Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. 2023.

</span>
<span class="ltx_bibblock">Stack more layers differently: High-rank training through low-rank updates.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2307.05695</em>.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2023)</span>
<span class="ltx_bibblock">
Mingjie Liu, Teodor-Dumitru Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, et al. 2023.

</span>
<span class="ltx_bibblock">Chipnemo: Domain-adapted llms for chip design.

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2311.00176</em>.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2024)</span>
<span class="ltx_bibblock">
Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024.

</span>
<span class="ltx_bibblock">Dora: Weight-decomposed low-rank adaptation.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.09353</em>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Maia et al. (2018)</span>
<span class="ltx_bibblock">
Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018.

</span>
<span class="ltx_bibblock">Www’18 open challenge: financial opinion mining and question answering.

</span>
<span class="ltx_bibblock">In <em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">Companion proceedings of the the web conference 2018</em>, pages 1941–1942.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Malo et al. (2014)</span>
<span class="ltx_bibblock">
Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala. 2014.

</span>
<span class="ltx_bibblock">Good debt or bad debt: Detecting semantic orientations in economic texts.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">Journal of the Association for Information Science and Technology</em>, 65(4):782–796.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Meng et al. (2024)</span>
<span class="ltx_bibblock">
Xiangdi Meng, Damai Dai, Weiyao Luo, Zhe Yang, Shaoxiang Wu, Xiaochen Wang, Peiyi Wang, Qingxiu Dong, Liang Chen, and Zhifang Sui. 2024.

</span>
<span class="ltx_bibblock">Periodiclora: Breaking the low-rank bottleneck in lora optimization.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.16141</em>.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Raffel et al. (2020)</span>
<span class="ltx_bibblock">
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.

</span>
<span class="ltx_bibblock">Exploring the limits of transfer learning with a unified text-to-text transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">Journal of machine learning research</em>, 21(140):1–67.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Renduchintala et al. (2023)</span>
<span class="ltx_bibblock">
Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev. 2023.

</span>
<span class="ltx_bibblock">Tied-lora: Enhacing parameter efficiency of lora with weight tying.

</span>
<span class="ltx_bibblock"><em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2311.09578</em>.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Salinas Alvarado et al. (2015)</span>
<span class="ltx_bibblock">
Julio Cesar Salinas Alvarado, Karin Verspoor, and Timothy Baldwin. 2015.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://aclanthology.org/U15-1010" title="" class="ltx_ref ltx_href">Domain adaption of named entity recognition to support credit risk assessment</a>.

</span>
<span class="ltx_bibblock">In <em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Australasian Language Technology Association Workshop 2015</em>, pages 84–90, Parramatta, Australia.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shazeer (2020)</span>
<span class="ltx_bibblock">
Noam Shazeer. 2020.

</span>
<span class="ltx_bibblock">Glu variants improve transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2002.05202</em>.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shi et al. (2024)</span>
<span class="ltx_bibblock">
Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang. 2024.

</span>
<span class="ltx_bibblock">Reslora: Identity residual mapping in low-rank adaption.

</span>
<span class="ltx_bibblock"><em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.18039</em>.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sinha and Khandait (2021)</span>
<span class="ltx_bibblock">
Ankur Sinha and Tanmay Khandait. 2021.

</span>
<span class="ltx_bibblock">Impact of news on the commodity market: Dataset and results.

</span>
<span class="ltx_bibblock">In <em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">Advances in Information and Communication: Proceedings of the 2021 Future of Information and Communication Conference (FICC), Volume 2</em>, pages 589–601. Springer.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Su et al. (2024)</span>
<span class="ltx_bibblock">
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024.

</span>
<span class="ltx_bibblock">Roformer: Enhanced transformer with rotary position embedding.

</span>
<span class="ltx_bibblock"><em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">Neurocomputing</em>, 568:127063.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2018)</span>
<span class="ltx_bibblock">
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018.

</span>
<span class="ltx_bibblock">Glue: A multi-task benchmark and analysis platform for natural language understanding.

</span>
<span class="ltx_bibblock"><em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1804.07461</em>.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2024)</span>
<span class="ltx_bibblock">
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. 2024.

</span>
<span class="ltx_bibblock">How far can camels go? exploring the state of instruction tuning on open resources.

</span>
<span class="ltx_bibblock"><em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wu et al. (2023)</span>
<span class="ltx_bibblock">
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023.

</span>
<span class="ltx_bibblock">Bloomberggpt: A large language model for finance.

</span>
<span class="ltx_bibblock"><em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2303.17564</em>.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yu et al. (2023)</span>
<span class="ltx_bibblock">
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023.

</span>
<span class="ltx_bibblock">Metamath: Bootstrap your own mathematical questions for large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2309.12284</em>.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang and Sennrich (2019)</span>
<span class="ltx_bibblock">
Biao Zhang and Rico Sennrich. 2019.

</span>
<span class="ltx_bibblock">Root mean square layer normalization.

</span>
<span class="ltx_bibblock"><em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 32.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. (2024)</span>
<span class="ltx_bibblock">
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2024.

</span>
<span class="ltx_bibblock">Lima: Less is more for alignment.

</span>
<span class="ltx_bibblock"><em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhu et al. (2024)</span>
<span class="ltx_bibblock">
Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez de Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. 2024.

</span>
<span class="ltx_bibblock">Asymmetry in low-rank adapters of foundation models.

</span>
<span class="ltx_bibblock"><em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.16842</em>.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Hyperparameters</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.1" class="ltx_p">We propose hyperparameters in Table <a href="#A1.T5" title="Table 5 ‣ Appendix A Hyperparameters ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>.</p>
</div>
<figure id="A1.T5" class="ltx_table">
<table id="A1.T5.40" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T5.4.4" class="ltx_tr">
<th id="A1.T5.4.4.5" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">Dataset</th>
<th id="A1.T5.4.4.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Method</th>
<th id="A1.T5.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><math id="A1.T5.1.1.1.m1.1" class="ltx_Math" alttext="r" display="inline"><semantics id="A1.T5.1.1.1.m1.1a"><mi id="A1.T5.1.1.1.m1.1.1" xref="A1.T5.1.1.1.m1.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="A1.T5.1.1.1.m1.1b"><ci id="A1.T5.1.1.1.m1.1.1.cmml" xref="A1.T5.1.1.1.m1.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.1.1.1.m1.1c">r</annotation></semantics></math></th>
<th id="A1.T5.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><math id="A1.T5.2.2.2.m1.1" class="ltx_Math" alttext="\alpha" display="inline"><semantics id="A1.T5.2.2.2.m1.1a"><mi id="A1.T5.2.2.2.m1.1.1" xref="A1.T5.2.2.2.m1.1.1.cmml">α</mi><annotation-xml encoding="MathML-Content" id="A1.T5.2.2.2.m1.1b"><ci id="A1.T5.2.2.2.m1.1.1.cmml" xref="A1.T5.2.2.2.m1.1.1">𝛼</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.2.2.2.m1.1c">\alpha</annotation></semantics></math></th>
<th id="A1.T5.4.4.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">LR</th>
<th id="A1.T5.4.4.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">LR Scheduler</th>
<th id="A1.T5.4.4.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Warmup</th>
<th id="A1.T5.4.4.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Epochs</th>
<th id="A1.T5.4.4.11" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Batch size</th>
<th id="A1.T5.4.4.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">
<math id="A1.T5.3.3.3.m1.1" class="ltx_Math" alttext="f_{\text{comp}}" display="inline"><semantics id="A1.T5.3.3.3.m1.1a"><msub id="A1.T5.3.3.3.m1.1.1" xref="A1.T5.3.3.3.m1.1.1.cmml"><mi id="A1.T5.3.3.3.m1.1.1.2" xref="A1.T5.3.3.3.m1.1.1.2.cmml">f</mi><mtext id="A1.T5.3.3.3.m1.1.1.3" xref="A1.T5.3.3.3.m1.1.1.3a.cmml">comp</mtext></msub><annotation-xml encoding="MathML-Content" id="A1.T5.3.3.3.m1.1b"><apply id="A1.T5.3.3.3.m1.1.1.cmml" xref="A1.T5.3.3.3.m1.1.1"><csymbol cd="ambiguous" id="A1.T5.3.3.3.m1.1.1.1.cmml" xref="A1.T5.3.3.3.m1.1.1">subscript</csymbol><ci id="A1.T5.3.3.3.m1.1.1.2.cmml" xref="A1.T5.3.3.3.m1.1.1.2">𝑓</ci><ci id="A1.T5.3.3.3.m1.1.1.3a.cmml" xref="A1.T5.3.3.3.m1.1.1.3"><mtext mathsize="70%" id="A1.T5.3.3.3.m1.1.1.3.cmml" xref="A1.T5.3.3.3.m1.1.1.3">comp</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.3.3.3.m1.1c">f_{\text{comp}}</annotation></semantics></math>, <math id="A1.T5.4.4.4.m2.1" class="ltx_Math" alttext="f_{\text{decomp}}" display="inline"><semantics id="A1.T5.4.4.4.m2.1a"><msub id="A1.T5.4.4.4.m2.1.1" xref="A1.T5.4.4.4.m2.1.1.cmml"><mi id="A1.T5.4.4.4.m2.1.1.2" xref="A1.T5.4.4.4.m2.1.1.2.cmml">f</mi><mtext id="A1.T5.4.4.4.m2.1.1.3" xref="A1.T5.4.4.4.m2.1.1.3a.cmml">decomp</mtext></msub><annotation-xml encoding="MathML-Content" id="A1.T5.4.4.4.m2.1b"><apply id="A1.T5.4.4.4.m2.1.1.cmml" xref="A1.T5.4.4.4.m2.1.1"><csymbol cd="ambiguous" id="A1.T5.4.4.4.m2.1.1.1.cmml" xref="A1.T5.4.4.4.m2.1.1">subscript</csymbol><ci id="A1.T5.4.4.4.m2.1.1.2.cmml" xref="A1.T5.4.4.4.m2.1.1.2">𝑓</ci><ci id="A1.T5.4.4.4.m2.1.1.3a.cmml" xref="A1.T5.4.4.4.m2.1.1.3"><mtext mathsize="70%" id="A1.T5.4.4.4.m2.1.1.3.cmml" xref="A1.T5.4.4.4.m2.1.1.3">decomp</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.4.4.4.m2.1c">f_{\text{decomp}}</annotation></semantics></math>
</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T5.40.41.1" class="ltx_tr">
<td id="A1.T5.40.41.1.1" class="ltx_td ltx_align_left ltx_border_t" rowspan="5"><span id="A1.T5.40.41.1.1.1" class="ltx_text">Tülu v2</span></td>
<td id="A1.T5.40.41.1.2" class="ltx_td ltx_align_center ltx_border_t">FFT</td>
<td id="A1.T5.40.41.1.3" class="ltx_td ltx_align_center ltx_border_t">-</td>
<td id="A1.T5.40.41.1.4" class="ltx_td ltx_align_center ltx_border_t">-</td>
<td id="A1.T5.40.41.1.5" class="ltx_td ltx_align_center ltx_border_t">2e-5</td>
<td id="A1.T5.40.41.1.6" class="ltx_td ltx_align_center ltx_border_t">cosine</td>
<td id="A1.T5.40.41.1.7" class="ltx_td ltx_align_center ltx_border_t">500</td>
<td id="A1.T5.40.41.1.8" class="ltx_td ltx_align_center ltx_border_t">2</td>
<td id="A1.T5.40.41.1.9" class="ltx_td ltx_align_center ltx_border_t">128</td>
<td id="A1.T5.40.41.1.10" class="ltx_td ltx_align_center ltx_border_t">-</td>
</tr>
<tr id="A1.T5.7.7" class="ltx_tr">
<td id="A1.T5.7.7.4" class="ltx_td ltx_align_center">LoRA-like</td>
<td id="A1.T5.7.7.5" class="ltx_td ltx_align_center">8</td>
<td id="A1.T5.7.7.6" class="ltx_td ltx_align_center">16</td>
<td id="A1.T5.7.7.3" class="ltx_td ltx_align_center">
<math id="A1.T5.5.5.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="A1.T5.5.5.1.m1.1a"><mo stretchy="false" id="A1.T5.5.5.1.m1.1.1" xref="A1.T5.5.5.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="A1.T5.5.5.1.m1.1b"><ci id="A1.T5.5.5.1.m1.1.1.cmml" xref="A1.T5.5.5.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.5.5.1.m1.1c">\{</annotation></semantics></math>1e-4<span id="A1.T5.7.7.3.1" class="ltx_text ltx_markedasmath">,</span>2e-4<math id="A1.T5.7.7.3.m3.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="A1.T5.7.7.3.m3.1a"><mo stretchy="false" id="A1.T5.7.7.3.m3.1.1" xref="A1.T5.7.7.3.m3.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="A1.T5.7.7.3.m3.1b"><ci id="A1.T5.7.7.3.m3.1.1.cmml" xref="A1.T5.7.7.3.m3.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.7.7.3.m3.1c">\}</annotation></semantics></math>
</td>
<td id="A1.T5.7.7.7" class="ltx_td ltx_align_center">cosine</td>
<td id="A1.T5.7.7.8" class="ltx_td ltx_align_center">500</td>
<td id="A1.T5.7.7.9" class="ltx_td ltx_align_center">2</td>
<td id="A1.T5.7.7.10" class="ltx_td ltx_align_center">128</td>
<td id="A1.T5.7.7.11" class="ltx_td ltx_align_center">-</td>
</tr>
<tr id="A1.T5.10.10" class="ltx_tr">
<td id="A1.T5.10.10.4" class="ltx_td ltx_align_center">MoRA</td>
<td id="A1.T5.10.10.5" class="ltx_td ltx_align_center">8</td>
<td id="A1.T5.10.10.6" class="ltx_td ltx_align_center">-</td>
<td id="A1.T5.10.10.3" class="ltx_td ltx_align_center">
<math id="A1.T5.8.8.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="A1.T5.8.8.1.m1.1a"><mo stretchy="false" id="A1.T5.8.8.1.m1.1.1" xref="A1.T5.8.8.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="A1.T5.8.8.1.m1.1b"><ci id="A1.T5.8.8.1.m1.1.1.cmml" xref="A1.T5.8.8.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.8.8.1.m1.1c">\{</annotation></semantics></math>2e-4<span id="A1.T5.10.10.3.1" class="ltx_text ltx_markedasmath">,</span>3e-4<math id="A1.T5.10.10.3.m3.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="A1.T5.10.10.3.m3.1a"><mo stretchy="false" id="A1.T5.10.10.3.m3.1.1" xref="A1.T5.10.10.3.m3.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="A1.T5.10.10.3.m3.1b"><ci id="A1.T5.10.10.3.m3.1.1.cmml" xref="A1.T5.10.10.3.m3.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.10.10.3.m3.1c">\}</annotation></semantics></math>
</td>
<td id="A1.T5.10.10.7" class="ltx_td ltx_align_center">cosine</td>
<td id="A1.T5.10.10.8" class="ltx_td ltx_align_center">500</td>
<td id="A1.T5.10.10.9" class="ltx_td ltx_align_center">2</td>
<td id="A1.T5.10.10.10" class="ltx_td ltx_align_center">128</td>
<td id="A1.T5.10.10.11" class="ltx_td ltx_align_center">Eq. <a href="#S4.E9" title="In 4 Method ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>
</td>
</tr>
<tr id="A1.T5.13.13" class="ltx_tr">
<td id="A1.T5.13.13.4" class="ltx_td ltx_align_center">LoRA-like</td>
<td id="A1.T5.13.13.5" class="ltx_td ltx_align_center">256</td>
<td id="A1.T5.13.13.6" class="ltx_td ltx_align_center">128</td>
<td id="A1.T5.13.13.3" class="ltx_td ltx_align_center">
<math id="A1.T5.11.11.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="A1.T5.11.11.1.m1.1a"><mo stretchy="false" id="A1.T5.11.11.1.m1.1.1" xref="A1.T5.11.11.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="A1.T5.11.11.1.m1.1b"><ci id="A1.T5.11.11.1.m1.1.1.cmml" xref="A1.T5.11.11.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.11.11.1.m1.1c">\{</annotation></semantics></math>1e-4<span id="A1.T5.13.13.3.1" class="ltx_text ltx_markedasmath">,</span>2e-4<math id="A1.T5.13.13.3.m3.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="A1.T5.13.13.3.m3.1a"><mo stretchy="false" id="A1.T5.13.13.3.m3.1.1" xref="A1.T5.13.13.3.m3.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="A1.T5.13.13.3.m3.1b"><ci id="A1.T5.13.13.3.m3.1.1.cmml" xref="A1.T5.13.13.3.m3.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.13.13.3.m3.1c">\}</annotation></semantics></math>
</td>
<td id="A1.T5.13.13.7" class="ltx_td ltx_align_center">cosine</td>
<td id="A1.T5.13.13.8" class="ltx_td ltx_align_center">500</td>
<td id="A1.T5.13.13.9" class="ltx_td ltx_align_center">2</td>
<td id="A1.T5.13.13.10" class="ltx_td ltx_align_center">128</td>
<td id="A1.T5.13.13.11" class="ltx_td ltx_align_center">-</td>
</tr>
<tr id="A1.T5.16.16" class="ltx_tr">
<td id="A1.T5.16.16.4" class="ltx_td ltx_align_center">MoRA</td>
<td id="A1.T5.16.16.5" class="ltx_td ltx_align_center">256</td>
<td id="A1.T5.16.16.6" class="ltx_td ltx_align_center">-</td>
<td id="A1.T5.16.16.3" class="ltx_td ltx_align_center">
<math id="A1.T5.14.14.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="A1.T5.14.14.1.m1.1a"><mo stretchy="false" id="A1.T5.14.14.1.m1.1.1" xref="A1.T5.14.14.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="A1.T5.14.14.1.m1.1b"><ci id="A1.T5.14.14.1.m1.1.1.cmml" xref="A1.T5.14.14.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.14.14.1.m1.1c">\{</annotation></semantics></math>3e-5<span id="A1.T5.16.16.3.1" class="ltx_text ltx_markedasmath">,</span>5e-5<math id="A1.T5.16.16.3.m3.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="A1.T5.16.16.3.m3.1a"><mo stretchy="false" id="A1.T5.16.16.3.m3.1.1" xref="A1.T5.16.16.3.m3.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="A1.T5.16.16.3.m3.1b"><ci id="A1.T5.16.16.3.m3.1.1.cmml" xref="A1.T5.16.16.3.m3.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.16.16.3.m3.1c">\}</annotation></semantics></math>
</td>
<td id="A1.T5.16.16.7" class="ltx_td ltx_align_center">cosine</td>
<td id="A1.T5.16.16.8" class="ltx_td ltx_align_center">500</td>
<td id="A1.T5.16.16.9" class="ltx_td ltx_align_center">2</td>
<td id="A1.T5.16.16.10" class="ltx_td ltx_align_center">128</td>
<td id="A1.T5.16.16.11" class="ltx_td ltx_align_center">Eq. <a href="#S4.E6" title="In 4 Method ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>
</td>
</tr>
<tr id="A1.T5.40.42.2" class="ltx_tr">
<td id="A1.T5.40.42.2.1" class="ltx_td ltx_align_left ltx_border_t" rowspan="5"><span id="A1.T5.40.42.2.1.1" class="ltx_text">MetaMath</span></td>
<td id="A1.T5.40.42.2.2" class="ltx_td ltx_align_center ltx_border_t">FFT</td>
<td id="A1.T5.40.42.2.3" class="ltx_td ltx_align_center ltx_border_t">-</td>
<td id="A1.T5.40.42.2.4" class="ltx_td ltx_align_center ltx_border_t">-</td>
<td id="A1.T5.40.42.2.5" class="ltx_td ltx_align_center ltx_border_t">2e-5</td>
<td id="A1.T5.40.42.2.6" class="ltx_td ltx_align_center ltx_border_t">cosine</td>
<td id="A1.T5.40.42.2.7" class="ltx_td ltx_align_center ltx_border_t">300</td>
<td id="A1.T5.40.42.2.8" class="ltx_td ltx_align_center ltx_border_t">3</td>
<td id="A1.T5.40.42.2.9" class="ltx_td ltx_align_center ltx_border_t">128</td>
<td id="A1.T5.40.42.2.10" class="ltx_td ltx_align_center ltx_border_t">-</td>
</tr>
<tr id="A1.T5.19.19" class="ltx_tr">
<td id="A1.T5.19.19.4" class="ltx_td ltx_align_center">LoRA-like</td>
<td id="A1.T5.19.19.5" class="ltx_td ltx_align_center">8</td>
<td id="A1.T5.19.19.6" class="ltx_td ltx_align_center">16</td>
<td id="A1.T5.19.19.3" class="ltx_td ltx_align_center">
<math id="A1.T5.17.17.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="A1.T5.17.17.1.m1.1a"><mo stretchy="false" id="A1.T5.17.17.1.m1.1.1" xref="A1.T5.17.17.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="A1.T5.17.17.1.m1.1b"><ci id="A1.T5.17.17.1.m1.1.1.cmml" xref="A1.T5.17.17.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.17.17.1.m1.1c">\{</annotation></semantics></math>1e-4<span id="A1.T5.19.19.3.1" class="ltx_text ltx_markedasmath">,</span>2e-4<math id="A1.T5.19.19.3.m3.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="A1.T5.19.19.3.m3.1a"><mo stretchy="false" id="A1.T5.19.19.3.m3.1.1" xref="A1.T5.19.19.3.m3.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="A1.T5.19.19.3.m3.1b"><ci id="A1.T5.19.19.3.m3.1.1.cmml" xref="A1.T5.19.19.3.m3.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.19.19.3.m3.1c">\}</annotation></semantics></math>
</td>
<td id="A1.T5.19.19.7" class="ltx_td ltx_align_center">cosine</td>
<td id="A1.T5.19.19.8" class="ltx_td ltx_align_center">300</td>
<td id="A1.T5.19.19.9" class="ltx_td ltx_align_center">3</td>
<td id="A1.T5.19.19.10" class="ltx_td ltx_align_center">128</td>
<td id="A1.T5.19.19.11" class="ltx_td ltx_align_center">-</td>
</tr>
<tr id="A1.T5.22.22" class="ltx_tr">
<td id="A1.T5.22.22.4" class="ltx_td ltx_align_center">MoRA</td>
<td id="A1.T5.22.22.5" class="ltx_td ltx_align_center">8</td>
<td id="A1.T5.22.22.6" class="ltx_td ltx_align_center">-</td>
<td id="A1.T5.22.22.3" class="ltx_td ltx_align_center">
<math id="A1.T5.20.20.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="A1.T5.20.20.1.m1.1a"><mo stretchy="false" id="A1.T5.20.20.1.m1.1.1" xref="A1.T5.20.20.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="A1.T5.20.20.1.m1.1b"><ci id="A1.T5.20.20.1.m1.1.1.cmml" xref="A1.T5.20.20.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.20.20.1.m1.1c">\{</annotation></semantics></math>2e-4<span id="A1.T5.22.22.3.1" class="ltx_text ltx_markedasmath">,</span>3e-4<math id="A1.T5.22.22.3.m3.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="A1.T5.22.22.3.m3.1a"><mo stretchy="false" id="A1.T5.22.22.3.m3.1.1" xref="A1.T5.22.22.3.m3.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="A1.T5.22.22.3.m3.1b"><ci id="A1.T5.22.22.3.m3.1.1.cmml" xref="A1.T5.22.22.3.m3.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.22.22.3.m3.1c">\}</annotation></semantics></math>
</td>
<td id="A1.T5.22.22.7" class="ltx_td ltx_align_center">cosine</td>
<td id="A1.T5.22.22.8" class="ltx_td ltx_align_center">300</td>
<td id="A1.T5.22.22.9" class="ltx_td ltx_align_center">3</td>
<td id="A1.T5.22.22.10" class="ltx_td ltx_align_center">128</td>
<td id="A1.T5.22.22.11" class="ltx_td ltx_align_center">Eq. <a href="#S4.E9" title="In 4 Method ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>
</td>
</tr>
<tr id="A1.T5.25.25" class="ltx_tr">
<td id="A1.T5.25.25.4" class="ltx_td ltx_align_center">LoRA-like</td>
<td id="A1.T5.25.25.5" class="ltx_td ltx_align_center">256</td>
<td id="A1.T5.25.25.6" class="ltx_td ltx_align_center">128</td>
<td id="A1.T5.25.25.3" class="ltx_td ltx_align_center">
<math id="A1.T5.23.23.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="A1.T5.23.23.1.m1.1a"><mo stretchy="false" id="A1.T5.23.23.1.m1.1.1" xref="A1.T5.23.23.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="A1.T5.23.23.1.m1.1b"><ci id="A1.T5.23.23.1.m1.1.1.cmml" xref="A1.T5.23.23.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.23.23.1.m1.1c">\{</annotation></semantics></math>1e-4<span id="A1.T5.25.25.3.1" class="ltx_text ltx_markedasmath">,</span>2e-4<math id="A1.T5.25.25.3.m3.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="A1.T5.25.25.3.m3.1a"><mo stretchy="false" id="A1.T5.25.25.3.m3.1.1" xref="A1.T5.25.25.3.m3.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="A1.T5.25.25.3.m3.1b"><ci id="A1.T5.25.25.3.m3.1.1.cmml" xref="A1.T5.25.25.3.m3.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.25.25.3.m3.1c">\}</annotation></semantics></math>
</td>
<td id="A1.T5.25.25.7" class="ltx_td ltx_align_center">cosine</td>
<td id="A1.T5.25.25.8" class="ltx_td ltx_align_center">300</td>
<td id="A1.T5.25.25.9" class="ltx_td ltx_align_center">3</td>
<td id="A1.T5.25.25.10" class="ltx_td ltx_align_center">128</td>
<td id="A1.T5.25.25.11" class="ltx_td ltx_align_center">-</td>
</tr>
<tr id="A1.T5.28.28" class="ltx_tr">
<td id="A1.T5.28.28.4" class="ltx_td ltx_align_center">MoRA</td>
<td id="A1.T5.28.28.5" class="ltx_td ltx_align_center">256</td>
<td id="A1.T5.28.28.6" class="ltx_td ltx_align_center">-</td>
<td id="A1.T5.28.28.3" class="ltx_td ltx_align_center">
<math id="A1.T5.26.26.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="A1.T5.26.26.1.m1.1a"><mo stretchy="false" id="A1.T5.26.26.1.m1.1.1" xref="A1.T5.26.26.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="A1.T5.26.26.1.m1.1b"><ci id="A1.T5.26.26.1.m1.1.1.cmml" xref="A1.T5.26.26.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.26.26.1.m1.1c">\{</annotation></semantics></math>3e-5<span id="A1.T5.28.28.3.1" class="ltx_text ltx_markedasmath">,</span>5e-5<math id="A1.T5.28.28.3.m3.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="A1.T5.28.28.3.m3.1a"><mo stretchy="false" id="A1.T5.28.28.3.m3.1.1" xref="A1.T5.28.28.3.m3.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="A1.T5.28.28.3.m3.1b"><ci id="A1.T5.28.28.3.m3.1.1.cmml" xref="A1.T5.28.28.3.m3.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.28.28.3.m3.1c">\}</annotation></semantics></math>
</td>
<td id="A1.T5.28.28.7" class="ltx_td ltx_align_center">cosine</td>
<td id="A1.T5.28.28.8" class="ltx_td ltx_align_center">300</td>
<td id="A1.T5.28.28.9" class="ltx_td ltx_align_center">3</td>
<td id="A1.T5.28.28.10" class="ltx_td ltx_align_center">128</td>
<td id="A1.T5.28.28.11" class="ltx_td ltx_align_center">Eq. <a href="#S4.E6" title="In 4 Method ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>
</td>
</tr>
<tr id="A1.T5.40.43.3" class="ltx_tr">
<td id="A1.T5.40.43.3.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" rowspan="5"><span id="A1.T5.40.43.3.1.1" class="ltx_text">BioMed./Fiance</span></td>
<td id="A1.T5.40.43.3.2" class="ltx_td ltx_align_center ltx_border_t">FFT</td>
<td id="A1.T5.40.43.3.3" class="ltx_td ltx_align_center ltx_border_t">-</td>
<td id="A1.T5.40.43.3.4" class="ltx_td ltx_align_center ltx_border_t">-</td>
<td id="A1.T5.40.43.3.5" class="ltx_td ltx_align_center ltx_border_t">3e-5</td>
<td id="A1.T5.40.43.3.6" class="ltx_td ltx_align_center ltx_border_t">linear</td>
<td id="A1.T5.40.43.3.7" class="ltx_td ltx_align_center ltx_border_t">150</td>
<td id="A1.T5.40.43.3.8" class="ltx_td ltx_align_center ltx_border_t">3</td>
<td id="A1.T5.40.43.3.9" class="ltx_td ltx_align_center ltx_border_t">128</td>
<td id="A1.T5.40.43.3.10" class="ltx_td ltx_align_center ltx_border_t">-</td>
</tr>
<tr id="A1.T5.31.31" class="ltx_tr">
<td id="A1.T5.31.31.4" class="ltx_td ltx_align_center">LoRA-like</td>
<td id="A1.T5.31.31.5" class="ltx_td ltx_align_center">8</td>
<td id="A1.T5.31.31.6" class="ltx_td ltx_align_center">16</td>
<td id="A1.T5.31.31.3" class="ltx_td ltx_align_center">
<math id="A1.T5.29.29.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="A1.T5.29.29.1.m1.1a"><mo stretchy="false" id="A1.T5.29.29.1.m1.1.1" xref="A1.T5.29.29.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="A1.T5.29.29.1.m1.1b"><ci id="A1.T5.29.29.1.m1.1.1.cmml" xref="A1.T5.29.29.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.29.29.1.m1.1c">\{</annotation></semantics></math>3e-4<span id="A1.T5.31.31.3.1" class="ltx_text ltx_markedasmath">,</span>4e-4<math id="A1.T5.31.31.3.m3.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="A1.T5.31.31.3.m3.1a"><mo stretchy="false" id="A1.T5.31.31.3.m3.1.1" xref="A1.T5.31.31.3.m3.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="A1.T5.31.31.3.m3.1b"><ci id="A1.T5.31.31.3.m3.1.1.cmml" xref="A1.T5.31.31.3.m3.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.31.31.3.m3.1c">\}</annotation></semantics></math>
</td>
<td id="A1.T5.31.31.7" class="ltx_td ltx_align_center">linear</td>
<td id="A1.T5.31.31.8" class="ltx_td ltx_align_center">150</td>
<td id="A1.T5.31.31.9" class="ltx_td ltx_align_center">3</td>
<td id="A1.T5.31.31.10" class="ltx_td ltx_align_center">128</td>
<td id="A1.T5.31.31.11" class="ltx_td ltx_align_center">-</td>
</tr>
<tr id="A1.T5.34.34" class="ltx_tr">
<td id="A1.T5.34.34.4" class="ltx_td ltx_align_center">MoRA</td>
<td id="A1.T5.34.34.5" class="ltx_td ltx_align_center">8</td>
<td id="A1.T5.34.34.6" class="ltx_td ltx_align_center">-</td>
<td id="A1.T5.34.34.3" class="ltx_td ltx_align_center">
<math id="A1.T5.32.32.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="A1.T5.32.32.1.m1.1a"><mo stretchy="false" id="A1.T5.32.32.1.m1.1.1" xref="A1.T5.32.32.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="A1.T5.32.32.1.m1.1b"><ci id="A1.T5.32.32.1.m1.1.1.cmml" xref="A1.T5.32.32.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.32.32.1.m1.1c">\{</annotation></semantics></math>4e-4<span id="A1.T5.34.34.3.1" class="ltx_text ltx_markedasmath">,</span>5e-4<math id="A1.T5.34.34.3.m3.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="A1.T5.34.34.3.m3.1a"><mo stretchy="false" id="A1.T5.34.34.3.m3.1.1" xref="A1.T5.34.34.3.m3.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="A1.T5.34.34.3.m3.1b"><ci id="A1.T5.34.34.3.m3.1.1.cmml" xref="A1.T5.34.34.3.m3.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.34.34.3.m3.1c">\}</annotation></semantics></math>
</td>
<td id="A1.T5.34.34.7" class="ltx_td ltx_align_center">linear</td>
<td id="A1.T5.34.34.8" class="ltx_td ltx_align_center">150</td>
<td id="A1.T5.34.34.9" class="ltx_td ltx_align_center">3</td>
<td id="A1.T5.34.34.10" class="ltx_td ltx_align_center">128</td>
<td id="A1.T5.34.34.11" class="ltx_td ltx_align_center">Eq. <a href="#S4.E9" title="In 4 Method ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>
</td>
</tr>
<tr id="A1.T5.37.37" class="ltx_tr">
<td id="A1.T5.37.37.4" class="ltx_td ltx_align_center">LoRA-like</td>
<td id="A1.T5.37.37.5" class="ltx_td ltx_align_center">256</td>
<td id="A1.T5.37.37.6" class="ltx_td ltx_align_center">128</td>
<td id="A1.T5.37.37.3" class="ltx_td ltx_align_center">
<math id="A1.T5.35.35.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="A1.T5.35.35.1.m1.1a"><mo stretchy="false" id="A1.T5.35.35.1.m1.1.1" xref="A1.T5.35.35.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="A1.T5.35.35.1.m1.1b"><ci id="A1.T5.35.35.1.m1.1.1.cmml" xref="A1.T5.35.35.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.35.35.1.m1.1c">\{</annotation></semantics></math>3e-4<span id="A1.T5.37.37.3.1" class="ltx_text ltx_markedasmath">,</span>4e-4<math id="A1.T5.37.37.3.m3.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="A1.T5.37.37.3.m3.1a"><mo stretchy="false" id="A1.T5.37.37.3.m3.1.1" xref="A1.T5.37.37.3.m3.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="A1.T5.37.37.3.m3.1b"><ci id="A1.T5.37.37.3.m3.1.1.cmml" xref="A1.T5.37.37.3.m3.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.37.37.3.m3.1c">\}</annotation></semantics></math>
</td>
<td id="A1.T5.37.37.7" class="ltx_td ltx_align_center">linear</td>
<td id="A1.T5.37.37.8" class="ltx_td ltx_align_center">150</td>
<td id="A1.T5.37.37.9" class="ltx_td ltx_align_center">3</td>
<td id="A1.T5.37.37.10" class="ltx_td ltx_align_center">128</td>
<td id="A1.T5.37.37.11" class="ltx_td ltx_align_center">-</td>
</tr>
<tr id="A1.T5.40.40" class="ltx_tr">
<td id="A1.T5.40.40.4" class="ltx_td ltx_align_center ltx_border_bb">MoRA</td>
<td id="A1.T5.40.40.5" class="ltx_td ltx_align_center ltx_border_bb">256</td>
<td id="A1.T5.40.40.6" class="ltx_td ltx_align_center ltx_border_bb">-</td>
<td id="A1.T5.40.40.3" class="ltx_td ltx_align_center ltx_border_bb">
<math id="A1.T5.38.38.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="A1.T5.38.38.1.m1.1a"><mo stretchy="false" id="A1.T5.38.38.1.m1.1.1" xref="A1.T5.38.38.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="A1.T5.38.38.1.m1.1b"><ci id="A1.T5.38.38.1.m1.1.1.cmml" xref="A1.T5.38.38.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.38.38.1.m1.1c">\{</annotation></semantics></math>5e-5<span id="A1.T5.40.40.3.1" class="ltx_text ltx_markedasmath">,</span>7e-5<math id="A1.T5.40.40.3.m3.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="A1.T5.40.40.3.m3.1a"><mo stretchy="false" id="A1.T5.40.40.3.m3.1.1" xref="A1.T5.40.40.3.m3.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="A1.T5.40.40.3.m3.1b"><ci id="A1.T5.40.40.3.m3.1.1.cmml" xref="A1.T5.40.40.3.m3.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T5.40.40.3.m3.1c">\}</annotation></semantics></math>
</td>
<td id="A1.T5.40.40.7" class="ltx_td ltx_align_center ltx_border_bb">linear</td>
<td id="A1.T5.40.40.8" class="ltx_td ltx_align_center ltx_border_bb">150</td>
<td id="A1.T5.40.40.9" class="ltx_td ltx_align_center ltx_border_bb">3</td>
<td id="A1.T5.40.40.10" class="ltx_td ltx_align_center ltx_border_bb">128</td>
<td id="A1.T5.40.40.11" class="ltx_td ltx_align_center ltx_border_bb">Eq. <a href="#S4.E6" title="In 4 Method ‣ MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>Hyperparameters for fine-tuning on three datasets.</figcaption>
</figure>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Implementation of ReMoRA</h2>

<div id="A2.p1" class="ltx_para">
<p id="A2.p1.9" class="ltx_p">We introduce detial implementation of ReMoRA in pretraining.
In this case, we simply define two kinds of <math id="A2.p1.1.m1.1" class="ltx_Math" alttext="g" display="inline"><semantics id="A2.p1.1.m1.1a"><mi id="A2.p1.1.m1.1.1" xref="A2.p1.1.m1.1.1.cmml">g</mi><annotation-xml encoding="MathML-Content" id="A2.p1.1.m1.1b"><ci id="A2.p1.1.m1.1.1.cmml" xref="A2.p1.1.m1.1.1">𝑔</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.1.m1.1c">g</annotation></semantics></math>.
The first kind is grouping every adjacent <math id="A2.p1.2.m2.1" class="ltx_Math" alttext="\hat{r}" display="inline"><semantics id="A2.p1.2.m2.1a"><mover accent="true" id="A2.p1.2.m2.1.1" xref="A2.p1.2.m2.1.1.cmml"><mi id="A2.p1.2.m2.1.1.2" xref="A2.p1.2.m2.1.1.2.cmml">r</mi><mo id="A2.p1.2.m2.1.1.1" xref="A2.p1.2.m2.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="A2.p1.2.m2.1b"><apply id="A2.p1.2.m2.1.1.cmml" xref="A2.p1.2.m2.1.1"><ci id="A2.p1.2.m2.1.1.1.cmml" xref="A2.p1.2.m2.1.1.1">^</ci><ci id="A2.p1.2.m2.1.1.2.cmml" xref="A2.p1.2.m2.1.1.2">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.2.m2.1c">\hat{r}</annotation></semantics></math> rows or columns together following the defined in fine-tuning, the first groups can be represented as <math id="A2.p1.3.m3.4" class="ltx_Math" alttext="\{1,2,\ldots,\hat{r}\}" display="inline"><semantics id="A2.p1.3.m3.4a"><mrow id="A2.p1.3.m3.4.5.2" xref="A2.p1.3.m3.4.5.1.cmml"><mo stretchy="false" id="A2.p1.3.m3.4.5.2.1" xref="A2.p1.3.m3.4.5.1.cmml">{</mo><mn id="A2.p1.3.m3.1.1" xref="A2.p1.3.m3.1.1.cmml">1</mn><mo id="A2.p1.3.m3.4.5.2.2" xref="A2.p1.3.m3.4.5.1.cmml">,</mo><mn id="A2.p1.3.m3.2.2" xref="A2.p1.3.m3.2.2.cmml">2</mn><mo id="A2.p1.3.m3.4.5.2.3" xref="A2.p1.3.m3.4.5.1.cmml">,</mo><mi mathvariant="normal" id="A2.p1.3.m3.3.3" xref="A2.p1.3.m3.3.3.cmml">…</mi><mo id="A2.p1.3.m3.4.5.2.4" xref="A2.p1.3.m3.4.5.1.cmml">,</mo><mover accent="true" id="A2.p1.3.m3.4.4" xref="A2.p1.3.m3.4.4.cmml"><mi id="A2.p1.3.m3.4.4.2" xref="A2.p1.3.m3.4.4.2.cmml">r</mi><mo id="A2.p1.3.m3.4.4.1" xref="A2.p1.3.m3.4.4.1.cmml">^</mo></mover><mo stretchy="false" id="A2.p1.3.m3.4.5.2.5" xref="A2.p1.3.m3.4.5.1.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="A2.p1.3.m3.4b"><set id="A2.p1.3.m3.4.5.1.cmml" xref="A2.p1.3.m3.4.5.2"><cn type="integer" id="A2.p1.3.m3.1.1.cmml" xref="A2.p1.3.m3.1.1">1</cn><cn type="integer" id="A2.p1.3.m3.2.2.cmml" xref="A2.p1.3.m3.2.2">2</cn><ci id="A2.p1.3.m3.3.3.cmml" xref="A2.p1.3.m3.3.3">…</ci><apply id="A2.p1.3.m3.4.4.cmml" xref="A2.p1.3.m3.4.4"><ci id="A2.p1.3.m3.4.4.1.cmml" xref="A2.p1.3.m3.4.4.1">^</ci><ci id="A2.p1.3.m3.4.4.2.cmml" xref="A2.p1.3.m3.4.4.2">𝑟</ci></apply></set></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.3.m3.4c">\{1,2,\ldots,\hat{r}\}</annotation></semantics></math>.
The second kind is grouping every neighboring <math id="A2.p1.4.m4.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A2.p1.4.m4.1a"><mi id="A2.p1.4.m4.1.1" xref="A2.p1.4.m4.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A2.p1.4.m4.1b"><ci id="A2.p1.4.m4.1.1.cmml" xref="A2.p1.4.m4.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.4.m4.1c">k</annotation></semantics></math> of the rows or columns together, the first groups can be represented as <math id="A2.p1.5.m5.4" class="ltx_Math" alttext="\{1,1+k,\ldots,1+\hat{r}k\}" display="inline"><semantics id="A2.p1.5.m5.4a"><mrow id="A2.p1.5.m5.4.4.2" xref="A2.p1.5.m5.4.4.3.cmml"><mo stretchy="false" id="A2.p1.5.m5.4.4.2.3" xref="A2.p1.5.m5.4.4.3.cmml">{</mo><mn id="A2.p1.5.m5.1.1" xref="A2.p1.5.m5.1.1.cmml">1</mn><mo id="A2.p1.5.m5.4.4.2.4" xref="A2.p1.5.m5.4.4.3.cmml">,</mo><mrow id="A2.p1.5.m5.3.3.1.1" xref="A2.p1.5.m5.3.3.1.1.cmml"><mn id="A2.p1.5.m5.3.3.1.1.2" xref="A2.p1.5.m5.3.3.1.1.2.cmml">1</mn><mo id="A2.p1.5.m5.3.3.1.1.1" xref="A2.p1.5.m5.3.3.1.1.1.cmml">+</mo><mi id="A2.p1.5.m5.3.3.1.1.3" xref="A2.p1.5.m5.3.3.1.1.3.cmml">k</mi></mrow><mo id="A2.p1.5.m5.4.4.2.5" xref="A2.p1.5.m5.4.4.3.cmml">,</mo><mi mathvariant="normal" id="A2.p1.5.m5.2.2" xref="A2.p1.5.m5.2.2.cmml">…</mi><mo id="A2.p1.5.m5.4.4.2.6" xref="A2.p1.5.m5.4.4.3.cmml">,</mo><mrow id="A2.p1.5.m5.4.4.2.2" xref="A2.p1.5.m5.4.4.2.2.cmml"><mn id="A2.p1.5.m5.4.4.2.2.2" xref="A2.p1.5.m5.4.4.2.2.2.cmml">1</mn><mo id="A2.p1.5.m5.4.4.2.2.1" xref="A2.p1.5.m5.4.4.2.2.1.cmml">+</mo><mrow id="A2.p1.5.m5.4.4.2.2.3" xref="A2.p1.5.m5.4.4.2.2.3.cmml"><mover accent="true" id="A2.p1.5.m5.4.4.2.2.3.2" xref="A2.p1.5.m5.4.4.2.2.3.2.cmml"><mi id="A2.p1.5.m5.4.4.2.2.3.2.2" xref="A2.p1.5.m5.4.4.2.2.3.2.2.cmml">r</mi><mo id="A2.p1.5.m5.4.4.2.2.3.2.1" xref="A2.p1.5.m5.4.4.2.2.3.2.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A2.p1.5.m5.4.4.2.2.3.1" xref="A2.p1.5.m5.4.4.2.2.3.1.cmml">​</mo><mi id="A2.p1.5.m5.4.4.2.2.3.3" xref="A2.p1.5.m5.4.4.2.2.3.3.cmml">k</mi></mrow></mrow><mo stretchy="false" id="A2.p1.5.m5.4.4.2.7" xref="A2.p1.5.m5.4.4.3.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="A2.p1.5.m5.4b"><set id="A2.p1.5.m5.4.4.3.cmml" xref="A2.p1.5.m5.4.4.2"><cn type="integer" id="A2.p1.5.m5.1.1.cmml" xref="A2.p1.5.m5.1.1">1</cn><apply id="A2.p1.5.m5.3.3.1.1.cmml" xref="A2.p1.5.m5.3.3.1.1"><plus id="A2.p1.5.m5.3.3.1.1.1.cmml" xref="A2.p1.5.m5.3.3.1.1.1"></plus><cn type="integer" id="A2.p1.5.m5.3.3.1.1.2.cmml" xref="A2.p1.5.m5.3.3.1.1.2">1</cn><ci id="A2.p1.5.m5.3.3.1.1.3.cmml" xref="A2.p1.5.m5.3.3.1.1.3">𝑘</ci></apply><ci id="A2.p1.5.m5.2.2.cmml" xref="A2.p1.5.m5.2.2">…</ci><apply id="A2.p1.5.m5.4.4.2.2.cmml" xref="A2.p1.5.m5.4.4.2.2"><plus id="A2.p1.5.m5.4.4.2.2.1.cmml" xref="A2.p1.5.m5.4.4.2.2.1"></plus><cn type="integer" id="A2.p1.5.m5.4.4.2.2.2.cmml" xref="A2.p1.5.m5.4.4.2.2.2">1</cn><apply id="A2.p1.5.m5.4.4.2.2.3.cmml" xref="A2.p1.5.m5.4.4.2.2.3"><times id="A2.p1.5.m5.4.4.2.2.3.1.cmml" xref="A2.p1.5.m5.4.4.2.2.3.1"></times><apply id="A2.p1.5.m5.4.4.2.2.3.2.cmml" xref="A2.p1.5.m5.4.4.2.2.3.2"><ci id="A2.p1.5.m5.4.4.2.2.3.2.1.cmml" xref="A2.p1.5.m5.4.4.2.2.3.2.1">^</ci><ci id="A2.p1.5.m5.4.4.2.2.3.2.2.cmml" xref="A2.p1.5.m5.4.4.2.2.3.2.2">𝑟</ci></apply><ci id="A2.p1.5.m5.4.4.2.2.3.3.cmml" xref="A2.p1.5.m5.4.4.2.2.3.3">𝑘</ci></apply></apply></set></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.5.m5.4c">\{1,1+k,\ldots,1+\hat{r}k\}</annotation></semantics></math>.
We propose a example code about compression and decompression functions in Algorithm <span id="A2.p1.9.1" class="ltx_text ltx_font_bold">1</span> and <span id="A2.p1.9.2" class="ltx_text ltx_font_bold">2</span>.
After merging, we can change the group type from <math id="A2.p1.6.m6.1" class="ltx_Math" alttext="0" display="inline"><semantics id="A2.p1.6.m6.1a"><mn id="A2.p1.6.m6.1.1" xref="A2.p1.6.m6.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="A2.p1.6.m6.1b"><cn type="integer" id="A2.p1.6.m6.1.1.cmml" xref="A2.p1.6.m6.1.1">0</cn></annotation-xml></semantics></math> to <math id="A2.p1.7.m7.1" class="ltx_Math" alttext="1" display="inline"><semantics id="A2.p1.7.m7.1a"><mn id="A2.p1.7.m7.1.1" xref="A2.p1.7.m7.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="A2.p1.7.m7.1b"><cn type="integer" id="A2.p1.7.m7.1.1.cmml" xref="A2.p1.7.m7.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.7.m7.1c">1</annotation></semantics></math> or <math id="A2.p1.8.m8.1" class="ltx_Math" alttext="1" display="inline"><semantics id="A2.p1.8.m8.1a"><mn id="A2.p1.8.m8.1.1" xref="A2.p1.8.m8.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="A2.p1.8.m8.1b"><cn type="integer" id="A2.p1.8.m8.1.1.cmml" xref="A2.p1.8.m8.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.8.m8.1c">1</annotation></semantics></math> to <math id="A2.p1.9.m9.1" class="ltx_Math" alttext="0" display="inline"><semantics id="A2.p1.9.m9.1a"><mn id="A2.p1.9.m9.1.1" xref="A2.p1.9.m9.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="A2.p1.9.m9.1b"><cn type="integer" id="A2.p1.9.m9.1.1.cmml" xref="A2.p1.9.m9.1.1">0</cn></annotation-xml></semantics></math>.</p>
</div>
<figure id="alg1" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg1.2.1.1" class="ltx_text ltx_font_bold">Algorithm 1</span> </span> Compression</figcaption>
<div id="alg1.3" class="ltx_listing ltx_listing">
<div id="alg1.l1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">1:</span><span id="alg1.l1.1" class="ltx_text ltx_font_bold">function</span> <span id="alg1.l1.2" class="ltx_text ltx_font_smallcaps">compress</span>(<math id="alg1.l1.m1.1" class="ltx_Math" alttext="x" display="inline"><semantics id="alg1.l1.m1.1a"><mi id="alg1.l1.m1.1.1" xref="alg1.l1.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="alg1.l1.m1.1b"><ci id="alg1.l1.m1.1.1.cmml" xref="alg1.l1.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m1.1c">x</annotation></semantics></math>, <math id="alg1.l1.m2.1" class="ltx_Math" alttext="\hat{r}" display="inline"><semantics id="alg1.l1.m2.1a"><mover accent="true" id="alg1.l1.m2.1.1" xref="alg1.l1.m2.1.1.cmml"><mi id="alg1.l1.m2.1.1.2" xref="alg1.l1.m2.1.1.2.cmml">r</mi><mo id="alg1.l1.m2.1.1.1" xref="alg1.l1.m2.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="alg1.l1.m2.1b"><apply id="alg1.l1.m2.1.1.cmml" xref="alg1.l1.m2.1.1"><ci id="alg1.l1.m2.1.1.1.cmml" xref="alg1.l1.m2.1.1.1">^</ci><ci id="alg1.l1.m2.1.1.2.cmml" xref="alg1.l1.m2.1.1.2">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m2.1c">\hat{r}</annotation></semantics></math>, <math id="alg1.l1.m3.1" class="ltx_Math" alttext="type" display="inline"><semantics id="alg1.l1.m3.1a"><mrow id="alg1.l1.m3.1.1" xref="alg1.l1.m3.1.1.cmml"><mi id="alg1.l1.m3.1.1.2" xref="alg1.l1.m3.1.1.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m3.1.1.1" xref="alg1.l1.m3.1.1.1.cmml">​</mo><mi id="alg1.l1.m3.1.1.3" xref="alg1.l1.m3.1.1.3.cmml">y</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m3.1.1.1a" xref="alg1.l1.m3.1.1.1.cmml">​</mo><mi id="alg1.l1.m3.1.1.4" xref="alg1.l1.m3.1.1.4.cmml">p</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m3.1.1.1b" xref="alg1.l1.m3.1.1.1.cmml">​</mo><mi id="alg1.l1.m3.1.1.5" xref="alg1.l1.m3.1.1.5.cmml">e</mi></mrow><annotation-xml encoding="MathML-Content" id="alg1.l1.m3.1b"><apply id="alg1.l1.m3.1.1.cmml" xref="alg1.l1.m3.1.1"><times id="alg1.l1.m3.1.1.1.cmml" xref="alg1.l1.m3.1.1.1"></times><ci id="alg1.l1.m3.1.1.2.cmml" xref="alg1.l1.m3.1.1.2">𝑡</ci><ci id="alg1.l1.m3.1.1.3.cmml" xref="alg1.l1.m3.1.1.3">𝑦</ci><ci id="alg1.l1.m3.1.1.4.cmml" xref="alg1.l1.m3.1.1.4">𝑝</ci><ci id="alg1.l1.m3.1.1.5.cmml" xref="alg1.l1.m3.1.1.5">𝑒</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m3.1c">type</annotation></semantics></math>)

</div>
<div id="alg1.l2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">2:</span>     # <math id="alg1.l2.m1.1" class="ltx_Math" alttext="x\in\mathbb{R}^{bsz\times l\times k}" display="inline"><semantics id="alg1.l2.m1.1a"><mrow id="alg1.l2.m1.1.1" xref="alg1.l2.m1.1.1.cmml"><mi id="alg1.l2.m1.1.1.2" xref="alg1.l2.m1.1.1.2.cmml">x</mi><mo id="alg1.l2.m1.1.1.1" xref="alg1.l2.m1.1.1.1.cmml">∈</mo><msup id="alg1.l2.m1.1.1.3" xref="alg1.l2.m1.1.1.3.cmml"><mi id="alg1.l2.m1.1.1.3.2" xref="alg1.l2.m1.1.1.3.2.cmml">ℝ</mi><mrow id="alg1.l2.m1.1.1.3.3" xref="alg1.l2.m1.1.1.3.3.cmml"><mrow id="alg1.l2.m1.1.1.3.3.2" xref="alg1.l2.m1.1.1.3.3.2.cmml"><mi id="alg1.l2.m1.1.1.3.3.2.2" xref="alg1.l2.m1.1.1.3.3.2.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.3.3.2.1" xref="alg1.l2.m1.1.1.3.3.2.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.3.3.2.3" xref="alg1.l2.m1.1.1.3.3.2.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.3.3.2.1a" xref="alg1.l2.m1.1.1.3.3.2.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.3.3.2.4" xref="alg1.l2.m1.1.1.3.3.2.4.cmml">z</mi></mrow><mo lspace="0.222em" rspace="0.222em" id="alg1.l2.m1.1.1.3.3.1" xref="alg1.l2.m1.1.1.3.3.1.cmml">×</mo><mi id="alg1.l2.m1.1.1.3.3.3" xref="alg1.l2.m1.1.1.3.3.3.cmml">l</mi><mo lspace="0.222em" rspace="0.222em" id="alg1.l2.m1.1.1.3.3.1a" xref="alg1.l2.m1.1.1.3.3.1.cmml">×</mo><mi id="alg1.l2.m1.1.1.3.3.4" xref="alg1.l2.m1.1.1.3.3.4.cmml">k</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="alg1.l2.m1.1b"><apply id="alg1.l2.m1.1.1.cmml" xref="alg1.l2.m1.1.1"><in id="alg1.l2.m1.1.1.1.cmml" xref="alg1.l2.m1.1.1.1"></in><ci id="alg1.l2.m1.1.1.2.cmml" xref="alg1.l2.m1.1.1.2">𝑥</ci><apply id="alg1.l2.m1.1.1.3.cmml" xref="alg1.l2.m1.1.1.3"><csymbol cd="ambiguous" id="alg1.l2.m1.1.1.3.1.cmml" xref="alg1.l2.m1.1.1.3">superscript</csymbol><ci id="alg1.l2.m1.1.1.3.2.cmml" xref="alg1.l2.m1.1.1.3.2">ℝ</ci><apply id="alg1.l2.m1.1.1.3.3.cmml" xref="alg1.l2.m1.1.1.3.3"><times id="alg1.l2.m1.1.1.3.3.1.cmml" xref="alg1.l2.m1.1.1.3.3.1"></times><apply id="alg1.l2.m1.1.1.3.3.2.cmml" xref="alg1.l2.m1.1.1.3.3.2"><times id="alg1.l2.m1.1.1.3.3.2.1.cmml" xref="alg1.l2.m1.1.1.3.3.2.1"></times><ci id="alg1.l2.m1.1.1.3.3.2.2.cmml" xref="alg1.l2.m1.1.1.3.3.2.2">𝑏</ci><ci id="alg1.l2.m1.1.1.3.3.2.3.cmml" xref="alg1.l2.m1.1.1.3.3.2.3">𝑠</ci><ci id="alg1.l2.m1.1.1.3.3.2.4.cmml" xref="alg1.l2.m1.1.1.3.3.2.4">𝑧</ci></apply><ci id="alg1.l2.m1.1.1.3.3.3.cmml" xref="alg1.l2.m1.1.1.3.3.3">𝑙</ci><ci id="alg1.l2.m1.1.1.3.3.4.cmml" xref="alg1.l2.m1.1.1.3.3.4">𝑘</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l2.m1.1c">x\in\mathbb{R}^{bsz\times l\times k}</annotation></semantics></math>: Input tensor

</div>
<div id="alg1.l3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">3:</span>     # <math id="alg1.l3.m1.1" class="ltx_Math" alttext="y\in\mathbb{R}^{bsz\times l\times\hat{r}}" display="inline"><semantics id="alg1.l3.m1.1a"><mrow id="alg1.l3.m1.1.1" xref="alg1.l3.m1.1.1.cmml"><mi id="alg1.l3.m1.1.1.2" xref="alg1.l3.m1.1.1.2.cmml">y</mi><mo id="alg1.l3.m1.1.1.1" xref="alg1.l3.m1.1.1.1.cmml">∈</mo><msup id="alg1.l3.m1.1.1.3" xref="alg1.l3.m1.1.1.3.cmml"><mi id="alg1.l3.m1.1.1.3.2" xref="alg1.l3.m1.1.1.3.2.cmml">ℝ</mi><mrow id="alg1.l3.m1.1.1.3.3" xref="alg1.l3.m1.1.1.3.3.cmml"><mrow id="alg1.l3.m1.1.1.3.3.2" xref="alg1.l3.m1.1.1.3.3.2.cmml"><mi id="alg1.l3.m1.1.1.3.3.2.2" xref="alg1.l3.m1.1.1.3.3.2.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l3.m1.1.1.3.3.2.1" xref="alg1.l3.m1.1.1.3.3.2.1.cmml">​</mo><mi id="alg1.l3.m1.1.1.3.3.2.3" xref="alg1.l3.m1.1.1.3.3.2.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l3.m1.1.1.3.3.2.1a" xref="alg1.l3.m1.1.1.3.3.2.1.cmml">​</mo><mi id="alg1.l3.m1.1.1.3.3.2.4" xref="alg1.l3.m1.1.1.3.3.2.4.cmml">z</mi></mrow><mo lspace="0.222em" rspace="0.222em" id="alg1.l3.m1.1.1.3.3.1" xref="alg1.l3.m1.1.1.3.3.1.cmml">×</mo><mi id="alg1.l3.m1.1.1.3.3.3" xref="alg1.l3.m1.1.1.3.3.3.cmml">l</mi><mo lspace="0.222em" rspace="0.222em" id="alg1.l3.m1.1.1.3.3.1a" xref="alg1.l3.m1.1.1.3.3.1.cmml">×</mo><mover accent="true" id="alg1.l3.m1.1.1.3.3.4" xref="alg1.l3.m1.1.1.3.3.4.cmml"><mi id="alg1.l3.m1.1.1.3.3.4.2" xref="alg1.l3.m1.1.1.3.3.4.2.cmml">r</mi><mo id="alg1.l3.m1.1.1.3.3.4.1" xref="alg1.l3.m1.1.1.3.3.4.1.cmml">^</mo></mover></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="alg1.l3.m1.1b"><apply id="alg1.l3.m1.1.1.cmml" xref="alg1.l3.m1.1.1"><in id="alg1.l3.m1.1.1.1.cmml" xref="alg1.l3.m1.1.1.1"></in><ci id="alg1.l3.m1.1.1.2.cmml" xref="alg1.l3.m1.1.1.2">𝑦</ci><apply id="alg1.l3.m1.1.1.3.cmml" xref="alg1.l3.m1.1.1.3"><csymbol cd="ambiguous" id="alg1.l3.m1.1.1.3.1.cmml" xref="alg1.l3.m1.1.1.3">superscript</csymbol><ci id="alg1.l3.m1.1.1.3.2.cmml" xref="alg1.l3.m1.1.1.3.2">ℝ</ci><apply id="alg1.l3.m1.1.1.3.3.cmml" xref="alg1.l3.m1.1.1.3.3"><times id="alg1.l3.m1.1.1.3.3.1.cmml" xref="alg1.l3.m1.1.1.3.3.1"></times><apply id="alg1.l3.m1.1.1.3.3.2.cmml" xref="alg1.l3.m1.1.1.3.3.2"><times id="alg1.l3.m1.1.1.3.3.2.1.cmml" xref="alg1.l3.m1.1.1.3.3.2.1"></times><ci id="alg1.l3.m1.1.1.3.3.2.2.cmml" xref="alg1.l3.m1.1.1.3.3.2.2">𝑏</ci><ci id="alg1.l3.m1.1.1.3.3.2.3.cmml" xref="alg1.l3.m1.1.1.3.3.2.3">𝑠</ci><ci id="alg1.l3.m1.1.1.3.3.2.4.cmml" xref="alg1.l3.m1.1.1.3.3.2.4">𝑧</ci></apply><ci id="alg1.l3.m1.1.1.3.3.3.cmml" xref="alg1.l3.m1.1.1.3.3.3">𝑙</ci><apply id="alg1.l3.m1.1.1.3.3.4.cmml" xref="alg1.l3.m1.1.1.3.3.4"><ci id="alg1.l3.m1.1.1.3.3.4.1.cmml" xref="alg1.l3.m1.1.1.3.3.4.1">^</ci><ci id="alg1.l3.m1.1.1.3.3.4.2.cmml" xref="alg1.l3.m1.1.1.3.3.4.2">𝑟</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l3.m1.1c">y\in\mathbb{R}^{bsz\times l\times\hat{r}}</annotation></semantics></math>: Output tensor

</div>
<div id="alg1.l4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">4:</span>     # <math id="alg1.l4.m1.2" class="ltx_Math" alttext="type\in\{0,1\}" display="inline"><semantics id="alg1.l4.m1.2a"><mrow id="alg1.l4.m1.2.3" xref="alg1.l4.m1.2.3.cmml"><mrow id="alg1.l4.m1.2.3.2" xref="alg1.l4.m1.2.3.2.cmml"><mi id="alg1.l4.m1.2.3.2.2" xref="alg1.l4.m1.2.3.2.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l4.m1.2.3.2.1" xref="alg1.l4.m1.2.3.2.1.cmml">​</mo><mi id="alg1.l4.m1.2.3.2.3" xref="alg1.l4.m1.2.3.2.3.cmml">y</mi><mo lspace="0em" rspace="0em" id="alg1.l4.m1.2.3.2.1a" xref="alg1.l4.m1.2.3.2.1.cmml">​</mo><mi id="alg1.l4.m1.2.3.2.4" xref="alg1.l4.m1.2.3.2.4.cmml">p</mi><mo lspace="0em" rspace="0em" id="alg1.l4.m1.2.3.2.1b" xref="alg1.l4.m1.2.3.2.1.cmml">​</mo><mi id="alg1.l4.m1.2.3.2.5" xref="alg1.l4.m1.2.3.2.5.cmml">e</mi></mrow><mo id="alg1.l4.m1.2.3.1" xref="alg1.l4.m1.2.3.1.cmml">∈</mo><mrow id="alg1.l4.m1.2.3.3.2" xref="alg1.l4.m1.2.3.3.1.cmml"><mo stretchy="false" id="alg1.l4.m1.2.3.3.2.1" xref="alg1.l4.m1.2.3.3.1.cmml">{</mo><mn id="alg1.l4.m1.1.1" xref="alg1.l4.m1.1.1.cmml">0</mn><mo id="alg1.l4.m1.2.3.3.2.2" xref="alg1.l4.m1.2.3.3.1.cmml">,</mo><mn id="alg1.l4.m1.2.2" xref="alg1.l4.m1.2.2.cmml">1</mn><mo stretchy="false" id="alg1.l4.m1.2.3.3.2.3" xref="alg1.l4.m1.2.3.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l4.m1.2b"><apply id="alg1.l4.m1.2.3.cmml" xref="alg1.l4.m1.2.3"><in id="alg1.l4.m1.2.3.1.cmml" xref="alg1.l4.m1.2.3.1"></in><apply id="alg1.l4.m1.2.3.2.cmml" xref="alg1.l4.m1.2.3.2"><times id="alg1.l4.m1.2.3.2.1.cmml" xref="alg1.l4.m1.2.3.2.1"></times><ci id="alg1.l4.m1.2.3.2.2.cmml" xref="alg1.l4.m1.2.3.2.2">𝑡</ci><ci id="alg1.l4.m1.2.3.2.3.cmml" xref="alg1.l4.m1.2.3.2.3">𝑦</ci><ci id="alg1.l4.m1.2.3.2.4.cmml" xref="alg1.l4.m1.2.3.2.4">𝑝</ci><ci id="alg1.l4.m1.2.3.2.5.cmml" xref="alg1.l4.m1.2.3.2.5">𝑒</ci></apply><set id="alg1.l4.m1.2.3.3.1.cmml" xref="alg1.l4.m1.2.3.3.2"><cn type="integer" id="alg1.l4.m1.1.1.cmml" xref="alg1.l4.m1.1.1">0</cn><cn type="integer" id="alg1.l4.m1.2.2.cmml" xref="alg1.l4.m1.2.2">1</cn></set></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l4.m1.2c">type\in\{0,1\}</annotation></semantics></math>: Group type 0 or 1

</div>
<div id="alg1.l5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">5:</span>     padding <math id="alg1.l5.m1.1" class="ltx_Math" alttext="x" display="inline"><semantics id="alg1.l5.m1.1a"><mi id="alg1.l5.m1.1.1" xref="alg1.l5.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="alg1.l5.m1.1b"><ci id="alg1.l5.m1.1.1.cmml" xref="alg1.l5.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m1.1c">x</annotation></semantics></math> to make <math id="alg1.l5.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="alg1.l5.m2.1a"><mi id="alg1.l5.m2.1.1" xref="alg1.l5.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="alg1.l5.m2.1b"><ci id="alg1.l5.m2.1.1.cmml" xref="alg1.l5.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m2.1c">k</annotation></semantics></math> divisible by <math id="alg1.l5.m3.1" class="ltx_Math" alttext="\hat{r}" display="inline"><semantics id="alg1.l5.m3.1a"><mover accent="true" id="alg1.l5.m3.1.1" xref="alg1.l5.m3.1.1.cmml"><mi id="alg1.l5.m3.1.1.2" xref="alg1.l5.m3.1.1.2.cmml">r</mi><mo id="alg1.l5.m3.1.1.1" xref="alg1.l5.m3.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="alg1.l5.m3.1b"><apply id="alg1.l5.m3.1.1.cmml" xref="alg1.l5.m3.1.1"><ci id="alg1.l5.m3.1.1.1.cmml" xref="alg1.l5.m3.1.1.1">^</ci><ci id="alg1.l5.m3.1.1.2.cmml" xref="alg1.l5.m3.1.1.2">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m3.1c">\hat{r}</annotation></semantics></math>

</div>
<div id="alg1.l6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">6:</span>     <span id="alg1.l6.1" class="ltx_text ltx_font_bold">if</span> <math id="alg1.l6.m1.1" class="ltx_Math" alttext="type=0" display="inline"><semantics id="alg1.l6.m1.1a"><mrow id="alg1.l6.m1.1.1" xref="alg1.l6.m1.1.1.cmml"><mrow id="alg1.l6.m1.1.1.2" xref="alg1.l6.m1.1.1.2.cmml"><mi id="alg1.l6.m1.1.1.2.2" xref="alg1.l6.m1.1.1.2.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l6.m1.1.1.2.1" xref="alg1.l6.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l6.m1.1.1.2.3" xref="alg1.l6.m1.1.1.2.3.cmml">y</mi><mo lspace="0em" rspace="0em" id="alg1.l6.m1.1.1.2.1a" xref="alg1.l6.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l6.m1.1.1.2.4" xref="alg1.l6.m1.1.1.2.4.cmml">p</mi><mo lspace="0em" rspace="0em" id="alg1.l6.m1.1.1.2.1b" xref="alg1.l6.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l6.m1.1.1.2.5" xref="alg1.l6.m1.1.1.2.5.cmml">e</mi></mrow><mo id="alg1.l6.m1.1.1.1" xref="alg1.l6.m1.1.1.1.cmml">=</mo><mn id="alg1.l6.m1.1.1.3" xref="alg1.l6.m1.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="alg1.l6.m1.1b"><apply id="alg1.l6.m1.1.1.cmml" xref="alg1.l6.m1.1.1"><eq id="alg1.l6.m1.1.1.1.cmml" xref="alg1.l6.m1.1.1.1"></eq><apply id="alg1.l6.m1.1.1.2.cmml" xref="alg1.l6.m1.1.1.2"><times id="alg1.l6.m1.1.1.2.1.cmml" xref="alg1.l6.m1.1.1.2.1"></times><ci id="alg1.l6.m1.1.1.2.2.cmml" xref="alg1.l6.m1.1.1.2.2">𝑡</ci><ci id="alg1.l6.m1.1.1.2.3.cmml" xref="alg1.l6.m1.1.1.2.3">𝑦</ci><ci id="alg1.l6.m1.1.1.2.4.cmml" xref="alg1.l6.m1.1.1.2.4">𝑝</ci><ci id="alg1.l6.m1.1.1.2.5.cmml" xref="alg1.l6.m1.1.1.2.5">𝑒</ci></apply><cn type="integer" id="alg1.l6.m1.1.1.3.cmml" xref="alg1.l6.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l6.m1.1c">type=0</annotation></semantics></math> <span id="alg1.l6.2" class="ltx_text ltx_font_bold">then</span>

</div>
<div id="alg1.l7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">7:</span>         <math id="alg1.l7.m1.1" class="ltx_Math" alttext="y" display="inline"><semantics id="alg1.l7.m1.1a"><mi id="alg1.l7.m1.1.1" xref="alg1.l7.m1.1.1.cmml">y</mi><annotation-xml encoding="MathML-Content" id="alg1.l7.m1.1b"><ci id="alg1.l7.m1.1.1.cmml" xref="alg1.l7.m1.1.1">𝑦</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l7.m1.1c">y</annotation></semantics></math> = <math id="alg1.l7.m2.1" class="ltx_Math" alttext="x" display="inline"><semantics id="alg1.l7.m2.1a"><mi id="alg1.l7.m2.1.1" xref="alg1.l7.m2.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="alg1.l7.m2.1b"><ci id="alg1.l7.m2.1.1.cmml" xref="alg1.l7.m2.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l7.m2.1c">x</annotation></semantics></math>.view(<math id="alg1.l7.m3.4" class="ltx_Math" alttext="bsz,l,k/\hat{r},\hat{r}" display="inline"><semantics id="alg1.l7.m3.4a"><mrow id="alg1.l7.m3.4.4.2" xref="alg1.l7.m3.4.4.3.cmml"><mrow id="alg1.l7.m3.3.3.1.1" xref="alg1.l7.m3.3.3.1.1.cmml"><mi id="alg1.l7.m3.3.3.1.1.2" xref="alg1.l7.m3.3.3.1.1.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m3.3.3.1.1.1" xref="alg1.l7.m3.3.3.1.1.1.cmml">​</mo><mi id="alg1.l7.m3.3.3.1.1.3" xref="alg1.l7.m3.3.3.1.1.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m3.3.3.1.1.1a" xref="alg1.l7.m3.3.3.1.1.1.cmml">​</mo><mi id="alg1.l7.m3.3.3.1.1.4" xref="alg1.l7.m3.3.3.1.1.4.cmml">z</mi></mrow><mo id="alg1.l7.m3.4.4.2.3" xref="alg1.l7.m3.4.4.3.cmml">,</mo><mi id="alg1.l7.m3.1.1" xref="alg1.l7.m3.1.1.cmml">l</mi><mo id="alg1.l7.m3.4.4.2.4" xref="alg1.l7.m3.4.4.3.cmml">,</mo><mrow id="alg1.l7.m3.4.4.2.2" xref="alg1.l7.m3.4.4.2.2.cmml"><mi id="alg1.l7.m3.4.4.2.2.2" xref="alg1.l7.m3.4.4.2.2.2.cmml">k</mi><mo id="alg1.l7.m3.4.4.2.2.1" xref="alg1.l7.m3.4.4.2.2.1.cmml">/</mo><mover accent="true" id="alg1.l7.m3.4.4.2.2.3" xref="alg1.l7.m3.4.4.2.2.3.cmml"><mi id="alg1.l7.m3.4.4.2.2.3.2" xref="alg1.l7.m3.4.4.2.2.3.2.cmml">r</mi><mo id="alg1.l7.m3.4.4.2.2.3.1" xref="alg1.l7.m3.4.4.2.2.3.1.cmml">^</mo></mover></mrow><mo id="alg1.l7.m3.4.4.2.5" xref="alg1.l7.m3.4.4.3.cmml">,</mo><mover accent="true" id="alg1.l7.m3.2.2" xref="alg1.l7.m3.2.2.cmml"><mi id="alg1.l7.m3.2.2.2" xref="alg1.l7.m3.2.2.2.cmml">r</mi><mo id="alg1.l7.m3.2.2.1" xref="alg1.l7.m3.2.2.1.cmml">^</mo></mover></mrow><annotation-xml encoding="MathML-Content" id="alg1.l7.m3.4b"><list id="alg1.l7.m3.4.4.3.cmml" xref="alg1.l7.m3.4.4.2"><apply id="alg1.l7.m3.3.3.1.1.cmml" xref="alg1.l7.m3.3.3.1.1"><times id="alg1.l7.m3.3.3.1.1.1.cmml" xref="alg1.l7.m3.3.3.1.1.1"></times><ci id="alg1.l7.m3.3.3.1.1.2.cmml" xref="alg1.l7.m3.3.3.1.1.2">𝑏</ci><ci id="alg1.l7.m3.3.3.1.1.3.cmml" xref="alg1.l7.m3.3.3.1.1.3">𝑠</ci><ci id="alg1.l7.m3.3.3.1.1.4.cmml" xref="alg1.l7.m3.3.3.1.1.4">𝑧</ci></apply><ci id="alg1.l7.m3.1.1.cmml" xref="alg1.l7.m3.1.1">𝑙</ci><apply id="alg1.l7.m3.4.4.2.2.cmml" xref="alg1.l7.m3.4.4.2.2"><divide id="alg1.l7.m3.4.4.2.2.1.cmml" xref="alg1.l7.m3.4.4.2.2.1"></divide><ci id="alg1.l7.m3.4.4.2.2.2.cmml" xref="alg1.l7.m3.4.4.2.2.2">𝑘</ci><apply id="alg1.l7.m3.4.4.2.2.3.cmml" xref="alg1.l7.m3.4.4.2.2.3"><ci id="alg1.l7.m3.4.4.2.2.3.1.cmml" xref="alg1.l7.m3.4.4.2.2.3.1">^</ci><ci id="alg1.l7.m3.4.4.2.2.3.2.cmml" xref="alg1.l7.m3.4.4.2.2.3.2">𝑟</ci></apply></apply><apply id="alg1.l7.m3.2.2.cmml" xref="alg1.l7.m3.2.2"><ci id="alg1.l7.m3.2.2.1.cmml" xref="alg1.l7.m3.2.2.1">^</ci><ci id="alg1.l7.m3.2.2.2.cmml" xref="alg1.l7.m3.2.2.2">𝑟</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg1.l7.m3.4c">bsz,l,k/\hat{r},\hat{r}</annotation></semantics></math>).sum(dim=<math id="alg1.l7.m4.1" class="ltx_Math" alttext="2" display="inline"><semantics id="alg1.l7.m4.1a"><mn id="alg1.l7.m4.1.1" xref="alg1.l7.m4.1.1.cmml">2</mn><annotation-xml encoding="MathML-Content" id="alg1.l7.m4.1b"><cn type="integer" id="alg1.l7.m4.1.1.cmml" xref="alg1.l7.m4.1.1">2</cn></annotation-xml><annotation encoding="application/x-tex" id="alg1.l7.m4.1c">2</annotation></semantics></math>) # first type of group

</div>
<div id="alg1.l8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">8:</span>     <span id="alg1.l8.1" class="ltx_text ltx_font_bold">else</span>
</div>
<div id="alg1.l9" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">9:</span>         <math id="alg1.l9.m1.1" class="ltx_Math" alttext="y" display="inline"><semantics id="alg1.l9.m1.1a"><mi id="alg1.l9.m1.1.1" xref="alg1.l9.m1.1.1.cmml">y</mi><annotation-xml encoding="MathML-Content" id="alg1.l9.m1.1b"><ci id="alg1.l9.m1.1.1.cmml" xref="alg1.l9.m1.1.1">𝑦</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l9.m1.1c">y</annotation></semantics></math> = <math id="alg1.l9.m2.1" class="ltx_Math" alttext="x" display="inline"><semantics id="alg1.l9.m2.1a"><mi id="alg1.l9.m2.1.1" xref="alg1.l9.m2.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="alg1.l9.m2.1b"><ci id="alg1.l9.m2.1.1.cmml" xref="alg1.l9.m2.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l9.m2.1c">x</annotation></semantics></math>.view(<math id="alg1.l9.m3.4" class="ltx_Math" alttext="bsz,l,\hat{r},k/\hat{r}" display="inline"><semantics id="alg1.l9.m3.4a"><mrow id="alg1.l9.m3.4.4.2" xref="alg1.l9.m3.4.4.3.cmml"><mrow id="alg1.l9.m3.3.3.1.1" xref="alg1.l9.m3.3.3.1.1.cmml"><mi id="alg1.l9.m3.3.3.1.1.2" xref="alg1.l9.m3.3.3.1.1.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l9.m3.3.3.1.1.1" xref="alg1.l9.m3.3.3.1.1.1.cmml">​</mo><mi id="alg1.l9.m3.3.3.1.1.3" xref="alg1.l9.m3.3.3.1.1.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l9.m3.3.3.1.1.1a" xref="alg1.l9.m3.3.3.1.1.1.cmml">​</mo><mi id="alg1.l9.m3.3.3.1.1.4" xref="alg1.l9.m3.3.3.1.1.4.cmml">z</mi></mrow><mo id="alg1.l9.m3.4.4.2.3" xref="alg1.l9.m3.4.4.3.cmml">,</mo><mi id="alg1.l9.m3.1.1" xref="alg1.l9.m3.1.1.cmml">l</mi><mo id="alg1.l9.m3.4.4.2.4" xref="alg1.l9.m3.4.4.3.cmml">,</mo><mover accent="true" id="alg1.l9.m3.2.2" xref="alg1.l9.m3.2.2.cmml"><mi id="alg1.l9.m3.2.2.2" xref="alg1.l9.m3.2.2.2.cmml">r</mi><mo id="alg1.l9.m3.2.2.1" xref="alg1.l9.m3.2.2.1.cmml">^</mo></mover><mo id="alg1.l9.m3.4.4.2.5" xref="alg1.l9.m3.4.4.3.cmml">,</mo><mrow id="alg1.l9.m3.4.4.2.2" xref="alg1.l9.m3.4.4.2.2.cmml"><mi id="alg1.l9.m3.4.4.2.2.2" xref="alg1.l9.m3.4.4.2.2.2.cmml">k</mi><mo id="alg1.l9.m3.4.4.2.2.1" xref="alg1.l9.m3.4.4.2.2.1.cmml">/</mo><mover accent="true" id="alg1.l9.m3.4.4.2.2.3" xref="alg1.l9.m3.4.4.2.2.3.cmml"><mi id="alg1.l9.m3.4.4.2.2.3.2" xref="alg1.l9.m3.4.4.2.2.3.2.cmml">r</mi><mo id="alg1.l9.m3.4.4.2.2.3.1" xref="alg1.l9.m3.4.4.2.2.3.1.cmml">^</mo></mover></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l9.m3.4b"><list id="alg1.l9.m3.4.4.3.cmml" xref="alg1.l9.m3.4.4.2"><apply id="alg1.l9.m3.3.3.1.1.cmml" xref="alg1.l9.m3.3.3.1.1"><times id="alg1.l9.m3.3.3.1.1.1.cmml" xref="alg1.l9.m3.3.3.1.1.1"></times><ci id="alg1.l9.m3.3.3.1.1.2.cmml" xref="alg1.l9.m3.3.3.1.1.2">𝑏</ci><ci id="alg1.l9.m3.3.3.1.1.3.cmml" xref="alg1.l9.m3.3.3.1.1.3">𝑠</ci><ci id="alg1.l9.m3.3.3.1.1.4.cmml" xref="alg1.l9.m3.3.3.1.1.4">𝑧</ci></apply><ci id="alg1.l9.m3.1.1.cmml" xref="alg1.l9.m3.1.1">𝑙</ci><apply id="alg1.l9.m3.2.2.cmml" xref="alg1.l9.m3.2.2"><ci id="alg1.l9.m3.2.2.1.cmml" xref="alg1.l9.m3.2.2.1">^</ci><ci id="alg1.l9.m3.2.2.2.cmml" xref="alg1.l9.m3.2.2.2">𝑟</ci></apply><apply id="alg1.l9.m3.4.4.2.2.cmml" xref="alg1.l9.m3.4.4.2.2"><divide id="alg1.l9.m3.4.4.2.2.1.cmml" xref="alg1.l9.m3.4.4.2.2.1"></divide><ci id="alg1.l9.m3.4.4.2.2.2.cmml" xref="alg1.l9.m3.4.4.2.2.2">𝑘</ci><apply id="alg1.l9.m3.4.4.2.2.3.cmml" xref="alg1.l9.m3.4.4.2.2.3"><ci id="alg1.l9.m3.4.4.2.2.3.1.cmml" xref="alg1.l9.m3.4.4.2.2.3.1">^</ci><ci id="alg1.l9.m3.4.4.2.2.3.2.cmml" xref="alg1.l9.m3.4.4.2.2.3.2">𝑟</ci></apply></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg1.l9.m3.4c">bsz,l,\hat{r},k/\hat{r}</annotation></semantics></math>).sum(dim=<math id="alg1.l9.m4.1" class="ltx_Math" alttext="3" display="inline"><semantics id="alg1.l9.m4.1a"><mn id="alg1.l9.m4.1.1" xref="alg1.l9.m4.1.1.cmml">3</mn><annotation-xml encoding="MathML-Content" id="alg1.l9.m4.1b"><cn type="integer" id="alg1.l9.m4.1.1.cmml" xref="alg1.l9.m4.1.1">3</cn></annotation-xml><annotation encoding="application/x-tex" id="alg1.l9.m4.1c">3</annotation></semantics></math>) # second type of group

</div>
<div id="alg1.l10" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">10:</span>     <span id="alg1.l10.1" class="ltx_text ltx_font_bold">end</span> <span id="alg1.l10.2" class="ltx_text ltx_font_bold">if</span>
</div>
<div id="alg1.l11" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">11:</span>     <span id="alg1.l11.1" class="ltx_text ltx_font_bold">return</span> <math id="alg1.l11.m1.1" class="ltx_Math" alttext="y" display="inline"><semantics id="alg1.l11.m1.1a"><mi id="alg1.l11.m1.1.1" xref="alg1.l11.m1.1.1.cmml">y</mi><annotation-xml encoding="MathML-Content" id="alg1.l11.m1.1b"><ci id="alg1.l11.m1.1.1.cmml" xref="alg1.l11.m1.1.1">𝑦</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l11.m1.1c">y</annotation></semantics></math>

</div>
<div id="alg1.l12" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">12:</span><span id="alg1.l12.1" class="ltx_text ltx_font_bold">end</span> <span id="alg1.l12.2" class="ltx_text ltx_font_bold">function</span>
</div>
</div>
</figure>
<figure id="alg2" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg2.2.1.1" class="ltx_text ltx_font_bold">Algorithm 2</span> </span> Decompression</figcaption>
<div id="alg2.3" class="ltx_listing ltx_listing">
<div id="alg2.l1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">1:</span><span id="alg2.l1.1" class="ltx_text ltx_font_bold">function</span> <span id="alg2.l1.2" class="ltx_text ltx_font_smallcaps">decompress</span>(<math id="alg2.l1.m1.1" class="ltx_Math" alttext="x" display="inline"><semantics id="alg2.l1.m1.1a"><mi id="alg2.l1.m1.1.1" xref="alg2.l1.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="alg2.l1.m1.1b"><ci id="alg2.l1.m1.1.1.cmml" xref="alg2.l1.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l1.m1.1c">x</annotation></semantics></math>, <math id="alg2.l1.m2.1" class="ltx_Math" alttext="\hat{r}" display="inline"><semantics id="alg2.l1.m2.1a"><mover accent="true" id="alg2.l1.m2.1.1" xref="alg2.l1.m2.1.1.cmml"><mi id="alg2.l1.m2.1.1.2" xref="alg2.l1.m2.1.1.2.cmml">r</mi><mo id="alg2.l1.m2.1.1.1" xref="alg2.l1.m2.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="alg2.l1.m2.1b"><apply id="alg2.l1.m2.1.1.cmml" xref="alg2.l1.m2.1.1"><ci id="alg2.l1.m2.1.1.1.cmml" xref="alg2.l1.m2.1.1.1">^</ci><ci id="alg2.l1.m2.1.1.2.cmml" xref="alg2.l1.m2.1.1.2">𝑟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l1.m2.1c">\hat{r}</annotation></semantics></math>, <math id="alg2.l1.m3.1" class="ltx_Math" alttext="type" display="inline"><semantics id="alg2.l1.m3.1a"><mrow id="alg2.l1.m3.1.1" xref="alg2.l1.m3.1.1.cmml"><mi id="alg2.l1.m3.1.1.2" xref="alg2.l1.m3.1.1.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg2.l1.m3.1.1.1" xref="alg2.l1.m3.1.1.1.cmml">​</mo><mi id="alg2.l1.m3.1.1.3" xref="alg2.l1.m3.1.1.3.cmml">y</mi><mo lspace="0em" rspace="0em" id="alg2.l1.m3.1.1.1a" xref="alg2.l1.m3.1.1.1.cmml">​</mo><mi id="alg2.l1.m3.1.1.4" xref="alg2.l1.m3.1.1.4.cmml">p</mi><mo lspace="0em" rspace="0em" id="alg2.l1.m3.1.1.1b" xref="alg2.l1.m3.1.1.1.cmml">​</mo><mi id="alg2.l1.m3.1.1.5" xref="alg2.l1.m3.1.1.5.cmml">e</mi></mrow><annotation-xml encoding="MathML-Content" id="alg2.l1.m3.1b"><apply id="alg2.l1.m3.1.1.cmml" xref="alg2.l1.m3.1.1"><times id="alg2.l1.m3.1.1.1.cmml" xref="alg2.l1.m3.1.1.1"></times><ci id="alg2.l1.m3.1.1.2.cmml" xref="alg2.l1.m3.1.1.2">𝑡</ci><ci id="alg2.l1.m3.1.1.3.cmml" xref="alg2.l1.m3.1.1.3">𝑦</ci><ci id="alg2.l1.m3.1.1.4.cmml" xref="alg2.l1.m3.1.1.4">𝑝</ci><ci id="alg2.l1.m3.1.1.5.cmml" xref="alg2.l1.m3.1.1.5">𝑒</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l1.m3.1c">type</annotation></semantics></math>)

</div>
<div id="alg2.l2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">2:</span>     # <math id="alg2.l2.m1.1" class="ltx_Math" alttext="x\in\mathbb{R}^{bsz\times l\times\hat{r}}" display="inline"><semantics id="alg2.l2.m1.1a"><mrow id="alg2.l2.m1.1.1" xref="alg2.l2.m1.1.1.cmml"><mi id="alg2.l2.m1.1.1.2" xref="alg2.l2.m1.1.1.2.cmml">x</mi><mo id="alg2.l2.m1.1.1.1" xref="alg2.l2.m1.1.1.1.cmml">∈</mo><msup id="alg2.l2.m1.1.1.3" xref="alg2.l2.m1.1.1.3.cmml"><mi id="alg2.l2.m1.1.1.3.2" xref="alg2.l2.m1.1.1.3.2.cmml">ℝ</mi><mrow id="alg2.l2.m1.1.1.3.3" xref="alg2.l2.m1.1.1.3.3.cmml"><mrow id="alg2.l2.m1.1.1.3.3.2" xref="alg2.l2.m1.1.1.3.3.2.cmml"><mi id="alg2.l2.m1.1.1.3.3.2.2" xref="alg2.l2.m1.1.1.3.3.2.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg2.l2.m1.1.1.3.3.2.1" xref="alg2.l2.m1.1.1.3.3.2.1.cmml">​</mo><mi id="alg2.l2.m1.1.1.3.3.2.3" xref="alg2.l2.m1.1.1.3.3.2.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg2.l2.m1.1.1.3.3.2.1a" xref="alg2.l2.m1.1.1.3.3.2.1.cmml">​</mo><mi id="alg2.l2.m1.1.1.3.3.2.4" xref="alg2.l2.m1.1.1.3.3.2.4.cmml">z</mi></mrow><mo lspace="0.222em" rspace="0.222em" id="alg2.l2.m1.1.1.3.3.1" xref="alg2.l2.m1.1.1.3.3.1.cmml">×</mo><mi id="alg2.l2.m1.1.1.3.3.3" xref="alg2.l2.m1.1.1.3.3.3.cmml">l</mi><mo lspace="0.222em" rspace="0.222em" id="alg2.l2.m1.1.1.3.3.1a" xref="alg2.l2.m1.1.1.3.3.1.cmml">×</mo><mover accent="true" id="alg2.l2.m1.1.1.3.3.4" xref="alg2.l2.m1.1.1.3.3.4.cmml"><mi id="alg2.l2.m1.1.1.3.3.4.2" xref="alg2.l2.m1.1.1.3.3.4.2.cmml">r</mi><mo id="alg2.l2.m1.1.1.3.3.4.1" xref="alg2.l2.m1.1.1.3.3.4.1.cmml">^</mo></mover></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="alg2.l2.m1.1b"><apply id="alg2.l2.m1.1.1.cmml" xref="alg2.l2.m1.1.1"><in id="alg2.l2.m1.1.1.1.cmml" xref="alg2.l2.m1.1.1.1"></in><ci id="alg2.l2.m1.1.1.2.cmml" xref="alg2.l2.m1.1.1.2">𝑥</ci><apply id="alg2.l2.m1.1.1.3.cmml" xref="alg2.l2.m1.1.1.3"><csymbol cd="ambiguous" id="alg2.l2.m1.1.1.3.1.cmml" xref="alg2.l2.m1.1.1.3">superscript</csymbol><ci id="alg2.l2.m1.1.1.3.2.cmml" xref="alg2.l2.m1.1.1.3.2">ℝ</ci><apply id="alg2.l2.m1.1.1.3.3.cmml" xref="alg2.l2.m1.1.1.3.3"><times id="alg2.l2.m1.1.1.3.3.1.cmml" xref="alg2.l2.m1.1.1.3.3.1"></times><apply id="alg2.l2.m1.1.1.3.3.2.cmml" xref="alg2.l2.m1.1.1.3.3.2"><times id="alg2.l2.m1.1.1.3.3.2.1.cmml" xref="alg2.l2.m1.1.1.3.3.2.1"></times><ci id="alg2.l2.m1.1.1.3.3.2.2.cmml" xref="alg2.l2.m1.1.1.3.3.2.2">𝑏</ci><ci id="alg2.l2.m1.1.1.3.3.2.3.cmml" xref="alg2.l2.m1.1.1.3.3.2.3">𝑠</ci><ci id="alg2.l2.m1.1.1.3.3.2.4.cmml" xref="alg2.l2.m1.1.1.3.3.2.4">𝑧</ci></apply><ci id="alg2.l2.m1.1.1.3.3.3.cmml" xref="alg2.l2.m1.1.1.3.3.3">𝑙</ci><apply id="alg2.l2.m1.1.1.3.3.4.cmml" xref="alg2.l2.m1.1.1.3.3.4"><ci id="alg2.l2.m1.1.1.3.3.4.1.cmml" xref="alg2.l2.m1.1.1.3.3.4.1">^</ci><ci id="alg2.l2.m1.1.1.3.3.4.2.cmml" xref="alg2.l2.m1.1.1.3.3.4.2">𝑟</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l2.m1.1c">x\in\mathbb{R}^{bsz\times l\times\hat{r}}</annotation></semantics></math>: Input tensor

</div>
<div id="alg2.l3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">3:</span>     # <math id="alg2.l3.m1.1" class="ltx_Math" alttext="y\in\mathbb{R}^{bsz\times l\times d}" display="inline"><semantics id="alg2.l3.m1.1a"><mrow id="alg2.l3.m1.1.1" xref="alg2.l3.m1.1.1.cmml"><mi id="alg2.l3.m1.1.1.2" xref="alg2.l3.m1.1.1.2.cmml">y</mi><mo id="alg2.l3.m1.1.1.1" xref="alg2.l3.m1.1.1.1.cmml">∈</mo><msup id="alg2.l3.m1.1.1.3" xref="alg2.l3.m1.1.1.3.cmml"><mi id="alg2.l3.m1.1.1.3.2" xref="alg2.l3.m1.1.1.3.2.cmml">ℝ</mi><mrow id="alg2.l3.m1.1.1.3.3" xref="alg2.l3.m1.1.1.3.3.cmml"><mrow id="alg2.l3.m1.1.1.3.3.2" xref="alg2.l3.m1.1.1.3.3.2.cmml"><mi id="alg2.l3.m1.1.1.3.3.2.2" xref="alg2.l3.m1.1.1.3.3.2.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg2.l3.m1.1.1.3.3.2.1" xref="alg2.l3.m1.1.1.3.3.2.1.cmml">​</mo><mi id="alg2.l3.m1.1.1.3.3.2.3" xref="alg2.l3.m1.1.1.3.3.2.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg2.l3.m1.1.1.3.3.2.1a" xref="alg2.l3.m1.1.1.3.3.2.1.cmml">​</mo><mi id="alg2.l3.m1.1.1.3.3.2.4" xref="alg2.l3.m1.1.1.3.3.2.4.cmml">z</mi></mrow><mo lspace="0.222em" rspace="0.222em" id="alg2.l3.m1.1.1.3.3.1" xref="alg2.l3.m1.1.1.3.3.1.cmml">×</mo><mi id="alg2.l3.m1.1.1.3.3.3" xref="alg2.l3.m1.1.1.3.3.3.cmml">l</mi><mo lspace="0.222em" rspace="0.222em" id="alg2.l3.m1.1.1.3.3.1a" xref="alg2.l3.m1.1.1.3.3.1.cmml">×</mo><mi id="alg2.l3.m1.1.1.3.3.4" xref="alg2.l3.m1.1.1.3.3.4.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="alg2.l3.m1.1b"><apply id="alg2.l3.m1.1.1.cmml" xref="alg2.l3.m1.1.1"><in id="alg2.l3.m1.1.1.1.cmml" xref="alg2.l3.m1.1.1.1"></in><ci id="alg2.l3.m1.1.1.2.cmml" xref="alg2.l3.m1.1.1.2">𝑦</ci><apply id="alg2.l3.m1.1.1.3.cmml" xref="alg2.l3.m1.1.1.3"><csymbol cd="ambiguous" id="alg2.l3.m1.1.1.3.1.cmml" xref="alg2.l3.m1.1.1.3">superscript</csymbol><ci id="alg2.l3.m1.1.1.3.2.cmml" xref="alg2.l3.m1.1.1.3.2">ℝ</ci><apply id="alg2.l3.m1.1.1.3.3.cmml" xref="alg2.l3.m1.1.1.3.3"><times id="alg2.l3.m1.1.1.3.3.1.cmml" xref="alg2.l3.m1.1.1.3.3.1"></times><apply id="alg2.l3.m1.1.1.3.3.2.cmml" xref="alg2.l3.m1.1.1.3.3.2"><times id="alg2.l3.m1.1.1.3.3.2.1.cmml" xref="alg2.l3.m1.1.1.3.3.2.1"></times><ci id="alg2.l3.m1.1.1.3.3.2.2.cmml" xref="alg2.l3.m1.1.1.3.3.2.2">𝑏</ci><ci id="alg2.l3.m1.1.1.3.3.2.3.cmml" xref="alg2.l3.m1.1.1.3.3.2.3">𝑠</ci><ci id="alg2.l3.m1.1.1.3.3.2.4.cmml" xref="alg2.l3.m1.1.1.3.3.2.4">𝑧</ci></apply><ci id="alg2.l3.m1.1.1.3.3.3.cmml" xref="alg2.l3.m1.1.1.3.3.3">𝑙</ci><ci id="alg2.l3.m1.1.1.3.3.4.cmml" xref="alg2.l3.m1.1.1.3.3.4">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l3.m1.1c">y\in\mathbb{R}^{bsz\times l\times d}</annotation></semantics></math>: Output tensor

</div>
<div id="alg2.l4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">4:</span>     # <math id="alg2.l4.m1.2" class="ltx_Math" alttext="type\in\{0,1\}" display="inline"><semantics id="alg2.l4.m1.2a"><mrow id="alg2.l4.m1.2.3" xref="alg2.l4.m1.2.3.cmml"><mrow id="alg2.l4.m1.2.3.2" xref="alg2.l4.m1.2.3.2.cmml"><mi id="alg2.l4.m1.2.3.2.2" xref="alg2.l4.m1.2.3.2.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg2.l4.m1.2.3.2.1" xref="alg2.l4.m1.2.3.2.1.cmml">​</mo><mi id="alg2.l4.m1.2.3.2.3" xref="alg2.l4.m1.2.3.2.3.cmml">y</mi><mo lspace="0em" rspace="0em" id="alg2.l4.m1.2.3.2.1a" xref="alg2.l4.m1.2.3.2.1.cmml">​</mo><mi id="alg2.l4.m1.2.3.2.4" xref="alg2.l4.m1.2.3.2.4.cmml">p</mi><mo lspace="0em" rspace="0em" id="alg2.l4.m1.2.3.2.1b" xref="alg2.l4.m1.2.3.2.1.cmml">​</mo><mi id="alg2.l4.m1.2.3.2.5" xref="alg2.l4.m1.2.3.2.5.cmml">e</mi></mrow><mo id="alg2.l4.m1.2.3.1" xref="alg2.l4.m1.2.3.1.cmml">∈</mo><mrow id="alg2.l4.m1.2.3.3.2" xref="alg2.l4.m1.2.3.3.1.cmml"><mo stretchy="false" id="alg2.l4.m1.2.3.3.2.1" xref="alg2.l4.m1.2.3.3.1.cmml">{</mo><mn id="alg2.l4.m1.1.1" xref="alg2.l4.m1.1.1.cmml">0</mn><mo id="alg2.l4.m1.2.3.3.2.2" xref="alg2.l4.m1.2.3.3.1.cmml">,</mo><mn id="alg2.l4.m1.2.2" xref="alg2.l4.m1.2.2.cmml">1</mn><mo stretchy="false" id="alg2.l4.m1.2.3.3.2.3" xref="alg2.l4.m1.2.3.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l4.m1.2b"><apply id="alg2.l4.m1.2.3.cmml" xref="alg2.l4.m1.2.3"><in id="alg2.l4.m1.2.3.1.cmml" xref="alg2.l4.m1.2.3.1"></in><apply id="alg2.l4.m1.2.3.2.cmml" xref="alg2.l4.m1.2.3.2"><times id="alg2.l4.m1.2.3.2.1.cmml" xref="alg2.l4.m1.2.3.2.1"></times><ci id="alg2.l4.m1.2.3.2.2.cmml" xref="alg2.l4.m1.2.3.2.2">𝑡</ci><ci id="alg2.l4.m1.2.3.2.3.cmml" xref="alg2.l4.m1.2.3.2.3">𝑦</ci><ci id="alg2.l4.m1.2.3.2.4.cmml" xref="alg2.l4.m1.2.3.2.4">𝑝</ci><ci id="alg2.l4.m1.2.3.2.5.cmml" xref="alg2.l4.m1.2.3.2.5">𝑒</ci></apply><set id="alg2.l4.m1.2.3.3.1.cmml" xref="alg2.l4.m1.2.3.3.2"><cn type="integer" id="alg2.l4.m1.1.1.cmml" xref="alg2.l4.m1.1.1">0</cn><cn type="integer" id="alg2.l4.m1.2.2.cmml" xref="alg2.l4.m1.2.2">1</cn></set></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l4.m1.2c">type\in\{0,1\}</annotation></semantics></math>: Group type 0 or 1

</div>
<div id="alg2.l5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">5:</span>     <span id="alg2.l5.1" class="ltx_text ltx_font_bold">if</span> <math id="alg2.l5.m1.1" class="ltx_Math" alttext="type=0" display="inline"><semantics id="alg2.l5.m1.1a"><mrow id="alg2.l5.m1.1.1" xref="alg2.l5.m1.1.1.cmml"><mrow id="alg2.l5.m1.1.1.2" xref="alg2.l5.m1.1.1.2.cmml"><mi id="alg2.l5.m1.1.1.2.2" xref="alg2.l5.m1.1.1.2.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg2.l5.m1.1.1.2.1" xref="alg2.l5.m1.1.1.2.1.cmml">​</mo><mi id="alg2.l5.m1.1.1.2.3" xref="alg2.l5.m1.1.1.2.3.cmml">y</mi><mo lspace="0em" rspace="0em" id="alg2.l5.m1.1.1.2.1a" xref="alg2.l5.m1.1.1.2.1.cmml">​</mo><mi id="alg2.l5.m1.1.1.2.4" xref="alg2.l5.m1.1.1.2.4.cmml">p</mi><mo lspace="0em" rspace="0em" id="alg2.l5.m1.1.1.2.1b" xref="alg2.l5.m1.1.1.2.1.cmml">​</mo><mi id="alg2.l5.m1.1.1.2.5" xref="alg2.l5.m1.1.1.2.5.cmml">e</mi></mrow><mo id="alg2.l5.m1.1.1.1" xref="alg2.l5.m1.1.1.1.cmml">=</mo><mn id="alg2.l5.m1.1.1.3" xref="alg2.l5.m1.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="alg2.l5.m1.1b"><apply id="alg2.l5.m1.1.1.cmml" xref="alg2.l5.m1.1.1"><eq id="alg2.l5.m1.1.1.1.cmml" xref="alg2.l5.m1.1.1.1"></eq><apply id="alg2.l5.m1.1.1.2.cmml" xref="alg2.l5.m1.1.1.2"><times id="alg2.l5.m1.1.1.2.1.cmml" xref="alg2.l5.m1.1.1.2.1"></times><ci id="alg2.l5.m1.1.1.2.2.cmml" xref="alg2.l5.m1.1.1.2.2">𝑡</ci><ci id="alg2.l5.m1.1.1.2.3.cmml" xref="alg2.l5.m1.1.1.2.3">𝑦</ci><ci id="alg2.l5.m1.1.1.2.4.cmml" xref="alg2.l5.m1.1.1.2.4">𝑝</ci><ci id="alg2.l5.m1.1.1.2.5.cmml" xref="alg2.l5.m1.1.1.2.5">𝑒</ci></apply><cn type="integer" id="alg2.l5.m1.1.1.3.cmml" xref="alg2.l5.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l5.m1.1c">type=0</annotation></semantics></math> <span id="alg2.l5.2" class="ltx_text ltx_font_bold">then</span>

</div>
<div id="alg2.l6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">6:</span>         <math id="alg2.l6.m1.1" class="ltx_Math" alttext="y" display="inline"><semantics id="alg2.l6.m1.1a"><mi id="alg2.l6.m1.1.1" xref="alg2.l6.m1.1.1.cmml">y</mi><annotation-xml encoding="MathML-Content" id="alg2.l6.m1.1b"><ci id="alg2.l6.m1.1.1.cmml" xref="alg2.l6.m1.1.1">𝑦</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l6.m1.1c">y</annotation></semantics></math> = repeat(x, <math id="alg2.l6.m2.1" class="ltx_Math" alttext="d/\hat{r}" display="inline"><semantics id="alg2.l6.m2.1a"><mrow id="alg2.l6.m2.1.1" xref="alg2.l6.m2.1.1.cmml"><mi id="alg2.l6.m2.1.1.2" xref="alg2.l6.m2.1.1.2.cmml">d</mi><mo id="alg2.l6.m2.1.1.1" xref="alg2.l6.m2.1.1.1.cmml">/</mo><mover accent="true" id="alg2.l6.m2.1.1.3" xref="alg2.l6.m2.1.1.3.cmml"><mi id="alg2.l6.m2.1.1.3.2" xref="alg2.l6.m2.1.1.3.2.cmml">r</mi><mo id="alg2.l6.m2.1.1.3.1" xref="alg2.l6.m2.1.1.3.1.cmml">^</mo></mover></mrow><annotation-xml encoding="MathML-Content" id="alg2.l6.m2.1b"><apply id="alg2.l6.m2.1.1.cmml" xref="alg2.l6.m2.1.1"><divide id="alg2.l6.m2.1.1.1.cmml" xref="alg2.l6.m2.1.1.1"></divide><ci id="alg2.l6.m2.1.1.2.cmml" xref="alg2.l6.m2.1.1.2">𝑑</ci><apply id="alg2.l6.m2.1.1.3.cmml" xref="alg2.l6.m2.1.1.3"><ci id="alg2.l6.m2.1.1.3.1.cmml" xref="alg2.l6.m2.1.1.3.1">^</ci><ci id="alg2.l6.m2.1.1.3.2.cmml" xref="alg2.l6.m2.1.1.3.2">𝑟</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l6.m2.1c">d/\hat{r}</annotation></semantics></math>, dim=<math id="alg2.l6.m3.1" class="ltx_Math" alttext="2" display="inline"><semantics id="alg2.l6.m3.1a"><mn id="alg2.l6.m3.1.1" xref="alg2.l6.m3.1.1.cmml">2</mn><annotation-xml encoding="MathML-Content" id="alg2.l6.m3.1b"><cn type="integer" id="alg2.l6.m3.1.1.cmml" xref="alg2.l6.m3.1.1">2</cn></annotation-xml><annotation encoding="application/x-tex" id="alg2.l6.m3.1c">2</annotation></semantics></math>) # first type of group

</div>
<div id="alg2.l7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">7:</span>     <span id="alg2.l7.1" class="ltx_text ltx_font_bold">else</span>
</div>
<div id="alg2.l8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">8:</span>         <math id="alg2.l8.m1.1" class="ltx_Math" alttext="y" display="inline"><semantics id="alg2.l8.m1.1a"><mi id="alg2.l8.m1.1.1" xref="alg2.l8.m1.1.1.cmml">y</mi><annotation-xml encoding="MathML-Content" id="alg2.l8.m1.1b"><ci id="alg2.l8.m1.1.1.cmml" xref="alg2.l8.m1.1.1">𝑦</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l8.m1.1c">y</annotation></semantics></math> = repeat-interleave(x, <math id="alg2.l8.m2.1" class="ltx_Math" alttext="d/\hat{r}" display="inline"><semantics id="alg2.l8.m2.1a"><mrow id="alg2.l8.m2.1.1" xref="alg2.l8.m2.1.1.cmml"><mi id="alg2.l8.m2.1.1.2" xref="alg2.l8.m2.1.1.2.cmml">d</mi><mo id="alg2.l8.m2.1.1.1" xref="alg2.l8.m2.1.1.1.cmml">/</mo><mover accent="true" id="alg2.l8.m2.1.1.3" xref="alg2.l8.m2.1.1.3.cmml"><mi id="alg2.l8.m2.1.1.3.2" xref="alg2.l8.m2.1.1.3.2.cmml">r</mi><mo id="alg2.l8.m2.1.1.3.1" xref="alg2.l8.m2.1.1.3.1.cmml">^</mo></mover></mrow><annotation-xml encoding="MathML-Content" id="alg2.l8.m2.1b"><apply id="alg2.l8.m2.1.1.cmml" xref="alg2.l8.m2.1.1"><divide id="alg2.l8.m2.1.1.1.cmml" xref="alg2.l8.m2.1.1.1"></divide><ci id="alg2.l8.m2.1.1.2.cmml" xref="alg2.l8.m2.1.1.2">𝑑</ci><apply id="alg2.l8.m2.1.1.3.cmml" xref="alg2.l8.m2.1.1.3"><ci id="alg2.l8.m2.1.1.3.1.cmml" xref="alg2.l8.m2.1.1.3.1">^</ci><ci id="alg2.l8.m2.1.1.3.2.cmml" xref="alg2.l8.m2.1.1.3.2">𝑟</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l8.m2.1c">d/\hat{r}</annotation></semantics></math>, dim=<math id="alg2.l8.m3.1" class="ltx_Math" alttext="2" display="inline"><semantics id="alg2.l8.m3.1a"><mn id="alg2.l8.m3.1.1" xref="alg2.l8.m3.1.1.cmml">2</mn><annotation-xml encoding="MathML-Content" id="alg2.l8.m3.1b"><cn type="integer" id="alg2.l8.m3.1.1.cmml" xref="alg2.l8.m3.1.1">2</cn></annotation-xml><annotation encoding="application/x-tex" id="alg2.l8.m3.1c">2</annotation></semantics></math>) # second type of group

</div>
<div id="alg2.l9" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">9:</span>     <span id="alg2.l9.1" class="ltx_text ltx_font_bold">end</span> <span id="alg2.l9.2" class="ltx_text ltx_font_bold">if</span>
</div>
<div id="alg2.l10" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">10:</span>     truncate <math id="alg2.l10.m1.1" class="ltx_Math" alttext="y" display="inline"><semantics id="alg2.l10.m1.1a"><mi id="alg2.l10.m1.1.1" xref="alg2.l10.m1.1.1.cmml">y</mi><annotation-xml encoding="MathML-Content" id="alg2.l10.m1.1b"><ci id="alg2.l10.m1.1.1.cmml" xref="alg2.l10.m1.1.1">𝑦</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l10.m1.1c">y</annotation></semantics></math> to <math id="alg2.l10.m2.1" class="ltx_Math" alttext="\mathbb{R}^{bsz\times l\times d}" display="inline"><semantics id="alg2.l10.m2.1a"><msup id="alg2.l10.m2.1.1" xref="alg2.l10.m2.1.1.cmml"><mi id="alg2.l10.m2.1.1.2" xref="alg2.l10.m2.1.1.2.cmml">ℝ</mi><mrow id="alg2.l10.m2.1.1.3" xref="alg2.l10.m2.1.1.3.cmml"><mrow id="alg2.l10.m2.1.1.3.2" xref="alg2.l10.m2.1.1.3.2.cmml"><mi id="alg2.l10.m2.1.1.3.2.2" xref="alg2.l10.m2.1.1.3.2.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg2.l10.m2.1.1.3.2.1" xref="alg2.l10.m2.1.1.3.2.1.cmml">​</mo><mi id="alg2.l10.m2.1.1.3.2.3" xref="alg2.l10.m2.1.1.3.2.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg2.l10.m2.1.1.3.2.1a" xref="alg2.l10.m2.1.1.3.2.1.cmml">​</mo><mi id="alg2.l10.m2.1.1.3.2.4" xref="alg2.l10.m2.1.1.3.2.4.cmml">z</mi></mrow><mo lspace="0.222em" rspace="0.222em" id="alg2.l10.m2.1.1.3.1" xref="alg2.l10.m2.1.1.3.1.cmml">×</mo><mi id="alg2.l10.m2.1.1.3.3" xref="alg2.l10.m2.1.1.3.3.cmml">l</mi><mo lspace="0.222em" rspace="0.222em" id="alg2.l10.m2.1.1.3.1a" xref="alg2.l10.m2.1.1.3.1.cmml">×</mo><mi id="alg2.l10.m2.1.1.3.4" xref="alg2.l10.m2.1.1.3.4.cmml">d</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="alg2.l10.m2.1b"><apply id="alg2.l10.m2.1.1.cmml" xref="alg2.l10.m2.1.1"><csymbol cd="ambiguous" id="alg2.l10.m2.1.1.1.cmml" xref="alg2.l10.m2.1.1">superscript</csymbol><ci id="alg2.l10.m2.1.1.2.cmml" xref="alg2.l10.m2.1.1.2">ℝ</ci><apply id="alg2.l10.m2.1.1.3.cmml" xref="alg2.l10.m2.1.1.3"><times id="alg2.l10.m2.1.1.3.1.cmml" xref="alg2.l10.m2.1.1.3.1"></times><apply id="alg2.l10.m2.1.1.3.2.cmml" xref="alg2.l10.m2.1.1.3.2"><times id="alg2.l10.m2.1.1.3.2.1.cmml" xref="alg2.l10.m2.1.1.3.2.1"></times><ci id="alg2.l10.m2.1.1.3.2.2.cmml" xref="alg2.l10.m2.1.1.3.2.2">𝑏</ci><ci id="alg2.l10.m2.1.1.3.2.3.cmml" xref="alg2.l10.m2.1.1.3.2.3">𝑠</ci><ci id="alg2.l10.m2.1.1.3.2.4.cmml" xref="alg2.l10.m2.1.1.3.2.4">𝑧</ci></apply><ci id="alg2.l10.m2.1.1.3.3.cmml" xref="alg2.l10.m2.1.1.3.3">𝑙</ci><ci id="alg2.l10.m2.1.1.3.4.cmml" xref="alg2.l10.m2.1.1.3.4">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l10.m2.1c">\mathbb{R}^{bsz\times l\times d}</annotation></semantics></math>

</div>
<div id="alg2.l11" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">11:</span>     <span id="alg2.l11.1" class="ltx_text ltx_font_bold">return</span> <math id="alg2.l11.m1.1" class="ltx_Math" alttext="y" display="inline"><semantics id="alg2.l11.m1.1a"><mi id="alg2.l11.m1.1.1" xref="alg2.l11.m1.1.1.cmml">y</mi><annotation-xml encoding="MathML-Content" id="alg2.l11.m1.1b"><ci id="alg2.l11.m1.1.1.cmml" xref="alg2.l11.m1.1.1">𝑦</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l11.m1.1c">y</annotation></semantics></math>

</div>
<div id="alg2.l12" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">12:</span><span id="alg2.l12.1" class="ltx_text ltx_font_bold">end</span> <span id="alg2.l12.2" class="ltx_text ltx_font_bold">function</span>
</div>
</div>
</figure>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Downstream Tasks of Continual Pretraining</h2>

<div id="A3.p1" class="ltx_para">
<p id="A3.p1.1" class="ltx_p">For biomedcine, we use PubMedQA <cite class="ltx_cite ltx_citemacro_cite">Jin et al. (<a href="#bib.bib19" title="" class="ltx_ref">2019</a>)</cite>, RCT <cite class="ltx_cite ltx_citemacro_cite">Dernoncourt and Lee (<a href="#bib.bib6" title="" class="ltx_ref">2017</a>)</cite>, USMLE <cite class="ltx_cite ltx_citemacro_cite">Jin et al. (<a href="#bib.bib18" title="" class="ltx_ref">2021</a>)</cite>, and selecting biomedicine subjects from MMLU to evaluate the performance.
For finance, following BloombergGPT <cite class="ltx_cite ltx_citemacro_cite">Wu et al. (<a href="#bib.bib37" title="" class="ltx_ref">2023</a>)</cite>,we use ConvFinQA <cite class="ltx_cite ltx_citemacro_cite">Chen et al. (<a href="#bib.bib2" title="" class="ltx_ref">2022</a>)</cite>, NER <cite class="ltx_cite ltx_citemacro_cite">Salinas Alvarado et al. (<a href="#bib.bib30" title="" class="ltx_ref">2015</a>)</cite>, Headline <cite class="ltx_cite ltx_citemacro_cite">Sinha and Khandait (<a href="#bib.bib33" title="" class="ltx_ref">2021</a>)</cite>, FiQA SA <cite class="ltx_cite ltx_citemacro_cite">Maia et al. (<a href="#bib.bib25" title="" class="ltx_ref">2018</a>)</cite> and FPB <cite class="ltx_cite ltx_citemacro_cite">Malo et al. (<a href="#bib.bib26" title="" class="ltx_ref">2014</a>)</cite>.
We report the detail performance of these tasks following:</p>
</div>
<figure id="A3.T6" class="ltx_table">
<table id="A3.T6.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A3.T6.1.1" class="ltx_tr">
<td id="A3.T6.1.1.2" class="ltx_td ltx_border_tt"></td>
<th id="A3.T6.1.1.1" class="ltx_td ltx_align_justify ltx_th ltx_th_column ltx_border_tt">
<span id="A3.T6.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.1.1.1.1" class="ltx_p"><math id="A3.T6.1.1.1.1.1.m1.1" class="ltx_Math" alttext="r" display="inline"><semantics id="A3.T6.1.1.1.1.1.m1.1a"><mi id="A3.T6.1.1.1.1.1.m1.1.1" xref="A3.T6.1.1.1.1.1.m1.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="A3.T6.1.1.1.1.1.m1.1b"><ci id="A3.T6.1.1.1.1.1.m1.1.1.cmml" xref="A3.T6.1.1.1.1.1.m1.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.T6.1.1.1.1.1.m1.1c">r</annotation></semantics></math></span>
</span>
</th>
<th id="A3.T6.1.1.3" class="ltx_td ltx_align_justify ltx_th ltx_th_column ltx_border_tt">
<span id="A3.T6.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.1.3.1.1" class="ltx_p"><span id="A3.T6.1.1.3.1.1.1" class="ltx_text ltx_font_bold">PubMedQA</span></span>
</span>
</th>
<th id="A3.T6.1.1.4" class="ltx_td ltx_align_justify ltx_th ltx_th_column ltx_border_tt">
<span id="A3.T6.1.1.4.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.1.4.1.1" class="ltx_p"><span id="A3.T6.1.1.4.1.1.1" class="ltx_text ltx_font_bold">USMLE</span></span>
</span>
</th>
<th id="A3.T6.1.1.5" class="ltx_td ltx_align_justify ltx_th ltx_th_column ltx_border_tt">
<span id="A3.T6.1.1.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.1.5.1.1" class="ltx_p"><span id="A3.T6.1.1.5.1.1.1" class="ltx_text ltx_font_bold">BioMMLU</span></span>
</span>
</th>
<th id="A3.T6.1.1.6" class="ltx_td ltx_align_justify ltx_th ltx_th_column ltx_border_tt">
<span id="A3.T6.1.1.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.1.6.1.1" class="ltx_p"><span id="A3.T6.1.1.6.1.1.1" class="ltx_text ltx_font_bold">RCT</span></span>
</span>
</th>
<th id="A3.T6.1.1.7" class="ltx_td ltx_align_justify ltx_th ltx_th_column ltx_border_tt">
<span id="A3.T6.1.1.7.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.1.7.1.1" class="ltx_p"><span id="A3.T6.1.1.7.1.1.1" class="ltx_text ltx_font_bold">Avg.</span></span>
</span>
</th>
</tr>
<tr id="A3.T6.1.2.1" class="ltx_tr">
<td id="A3.T6.1.2.1.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A3.T6.1.2.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.2.1.1.1.1" class="ltx_p">FFT</span>
</span>
</td>
<td id="A3.T6.1.2.1.2" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A3.T6.1.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.2.1.2.1.1" class="ltx_p">-</span>
</span>
</td>
<td id="A3.T6.1.2.1.3" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A3.T6.1.2.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.2.1.3.1.1" class="ltx_p">74.1</span>
</span>
</td>
<td id="A3.T6.1.2.1.4" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A3.T6.1.2.1.4.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.2.1.4.1.1" class="ltx_p">41.2</span>
</span>
</td>
<td id="A3.T6.1.2.1.5" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A3.T6.1.2.1.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.2.1.5.1.1" class="ltx_p">47.5</span>
</span>
</td>
<td id="A3.T6.1.2.1.6" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A3.T6.1.2.1.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.2.1.6.1.1" class="ltx_p">62.7</span>
</span>
</td>
<td id="A3.T6.1.2.1.7" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A3.T6.1.2.1.7.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.2.1.7.1.1" class="ltx_p">56.4</span>
</span>
</td>
</tr>
<tr id="A3.T6.1.3.2" class="ltx_tr">
<td id="A3.T6.1.3.2.1" class="ltx_td ltx_align_justify">
<span id="A3.T6.1.3.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.3.2.1.1.1" class="ltx_p">LoRA</span>
</span>
</td>
<td id="A3.T6.1.3.2.2" class="ltx_td ltx_align_justify">
<span id="A3.T6.1.3.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.3.2.2.1.1" class="ltx_p">8</span>
</span>
</td>
<td id="A3.T6.1.3.2.3" class="ltx_td ltx_align_justify">
<span id="A3.T6.1.3.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.3.2.3.1.1" class="ltx_p">73.1</span>
</span>
</td>
<td id="A3.T6.1.3.2.4" class="ltx_td ltx_align_justify">
<span id="A3.T6.1.3.2.4.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.3.2.4.1.1" class="ltx_p">34.9</span>
</span>
</td>
<td id="A3.T6.1.3.2.5" class="ltx_td ltx_align_justify">
<span id="A3.T6.1.3.2.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.3.2.5.1.1" class="ltx_p">45.3</span>
</span>
</td>
<td id="A3.T6.1.3.2.6" class="ltx_td ltx_align_justify">
<span id="A3.T6.1.3.2.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.3.2.6.1.1" class="ltx_p">54.9</span>
</span>
</td>
<td id="A3.T6.1.3.2.7" class="ltx_td ltx_align_justify">
<span id="A3.T6.1.3.2.7.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.3.2.7.1.1" class="ltx_p">51.9</span>
</span>
</td>
</tr>
<tr id="A3.T6.1.4.3" class="ltx_tr">
<td id="A3.T6.1.4.3.1" class="ltx_td ltx_align_justify">
<span id="A3.T6.1.4.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.4.3.1.1.1" class="ltx_p">MoRA</span>
</span>
</td>
<td id="A3.T6.1.4.3.2" class="ltx_td ltx_align_justify">
<span id="A3.T6.1.4.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.4.3.2.1.1" class="ltx_p">8</span>
</span>
</td>
<td id="A3.T6.1.4.3.3" class="ltx_td ltx_align_justify">
<span id="A3.T6.1.4.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.4.3.3.1.1" class="ltx_p">73.3</span>
</span>
</td>
<td id="A3.T6.1.4.3.4" class="ltx_td ltx_align_justify">
<span id="A3.T6.1.4.3.4.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.4.3.4.1.1" class="ltx_p">34.7</span>
</span>
</td>
<td id="A3.T6.1.4.3.5" class="ltx_td ltx_align_justify">
<span id="A3.T6.1.4.3.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.4.3.5.1.1" class="ltx_p">45.3</span>
</span>
</td>
<td id="A3.T6.1.4.3.6" class="ltx_td ltx_align_justify">
<span id="A3.T6.1.4.3.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.4.3.6.1.1" class="ltx_p">59.9</span>
</span>
</td>
<td id="A3.T6.1.4.3.7" class="ltx_td ltx_align_justify">
<span id="A3.T6.1.4.3.7.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.4.3.7.1.1" class="ltx_p">53.3</span>
</span>
</td>
</tr>
<tr id="A3.T6.1.5.4" class="ltx_tr">
<td id="A3.T6.1.5.4.1" class="ltx_td ltx_align_justify">
<span id="A3.T6.1.5.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.5.4.1.1.1" class="ltx_p">LoRA</span>
</span>
</td>
<td id="A3.T6.1.5.4.2" class="ltx_td ltx_align_justify">
<span id="A3.T6.1.5.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.5.4.2.1.1" class="ltx_p">256</span>
</span>
</td>
<td id="A3.T6.1.5.4.3" class="ltx_td ltx_align_justify">
<span id="A3.T6.1.5.4.3.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.5.4.3.1.1" class="ltx_p">73.8</span>
</span>
</td>
<td id="A3.T6.1.5.4.4" class="ltx_td ltx_align_justify">
<span id="A3.T6.1.5.4.4.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.5.4.4.1.1" class="ltx_p">39.7</span>
</span>
</td>
<td id="A3.T6.1.5.4.5" class="ltx_td ltx_align_justify">
<span id="A3.T6.1.5.4.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.5.4.5.1.1" class="ltx_p">46.0</span>
</span>
</td>
<td id="A3.T6.1.5.4.6" class="ltx_td ltx_align_justify">
<span id="A3.T6.1.5.4.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.5.4.6.1.1" class="ltx_p">56.9</span>
</span>
</td>
<td id="A3.T6.1.5.4.7" class="ltx_td ltx_align_justify">
<span id="A3.T6.1.5.4.7.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.5.4.7.1.1" class="ltx_p">54.1</span>
</span>
</td>
</tr>
<tr id="A3.T6.1.6.5" class="ltx_tr">
<td id="A3.T6.1.6.5.1" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A3.T6.1.6.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.6.5.1.1.1" class="ltx_p">MoRA</span>
</span>
</td>
<td id="A3.T6.1.6.5.2" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A3.T6.1.6.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.6.5.2.1.1" class="ltx_p">256</span>
</span>
</td>
<td id="A3.T6.1.6.5.3" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A3.T6.1.6.5.3.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.6.5.3.1.1" class="ltx_p">74.4</span>
</span>
</td>
<td id="A3.T6.1.6.5.4" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A3.T6.1.6.5.4.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.6.5.4.1.1" class="ltx_p">40.4</span>
</span>
</td>
<td id="A3.T6.1.6.5.5" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A3.T6.1.6.5.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.6.5.5.1.1" class="ltx_p">46.1</span>
</span>
</td>
<td id="A3.T6.1.6.5.6" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A3.T6.1.6.5.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.6.5.6.1.1" class="ltx_p">60.6</span>
</span>
</td>
<td id="A3.T6.1.6.5.7" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A3.T6.1.6.5.7.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T6.1.6.5.7.1.1" class="ltx_p">55.4</span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 6: </span>Performance on biomedical tasks.</figcaption>
</figure>
<figure id="A3.T7" class="ltx_table">
<table id="A3.T7.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A3.T7.1.1" class="ltx_tr">
<td id="A3.T7.1.1.2" class="ltx_td ltx_border_tt"></td>
<th id="A3.T7.1.1.1" class="ltx_td ltx_align_justify ltx_th ltx_th_column ltx_border_tt">
<span id="A3.T7.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.1.1.1.1" class="ltx_p"><math id="A3.T7.1.1.1.1.1.m1.1" class="ltx_Math" alttext="r" display="inline"><semantics id="A3.T7.1.1.1.1.1.m1.1a"><mi id="A3.T7.1.1.1.1.1.m1.1.1" xref="A3.T7.1.1.1.1.1.m1.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="A3.T7.1.1.1.1.1.m1.1b"><ci id="A3.T7.1.1.1.1.1.m1.1.1.cmml" xref="A3.T7.1.1.1.1.1.m1.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.T7.1.1.1.1.1.m1.1c">r</annotation></semantics></math></span>
</span>
</th>
<th id="A3.T7.1.1.3" class="ltx_td ltx_align_justify ltx_th ltx_th_column ltx_border_tt">
<span id="A3.T7.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.1.3.1.1" class="ltx_p"><span id="A3.T7.1.1.3.1.1.1" class="ltx_text ltx_font_bold">ConvFinQA</span></span>
</span>
</th>
<th id="A3.T7.1.1.4" class="ltx_td ltx_align_justify ltx_th ltx_th_column ltx_border_tt">
<span id="A3.T7.1.1.4.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.1.4.1.1" class="ltx_p"><span id="A3.T7.1.1.4.1.1.1" class="ltx_text ltx_font_bold">FiQA SA</span></span>
</span>
</th>
<th id="A3.T7.1.1.5" class="ltx_td ltx_align_justify ltx_th ltx_th_column ltx_border_tt">
<span id="A3.T7.1.1.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.1.5.1.1" class="ltx_p"><span id="A3.T7.1.1.5.1.1.1" class="ltx_text ltx_font_bold">Headline</span></span>
</span>
</th>
<th id="A3.T7.1.1.6" class="ltx_td ltx_align_justify ltx_th ltx_th_column ltx_border_tt">
<span id="A3.T7.1.1.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.1.6.1.1" class="ltx_p"><span id="A3.T7.1.1.6.1.1.1" class="ltx_text ltx_font_bold">NER</span></span>
</span>
</th>
<th id="A3.T7.1.1.7" class="ltx_td ltx_align_justify ltx_th ltx_th_column ltx_border_tt">
<span id="A3.T7.1.1.7.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.1.7.1.1" class="ltx_p"><span id="A3.T7.1.1.7.1.1.1" class="ltx_text ltx_font_bold">FPB</span></span>
</span>
</th>
<th id="A3.T7.1.1.8" class="ltx_td ltx_align_justify ltx_th ltx_th_column ltx_border_tt">
<span id="A3.T7.1.1.8.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.1.8.1.1" class="ltx_p"><span id="A3.T7.1.1.8.1.1.1" class="ltx_text ltx_font_bold">Avg.</span></span>
</span>
</th>
</tr>
<tr id="A3.T7.1.2.1" class="ltx_tr">
<td id="A3.T7.1.2.1.1" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A3.T7.1.2.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.2.1.1.1.1" class="ltx_p">FFT</span>
</span>
</td>
<td id="A3.T7.1.2.1.2" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A3.T7.1.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.2.1.2.1.1" class="ltx_p">-</span>
</span>
</td>
<td id="A3.T7.1.2.1.3" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A3.T7.1.2.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.2.1.3.1.1" class="ltx_p">44.4</span>
</span>
</td>
<td id="A3.T7.1.2.1.4" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A3.T7.1.2.1.4.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.2.1.4.1.1" class="ltx_p">78.8</span>
</span>
</td>
<td id="A3.T7.1.2.1.5" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A3.T7.1.2.1.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.2.1.5.1.1" class="ltx_p">82.3</span>
</span>
</td>
<td id="A3.T7.1.2.1.6" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A3.T7.1.2.1.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.2.1.6.1.1" class="ltx_p">68.1</span>
</span>
</td>
<td id="A3.T7.1.2.1.7" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A3.T7.1.2.1.7.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.2.1.7.1.1" class="ltx_p">74.3</span>
</span>
</td>
<td id="A3.T7.1.2.1.8" class="ltx_td ltx_align_justify ltx_border_t">
<span id="A3.T7.1.2.1.8.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.2.1.8.1.1" class="ltx_p">69.6</span>
</span>
</td>
</tr>
<tr id="A3.T7.1.3.2" class="ltx_tr">
<td id="A3.T7.1.3.2.1" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.3.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.3.2.1.1.1" class="ltx_p">LoRA</span>
</span>
</td>
<td id="A3.T7.1.3.2.2" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.3.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.3.2.2.1.1" class="ltx_p">8</span>
</span>
</td>
<td id="A3.T7.1.3.2.3" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.3.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.3.2.3.1.1" class="ltx_p">44.5</span>
</span>
</td>
<td id="A3.T7.1.3.2.4" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.3.2.4.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.3.2.4.1.1" class="ltx_p">76.2</span>
</span>
</td>
<td id="A3.T7.1.3.2.5" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.3.2.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.3.2.5.1.1" class="ltx_p">72.4</span>
</span>
</td>
<td id="A3.T7.1.3.2.6" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.3.2.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.3.2.6.1.1" class="ltx_p">61.6</span>
</span>
</td>
<td id="A3.T7.1.3.2.7" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.3.2.7.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.3.2.7.1.1" class="ltx_p">65.1</span>
</span>
</td>
<td id="A3.T7.1.3.2.8" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.3.2.8.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.3.2.8.1.1" class="ltx_p">64.0</span>
</span>
</td>
</tr>
<tr id="A3.T7.1.4.3" class="ltx_tr">
<td id="A3.T7.1.4.3.1" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.4.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.4.3.1.1.1" class="ltx_p">MoRA</span>
</span>
</td>
<td id="A3.T7.1.4.3.2" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.4.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.4.3.2.1.1" class="ltx_p">8</span>
</span>
</td>
<td id="A3.T7.1.4.3.3" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.4.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.4.3.3.1.1" class="ltx_p">45.8</span>
</span>
</td>
<td id="A3.T7.1.4.3.4" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.4.3.4.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.4.3.4.1.1" class="ltx_p">76.6</span>
</span>
</td>
<td id="A3.T7.1.4.3.5" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.4.3.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.4.3.5.1.1" class="ltx_p">76.3</span>
</span>
</td>
<td id="A3.T7.1.4.3.6" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.4.3.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.4.3.6.1.1" class="ltx_p">68.9</span>
</span>
</td>
<td id="A3.T7.1.4.3.7" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.4.3.7.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.4.3.7.1.1" class="ltx_p">68.2</span>
</span>
</td>
<td id="A3.T7.1.4.3.8" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.4.3.8.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.4.3.8.1.1" class="ltx_p">67.1</span>
</span>
</td>
</tr>
<tr id="A3.T7.1.5.4" class="ltx_tr">
<td id="A3.T7.1.5.4.1" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.5.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.5.4.1.1.1" class="ltx_p">LoRA</span>
</span>
</td>
<td id="A3.T7.1.5.4.2" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.5.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.5.4.2.1.1" class="ltx_p">256</span>
</span>
</td>
<td id="A3.T7.1.5.4.3" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.5.4.3.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.5.4.3.1.1" class="ltx_p">41.4</span>
</span>
</td>
<td id="A3.T7.1.5.4.4" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.5.4.4.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.5.4.4.1.1" class="ltx_p">78.3</span>
</span>
</td>
<td id="A3.T7.1.5.4.5" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.5.4.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.5.4.5.1.1" class="ltx_p">83.0</span>
</span>
</td>
<td id="A3.T7.1.5.4.6" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.5.4.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.5.4.6.1.1" class="ltx_p">66.8</span>
</span>
</td>
<td id="A3.T7.1.5.4.7" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.5.4.7.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.5.4.7.1.1" class="ltx_p">66.7</span>
</span>
</td>
<td id="A3.T7.1.5.4.8" class="ltx_td ltx_align_justify">
<span id="A3.T7.1.5.4.8.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.5.4.8.1.1" class="ltx_p">67.3</span>
</span>
</td>
</tr>
<tr id="A3.T7.1.6.5" class="ltx_tr">
<td id="A3.T7.1.6.5.1" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A3.T7.1.6.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.6.5.1.1.1" class="ltx_p">MoRA</span>
</span>
</td>
<td id="A3.T7.1.6.5.2" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A3.T7.1.6.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.6.5.2.1.1" class="ltx_p">256</span>
</span>
</td>
<td id="A3.T7.1.6.5.3" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A3.T7.1.6.5.3.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.6.5.3.1.1" class="ltx_p">47.7</span>
</span>
</td>
<td id="A3.T7.1.6.5.4" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A3.T7.1.6.5.4.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.6.5.4.1.1" class="ltx_p">76.3</span>
</span>
</td>
<td id="A3.T7.1.6.5.5" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A3.T7.1.6.5.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.6.5.5.1.1" class="ltx_p">83.4</span>
</span>
</td>
<td id="A3.T7.1.6.5.6" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A3.T7.1.6.5.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.6.5.6.1.1" class="ltx_p">68.0</span>
</span>
</td>
<td id="A3.T7.1.6.5.7" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A3.T7.1.6.5.7.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.6.5.7.1.1" class="ltx_p">68.1</span>
</span>
</td>
<td id="A3.T7.1.6.5.8" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="A3.T7.1.6.5.8.1" class="ltx_inline-block ltx_align_top">
<span id="A3.T7.1.6.5.8.1.1" class="ltx_p">68.7</span>
</span>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 7: </span>Performance on finicial tasks.</figcaption>
</figure>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2405.12129" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2405.12130" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2405.12130">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2405.12130" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2405.12131" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Wed Jun  5 17:15:59 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2405.12130
Low-rank adaptation (LoRA)  is a popular parameter-efficient fine-tuning (PEFT)  method for large language models (LLMs) .
In this paper, we analyze the impact of low-rank updating, as implemented in LoRA. Our findings suggest that the low-rank updating mechanism may limit the ability of LLMs to effectively learn and memorize new knowledge.
Inspired by this observation, we propose a new method called MoRA, which employs a square matrix to achieve high-rank updating while maintaining the same number of trainable parameters.
To achieve it,
we introduce the corresponding non-parameter operators to reduce the input dimension and increase the output dimension for the square matrix.
Furthermore, these operators ensure that the weight can be merged back into LLMs, which makes our method can be deployed like LoRA.
We perform a comprehensive evaluation of our method across five tasks: instruction tuning, mathematical reasoning, continual pretraining, memory and pretraining. Our method outperforms LoRA on memory-intensive tasks and achieves comparable performance on other tasks.
Our code will be available athttps://github.com/kongds/MoRA.
abstract
MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
Abstract
2405.12130
1IntroductionAs the size of language models increases, parameter-efficient fine-tuning (PEFT) has emerged as a popular technique to adapt these models to specific downstream tasks.
Compared to Full Fine-Tuning (FFT) , which updates all model parameters, PEFT modifies only a small part of the parameters. For example, it can achieve similar performance with FFT by updating less than 1% of the parameters in some tasks, which significantly reduces the memory requirements for the optimizer and facilitates the storage and deployment of fine-tuned models.(a) LoRA () (b) MoRA () Figure 1:An overview of our method compared to LoRA undersamenumber of trainable parameters.is the frozen weight from model.andare trainable low-rank matrices in LoRA.is the trainable matrix in our method.
Gray parts are non-parameter operators to reducing the input dimension and increasing the output dimension.rrepresents the rank in two methods.Among the existing PEFT methods, Low-Rank Adaptation (LoRA) is particularly prevalent for LLMs. LoRA enhances performance over other PEFT methods such as prompt tuningor adaptersby updating parameters via low-rank matrices. These matrices can be merged into the original model parameters, thereby avoiding additional computational costs during inference. There are numerous methods that aim to improve LoRA for LLMs. However,
most methods primarily validate their efficiency based on GLUE, either by achieving better performance or by requiring fewer trainable parameters.
Recent methodsleverage instruction tuning task such as Alpacaor reasoning tasks like GSM8Kto better evaluate their performance on LLMs.
However, the diverse settings and datasets used in the evaluation complicate the understanding of their progression.In this paper, we conduct a comprehensive evaluation of LoRA across various tasks under the same settings, including instruction tuning, mathematical reasoning, and continual pretraining. We find that LoRA-like methods demonstrate similar performance across these tasks and they perform comparably to FFT in instruction tuning but fall short in mathematical reasoning and continual pretraining. Among these tasks, instruction tuning primarily focuses on interacting with the format, rather than acquiring knowledge and capabilities, which are learned almost entirely during pretraining. We observe that LoRA is easily adapted to follow response formats in instruction tuning but struggles with other tasks that require enhancing knowledge and capabilities through fine-tuning.One plausible explanation for this limitation observed with LoRA could be its reliance on low-rank updates. The low-rank update matrix,, struggles to estimate the full-rank updates in FFT, particularly in memory-intensive tasks like continual pretraining that require memorizing domain-specific knowledge.
Since the rank ofis significantly smaller than the full rank, this limitation restricts capacity to store new information via fine-tuning. Moreover, current variants of LoRA cannot alter the inherent characteristic of low-rank updates.
To validate this, we conducted a memorization task using pseudo-data to assess the performance of LoRA in memorizing new knowledge. We found that LoRA performed significantly worse than FFT, even with a large rank such as 256.Given these observations, we introduce a method called MoRA, which employs a square matrix as opposed to low-rank matrices, aiming to maximize the rank inwhile maintaining the same number of trainable parameters. For instance, when utilizing 8 rank with the hidden size 4096, LoRA employs two low-rank matricesand, with. Under same number of parameters, our method uses a square matrix, with, as depicted in Figure1.
Notably, our method exhibits a greater capacity than LoRA with a large rank.
To decrease the input dimension and increase the output dimension for, we develop corresponding non-parameter operators.
Furthermore, these operators andcan be substituted by a, ensuring our method can be merged back into LLM like LoRA.Our contributions are as follows:1.We introduce MoRA, a novel method that employs a square matrix instead of low-rank matrices in LoRA to achieve high-rank updating, while maintaining the same number of trainable parameters.2.We discuss four kinds of non-parameter operators of MoRA to reduce the input dimension and increase the output dimension for the square matrix, while ensures that the weight can be merged back into LLMs.3.We evaluate MoRA across five tasks: memory, instruction tuning, mathematical reasoning, continual pretraining, and pretraining. Our method outperforms LoRA on memory-intensive tasks and achieves comparable performance on other tasks, which demonstrates the effectiveness of high-rank updating.
introduction
MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
1Introduction
2405.12130
2Related Work2.1LoRALoRA is one of the most popular PEFT methods for fine-tuning LLM, owing to its broad applicability and robust performance in comparison to other methods.
To approximate the updated weightin FFT, LoRA employs two low-rank matrices for its decomposition. By adjusting the rank of these two matrices, LoRA can accordingly modify the trainable parameters. Benefit from it, LoRA can merge these matrices after fine-tuning without the inference latency compared to FFT.
There are many methods to further improve LoRA, particularly for the application in LLMs. DoRAfurther decomposes the original weight into magnitude and direction components and uses LoRA to update the direction component. LoRA+employs different learning rates for the two low-rank matrices to improve learning efficiency. ReLoRAintegrates LoRA into the LLM during training to increase the rank of the final.2.2Fine-Tuning with LLMsDespite the impressive performance of LLMs with in-context learning, certain scenarios still necessitate fine-tuning, which can be broadly categorized into three types.
The first type, instruction tuning, aims to better align LLMs with end tasks and user preferences, without significantly enhancing the knowledge and capabilities of LLMs. This approach simplifies the process of dealing with varied tasks and understanding complex instructions.
The second type involves complex reasoning tasks such as mathematical problem-solving, where general instruction tuning often falls short in handling complex, symbolic, multi-step reasoning tasks. To improve the reasoning abilities of LLMs, the majority of research focuses on creating corresponding training datasets, either by leveraging larger teacher models like GPT-4, or by rephrasing questions along a reasoning path.
The third type, continual pretraining, aims to enhance the domain-specific capabilities of LLMs. Unlike instruction tuning, it necessitates fine-tuning to augment the corresponding domain-specific knowledge and capabilities.However, most variants of LoRApredominantly employ instruction tuning or text classification tasks from GLUEto validate their efficacy on LLMs. Given that instruction tuning requires the least capacity for fine-tuning compared to other types, it may not accurately reflect the effectiveness of LoRA variants.
To better evaluate their methods, recent workshave employed reasoning tasks to test their methods.
But the training sets used are often too small for LLMs to effectively learn reasoning. For instance, some methodsutilize the GSM8Kwith only 7.5K training samples. Compare to the SOTA method with 395K training samples, this small training set achieves worse performance on reasoning and makes it hard to evaluate the effectiveness of these methods.
body
MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
2Related Work
2405.12130
3Analysis the Influence of Low-rank UpdatingThe key idea of LoRAinvolves the use of low-rank updates to estimate full-rank updates in FFT. Formally, given a pretrained parameter matrix, LoRA employs two low-rank matrices to calculate the weight update:(1) whereandrepresent the low-rank matrices in LoRA. To ensure thatat the beginning of training, LoRA initializeswith a Gaussian distribution andwith zero. Due to the low-rank decomposition ofinto, the.
The weight update in LoRA exhibits a markedly low rank,, in comparison to the full-rank updating in FFT.
Low-rank updating by LoRA shows on-par performance with full-rank updating in some tasks such as text classification or instruction tuning.
However, for tasks like complex reasoning or continual pretraining, LoRA tends to show worse performance.Based on these observations, we propose a hypothesis that low-rank updating is easy to leverage original knowledge and capabilities of LLM to solve task, but it is struggle to handle tasks that require enhancing knowledge and capabilities of LLM.Figure 2:Performance of memorizing UUID pairs through fine-tuning with FFT and LoRA.To substantiate this hypothesis, we examine the differences between LoRA and FFT in terms of memorizing new knowledge through fine-tuning. In order to circumvent leveraging the original knowledge of the LLM, we randomly generate 10K pairs of Universally Unique Identifiers (UUIDs) , each pair comprising two UUIDs with 32 hexadecimal values. The task requires the LLM to generate the corresponding UUID based on the input UUID. For instance, given a UUID such as “205f3777-52b6-4270-9f67-c5125867d358”, the model should generate the corresponding UUID based on 10K training pairs.
This task can also be viewed as a question-answering task, while the knowledge indispensable for accomplishing it is exclusively from the training datasets rather than the LLM itself.For the training settings, we employ LLaMA-2 7B as base model, utilizing 1,000 pairs per batch and conducting 100 epochs. For the LoRA, we apply low-rank matrices to all linear layers and search learning rate from1e-4,2e-4,3e-4to enhance performances.
We conduct the experiment on LoRA using various ranks.
For the FFT, we directly use a learning rate of 3e-5.
Based on Figure2, we observe low-rank updating are hard to memorizing new knowledge compared to FFT.
Although constantly increasing the rank of LoRA can alleviate this problem, the gap still exists.In contrast to the memory task, we also evaluate the performance gap between LoRA and FFT on instruction tuning, which merely introduces new knowledge.
Similar to previous results, we also find that LoRA matches the performance of FFT with small rankin Table1.
This indicates that LoRA can easily leverage the original knowledge of LLMs by fine-tuning like FFT.
body
MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
3Analysis the Influence of Low-rank Updating
2405.12130
4MethodBased on the above analysis, we propose a new method to alleviate the negative effects of low-rank updating. The main idea of our method is to utilize the same trainable parameters as much as possible to achieve a higher rank in.
Consider to the pretrained weight, LoRA uses two low-rank matricesandwithtotal trainable parameters for rank.
Under same trainable parameters, a square matrixwherecan achieve the highest rank due to.To accomplish this, we need to reduce the input dimension and increase the output dimension for. Formally,(2) wheredenotes the function that decreases the input dimension offromto, andrepresents the function that enhances the output dimension fromto.
Furthermore, these two functions ought to be non-parameterized operators and expected to execute in linear time corresponding to the dimension. They should also have corresponding function,and, to transforminto. For any, the following should hold:(3) where.
If Eq.3holds,can be losslessly expanded tobased onand. This allows our method to merge back into the LLM like LoRA.For the design ofand, we explore several methods to implement these functions.
One straightforward method is truncating the dimension and subsequently add it in corresponding dimension.
Formally, this can be represented as:(4) and the correspondingis:(5) However, this method leads to a significant loss of information during compression and only modifies a segment of the output by appending a zero vector during decompression.
To improve it, we can share the rows and columns ofto achieve a more efficient compression and decompression. Formally, this can be represented as:(6) Here,andrepresent predefined groups that share the same row and column in, respectively. Theindicates that the-th dimension belongs to the-th group in. The termis the reverse of, referring to the-th dimension associated with the-th group in. The correspondingis as follows:(7) Sharing rows and columns can be efficient for larger ranks such asor, as only a few rows or columns inshare a common row or column.
For instance, considering tofor, which hasand. In this situation, only 4 rows or columns share the same row or column.
Conversely, for smaller ranks such as, where, it requires average 16 rows or columns in a group to share the same row or column in.
It can lead to inefficiencies due to the significant information loss during compression in Eq.6.To enhance performance for smaller ranks, we reshapeinstead of directly compressing it, to preserve the input information. In this context,and.
Corresponding,andare as follows:(8) whererefers to concatenate the rows ofinto a vector.
For simplicity, we omit the padding and truncation operators in above functions and focus on the case where. In comparison to sharing columns and rows, this method incurs additional computational overhead by reshapingintoinstead of. However, given that the size ofis significantly smaller than, this additional computation is very small for rank like 8. For instance, when fine-tuning the 7B model with rank of 8 () , this method is only 1.03 times slower than previous methods.Inspired by RoPE, we can further refine this method by incorporating rotation operators intoto augment the expressiveness ofby enable it to differentiate between variousby rotating them.
We can modify Eq.8as follows:(9) whereandrepresent the corresponding values ofandpost-rotation, respectively.
Following RoPE, we use ablock diagonal matrix to achieve the rotation.
However, our method use rotation information to enabledistinguish theinstead of token position in RoPE.
We can defineandas follows:(10) whereandis a rotation matrix:(11) 
body
MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
4Method
2405.12130
5ExperimentWe evaluate our method on various tasks to understand the influence of high-rank updating.
In Section5.1, we evaluate our method with LoRA and our method on memorizing UUID pairs to show the benefit of high-rank updating on memorizing.
In Section5.2, we reproduce LoRA, LoRA variants and FFT on three fine-tuning tasks: instruction tuning, mathematical reasoning and continual pretraining.
In Section5.3, we compare our method with LoRA and ReLoRA on pretraining by training transformer from scratch.Figure 3:Performance of memorizing UUID pairs with LoRA and our method on rank 8 and 256.Instruction TuningMathematical ReasoningContinual PretrainingMethodRankMMLU 0MMLU 5GSM8KMATHBioMed.FinanceFFT-50.651.366.620.156.469.6LoRA850.251.564.615.152.364.0LoRA+849.251.164.115.852.264.9ReLoRA849.350.261.514.546.361.0AsyLoRA850.352.264.515.052.563.5DoRA850.251.564.514.652.563.9MoRA (Ours) 849.751.564.215.453.367.1LoRA25649.750.867.919.954.167.3LoRA+25649.251.368.217.154.266.7ReLoRA256--64.018.152.957.9AsyLoRA25650.152.066.919.354.166.9DoRA25649.651.167.419.554.266.0MoRA (Ours) 25649.951.467.919.255.468.7Table 1:Performance of FFT, LoRA, LoRA variants and our method on instruction tuning, mathematical reasoning and continual pretraining tasks.5.1Memorizing UUID PairsWe first compare our method with LoRA and FFT on memorizing UUID pairs to demonstrate improvements through high-rank updating.
Following the training settings in Section3, we search learning rate from5e-5,1e-4,2e-4and use decompress and compress functions in Eq.8, sharing rows and columns in.
Due to use one matrixinstead of two matricesand, we can directly initializewith zeros.
For the predefined groupsand, we group every adjacentrows or columns together.
The training loss is presented in Figure3. Our method shows significant improvements over LoRA with the same number of trainable parameters, benefiting from high-rank updating. We also report character-level accuracy at various training steps in Table2. MoRA requires fewer training steps to memorize these UUID pairs compared to LoRA.
Compared to FFT, MoRA with 256 rank can achieve similar performance and both method can memorize all UUID pairs in 500 steps.Rank300500700900FFT-42.5100100100LoRA89.910.010.754.2MoRA810.115.787.4100LoRA2569.970.6100100MoRA25641.6100100100Table 2:Character-level accuracy of memorizing UUID pairs by generating the value of corresponding key in 300, 500, 700 and 900 training steps.5.2Fine-tuning Tasks5.2.1SetupWe evaluate our method across three fine-tuning tasks for large language models (LLMs) : instruction tuning, mathematical reasoning, and continual pretraining.
For these tasks, we select high-quality corresponding datasets to test both LoRA and our method.
In instruction tuning, we utilize Tülu v2, a blend of several high-quality instruction datasets, containing 326k filtered samples. We assess instruction performance using the MMLUin both zero-shot and five-shot settings. For mathematical reasoning, we employ the MetaMathwith its 395k samples to enhance mathematical reasoning capabilities and also use GSM8Kand MATHfor further evaluation. In continual pretraining, we adapt an LLM to the biomedicine and finance using PubMed abstracts from the Pileand finicial news, complemented by data preprocessing methods from AdaptLLMto boost performance. We report the average performance of corresponding tasks for continual pretraining. More details can be found in AppendixC.(a) Pretraining loss at 250M models.(b) Pretraining loss at 1.3B models.Figure 4:Pretraining loss with LoRA and MoRA on 250M and 1B models from scratch.
Both LoRA and MoRA use same amount of trainable parameters with.
ReMoRA and ReLoRA refer to merge MoRA or LoRA back to the model during training to increase the rank of.5.2.2Baselines and ImplementsFor LoRA-like methods and MoRA, we conducted experiments atand, and reproduce following methods across three tasks: FFT, LoRA, LoRA+, AsyLoRA, ReLoRAand DoRA.
LoRA+ enhances the learning rate of matrixin LoRA to facilitate efficient feature learning based on theoretical analysis.
We search the corresponding the hyperparameterfrom2,4.
AsyLoRA also analyzes asymmetry in theandmatrices, and we adopted their initialization strategy.
ReLoRA proposes a method to merge low-rank matrices into the model during training to increase the rank of.
we search merge steps from {k,k} and use 50 steps restarts warmup.
DoRA leverages weight decomposition to enhance performance as a robust baseline.
For FFT, we follow the settings proposed by corresponding datasets.
For MoRA, we employed rotation operators as outlined in Eq.9to implement compression and decompression for, and for, we utilized shared rows and columns as specified in Eq.6and group every adjacentrows or columns together.
The details hyperparameters about fine-tuning can be found in AppendixA.5.2.3Results and AnalysisWe present the results of fine-tuning tasks in Table1.
We report the results of MMLU with zero-shot and 5-shot settings for instruction tuning, GSM8K and MATH for mathematical reasoning, and average performance on biomedical tasks and financial tasks for continual pretraining.MoRA shows on par performances with LoRA on instruction tuning and mathematical reasoning.
Benefit from high-rank updating to memorize new knowledge, MoRA outperforms LoRA on both biomedical and financial domains for continual pretraining.We also find that LoRA variants exhibit similar performances on these fine-tuning tasks as compared to LoRA. Although AsyLoRA achieves the best performance in instruction tuning, it demonstrates poor performance in mathematical reasoning. For ReLoRA, merging low-rank matrices during training can harm performance, particularly at the the high rank like 256.Consider the difference between three tasks, they show different requirements for fine-tuning capabilities.
For instruction tuning, which does not learn new knowledge from fine-tuning, rank 8 is enough to achieve performance similar to FFT. For mathematical reasoning, rank 8 is unable to match FFT performance.
However, increasing the rank from 8 to 256 can eliminate the performance gap. For continual pretraining, LoRA with rank 256 still underperforms FFT.5.3Pretraining250M1.3BLoRA33.4028.56MoRA (Ours) 28.5425.25ReLoRA32.1927.80ReMoRA (Ours) 26.7423.34Table 3:Perplexity on C4 validation dataset.To understand the influence of high-rank updating,
we train transformer from scratch on the C4 datasets.
For the model architeture, we train LLaMA-based model with RMSNorm, SwiGLUand RoPEon 250M and 1.3B sizes.
For the hyperparameters, we use 10k steps, 1024 batch size, 512 sequence length and follow, using rankfor LoRA and our methods and also keep modules without applying LoRA-like layernorm or embeddings unfreezed.
We compare our method with LoRA and ReLoRA.
To better show the difference between high-rank and low-rank updating, we reproduce ReLoRA and other methods without full-rank training warmup.
For MoRA, we use compression and decompression functions in Eq.6by sharing columns and rows.We also combine merge-and-reint in ReLoRA with our method called ReMoRA by mergingback into the original parameters during training to increase the rank of.
However, if we directly mergewithandin Eq.6, the final rank ofis unchanged due to the same expand pattern.
To solve this problem, we can changeandafter merging to ensure the rank ofincreasing.
More details about ReMoRA can be found in AppendixB.
For the hyperparameters corresponding to ReLoRA and ReMoRA, we merge every 2k steps and use 50 steps restarts warmup with optimizer reseting and jagged scheduler.We show pretraining loss in Figure4and corresponding perplexity on C4 validation dataset in Table3.
Our method show better performance on pretraining compared to LoRA and ReLoRA with same amount of trainable parameters.
Benefiting from high-rank updating, ReMoRA also achieves more improvements on MoRA compared to ReLoRA, which
demonstrates the effectiveness of merge-and-reint strategy in ReMoRA.
body
MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
5Experiment
2405.12130
6AnalysisFigure 5:The number of singular valuesinon the 250M pretraining model.6.1High-rank UpdatingTo demonstrate the impact of high-rank updating on the rank of, we analyzed the spectrum of singular values for the learnedon 250M pretraining 250M model.
We present the average count of singular values exceeding 0.1 across all layers for,,,,,, andin Figure5following.
MoRA and ReMoRA exhibit a substantially higher number of significant singular values compared to LoRA and ReLoRA, highlighting the effectiveness of our methods in increasing the rank of. We find the quantity of singular values shown in Figure5can be correlated with the perplexity metrics listed in Table3. Moreover, MoRA, without merge-and-reint strategy in ReLoRA and ReMoRA, can achieve a lower perplexity than ReLoRA along with a higher significant singular values.6.2Influence of Decompression and CompressionTo explore the impact of decompression and compression functions in MoRA, we report the performance on GSM8K using various methods: truncation, sharing, decoupling, and rotation in Table4.
Among these methods, truncation shows the worst performance due to the significant information loss during compression.
Sharing can achieve better performance than truncation by leveraging the shared rows or columns to preserve the input information.
But in the case of, sharing shows worse performance than decouple and rotation due to the large number of sharing rows or columns, as we discussed in Section4.
Rotation is more efficient than decouple, due to the rotation information can help the square matrix to distinguish the input information.,TruncationEq.459.566.6SharingEq.662.567.9DecoupleEq.863.667.8RotationEq.964.267.9Table 4:Influence of decompression and compression functions onandon GSM8K.
body
MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
6Analysis
2405.12130
7ConclusionIn this paper, we analyze the impact of low-rank updating through LoRA and observe that such updating struggles for memory-intensive tasks, which also limits current LoRA variants. To overcome this limitation, we introduce MoRA, a method that utilizes non-parameterized operators for high-rank updating. Within the MoRA framework, we explore various methods to implement decompression and compression functions. Performance comparisons indicate that MoRA matches LoRA in instruction tuning and mathematical reasoning, and exhibits superior performance in continual pretraining and memory tasks. Additionally, we conduct pretraining experiments to further demonstrate the effectiveness of high-rank updating and show superior results compared to ReLoRA.
conclusion
MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
7Conclusion