<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2310.11451] Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective</title><meta property="og:description" content="Large Language Models (LLMs) inherently encode a wealth of knowledge within their parameters through pre-training on extensive corpora. While prior research has delved into operations on these parameters to manipulate …">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2310.11451">

<!--Generated on Wed Feb 28 00:15:36 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Seeking Neural Nuggets: Knowledge
<br class="ltx_break">Transfer in Large Language Models from a Parametric Perspective</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Ming Zhong<sup id="id9.9.id1" class="ltx_sup">1</sup>, Chenxin An<sup id="id10.10.id2" class="ltx_sup">2</sup>, Weizhu Chen<sup id="id11.11.id3" class="ltx_sup">3</sup>, Jiawei Han<sup id="id12.12.id4" class="ltx_sup">1</sup>, Pengcheng He<sup id="id13.13.id5" class="ltx_sup">3</sup> 
<br class="ltx_break"><sup id="id14.14.id6" class="ltx_sup">1</sup>University of Illinois Urbana-Champaign, <sup id="id15.15.id7" class="ltx_sup">2</sup>The University of Hong Kong, <sup id="id16.16.id8" class="ltx_sup">3</sup>Microsoft Azure AI 
<br class="ltx_break"><span id="id17.17.id9" class="ltx_text ltx_font_typewriter">{mingz5, hanj}@illinois.edu</span>, <span id="id18.18.id10" class="ltx_text ltx_font_typewriter">cxan23@connect.hku.hk</span> 
<br class="ltx_break"><span id="id19.19.id11" class="ltx_text ltx_font_typewriter">wzchen@microsoft.com</span>, <span id="id20.20.id12" class="ltx_text ltx_font_typewriter">Herbert.he@gmail.com</span>
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id21.id1" class="ltx_p">Large Language Models (LLMs) inherently encode a wealth of knowledge within their parameters through pre-training on extensive corpora. While prior research has delved into operations on these parameters to manipulate the underlying implicit knowledge—encompassing detection, editing, and merging—there remains an ambiguous understanding regarding their transferability across models with varying scales. In this paper, we seek to empirically investigate knowledge transfer from larger to smaller models through a parametric perspective. To achieve this, we employ sensitivity-based techniques to extract and align knowledge-specific parameters between different LLMs. Moreover, the LoRA module is used as the intermediary mechanism for injecting the extracted knowledge into smaller models. Evaluations across four benchmarks validate the efficacy of our proposed method. Our findings highlight the critical factors contributing to the process of parametric knowledge transfer, underscoring the transferability of model parameters across LLMs of different scales. We release code and data at <a target="_blank" href="https://github.com/maszhongming/ParaKnowTransfer" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/maszhongming/ParaKnowTransfer</a>.</p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para ltx_noindent">
<p id="S1.p1.1" class="ltx_p">Driven by the advancements of Large Language Models (LLMs) <cite class="ltx_cite ltx_citemacro_citep">(Brown et al., <a href="#bib.bib4" title="" class="ltx_ref">2020</a>; Chowdhery et al., <a href="#bib.bib10" title="" class="ltx_ref">2022</a>; OpenAI, <a href="#bib.bib42" title="" class="ltx_ref">2023</a>; Touvron et al., <a href="#bib.bib56" title="" class="ltx_ref">2023a</a>)</cite>, a transformative wave has reshaped the landscape in multiple areas of Artificial Intelligence, elevating performance across diverse tasks.
From a parametric perspective, the objective of pre-training is to encode substantial amounts of knowledge into model parameters through language modeling on extensive corpora <cite class="ltx_cite ltx_citemacro_citep">(Peters et al., <a href="#bib.bib46" title="" class="ltx_ref">2018</a>; <a href="#bib.bib50" title="" class="ltx_ref">Radford et al., </a>; Devlin et al., <a href="#bib.bib14" title="" class="ltx_ref">2019</a>; Delétang et al., <a href="#bib.bib13" title="" class="ltx_ref">2023</a>)</cite>.
In a quest to unravel the intricate workings of LLMs, a multitude of research efforts have been directed toward the detailed exploration and nuanced manipulation of this reservoir of implicit knowledge.</p>
</div>
<div id="S1.p2" class="ltx_para ltx_noindent">
<p id="S1.p2.1" class="ltx_p">Early research efforts sought to detect this parametric knowledge, typically probing the concrete facts by using the “fill-in-the-blank” task under a closed-book setting <cite class="ltx_cite ltx_citemacro_citep">(Petroni et al., <a href="#bib.bib47" title="" class="ltx_ref">2019</a>; Jiang et al., <a href="#bib.bib28" title="" class="ltx_ref">2020</a>; Roberts et al., <a href="#bib.bib51" title="" class="ltx_ref">2020</a>)</cite>.
Subsequent studies delved into the feasibility of executing operations on model knowledge, including knowledge editing <cite class="ltx_cite ltx_citemacro_citep">(Cao et al., <a href="#bib.bib5" title="" class="ltx_ref">2021</a>; Mitchell et al., <a href="#bib.bib39" title="" class="ltx_ref">2022</a>; Meng et al., <a href="#bib.bib36" title="" class="ltx_ref">2022</a>)</cite>, a technique designed to modify targeted knowledge while preserving the integrity of the remaining information, and model merging <cite class="ltx_cite ltx_citemacro_citep">(Izmailov et al., <a href="#bib.bib27" title="" class="ltx_ref">2018</a>; Ainsworth et al., <a href="#bib.bib2" title="" class="ltx_ref">2023</a>; Stoica et al., <a href="#bib.bib54" title="" class="ltx_ref">2023</a>)</cite>, a strategy that combines multiple models to enhance robustness or facilitate multitasking abilities.
While these investigations exhibited that such parametric knowledge is both <span id="S1.p2.1.1" class="ltx_text ltx_font_italic">detectable</span> and <span id="S1.p2.1.2" class="ltx_text ltx_font_italic">editable</span> within a single model, the broader question of whether it is <span id="S1.p2.1.3" class="ltx_text ltx_font_italic">transferable</span> across different LLMs remains an open and under-explored topic.</p>
</div>
<div id="S1.p3" class="ltx_para ltx_noindent">
<p id="S1.p3.1" class="ltx_p">Knowledge transfer refers to distilling the expertise of larger teacher models into smaller, more manageable counterparts, thereby democratizing access to cutting-edge machine learning capabilities.
As illustrated in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, online and offline distillation currently stand as the primary paradigms.
The former, especially prevalent before the LLM era, capitalizes on teacher models to guide the learning trajectory of student models <cite class="ltx_cite ltx_citemacro_citep">(Hinton et al., <a href="#bib.bib22" title="" class="ltx_ref">2015</a>; Sanh et al., <a href="#bib.bib52" title="" class="ltx_ref">2019</a>; Gou et al., <a href="#bib.bib19" title="" class="ltx_ref">2021</a>)</cite>.
Yet, as LLMs grow in scale, the inherent demand for the teacher model to undergo fine-tuning or participate in student training becomes increasingly cost-prohibitive.
In contrast, offline distillation calls upon the teacher model merely to generate answers to open-ended queries, creating a distilled training dataset for students <cite class="ltx_cite ltx_citemacro_citep">(Honovich et al., <a href="#bib.bib23" title="" class="ltx_ref">2023</a>; Wang et al., <a href="#bib.bib62" title="" class="ltx_ref">2023c</a>; Taori et al., <a href="#bib.bib55" title="" class="ltx_ref">2023</a>)</cite>.
Despite reducing the overhead to thousands of inferences, it completely overlooks the rich knowledge implicitly stored within the teacher’s parameters.</p>
</div>
<div id="S1.p4" class="ltx_para ltx_noindent">
<p id="S1.p4.1" class="ltx_p">In this paper, we empirically investigate knowledge transfer from a distinct parametric perspective, dedicated to selecting static parameters directly from the teacher model and exploring their transferability.
Specifically, we introduce a new parametric knowledge transfer paradigm designed to extract task-specific parameters from the teacher model and subsequently inject them into the student model, thereby enhancing performance on downstream tasks.
Through decoding on a limited set of seed samples (e.g., 32 samples) with the teacher model, we calculate sensitivity metrics that serve as the basis for knowledge extraction.
Given the discrepancies in the number of layers and dimensions across varied LLM scales, we employ sensitivity-based layer mapping and dimensionality reduction techniques to establish alignment between the teacher and student models.
Lastly, we leverage LoRA <cite class="ltx_cite ltx_citemacro_citep">(Hu et al., <a href="#bib.bib24" title="" class="ltx_ref">2022</a>)</cite> as a bridge to inject these extracted parameters into student models, facilitating its fine-tuning on downstream tasks and thus achieving the knowledge transfer process.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2310.11451/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="139" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>Different paradigms of knowledge transfer from teacher models to student models. (a) Online Distillation: utilizing soft logits from the fine-tuned teacher model to guide the training of the student model; (b) Offline Distillation: generating a distilled dataset that encapsulates the knowledge of the teacher model to fine-tune the student model. (c) Parametric Knowledge Transfer: extracting knowledge-specific parameters from the vanilla teacher model and injecting them into the student model to enhance its training efficacy.</figcaption>
</figure>
<div id="S1.p5" class="ltx_para ltx_noindent">
<p id="S1.p5.1" class="ltx_p">Experimentally, we evaluate the parametric knowledge transfer framework across four benchmark categories: reasoning <cite class="ltx_cite ltx_citemacro_citep">(Cobbe et al., <a href="#bib.bib11" title="" class="ltx_ref">2021</a>)</cite>, professional knowledge <cite class="ltx_cite ltx_citemacro_citep">(Hendrycks et al., <a href="#bib.bib21" title="" class="ltx_ref">2021</a>)</cite>, instruction-driven NLP tasks <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib60" title="" class="ltx_ref">2022</a>)</cite>, and open-ended conversation <cite class="ltx_cite ltx_citemacro_citep">(Dubois et al., <a href="#bib.bib15" title="" class="ltx_ref">2023</a>)</cite>, using various sizes of LLaMA models <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib56" title="" class="ltx_ref">2023a</a>; <a href="#bib.bib57" title="" class="ltx_ref">b</a>)</cite>.
The results indicate that upon transferring the teacher model’s parameters, the student performance demonstrates consistent improvements across all benchmarks, affirming the transferability of parametric knowledge.
Furthermore, our detailed analyses illustrate the underlying factors that contribute to effective parametric knowledge transfer, discovering that the teacher scales, initialization strategies, number of seed samples, and the origin and structure of the extracted parameters all play crucial roles.</p>
</div>
<div id="S1.p6" class="ltx_para ltx_noindent">
<p id="S1.p6.1" class="ltx_p">To summarize, the key contributions of this paper are threefold: (1) From a distinct perspective, we introduce a parametric knowledge transfer paradigm that encompasses stages of sensitivity-based knowledge extraction and LoRA-driven knowledge injection. (2) Through comprehensive evaluations, we provide empirical evidence that implicit model knowledge is indeed <span id="S1.p6.1.1" class="ltx_text ltx_font_italic">transferable</span> across varying scales of LLMs. (3) Further enriching our insights into parametric knowledge transfer, we undertake a thorough analysis to pinpoint the pivotal factors that dictate its efficacy.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Manipulation of Implicit Model Knowledge</h3>

<div id="S2.SS1.p1" class="ltx_para ltx_noindent">
<p id="S2.SS1.p1.1" class="ltx_p">With the recognition of the vast repository of knowledge embedded in model parameters <cite class="ltx_cite ltx_citemacro_citep">(Petroni et al., <a href="#bib.bib47" title="" class="ltx_ref">2019</a>; Jiang et al., <a href="#bib.bib28" title="" class="ltx_ref">2020</a>; Roberts et al., <a href="#bib.bib51" title="" class="ltx_ref">2020</a>; Dai et al., <a href="#bib.bib12" title="" class="ltx_ref">2022</a>)</cite>, ensuing research has sought to execute diverse operations on these parameters, aiming to manipulate the implicit knowledge.
For instance, knowledge editing endeavors to modify or update specific facts by editing the associated parameters, all the while ensuring the broader knowledge base remains untouched <cite class="ltx_cite ltx_citemacro_citep">(Cao et al., <a href="#bib.bib5" title="" class="ltx_ref">2021</a>; Mitchell et al., <a href="#bib.bib39" title="" class="ltx_ref">2022</a>; Meng et al., <a href="#bib.bib36" title="" class="ltx_ref">2022</a>; <a href="#bib.bib37" title="" class="ltx_ref">2023</a>; Yao et al., <a href="#bib.bib68" title="" class="ltx_ref">2023b</a>)</cite>.
Another avenue, model merging, combines the weights of two or more models into a unified weight set for greater robustness or multitasking capabilities <cite class="ltx_cite ltx_citemacro_citep">(Huang et al., <a href="#bib.bib25" title="" class="ltx_ref">2017</a>; Izmailov et al., <a href="#bib.bib27" title="" class="ltx_ref">2018</a>; Ainsworth et al., <a href="#bib.bib2" title="" class="ltx_ref">2023</a>; Stoica et al., <a href="#bib.bib54" title="" class="ltx_ref">2023</a>)</cite>.
Additionally, a strand of studies probes into performing arithmetic operations on the pre-trained weights of a model or specific modules, thus enabling the model to augment or diminish particular functionalities <cite class="ltx_cite ltx_citemacro_citep">(Ilharco et al., <a href="#bib.bib26" title="" class="ltx_ref">2023</a>; Ortiz-Jiménez et al., <a href="#bib.bib43" title="" class="ltx_ref">2023</a>; Zhang et al., <a href="#bib.bib69" title="" class="ltx_ref">2023</a>)</cite>.
However, these explorations are limited to operations within individual models or the merging of models with identical architectures, without investigating whether implicit knowledge between different scale models can be manipulated and transferred.</p>
</div>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Inheritance of Model Knowledge</h3>

<div id="S2.SS2.p1" class="ltx_para ltx_noindent">
<p id="S2.SS2.p1.1" class="ltx_p">Another line of research that aligns more closely with our work concerns the operation of model parameters across scales, specifically the concept of model growth <cite class="ltx_cite ltx_citemacro_citep">(Yao et al., <a href="#bib.bib67" title="" class="ltx_ref">2023a</a>; Li et al., <a href="#bib.bib33" title="" class="ltx_ref">2023</a>)</cite>.
This refers to accelerating the pre-training of LLMs by incrementally growing and expanding the parameters of a smaller model, using them as an initialization for the larger one.
The majority of existing work is concentrated on devising function-preserving methods <cite class="ltx_cite ltx_citemacro_citep">(Chen et al., <a href="#bib.bib8" title="" class="ltx_ref">2016</a>)</cite>, ensuring that the initialized larger model replicates the behaviors of the original smaller model <cite class="ltx_cite ltx_citemacro_citep">(Wei et al., <a href="#bib.bib64" title="" class="ltx_ref">2016</a>; Gu et al., <a href="#bib.bib20" title="" class="ltx_ref">2021</a>; Chen et al., <a href="#bib.bib7" title="" class="ltx_ref">2022</a>; Evci et al., <a href="#bib.bib16" title="" class="ltx_ref">2022</a>; Shen et al., <a href="#bib.bib53" title="" class="ltx_ref">2022</a>; Gesmundo &amp; Maile, <a href="#bib.bib18" title="" class="ltx_ref">2023</a>)</cite>.
Concurrently, several studies adopt data-driven strategies, investigating reverse distillation <cite class="ltx_cite ltx_citemacro_citep">(Qin et al., <a href="#bib.bib49" title="" class="ltx_ref">2022</a>)</cite> or mapping learned weights from smaller models to their larger counterparts <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib59" title="" class="ltx_ref">2023a</a>)</cite>.
In contrast to this research direction, our emphasis is on the transfer of knowledge from larger teacher to smaller student models, with the aim of exploring not only the efficiency of training, but also the transferability of parametric knowledge across different scenarios.</p>
</div>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3 </span>Transfer of Model Knowledge</h3>

<div id="S2.SS3.p1" class="ltx_para ltx_noindent">
<p id="S2.SS3.p1.1" class="ltx_p">Knowledge transfer is a research area dedicated to training a smaller student model to mimic the behavior of a larger pre-trained teacher model to achieve similar performance with fewer parameters <cite class="ltx_cite ltx_citemacro_citep">(Hinton et al., <a href="#bib.bib22" title="" class="ltx_ref">2015</a>)</cite>.
Despite progress in improving the online distillation paradigm <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib70" title="" class="ltx_ref">2018</a>; Lan et al., <a href="#bib.bib31" title="" class="ltx_ref">2018</a>; Jin et al., <a href="#bib.bib29" title="" class="ltx_ref">2019</a>; Mirzadeh et al., <a href="#bib.bib38" title="" class="ltx_ref">2020</a>; Park et al., <a href="#bib.bib44" title="" class="ltx_ref">2021</a>; Pham et al., <a href="#bib.bib48" title="" class="ltx_ref">2021</a>; Zhou et al., <a href="#bib.bib72" title="" class="ltx_ref">2022</a>)</cite> and optimizing the efficiency of offline distillation <cite class="ltx_cite ltx_citemacro_citep">(Honovich et al., <a href="#bib.bib23" title="" class="ltx_ref">2023</a>; Wang et al., <a href="#bib.bib62" title="" class="ltx_ref">2023c</a>; Wu et al., <a href="#bib.bib65" title="" class="ltx_ref">2023</a>; Taori et al., <a href="#bib.bib55" title="" class="ltx_ref">2023</a>; Peng et al., <a href="#bib.bib45" title="" class="ltx_ref">2023</a>; Xu et al., <a href="#bib.bib66" title="" class="ltx_ref">2023</a>)</cite>, they both completely ignore the implicit knowledge embedded inherently in the teacher model.
In this paper, we are grounded in a fresh perspective and propose a parametric knowledge transfer framework. It seeks to affirm the transferability of such implicit knowledge by directly extracting parameters from the teacher model and injecting them into the student model. Furthermore, by leveraging the static parameters of the teacher model, our approach naturally holds an efficiency advantage over preceding methodologies.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Parametric Knowledge Transfer</h2>

<div id="S3.p1" class="ltx_para ltx_noindent">
<p id="S3.p1.1" class="ltx_p">In this section, we first outline the task formulation for parametric knowledge transfer. Following this, we delve into a detailed description of our proposed method, as depicted in Figure <a href="#S3.F2" title="Figure 2 ‣ 3.1 Task Formulation ‣ 3 Parametric Knowledge Transfer ‣ Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Task Formulation</h3>

<div id="S3.SS1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS1.p1.5" class="ltx_p">The core objective of parametric knowledge transfer is to enhance a student model by selectively transferring task-specific parametric knowledge from a more knowledgeable teacher model. Given a task <math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><ci id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">\mathcal{T}</annotation></semantics></math>, the transfer process begins with a teacher model <math id="S3.SS1.p1.2.m2.1" class="ltx_Math" alttext="M_{T}" display="inline"><semantics id="S3.SS1.p1.2.m2.1a"><msub id="S3.SS1.p1.2.m2.1.1" xref="S3.SS1.p1.2.m2.1.1.cmml"><mi id="S3.SS1.p1.2.m2.1.1.2" xref="S3.SS1.p1.2.m2.1.1.2.cmml">M</mi><mi id="S3.SS1.p1.2.m2.1.1.3" xref="S3.SS1.p1.2.m2.1.1.3.cmml">T</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.2.m2.1b"><apply id="S3.SS1.p1.2.m2.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.2.m2.1.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1">subscript</csymbol><ci id="S3.SS1.p1.2.m2.1.1.2.cmml" xref="S3.SS1.p1.2.m2.1.1.2">𝑀</ci><ci id="S3.SS1.p1.2.m2.1.1.3.cmml" xref="S3.SS1.p1.2.m2.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.2.m2.1c">M_{T}</annotation></semantics></math> endowed with parameters <math id="S3.SS1.p1.3.m3.1" class="ltx_Math" alttext="\mathbf{\Theta}_{T}" display="inline"><semantics id="S3.SS1.p1.3.m3.1a"><msub id="S3.SS1.p1.3.m3.1.1" xref="S3.SS1.p1.3.m3.1.1.cmml"><mi id="S3.SS1.p1.3.m3.1.1.2" xref="S3.SS1.p1.3.m3.1.1.2.cmml">𝚯</mi><mi id="S3.SS1.p1.3.m3.1.1.3" xref="S3.SS1.p1.3.m3.1.1.3.cmml">T</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.3.m3.1b"><apply id="S3.SS1.p1.3.m3.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.3.m3.1.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1">subscript</csymbol><ci id="S3.SS1.p1.3.m3.1.1.2.cmml" xref="S3.SS1.p1.3.m3.1.1.2">𝚯</ci><ci id="S3.SS1.p1.3.m3.1.1.3.cmml" xref="S3.SS1.p1.3.m3.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.3.m3.1c">\mathbf{\Theta}_{T}</annotation></semantics></math> and a student model <math id="S3.SS1.p1.4.m4.1" class="ltx_Math" alttext="M_{S}" display="inline"><semantics id="S3.SS1.p1.4.m4.1a"><msub id="S3.SS1.p1.4.m4.1.1" xref="S3.SS1.p1.4.m4.1.1.cmml"><mi id="S3.SS1.p1.4.m4.1.1.2" xref="S3.SS1.p1.4.m4.1.1.2.cmml">M</mi><mi id="S3.SS1.p1.4.m4.1.1.3" xref="S3.SS1.p1.4.m4.1.1.3.cmml">S</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.4.m4.1b"><apply id="S3.SS1.p1.4.m4.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.4.m4.1.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1">subscript</csymbol><ci id="S3.SS1.p1.4.m4.1.1.2.cmml" xref="S3.SS1.p1.4.m4.1.1.2">𝑀</ci><ci id="S3.SS1.p1.4.m4.1.1.3.cmml" xref="S3.SS1.p1.4.m4.1.1.3">𝑆</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.4.m4.1c">M_{S}</annotation></semantics></math> characterized by parameters <math id="S3.SS1.p1.5.m5.1" class="ltx_Math" alttext="\mathbf{\Theta}_{S}" display="inline"><semantics id="S3.SS1.p1.5.m5.1a"><msub id="S3.SS1.p1.5.m5.1.1" xref="S3.SS1.p1.5.m5.1.1.cmml"><mi id="S3.SS1.p1.5.m5.1.1.2" xref="S3.SS1.p1.5.m5.1.1.2.cmml">𝚯</mi><mi id="S3.SS1.p1.5.m5.1.1.3" xref="S3.SS1.p1.5.m5.1.1.3.cmml">S</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.5.m5.1b"><apply id="S3.SS1.p1.5.m5.1.1.cmml" xref="S3.SS1.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.5.m5.1.1.1.cmml" xref="S3.SS1.p1.5.m5.1.1">subscript</csymbol><ci id="S3.SS1.p1.5.m5.1.1.2.cmml" xref="S3.SS1.p1.5.m5.1.1.2">𝚯</ci><ci id="S3.SS1.p1.5.m5.1.1.3.cmml" xref="S3.SS1.p1.5.m5.1.1.3">𝑆</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.5.m5.1c">\mathbf{\Theta}_{S}</annotation></semantics></math>.</p>
</div>
<div id="S3.SS1.p2" class="ltx_para ltx_noindent">
<p id="S3.SS1.p2.5" class="ltx_p">The first step in this procedure involves extraction, where task-relevant parameters are identified from the teacher model and resized to a desired scale based on the student model’s parameter dimensions. This can be expressed as:</p>
<table id="S3.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E1.m1.2" class="ltx_Math" alttext="\text{Extract}(\mathbf{\Theta}_{T};\mathbf{\Theta}_{S};\mathcal{T})=\mathbf{\Theta}_{T_{\text{extract}}}," display="block"><semantics id="S3.E1.m1.2a"><mrow id="S3.E1.m1.2.2.1" xref="S3.E1.m1.2.2.1.1.cmml"><mrow id="S3.E1.m1.2.2.1.1" xref="S3.E1.m1.2.2.1.1.cmml"><mrow id="S3.E1.m1.2.2.1.1.2" xref="S3.E1.m1.2.2.1.1.2.cmml"><mtext id="S3.E1.m1.2.2.1.1.2.4" xref="S3.E1.m1.2.2.1.1.2.4a.cmml">Extract</mtext><mo lspace="0em" rspace="0em" id="S3.E1.m1.2.2.1.1.2.3" xref="S3.E1.m1.2.2.1.1.2.3.cmml">​</mo><mrow id="S3.E1.m1.2.2.1.1.2.2.2" xref="S3.E1.m1.2.2.1.1.2.2.3.cmml"><mo stretchy="false" id="S3.E1.m1.2.2.1.1.2.2.2.3" xref="S3.E1.m1.2.2.1.1.2.2.3.cmml">(</mo><msub id="S3.E1.m1.2.2.1.1.1.1.1.1" xref="S3.E1.m1.2.2.1.1.1.1.1.1.cmml"><mi id="S3.E1.m1.2.2.1.1.1.1.1.1.2" xref="S3.E1.m1.2.2.1.1.1.1.1.1.2.cmml">𝚯</mi><mi id="S3.E1.m1.2.2.1.1.1.1.1.1.3" xref="S3.E1.m1.2.2.1.1.1.1.1.1.3.cmml">T</mi></msub><mo id="S3.E1.m1.2.2.1.1.2.2.2.4" xref="S3.E1.m1.2.2.1.1.2.2.3.cmml">;</mo><msub id="S3.E1.m1.2.2.1.1.2.2.2.2" xref="S3.E1.m1.2.2.1.1.2.2.2.2.cmml"><mi id="S3.E1.m1.2.2.1.1.2.2.2.2.2" xref="S3.E1.m1.2.2.1.1.2.2.2.2.2.cmml">𝚯</mi><mi id="S3.E1.m1.2.2.1.1.2.2.2.2.3" xref="S3.E1.m1.2.2.1.1.2.2.2.2.3.cmml">S</mi></msub><mo id="S3.E1.m1.2.2.1.1.2.2.2.5" xref="S3.E1.m1.2.2.1.1.2.2.3.cmml">;</mo><mi class="ltx_font_mathcaligraphic" id="S3.E1.m1.1.1" xref="S3.E1.m1.1.1.cmml">𝒯</mi><mo stretchy="false" id="S3.E1.m1.2.2.1.1.2.2.2.6" xref="S3.E1.m1.2.2.1.1.2.2.3.cmml">)</mo></mrow></mrow><mo id="S3.E1.m1.2.2.1.1.3" xref="S3.E1.m1.2.2.1.1.3.cmml">=</mo><msub id="S3.E1.m1.2.2.1.1.4" xref="S3.E1.m1.2.2.1.1.4.cmml"><mi id="S3.E1.m1.2.2.1.1.4.2" xref="S3.E1.m1.2.2.1.1.4.2.cmml">𝚯</mi><msub id="S3.E1.m1.2.2.1.1.4.3" xref="S3.E1.m1.2.2.1.1.4.3.cmml"><mi id="S3.E1.m1.2.2.1.1.4.3.2" xref="S3.E1.m1.2.2.1.1.4.3.2.cmml">T</mi><mtext id="S3.E1.m1.2.2.1.1.4.3.3" xref="S3.E1.m1.2.2.1.1.4.3.3a.cmml">extract</mtext></msub></msub></mrow><mo id="S3.E1.m1.2.2.1.2" xref="S3.E1.m1.2.2.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E1.m1.2b"><apply id="S3.E1.m1.2.2.1.1.cmml" xref="S3.E1.m1.2.2.1"><eq id="S3.E1.m1.2.2.1.1.3.cmml" xref="S3.E1.m1.2.2.1.1.3"></eq><apply id="S3.E1.m1.2.2.1.1.2.cmml" xref="S3.E1.m1.2.2.1.1.2"><times id="S3.E1.m1.2.2.1.1.2.3.cmml" xref="S3.E1.m1.2.2.1.1.2.3"></times><ci id="S3.E1.m1.2.2.1.1.2.4a.cmml" xref="S3.E1.m1.2.2.1.1.2.4"><mtext id="S3.E1.m1.2.2.1.1.2.4.cmml" xref="S3.E1.m1.2.2.1.1.2.4">Extract</mtext></ci><list id="S3.E1.m1.2.2.1.1.2.2.3.cmml" xref="S3.E1.m1.2.2.1.1.2.2.2"><apply id="S3.E1.m1.2.2.1.1.1.1.1.1.cmml" xref="S3.E1.m1.2.2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E1.m1.2.2.1.1.1.1.1.1.1.cmml" xref="S3.E1.m1.2.2.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E1.m1.2.2.1.1.1.1.1.1.2.cmml" xref="S3.E1.m1.2.2.1.1.1.1.1.1.2">𝚯</ci><ci id="S3.E1.m1.2.2.1.1.1.1.1.1.3.cmml" xref="S3.E1.m1.2.2.1.1.1.1.1.1.3">𝑇</ci></apply><apply id="S3.E1.m1.2.2.1.1.2.2.2.2.cmml" xref="S3.E1.m1.2.2.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S3.E1.m1.2.2.1.1.2.2.2.2.1.cmml" xref="S3.E1.m1.2.2.1.1.2.2.2.2">subscript</csymbol><ci id="S3.E1.m1.2.2.1.1.2.2.2.2.2.cmml" xref="S3.E1.m1.2.2.1.1.2.2.2.2.2">𝚯</ci><ci id="S3.E1.m1.2.2.1.1.2.2.2.2.3.cmml" xref="S3.E1.m1.2.2.1.1.2.2.2.2.3">𝑆</ci></apply><ci id="S3.E1.m1.1.1.cmml" xref="S3.E1.m1.1.1">𝒯</ci></list></apply><apply id="S3.E1.m1.2.2.1.1.4.cmml" xref="S3.E1.m1.2.2.1.1.4"><csymbol cd="ambiguous" id="S3.E1.m1.2.2.1.1.4.1.cmml" xref="S3.E1.m1.2.2.1.1.4">subscript</csymbol><ci id="S3.E1.m1.2.2.1.1.4.2.cmml" xref="S3.E1.m1.2.2.1.1.4.2">𝚯</ci><apply id="S3.E1.m1.2.2.1.1.4.3.cmml" xref="S3.E1.m1.2.2.1.1.4.3"><csymbol cd="ambiguous" id="S3.E1.m1.2.2.1.1.4.3.1.cmml" xref="S3.E1.m1.2.2.1.1.4.3">subscript</csymbol><ci id="S3.E1.m1.2.2.1.1.4.3.2.cmml" xref="S3.E1.m1.2.2.1.1.4.3.2">𝑇</ci><ci id="S3.E1.m1.2.2.1.1.4.3.3a.cmml" xref="S3.E1.m1.2.2.1.1.4.3.3"><mtext mathsize="50%" id="S3.E1.m1.2.2.1.1.4.3.3.cmml" xref="S3.E1.m1.2.2.1.1.4.3.3">extract</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E1.m1.2c">\text{Extract}(\mathbf{\Theta}_{T};\mathbf{\Theta}_{S};\mathcal{T})=\mathbf{\Theta}_{T_{\text{extract}}},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<p id="S3.SS1.p2.1" class="ltx_p">with <math id="S3.SS1.p2.1.m1.1" class="ltx_Math" alttext="\text{Extract}(\cdot)" display="inline"><semantics id="S3.SS1.p2.1.m1.1a"><mrow id="S3.SS1.p2.1.m1.1.2" xref="S3.SS1.p2.1.m1.1.2.cmml"><mtext id="S3.SS1.p2.1.m1.1.2.2" xref="S3.SS1.p2.1.m1.1.2.2a.cmml">Extract</mtext><mo lspace="0em" rspace="0em" id="S3.SS1.p2.1.m1.1.2.1" xref="S3.SS1.p2.1.m1.1.2.1.cmml">​</mo><mrow id="S3.SS1.p2.1.m1.1.2.3.2" xref="S3.SS1.p2.1.m1.1.2.cmml"><mo stretchy="false" id="S3.SS1.p2.1.m1.1.2.3.2.1" xref="S3.SS1.p2.1.m1.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S3.SS1.p2.1.m1.1.1" xref="S3.SS1.p2.1.m1.1.1.cmml">⋅</mo><mo stretchy="false" id="S3.SS1.p2.1.m1.1.2.3.2.2" xref="S3.SS1.p2.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.1.m1.1b"><apply id="S3.SS1.p2.1.m1.1.2.cmml" xref="S3.SS1.p2.1.m1.1.2"><times id="S3.SS1.p2.1.m1.1.2.1.cmml" xref="S3.SS1.p2.1.m1.1.2.1"></times><ci id="S3.SS1.p2.1.m1.1.2.2a.cmml" xref="S3.SS1.p2.1.m1.1.2.2"><mtext id="S3.SS1.p2.1.m1.1.2.2.cmml" xref="S3.SS1.p2.1.m1.1.2.2">Extract</mtext></ci><ci id="S3.SS1.p2.1.m1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.1.m1.1c">\text{Extract}(\cdot)</annotation></semantics></math> encapsulating the logic for both parameter extraction and downscaling.
Following extraction, the next step is the injection of these extracted parameters into the student model:</p>
<table id="S3.E2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E2.m1.1" class="ltx_Math" alttext="\text{Inject}(\mathbf{\Theta}_{S};\mathbf{\Theta}_{T_{\text{extract}}})=\mathbf{\Theta^{\prime}}_{S}," display="block"><semantics id="S3.E2.m1.1a"><mrow id="S3.E2.m1.1.1.1" xref="S3.E2.m1.1.1.1.1.cmml"><mrow id="S3.E2.m1.1.1.1.1" xref="S3.E2.m1.1.1.1.1.cmml"><mrow id="S3.E2.m1.1.1.1.1.2" xref="S3.E2.m1.1.1.1.1.2.cmml"><mtext id="S3.E2.m1.1.1.1.1.2.4" xref="S3.E2.m1.1.1.1.1.2.4a.cmml">Inject</mtext><mo lspace="0em" rspace="0em" id="S3.E2.m1.1.1.1.1.2.3" xref="S3.E2.m1.1.1.1.1.2.3.cmml">​</mo><mrow id="S3.E2.m1.1.1.1.1.2.2.2" xref="S3.E2.m1.1.1.1.1.2.2.3.cmml"><mo stretchy="false" id="S3.E2.m1.1.1.1.1.2.2.2.3" xref="S3.E2.m1.1.1.1.1.2.2.3.cmml">(</mo><msub id="S3.E2.m1.1.1.1.1.1.1.1.1" xref="S3.E2.m1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E2.m1.1.1.1.1.1.1.1.1.2" xref="S3.E2.m1.1.1.1.1.1.1.1.1.2.cmml">𝚯</mi><mi id="S3.E2.m1.1.1.1.1.1.1.1.1.3" xref="S3.E2.m1.1.1.1.1.1.1.1.1.3.cmml">S</mi></msub><mo id="S3.E2.m1.1.1.1.1.2.2.2.4" xref="S3.E2.m1.1.1.1.1.2.2.3.cmml">;</mo><msub id="S3.E2.m1.1.1.1.1.2.2.2.2" xref="S3.E2.m1.1.1.1.1.2.2.2.2.cmml"><mi id="S3.E2.m1.1.1.1.1.2.2.2.2.2" xref="S3.E2.m1.1.1.1.1.2.2.2.2.2.cmml">𝚯</mi><msub id="S3.E2.m1.1.1.1.1.2.2.2.2.3" xref="S3.E2.m1.1.1.1.1.2.2.2.2.3.cmml"><mi id="S3.E2.m1.1.1.1.1.2.2.2.2.3.2" xref="S3.E2.m1.1.1.1.1.2.2.2.2.3.2.cmml">T</mi><mtext id="S3.E2.m1.1.1.1.1.2.2.2.2.3.3" xref="S3.E2.m1.1.1.1.1.2.2.2.2.3.3a.cmml">extract</mtext></msub></msub><mo stretchy="false" id="S3.E2.m1.1.1.1.1.2.2.2.5" xref="S3.E2.m1.1.1.1.1.2.2.3.cmml">)</mo></mrow></mrow><mo id="S3.E2.m1.1.1.1.1.3" xref="S3.E2.m1.1.1.1.1.3.cmml">=</mo><mmultiscripts id="S3.E2.m1.1.1.1.1.4" xref="S3.E2.m1.1.1.1.1.4.cmml"><mi id="S3.E2.m1.1.1.1.1.4.2.2" xref="S3.E2.m1.1.1.1.1.4.2.2.cmml">𝚯</mi><mrow id="S3.E2.m1.1.1.1.1.4a" xref="S3.E2.m1.1.1.1.1.4.cmml"></mrow><mo id="S3.E2.m1.1.1.1.1.4.2.3" xref="S3.E2.m1.1.1.1.1.4.2.3.cmml">′</mo><mi id="S3.E2.m1.1.1.1.1.4.3" xref="S3.E2.m1.1.1.1.1.4.3.cmml">S</mi><mrow id="S3.E2.m1.1.1.1.1.4b" xref="S3.E2.m1.1.1.1.1.4.cmml"></mrow></mmultiscripts></mrow><mo id="S3.E2.m1.1.1.1.2" xref="S3.E2.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E2.m1.1b"><apply id="S3.E2.m1.1.1.1.1.cmml" xref="S3.E2.m1.1.1.1"><eq id="S3.E2.m1.1.1.1.1.3.cmml" xref="S3.E2.m1.1.1.1.1.3"></eq><apply id="S3.E2.m1.1.1.1.1.2.cmml" xref="S3.E2.m1.1.1.1.1.2"><times id="S3.E2.m1.1.1.1.1.2.3.cmml" xref="S3.E2.m1.1.1.1.1.2.3"></times><ci id="S3.E2.m1.1.1.1.1.2.4a.cmml" xref="S3.E2.m1.1.1.1.1.2.4"><mtext id="S3.E2.m1.1.1.1.1.2.4.cmml" xref="S3.E2.m1.1.1.1.1.2.4">Inject</mtext></ci><list id="S3.E2.m1.1.1.1.1.2.2.3.cmml" xref="S3.E2.m1.1.1.1.1.2.2.2"><apply id="S3.E2.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E2.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1.2">𝚯</ci><ci id="S3.E2.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E2.m1.1.1.1.1.1.1.1.1.3">𝑆</ci></apply><apply id="S3.E2.m1.1.1.1.1.2.2.2.2.cmml" xref="S3.E2.m1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.1.1.2.2.2.2.1.cmml" xref="S3.E2.m1.1.1.1.1.2.2.2.2">subscript</csymbol><ci id="S3.E2.m1.1.1.1.1.2.2.2.2.2.cmml" xref="S3.E2.m1.1.1.1.1.2.2.2.2.2">𝚯</ci><apply id="S3.E2.m1.1.1.1.1.2.2.2.2.3.cmml" xref="S3.E2.m1.1.1.1.1.2.2.2.2.3"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.1.1.2.2.2.2.3.1.cmml" xref="S3.E2.m1.1.1.1.1.2.2.2.2.3">subscript</csymbol><ci id="S3.E2.m1.1.1.1.1.2.2.2.2.3.2.cmml" xref="S3.E2.m1.1.1.1.1.2.2.2.2.3.2">𝑇</ci><ci id="S3.E2.m1.1.1.1.1.2.2.2.2.3.3a.cmml" xref="S3.E2.m1.1.1.1.1.2.2.2.2.3.3"><mtext mathsize="50%" id="S3.E2.m1.1.1.1.1.2.2.2.2.3.3.cmml" xref="S3.E2.m1.1.1.1.1.2.2.2.2.3.3">extract</mtext></ci></apply></apply></list></apply><apply id="S3.E2.m1.1.1.1.1.4.cmml" xref="S3.E2.m1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.1.1.4.1.cmml" xref="S3.E2.m1.1.1.1.1.4">subscript</csymbol><apply id="S3.E2.m1.1.1.1.1.4.2.cmml" xref="S3.E2.m1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.1.1.4.2.1.cmml" xref="S3.E2.m1.1.1.1.1.4">superscript</csymbol><ci id="S3.E2.m1.1.1.1.1.4.2.2.cmml" xref="S3.E2.m1.1.1.1.1.4.2.2">𝚯</ci><ci id="S3.E2.m1.1.1.1.1.4.2.3.cmml" xref="S3.E2.m1.1.1.1.1.4.2.3">′</ci></apply><ci id="S3.E2.m1.1.1.1.1.4.3.cmml" xref="S3.E2.m1.1.1.1.1.4.3">𝑆</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E2.m1.1c">\text{Inject}(\mathbf{\Theta}_{S};\mathbf{\Theta}_{T_{\text{extract}}})=\mathbf{\Theta^{\prime}}_{S},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
<p id="S3.SS1.p2.4" class="ltx_p">yielding a student model now characterized by the modified parameter set <math id="S3.SS1.p2.2.m1.1" class="ltx_Math" alttext="\mathbf{\Theta^{\prime}}_{S}" display="inline"><semantics id="S3.SS1.p2.2.m1.1a"><mmultiscripts id="S3.SS1.p2.2.m1.1.1" xref="S3.SS1.p2.2.m1.1.1.cmml"><mi id="S3.SS1.p2.2.m1.1.1.2.2" xref="S3.SS1.p2.2.m1.1.1.2.2.cmml">𝚯</mi><mrow id="S3.SS1.p2.2.m1.1.1a" xref="S3.SS1.p2.2.m1.1.1.cmml"></mrow><mo id="S3.SS1.p2.2.m1.1.1.2.3" xref="S3.SS1.p2.2.m1.1.1.2.3.cmml">′</mo><mi id="S3.SS1.p2.2.m1.1.1.3" xref="S3.SS1.p2.2.m1.1.1.3.cmml">S</mi><mrow id="S3.SS1.p2.2.m1.1.1b" xref="S3.SS1.p2.2.m1.1.1.cmml"></mrow></mmultiscripts><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.2.m1.1b"><apply id="S3.SS1.p2.2.m1.1.1.cmml" xref="S3.SS1.p2.2.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.2.m1.1.1.1.cmml" xref="S3.SS1.p2.2.m1.1.1">subscript</csymbol><apply id="S3.SS1.p2.2.m1.1.1.2.cmml" xref="S3.SS1.p2.2.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.2.m1.1.1.2.1.cmml" xref="S3.SS1.p2.2.m1.1.1">superscript</csymbol><ci id="S3.SS1.p2.2.m1.1.1.2.2.cmml" xref="S3.SS1.p2.2.m1.1.1.2.2">𝚯</ci><ci id="S3.SS1.p2.2.m1.1.1.2.3.cmml" xref="S3.SS1.p2.2.m1.1.1.2.3">′</ci></apply><ci id="S3.SS1.p2.2.m1.1.1.3.cmml" xref="S3.SS1.p2.2.m1.1.1.3">𝑆</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.2.m1.1c">\mathbf{\Theta^{\prime}}_{S}</annotation></semantics></math>.
Upon completing the knowledge injection, there remains an optional phase wherein the student model fine-tunes the newly incorporated parameters <math id="S3.SS1.p2.3.m2.1" class="ltx_Math" alttext="\mathbf{\Theta^{\prime}}_{S}" display="inline"><semantics id="S3.SS1.p2.3.m2.1a"><mmultiscripts id="S3.SS1.p2.3.m2.1.1" xref="S3.SS1.p2.3.m2.1.1.cmml"><mi id="S3.SS1.p2.3.m2.1.1.2.2" xref="S3.SS1.p2.3.m2.1.1.2.2.cmml">𝚯</mi><mrow id="S3.SS1.p2.3.m2.1.1a" xref="S3.SS1.p2.3.m2.1.1.cmml"></mrow><mo id="S3.SS1.p2.3.m2.1.1.2.3" xref="S3.SS1.p2.3.m2.1.1.2.3.cmml">′</mo><mi id="S3.SS1.p2.3.m2.1.1.3" xref="S3.SS1.p2.3.m2.1.1.3.cmml">S</mi><mrow id="S3.SS1.p2.3.m2.1.1b" xref="S3.SS1.p2.3.m2.1.1.cmml"></mrow></mmultiscripts><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.3.m2.1b"><apply id="S3.SS1.p2.3.m2.1.1.cmml" xref="S3.SS1.p2.3.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.3.m2.1.1.1.cmml" xref="S3.SS1.p2.3.m2.1.1">subscript</csymbol><apply id="S3.SS1.p2.3.m2.1.1.2.cmml" xref="S3.SS1.p2.3.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.3.m2.1.1.2.1.cmml" xref="S3.SS1.p2.3.m2.1.1">superscript</csymbol><ci id="S3.SS1.p2.3.m2.1.1.2.2.cmml" xref="S3.SS1.p2.3.m2.1.1.2.2">𝚯</ci><ci id="S3.SS1.p2.3.m2.1.1.2.3.cmml" xref="S3.SS1.p2.3.m2.1.1.2.3">′</ci></apply><ci id="S3.SS1.p2.3.m2.1.1.3.cmml" xref="S3.SS1.p2.3.m2.1.1.3">𝑆</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.3.m2.1c">\mathbf{\Theta^{\prime}}_{S}</annotation></semantics></math> with respect to the task <math id="S3.SS1.p2.4.m3.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="S3.SS1.p2.4.m3.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p2.4.m3.1.1" xref="S3.SS1.p2.4.m3.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.4.m3.1b"><ci id="S3.SS1.p2.4.m3.1.1.cmml" xref="S3.SS1.p2.4.m3.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.4.m3.1c">\mathcal{T}</annotation></semantics></math>.</p>
</div>
<figure id="S3.F2" class="ltx_figure"><img src="/html/2310.11451/assets/x2.png" id="S3.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="415" height="174" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>Overview of our parametric knowledge transfer framework. Starting with the teacher model, we compute sensitivity metrics using a set of seed samples, which aids in the extraction of task-specific knowledge. Subsequently, the extracted parameter matrices are factorized to initialize the student model’s LoRA module, serving as a bridge for knowledge injection.</figcaption>
</figure>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Knowledge Extraction</h3>

<div id="S3.SS2.p1" class="ltx_para ltx_noindent">
<p id="S3.SS2.p1.1" class="ltx_p">In implementing our <math id="S3.SS2.p1.1.m1.1" class="ltx_Math" alttext="\text{Extract}(\cdot)" display="inline"><semantics id="S3.SS2.p1.1.m1.1a"><mrow id="S3.SS2.p1.1.m1.1.2" xref="S3.SS2.p1.1.m1.1.2.cmml"><mtext id="S3.SS2.p1.1.m1.1.2.2" xref="S3.SS2.p1.1.m1.1.2.2a.cmml">Extract</mtext><mo lspace="0em" rspace="0em" id="S3.SS2.p1.1.m1.1.2.1" xref="S3.SS2.p1.1.m1.1.2.1.cmml">​</mo><mrow id="S3.SS2.p1.1.m1.1.2.3.2" xref="S3.SS2.p1.1.m1.1.2.cmml"><mo stretchy="false" id="S3.SS2.p1.1.m1.1.2.3.2.1" xref="S3.SS2.p1.1.m1.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S3.SS2.p1.1.m1.1.1" xref="S3.SS2.p1.1.m1.1.1.cmml">⋅</mo><mo stretchy="false" id="S3.SS2.p1.1.m1.1.2.3.2.2" xref="S3.SS2.p1.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.1.m1.1b"><apply id="S3.SS2.p1.1.m1.1.2.cmml" xref="S3.SS2.p1.1.m1.1.2"><times id="S3.SS2.p1.1.m1.1.2.1.cmml" xref="S3.SS2.p1.1.m1.1.2.1"></times><ci id="S3.SS2.p1.1.m1.1.2.2a.cmml" xref="S3.SS2.p1.1.m1.1.2.2"><mtext id="S3.SS2.p1.1.m1.1.2.2.cmml" xref="S3.SS2.p1.1.m1.1.2.2">Extract</mtext></ci><ci id="S3.SS2.p1.1.m1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.1.m1.1c">\text{Extract}(\cdot)</annotation></semantics></math> function, we employ parameter sensitivity as the foundational metric to guide the extraction process.</p>
</div>
<section id="S3.SS2.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Sensitivity of the Parameters.</h4>

<div id="S3.SS2.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS2.SSS0.Px1.p1.10" class="ltx_p">Parameter sensitivity serves as a mechanism to measure the variation in the loss upon setting a particular parameter to zero <cite class="ltx_cite ltx_citemacro_citep">(Mozer &amp; Smolensky, <a href="#bib.bib41" title="" class="ltx_ref">1988</a>; Lee et al., <a href="#bib.bib32" title="" class="ltx_ref">2019</a>; Lubana &amp; Dick, <a href="#bib.bib35" title="" class="ltx_ref">2021</a>; Liang et al., <a href="#bib.bib34" title="" class="ltx_ref">2022</a>)</cite>. When this removal elicits a significant shift in the loss, such a parameter is deemed to be of high sensitivity. For a teacher model <math id="S3.SS2.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="M_{T}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p1.1.m1.1a"><msub id="S3.SS2.SSS0.Px1.p1.1.m1.1.1" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.1.cmml"><mi id="S3.SS2.SSS0.Px1.p1.1.m1.1.1.2" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.1.2.cmml">M</mi><mi id="S3.SS2.SSS0.Px1.p1.1.m1.1.1.3" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.1.3.cmml">T</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p1.1.m1.1b"><apply id="S3.SS2.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.1.2">𝑀</ci><ci id="S3.SS2.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p1.1.m1.1c">M_{T}</annotation></semantics></math> with parameters <math id="S3.SS2.SSS0.Px1.p1.2.m2.3" class="ltx_Math" alttext="\mathbf{\Theta}_{T}=[\theta_{1},\dots,\theta_{N_{T}}]" display="inline"><semantics id="S3.SS2.SSS0.Px1.p1.2.m2.3a"><mrow id="S3.SS2.SSS0.Px1.p1.2.m2.3.3" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.cmml"><msub id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.4" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.4.cmml"><mi id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.4.2" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.4.2.cmml">𝚯</mi><mi id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.4.3" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.4.3.cmml">T</mi></msub><mo id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.3" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.3.cmml">=</mo><mrow id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.3.cmml"><mo stretchy="false" id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.3" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.3.cmml">[</mo><msub id="S3.SS2.SSS0.Px1.p1.2.m2.2.2.1.1.1" xref="S3.SS2.SSS0.Px1.p1.2.m2.2.2.1.1.1.cmml"><mi id="S3.SS2.SSS0.Px1.p1.2.m2.2.2.1.1.1.2" xref="S3.SS2.SSS0.Px1.p1.2.m2.2.2.1.1.1.2.cmml">θ</mi><mn id="S3.SS2.SSS0.Px1.p1.2.m2.2.2.1.1.1.3" xref="S3.SS2.SSS0.Px1.p1.2.m2.2.2.1.1.1.3.cmml">1</mn></msub><mo id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.4" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="S3.SS2.SSS0.Px1.p1.2.m2.1.1" xref="S3.SS2.SSS0.Px1.p1.2.m2.1.1.cmml">…</mi><mo id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.5" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.3.cmml">,</mo><msub id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2.cmml"><mi id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2.2" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2.2.cmml">θ</mi><msub id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2.3" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2.3.cmml"><mi id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2.3.2" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2.3.2.cmml">N</mi><mi id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2.3.3" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2.3.3.cmml">T</mi></msub></msub><mo stretchy="false" id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.6" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.3.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p1.2.m2.3b"><apply id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3"><eq id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.3.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.3"></eq><apply id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.4.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.4"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.4.1.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.4">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.4.2.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.4.2">𝚯</ci><ci id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.4.3.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.4.3">𝑇</ci></apply><list id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.3.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2"><apply id="S3.SS2.SSS0.Px1.p1.2.m2.2.2.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.2.m2.2.2.1.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.2.2.1.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.2.m2.2.2.1.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.2.2.1.1.1.2">𝜃</ci><cn type="integer" id="S3.SS2.SSS0.Px1.p1.2.m2.2.2.1.1.1.3.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.2.2.1.1.1.3">1</cn></apply><ci id="S3.SS2.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.1.1">…</ci><apply id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2.1.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2.2.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2.2">𝜃</ci><apply id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2.3.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2.3.1.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2.3.2.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2.3.2">𝑁</ci><ci id="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2.3.3.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.3.3.2.2.2.3.3">𝑇</ci></apply></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p1.2.m2.3c">\mathbf{\Theta}_{T}=[\theta_{1},\dots,\theta_{N_{T}}]</annotation></semantics></math>, where <math id="S3.SS2.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="N_{T}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p1.3.m3.1a"><msub id="S3.SS2.SSS0.Px1.p1.3.m3.1.1" xref="S3.SS2.SSS0.Px1.p1.3.m3.1.1.cmml"><mi id="S3.SS2.SSS0.Px1.p1.3.m3.1.1.2" xref="S3.SS2.SSS0.Px1.p1.3.m3.1.1.2.cmml">N</mi><mi id="S3.SS2.SSS0.Px1.p1.3.m3.1.1.3" xref="S3.SS2.SSS0.Px1.p1.3.m3.1.1.3.cmml">T</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p1.3.m3.1b"><apply id="S3.SS2.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.3.m3.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.3.m3.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.3.m3.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p1.3.m3.1.1.2">𝑁</ci><ci id="S3.SS2.SSS0.Px1.p1.3.m3.1.1.3.cmml" xref="S3.SS2.SSS0.Px1.p1.3.m3.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p1.3.m3.1c">N_{T}</annotation></semantics></math> represents the total number of parameters, the <math id="S3.SS2.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S3.SS2.SSS0.Px1.p1.4.m4.1a"><mi id="S3.SS2.SSS0.Px1.p1.4.m4.1.1" xref="S3.SS2.SSS0.Px1.p1.4.m4.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p1.4.m4.1b"><ci id="S3.SS2.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.4.m4.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p1.4.m4.1c">i</annotation></semantics></math>-th parameter can be expressed as <math id="S3.SS2.SSS0.Px1.p1.5.m5.5" class="ltx_Math" alttext="\mathbf{\Theta}_{T_{i}}=[0,\dots,\theta_{i},\dots,0]" display="inline"><semantics id="S3.SS2.SSS0.Px1.p1.5.m5.5a"><mrow id="S3.SS2.SSS0.Px1.p1.5.m5.5.5" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.cmml"><msub id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3.cmml"><mi id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3.2" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3.2.cmml">𝚯</mi><msub id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3.3" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3.3.cmml"><mi id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3.3.2" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3.3.2.cmml">T</mi><mi id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3.3.3" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3.3.3.cmml">i</mi></msub></msub><mo id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.2" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.2.cmml">=</mo><mrow id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.2.cmml"><mo stretchy="false" id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1.2" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.2.cmml">[</mo><mn id="S3.SS2.SSS0.Px1.p1.5.m5.1.1" xref="S3.SS2.SSS0.Px1.p1.5.m5.1.1.cmml">0</mn><mo id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1.3" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.2.cmml">,</mo><mi mathvariant="normal" id="S3.SS2.SSS0.Px1.p1.5.m5.2.2" xref="S3.SS2.SSS0.Px1.p1.5.m5.2.2.cmml">…</mi><mo id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1.4" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.2.cmml">,</mo><msub id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1.1" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1.1.cmml"><mi id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1.1.2" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1.1.2.cmml">θ</mi><mi id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1.1.3" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1.1.3.cmml">i</mi></msub><mo id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1.5" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.2.cmml">,</mo><mi mathvariant="normal" id="S3.SS2.SSS0.Px1.p1.5.m5.3.3" xref="S3.SS2.SSS0.Px1.p1.5.m5.3.3.cmml">…</mi><mo id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1.6" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.2.cmml">,</mo><mn id="S3.SS2.SSS0.Px1.p1.5.m5.4.4" xref="S3.SS2.SSS0.Px1.p1.5.m5.4.4.cmml">0</mn><mo stretchy="false" id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1.7" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.2.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p1.5.m5.5b"><apply id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5"><eq id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.2.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.2"></eq><apply id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3.1.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3.2.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3.2">𝚯</ci><apply id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3.3.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3.3.1.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3.3.2.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3.3.2">𝑇</ci><ci id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3.3.3.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.3.3.3">𝑖</ci></apply></apply><list id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.2.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1"><cn type="integer" id="S3.SS2.SSS0.Px1.p1.5.m5.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m5.1.1">0</cn><ci id="S3.SS2.SSS0.Px1.p1.5.m5.2.2.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m5.2.2">…</ci><apply id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1.1.2">𝜃</ci><ci id="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1.1.3.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m5.5.5.1.1.1.3">𝑖</ci></apply><ci id="S3.SS2.SSS0.Px1.p1.5.m5.3.3.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m5.3.3">…</ci><cn type="integer" id="S3.SS2.SSS0.Px1.p1.5.m5.4.4.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m5.4.4">0</cn></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p1.5.m5.5c">\mathbf{\Theta}_{T_{i}}=[0,\dots,\theta_{i},\dots,0]</annotation></semantics></math>. With gradients of the loss relative to <math id="S3.SS2.SSS0.Px1.p1.6.m6.1" class="ltx_Math" alttext="\mathbf{\Theta}_{T}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p1.6.m6.1a"><msub id="S3.SS2.SSS0.Px1.p1.6.m6.1.1" xref="S3.SS2.SSS0.Px1.p1.6.m6.1.1.cmml"><mi id="S3.SS2.SSS0.Px1.p1.6.m6.1.1.2" xref="S3.SS2.SSS0.Px1.p1.6.m6.1.1.2.cmml">𝚯</mi><mi id="S3.SS2.SSS0.Px1.p1.6.m6.1.1.3" xref="S3.SS2.SSS0.Px1.p1.6.m6.1.1.3.cmml">T</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p1.6.m6.1b"><apply id="S3.SS2.SSS0.Px1.p1.6.m6.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.6.m6.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.6.m6.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.6.m6.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p1.6.m6.1.1.2">𝚯</ci><ci id="S3.SS2.SSS0.Px1.p1.6.m6.1.1.3.cmml" xref="S3.SS2.SSS0.Px1.p1.6.m6.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p1.6.m6.1c">\mathbf{\Theta}_{T}</annotation></semantics></math> represented as <math id="S3.SS2.SSS0.Px1.p1.7.m7.1" class="ltx_Math" alttext="\nabla_{\mathbf{\Theta}_{T}}\mathcal{L}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p1.7.m7.1a"><mrow id="S3.SS2.SSS0.Px1.p1.7.m7.1.1" xref="S3.SS2.SSS0.Px1.p1.7.m7.1.1.cmml"><msub id="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1" xref="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1.cmml"><mo id="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1.2" xref="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1.2.cmml">∇</mo><msub id="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1.3" xref="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1.3.cmml"><mi id="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1.3.2" xref="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1.3.2.cmml">𝚯</mi><mi id="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1.3.3" xref="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1.3.3.cmml">T</mi></msub></msub><mi class="ltx_font_mathcaligraphic" id="S3.SS2.SSS0.Px1.p1.7.m7.1.1.2" xref="S3.SS2.SSS0.Px1.p1.7.m7.1.1.2.cmml">ℒ</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p1.7.m7.1b"><apply id="S3.SS2.SSS0.Px1.p1.7.m7.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.7.m7.1.1"><apply id="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1.2">∇</ci><apply id="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1.3.cmml" xref="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1.3.1.cmml" xref="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1.3.2.cmml" xref="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1.3.2">𝚯</ci><ci id="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1.3.3.cmml" xref="S3.SS2.SSS0.Px1.p1.7.m7.1.1.1.3.3">𝑇</ci></apply></apply><ci id="S3.SS2.SSS0.Px1.p1.7.m7.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p1.7.m7.1.1.2">ℒ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p1.7.m7.1c">\nabla_{\mathbf{\Theta}_{T}}\mathcal{L}</annotation></semantics></math>, the sensitivity of the <math id="S3.SS2.SSS0.Px1.p1.8.m8.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S3.SS2.SSS0.Px1.p1.8.m8.1a"><mi id="S3.SS2.SSS0.Px1.p1.8.m8.1.1" xref="S3.SS2.SSS0.Px1.p1.8.m8.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p1.8.m8.1b"><ci id="S3.SS2.SSS0.Px1.p1.8.m8.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.8.m8.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p1.8.m8.1c">i</annotation></semantics></math>-th parameter for a specific sample <math id="S3.SS2.SSS0.Px1.p1.9.m9.1" class="ltx_Math" alttext="x_{j}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p1.9.m9.1a"><msub id="S3.SS2.SSS0.Px1.p1.9.m9.1.1" xref="S3.SS2.SSS0.Px1.p1.9.m9.1.1.cmml"><mi id="S3.SS2.SSS0.Px1.p1.9.m9.1.1.2" xref="S3.SS2.SSS0.Px1.p1.9.m9.1.1.2.cmml">x</mi><mi id="S3.SS2.SSS0.Px1.p1.9.m9.1.1.3" xref="S3.SS2.SSS0.Px1.p1.9.m9.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p1.9.m9.1b"><apply id="S3.SS2.SSS0.Px1.p1.9.m9.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.9.m9.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.9.m9.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.9.m9.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.9.m9.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p1.9.m9.1.1.2">𝑥</ci><ci id="S3.SS2.SSS0.Px1.p1.9.m9.1.1.3.cmml" xref="S3.SS2.SSS0.Px1.p1.9.m9.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p1.9.m9.1c">x_{j}</annotation></semantics></math> from task <math id="S3.SS2.SSS0.Px1.p1.10.m10.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p1.10.m10.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.SSS0.Px1.p1.10.m10.1.1" xref="S3.SS2.SSS0.Px1.p1.10.m10.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p1.10.m10.1b"><ci id="S3.SS2.SSS0.Px1.p1.10.m10.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.10.m10.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p1.10.m10.1c">\mathcal{T}</annotation></semantics></math> is determined as:</p>
</div>
<div id="S3.SS2.SSS0.Px1.p2" class="ltx_para ltx_noindent">
<table id="S3.E3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E3.m1.3" class="ltx_Math" alttext="S_{i,j}=\left|\mathbf{\Theta}_{T_{i}}^{\top}\nabla_{\mathbf{\Theta}_{T}}\mathcal{L}(x_{j})\right|." display="block"><semantics id="S3.E3.m1.3a"><mrow id="S3.E3.m1.3.3.1" xref="S3.E3.m1.3.3.1.1.cmml"><mrow id="S3.E3.m1.3.3.1.1" xref="S3.E3.m1.3.3.1.1.cmml"><msub id="S3.E3.m1.3.3.1.1.3" xref="S3.E3.m1.3.3.1.1.3.cmml"><mi id="S3.E3.m1.3.3.1.1.3.2" xref="S3.E3.m1.3.3.1.1.3.2.cmml">S</mi><mrow id="S3.E3.m1.2.2.2.4" xref="S3.E3.m1.2.2.2.3.cmml"><mi id="S3.E3.m1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.cmml">i</mi><mo id="S3.E3.m1.2.2.2.4.1" xref="S3.E3.m1.2.2.2.3.cmml">,</mo><mi id="S3.E3.m1.2.2.2.2" xref="S3.E3.m1.2.2.2.2.cmml">j</mi></mrow></msub><mo id="S3.E3.m1.3.3.1.1.2" xref="S3.E3.m1.3.3.1.1.2.cmml">=</mo><mrow id="S3.E3.m1.3.3.1.1.1.1" xref="S3.E3.m1.3.3.1.1.1.2.cmml"><mo id="S3.E3.m1.3.3.1.1.1.1.2" xref="S3.E3.m1.3.3.1.1.1.2.1.cmml">|</mo><mrow id="S3.E3.m1.3.3.1.1.1.1.1" xref="S3.E3.m1.3.3.1.1.1.1.1.cmml"><msubsup id="S3.E3.m1.3.3.1.1.1.1.1.3" xref="S3.E3.m1.3.3.1.1.1.1.1.3.cmml"><mi id="S3.E3.m1.3.3.1.1.1.1.1.3.2.2" xref="S3.E3.m1.3.3.1.1.1.1.1.3.2.2.cmml">𝚯</mi><msub id="S3.E3.m1.3.3.1.1.1.1.1.3.2.3" xref="S3.E3.m1.3.3.1.1.1.1.1.3.2.3.cmml"><mi id="S3.E3.m1.3.3.1.1.1.1.1.3.2.3.2" xref="S3.E3.m1.3.3.1.1.1.1.1.3.2.3.2.cmml">T</mi><mi id="S3.E3.m1.3.3.1.1.1.1.1.3.2.3.3" xref="S3.E3.m1.3.3.1.1.1.1.1.3.2.3.3.cmml">i</mi></msub><mo id="S3.E3.m1.3.3.1.1.1.1.1.3.3" xref="S3.E3.m1.3.3.1.1.1.1.1.3.3.cmml">⊤</mo></msubsup><mo lspace="0.167em" rspace="0em" id="S3.E3.m1.3.3.1.1.1.1.1.2" xref="S3.E3.m1.3.3.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.E3.m1.3.3.1.1.1.1.1.4" xref="S3.E3.m1.3.3.1.1.1.1.1.4.cmml"><msub id="S3.E3.m1.3.3.1.1.1.1.1.4.1" xref="S3.E3.m1.3.3.1.1.1.1.1.4.1.cmml"><mo rspace="0.167em" id="S3.E3.m1.3.3.1.1.1.1.1.4.1.2" xref="S3.E3.m1.3.3.1.1.1.1.1.4.1.2.cmml">∇</mo><msub id="S3.E3.m1.3.3.1.1.1.1.1.4.1.3" xref="S3.E3.m1.3.3.1.1.1.1.1.4.1.3.cmml"><mi id="S3.E3.m1.3.3.1.1.1.1.1.4.1.3.2" xref="S3.E3.m1.3.3.1.1.1.1.1.4.1.3.2.cmml">𝚯</mi><mi id="S3.E3.m1.3.3.1.1.1.1.1.4.1.3.3" xref="S3.E3.m1.3.3.1.1.1.1.1.4.1.3.3.cmml">T</mi></msub></msub><mi class="ltx_font_mathcaligraphic" id="S3.E3.m1.3.3.1.1.1.1.1.4.2" xref="S3.E3.m1.3.3.1.1.1.1.1.4.2.cmml">ℒ</mi></mrow><mo lspace="0em" rspace="0em" id="S3.E3.m1.3.3.1.1.1.1.1.2a" xref="S3.E3.m1.3.3.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.E3.m1.3.3.1.1.1.1.1.1.1" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E3.m1.3.3.1.1.1.1.1.1.1.2" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="S3.E3.m1.3.3.1.1.1.1.1.1.1.1" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E3.m1.3.3.1.1.1.1.1.1.1.1.2" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="S3.E3.m1.3.3.1.1.1.1.1.1.1.1.3" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.1.3.cmml">j</mi></msub><mo stretchy="false" id="S3.E3.m1.3.3.1.1.1.1.1.1.1.3" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.E3.m1.3.3.1.1.1.1.3" xref="S3.E3.m1.3.3.1.1.1.2.1.cmml">|</mo></mrow></mrow><mo lspace="0em" id="S3.E3.m1.3.3.1.2" xref="S3.E3.m1.3.3.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E3.m1.3b"><apply id="S3.E3.m1.3.3.1.1.cmml" xref="S3.E3.m1.3.3.1"><eq id="S3.E3.m1.3.3.1.1.2.cmml" xref="S3.E3.m1.3.3.1.1.2"></eq><apply id="S3.E3.m1.3.3.1.1.3.cmml" xref="S3.E3.m1.3.3.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.3.3.1.1.3.1.cmml" xref="S3.E3.m1.3.3.1.1.3">subscript</csymbol><ci id="S3.E3.m1.3.3.1.1.3.2.cmml" xref="S3.E3.m1.3.3.1.1.3.2">𝑆</ci><list id="S3.E3.m1.2.2.2.3.cmml" xref="S3.E3.m1.2.2.2.4"><ci id="S3.E3.m1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1.1">𝑖</ci><ci id="S3.E3.m1.2.2.2.2.cmml" xref="S3.E3.m1.2.2.2.2">𝑗</ci></list></apply><apply id="S3.E3.m1.3.3.1.1.1.2.cmml" xref="S3.E3.m1.3.3.1.1.1.1"><abs id="S3.E3.m1.3.3.1.1.1.2.1.cmml" xref="S3.E3.m1.3.3.1.1.1.1.2"></abs><apply id="S3.E3.m1.3.3.1.1.1.1.1.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1"><times id="S3.E3.m1.3.3.1.1.1.1.1.2.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.2"></times><apply id="S3.E3.m1.3.3.1.1.1.1.1.3.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.3.3.1.1.1.1.1.3.1.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.3">superscript</csymbol><apply id="S3.E3.m1.3.3.1.1.1.1.1.3.2.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.3.3.1.1.1.1.1.3.2.1.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E3.m1.3.3.1.1.1.1.1.3.2.2.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.3.2.2">𝚯</ci><apply id="S3.E3.m1.3.3.1.1.1.1.1.3.2.3.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.3.2.3"><csymbol cd="ambiguous" id="S3.E3.m1.3.3.1.1.1.1.1.3.2.3.1.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.3.2.3">subscript</csymbol><ci id="S3.E3.m1.3.3.1.1.1.1.1.3.2.3.2.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.3.2.3.2">𝑇</ci><ci id="S3.E3.m1.3.3.1.1.1.1.1.3.2.3.3.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.3.2.3.3">𝑖</ci></apply></apply><csymbol cd="latexml" id="S3.E3.m1.3.3.1.1.1.1.1.3.3.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.3.3">top</csymbol></apply><apply id="S3.E3.m1.3.3.1.1.1.1.1.4.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.4"><apply id="S3.E3.m1.3.3.1.1.1.1.1.4.1.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.4.1"><csymbol cd="ambiguous" id="S3.E3.m1.3.3.1.1.1.1.1.4.1.1.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.4.1">subscript</csymbol><ci id="S3.E3.m1.3.3.1.1.1.1.1.4.1.2.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.4.1.2">∇</ci><apply id="S3.E3.m1.3.3.1.1.1.1.1.4.1.3.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.4.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.3.3.1.1.1.1.1.4.1.3.1.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.4.1.3">subscript</csymbol><ci id="S3.E3.m1.3.3.1.1.1.1.1.4.1.3.2.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.4.1.3.2">𝚯</ci><ci id="S3.E3.m1.3.3.1.1.1.1.1.4.1.3.3.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.4.1.3.3">𝑇</ci></apply></apply><ci id="S3.E3.m1.3.3.1.1.1.1.1.4.2.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.4.2">ℒ</ci></apply><apply id="S3.E3.m1.3.3.1.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E3.m1.3.3.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E3.m1.3.3.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="S3.E3.m1.3.3.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E3.m1.3.3.1.1.1.1.1.1.1.1.3">𝑗</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3.m1.3c">S_{i,j}=\left|\mathbf{\Theta}_{T_{i}}^{\top}\nabla_{\mathbf{\Theta}_{T}}\mathcal{L}(x_{j})\right|.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
</div>
<div id="S3.SS2.SSS0.Px1.p3" class="ltx_para ltx_noindent">
<p id="S3.SS2.SSS0.Px1.p3.5" class="ltx_p">The rationale behind this sensitivity definition stems from the first-order Taylor expansion of <math id="S3.SS2.SSS0.Px1.p3.1.m1.1" class="ltx_Math" alttext="\mathcal{L}(\cdot)" display="inline"><semantics id="S3.SS2.SSS0.Px1.p3.1.m1.1a"><mrow id="S3.SS2.SSS0.Px1.p3.1.m1.1.2" xref="S3.SS2.SSS0.Px1.p3.1.m1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.SSS0.Px1.p3.1.m1.1.2.2" xref="S3.SS2.SSS0.Px1.p3.1.m1.1.2.2.cmml">ℒ</mi><mo lspace="0em" rspace="0em" id="S3.SS2.SSS0.Px1.p3.1.m1.1.2.1" xref="S3.SS2.SSS0.Px1.p3.1.m1.1.2.1.cmml">​</mo><mrow id="S3.SS2.SSS0.Px1.p3.1.m1.1.2.3.2" xref="S3.SS2.SSS0.Px1.p3.1.m1.1.2.cmml"><mo stretchy="false" id="S3.SS2.SSS0.Px1.p3.1.m1.1.2.3.2.1" xref="S3.SS2.SSS0.Px1.p3.1.m1.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S3.SS2.SSS0.Px1.p3.1.m1.1.1" xref="S3.SS2.SSS0.Px1.p3.1.m1.1.1.cmml">⋅</mo><mo stretchy="false" id="S3.SS2.SSS0.Px1.p3.1.m1.1.2.3.2.2" xref="S3.SS2.SSS0.Px1.p3.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p3.1.m1.1b"><apply id="S3.SS2.SSS0.Px1.p3.1.m1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p3.1.m1.1.2"><times id="S3.SS2.SSS0.Px1.p3.1.m1.1.2.1.cmml" xref="S3.SS2.SSS0.Px1.p3.1.m1.1.2.1"></times><ci id="S3.SS2.SSS0.Px1.p3.1.m1.1.2.2.cmml" xref="S3.SS2.SSS0.Px1.p3.1.m1.1.2.2">ℒ</ci><ci id="S3.SS2.SSS0.Px1.p3.1.m1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p3.1.m1.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p3.1.m1.1c">\mathcal{L}(\cdot)</annotation></semantics></math> relative to <math id="S3.SS2.SSS0.Px1.p3.2.m2.1" class="ltx_Math" alttext="\theta_{i}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p3.2.m2.1a"><msub id="S3.SS2.SSS0.Px1.p3.2.m2.1.1" xref="S3.SS2.SSS0.Px1.p3.2.m2.1.1.cmml"><mi id="S3.SS2.SSS0.Px1.p3.2.m2.1.1.2" xref="S3.SS2.SSS0.Px1.p3.2.m2.1.1.2.cmml">θ</mi><mi id="S3.SS2.SSS0.Px1.p3.2.m2.1.1.3" xref="S3.SS2.SSS0.Px1.p3.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p3.2.m2.1b"><apply id="S3.SS2.SSS0.Px1.p3.2.m2.1.1.cmml" xref="S3.SS2.SSS0.Px1.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p3.2.m2.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p3.2.m2.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p3.2.m2.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p3.2.m2.1.1.2">𝜃</ci><ci id="S3.SS2.SSS0.Px1.p3.2.m2.1.1.3.cmml" xref="S3.SS2.SSS0.Px1.p3.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p3.2.m2.1c">\theta_{i}</annotation></semantics></math> at <math id="S3.SS2.SSS0.Px1.p3.3.m3.1" class="ltx_Math" alttext="\mathbf{\Theta}_{T_{i}}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p3.3.m3.1a"><msub id="S3.SS2.SSS0.Px1.p3.3.m3.1.1" xref="S3.SS2.SSS0.Px1.p3.3.m3.1.1.cmml"><mi id="S3.SS2.SSS0.Px1.p3.3.m3.1.1.2" xref="S3.SS2.SSS0.Px1.p3.3.m3.1.1.2.cmml">𝚯</mi><msub id="S3.SS2.SSS0.Px1.p3.3.m3.1.1.3" xref="S3.SS2.SSS0.Px1.p3.3.m3.1.1.3.cmml"><mi id="S3.SS2.SSS0.Px1.p3.3.m3.1.1.3.2" xref="S3.SS2.SSS0.Px1.p3.3.m3.1.1.3.2.cmml">T</mi><mi id="S3.SS2.SSS0.Px1.p3.3.m3.1.1.3.3" xref="S3.SS2.SSS0.Px1.p3.3.m3.1.1.3.3.cmml">i</mi></msub></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p3.3.m3.1b"><apply id="S3.SS2.SSS0.Px1.p3.3.m3.1.1.cmml" xref="S3.SS2.SSS0.Px1.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p3.3.m3.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p3.3.m3.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p3.3.m3.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p3.3.m3.1.1.2">𝚯</ci><apply id="S3.SS2.SSS0.Px1.p3.3.m3.1.1.3.cmml" xref="S3.SS2.SSS0.Px1.p3.3.m3.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p3.3.m3.1.1.3.1.cmml" xref="S3.SS2.SSS0.Px1.p3.3.m3.1.1.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p3.3.m3.1.1.3.2.cmml" xref="S3.SS2.SSS0.Px1.p3.3.m3.1.1.3.2">𝑇</ci><ci id="S3.SS2.SSS0.Px1.p3.3.m3.1.1.3.3.cmml" xref="S3.SS2.SSS0.Px1.p3.3.m3.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p3.3.m3.1c">\mathbf{\Theta}_{T_{i}}</annotation></semantics></math> <cite class="ltx_cite ltx_citemacro_citep">(Molchanov et al., <a href="#bib.bib40" title="" class="ltx_ref">2017</a>)</cite>. In essence, <math id="S3.SS2.SSS0.Px1.p3.4.m4.2" class="ltx_Math" alttext="S_{i,j}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p3.4.m4.2a"><msub id="S3.SS2.SSS0.Px1.p3.4.m4.2.3" xref="S3.SS2.SSS0.Px1.p3.4.m4.2.3.cmml"><mi id="S3.SS2.SSS0.Px1.p3.4.m4.2.3.2" xref="S3.SS2.SSS0.Px1.p3.4.m4.2.3.2.cmml">S</mi><mrow id="S3.SS2.SSS0.Px1.p3.4.m4.2.2.2.4" xref="S3.SS2.SSS0.Px1.p3.4.m4.2.2.2.3.cmml"><mi id="S3.SS2.SSS0.Px1.p3.4.m4.1.1.1.1" xref="S3.SS2.SSS0.Px1.p3.4.m4.1.1.1.1.cmml">i</mi><mo id="S3.SS2.SSS0.Px1.p3.4.m4.2.2.2.4.1" xref="S3.SS2.SSS0.Px1.p3.4.m4.2.2.2.3.cmml">,</mo><mi id="S3.SS2.SSS0.Px1.p3.4.m4.2.2.2.2" xref="S3.SS2.SSS0.Px1.p3.4.m4.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p3.4.m4.2b"><apply id="S3.SS2.SSS0.Px1.p3.4.m4.2.3.cmml" xref="S3.SS2.SSS0.Px1.p3.4.m4.2.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p3.4.m4.2.3.1.cmml" xref="S3.SS2.SSS0.Px1.p3.4.m4.2.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p3.4.m4.2.3.2.cmml" xref="S3.SS2.SSS0.Px1.p3.4.m4.2.3.2">𝑆</ci><list id="S3.SS2.SSS0.Px1.p3.4.m4.2.2.2.3.cmml" xref="S3.SS2.SSS0.Px1.p3.4.m4.2.2.2.4"><ci id="S3.SS2.SSS0.Px1.p3.4.m4.1.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p3.4.m4.1.1.1.1">𝑖</ci><ci id="S3.SS2.SSS0.Px1.p3.4.m4.2.2.2.2.cmml" xref="S3.SS2.SSS0.Px1.p3.4.m4.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p3.4.m4.2c">S_{i,j}</annotation></semantics></math> provides an approximation for how the loss might change in the absence of <math id="S3.SS2.SSS0.Px1.p3.5.m5.1" class="ltx_Math" alttext="\theta_{i}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p3.5.m5.1a"><msub id="S3.SS2.SSS0.Px1.p3.5.m5.1.1" xref="S3.SS2.SSS0.Px1.p3.5.m5.1.1.cmml"><mi id="S3.SS2.SSS0.Px1.p3.5.m5.1.1.2" xref="S3.SS2.SSS0.Px1.p3.5.m5.1.1.2.cmml">θ</mi><mi id="S3.SS2.SSS0.Px1.p3.5.m5.1.1.3" xref="S3.SS2.SSS0.Px1.p3.5.m5.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p3.5.m5.1b"><apply id="S3.SS2.SSS0.Px1.p3.5.m5.1.1.cmml" xref="S3.SS2.SSS0.Px1.p3.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p3.5.m5.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p3.5.m5.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p3.5.m5.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p3.5.m5.1.1.2">𝜃</ci><ci id="S3.SS2.SSS0.Px1.p3.5.m5.1.1.3.cmml" xref="S3.SS2.SSS0.Px1.p3.5.m5.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p3.5.m5.1c">\theta_{i}</annotation></semantics></math>:</p>
</div>
<div id="S3.SS2.SSS0.Px1.p4" class="ltx_para ltx_noindent">
<table id="S3.E4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E4.m1.1" class="ltx_Math" alttext="\mathbf{\Theta}_{T_{i}}^{\top}\nabla_{\mathbf{\Theta}_{T}}\mathcal{L}(x_{j})\approx\mathcal{L}(\mathbf{\Theta}_{T})-\mathcal{L}(\mathbf{\Theta}_{T}-\mathbf{\Theta}_{T_{i}})." display="block"><semantics id="S3.E4.m1.1a"><mrow id="S3.E4.m1.1.1.1" xref="S3.E4.m1.1.1.1.1.cmml"><mrow id="S3.E4.m1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.cmml"><mrow id="S3.E4.m1.1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.1.cmml"><msubsup id="S3.E4.m1.1.1.1.1.1.3" xref="S3.E4.m1.1.1.1.1.1.3.cmml"><mi id="S3.E4.m1.1.1.1.1.1.3.2.2" xref="S3.E4.m1.1.1.1.1.1.3.2.2.cmml">𝚯</mi><msub id="S3.E4.m1.1.1.1.1.1.3.2.3" xref="S3.E4.m1.1.1.1.1.1.3.2.3.cmml"><mi id="S3.E4.m1.1.1.1.1.1.3.2.3.2" xref="S3.E4.m1.1.1.1.1.1.3.2.3.2.cmml">T</mi><mi id="S3.E4.m1.1.1.1.1.1.3.2.3.3" xref="S3.E4.m1.1.1.1.1.1.3.2.3.3.cmml">i</mi></msub><mo id="S3.E4.m1.1.1.1.1.1.3.3" xref="S3.E4.m1.1.1.1.1.1.3.3.cmml">⊤</mo></msubsup><mo lspace="0.167em" rspace="0em" id="S3.E4.m1.1.1.1.1.1.2" xref="S3.E4.m1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.E4.m1.1.1.1.1.1.4" xref="S3.E4.m1.1.1.1.1.1.4.cmml"><msub id="S3.E4.m1.1.1.1.1.1.4.1" xref="S3.E4.m1.1.1.1.1.1.4.1.cmml"><mo rspace="0.167em" id="S3.E4.m1.1.1.1.1.1.4.1.2" xref="S3.E4.m1.1.1.1.1.1.4.1.2.cmml">∇</mo><msub id="S3.E4.m1.1.1.1.1.1.4.1.3" xref="S3.E4.m1.1.1.1.1.1.4.1.3.cmml"><mi id="S3.E4.m1.1.1.1.1.1.4.1.3.2" xref="S3.E4.m1.1.1.1.1.1.4.1.3.2.cmml">𝚯</mi><mi id="S3.E4.m1.1.1.1.1.1.4.1.3.3" xref="S3.E4.m1.1.1.1.1.1.4.1.3.3.cmml">T</mi></msub></msub><mi class="ltx_font_mathcaligraphic" id="S3.E4.m1.1.1.1.1.1.4.2" xref="S3.E4.m1.1.1.1.1.1.4.2.cmml">ℒ</mi></mrow><mo lspace="0em" rspace="0em" id="S3.E4.m1.1.1.1.1.1.2a" xref="S3.E4.m1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.E4.m1.1.1.1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E4.m1.1.1.1.1.1.1.1.2" xref="S3.E4.m1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="S3.E4.m1.1.1.1.1.1.1.1.1" xref="S3.E4.m1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E4.m1.1.1.1.1.1.1.1.1.2" xref="S3.E4.m1.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="S3.E4.m1.1.1.1.1.1.1.1.1.3" xref="S3.E4.m1.1.1.1.1.1.1.1.1.3.cmml">j</mi></msub><mo stretchy="false" id="S3.E4.m1.1.1.1.1.1.1.1.3" xref="S3.E4.m1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.E4.m1.1.1.1.1.4" xref="S3.E4.m1.1.1.1.1.4.cmml">≈</mo><mrow id="S3.E4.m1.1.1.1.1.3" xref="S3.E4.m1.1.1.1.1.3.cmml"><mrow id="S3.E4.m1.1.1.1.1.2.1" xref="S3.E4.m1.1.1.1.1.2.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E4.m1.1.1.1.1.2.1.3" xref="S3.E4.m1.1.1.1.1.2.1.3.cmml">ℒ</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.1.1.1.1.2.1.2" xref="S3.E4.m1.1.1.1.1.2.1.2.cmml">​</mo><mrow id="S3.E4.m1.1.1.1.1.2.1.1.1" xref="S3.E4.m1.1.1.1.1.2.1.1.1.1.cmml"><mo stretchy="false" id="S3.E4.m1.1.1.1.1.2.1.1.1.2" xref="S3.E4.m1.1.1.1.1.2.1.1.1.1.cmml">(</mo><msub id="S3.E4.m1.1.1.1.1.2.1.1.1.1" xref="S3.E4.m1.1.1.1.1.2.1.1.1.1.cmml"><mi id="S3.E4.m1.1.1.1.1.2.1.1.1.1.2" xref="S3.E4.m1.1.1.1.1.2.1.1.1.1.2.cmml">𝚯</mi><mi id="S3.E4.m1.1.1.1.1.2.1.1.1.1.3" xref="S3.E4.m1.1.1.1.1.2.1.1.1.1.3.cmml">T</mi></msub><mo stretchy="false" id="S3.E4.m1.1.1.1.1.2.1.1.1.3" xref="S3.E4.m1.1.1.1.1.2.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.E4.m1.1.1.1.1.3.3" xref="S3.E4.m1.1.1.1.1.3.3.cmml">−</mo><mrow id="S3.E4.m1.1.1.1.1.3.2" xref="S3.E4.m1.1.1.1.1.3.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E4.m1.1.1.1.1.3.2.3" xref="S3.E4.m1.1.1.1.1.3.2.3.cmml">ℒ</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.1.1.1.1.3.2.2" xref="S3.E4.m1.1.1.1.1.3.2.2.cmml">​</mo><mrow id="S3.E4.m1.1.1.1.1.3.2.1.1" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.cmml"><mo stretchy="false" id="S3.E4.m1.1.1.1.1.3.2.1.1.2" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.cmml">(</mo><mrow id="S3.E4.m1.1.1.1.1.3.2.1.1.1" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.cmml"><msub id="S3.E4.m1.1.1.1.1.3.2.1.1.1.2" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.2.cmml"><mi id="S3.E4.m1.1.1.1.1.3.2.1.1.1.2.2" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.2.2.cmml">𝚯</mi><mi id="S3.E4.m1.1.1.1.1.3.2.1.1.1.2.3" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.2.3.cmml">T</mi></msub><mo id="S3.E4.m1.1.1.1.1.3.2.1.1.1.1" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.1.cmml">−</mo><msub id="S3.E4.m1.1.1.1.1.3.2.1.1.1.3" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.3.cmml"><mi id="S3.E4.m1.1.1.1.1.3.2.1.1.1.3.2" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.3.2.cmml">𝚯</mi><msub id="S3.E4.m1.1.1.1.1.3.2.1.1.1.3.3" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.3.3.cmml"><mi id="S3.E4.m1.1.1.1.1.3.2.1.1.1.3.3.2" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.3.3.2.cmml">T</mi><mi id="S3.E4.m1.1.1.1.1.3.2.1.1.1.3.3.3" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.3.3.3.cmml">i</mi></msub></msub></mrow><mo stretchy="false" id="S3.E4.m1.1.1.1.1.3.2.1.1.3" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><mo lspace="0em" id="S3.E4.m1.1.1.1.2" xref="S3.E4.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E4.m1.1b"><apply id="S3.E4.m1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1"><approx id="S3.E4.m1.1.1.1.1.4.cmml" xref="S3.E4.m1.1.1.1.1.4"></approx><apply id="S3.E4.m1.1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1.1"><times id="S3.E4.m1.1.1.1.1.1.2.cmml" xref="S3.E4.m1.1.1.1.1.1.2"></times><apply id="S3.E4.m1.1.1.1.1.1.3.cmml" xref="S3.E4.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.1.1.1.3.1.cmml" xref="S3.E4.m1.1.1.1.1.1.3">superscript</csymbol><apply id="S3.E4.m1.1.1.1.1.1.3.2.cmml" xref="S3.E4.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.1.1.1.3.2.1.cmml" xref="S3.E4.m1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E4.m1.1.1.1.1.1.3.2.2.cmml" xref="S3.E4.m1.1.1.1.1.1.3.2.2">𝚯</ci><apply id="S3.E4.m1.1.1.1.1.1.3.2.3.cmml" xref="S3.E4.m1.1.1.1.1.1.3.2.3"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.1.1.1.3.2.3.1.cmml" xref="S3.E4.m1.1.1.1.1.1.3.2.3">subscript</csymbol><ci id="S3.E4.m1.1.1.1.1.1.3.2.3.2.cmml" xref="S3.E4.m1.1.1.1.1.1.3.2.3.2">𝑇</ci><ci id="S3.E4.m1.1.1.1.1.1.3.2.3.3.cmml" xref="S3.E4.m1.1.1.1.1.1.3.2.3.3">𝑖</ci></apply></apply><csymbol cd="latexml" id="S3.E4.m1.1.1.1.1.1.3.3.cmml" xref="S3.E4.m1.1.1.1.1.1.3.3">top</csymbol></apply><apply id="S3.E4.m1.1.1.1.1.1.4.cmml" xref="S3.E4.m1.1.1.1.1.1.4"><apply id="S3.E4.m1.1.1.1.1.1.4.1.cmml" xref="S3.E4.m1.1.1.1.1.1.4.1"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.1.1.1.4.1.1.cmml" xref="S3.E4.m1.1.1.1.1.1.4.1">subscript</csymbol><ci id="S3.E4.m1.1.1.1.1.1.4.1.2.cmml" xref="S3.E4.m1.1.1.1.1.1.4.1.2">∇</ci><apply id="S3.E4.m1.1.1.1.1.1.4.1.3.cmml" xref="S3.E4.m1.1.1.1.1.1.4.1.3"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.1.1.1.4.1.3.1.cmml" xref="S3.E4.m1.1.1.1.1.1.4.1.3">subscript</csymbol><ci id="S3.E4.m1.1.1.1.1.1.4.1.3.2.cmml" xref="S3.E4.m1.1.1.1.1.1.4.1.3.2">𝚯</ci><ci id="S3.E4.m1.1.1.1.1.1.4.1.3.3.cmml" xref="S3.E4.m1.1.1.1.1.1.4.1.3.3">𝑇</ci></apply></apply><ci id="S3.E4.m1.1.1.1.1.1.4.2.cmml" xref="S3.E4.m1.1.1.1.1.1.4.2">ℒ</ci></apply><apply id="S3.E4.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E4.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="S3.E4.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E4.m1.1.1.1.1.1.1.1.1.3">𝑗</ci></apply></apply><apply id="S3.E4.m1.1.1.1.1.3.cmml" xref="S3.E4.m1.1.1.1.1.3"><minus id="S3.E4.m1.1.1.1.1.3.3.cmml" xref="S3.E4.m1.1.1.1.1.3.3"></minus><apply id="S3.E4.m1.1.1.1.1.2.1.cmml" xref="S3.E4.m1.1.1.1.1.2.1"><times id="S3.E4.m1.1.1.1.1.2.1.2.cmml" xref="S3.E4.m1.1.1.1.1.2.1.2"></times><ci id="S3.E4.m1.1.1.1.1.2.1.3.cmml" xref="S3.E4.m1.1.1.1.1.2.1.3">ℒ</ci><apply id="S3.E4.m1.1.1.1.1.2.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1.2.1.1.1"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.1.1.2.1.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1.2.1.1.1">subscript</csymbol><ci id="S3.E4.m1.1.1.1.1.2.1.1.1.1.2.cmml" xref="S3.E4.m1.1.1.1.1.2.1.1.1.1.2">𝚯</ci><ci id="S3.E4.m1.1.1.1.1.2.1.1.1.1.3.cmml" xref="S3.E4.m1.1.1.1.1.2.1.1.1.1.3">𝑇</ci></apply></apply><apply id="S3.E4.m1.1.1.1.1.3.2.cmml" xref="S3.E4.m1.1.1.1.1.3.2"><times id="S3.E4.m1.1.1.1.1.3.2.2.cmml" xref="S3.E4.m1.1.1.1.1.3.2.2"></times><ci id="S3.E4.m1.1.1.1.1.3.2.3.cmml" xref="S3.E4.m1.1.1.1.1.3.2.3">ℒ</ci><apply id="S3.E4.m1.1.1.1.1.3.2.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1.3.2.1.1"><minus id="S3.E4.m1.1.1.1.1.3.2.1.1.1.1.cmml" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.1"></minus><apply id="S3.E4.m1.1.1.1.1.3.2.1.1.1.2.cmml" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.2"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.1.1.3.2.1.1.1.2.1.cmml" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.2">subscript</csymbol><ci id="S3.E4.m1.1.1.1.1.3.2.1.1.1.2.2.cmml" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.2.2">𝚯</ci><ci id="S3.E4.m1.1.1.1.1.3.2.1.1.1.2.3.cmml" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.2.3">𝑇</ci></apply><apply id="S3.E4.m1.1.1.1.1.3.2.1.1.1.3.cmml" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.1.1.3.2.1.1.1.3.1.cmml" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.3">subscript</csymbol><ci id="S3.E4.m1.1.1.1.1.3.2.1.1.1.3.2.cmml" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.3.2">𝚯</ci><apply id="S3.E4.m1.1.1.1.1.3.2.1.1.1.3.3.cmml" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.3.3"><csymbol cd="ambiguous" id="S3.E4.m1.1.1.1.1.3.2.1.1.1.3.3.1.cmml" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.3.3">subscript</csymbol><ci id="S3.E4.m1.1.1.1.1.3.2.1.1.1.3.3.2.cmml" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.3.3.2">𝑇</ci><ci id="S3.E4.m1.1.1.1.1.3.2.1.1.1.3.3.3.cmml" xref="S3.E4.m1.1.1.1.1.3.2.1.1.1.3.3.3">𝑖</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E4.m1.1c">\mathbf{\Theta}_{T_{i}}^{\top}\nabla_{\mathbf{\Theta}_{T}}\mathcal{L}(x_{j})\approx\mathcal{L}(\mathbf{\Theta}_{T})-\mathcal{L}(\mathbf{\Theta}_{T}-\mathbf{\Theta}_{T_{i}}).</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
</div>
<div id="S3.SS2.SSS0.Px1.p5" class="ltx_para ltx_noindent">
<p id="S3.SS2.SSS0.Px1.p5.7" class="ltx_p">To ascertain <math id="S3.SS2.SSS0.Px1.p5.1.m1.1" class="ltx_Math" alttext="S_{i}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p5.1.m1.1a"><msub id="S3.SS2.SSS0.Px1.p5.1.m1.1.1" xref="S3.SS2.SSS0.Px1.p5.1.m1.1.1.cmml"><mi id="S3.SS2.SSS0.Px1.p5.1.m1.1.1.2" xref="S3.SS2.SSS0.Px1.p5.1.m1.1.1.2.cmml">S</mi><mi id="S3.SS2.SSS0.Px1.p5.1.m1.1.1.3" xref="S3.SS2.SSS0.Px1.p5.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p5.1.m1.1b"><apply id="S3.SS2.SSS0.Px1.p5.1.m1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p5.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p5.1.m1.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p5.1.m1.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p5.1.m1.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p5.1.m1.1.1.2">𝑆</ci><ci id="S3.SS2.SSS0.Px1.p5.1.m1.1.1.3.cmml" xref="S3.SS2.SSS0.Px1.p5.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p5.1.m1.1c">S_{i}</annotation></semantics></math> for parameter <math id="S3.SS2.SSS0.Px1.p5.2.m2.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S3.SS2.SSS0.Px1.p5.2.m2.1a"><mi id="S3.SS2.SSS0.Px1.p5.2.m2.1.1" xref="S3.SS2.SSS0.Px1.p5.2.m2.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p5.2.m2.1b"><ci id="S3.SS2.SSS0.Px1.p5.2.m2.1.1.cmml" xref="S3.SS2.SSS0.Px1.p5.2.m2.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p5.2.m2.1c">i</annotation></semantics></math> pertaining to task <math id="S3.SS2.SSS0.Px1.p5.3.m3.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p5.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.SSS0.Px1.p5.3.m3.1.1" xref="S3.SS2.SSS0.Px1.p5.3.m3.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p5.3.m3.1b"><ci id="S3.SS2.SSS0.Px1.p5.3.m3.1.1.cmml" xref="S3.SS2.SSS0.Px1.p5.3.m3.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p5.3.m3.1c">\mathcal{T}</annotation></semantics></math>, we randomly sample <math id="S3.SS2.SSS0.Px1.p5.4.m4.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS2.SSS0.Px1.p5.4.m4.1a"><mi id="S3.SS2.SSS0.Px1.p5.4.m4.1.1" xref="S3.SS2.SSS0.Px1.p5.4.m4.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p5.4.m4.1b"><ci id="S3.SS2.SSS0.Px1.p5.4.m4.1.1.cmml" xref="S3.SS2.SSS0.Px1.p5.4.m4.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p5.4.m4.1c">k</annotation></semantics></math> instances as seed samples for an efficient and representative estimate. Thus, the final formulation <math id="S3.SS2.SSS0.Px1.p5.5.m5.1" class="ltx_Math" alttext="S_{i}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p5.5.m5.1a"><msub id="S3.SS2.SSS0.Px1.p5.5.m5.1.1" xref="S3.SS2.SSS0.Px1.p5.5.m5.1.1.cmml"><mi id="S3.SS2.SSS0.Px1.p5.5.m5.1.1.2" xref="S3.SS2.SSS0.Px1.p5.5.m5.1.1.2.cmml">S</mi><mi id="S3.SS2.SSS0.Px1.p5.5.m5.1.1.3" xref="S3.SS2.SSS0.Px1.p5.5.m5.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p5.5.m5.1b"><apply id="S3.SS2.SSS0.Px1.p5.5.m5.1.1.cmml" xref="S3.SS2.SSS0.Px1.p5.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p5.5.m5.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p5.5.m5.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p5.5.m5.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p5.5.m5.1.1.2">𝑆</ci><ci id="S3.SS2.SSS0.Px1.p5.5.m5.1.1.3.cmml" xref="S3.SS2.SSS0.Px1.p5.5.m5.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p5.5.m5.1c">S_{i}</annotation></semantics></math> for task <math id="S3.SS2.SSS0.Px1.p5.6.m6.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p5.6.m6.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.SSS0.Px1.p5.6.m6.1.1" xref="S3.SS2.SSS0.Px1.p5.6.m6.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p5.6.m6.1b"><ci id="S3.SS2.SSS0.Px1.p5.6.m6.1.1.cmml" xref="S3.SS2.SSS0.Px1.p5.6.m6.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p5.6.m6.1c">\mathcal{T}</annotation></semantics></math> integrates the cumulative sensitivity over the sampled instances, calculated as <math id="S3.SS2.SSS0.Px1.p5.7.m7.2" class="ltx_Math" alttext="S_{i}=\sum_{j=1}^{k}S_{i,j}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p5.7.m7.2a"><mrow id="S3.SS2.SSS0.Px1.p5.7.m7.2.3" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.cmml"><msub id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.2" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.2.cmml"><mi id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.2.2" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.2.2.cmml">S</mi><mi id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.2.3" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.2.3.cmml">i</mi></msub><mo rspace="0.111em" id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.1" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.1.cmml">=</mo><mrow id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.cmml"><msubsup id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.cmml"><mo id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.2" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.2.cmml">∑</mo><mrow id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.3" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.3.cmml"><mi id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.3.2" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.3.2.cmml">j</mi><mo id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.3.1" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.3.1.cmml">=</mo><mn id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.3.3" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.3.3.cmml">1</mn></mrow><mi id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.3" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.3.cmml">k</mi></msubsup><msub id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.2" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.2.cmml"><mi id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.2.2" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.2.2.cmml">S</mi><mrow id="S3.SS2.SSS0.Px1.p5.7.m7.2.2.2.4" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.2.2.3.cmml"><mi id="S3.SS2.SSS0.Px1.p5.7.m7.1.1.1.1" xref="S3.SS2.SSS0.Px1.p5.7.m7.1.1.1.1.cmml">i</mi><mo id="S3.SS2.SSS0.Px1.p5.7.m7.2.2.2.4.1" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.2.2.3.cmml">,</mo><mi id="S3.SS2.SSS0.Px1.p5.7.m7.2.2.2.2" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.2.2.2.cmml">j</mi></mrow></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p5.7.m7.2b"><apply id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3"><eq id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.1.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.1"></eq><apply id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.2.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.2.1.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.2">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.2.2.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.2.2">𝑆</ci><ci id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.2.3.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.2.3">𝑖</ci></apply><apply id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3"><apply id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.1.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1">superscript</csymbol><apply id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.1.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1">subscript</csymbol><sum id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.2.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.2"></sum><apply id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.3.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.3"><eq id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.3.1.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.3.1"></eq><ci id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.3.2.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.3.2">𝑗</ci><cn type="integer" id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.3.3.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.2.3.3">1</cn></apply></apply><ci id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.3.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.1.3">𝑘</ci></apply><apply id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.2.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.2.1.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.2">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.2.2.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.3.3.2.2">𝑆</ci><list id="S3.SS2.SSS0.Px1.p5.7.m7.2.2.2.3.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.2.2.4"><ci id="S3.SS2.SSS0.Px1.p5.7.m7.1.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.1.1.1.1">𝑖</ci><ci id="S3.SS2.SSS0.Px1.p5.7.m7.2.2.2.2.cmml" xref="S3.SS2.SSS0.Px1.p5.7.m7.2.2.2.2">𝑗</ci></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p5.7.m7.2c">S_{i}=\sum_{j=1}^{k}S_{i,j}</annotation></semantics></math>.</p>
</div>
</section>
<section id="S3.SS2.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Layer Selection and Dimensionality Reduction.</h4>

<div id="S3.SS2.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S3.SS2.SSS0.Px2.p1.12" class="ltx_p">Given that models of varying scales often differ in both the number of layers and their dimensions, we adopt a method of layer selection and dimensionality reduction based on sensitivity scores. Our first step is to assess the layers of the teacher model, <math id="S3.SS2.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="M_{T}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.1.m1.1a"><msub id="S3.SS2.SSS0.Px2.p1.1.m1.1.1" xref="S3.SS2.SSS0.Px2.p1.1.m1.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p1.1.m1.1.1.2" xref="S3.SS2.SSS0.Px2.p1.1.m1.1.1.2.cmml">M</mi><mi id="S3.SS2.SSS0.Px2.p1.1.m1.1.1.3" xref="S3.SS2.SSS0.Px2.p1.1.m1.1.1.3.cmml">T</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.1.m1.1b"><apply id="S3.SS2.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.1.m1.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.1.m1.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.1.m1.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.1.m1.1.1.2">𝑀</ci><ci id="S3.SS2.SSS0.Px2.p1.1.m1.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.1.m1.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.1.m1.1c">M_{T}</annotation></semantics></math>, with respect to their relevance to task <math id="S3.SS2.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.SSS0.Px2.p1.2.m2.1.1" xref="S3.SS2.SSS0.Px2.p1.2.m2.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.2.m2.1b"><ci id="S3.SS2.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.2.m2.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.2.m2.1c">\mathcal{T}</annotation></semantics></math>. Let <math id="S3.SS2.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="L_{T}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.3.m3.1a"><msub id="S3.SS2.SSS0.Px2.p1.3.m3.1.1" xref="S3.SS2.SSS0.Px2.p1.3.m3.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p1.3.m3.1.1.2" xref="S3.SS2.SSS0.Px2.p1.3.m3.1.1.2.cmml">L</mi><mi id="S3.SS2.SSS0.Px2.p1.3.m3.1.1.3" xref="S3.SS2.SSS0.Px2.p1.3.m3.1.1.3.cmml">T</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.3.m3.1b"><apply id="S3.SS2.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.3.m3.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.3.m3.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.3.m3.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.3.m3.1.1.2">𝐿</ci><ci id="S3.SS2.SSS0.Px2.p1.3.m3.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.3.m3.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.3.m3.1c">L_{T}</annotation></semantics></math> and <math id="S3.SS2.SSS0.Px2.p1.4.m4.1" class="ltx_Math" alttext="L_{S}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.4.m4.1a"><msub id="S3.SS2.SSS0.Px2.p1.4.m4.1.1" xref="S3.SS2.SSS0.Px2.p1.4.m4.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p1.4.m4.1.1.2" xref="S3.SS2.SSS0.Px2.p1.4.m4.1.1.2.cmml">L</mi><mi id="S3.SS2.SSS0.Px2.p1.4.m4.1.1.3" xref="S3.SS2.SSS0.Px2.p1.4.m4.1.1.3.cmml">S</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.4.m4.1b"><apply id="S3.SS2.SSS0.Px2.p1.4.m4.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.4.m4.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.4.m4.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.4.m4.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.4.m4.1.1.2">𝐿</ci><ci id="S3.SS2.SSS0.Px2.p1.4.m4.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.4.m4.1.1.3">𝑆</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.4.m4.1c">L_{S}</annotation></semantics></math> represent the total number of layers in the teacher and student models, respectively, with <math id="S3.SS2.SSS0.Px2.p1.5.m5.1" class="ltx_Math" alttext="L_{S}\leq L_{T}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.5.m5.1a"><mrow id="S3.SS2.SSS0.Px2.p1.5.m5.1.1" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.cmml"><msub id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.2" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.2.cmml"><mi id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.2.2" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.2.2.cmml">L</mi><mi id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.2.3" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.2.3.cmml">S</mi></msub><mo id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.1" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.1.cmml">≤</mo><msub id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.3" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.3.cmml"><mi id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.3.2" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.3.2.cmml">L</mi><mi id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.3.3" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.3.3.cmml">T</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.5.m5.1b"><apply id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1"><leq id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.1"></leq><apply id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.2.1.cmml" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.2.2.cmml" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.2.2">𝐿</ci><ci id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.2.3.cmml" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.2.3">𝑆</ci></apply><apply id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.3.1.cmml" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.3.2.cmml" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.3.2">𝐿</ci><ci id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.3.3.cmml" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.3.3">𝑇</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.5.m5.1c">L_{S}\leq L_{T}</annotation></semantics></math>. For each layer <math id="S3.SS2.SSS0.Px2.p1.6.m6.1" class="ltx_Math" alttext="l" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.6.m6.1a"><mi id="S3.SS2.SSS0.Px2.p1.6.m6.1.1" xref="S3.SS2.SSS0.Px2.p1.6.m6.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.6.m6.1b"><ci id="S3.SS2.SSS0.Px2.p1.6.m6.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.6.m6.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.6.m6.1c">l</annotation></semantics></math> in <math id="S3.SS2.SSS0.Px2.p1.7.m7.1" class="ltx_Math" alttext="M_{T}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.7.m7.1a"><msub id="S3.SS2.SSS0.Px2.p1.7.m7.1.1" xref="S3.SS2.SSS0.Px2.p1.7.m7.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p1.7.m7.1.1.2" xref="S3.SS2.SSS0.Px2.p1.7.m7.1.1.2.cmml">M</mi><mi id="S3.SS2.SSS0.Px2.p1.7.m7.1.1.3" xref="S3.SS2.SSS0.Px2.p1.7.m7.1.1.3.cmml">T</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.7.m7.1b"><apply id="S3.SS2.SSS0.Px2.p1.7.m7.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.7.m7.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.7.m7.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.7.m7.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.7.m7.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.7.m7.1.1.2">𝑀</ci><ci id="S3.SS2.SSS0.Px2.p1.7.m7.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.7.m7.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.7.m7.1c">M_{T}</annotation></semantics></math>, we calculate a layer-specific sensitivity score, <math id="S3.SS2.SSS0.Px2.p1.8.m8.1" class="ltx_Math" alttext="S_{T_{l}}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.8.m8.1a"><msub id="S3.SS2.SSS0.Px2.p1.8.m8.1.1" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.2" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1.2.cmml">S</mi><msub id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.cmml"><mi id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.2" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.2.cmml">T</mi><mi id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.3" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.3.cmml">l</mi></msub></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.8.m8.1b"><apply id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1.2">𝑆</ci><apply id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.1.cmml" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.2.cmml" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.2">𝑇</ci><ci id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.3.cmml" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.3">𝑙</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.8.m8.1c">S_{T_{l}}</annotation></semantics></math>, by aggregating the sensitivity scores of all parameters within that layer, represented as: <math id="S3.SS2.SSS0.Px2.p1.9.m9.1" class="ltx_Math" alttext="S_{T_{l}}=\sum_{\theta_{i}\in\mathbf{\Theta}_{T_{l}}}S_{i}." display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.9.m9.1a"><mrow id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.cmml"><mrow id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.cmml"><msub id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2.cmml"><mi id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2.2" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2.2.cmml">S</mi><msub id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2.3" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2.3.cmml"><mi id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2.3.2" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2.3.2.cmml">T</mi><mi id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2.3.3" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2.3.3.cmml">l</mi></msub></msub><mo rspace="0.111em" id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.1" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.1.cmml">=</mo><mrow id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.cmml"><msub id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.cmml"><mo id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.2" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.2.cmml">∑</mo><mrow id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.cmml"><msub id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.2" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.2.cmml"><mi id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.2.2" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.2.2.cmml">θ</mi><mi id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.2.3" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.2.3.cmml">i</mi></msub><mo id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.1" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.1.cmml">∈</mo><msub id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3.cmml"><mi id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3.2" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3.2.cmml">𝚯</mi><msub id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3.3" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3.3.cmml"><mi id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3.3.2" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3.3.2.cmml">T</mi><mi id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3.3.3" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3.3.3.cmml">l</mi></msub></msub></mrow></msub><msub id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.2" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.2.cmml"><mi id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.2.2" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.2.2.cmml">S</mi><mi id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.2.3" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.2.3.cmml">i</mi></msub></mrow></mrow><mo lspace="0em" id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.2" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.9.m9.1b"><apply id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1"><eq id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.1"></eq><apply id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2.1.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2.2.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2.2">𝑆</ci><apply id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2.3.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2.3.1.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2.3.2.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2.3.2">𝑇</ci><ci id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2.3.3.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.2.3.3">𝑙</ci></apply></apply><apply id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3"><apply id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1">subscript</csymbol><sum id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.2"></sum><apply id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3"><in id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.1.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.1"></in><apply id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.2.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.2.1.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.2">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.2.2.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.2.2">𝜃</ci><ci id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.2.3.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.2.3">𝑖</ci></apply><apply id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3.1.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3.2.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3.2">𝚯</ci><apply id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3.3.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3.3.1.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3.3.2.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3.3.2">𝑇</ci><ci id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3.3.3.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.1.3.3.3.3">𝑙</ci></apply></apply></apply></apply><apply id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.2.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.2.1.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.2">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.2.2.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.2.2">𝑆</ci><ci id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.2.3.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.1.1.3.2.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.9.m9.1c">S_{T_{l}}=\sum_{\theta_{i}\in\mathbf{\Theta}_{T_{l}}}S_{i}.</annotation></semantics></math>
Having computed the sensitivity scores for layer <math id="S3.SS2.SSS0.Px2.p1.10.m10.1" class="ltx_Math" alttext="l" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.10.m10.1a"><mi id="S3.SS2.SSS0.Px2.p1.10.m10.1.1" xref="S3.SS2.SSS0.Px2.p1.10.m10.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.10.m10.1b"><ci id="S3.SS2.SSS0.Px2.p1.10.m10.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.10.m10.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.10.m10.1c">l</annotation></semantics></math> in <math id="S3.SS2.SSS0.Px2.p1.11.m11.1" class="ltx_Math" alttext="M_{T}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.11.m11.1a"><msub id="S3.SS2.SSS0.Px2.p1.11.m11.1.1" xref="S3.SS2.SSS0.Px2.p1.11.m11.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p1.11.m11.1.1.2" xref="S3.SS2.SSS0.Px2.p1.11.m11.1.1.2.cmml">M</mi><mi id="S3.SS2.SSS0.Px2.p1.11.m11.1.1.3" xref="S3.SS2.SSS0.Px2.p1.11.m11.1.1.3.cmml">T</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.11.m11.1b"><apply id="S3.SS2.SSS0.Px2.p1.11.m11.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.11.m11.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.11.m11.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.11.m11.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.11.m11.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.11.m11.1.1.2">𝑀</ci><ci id="S3.SS2.SSS0.Px2.p1.11.m11.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.11.m11.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.11.m11.1c">M_{T}</annotation></semantics></math>, we proceed to arrange them in descending order and select the top <math id="S3.SS2.SSS0.Px2.p1.12.m12.1" class="ltx_Math" alttext="L_{S}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.12.m12.1a"><msub id="S3.SS2.SSS0.Px2.p1.12.m12.1.1" xref="S3.SS2.SSS0.Px2.p1.12.m12.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p1.12.m12.1.1.2" xref="S3.SS2.SSS0.Px2.p1.12.m12.1.1.2.cmml">L</mi><mi id="S3.SS2.SSS0.Px2.p1.12.m12.1.1.3" xref="S3.SS2.SSS0.Px2.p1.12.m12.1.1.3.cmml">S</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.12.m12.1b"><apply id="S3.SS2.SSS0.Px2.p1.12.m12.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.12.m12.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.12.m12.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.12.m12.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.12.m12.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.12.m12.1.1.2">𝐿</ci><ci id="S3.SS2.SSS0.Px2.p1.12.m12.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.12.m12.1.1.3">𝑆</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.12.m12.1c">L_{S}</annotation></semantics></math> layers with the highest scores. To preserve the inherent structure of the teacher model, the chosen layers are subsequently mapped to the student model maintaining their original sequential order.</p>
</div>
<div id="S3.SS2.SSS0.Px2.p2" class="ltx_para ltx_noindent">
<p id="S3.SS2.SSS0.Px2.p2.5" class="ltx_p">Upon alignment of the layers, the parameter dimensions of each layer in the teacher model typically continue to surpass those of the student model. During this phase of the transfer process, our focus is primarily on all the two-dimensional matrices in the teacher model, which are denoted as <math id="S3.SS2.SSS0.Px2.p2.1.m1.1" class="ltx_Math" alttext="\bm{\mathsfit{W_{T}}}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p2.1.m1.1a"><msub id="S3.SS2.SSS0.Px2.p2.1.m1.1.1" xref="S3.SS2.SSS0.Px2.p2.1.m1.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p2.1.m1.1.1.2" xref="S3.SS2.SSS0.Px2.p2.1.m1.1.1.2.cmml">𝑾</mi><mi id="S3.SS2.SSS0.Px2.p2.1.m1.1.1.3" xref="S3.SS2.SSS0.Px2.p2.1.m1.1.1.3.cmml">𝑻</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p2.1.m1.1b"><apply id="S3.SS2.SSS0.Px2.p2.1.m1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p2.1.m1.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p2.1.m1.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p2.1.m1.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p2.1.m1.1.1.2">𝑾</ci><ci id="S3.SS2.SSS0.Px2.p2.1.m1.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p2.1.m1.1.1.3">𝑻</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p2.1.m1.1c">\bm{\mathsfit{W_{T}}}</annotation></semantics></math>.
To preserve the structural information in each <math id="S3.SS2.SSS0.Px2.p2.2.m2.1" class="ltx_Math" alttext="\bm{W}_{T_{i}}\in\bm{\mathsfit{W_{T}}}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p2.2.m2.1a"><mrow id="S3.SS2.SSS0.Px2.p2.2.m2.1.1" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.cmml"><msub id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.cmml"><mi id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.2" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.2.cmml">𝑾</mi><msub id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.3" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.3.cmml"><mi id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.3.2" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.3.2.cmml">T</mi><mi id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.3.3" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.3.3.cmml">i</mi></msub></msub><mo id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.1" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.1.cmml">∈</mo><msub id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.3" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.3.cmml"><mi id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.3.2" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.3.2.cmml">𝑾</mi><mi id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.3.3" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.3.3.cmml">𝑻</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p2.2.m2.1b"><apply id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.cmml" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1"><in id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.1"></in><apply id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.1.cmml" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.2.cmml" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.2">𝑾</ci><apply id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.3.cmml" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.3.1.cmml" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.3.2.cmml" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.3.2">𝑇</ci><ci id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.3.3.cmml" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.2.3.3">𝑖</ci></apply></apply><apply id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.3.1.cmml" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.3.2.cmml" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.3.2">𝑾</ci><ci id="S3.SS2.SSS0.Px2.p2.2.m2.1.1.3.3.cmml" xref="S3.SS2.SSS0.Px2.p2.2.m2.1.1.3.3">𝑻</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p2.2.m2.1c">\bm{W}_{T_{i}}\in\bm{\mathsfit{W_{T}}}</annotation></semantics></math> with the dimensions <math id="S3.SS2.SSS0.Px2.p2.3.m3.1" class="ltx_Math" alttext="\mathbb{R}^{n_{T}\times m_{T}}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p2.3.m3.1a"><msup id="S3.SS2.SSS0.Px2.p2.3.m3.1.1" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.2" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1.2.cmml">ℝ</mi><mrow id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.cmml"><msub id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.2" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.2.cmml"><mi id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.2.2" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.2.2.cmml">n</mi><mi id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.2.3" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.2.3.cmml">T</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.1" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.1.cmml">×</mo><msub id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.3" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.3.cmml"><mi id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.3.2" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.3.2.cmml">m</mi><mi id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.3.3" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.3.3.cmml">T</mi></msub></mrow></msup><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p2.3.m3.1b"><apply id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.cmml" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1">superscript</csymbol><ci id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1.2">ℝ</ci><apply id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3"><times id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.1.cmml" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.1"></times><apply id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.2.cmml" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.2.1.cmml" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.2">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.2.2.cmml" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.2.2">𝑛</ci><ci id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.2.3.cmml" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.2.3">𝑇</ci></apply><apply id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.3.cmml" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.3.1.cmml" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.3.2.cmml" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.3.2">𝑚</ci><ci id="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.3.3.cmml" xref="S3.SS2.SSS0.Px2.p2.3.m3.1.1.3.3.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p2.3.m3.1c">\mathbb{R}^{n_{T}\times m_{T}}</annotation></semantics></math>, we extract the submatrix with the highest cumulative sensitivity to align with the student model’s corresponding matrix with dimensions <math id="S3.SS2.SSS0.Px2.p2.4.m4.1" class="ltx_Math" alttext="\mathbb{R}^{n_{S}\times m_{S}}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p2.4.m4.1a"><msup id="S3.SS2.SSS0.Px2.p2.4.m4.1.1" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.2" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1.2.cmml">ℝ</mi><mrow id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.cmml"><msub id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.2" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.2.cmml"><mi id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.2.2" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.2.2.cmml">n</mi><mi id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.2.3" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.2.3.cmml">S</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.1" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.1.cmml">×</mo><msub id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.3" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.3.cmml"><mi id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.3.2" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.3.2.cmml">m</mi><mi id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.3.3" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.3.3.cmml">S</mi></msub></mrow></msup><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p2.4.m4.1b"><apply id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.cmml" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1">superscript</csymbol><ci id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1.2">ℝ</ci><apply id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3"><times id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.1.cmml" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.1"></times><apply id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.2.cmml" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.2.1.cmml" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.2">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.2.2.cmml" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.2.2">𝑛</ci><ci id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.2.3.cmml" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.2.3">𝑆</ci></apply><apply id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.3.cmml" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.3.1.cmml" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.3.2.cmml" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.3.2">𝑚</ci><ci id="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.3.3.cmml" xref="S3.SS2.SSS0.Px2.p2.4.m4.1.1.3.3.3">𝑆</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p2.4.m4.1c">\mathbb{R}^{n_{S}\times m_{S}}</annotation></semantics></math> (<math id="S3.SS2.SSS0.Px2.p2.5.m5.2" class="ltx_Math" alttext="n_{S}\leq n_{T},m_{S}\leq m_{T}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p2.5.m5.2a"><mrow id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.3.cmml"><mrow id="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1" xref="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.cmml"><msub id="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.2" xref="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.2.cmml"><mi id="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.2.2" xref="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.2.2.cmml">n</mi><mi id="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.2.3" xref="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.2.3.cmml">S</mi></msub><mo id="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1" xref="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.cmml">≤</mo><msub id="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.3" xref="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.3.cmml"><mi id="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.3.2" xref="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.3.2.cmml">n</mi><mi id="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.3.3" xref="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.3.3.cmml">T</mi></msub></mrow><mo id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.3" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.3a.cmml">,</mo><mrow id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.cmml"><msub id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.2" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.2.cmml"><mi id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.2.2" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.2.2.cmml">m</mi><mi id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.2.3" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.2.3.cmml">S</mi></msub><mo id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.1" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.1.cmml">≤</mo><msub id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.3" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.3.cmml"><mi id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.3.2" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.3.2.cmml">m</mi><mi id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.3.3" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.3.3.cmml">T</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p2.5.m5.2b"><apply id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.3.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.3a.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.3">formulae-sequence</csymbol><apply id="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1"><leq id="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.1"></leq><apply id="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.2.1.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.2.2.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.2.2">𝑛</ci><ci id="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.2.3.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.2.3">𝑆</ci></apply><apply id="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.3.1.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.3.2.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.3.2">𝑛</ci><ci id="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.3.3.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.1.1.1.1.3.3">𝑇</ci></apply></apply><apply id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2"><leq id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.1.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.1"></leq><apply id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.2.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.2.1.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.2">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.2.2.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.2.2">𝑚</ci><ci id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.2.3.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.2.3">𝑆</ci></apply><apply id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.3.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.3.1.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.3.2.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.3.2">𝑚</ci><ci id="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.3.3.cmml" xref="S3.SS2.SSS0.Px2.p2.5.m5.2.2.2.2.3.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p2.5.m5.2c">n_{S}\leq n_{T},m_{S}\leq m_{T}</annotation></semantics></math>).
Formally, this objective is expressed as:</p>
<table id="S3.E5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E5.m1.3" class="ltx_Math" alttext="\bm{W}_{T_{i},\text{extract}}=\arg\max_{\bm{W}^{\prime}\subseteq\bm{W}_{T_{i}}}\sum_{\theta_{i}\in\bm{W}^{\prime}}S_{i}." display="block"><semantics id="S3.E5.m1.3a"><mrow id="S3.E5.m1.3.3.1" xref="S3.E5.m1.3.3.1.1.cmml"><mrow id="S3.E5.m1.3.3.1.1" xref="S3.E5.m1.3.3.1.1.cmml"><msub id="S3.E5.m1.3.3.1.1.2" xref="S3.E5.m1.3.3.1.1.2.cmml"><mi id="S3.E5.m1.3.3.1.1.2.2" xref="S3.E5.m1.3.3.1.1.2.2.cmml">𝑾</mi><mrow id="S3.E5.m1.2.2.2.2" xref="S3.E5.m1.2.2.2.3.cmml"><msub id="S3.E5.m1.2.2.2.2.1" xref="S3.E5.m1.2.2.2.2.1.cmml"><mi id="S3.E5.m1.2.2.2.2.1.2" xref="S3.E5.m1.2.2.2.2.1.2.cmml">T</mi><mi id="S3.E5.m1.2.2.2.2.1.3" xref="S3.E5.m1.2.2.2.2.1.3.cmml">i</mi></msub><mo id="S3.E5.m1.2.2.2.2.2" xref="S3.E5.m1.2.2.2.3.cmml">,</mo><mtext id="S3.E5.m1.1.1.1.1" xref="S3.E5.m1.1.1.1.1a.cmml">extract</mtext></mrow></msub><mo id="S3.E5.m1.3.3.1.1.1" xref="S3.E5.m1.3.3.1.1.1.cmml">=</mo><mrow id="S3.E5.m1.3.3.1.1.3" xref="S3.E5.m1.3.3.1.1.3.cmml"><mrow id="S3.E5.m1.3.3.1.1.3.2" xref="S3.E5.m1.3.3.1.1.3.2.cmml"><mi id="S3.E5.m1.3.3.1.1.3.2.1" xref="S3.E5.m1.3.3.1.1.3.2.1.cmml">arg</mi><mo lspace="0.167em" id="S3.E5.m1.3.3.1.1.3.2a" xref="S3.E5.m1.3.3.1.1.3.2.cmml">⁡</mo><munder id="S3.E5.m1.3.3.1.1.3.2.2" xref="S3.E5.m1.3.3.1.1.3.2.2.cmml"><mi id="S3.E5.m1.3.3.1.1.3.2.2.2" xref="S3.E5.m1.3.3.1.1.3.2.2.2.cmml">max</mi><mrow id="S3.E5.m1.3.3.1.1.3.2.2.3" xref="S3.E5.m1.3.3.1.1.3.2.2.3.cmml"><msup id="S3.E5.m1.3.3.1.1.3.2.2.3.2" xref="S3.E5.m1.3.3.1.1.3.2.2.3.2.cmml"><mi id="S3.E5.m1.3.3.1.1.3.2.2.3.2.2" xref="S3.E5.m1.3.3.1.1.3.2.2.3.2.2.cmml">𝑾</mi><mo id="S3.E5.m1.3.3.1.1.3.2.2.3.2.3" xref="S3.E5.m1.3.3.1.1.3.2.2.3.2.3.cmml">′</mo></msup><mo id="S3.E5.m1.3.3.1.1.3.2.2.3.1" xref="S3.E5.m1.3.3.1.1.3.2.2.3.1.cmml">⊆</mo><msub id="S3.E5.m1.3.3.1.1.3.2.2.3.3" xref="S3.E5.m1.3.3.1.1.3.2.2.3.3.cmml"><mi id="S3.E5.m1.3.3.1.1.3.2.2.3.3.2" xref="S3.E5.m1.3.3.1.1.3.2.2.3.3.2.cmml">𝑾</mi><msub id="S3.E5.m1.3.3.1.1.3.2.2.3.3.3" xref="S3.E5.m1.3.3.1.1.3.2.2.3.3.3.cmml"><mi id="S3.E5.m1.3.3.1.1.3.2.2.3.3.3.2" xref="S3.E5.m1.3.3.1.1.3.2.2.3.3.3.2.cmml">T</mi><mi id="S3.E5.m1.3.3.1.1.3.2.2.3.3.3.3" xref="S3.E5.m1.3.3.1.1.3.2.2.3.3.3.3.cmml">i</mi></msub></msub></mrow></munder></mrow><mo lspace="0em" rspace="0em" id="S3.E5.m1.3.3.1.1.3.1" xref="S3.E5.m1.3.3.1.1.3.1.cmml">​</mo><mrow id="S3.E5.m1.3.3.1.1.3.3" xref="S3.E5.m1.3.3.1.1.3.3.cmml"><munder id="S3.E5.m1.3.3.1.1.3.3.1" xref="S3.E5.m1.3.3.1.1.3.3.1.cmml"><mo movablelimits="false" id="S3.E5.m1.3.3.1.1.3.3.1.2" xref="S3.E5.m1.3.3.1.1.3.3.1.2.cmml">∑</mo><mrow id="S3.E5.m1.3.3.1.1.3.3.1.3" xref="S3.E5.m1.3.3.1.1.3.3.1.3.cmml"><msub id="S3.E5.m1.3.3.1.1.3.3.1.3.2" xref="S3.E5.m1.3.3.1.1.3.3.1.3.2.cmml"><mi id="S3.E5.m1.3.3.1.1.3.3.1.3.2.2" xref="S3.E5.m1.3.3.1.1.3.3.1.3.2.2.cmml">θ</mi><mi id="S3.E5.m1.3.3.1.1.3.3.1.3.2.3" xref="S3.E5.m1.3.3.1.1.3.3.1.3.2.3.cmml">i</mi></msub><mo id="S3.E5.m1.3.3.1.1.3.3.1.3.1" xref="S3.E5.m1.3.3.1.1.3.3.1.3.1.cmml">∈</mo><msup id="S3.E5.m1.3.3.1.1.3.3.1.3.3" xref="S3.E5.m1.3.3.1.1.3.3.1.3.3.cmml"><mi id="S3.E5.m1.3.3.1.1.3.3.1.3.3.2" xref="S3.E5.m1.3.3.1.1.3.3.1.3.3.2.cmml">𝑾</mi><mo id="S3.E5.m1.3.3.1.1.3.3.1.3.3.3" xref="S3.E5.m1.3.3.1.1.3.3.1.3.3.3.cmml">′</mo></msup></mrow></munder><msub id="S3.E5.m1.3.3.1.1.3.3.2" xref="S3.E5.m1.3.3.1.1.3.3.2.cmml"><mi id="S3.E5.m1.3.3.1.1.3.3.2.2" xref="S3.E5.m1.3.3.1.1.3.3.2.2.cmml">S</mi><mi id="S3.E5.m1.3.3.1.1.3.3.2.3" xref="S3.E5.m1.3.3.1.1.3.3.2.3.cmml">i</mi></msub></mrow></mrow></mrow><mo lspace="0em" id="S3.E5.m1.3.3.1.2" xref="S3.E5.m1.3.3.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E5.m1.3b"><apply id="S3.E5.m1.3.3.1.1.cmml" xref="S3.E5.m1.3.3.1"><eq id="S3.E5.m1.3.3.1.1.1.cmml" xref="S3.E5.m1.3.3.1.1.1"></eq><apply id="S3.E5.m1.3.3.1.1.2.cmml" xref="S3.E5.m1.3.3.1.1.2"><csymbol cd="ambiguous" id="S3.E5.m1.3.3.1.1.2.1.cmml" xref="S3.E5.m1.3.3.1.1.2">subscript</csymbol><ci id="S3.E5.m1.3.3.1.1.2.2.cmml" xref="S3.E5.m1.3.3.1.1.2.2">𝑾</ci><list id="S3.E5.m1.2.2.2.3.cmml" xref="S3.E5.m1.2.2.2.2"><apply id="S3.E5.m1.2.2.2.2.1.cmml" xref="S3.E5.m1.2.2.2.2.1"><csymbol cd="ambiguous" id="S3.E5.m1.2.2.2.2.1.1.cmml" xref="S3.E5.m1.2.2.2.2.1">subscript</csymbol><ci id="S3.E5.m1.2.2.2.2.1.2.cmml" xref="S3.E5.m1.2.2.2.2.1.2">𝑇</ci><ci id="S3.E5.m1.2.2.2.2.1.3.cmml" xref="S3.E5.m1.2.2.2.2.1.3">𝑖</ci></apply><ci id="S3.E5.m1.1.1.1.1a.cmml" xref="S3.E5.m1.1.1.1.1"><mtext mathsize="70%" id="S3.E5.m1.1.1.1.1.cmml" xref="S3.E5.m1.1.1.1.1">extract</mtext></ci></list></apply><apply id="S3.E5.m1.3.3.1.1.3.cmml" xref="S3.E5.m1.3.3.1.1.3"><times id="S3.E5.m1.3.3.1.1.3.1.cmml" xref="S3.E5.m1.3.3.1.1.3.1"></times><apply id="S3.E5.m1.3.3.1.1.3.2.cmml" xref="S3.E5.m1.3.3.1.1.3.2"><arg id="S3.E5.m1.3.3.1.1.3.2.1.cmml" xref="S3.E5.m1.3.3.1.1.3.2.1"></arg><apply id="S3.E5.m1.3.3.1.1.3.2.2.cmml" xref="S3.E5.m1.3.3.1.1.3.2.2"><csymbol cd="ambiguous" id="S3.E5.m1.3.3.1.1.3.2.2.1.cmml" xref="S3.E5.m1.3.3.1.1.3.2.2">subscript</csymbol><max id="S3.E5.m1.3.3.1.1.3.2.2.2.cmml" xref="S3.E5.m1.3.3.1.1.3.2.2.2"></max><apply id="S3.E5.m1.3.3.1.1.3.2.2.3.cmml" xref="S3.E5.m1.3.3.1.1.3.2.2.3"><subset id="S3.E5.m1.3.3.1.1.3.2.2.3.1.cmml" xref="S3.E5.m1.3.3.1.1.3.2.2.3.1"></subset><apply id="S3.E5.m1.3.3.1.1.3.2.2.3.2.cmml" xref="S3.E5.m1.3.3.1.1.3.2.2.3.2"><csymbol cd="ambiguous" id="S3.E5.m1.3.3.1.1.3.2.2.3.2.1.cmml" xref="S3.E5.m1.3.3.1.1.3.2.2.3.2">superscript</csymbol><ci id="S3.E5.m1.3.3.1.1.3.2.2.3.2.2.cmml" xref="S3.E5.m1.3.3.1.1.3.2.2.3.2.2">𝑾</ci><ci id="S3.E5.m1.3.3.1.1.3.2.2.3.2.3.cmml" xref="S3.E5.m1.3.3.1.1.3.2.2.3.2.3">′</ci></apply><apply id="S3.E5.m1.3.3.1.1.3.2.2.3.3.cmml" xref="S3.E5.m1.3.3.1.1.3.2.2.3.3"><csymbol cd="ambiguous" id="S3.E5.m1.3.3.1.1.3.2.2.3.3.1.cmml" xref="S3.E5.m1.3.3.1.1.3.2.2.3.3">subscript</csymbol><ci id="S3.E5.m1.3.3.1.1.3.2.2.3.3.2.cmml" xref="S3.E5.m1.3.3.1.1.3.2.2.3.3.2">𝑾</ci><apply id="S3.E5.m1.3.3.1.1.3.2.2.3.3.3.cmml" xref="S3.E5.m1.3.3.1.1.3.2.2.3.3.3"><csymbol cd="ambiguous" id="S3.E5.m1.3.3.1.1.3.2.2.3.3.3.1.cmml" xref="S3.E5.m1.3.3.1.1.3.2.2.3.3.3">subscript</csymbol><ci id="S3.E5.m1.3.3.1.1.3.2.2.3.3.3.2.cmml" xref="S3.E5.m1.3.3.1.1.3.2.2.3.3.3.2">𝑇</ci><ci id="S3.E5.m1.3.3.1.1.3.2.2.3.3.3.3.cmml" xref="S3.E5.m1.3.3.1.1.3.2.2.3.3.3.3">𝑖</ci></apply></apply></apply></apply></apply><apply id="S3.E5.m1.3.3.1.1.3.3.cmml" xref="S3.E5.m1.3.3.1.1.3.3"><apply id="S3.E5.m1.3.3.1.1.3.3.1.cmml" xref="S3.E5.m1.3.3.1.1.3.3.1"><csymbol cd="ambiguous" id="S3.E5.m1.3.3.1.1.3.3.1.1.cmml" xref="S3.E5.m1.3.3.1.1.3.3.1">subscript</csymbol><sum id="S3.E5.m1.3.3.1.1.3.3.1.2.cmml" xref="S3.E5.m1.3.3.1.1.3.3.1.2"></sum><apply id="S3.E5.m1.3.3.1.1.3.3.1.3.cmml" xref="S3.E5.m1.3.3.1.1.3.3.1.3"><in id="S3.E5.m1.3.3.1.1.3.3.1.3.1.cmml" xref="S3.E5.m1.3.3.1.1.3.3.1.3.1"></in><apply id="S3.E5.m1.3.3.1.1.3.3.1.3.2.cmml" xref="S3.E5.m1.3.3.1.1.3.3.1.3.2"><csymbol cd="ambiguous" id="S3.E5.m1.3.3.1.1.3.3.1.3.2.1.cmml" xref="S3.E5.m1.3.3.1.1.3.3.1.3.2">subscript</csymbol><ci id="S3.E5.m1.3.3.1.1.3.3.1.3.2.2.cmml" xref="S3.E5.m1.3.3.1.1.3.3.1.3.2.2">𝜃</ci><ci id="S3.E5.m1.3.3.1.1.3.3.1.3.2.3.cmml" xref="S3.E5.m1.3.3.1.1.3.3.1.3.2.3">𝑖</ci></apply><apply id="S3.E5.m1.3.3.1.1.3.3.1.3.3.cmml" xref="S3.E5.m1.3.3.1.1.3.3.1.3.3"><csymbol cd="ambiguous" id="S3.E5.m1.3.3.1.1.3.3.1.3.3.1.cmml" xref="S3.E5.m1.3.3.1.1.3.3.1.3.3">superscript</csymbol><ci id="S3.E5.m1.3.3.1.1.3.3.1.3.3.2.cmml" xref="S3.E5.m1.3.3.1.1.3.3.1.3.3.2">𝑾</ci><ci id="S3.E5.m1.3.3.1.1.3.3.1.3.3.3.cmml" xref="S3.E5.m1.3.3.1.1.3.3.1.3.3.3">′</ci></apply></apply></apply><apply id="S3.E5.m1.3.3.1.1.3.3.2.cmml" xref="S3.E5.m1.3.3.1.1.3.3.2"><csymbol cd="ambiguous" id="S3.E5.m1.3.3.1.1.3.3.2.1.cmml" xref="S3.E5.m1.3.3.1.1.3.3.2">subscript</csymbol><ci id="S3.E5.m1.3.3.1.1.3.3.2.2.cmml" xref="S3.E5.m1.3.3.1.1.3.3.2.2">𝑆</ci><ci id="S3.E5.m1.3.3.1.1.3.3.2.3.cmml" xref="S3.E5.m1.3.3.1.1.3.3.2.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E5.m1.3c">\bm{W}_{T_{i},\text{extract}}=\arg\max_{\bm{W}^{\prime}\subseteq\bm{W}_{T_{i}}}\sum_{\theta_{i}\in\bm{W}^{\prime}}S_{i}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(5)</span></td>
</tr></tbody>
</table>
<p id="S3.SS2.SSS0.Px2.p2.8" class="ltx_p">By aggregating <math id="S3.SS2.SSS0.Px2.p2.6.m1.2" class="ltx_Math" alttext="\bm{W}_{T_{i},\text{extract}}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p2.6.m1.2a"><msub id="S3.SS2.SSS0.Px2.p2.6.m1.2.3" xref="S3.SS2.SSS0.Px2.p2.6.m1.2.3.cmml"><mi id="S3.SS2.SSS0.Px2.p2.6.m1.2.3.2" xref="S3.SS2.SSS0.Px2.p2.6.m1.2.3.2.cmml">𝑾</mi><mrow id="S3.SS2.SSS0.Px2.p2.6.m1.2.2.2.2" xref="S3.SS2.SSS0.Px2.p2.6.m1.2.2.2.3.cmml"><msub id="S3.SS2.SSS0.Px2.p2.6.m1.2.2.2.2.1" xref="S3.SS2.SSS0.Px2.p2.6.m1.2.2.2.2.1.cmml"><mi id="S3.SS2.SSS0.Px2.p2.6.m1.2.2.2.2.1.2" xref="S3.SS2.SSS0.Px2.p2.6.m1.2.2.2.2.1.2.cmml">T</mi><mi id="S3.SS2.SSS0.Px2.p2.6.m1.2.2.2.2.1.3" xref="S3.SS2.SSS0.Px2.p2.6.m1.2.2.2.2.1.3.cmml">i</mi></msub><mo id="S3.SS2.SSS0.Px2.p2.6.m1.2.2.2.2.2" xref="S3.SS2.SSS0.Px2.p2.6.m1.2.2.2.3.cmml">,</mo><mtext id="S3.SS2.SSS0.Px2.p2.6.m1.1.1.1.1" xref="S3.SS2.SSS0.Px2.p2.6.m1.1.1.1.1a.cmml">extract</mtext></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p2.6.m1.2b"><apply id="S3.SS2.SSS0.Px2.p2.6.m1.2.3.cmml" xref="S3.SS2.SSS0.Px2.p2.6.m1.2.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p2.6.m1.2.3.1.cmml" xref="S3.SS2.SSS0.Px2.p2.6.m1.2.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p2.6.m1.2.3.2.cmml" xref="S3.SS2.SSS0.Px2.p2.6.m1.2.3.2">𝑾</ci><list id="S3.SS2.SSS0.Px2.p2.6.m1.2.2.2.3.cmml" xref="S3.SS2.SSS0.Px2.p2.6.m1.2.2.2.2"><apply id="S3.SS2.SSS0.Px2.p2.6.m1.2.2.2.2.1.cmml" xref="S3.SS2.SSS0.Px2.p2.6.m1.2.2.2.2.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p2.6.m1.2.2.2.2.1.1.cmml" xref="S3.SS2.SSS0.Px2.p2.6.m1.2.2.2.2.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p2.6.m1.2.2.2.2.1.2.cmml" xref="S3.SS2.SSS0.Px2.p2.6.m1.2.2.2.2.1.2">𝑇</ci><ci id="S3.SS2.SSS0.Px2.p2.6.m1.2.2.2.2.1.3.cmml" xref="S3.SS2.SSS0.Px2.p2.6.m1.2.2.2.2.1.3">𝑖</ci></apply><ci id="S3.SS2.SSS0.Px2.p2.6.m1.1.1.1.1a.cmml" xref="S3.SS2.SSS0.Px2.p2.6.m1.1.1.1.1"><mtext mathsize="70%" id="S3.SS2.SSS0.Px2.p2.6.m1.1.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p2.6.m1.1.1.1.1">extract</mtext></ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p2.6.m1.2c">\bm{W}_{T_{i},\text{extract}}</annotation></semantics></math> across all layers, we derive the extracted parameters <math id="S3.SS2.SSS0.Px2.p2.7.m2.1" class="ltx_Math" alttext="\mathbf{\Theta}_{T_{\text{extract}}}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p2.7.m2.1a"><msub id="S3.SS2.SSS0.Px2.p2.7.m2.1.1" xref="S3.SS2.SSS0.Px2.p2.7.m2.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p2.7.m2.1.1.2" xref="S3.SS2.SSS0.Px2.p2.7.m2.1.1.2.cmml">𝚯</mi><msub id="S3.SS2.SSS0.Px2.p2.7.m2.1.1.3" xref="S3.SS2.SSS0.Px2.p2.7.m2.1.1.3.cmml"><mi id="S3.SS2.SSS0.Px2.p2.7.m2.1.1.3.2" xref="S3.SS2.SSS0.Px2.p2.7.m2.1.1.3.2.cmml">T</mi><mtext id="S3.SS2.SSS0.Px2.p2.7.m2.1.1.3.3" xref="S3.SS2.SSS0.Px2.p2.7.m2.1.1.3.3a.cmml">extract</mtext></msub></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p2.7.m2.1b"><apply id="S3.SS2.SSS0.Px2.p2.7.m2.1.1.cmml" xref="S3.SS2.SSS0.Px2.p2.7.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p2.7.m2.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p2.7.m2.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p2.7.m2.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p2.7.m2.1.1.2">𝚯</ci><apply id="S3.SS2.SSS0.Px2.p2.7.m2.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p2.7.m2.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p2.7.m2.1.1.3.1.cmml" xref="S3.SS2.SSS0.Px2.p2.7.m2.1.1.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p2.7.m2.1.1.3.2.cmml" xref="S3.SS2.SSS0.Px2.p2.7.m2.1.1.3.2">𝑇</ci><ci id="S3.SS2.SSS0.Px2.p2.7.m2.1.1.3.3a.cmml" xref="S3.SS2.SSS0.Px2.p2.7.m2.1.1.3.3"><mtext mathsize="50%" id="S3.SS2.SSS0.Px2.p2.7.m2.1.1.3.3.cmml" xref="S3.SS2.SSS0.Px2.p2.7.m2.1.1.3.3">extract</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p2.7.m2.1c">\mathbf{\Theta}_{T_{\text{extract}}}</annotation></semantics></math> from <math id="S3.SS2.SSS0.Px2.p2.8.m3.1" class="ltx_Math" alttext="M_{T}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p2.8.m3.1a"><msub id="S3.SS2.SSS0.Px2.p2.8.m3.1.1" xref="S3.SS2.SSS0.Px2.p2.8.m3.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p2.8.m3.1.1.2" xref="S3.SS2.SSS0.Px2.p2.8.m3.1.1.2.cmml">M</mi><mi id="S3.SS2.SSS0.Px2.p2.8.m3.1.1.3" xref="S3.SS2.SSS0.Px2.p2.8.m3.1.1.3.cmml">T</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p2.8.m3.1b"><apply id="S3.SS2.SSS0.Px2.p2.8.m3.1.1.cmml" xref="S3.SS2.SSS0.Px2.p2.8.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p2.8.m3.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p2.8.m3.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p2.8.m3.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p2.8.m3.1.1.2">𝑀</ci><ci id="S3.SS2.SSS0.Px2.p2.8.m3.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p2.8.m3.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p2.8.m3.1c">M_{T}</annotation></semantics></math>.</p>
</div>
</section>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Knowledge Injection</h3>

<div id="S3.SS3.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.p1.1" class="ltx_p">To keep the architecture and the number of parameters of the student model unchanged during the knowledge transfer process, we employ the LoRA approach to instantiate the <math id="S3.SS3.p1.1.m1.1" class="ltx_Math" alttext="\text{Inject}(\cdot)" display="inline"><semantics id="S3.SS3.p1.1.m1.1a"><mrow id="S3.SS3.p1.1.m1.1.2" xref="S3.SS3.p1.1.m1.1.2.cmml"><mtext id="S3.SS3.p1.1.m1.1.2.2" xref="S3.SS3.p1.1.m1.1.2.2a.cmml">Inject</mtext><mo lspace="0em" rspace="0em" id="S3.SS3.p1.1.m1.1.2.1" xref="S3.SS3.p1.1.m1.1.2.1.cmml">​</mo><mrow id="S3.SS3.p1.1.m1.1.2.3.2" xref="S3.SS3.p1.1.m1.1.2.cmml"><mo stretchy="false" id="S3.SS3.p1.1.m1.1.2.3.2.1" xref="S3.SS3.p1.1.m1.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S3.SS3.p1.1.m1.1.1" xref="S3.SS3.p1.1.m1.1.1.cmml">⋅</mo><mo stretchy="false" id="S3.SS3.p1.1.m1.1.2.3.2.2" xref="S3.SS3.p1.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.1.m1.1b"><apply id="S3.SS3.p1.1.m1.1.2.cmml" xref="S3.SS3.p1.1.m1.1.2"><times id="S3.SS3.p1.1.m1.1.2.1.cmml" xref="S3.SS3.p1.1.m1.1.2.1"></times><ci id="S3.SS3.p1.1.m1.1.2.2a.cmml" xref="S3.SS3.p1.1.m1.1.2.2"><mtext id="S3.SS3.p1.1.m1.1.2.2.cmml" xref="S3.SS3.p1.1.m1.1.2.2">Inject</mtext></ci><ci id="S3.SS3.p1.1.m1.1.1.cmml" xref="S3.SS3.p1.1.m1.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.1.m1.1c">\text{Inject}(\cdot)</annotation></semantics></math> function.</p>
</div>
<section id="S3.SS3.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">LoRA Module.</h4>

<div id="S3.SS3.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS0.Px1.p1.1" class="ltx_p">LoRA <cite class="ltx_cite ltx_citemacro_citep">(Hu et al., <a href="#bib.bib24" title="" class="ltx_ref">2022</a>)</cite>, which stands for Low-Rank Adaptation, is a method designed to optimize parameter efficiency by freezing the pre-trained model weights and inserting trainable rank decomposition matrices into the deep neural network. The guiding intuition is that pre-trained language models possess low “intrinsic dimensions” <cite class="ltx_cite ltx_citemacro_citep">(Aghajanyan et al., <a href="#bib.bib1" title="" class="ltx_ref">2021</a>)</cite>. This means that even when projected to a smaller subspace, these models can still exhibit efficient learning. Consequently, it can be hypothesized that weight updates during adaptation also exhibit low “intrinsic ranks”. For a given pre-trained weight matrix <math id="S3.SS3.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="\bm{W}_{i}\in\mathbb{R}^{n\times m}" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.1.m1.1a"><mrow id="S3.SS3.SSS0.Px1.p1.1.m1.1.1" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.cmml"><msub id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.2" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.2.cmml">𝑾</mi><mi id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.3" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.3.cmml">i</mi></msub><mo id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.1" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.1.cmml">∈</mo><msup id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.cmml"><mi id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.2" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.3" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.3.cmml"><mi id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.3.2" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.3.2.cmml">n</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.3.1" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.3.1.cmml">×</mo><mi id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.3.3" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.3.3.cmml">m</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.1.m1.1b"><apply id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1"><in id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.1"></in><apply id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.2">𝑾</ci><ci id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.3">𝑖</ci></apply><apply id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.1.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3">superscript</csymbol><ci id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.2.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.2">ℝ</ci><apply id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.3.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.3"><times id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.3.1.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.3.1"></times><ci id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.3.2.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.3.2">𝑛</ci><ci id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.3.3.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.3.3">𝑚</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.1.m1.1c">\bm{W}_{i}\in\mathbb{R}^{n\times m}</annotation></semantics></math>, it can be updated as:</p>
</div>
<div id="S3.SS3.SSS0.Px1.p2" class="ltx_para ltx_noindent">
<table id="S3.E6" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E6.m1.1" class="ltx_Math" alttext="\bm{W}_{i}^{*}=\bm{W}_{i}+\Delta\bm{W}=\bm{W}_{i}+\bm{B}_{i}\bm{A}_{i},\quad\bm{B}_{i}\in\mathbb{R}^{n\times r},\quad\bm{A}_{i}\in\mathbb{R}^{r\times m}," display="block"><semantics id="S3.E6.m1.1a"><mrow id="S3.E6.m1.1.1.1"><mrow id="S3.E6.m1.1.1.1.1.2" xref="S3.E6.m1.1.1.1.1.3.cmml"><mrow id="S3.E6.m1.1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.1.cmml"><msubsup id="S3.E6.m1.1.1.1.1.1.1.2" xref="S3.E6.m1.1.1.1.1.1.1.2.cmml"><mi id="S3.E6.m1.1.1.1.1.1.1.2.2.2" xref="S3.E6.m1.1.1.1.1.1.1.2.2.2.cmml">𝑾</mi><mi id="S3.E6.m1.1.1.1.1.1.1.2.2.3" xref="S3.E6.m1.1.1.1.1.1.1.2.2.3.cmml">i</mi><mo id="S3.E6.m1.1.1.1.1.1.1.2.3" xref="S3.E6.m1.1.1.1.1.1.1.2.3.cmml">∗</mo></msubsup><mo id="S3.E6.m1.1.1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.1.1.3.cmml">=</mo><mrow id="S3.E6.m1.1.1.1.1.1.1.4" xref="S3.E6.m1.1.1.1.1.1.1.4.cmml"><msub id="S3.E6.m1.1.1.1.1.1.1.4.2" xref="S3.E6.m1.1.1.1.1.1.1.4.2.cmml"><mi id="S3.E6.m1.1.1.1.1.1.1.4.2.2" xref="S3.E6.m1.1.1.1.1.1.1.4.2.2.cmml">𝑾</mi><mi id="S3.E6.m1.1.1.1.1.1.1.4.2.3" xref="S3.E6.m1.1.1.1.1.1.1.4.2.3.cmml">i</mi></msub><mo id="S3.E6.m1.1.1.1.1.1.1.4.1" xref="S3.E6.m1.1.1.1.1.1.1.4.1.cmml">+</mo><mrow id="S3.E6.m1.1.1.1.1.1.1.4.3" xref="S3.E6.m1.1.1.1.1.1.1.4.3.cmml"><mi mathvariant="normal" id="S3.E6.m1.1.1.1.1.1.1.4.3.2" xref="S3.E6.m1.1.1.1.1.1.1.4.3.2.cmml">Δ</mi><mo lspace="0em" rspace="0em" id="S3.E6.m1.1.1.1.1.1.1.4.3.1" xref="S3.E6.m1.1.1.1.1.1.1.4.3.1.cmml">​</mo><mi id="S3.E6.m1.1.1.1.1.1.1.4.3.3" xref="S3.E6.m1.1.1.1.1.1.1.4.3.3.cmml">𝑾</mi></mrow></mrow><mo id="S3.E6.m1.1.1.1.1.1.1.5" xref="S3.E6.m1.1.1.1.1.1.1.5.cmml">=</mo><mrow id="S3.E6.m1.1.1.1.1.1.1.6" xref="S3.E6.m1.1.1.1.1.1.1.6.cmml"><msub id="S3.E6.m1.1.1.1.1.1.1.6.2" xref="S3.E6.m1.1.1.1.1.1.1.6.2.cmml"><mi id="S3.E6.m1.1.1.1.1.1.1.6.2.2" xref="S3.E6.m1.1.1.1.1.1.1.6.2.2.cmml">𝑾</mi><mi id="S3.E6.m1.1.1.1.1.1.1.6.2.3" xref="S3.E6.m1.1.1.1.1.1.1.6.2.3.cmml">i</mi></msub><mo id="S3.E6.m1.1.1.1.1.1.1.6.1" xref="S3.E6.m1.1.1.1.1.1.1.6.1.cmml">+</mo><mrow id="S3.E6.m1.1.1.1.1.1.1.6.3" xref="S3.E6.m1.1.1.1.1.1.1.6.3.cmml"><msub id="S3.E6.m1.1.1.1.1.1.1.6.3.2" xref="S3.E6.m1.1.1.1.1.1.1.6.3.2.cmml"><mi id="S3.E6.m1.1.1.1.1.1.1.6.3.2.2" xref="S3.E6.m1.1.1.1.1.1.1.6.3.2.2.cmml">𝑩</mi><mi id="S3.E6.m1.1.1.1.1.1.1.6.3.2.3" xref="S3.E6.m1.1.1.1.1.1.1.6.3.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.E6.m1.1.1.1.1.1.1.6.3.1" xref="S3.E6.m1.1.1.1.1.1.1.6.3.1.cmml">​</mo><msub id="S3.E6.m1.1.1.1.1.1.1.6.3.3" xref="S3.E6.m1.1.1.1.1.1.1.6.3.3.cmml"><mi id="S3.E6.m1.1.1.1.1.1.1.6.3.3.2" xref="S3.E6.m1.1.1.1.1.1.1.6.3.3.2.cmml">𝑨</mi><mi id="S3.E6.m1.1.1.1.1.1.1.6.3.3.3" xref="S3.E6.m1.1.1.1.1.1.1.6.3.3.3.cmml">i</mi></msub></mrow></mrow></mrow><mo rspace="1.167em" id="S3.E6.m1.1.1.1.1.2.3" xref="S3.E6.m1.1.1.1.1.3a.cmml">,</mo><mrow id="S3.E6.m1.1.1.1.1.2.2.2" xref="S3.E6.m1.1.1.1.1.2.2.3.cmml"><mrow id="S3.E6.m1.1.1.1.1.2.2.1.1" xref="S3.E6.m1.1.1.1.1.2.2.1.1.cmml"><msub id="S3.E6.m1.1.1.1.1.2.2.1.1.2" xref="S3.E6.m1.1.1.1.1.2.2.1.1.2.cmml"><mi id="S3.E6.m1.1.1.1.1.2.2.1.1.2.2" xref="S3.E6.m1.1.1.1.1.2.2.1.1.2.2.cmml">𝑩</mi><mi id="S3.E6.m1.1.1.1.1.2.2.1.1.2.3" xref="S3.E6.m1.1.1.1.1.2.2.1.1.2.3.cmml">i</mi></msub><mo id="S3.E6.m1.1.1.1.1.2.2.1.1.1" xref="S3.E6.m1.1.1.1.1.2.2.1.1.1.cmml">∈</mo><msup id="S3.E6.m1.1.1.1.1.2.2.1.1.3" xref="S3.E6.m1.1.1.1.1.2.2.1.1.3.cmml"><mi id="S3.E6.m1.1.1.1.1.2.2.1.1.3.2" xref="S3.E6.m1.1.1.1.1.2.2.1.1.3.2.cmml">ℝ</mi><mrow id="S3.E6.m1.1.1.1.1.2.2.1.1.3.3" xref="S3.E6.m1.1.1.1.1.2.2.1.1.3.3.cmml"><mi id="S3.E6.m1.1.1.1.1.2.2.1.1.3.3.2" xref="S3.E6.m1.1.1.1.1.2.2.1.1.3.3.2.cmml">n</mi><mo lspace="0.222em" rspace="0.222em" id="S3.E6.m1.1.1.1.1.2.2.1.1.3.3.1" xref="S3.E6.m1.1.1.1.1.2.2.1.1.3.3.1.cmml">×</mo><mi id="S3.E6.m1.1.1.1.1.2.2.1.1.3.3.3" xref="S3.E6.m1.1.1.1.1.2.2.1.1.3.3.3.cmml">r</mi></mrow></msup></mrow><mo rspace="1.167em" id="S3.E6.m1.1.1.1.1.2.2.2.3" xref="S3.E6.m1.1.1.1.1.2.2.3a.cmml">,</mo><mrow id="S3.E6.m1.1.1.1.1.2.2.2.2" xref="S3.E6.m1.1.1.1.1.2.2.2.2.cmml"><msub id="S3.E6.m1.1.1.1.1.2.2.2.2.2" xref="S3.E6.m1.1.1.1.1.2.2.2.2.2.cmml"><mi id="S3.E6.m1.1.1.1.1.2.2.2.2.2.2" xref="S3.E6.m1.1.1.1.1.2.2.2.2.2.2.cmml">𝑨</mi><mi id="S3.E6.m1.1.1.1.1.2.2.2.2.2.3" xref="S3.E6.m1.1.1.1.1.2.2.2.2.2.3.cmml">i</mi></msub><mo id="S3.E6.m1.1.1.1.1.2.2.2.2.1" xref="S3.E6.m1.1.1.1.1.2.2.2.2.1.cmml">∈</mo><msup id="S3.E6.m1.1.1.1.1.2.2.2.2.3" xref="S3.E6.m1.1.1.1.1.2.2.2.2.3.cmml"><mi id="S3.E6.m1.1.1.1.1.2.2.2.2.3.2" xref="S3.E6.m1.1.1.1.1.2.2.2.2.3.2.cmml">ℝ</mi><mrow id="S3.E6.m1.1.1.1.1.2.2.2.2.3.3" xref="S3.E6.m1.1.1.1.1.2.2.2.2.3.3.cmml"><mi id="S3.E6.m1.1.1.1.1.2.2.2.2.3.3.2" xref="S3.E6.m1.1.1.1.1.2.2.2.2.3.3.2.cmml">r</mi><mo lspace="0.222em" rspace="0.222em" id="S3.E6.m1.1.1.1.1.2.2.2.2.3.3.1" xref="S3.E6.m1.1.1.1.1.2.2.2.2.3.3.1.cmml">×</mo><mi id="S3.E6.m1.1.1.1.1.2.2.2.2.3.3.3" xref="S3.E6.m1.1.1.1.1.2.2.2.2.3.3.3.cmml">m</mi></mrow></msup></mrow></mrow></mrow><mo id="S3.E6.m1.1.1.1.2">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E6.m1.1b"><apply id="S3.E6.m1.1.1.1.1.3.cmml" xref="S3.E6.m1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.3a.cmml" xref="S3.E6.m1.1.1.1.1.2.3">formulae-sequence</csymbol><apply id="S3.E6.m1.1.1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1"><and id="S3.E6.m1.1.1.1.1.1.1a.cmml" xref="S3.E6.m1.1.1.1.1.1.1"></and><apply id="S3.E6.m1.1.1.1.1.1.1b.cmml" xref="S3.E6.m1.1.1.1.1.1.1"><eq id="S3.E6.m1.1.1.1.1.1.1.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3"></eq><apply id="S3.E6.m1.1.1.1.1.1.1.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.1.2.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.2">superscript</csymbol><apply id="S3.E6.m1.1.1.1.1.1.1.2.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.1.2.2.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.E6.m1.1.1.1.1.1.1.2.2.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.2.2.2">𝑾</ci><ci id="S3.E6.m1.1.1.1.1.1.1.2.2.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.2.2.3">𝑖</ci></apply><times id="S3.E6.m1.1.1.1.1.1.1.2.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.2.3"></times></apply><apply id="S3.E6.m1.1.1.1.1.1.1.4.cmml" xref="S3.E6.m1.1.1.1.1.1.1.4"><plus id="S3.E6.m1.1.1.1.1.1.1.4.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.4.1"></plus><apply id="S3.E6.m1.1.1.1.1.1.1.4.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.4.2"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.1.4.2.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.4.2">subscript</csymbol><ci id="S3.E6.m1.1.1.1.1.1.1.4.2.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.4.2.2">𝑾</ci><ci id="S3.E6.m1.1.1.1.1.1.1.4.2.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.4.2.3">𝑖</ci></apply><apply id="S3.E6.m1.1.1.1.1.1.1.4.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.4.3"><times id="S3.E6.m1.1.1.1.1.1.1.4.3.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.4.3.1"></times><ci id="S3.E6.m1.1.1.1.1.1.1.4.3.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.4.3.2">Δ</ci><ci id="S3.E6.m1.1.1.1.1.1.1.4.3.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.4.3.3">𝑾</ci></apply></apply></apply><apply id="S3.E6.m1.1.1.1.1.1.1c.cmml" xref="S3.E6.m1.1.1.1.1.1.1"><eq id="S3.E6.m1.1.1.1.1.1.1.5.cmml" xref="S3.E6.m1.1.1.1.1.1.1.5"></eq><share href="#S3.E6.m1.1.1.1.1.1.1.4.cmml" id="S3.E6.m1.1.1.1.1.1.1d.cmml" xref="S3.E6.m1.1.1.1.1.1.1"></share><apply id="S3.E6.m1.1.1.1.1.1.1.6.cmml" xref="S3.E6.m1.1.1.1.1.1.1.6"><plus id="S3.E6.m1.1.1.1.1.1.1.6.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.6.1"></plus><apply id="S3.E6.m1.1.1.1.1.1.1.6.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.6.2"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.1.6.2.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.6.2">subscript</csymbol><ci id="S3.E6.m1.1.1.1.1.1.1.6.2.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.6.2.2">𝑾</ci><ci id="S3.E6.m1.1.1.1.1.1.1.6.2.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.6.2.3">𝑖</ci></apply><apply id="S3.E6.m1.1.1.1.1.1.1.6.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.6.3"><times id="S3.E6.m1.1.1.1.1.1.1.6.3.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.6.3.1"></times><apply id="S3.E6.m1.1.1.1.1.1.1.6.3.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.6.3.2"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.1.6.3.2.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.6.3.2">subscript</csymbol><ci id="S3.E6.m1.1.1.1.1.1.1.6.3.2.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.6.3.2.2">𝑩</ci><ci id="S3.E6.m1.1.1.1.1.1.1.6.3.2.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.6.3.2.3">𝑖</ci></apply><apply id="S3.E6.m1.1.1.1.1.1.1.6.3.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.6.3.3"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.1.6.3.3.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.6.3.3">subscript</csymbol><ci id="S3.E6.m1.1.1.1.1.1.1.6.3.3.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.6.3.3.2">𝑨</ci><ci id="S3.E6.m1.1.1.1.1.1.1.6.3.3.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.6.3.3.3">𝑖</ci></apply></apply></apply></apply></apply><apply id="S3.E6.m1.1.1.1.1.2.2.3.cmml" xref="S3.E6.m1.1.1.1.1.2.2.2"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.2.2.3a.cmml" xref="S3.E6.m1.1.1.1.1.2.2.2.3">formulae-sequence</csymbol><apply id="S3.E6.m1.1.1.1.1.2.2.1.1.cmml" xref="S3.E6.m1.1.1.1.1.2.2.1.1"><in id="S3.E6.m1.1.1.1.1.2.2.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1.2.2.1.1.1"></in><apply id="S3.E6.m1.1.1.1.1.2.2.1.1.2.cmml" xref="S3.E6.m1.1.1.1.1.2.2.1.1.2"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.2.2.1.1.2.1.cmml" xref="S3.E6.m1.1.1.1.1.2.2.1.1.2">subscript</csymbol><ci id="S3.E6.m1.1.1.1.1.2.2.1.1.2.2.cmml" xref="S3.E6.m1.1.1.1.1.2.2.1.1.2.2">𝑩</ci><ci id="S3.E6.m1.1.1.1.1.2.2.1.1.2.3.cmml" xref="S3.E6.m1.1.1.1.1.2.2.1.1.2.3">𝑖</ci></apply><apply id="S3.E6.m1.1.1.1.1.2.2.1.1.3.cmml" xref="S3.E6.m1.1.1.1.1.2.2.1.1.3"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.2.2.1.1.3.1.cmml" xref="S3.E6.m1.1.1.1.1.2.2.1.1.3">superscript</csymbol><ci id="S3.E6.m1.1.1.1.1.2.2.1.1.3.2.cmml" xref="S3.E6.m1.1.1.1.1.2.2.1.1.3.2">ℝ</ci><apply id="S3.E6.m1.1.1.1.1.2.2.1.1.3.3.cmml" xref="S3.E6.m1.1.1.1.1.2.2.1.1.3.3"><times id="S3.E6.m1.1.1.1.1.2.2.1.1.3.3.1.cmml" xref="S3.E6.m1.1.1.1.1.2.2.1.1.3.3.1"></times><ci id="S3.E6.m1.1.1.1.1.2.2.1.1.3.3.2.cmml" xref="S3.E6.m1.1.1.1.1.2.2.1.1.3.3.2">𝑛</ci><ci id="S3.E6.m1.1.1.1.1.2.2.1.1.3.3.3.cmml" xref="S3.E6.m1.1.1.1.1.2.2.1.1.3.3.3">𝑟</ci></apply></apply></apply><apply id="S3.E6.m1.1.1.1.1.2.2.2.2.cmml" xref="S3.E6.m1.1.1.1.1.2.2.2.2"><in id="S3.E6.m1.1.1.1.1.2.2.2.2.1.cmml" xref="S3.E6.m1.1.1.1.1.2.2.2.2.1"></in><apply id="S3.E6.m1.1.1.1.1.2.2.2.2.2.cmml" xref="S3.E6.m1.1.1.1.1.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.2.2.2.2.2.1.cmml" xref="S3.E6.m1.1.1.1.1.2.2.2.2.2">subscript</csymbol><ci id="S3.E6.m1.1.1.1.1.2.2.2.2.2.2.cmml" xref="S3.E6.m1.1.1.1.1.2.2.2.2.2.2">𝑨</ci><ci id="S3.E6.m1.1.1.1.1.2.2.2.2.2.3.cmml" xref="S3.E6.m1.1.1.1.1.2.2.2.2.2.3">𝑖</ci></apply><apply id="S3.E6.m1.1.1.1.1.2.2.2.2.3.cmml" xref="S3.E6.m1.1.1.1.1.2.2.2.2.3"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.2.2.2.2.3.1.cmml" xref="S3.E6.m1.1.1.1.1.2.2.2.2.3">superscript</csymbol><ci id="S3.E6.m1.1.1.1.1.2.2.2.2.3.2.cmml" xref="S3.E6.m1.1.1.1.1.2.2.2.2.3.2">ℝ</ci><apply id="S3.E6.m1.1.1.1.1.2.2.2.2.3.3.cmml" xref="S3.E6.m1.1.1.1.1.2.2.2.2.3.3"><times id="S3.E6.m1.1.1.1.1.2.2.2.2.3.3.1.cmml" xref="S3.E6.m1.1.1.1.1.2.2.2.2.3.3.1"></times><ci id="S3.E6.m1.1.1.1.1.2.2.2.2.3.3.2.cmml" xref="S3.E6.m1.1.1.1.1.2.2.2.2.3.3.2">𝑟</ci><ci id="S3.E6.m1.1.1.1.1.2.2.2.2.3.3.3.cmml" xref="S3.E6.m1.1.1.1.1.2.2.2.2.3.3.3">𝑚</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E6.m1.1c">\bm{W}_{i}^{*}=\bm{W}_{i}+\Delta\bm{W}=\bm{W}_{i}+\bm{B}_{i}\bm{A}_{i},\quad\bm{B}_{i}\in\mathbb{R}^{n\times r},\quad\bm{A}_{i}\in\mathbb{R}^{r\times m},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(6)</span></td>
</tr></tbody>
</table>
</div>
<div id="S3.SS3.SSS0.Px1.p3" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS0.Px1.p3.7" class="ltx_p">where <math id="S3.SS3.SSS0.Px1.p3.1.m1.1" class="ltx_Math" alttext="r" display="inline"><semantics id="S3.SS3.SSS0.Px1.p3.1.m1.1a"><mi id="S3.SS3.SSS0.Px1.p3.1.m1.1.1" xref="S3.SS3.SSS0.Px1.p3.1.m1.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p3.1.m1.1b"><ci id="S3.SS3.SSS0.Px1.p3.1.m1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p3.1.m1.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p3.1.m1.1c">r</annotation></semantics></math> represents the low rank with <math id="S3.SS3.SSS0.Px1.p3.2.m2.3" class="ltx_Math" alttext="r\ll\min(n,m)" display="inline"><semantics id="S3.SS3.SSS0.Px1.p3.2.m2.3a"><mrow id="S3.SS3.SSS0.Px1.p3.2.m2.3.4" xref="S3.SS3.SSS0.Px1.p3.2.m2.3.4.cmml"><mi id="S3.SS3.SSS0.Px1.p3.2.m2.3.4.2" xref="S3.SS3.SSS0.Px1.p3.2.m2.3.4.2.cmml">r</mi><mo id="S3.SS3.SSS0.Px1.p3.2.m2.3.4.1" xref="S3.SS3.SSS0.Px1.p3.2.m2.3.4.1.cmml">≪</mo><mrow id="S3.SS3.SSS0.Px1.p3.2.m2.3.4.3.2" xref="S3.SS3.SSS0.Px1.p3.2.m2.3.4.3.1.cmml"><mi id="S3.SS3.SSS0.Px1.p3.2.m2.1.1" xref="S3.SS3.SSS0.Px1.p3.2.m2.1.1.cmml">min</mi><mo id="S3.SS3.SSS0.Px1.p3.2.m2.3.4.3.2a" xref="S3.SS3.SSS0.Px1.p3.2.m2.3.4.3.1.cmml">⁡</mo><mrow id="S3.SS3.SSS0.Px1.p3.2.m2.3.4.3.2.1" xref="S3.SS3.SSS0.Px1.p3.2.m2.3.4.3.1.cmml"><mo stretchy="false" id="S3.SS3.SSS0.Px1.p3.2.m2.3.4.3.2.1.1" xref="S3.SS3.SSS0.Px1.p3.2.m2.3.4.3.1.cmml">(</mo><mi id="S3.SS3.SSS0.Px1.p3.2.m2.2.2" xref="S3.SS3.SSS0.Px1.p3.2.m2.2.2.cmml">n</mi><mo id="S3.SS3.SSS0.Px1.p3.2.m2.3.4.3.2.1.2" xref="S3.SS3.SSS0.Px1.p3.2.m2.3.4.3.1.cmml">,</mo><mi id="S3.SS3.SSS0.Px1.p3.2.m2.3.3" xref="S3.SS3.SSS0.Px1.p3.2.m2.3.3.cmml">m</mi><mo stretchy="false" id="S3.SS3.SSS0.Px1.p3.2.m2.3.4.3.2.1.3" xref="S3.SS3.SSS0.Px1.p3.2.m2.3.4.3.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p3.2.m2.3b"><apply id="S3.SS3.SSS0.Px1.p3.2.m2.3.4.cmml" xref="S3.SS3.SSS0.Px1.p3.2.m2.3.4"><csymbol cd="latexml" id="S3.SS3.SSS0.Px1.p3.2.m2.3.4.1.cmml" xref="S3.SS3.SSS0.Px1.p3.2.m2.3.4.1">much-less-than</csymbol><ci id="S3.SS3.SSS0.Px1.p3.2.m2.3.4.2.cmml" xref="S3.SS3.SSS0.Px1.p3.2.m2.3.4.2">𝑟</ci><apply id="S3.SS3.SSS0.Px1.p3.2.m2.3.4.3.1.cmml" xref="S3.SS3.SSS0.Px1.p3.2.m2.3.4.3.2"><min id="S3.SS3.SSS0.Px1.p3.2.m2.1.1.cmml" xref="S3.SS3.SSS0.Px1.p3.2.m2.1.1"></min><ci id="S3.SS3.SSS0.Px1.p3.2.m2.2.2.cmml" xref="S3.SS3.SSS0.Px1.p3.2.m2.2.2">𝑛</ci><ci id="S3.SS3.SSS0.Px1.p3.2.m2.3.3.cmml" xref="S3.SS3.SSS0.Px1.p3.2.m2.3.3">𝑚</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p3.2.m2.3c">r\ll\min(n,m)</annotation></semantics></math>. The matrix <math id="S3.SS3.SSS0.Px1.p3.3.m3.1" class="ltx_Math" alttext="\bm{W}_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px1.p3.3.m3.1a"><msub id="S3.SS3.SSS0.Px1.p3.3.m3.1.1" xref="S3.SS3.SSS0.Px1.p3.3.m3.1.1.cmml"><mi id="S3.SS3.SSS0.Px1.p3.3.m3.1.1.2" xref="S3.SS3.SSS0.Px1.p3.3.m3.1.1.2.cmml">𝑾</mi><mi id="S3.SS3.SSS0.Px1.p3.3.m3.1.1.3" xref="S3.SS3.SSS0.Px1.p3.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p3.3.m3.1b"><apply id="S3.SS3.SSS0.Px1.p3.3.m3.1.1.cmml" xref="S3.SS3.SSS0.Px1.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p3.3.m3.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p3.3.m3.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p3.3.m3.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p3.3.m3.1.1.2">𝑾</ci><ci id="S3.SS3.SSS0.Px1.p3.3.m3.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p3.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p3.3.m3.1c">\bm{W}_{i}</annotation></semantics></math> remains constant during this operation, implying that only <math id="S3.SS3.SSS0.Px1.p3.4.m4.1" class="ltx_Math" alttext="\bm{B}_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px1.p3.4.m4.1a"><msub id="S3.SS3.SSS0.Px1.p3.4.m4.1.1" xref="S3.SS3.SSS0.Px1.p3.4.m4.1.1.cmml"><mi id="S3.SS3.SSS0.Px1.p3.4.m4.1.1.2" xref="S3.SS3.SSS0.Px1.p3.4.m4.1.1.2.cmml">𝑩</mi><mi id="S3.SS3.SSS0.Px1.p3.4.m4.1.1.3" xref="S3.SS3.SSS0.Px1.p3.4.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p3.4.m4.1b"><apply id="S3.SS3.SSS0.Px1.p3.4.m4.1.1.cmml" xref="S3.SS3.SSS0.Px1.p3.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p3.4.m4.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p3.4.m4.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p3.4.m4.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p3.4.m4.1.1.2">𝑩</ci><ci id="S3.SS3.SSS0.Px1.p3.4.m4.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p3.4.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p3.4.m4.1c">\bm{B}_{i}</annotation></semantics></math> and <math id="S3.SS3.SSS0.Px1.p3.5.m5.1" class="ltx_Math" alttext="\bm{A}_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px1.p3.5.m5.1a"><msub id="S3.SS3.SSS0.Px1.p3.5.m5.1.1" xref="S3.SS3.SSS0.Px1.p3.5.m5.1.1.cmml"><mi id="S3.SS3.SSS0.Px1.p3.5.m5.1.1.2" xref="S3.SS3.SSS0.Px1.p3.5.m5.1.1.2.cmml">𝑨</mi><mi id="S3.SS3.SSS0.Px1.p3.5.m5.1.1.3" xref="S3.SS3.SSS0.Px1.p3.5.m5.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p3.5.m5.1b"><apply id="S3.SS3.SSS0.Px1.p3.5.m5.1.1.cmml" xref="S3.SS3.SSS0.Px1.p3.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p3.5.m5.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p3.5.m5.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p3.5.m5.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p3.5.m5.1.1.2">𝑨</ci><ci id="S3.SS3.SSS0.Px1.p3.5.m5.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p3.5.m5.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p3.5.m5.1c">\bm{A}_{i}</annotation></semantics></math> are updated in the training phase. To ensure that training commences from the original pre-trained weights, either <math id="S3.SS3.SSS0.Px1.p3.6.m6.1" class="ltx_Math" alttext="\bm{B}_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px1.p3.6.m6.1a"><msub id="S3.SS3.SSS0.Px1.p3.6.m6.1.1" xref="S3.SS3.SSS0.Px1.p3.6.m6.1.1.cmml"><mi id="S3.SS3.SSS0.Px1.p3.6.m6.1.1.2" xref="S3.SS3.SSS0.Px1.p3.6.m6.1.1.2.cmml">𝑩</mi><mi id="S3.SS3.SSS0.Px1.p3.6.m6.1.1.3" xref="S3.SS3.SSS0.Px1.p3.6.m6.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p3.6.m6.1b"><apply id="S3.SS3.SSS0.Px1.p3.6.m6.1.1.cmml" xref="S3.SS3.SSS0.Px1.p3.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p3.6.m6.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p3.6.m6.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p3.6.m6.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p3.6.m6.1.1.2">𝑩</ci><ci id="S3.SS3.SSS0.Px1.p3.6.m6.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p3.6.m6.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p3.6.m6.1c">\bm{B}_{i}</annotation></semantics></math> or <math id="S3.SS3.SSS0.Px1.p3.7.m7.1" class="ltx_Math" alttext="\bm{A}_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px1.p3.7.m7.1a"><msub id="S3.SS3.SSS0.Px1.p3.7.m7.1.1" xref="S3.SS3.SSS0.Px1.p3.7.m7.1.1.cmml"><mi id="S3.SS3.SSS0.Px1.p3.7.m7.1.1.2" xref="S3.SS3.SSS0.Px1.p3.7.m7.1.1.2.cmml">𝑨</mi><mi id="S3.SS3.SSS0.Px1.p3.7.m7.1.1.3" xref="S3.SS3.SSS0.Px1.p3.7.m7.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p3.7.m7.1b"><apply id="S3.SS3.SSS0.Px1.p3.7.m7.1.1.cmml" xref="S3.SS3.SSS0.Px1.p3.7.m7.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p3.7.m7.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p3.7.m7.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px1.p3.7.m7.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p3.7.m7.1.1.2">𝑨</ci><ci id="S3.SS3.SSS0.Px1.p3.7.m7.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p3.7.m7.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p3.7.m7.1c">\bm{A}_{i}</annotation></semantics></math> is initialized with zeros.</p>
</div>
</section>
<section id="S3.SS3.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Knowledge Injection with LoRA.</h4>

<div id="S3.SS3.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS0.Px2.p1.2" class="ltx_p">The main goal of this step is to integrate knowledge from the teacher model by incorporating extracted parameters into the student’s LoRA module. Initially, SVD is adopted to factorize each matrix <math id="S3.SS3.SSS0.Px2.p1.1.m1.2" class="ltx_Math" alttext="\bm{W}_{T_{i},\text{extract}}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p1.1.m1.2a"><msub id="S3.SS3.SSS0.Px2.p1.1.m1.2.3" xref="S3.SS3.SSS0.Px2.p1.1.m1.2.3.cmml"><mi id="S3.SS3.SSS0.Px2.p1.1.m1.2.3.2" xref="S3.SS3.SSS0.Px2.p1.1.m1.2.3.2.cmml">𝑾</mi><mrow id="S3.SS3.SSS0.Px2.p1.1.m1.2.2.2.2" xref="S3.SS3.SSS0.Px2.p1.1.m1.2.2.2.3.cmml"><msub id="S3.SS3.SSS0.Px2.p1.1.m1.2.2.2.2.1" xref="S3.SS3.SSS0.Px2.p1.1.m1.2.2.2.2.1.cmml"><mi id="S3.SS3.SSS0.Px2.p1.1.m1.2.2.2.2.1.2" xref="S3.SS3.SSS0.Px2.p1.1.m1.2.2.2.2.1.2.cmml">T</mi><mi id="S3.SS3.SSS0.Px2.p1.1.m1.2.2.2.2.1.3" xref="S3.SS3.SSS0.Px2.p1.1.m1.2.2.2.2.1.3.cmml">i</mi></msub><mo id="S3.SS3.SSS0.Px2.p1.1.m1.2.2.2.2.2" xref="S3.SS3.SSS0.Px2.p1.1.m1.2.2.2.3.cmml">,</mo><mtext id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.1.1" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.1.1a.cmml">extract</mtext></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p1.1.m1.2b"><apply id="S3.SS3.SSS0.Px2.p1.1.m1.2.3.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.2.3"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p1.1.m1.2.3.1.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.2.3">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p1.1.m1.2.3.2.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.2.3.2">𝑾</ci><list id="S3.SS3.SSS0.Px2.p1.1.m1.2.2.2.3.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.2.2.2.2"><apply id="S3.SS3.SSS0.Px2.p1.1.m1.2.2.2.2.1.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.2.2.2.2.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p1.1.m1.2.2.2.2.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.2.2.2.2.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p1.1.m1.2.2.2.2.1.2.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.2.2.2.2.1.2">𝑇</ci><ci id="S3.SS3.SSS0.Px2.p1.1.m1.2.2.2.2.1.3.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.2.2.2.2.1.3">𝑖</ci></apply><ci id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.1.1a.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.1.1"><mtext mathsize="70%" id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.1.1">extract</mtext></ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p1.1.m1.2c">\bm{W}_{T_{i},\text{extract}}</annotation></semantics></math> in <math id="S3.SS3.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{\Theta}_{T_{\text{extract}}}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p1.2.m2.1a"><msub id="S3.SS3.SSS0.Px2.p1.2.m2.1.1" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.2" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.2.cmml">𝚯</mi><msub id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.3" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.3.cmml"><mi id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.3.2" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.3.2.cmml">T</mi><mtext id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.3.3" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.3.3a.cmml">extract</mtext></msub></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p1.2.m2.1b"><apply id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.2">𝚯</ci><apply id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.3.1.cmml" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.3">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.3.2.cmml" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.3.2">𝑇</ci><ci id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.3.3a.cmml" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.3.3"><mtext mathsize="50%" id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.3.3.cmml" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.3.3">extract</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p1.2.m2.1c">\mathbf{\Theta}_{T_{\text{extract}}}</annotation></semantics></math> into three constituent matrices as:</p>
</div>
<div id="S3.SS3.SSS0.Px2.p2" class="ltx_para ltx_noindent">
<table id="S3.E7" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E7.m1.3" class="ltx_Math" alttext="\bm{W}_{T_{i},\text{extract}}=\bm{U}_{i}\bm{\Sigma}_{i}\bm{V}_{i}^{\top}." display="block"><semantics id="S3.E7.m1.3a"><mrow id="S3.E7.m1.3.3.1" xref="S3.E7.m1.3.3.1.1.cmml"><mrow id="S3.E7.m1.3.3.1.1" xref="S3.E7.m1.3.3.1.1.cmml"><msub id="S3.E7.m1.3.3.1.1.2" xref="S3.E7.m1.3.3.1.1.2.cmml"><mi id="S3.E7.m1.3.3.1.1.2.2" xref="S3.E7.m1.3.3.1.1.2.2.cmml">𝑾</mi><mrow id="S3.E7.m1.2.2.2.2" xref="S3.E7.m1.2.2.2.3.cmml"><msub id="S3.E7.m1.2.2.2.2.1" xref="S3.E7.m1.2.2.2.2.1.cmml"><mi id="S3.E7.m1.2.2.2.2.1.2" xref="S3.E7.m1.2.2.2.2.1.2.cmml">T</mi><mi id="S3.E7.m1.2.2.2.2.1.3" xref="S3.E7.m1.2.2.2.2.1.3.cmml">i</mi></msub><mo id="S3.E7.m1.2.2.2.2.2" xref="S3.E7.m1.2.2.2.3.cmml">,</mo><mtext id="S3.E7.m1.1.1.1.1" xref="S3.E7.m1.1.1.1.1a.cmml">extract</mtext></mrow></msub><mo id="S3.E7.m1.3.3.1.1.1" xref="S3.E7.m1.3.3.1.1.1.cmml">=</mo><mrow id="S3.E7.m1.3.3.1.1.3" xref="S3.E7.m1.3.3.1.1.3.cmml"><msub id="S3.E7.m1.3.3.1.1.3.2" xref="S3.E7.m1.3.3.1.1.3.2.cmml"><mi id="S3.E7.m1.3.3.1.1.3.2.2" xref="S3.E7.m1.3.3.1.1.3.2.2.cmml">𝑼</mi><mi id="S3.E7.m1.3.3.1.1.3.2.3" xref="S3.E7.m1.3.3.1.1.3.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.E7.m1.3.3.1.1.3.1" xref="S3.E7.m1.3.3.1.1.3.1.cmml">​</mo><msub id="S3.E7.m1.3.3.1.1.3.3" xref="S3.E7.m1.3.3.1.1.3.3.cmml"><mi id="S3.E7.m1.3.3.1.1.3.3.2" xref="S3.E7.m1.3.3.1.1.3.3.2.cmml">𝚺</mi><mi id="S3.E7.m1.3.3.1.1.3.3.3" xref="S3.E7.m1.3.3.1.1.3.3.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.E7.m1.3.3.1.1.3.1a" xref="S3.E7.m1.3.3.1.1.3.1.cmml">​</mo><msubsup id="S3.E7.m1.3.3.1.1.3.4" xref="S3.E7.m1.3.3.1.1.3.4.cmml"><mi id="S3.E7.m1.3.3.1.1.3.4.2.2" xref="S3.E7.m1.3.3.1.1.3.4.2.2.cmml">𝑽</mi><mi id="S3.E7.m1.3.3.1.1.3.4.2.3" xref="S3.E7.m1.3.3.1.1.3.4.2.3.cmml">i</mi><mo id="S3.E7.m1.3.3.1.1.3.4.3" xref="S3.E7.m1.3.3.1.1.3.4.3.cmml">⊤</mo></msubsup></mrow></mrow><mo lspace="0em" id="S3.E7.m1.3.3.1.2" xref="S3.E7.m1.3.3.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E7.m1.3b"><apply id="S3.E7.m1.3.3.1.1.cmml" xref="S3.E7.m1.3.3.1"><eq id="S3.E7.m1.3.3.1.1.1.cmml" xref="S3.E7.m1.3.3.1.1.1"></eq><apply id="S3.E7.m1.3.3.1.1.2.cmml" xref="S3.E7.m1.3.3.1.1.2"><csymbol cd="ambiguous" id="S3.E7.m1.3.3.1.1.2.1.cmml" xref="S3.E7.m1.3.3.1.1.2">subscript</csymbol><ci id="S3.E7.m1.3.3.1.1.2.2.cmml" xref="S3.E7.m1.3.3.1.1.2.2">𝑾</ci><list id="S3.E7.m1.2.2.2.3.cmml" xref="S3.E7.m1.2.2.2.2"><apply id="S3.E7.m1.2.2.2.2.1.cmml" xref="S3.E7.m1.2.2.2.2.1"><csymbol cd="ambiguous" id="S3.E7.m1.2.2.2.2.1.1.cmml" xref="S3.E7.m1.2.2.2.2.1">subscript</csymbol><ci id="S3.E7.m1.2.2.2.2.1.2.cmml" xref="S3.E7.m1.2.2.2.2.1.2">𝑇</ci><ci id="S3.E7.m1.2.2.2.2.1.3.cmml" xref="S3.E7.m1.2.2.2.2.1.3">𝑖</ci></apply><ci id="S3.E7.m1.1.1.1.1a.cmml" xref="S3.E7.m1.1.1.1.1"><mtext mathsize="70%" id="S3.E7.m1.1.1.1.1.cmml" xref="S3.E7.m1.1.1.1.1">extract</mtext></ci></list></apply><apply id="S3.E7.m1.3.3.1.1.3.cmml" xref="S3.E7.m1.3.3.1.1.3"><times id="S3.E7.m1.3.3.1.1.3.1.cmml" xref="S3.E7.m1.3.3.1.1.3.1"></times><apply id="S3.E7.m1.3.3.1.1.3.2.cmml" xref="S3.E7.m1.3.3.1.1.3.2"><csymbol cd="ambiguous" id="S3.E7.m1.3.3.1.1.3.2.1.cmml" xref="S3.E7.m1.3.3.1.1.3.2">subscript</csymbol><ci id="S3.E7.m1.3.3.1.1.3.2.2.cmml" xref="S3.E7.m1.3.3.1.1.3.2.2">𝑼</ci><ci id="S3.E7.m1.3.3.1.1.3.2.3.cmml" xref="S3.E7.m1.3.3.1.1.3.2.3">𝑖</ci></apply><apply id="S3.E7.m1.3.3.1.1.3.3.cmml" xref="S3.E7.m1.3.3.1.1.3.3"><csymbol cd="ambiguous" id="S3.E7.m1.3.3.1.1.3.3.1.cmml" xref="S3.E7.m1.3.3.1.1.3.3">subscript</csymbol><ci id="S3.E7.m1.3.3.1.1.3.3.2.cmml" xref="S3.E7.m1.3.3.1.1.3.3.2">𝚺</ci><ci id="S3.E7.m1.3.3.1.1.3.3.3.cmml" xref="S3.E7.m1.3.3.1.1.3.3.3">𝑖</ci></apply><apply id="S3.E7.m1.3.3.1.1.3.4.cmml" xref="S3.E7.m1.3.3.1.1.3.4"><csymbol cd="ambiguous" id="S3.E7.m1.3.3.1.1.3.4.1.cmml" xref="S3.E7.m1.3.3.1.1.3.4">superscript</csymbol><apply id="S3.E7.m1.3.3.1.1.3.4.2.cmml" xref="S3.E7.m1.3.3.1.1.3.4"><csymbol cd="ambiguous" id="S3.E7.m1.3.3.1.1.3.4.2.1.cmml" xref="S3.E7.m1.3.3.1.1.3.4">subscript</csymbol><ci id="S3.E7.m1.3.3.1.1.3.4.2.2.cmml" xref="S3.E7.m1.3.3.1.1.3.4.2.2">𝑽</ci><ci id="S3.E7.m1.3.3.1.1.3.4.2.3.cmml" xref="S3.E7.m1.3.3.1.1.3.4.2.3">𝑖</ci></apply><csymbol cd="latexml" id="S3.E7.m1.3.3.1.1.3.4.3.cmml" xref="S3.E7.m1.3.3.1.1.3.4.3">top</csymbol></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E7.m1.3c">\bm{W}_{T_{i},\text{extract}}=\bm{U}_{i}\bm{\Sigma}_{i}\bm{V}_{i}^{\top}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(7)</span></td>
</tr></tbody>
</table>
</div>
<div id="S3.SS3.SSS0.Px2.p3" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS0.Px2.p3.4" class="ltx_p">Here, <math id="S3.SS3.SSS0.Px2.p3.1.m1.1" class="ltx_Math" alttext="\bm{U}_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p3.1.m1.1a"><msub id="S3.SS3.SSS0.Px2.p3.1.m1.1.1" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.2" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.2.cmml">𝑼</mi><mi id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p3.1.m1.1b"><apply id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.2">𝑼</ci><ci id="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p3.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p3.1.m1.1c">\bm{U}_{i}</annotation></semantics></math> and <math id="S3.SS3.SSS0.Px2.p3.2.m2.1" class="ltx_Math" alttext="\bm{V}_{i}^{\top}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p3.2.m2.1a"><msubsup id="S3.SS3.SSS0.Px2.p3.2.m2.1.1" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.2" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.2.cmml">𝑽</mi><mi id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.3" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.3.cmml">i</mi><mo id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.3" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1.3.cmml">⊤</mo></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p3.2.m2.1b"><apply id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.cmml" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1">superscript</csymbol><apply id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.2">𝑽</ci><ci id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1.2.3">𝑖</ci></apply><csymbol cd="latexml" id="S3.SS3.SSS0.Px2.p3.2.m2.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p3.2.m2.1.1.3">top</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p3.2.m2.1c">\bm{V}_{i}^{\top}</annotation></semantics></math> are orthogonal matrices containing left and right singular vectors, respectively, while <math id="S3.SS3.SSS0.Px2.p3.3.m3.1" class="ltx_Math" alttext="\bm{\Sigma}_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p3.3.m3.1a"><msub id="S3.SS3.SSS0.Px2.p3.3.m3.1.1" xref="S3.SS3.SSS0.Px2.p3.3.m3.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p3.3.m3.1.1.2" xref="S3.SS3.SSS0.Px2.p3.3.m3.1.1.2.cmml">𝚺</mi><mi id="S3.SS3.SSS0.Px2.p3.3.m3.1.1.3" xref="S3.SS3.SSS0.Px2.p3.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p3.3.m3.1b"><apply id="S3.SS3.SSS0.Px2.p3.3.m3.1.1.cmml" xref="S3.SS3.SSS0.Px2.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p3.3.m3.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p3.3.m3.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p3.3.m3.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p3.3.m3.1.1.2">𝚺</ci><ci id="S3.SS3.SSS0.Px2.p3.3.m3.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p3.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p3.3.m3.1c">\bm{\Sigma}_{i}</annotation></semantics></math> is a diagonal matrix that hosts the singular values in descending order. To capture the first <math id="S3.SS3.SSS0.Px2.p3.4.m4.1" class="ltx_Math" alttext="r" display="inline"><semantics id="S3.SS3.SSS0.Px2.p3.4.m4.1a"><mi id="S3.SS3.SSS0.Px2.p3.4.m4.1.1" xref="S3.SS3.SSS0.Px2.p3.4.m4.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p3.4.m4.1b"><ci id="S3.SS3.SSS0.Px2.p3.4.m4.1.1.cmml" xref="S3.SS3.SSS0.Px2.p3.4.m4.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p3.4.m4.1c">r</annotation></semantics></math> ranks, we then formulate:</p>
</div>
<div id="S3.SS3.SSS0.Px2.p4" class="ltx_para ltx_noindent">
<table id="S3.E8" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E8.m1.3" class="ltx_math_unparsed" alttext="\bm{W}_{T_{i},\text{extract},r}=\bm{U}_{i}[:,:r]\bm{\Sigma}_{i}[:r,:r]\bm{V}_{i}^{\top}[:r,:]." display="block"><semantics id="S3.E8.m1.3a"><mrow id="S3.E8.m1.3b"><msub id="S3.E8.m1.3.4"><mi id="S3.E8.m1.3.4.2">𝑾</mi><mrow id="S3.E8.m1.3.3.3.3"><msub id="S3.E8.m1.3.3.3.3.1"><mi id="S3.E8.m1.3.3.3.3.1.2">T</mi><mi id="S3.E8.m1.3.3.3.3.1.3">i</mi></msub><mo id="S3.E8.m1.3.3.3.3.2">,</mo><mtext id="S3.E8.m1.1.1.1.1">extract</mtext><mo id="S3.E8.m1.3.3.3.3.3">,</mo><mi id="S3.E8.m1.2.2.2.2">r</mi></mrow></msub><mo id="S3.E8.m1.3.5">=</mo><msub id="S3.E8.m1.3.6"><mi id="S3.E8.m1.3.6.2">𝑼</mi><mi id="S3.E8.m1.3.6.3">i</mi></msub><mrow id="S3.E8.m1.3.7"><mo stretchy="false" id="S3.E8.m1.3.7.1">[</mo><mo rspace="0em" id="S3.E8.m1.3.7.2">:</mo><mo id="S3.E8.m1.3.7.3">,</mo><mo rspace="0.278em" id="S3.E8.m1.3.7.4">:</mo><mi id="S3.E8.m1.3.7.5">r</mi><mo stretchy="false" id="S3.E8.m1.3.7.6">]</mo></mrow><msub id="S3.E8.m1.3.8"><mi id="S3.E8.m1.3.8.2">𝚺</mi><mi id="S3.E8.m1.3.8.3">i</mi></msub><mrow id="S3.E8.m1.3.9"><mo stretchy="false" id="S3.E8.m1.3.9.1">[</mo><mo rspace="0.278em" id="S3.E8.m1.3.9.2">:</mo><mi id="S3.E8.m1.3.9.3">r</mi><mo id="S3.E8.m1.3.9.4">,</mo><mo rspace="0.278em" id="S3.E8.m1.3.9.5">:</mo><mi id="S3.E8.m1.3.9.6">r</mi><mo stretchy="false" id="S3.E8.m1.3.9.7">]</mo></mrow><msubsup id="S3.E8.m1.3.10"><mi id="S3.E8.m1.3.10.2.2">𝑽</mi><mi id="S3.E8.m1.3.10.2.3">i</mi><mo id="S3.E8.m1.3.10.3">⊤</mo></msubsup><mrow id="S3.E8.m1.3.11"><mo stretchy="false" id="S3.E8.m1.3.11.1">[</mo><mo rspace="0.278em" id="S3.E8.m1.3.11.2">:</mo><mi id="S3.E8.m1.3.11.3">r</mi><mo id="S3.E8.m1.3.11.4">,</mo><mo rspace="0em" id="S3.E8.m1.3.11.5">:</mo><mo stretchy="false" id="S3.E8.m1.3.11.6">]</mo></mrow><mo lspace="0em" id="S3.E8.m1.3.12">.</mo></mrow><annotation encoding="application/x-tex" id="S3.E8.m1.3c">\bm{W}_{T_{i},\text{extract},r}=\bm{U}_{i}[:,:r]\bm{\Sigma}_{i}[:r,:r]\bm{V}_{i}^{\top}[:r,:].</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(8)</span></td>
</tr></tbody>
</table>
</div>
<div id="S3.SS3.SSS0.Px2.p5" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS0.Px2.p5.8" class="ltx_p">The symbols <math id="S3.SS3.SSS0.Px2.p5.1.m1.1" class="ltx_math_unparsed" alttext="\bm{U}_{i}[:,:r]" display="inline"><semantics id="S3.SS3.SSS0.Px2.p5.1.m1.1a"><mrow id="S3.SS3.SSS0.Px2.p5.1.m1.1b"><msub id="S3.SS3.SSS0.Px2.p5.1.m1.1.1"><mi id="S3.SS3.SSS0.Px2.p5.1.m1.1.1.2">𝑼</mi><mi id="S3.SS3.SSS0.Px2.p5.1.m1.1.1.3">i</mi></msub><mrow id="S3.SS3.SSS0.Px2.p5.1.m1.1.2"><mo stretchy="false" id="S3.SS3.SSS0.Px2.p5.1.m1.1.2.1">[</mo><mo rspace="0em" id="S3.SS3.SSS0.Px2.p5.1.m1.1.2.2">:</mo><mo id="S3.SS3.SSS0.Px2.p5.1.m1.1.2.3">,</mo><mo rspace="0.278em" id="S3.SS3.SSS0.Px2.p5.1.m1.1.2.4">:</mo><mi id="S3.SS3.SSS0.Px2.p5.1.m1.1.2.5">r</mi><mo stretchy="false" id="S3.SS3.SSS0.Px2.p5.1.m1.1.2.6">]</mo></mrow></mrow><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p5.1.m1.1c">\bm{U}_{i}[:,:r]</annotation></semantics></math> and <math id="S3.SS3.SSS0.Px2.p5.2.m2.1" class="ltx_math_unparsed" alttext="\bm{V}_{i}^{\top}[:r,:]" display="inline"><semantics id="S3.SS3.SSS0.Px2.p5.2.m2.1a"><mrow id="S3.SS3.SSS0.Px2.p5.2.m2.1b"><msubsup id="S3.SS3.SSS0.Px2.p5.2.m2.1.1"><mi id="S3.SS3.SSS0.Px2.p5.2.m2.1.1.2.2">𝑽</mi><mi id="S3.SS3.SSS0.Px2.p5.2.m2.1.1.2.3">i</mi><mo id="S3.SS3.SSS0.Px2.p5.2.m2.1.1.3">⊤</mo></msubsup><mrow id="S3.SS3.SSS0.Px2.p5.2.m2.1.2"><mo stretchy="false" id="S3.SS3.SSS0.Px2.p5.2.m2.1.2.1">[</mo><mo rspace="0.278em" id="S3.SS3.SSS0.Px2.p5.2.m2.1.2.2">:</mo><mi id="S3.SS3.SSS0.Px2.p5.2.m2.1.2.3">r</mi><mo id="S3.SS3.SSS0.Px2.p5.2.m2.1.2.4">,</mo><mo rspace="0em" id="S3.SS3.SSS0.Px2.p5.2.m2.1.2.5">:</mo><mo stretchy="false" id="S3.SS3.SSS0.Px2.p5.2.m2.1.2.6">]</mo></mrow></mrow><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p5.2.m2.1c">\bm{V}_{i}^{\top}[:r,:]</annotation></semantics></math> represent the initial <math id="S3.SS3.SSS0.Px2.p5.3.m3.1" class="ltx_Math" alttext="r" display="inline"><semantics id="S3.SS3.SSS0.Px2.p5.3.m3.1a"><mi id="S3.SS3.SSS0.Px2.p5.3.m3.1.1" xref="S3.SS3.SSS0.Px2.p5.3.m3.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p5.3.m3.1b"><ci id="S3.SS3.SSS0.Px2.p5.3.m3.1.1.cmml" xref="S3.SS3.SSS0.Px2.p5.3.m3.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p5.3.m3.1c">r</annotation></semantics></math> columns of <math id="S3.SS3.SSS0.Px2.p5.4.m4.1" class="ltx_Math" alttext="\bm{U}_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p5.4.m4.1a"><msub id="S3.SS3.SSS0.Px2.p5.4.m4.1.1" xref="S3.SS3.SSS0.Px2.p5.4.m4.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p5.4.m4.1.1.2" xref="S3.SS3.SSS0.Px2.p5.4.m4.1.1.2.cmml">𝑼</mi><mi id="S3.SS3.SSS0.Px2.p5.4.m4.1.1.3" xref="S3.SS3.SSS0.Px2.p5.4.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p5.4.m4.1b"><apply id="S3.SS3.SSS0.Px2.p5.4.m4.1.1.cmml" xref="S3.SS3.SSS0.Px2.p5.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p5.4.m4.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p5.4.m4.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p5.4.m4.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p5.4.m4.1.1.2">𝑼</ci><ci id="S3.SS3.SSS0.Px2.p5.4.m4.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p5.4.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p5.4.m4.1c">\bm{U}_{i}</annotation></semantics></math> and <math id="S3.SS3.SSS0.Px2.p5.5.m5.1" class="ltx_Math" alttext="\bm{V}_{i}^{\top}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p5.5.m5.1a"><msubsup id="S3.SS3.SSS0.Px2.p5.5.m5.1.1" xref="S3.SS3.SSS0.Px2.p5.5.m5.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p5.5.m5.1.1.2.2" xref="S3.SS3.SSS0.Px2.p5.5.m5.1.1.2.2.cmml">𝑽</mi><mi id="S3.SS3.SSS0.Px2.p5.5.m5.1.1.2.3" xref="S3.SS3.SSS0.Px2.p5.5.m5.1.1.2.3.cmml">i</mi><mo id="S3.SS3.SSS0.Px2.p5.5.m5.1.1.3" xref="S3.SS3.SSS0.Px2.p5.5.m5.1.1.3.cmml">⊤</mo></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p5.5.m5.1b"><apply id="S3.SS3.SSS0.Px2.p5.5.m5.1.1.cmml" xref="S3.SS3.SSS0.Px2.p5.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p5.5.m5.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p5.5.m5.1.1">superscript</csymbol><apply id="S3.SS3.SSS0.Px2.p5.5.m5.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p5.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p5.5.m5.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px2.p5.5.m5.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p5.5.m5.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px2.p5.5.m5.1.1.2.2">𝑽</ci><ci id="S3.SS3.SSS0.Px2.p5.5.m5.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px2.p5.5.m5.1.1.2.3">𝑖</ci></apply><csymbol cd="latexml" id="S3.SS3.SSS0.Px2.p5.5.m5.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p5.5.m5.1.1.3">top</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p5.5.m5.1c">\bm{V}_{i}^{\top}</annotation></semantics></math>, respectively, while <math id="S3.SS3.SSS0.Px2.p5.6.m6.1" class="ltx_math_unparsed" alttext="\bm{\Sigma}_{i}[:r,:r]" display="inline"><semantics id="S3.SS3.SSS0.Px2.p5.6.m6.1a"><mrow id="S3.SS3.SSS0.Px2.p5.6.m6.1b"><msub id="S3.SS3.SSS0.Px2.p5.6.m6.1.1"><mi id="S3.SS3.SSS0.Px2.p5.6.m6.1.1.2">𝚺</mi><mi id="S3.SS3.SSS0.Px2.p5.6.m6.1.1.3">i</mi></msub><mrow id="S3.SS3.SSS0.Px2.p5.6.m6.1.2"><mo stretchy="false" id="S3.SS3.SSS0.Px2.p5.6.m6.1.2.1">[</mo><mo rspace="0.278em" id="S3.SS3.SSS0.Px2.p5.6.m6.1.2.2">:</mo><mi id="S3.SS3.SSS0.Px2.p5.6.m6.1.2.3">r</mi><mo id="S3.SS3.SSS0.Px2.p5.6.m6.1.2.4">,</mo><mo rspace="0.278em" id="S3.SS3.SSS0.Px2.p5.6.m6.1.2.5">:</mo><mi id="S3.SS3.SSS0.Px2.p5.6.m6.1.2.6">r</mi><mo stretchy="false" id="S3.SS3.SSS0.Px2.p5.6.m6.1.2.7">]</mo></mrow></mrow><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p5.6.m6.1c">\bm{\Sigma}_{i}[:r,:r]</annotation></semantics></math> captures the top <math id="S3.SS3.SSS0.Px2.p5.7.m7.1" class="ltx_Math" alttext="r" display="inline"><semantics id="S3.SS3.SSS0.Px2.p5.7.m7.1a"><mi id="S3.SS3.SSS0.Px2.p5.7.m7.1.1" xref="S3.SS3.SSS0.Px2.p5.7.m7.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p5.7.m7.1b"><ci id="S3.SS3.SSS0.Px2.p5.7.m7.1.1.cmml" xref="S3.SS3.SSS0.Px2.p5.7.m7.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p5.7.m7.1c">r</annotation></semantics></math> singular values. Subsequently, for each corresponding matrix <math id="S3.SS3.SSS0.Px2.p5.8.m8.1" class="ltx_Math" alttext="\bm{W}_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p5.8.m8.1a"><msub id="S3.SS3.SSS0.Px2.p5.8.m8.1.1" xref="S3.SS3.SSS0.Px2.p5.8.m8.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p5.8.m8.1.1.2" xref="S3.SS3.SSS0.Px2.p5.8.m8.1.1.2.cmml">𝑾</mi><mi id="S3.SS3.SSS0.Px2.p5.8.m8.1.1.3" xref="S3.SS3.SSS0.Px2.p5.8.m8.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p5.8.m8.1b"><apply id="S3.SS3.SSS0.Px2.p5.8.m8.1.1.cmml" xref="S3.SS3.SSS0.Px2.p5.8.m8.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p5.8.m8.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p5.8.m8.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p5.8.m8.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p5.8.m8.1.1.2">𝑾</ci><ci id="S3.SS3.SSS0.Px2.p5.8.m8.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p5.8.m8.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p5.8.m8.1c">\bm{W}_{i}</annotation></semantics></math> in the student model, its update can be given as in two ways:</p>
</div>
<div id="S3.SS3.SSS0.Px2.p6" class="ltx_para ltx_noindent">
<table id="S5.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E9"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E9.m1.1" class="ltx_Math" alttext="\displaystyle\bm{W}_{i}^{*}" display="inline"><semantics id="S3.E9.m1.1a"><msubsup id="S3.E9.m1.1.1" xref="S3.E9.m1.1.1.cmml"><mi id="S3.E9.m1.1.1.2.2" xref="S3.E9.m1.1.1.2.2.cmml">𝑾</mi><mi id="S3.E9.m1.1.1.2.3" xref="S3.E9.m1.1.1.2.3.cmml">i</mi><mo id="S3.E9.m1.1.1.3" xref="S3.E9.m1.1.1.3.cmml">∗</mo></msubsup><annotation-xml encoding="MathML-Content" id="S3.E9.m1.1b"><apply id="S3.E9.m1.1.1.cmml" xref="S3.E9.m1.1.1"><csymbol cd="ambiguous" id="S3.E9.m1.1.1.1.cmml" xref="S3.E9.m1.1.1">superscript</csymbol><apply id="S3.E9.m1.1.1.2.cmml" xref="S3.E9.m1.1.1"><csymbol cd="ambiguous" id="S3.E9.m1.1.1.2.1.cmml" xref="S3.E9.m1.1.1">subscript</csymbol><ci id="S3.E9.m1.1.1.2.2.cmml" xref="S3.E9.m1.1.1.2.2">𝑾</ci><ci id="S3.E9.m1.1.1.2.3.cmml" xref="S3.E9.m1.1.1.2.3">𝑖</ci></apply><times id="S3.E9.m1.1.1.3.cmml" xref="S3.E9.m1.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E9.m1.1c">\displaystyle\bm{W}_{i}^{*}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E9.m2.1" class="ltx_Math" alttext="\displaystyle=\bm{W}_{i}+\bm{B}_{i}\bm{A}_{i}," display="inline"><semantics id="S3.E9.m2.1a"><mrow id="S3.E9.m2.1.1.1" xref="S3.E9.m2.1.1.1.1.cmml"><mrow id="S3.E9.m2.1.1.1.1" xref="S3.E9.m2.1.1.1.1.cmml"><mi id="S3.E9.m2.1.1.1.1.2" xref="S3.E9.m2.1.1.1.1.2.cmml"></mi><mo id="S3.E9.m2.1.1.1.1.1" xref="S3.E9.m2.1.1.1.1.1.cmml">=</mo><mrow id="S3.E9.m2.1.1.1.1.3" xref="S3.E9.m2.1.1.1.1.3.cmml"><msub id="S3.E9.m2.1.1.1.1.3.2" xref="S3.E9.m2.1.1.1.1.3.2.cmml"><mi id="S3.E9.m2.1.1.1.1.3.2.2" xref="S3.E9.m2.1.1.1.1.3.2.2.cmml">𝑾</mi><mi id="S3.E9.m2.1.1.1.1.3.2.3" xref="S3.E9.m2.1.1.1.1.3.2.3.cmml">i</mi></msub><mo id="S3.E9.m2.1.1.1.1.3.1" xref="S3.E9.m2.1.1.1.1.3.1.cmml">+</mo><mrow id="S3.E9.m2.1.1.1.1.3.3" xref="S3.E9.m2.1.1.1.1.3.3.cmml"><msub id="S3.E9.m2.1.1.1.1.3.3.2" xref="S3.E9.m2.1.1.1.1.3.3.2.cmml"><mi id="S3.E9.m2.1.1.1.1.3.3.2.2" xref="S3.E9.m2.1.1.1.1.3.3.2.2.cmml">𝑩</mi><mi id="S3.E9.m2.1.1.1.1.3.3.2.3" xref="S3.E9.m2.1.1.1.1.3.3.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.E9.m2.1.1.1.1.3.3.1" xref="S3.E9.m2.1.1.1.1.3.3.1.cmml">​</mo><msub id="S3.E9.m2.1.1.1.1.3.3.3" xref="S3.E9.m2.1.1.1.1.3.3.3.cmml"><mi id="S3.E9.m2.1.1.1.1.3.3.3.2" xref="S3.E9.m2.1.1.1.1.3.3.3.2.cmml">𝑨</mi><mi id="S3.E9.m2.1.1.1.1.3.3.3.3" xref="S3.E9.m2.1.1.1.1.3.3.3.3.cmml">i</mi></msub></mrow></mrow></mrow><mo id="S3.E9.m2.1.1.1.2" xref="S3.E9.m2.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E9.m2.1b"><apply id="S3.E9.m2.1.1.1.1.cmml" xref="S3.E9.m2.1.1.1"><eq id="S3.E9.m2.1.1.1.1.1.cmml" xref="S3.E9.m2.1.1.1.1.1"></eq><csymbol cd="latexml" id="S3.E9.m2.1.1.1.1.2.cmml" xref="S3.E9.m2.1.1.1.1.2">absent</csymbol><apply id="S3.E9.m2.1.1.1.1.3.cmml" xref="S3.E9.m2.1.1.1.1.3"><plus id="S3.E9.m2.1.1.1.1.3.1.cmml" xref="S3.E9.m2.1.1.1.1.3.1"></plus><apply id="S3.E9.m2.1.1.1.1.3.2.cmml" xref="S3.E9.m2.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S3.E9.m2.1.1.1.1.3.2.1.cmml" xref="S3.E9.m2.1.1.1.1.3.2">subscript</csymbol><ci id="S3.E9.m2.1.1.1.1.3.2.2.cmml" xref="S3.E9.m2.1.1.1.1.3.2.2">𝑾</ci><ci id="S3.E9.m2.1.1.1.1.3.2.3.cmml" xref="S3.E9.m2.1.1.1.1.3.2.3">𝑖</ci></apply><apply id="S3.E9.m2.1.1.1.1.3.3.cmml" xref="S3.E9.m2.1.1.1.1.3.3"><times id="S3.E9.m2.1.1.1.1.3.3.1.cmml" xref="S3.E9.m2.1.1.1.1.3.3.1"></times><apply id="S3.E9.m2.1.1.1.1.3.3.2.cmml" xref="S3.E9.m2.1.1.1.1.3.3.2"><csymbol cd="ambiguous" id="S3.E9.m2.1.1.1.1.3.3.2.1.cmml" xref="S3.E9.m2.1.1.1.1.3.3.2">subscript</csymbol><ci id="S3.E9.m2.1.1.1.1.3.3.2.2.cmml" xref="S3.E9.m2.1.1.1.1.3.3.2.2">𝑩</ci><ci id="S3.E9.m2.1.1.1.1.3.3.2.3.cmml" xref="S3.E9.m2.1.1.1.1.3.3.2.3">𝑖</ci></apply><apply id="S3.E9.m2.1.1.1.1.3.3.3.cmml" xref="S3.E9.m2.1.1.1.1.3.3.3"><csymbol cd="ambiguous" id="S3.E9.m2.1.1.1.1.3.3.3.1.cmml" xref="S3.E9.m2.1.1.1.1.3.3.3">subscript</csymbol><ci id="S3.E9.m2.1.1.1.1.3.3.3.2.cmml" xref="S3.E9.m2.1.1.1.1.3.3.3.2">𝑨</ci><ci id="S3.E9.m2.1.1.1.1.3.3.3.3.cmml" xref="S3.E9.m2.1.1.1.1.3.3.3.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E9.m2.1c">\displaystyle=\bm{W}_{i}+\bm{B}_{i}\bm{A}_{i},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(9)</span></td>
</tr></tbody>
<tbody id="S3.E10"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E10.m1.1" class="ltx_Math" alttext="\displaystyle\bm{W}_{i}^{*}" display="inline"><semantics id="S3.E10.m1.1a"><msubsup id="S3.E10.m1.1.1" xref="S3.E10.m1.1.1.cmml"><mi id="S3.E10.m1.1.1.2.2" xref="S3.E10.m1.1.1.2.2.cmml">𝑾</mi><mi id="S3.E10.m1.1.1.2.3" xref="S3.E10.m1.1.1.2.3.cmml">i</mi><mo id="S3.E10.m1.1.1.3" xref="S3.E10.m1.1.1.3.cmml">∗</mo></msubsup><annotation-xml encoding="MathML-Content" id="S3.E10.m1.1b"><apply id="S3.E10.m1.1.1.cmml" xref="S3.E10.m1.1.1"><csymbol cd="ambiguous" id="S3.E10.m1.1.1.1.cmml" xref="S3.E10.m1.1.1">superscript</csymbol><apply id="S3.E10.m1.1.1.2.cmml" xref="S3.E10.m1.1.1"><csymbol cd="ambiguous" id="S3.E10.m1.1.1.2.1.cmml" xref="S3.E10.m1.1.1">subscript</csymbol><ci id="S3.E10.m1.1.1.2.2.cmml" xref="S3.E10.m1.1.1.2.2">𝑾</ci><ci id="S3.E10.m1.1.1.2.3.cmml" xref="S3.E10.m1.1.1.2.3">𝑖</ci></apply><times id="S3.E10.m1.1.1.3.cmml" xref="S3.E10.m1.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E10.m1.1c">\displaystyle\bm{W}_{i}^{*}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E10.m2.4" class="ltx_Math" alttext="\displaystyle=\bm{W}_{i}-\bm{W}_{T_{i},\text{extract},r}+\bm{B}_{i}\bm{A}_{i}," display="inline"><semantics id="S3.E10.m2.4a"><mrow id="S3.E10.m2.4.4.1" xref="S3.E10.m2.4.4.1.1.cmml"><mrow id="S3.E10.m2.4.4.1.1" xref="S3.E10.m2.4.4.1.1.cmml"><mi id="S3.E10.m2.4.4.1.1.2" xref="S3.E10.m2.4.4.1.1.2.cmml"></mi><mo id="S3.E10.m2.4.4.1.1.1" xref="S3.E10.m2.4.4.1.1.1.cmml">=</mo><mrow id="S3.E10.m2.4.4.1.1.3" xref="S3.E10.m2.4.4.1.1.3.cmml"><mrow id="S3.E10.m2.4.4.1.1.3.2" xref="S3.E10.m2.4.4.1.1.3.2.cmml"><msub id="S3.E10.m2.4.4.1.1.3.2.2" xref="S3.E10.m2.4.4.1.1.3.2.2.cmml"><mi id="S3.E10.m2.4.4.1.1.3.2.2.2" xref="S3.E10.m2.4.4.1.1.3.2.2.2.cmml">𝑾</mi><mi id="S3.E10.m2.4.4.1.1.3.2.2.3" xref="S3.E10.m2.4.4.1.1.3.2.2.3.cmml">i</mi></msub><mo id="S3.E10.m2.4.4.1.1.3.2.1" xref="S3.E10.m2.4.4.1.1.3.2.1.cmml">−</mo><msub id="S3.E10.m2.4.4.1.1.3.2.3" xref="S3.E10.m2.4.4.1.1.3.2.3.cmml"><mi id="S3.E10.m2.4.4.1.1.3.2.3.2" xref="S3.E10.m2.4.4.1.1.3.2.3.2.cmml">𝑾</mi><mrow id="S3.E10.m2.3.3.3.3" xref="S3.E10.m2.3.3.3.4.cmml"><msub id="S3.E10.m2.3.3.3.3.1" xref="S3.E10.m2.3.3.3.3.1.cmml"><mi id="S3.E10.m2.3.3.3.3.1.2" xref="S3.E10.m2.3.3.3.3.1.2.cmml">T</mi><mi id="S3.E10.m2.3.3.3.3.1.3" xref="S3.E10.m2.3.3.3.3.1.3.cmml">i</mi></msub><mo id="S3.E10.m2.3.3.3.3.2" xref="S3.E10.m2.3.3.3.4.cmml">,</mo><mtext id="S3.E10.m2.1.1.1.1" xref="S3.E10.m2.1.1.1.1a.cmml">extract</mtext><mo id="S3.E10.m2.3.3.3.3.3" xref="S3.E10.m2.3.3.3.4.cmml">,</mo><mi id="S3.E10.m2.2.2.2.2" xref="S3.E10.m2.2.2.2.2.cmml">r</mi></mrow></msub></mrow><mo id="S3.E10.m2.4.4.1.1.3.1" xref="S3.E10.m2.4.4.1.1.3.1.cmml">+</mo><mrow id="S3.E10.m2.4.4.1.1.3.3" xref="S3.E10.m2.4.4.1.1.3.3.cmml"><msub id="S3.E10.m2.4.4.1.1.3.3.2" xref="S3.E10.m2.4.4.1.1.3.3.2.cmml"><mi id="S3.E10.m2.4.4.1.1.3.3.2.2" xref="S3.E10.m2.4.4.1.1.3.3.2.2.cmml">𝑩</mi><mi id="S3.E10.m2.4.4.1.1.3.3.2.3" xref="S3.E10.m2.4.4.1.1.3.3.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.E10.m2.4.4.1.1.3.3.1" xref="S3.E10.m2.4.4.1.1.3.3.1.cmml">​</mo><msub id="S3.E10.m2.4.4.1.1.3.3.3" xref="S3.E10.m2.4.4.1.1.3.3.3.cmml"><mi id="S3.E10.m2.4.4.1.1.3.3.3.2" xref="S3.E10.m2.4.4.1.1.3.3.3.2.cmml">𝑨</mi><mi id="S3.E10.m2.4.4.1.1.3.3.3.3" xref="S3.E10.m2.4.4.1.1.3.3.3.3.cmml">i</mi></msub></mrow></mrow></mrow><mo id="S3.E10.m2.4.4.1.2" xref="S3.E10.m2.4.4.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E10.m2.4b"><apply id="S3.E10.m2.4.4.1.1.cmml" xref="S3.E10.m2.4.4.1"><eq id="S3.E10.m2.4.4.1.1.1.cmml" xref="S3.E10.m2.4.4.1.1.1"></eq><csymbol cd="latexml" id="S3.E10.m2.4.4.1.1.2.cmml" xref="S3.E10.m2.4.4.1.1.2">absent</csymbol><apply id="S3.E10.m2.4.4.1.1.3.cmml" xref="S3.E10.m2.4.4.1.1.3"><plus id="S3.E10.m2.4.4.1.1.3.1.cmml" xref="S3.E10.m2.4.4.1.1.3.1"></plus><apply id="S3.E10.m2.4.4.1.1.3.2.cmml" xref="S3.E10.m2.4.4.1.1.3.2"><minus id="S3.E10.m2.4.4.1.1.3.2.1.cmml" xref="S3.E10.m2.4.4.1.1.3.2.1"></minus><apply id="S3.E10.m2.4.4.1.1.3.2.2.cmml" xref="S3.E10.m2.4.4.1.1.3.2.2"><csymbol cd="ambiguous" id="S3.E10.m2.4.4.1.1.3.2.2.1.cmml" xref="S3.E10.m2.4.4.1.1.3.2.2">subscript</csymbol><ci id="S3.E10.m2.4.4.1.1.3.2.2.2.cmml" xref="S3.E10.m2.4.4.1.1.3.2.2.2">𝑾</ci><ci id="S3.E10.m2.4.4.1.1.3.2.2.3.cmml" xref="S3.E10.m2.4.4.1.1.3.2.2.3">𝑖</ci></apply><apply id="S3.E10.m2.4.4.1.1.3.2.3.cmml" xref="S3.E10.m2.4.4.1.1.3.2.3"><csymbol cd="ambiguous" id="S3.E10.m2.4.4.1.1.3.2.3.1.cmml" xref="S3.E10.m2.4.4.1.1.3.2.3">subscript</csymbol><ci id="S3.E10.m2.4.4.1.1.3.2.3.2.cmml" xref="S3.E10.m2.4.4.1.1.3.2.3.2">𝑾</ci><list id="S3.E10.m2.3.3.3.4.cmml" xref="S3.E10.m2.3.3.3.3"><apply id="S3.E10.m2.3.3.3.3.1.cmml" xref="S3.E10.m2.3.3.3.3.1"><csymbol cd="ambiguous" id="S3.E10.m2.3.3.3.3.1.1.cmml" xref="S3.E10.m2.3.3.3.3.1">subscript</csymbol><ci id="S3.E10.m2.3.3.3.3.1.2.cmml" xref="S3.E10.m2.3.3.3.3.1.2">𝑇</ci><ci id="S3.E10.m2.3.3.3.3.1.3.cmml" xref="S3.E10.m2.3.3.3.3.1.3">𝑖</ci></apply><ci id="S3.E10.m2.1.1.1.1a.cmml" xref="S3.E10.m2.1.1.1.1"><mtext mathsize="70%" id="S3.E10.m2.1.1.1.1.cmml" xref="S3.E10.m2.1.1.1.1">extract</mtext></ci><ci id="S3.E10.m2.2.2.2.2.cmml" xref="S3.E10.m2.2.2.2.2">𝑟</ci></list></apply></apply><apply id="S3.E10.m2.4.4.1.1.3.3.cmml" xref="S3.E10.m2.4.4.1.1.3.3"><times id="S3.E10.m2.4.4.1.1.3.3.1.cmml" xref="S3.E10.m2.4.4.1.1.3.3.1"></times><apply id="S3.E10.m2.4.4.1.1.3.3.2.cmml" xref="S3.E10.m2.4.4.1.1.3.3.2"><csymbol cd="ambiguous" id="S3.E10.m2.4.4.1.1.3.3.2.1.cmml" xref="S3.E10.m2.4.4.1.1.3.3.2">subscript</csymbol><ci id="S3.E10.m2.4.4.1.1.3.3.2.2.cmml" xref="S3.E10.m2.4.4.1.1.3.3.2.2">𝑩</ci><ci id="S3.E10.m2.4.4.1.1.3.3.2.3.cmml" xref="S3.E10.m2.4.4.1.1.3.3.2.3">𝑖</ci></apply><apply id="S3.E10.m2.4.4.1.1.3.3.3.cmml" xref="S3.E10.m2.4.4.1.1.3.3.3"><csymbol cd="ambiguous" id="S3.E10.m2.4.4.1.1.3.3.3.1.cmml" xref="S3.E10.m2.4.4.1.1.3.3.3">subscript</csymbol><ci id="S3.E10.m2.4.4.1.1.3.3.3.2.cmml" xref="S3.E10.m2.4.4.1.1.3.3.3.2">𝑨</ci><ci id="S3.E10.m2.4.4.1.1.3.3.3.3.cmml" xref="S3.E10.m2.4.4.1.1.3.3.3.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E10.m2.4c">\displaystyle=\bm{W}_{i}-\bm{W}_{T_{i},\text{extract},r}+\bm{B}_{i}\bm{A}_{i},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(10)</span></td>
</tr></tbody>
</table>
</div>
<div id="S3.SS3.SSS0.Px2.p7" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS0.Px2.p7.8" class="ltx_p">where <math id="S3.SS3.SSS0.Px2.p7.1.m1.1" class="ltx_Math" alttext="\bm{B}_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p7.1.m1.1a"><msub id="S3.SS3.SSS0.Px2.p7.1.m1.1.1" xref="S3.SS3.SSS0.Px2.p7.1.m1.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p7.1.m1.1.1.2" xref="S3.SS3.SSS0.Px2.p7.1.m1.1.1.2.cmml">𝑩</mi><mi id="S3.SS3.SSS0.Px2.p7.1.m1.1.1.3" xref="S3.SS3.SSS0.Px2.p7.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p7.1.m1.1b"><apply id="S3.SS3.SSS0.Px2.p7.1.m1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p7.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p7.1.m1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p7.1.m1.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p7.1.m1.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p7.1.m1.1.1.2">𝑩</ci><ci id="S3.SS3.SSS0.Px2.p7.1.m1.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p7.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p7.1.m1.1c">\bm{B}_{i}</annotation></semantics></math> is initialized using <math id="S3.SS3.SSS0.Px2.p7.2.m2.1" class="ltx_math_unparsed" alttext="\bm{U}_{i}[:,:r]\bm{\Sigma}_{i}[:r,:r]" display="inline"><semantics id="S3.SS3.SSS0.Px2.p7.2.m2.1a"><mrow id="S3.SS3.SSS0.Px2.p7.2.m2.1b"><msub id="S3.SS3.SSS0.Px2.p7.2.m2.1.1"><mi id="S3.SS3.SSS0.Px2.p7.2.m2.1.1.2">𝑼</mi><mi id="S3.SS3.SSS0.Px2.p7.2.m2.1.1.3">i</mi></msub><mrow id="S3.SS3.SSS0.Px2.p7.2.m2.1.2"><mo stretchy="false" id="S3.SS3.SSS0.Px2.p7.2.m2.1.2.1">[</mo><mo rspace="0em" id="S3.SS3.SSS0.Px2.p7.2.m2.1.2.2">:</mo><mo id="S3.SS3.SSS0.Px2.p7.2.m2.1.2.3">,</mo><mo rspace="0.278em" id="S3.SS3.SSS0.Px2.p7.2.m2.1.2.4">:</mo><mi id="S3.SS3.SSS0.Px2.p7.2.m2.1.2.5">r</mi><mo stretchy="false" id="S3.SS3.SSS0.Px2.p7.2.m2.1.2.6">]</mo></mrow><msub id="S3.SS3.SSS0.Px2.p7.2.m2.1.3"><mi id="S3.SS3.SSS0.Px2.p7.2.m2.1.3.2">𝚺</mi><mi id="S3.SS3.SSS0.Px2.p7.2.m2.1.3.3">i</mi></msub><mrow id="S3.SS3.SSS0.Px2.p7.2.m2.1.4"><mo stretchy="false" id="S3.SS3.SSS0.Px2.p7.2.m2.1.4.1">[</mo><mo rspace="0.278em" id="S3.SS3.SSS0.Px2.p7.2.m2.1.4.2">:</mo><mi id="S3.SS3.SSS0.Px2.p7.2.m2.1.4.3">r</mi><mo id="S3.SS3.SSS0.Px2.p7.2.m2.1.4.4">,</mo><mo rspace="0.278em" id="S3.SS3.SSS0.Px2.p7.2.m2.1.4.5">:</mo><mi id="S3.SS3.SSS0.Px2.p7.2.m2.1.4.6">r</mi><mo stretchy="false" id="S3.SS3.SSS0.Px2.p7.2.m2.1.4.7">]</mo></mrow></mrow><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p7.2.m2.1c">\bm{U}_{i}[:,:r]\bm{\Sigma}_{i}[:r,:r]</annotation></semantics></math>, and <math id="S3.SS3.SSS0.Px2.p7.3.m3.1" class="ltx_Math" alttext="\bm{A}_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p7.3.m3.1a"><msub id="S3.SS3.SSS0.Px2.p7.3.m3.1.1" xref="S3.SS3.SSS0.Px2.p7.3.m3.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p7.3.m3.1.1.2" xref="S3.SS3.SSS0.Px2.p7.3.m3.1.1.2.cmml">𝑨</mi><mi id="S3.SS3.SSS0.Px2.p7.3.m3.1.1.3" xref="S3.SS3.SSS0.Px2.p7.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p7.3.m3.1b"><apply id="S3.SS3.SSS0.Px2.p7.3.m3.1.1.cmml" xref="S3.SS3.SSS0.Px2.p7.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p7.3.m3.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p7.3.m3.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p7.3.m3.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p7.3.m3.1.1.2">𝑨</ci><ci id="S3.SS3.SSS0.Px2.p7.3.m3.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p7.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p7.3.m3.1c">\bm{A}_{i}</annotation></semantics></math> is initialized with <math id="S3.SS3.SSS0.Px2.p7.4.m4.1" class="ltx_math_unparsed" alttext="\bm{V}_{i}^{\top}[:r,:]" display="inline"><semantics id="S3.SS3.SSS0.Px2.p7.4.m4.1a"><mrow id="S3.SS3.SSS0.Px2.p7.4.m4.1b"><msubsup id="S3.SS3.SSS0.Px2.p7.4.m4.1.1"><mi id="S3.SS3.SSS0.Px2.p7.4.m4.1.1.2.2">𝑽</mi><mi id="S3.SS3.SSS0.Px2.p7.4.m4.1.1.2.3">i</mi><mo id="S3.SS3.SSS0.Px2.p7.4.m4.1.1.3">⊤</mo></msubsup><mrow id="S3.SS3.SSS0.Px2.p7.4.m4.1.2"><mo stretchy="false" id="S3.SS3.SSS0.Px2.p7.4.m4.1.2.1">[</mo><mo rspace="0.278em" id="S3.SS3.SSS0.Px2.p7.4.m4.1.2.2">:</mo><mi id="S3.SS3.SSS0.Px2.p7.4.m4.1.2.3">r</mi><mo id="S3.SS3.SSS0.Px2.p7.4.m4.1.2.4">,</mo><mo rspace="0em" id="S3.SS3.SSS0.Px2.p7.4.m4.1.2.5">:</mo><mo stretchy="false" id="S3.SS3.SSS0.Px2.p7.4.m4.1.2.6">]</mo></mrow></mrow><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p7.4.m4.1c">\bm{V}_{i}^{\top}[:r,:]</annotation></semantics></math>. During training, both <math id="S3.SS3.SSS0.Px2.p7.5.m5.1" class="ltx_Math" alttext="\bm{W}_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p7.5.m5.1a"><msub id="S3.SS3.SSS0.Px2.p7.5.m5.1.1" xref="S3.SS3.SSS0.Px2.p7.5.m5.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p7.5.m5.1.1.2" xref="S3.SS3.SSS0.Px2.p7.5.m5.1.1.2.cmml">𝑾</mi><mi id="S3.SS3.SSS0.Px2.p7.5.m5.1.1.3" xref="S3.SS3.SSS0.Px2.p7.5.m5.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p7.5.m5.1b"><apply id="S3.SS3.SSS0.Px2.p7.5.m5.1.1.cmml" xref="S3.SS3.SSS0.Px2.p7.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p7.5.m5.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p7.5.m5.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p7.5.m5.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p7.5.m5.1.1.2">𝑾</ci><ci id="S3.SS3.SSS0.Px2.p7.5.m5.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p7.5.m5.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p7.5.m5.1c">\bm{W}_{i}</annotation></semantics></math> and <math id="S3.SS3.SSS0.Px2.p7.6.m6.3" class="ltx_Math" alttext="\bm{W}_{T_{i},\text{extract},r}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p7.6.m6.3a"><msub id="S3.SS3.SSS0.Px2.p7.6.m6.3.4" xref="S3.SS3.SSS0.Px2.p7.6.m6.3.4.cmml"><mi id="S3.SS3.SSS0.Px2.p7.6.m6.3.4.2" xref="S3.SS3.SSS0.Px2.p7.6.m6.3.4.2.cmml">𝑾</mi><mrow id="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.3" xref="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.4.cmml"><msub id="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.3.1" xref="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.3.1.cmml"><mi id="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.3.1.2" xref="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.3.1.2.cmml">T</mi><mi id="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.3.1.3" xref="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.3.1.3.cmml">i</mi></msub><mo id="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.3.2" xref="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.4.cmml">,</mo><mtext id="S3.SS3.SSS0.Px2.p7.6.m6.1.1.1.1" xref="S3.SS3.SSS0.Px2.p7.6.m6.1.1.1.1a.cmml">extract</mtext><mo id="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.3.3" xref="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.4.cmml">,</mo><mi id="S3.SS3.SSS0.Px2.p7.6.m6.2.2.2.2" xref="S3.SS3.SSS0.Px2.p7.6.m6.2.2.2.2.cmml">r</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p7.6.m6.3b"><apply id="S3.SS3.SSS0.Px2.p7.6.m6.3.4.cmml" xref="S3.SS3.SSS0.Px2.p7.6.m6.3.4"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p7.6.m6.3.4.1.cmml" xref="S3.SS3.SSS0.Px2.p7.6.m6.3.4">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p7.6.m6.3.4.2.cmml" xref="S3.SS3.SSS0.Px2.p7.6.m6.3.4.2">𝑾</ci><list id="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.4.cmml" xref="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.3"><apply id="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.3.1.cmml" xref="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.3.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.3.1.1.cmml" xref="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.3.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.3.1.2.cmml" xref="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.3.1.2">𝑇</ci><ci id="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.3.1.3.cmml" xref="S3.SS3.SSS0.Px2.p7.6.m6.3.3.3.3.1.3">𝑖</ci></apply><ci id="S3.SS3.SSS0.Px2.p7.6.m6.1.1.1.1a.cmml" xref="S3.SS3.SSS0.Px2.p7.6.m6.1.1.1.1"><mtext mathsize="70%" id="S3.SS3.SSS0.Px2.p7.6.m6.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p7.6.m6.1.1.1.1">extract</mtext></ci><ci id="S3.SS3.SSS0.Px2.p7.6.m6.2.2.2.2.cmml" xref="S3.SS3.SSS0.Px2.p7.6.m6.2.2.2.2">𝑟</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p7.6.m6.3c">\bm{W}_{T_{i},\text{extract},r}</annotation></semantics></math> remain constant, with only the parameters in <math id="S3.SS3.SSS0.Px2.p7.7.m7.1" class="ltx_Math" alttext="\bm{B}_{i}\bm{A}_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p7.7.m7.1a"><mrow id="S3.SS3.SSS0.Px2.p7.7.m7.1.1" xref="S3.SS3.SSS0.Px2.p7.7.m7.1.1.cmml"><msub id="S3.SS3.SSS0.Px2.p7.7.m7.1.1.2" xref="S3.SS3.SSS0.Px2.p7.7.m7.1.1.2.cmml"><mi id="S3.SS3.SSS0.Px2.p7.7.m7.1.1.2.2" xref="S3.SS3.SSS0.Px2.p7.7.m7.1.1.2.2.cmml">𝑩</mi><mi id="S3.SS3.SSS0.Px2.p7.7.m7.1.1.2.3" xref="S3.SS3.SSS0.Px2.p7.7.m7.1.1.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS3.SSS0.Px2.p7.7.m7.1.1.1" xref="S3.SS3.SSS0.Px2.p7.7.m7.1.1.1.cmml">​</mo><msub id="S3.SS3.SSS0.Px2.p7.7.m7.1.1.3" xref="S3.SS3.SSS0.Px2.p7.7.m7.1.1.3.cmml"><mi id="S3.SS3.SSS0.Px2.p7.7.m7.1.1.3.2" xref="S3.SS3.SSS0.Px2.p7.7.m7.1.1.3.2.cmml">𝑨</mi><mi id="S3.SS3.SSS0.Px2.p7.7.m7.1.1.3.3" xref="S3.SS3.SSS0.Px2.p7.7.m7.1.1.3.3.cmml">i</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p7.7.m7.1b"><apply id="S3.SS3.SSS0.Px2.p7.7.m7.1.1.cmml" xref="S3.SS3.SSS0.Px2.p7.7.m7.1.1"><times id="S3.SS3.SSS0.Px2.p7.7.m7.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p7.7.m7.1.1.1"></times><apply id="S3.SS3.SSS0.Px2.p7.7.m7.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p7.7.m7.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p7.7.m7.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px2.p7.7.m7.1.1.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p7.7.m7.1.1.2.2.cmml" xref="S3.SS3.SSS0.Px2.p7.7.m7.1.1.2.2">𝑩</ci><ci id="S3.SS3.SSS0.Px2.p7.7.m7.1.1.2.3.cmml" xref="S3.SS3.SSS0.Px2.p7.7.m7.1.1.2.3">𝑖</ci></apply><apply id="S3.SS3.SSS0.Px2.p7.7.m7.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p7.7.m7.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p7.7.m7.1.1.3.1.cmml" xref="S3.SS3.SSS0.Px2.p7.7.m7.1.1.3">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p7.7.m7.1.1.3.2.cmml" xref="S3.SS3.SSS0.Px2.p7.7.m7.1.1.3.2">𝑨</ci><ci id="S3.SS3.SSS0.Px2.p7.7.m7.1.1.3.3.cmml" xref="S3.SS3.SSS0.Px2.p7.7.m7.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p7.7.m7.1c">\bm{B}_{i}\bm{A}_{i}</annotation></semantics></math> being updated.
The former is the initialization from LoRA paper, while the latter ensures that training starts from the original weights of <math id="S3.SS3.SSS0.Px2.p7.8.m8.1" class="ltx_Math" alttext="\bm{W}_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p7.8.m8.1a"><msub id="S3.SS3.SSS0.Px2.p7.8.m8.1.1" xref="S3.SS3.SSS0.Px2.p7.8.m8.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p7.8.m8.1.1.2" xref="S3.SS3.SSS0.Px2.p7.8.m8.1.1.2.cmml">𝑾</mi><mi id="S3.SS3.SSS0.Px2.p7.8.m8.1.1.3" xref="S3.SS3.SSS0.Px2.p7.8.m8.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p7.8.m8.1b"><apply id="S3.SS3.SSS0.Px2.p7.8.m8.1.1.cmml" xref="S3.SS3.SSS0.Px2.p7.8.m8.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p7.8.m8.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p7.8.m8.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px2.p7.8.m8.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p7.8.m8.1.1.2">𝑾</ci><ci id="S3.SS3.SSS0.Px2.p7.8.m8.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p7.8.m8.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p7.8.m8.1c">\bm{W}_{i}</annotation></semantics></math>.
We provide an empirical comparison of the two initialization strategies in Section <a href="#S4.SS3" title="4.3 Analysis: Key Factors for Parametric Knowledge Transfer ‣ 4 Experiments ‣ Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.3</span></a>.
Here, the introduction of LoRA module aims to effectively utilize the most salient features of the extracted knowledge in a concise and efficient manner.</p>
</div>
</section>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Experiments</h2>

<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Experimental Setup</h3>

<section id="S4.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Facets of Evaluation.</h4>

<div id="S4.SS1.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px1.p1.1" class="ltx_p">To validate the efficacy of our proposed framework, we conduct evaluations across four distinct benchmark categories:</p>
</div>
<div id="S4.SS1.SSS0.Px1.p2" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px1.p2.1" class="ltx_p">(1) <span id="S4.SS1.SSS0.Px1.p2.1.1" class="ltx_text ltx_framed ltx_framed_underline">Reasoning:</span> Reasoning stands as a foundational capability for models, particularly when tackling intricate tasks. We leverage the Grade School Math dataset (GSM) <cite class="ltx_cite ltx_citemacro_citep">(Cobbe et al., <a href="#bib.bib11" title="" class="ltx_ref">2021</a>)</cite> to assess the reasoning proficiency of models. The evaluation format requires models, given a math problem, to produce the chain-of-thought process <cite class="ltx_cite ltx_citemacro_citep">(Wei et al., <a href="#bib.bib63" title="" class="ltx_ref">2022</a>)</cite> and the final numerical answer.</p>
</div>
<div id="S4.SS1.SSS0.Px1.p3" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px1.p3.1" class="ltx_p">(2) <span id="S4.SS1.SSS0.Px1.p3.1.1" class="ltx_text ltx_framed ltx_framed_underline">Professional Knowledge:</span> For language models to effectively cater to users’ informational needs, possessing a robust repository of professional knowledge is crucial. We measure this knowledge reservoir using the Massive Multitask Language Understanding dataset (MMLU) <cite class="ltx_cite ltx_citemacro_citep">(Hendrycks et al., <a href="#bib.bib21" title="" class="ltx_ref">2021</a>)</cite>. This dataset encompasses questions about 57 subjects, spanning a spectrum of difficulty levels from elementary to advanced professional tiers, all formatted as multiple-choice questions.</p>
</div>
<div id="S4.SS1.SSS0.Px1.p4" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px1.p4.1" class="ltx_p">(3) <span id="S4.SS1.SSS0.Px1.p4.1.1" class="ltx_text ltx_framed ltx_framed_underline">Instruction-driven NLP Tasks:</span> This set of tasks evaluates a model’s capability in adhering to instructions. Typically, the language model receives both a task definition and an input text, and it must perform the specified classification or generation tasks as directed. Our chosen benchmark for this category is the Super Natural Instructions (Super NI) <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib60" title="" class="ltx_ref">2022</a>)</cite>, a rich dataset comprising 1,616 varied NLP tasks alongside their expert-written instructions.</p>
</div>
<div id="S4.SS1.SSS0.Px1.p5" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px1.p5.1" class="ltx_p">(4) <span id="S4.SS1.SSS0.Px1.p5.1.1" class="ltx_text ltx_framed ltx_framed_underline">Open-ended Conversation:</span> This represents the primary interface through which models interact with users in real-world scenarios. To evaluate such instructability, we employ AlpacaFarm <cite class="ltx_cite ltx_citemacro_citep">(Dubois et al., <a href="#bib.bib15" title="" class="ltx_ref">2023</a>)</cite>, which contains 805 instructions including subsets from various evaluations like Self-Instruct <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib62" title="" class="ltx_ref">2023c</a>)</cite>, Open Assistant <cite class="ltx_cite ltx_citemacro_citep">(Köpf et al., <a href="#bib.bib30" title="" class="ltx_ref">2023</a>)</cite>, Anthropic <cite class="ltx_cite ltx_citemacro_citep">(Bai et al., <a href="#bib.bib3" title="" class="ltx_ref">2022</a>)</cite>, Vicuna <cite class="ltx_cite ltx_citemacro_citep">(Chiang et al., <a href="#bib.bib9" title="" class="ltx_ref">2023</a>)</cite>, and Koala <cite class="ltx_cite ltx_citemacro_citep">(Geng et al., <a href="#bib.bib17" title="" class="ltx_ref">2023</a>)</cite>. GPT4-32K serves as the evaluator, determining the win rate of the testing model against the outputs generated by Davinci-003.</p>
</div>
<div id="S4.SS1.SSS0.Px1.p6" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px1.p6.1" class="ltx_p">Throughout all evaluations, we adhere to established metrics and prompts, utilizing the evaluation scripts sourced from <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib61" title="" class="ltx_ref">2023b</a>)</cite>.</p>
</div>
</section>
<section id="S4.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Implementation Details.</h4>

<div id="S4.SS1.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px2.p1.1" class="ltx_p">For all our experiments, the larger-scale LLaMA model <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib56" title="" class="ltx_ref">2023a</a>; <a href="#bib.bib57" title="" class="ltx_ref">b</a>)</cite> serves as the teacher, and its smaller-scale counterpart acts as the student. For the fine-tuning of the student model, we draw a random subset of 1,000 instances from the respective training datasets of each benchmark. In the case of AlpacaFarm, due to the absence of a training set, we utilize LIMA data <cite class="ltx_cite ltx_citemacro_citep">(Zhou et al., <a href="#bib.bib71" title="" class="ltx_ref">2023</a>)</cite> as a substitute, which is composed of 1,000 carefully curated open-ended conversations. For each experiment, 32 seed samples are randomly selected from the corresponding training sets. The student model is trained for 3 epochs with a batch size of 64 and a learning rate of 3e-4. Regarding LoRA, we set the rank as 16, and insert LoRA module into the embedding layer, FFN, and self-attention layer in the Transformer architecture <cite class="ltx_cite ltx_citemacro_citep">(Vaswani et al., <a href="#bib.bib58" title="" class="ltx_ref">2017</a>)</cite>. Notably, all results presented in this paper are mean values derived from three separate runs, with each run using a new random set of seed samples.</p>
</div>
</section>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Experimental Results</h3>

<figure id="S4.T1" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>Results for parametric knowledge transfer. “7B-LoRA + 13B Param.” represents that we extract parameters from the 13B teacher model and transfer them to the 7B student model.</figcaption>
<table id="S4.T1.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S4.T1.1.1" class="ltx_tr">
<td id="S4.T1.1.1.1" class="ltx_td ltx_align_left ltx_border_tt" style="padding:0.5pt 7.2pt;" rowspan="2"><span id="S4.T1.1.1.1.1" class="ltx_text ltx_font_bold">Models</span></td>
<td id="S4.T1.1.1.2" class="ltx_td ltx_align_center ltx_border_tt" style="padding:0.5pt 7.2pt;" colspan="2"><span id="S4.T1.1.1.2.1" class="ltx_text ltx_font_bold">GSM</span></td>
<td id="S4.T1.1.1.3" class="ltx_td ltx_align_center ltx_border_tt" style="padding:0.5pt 7.2pt;" colspan="2"><span id="S4.T1.1.1.3.1" class="ltx_text ltx_font_bold">MMLU</span></td>
<td id="S4.T1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt" style="padding:0.5pt 7.2pt;" colspan="2"><span id="S4.T1.1.1.4.1" class="ltx_text ltx_font_bold">Super NI</span></td>
<td id="S4.T1.1.1.5" class="ltx_td ltx_align_center ltx_border_tt" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.1.5.1" class="ltx_text ltx_font_bold">AlpacaFarm</span></td>
<td id="S4.T1.1.1.6" class="ltx_td ltx_align_center ltx_border_tt" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.1.6.1" class="ltx_text ltx_font_bold">Average</span></td>
</tr>
<tr id="S4.T1.1.2" class="ltx_tr">
<td id="S4.T1.1.2.1" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">0-shot</td>
<td id="S4.T1.1.2.2" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">8-shot</td>
<td id="S4.T1.1.2.3" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">0-shot</td>
<td id="S4.T1.1.2.4" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">5-shot</td>
<td id="S4.T1.1.2.5" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">EM</td>
<td id="S4.T1.1.2.6" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">R-L</td>
<td id="S4.T1.1.2.7" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">Win Rate%</td>
<td id="S4.T1.1.2.8" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">-</td>
</tr>
<tr id="S4.T1.1.3" class="ltx_tr">
<td id="S4.T1.1.3.1" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FAFAFF;padding:0.5pt 7.2pt;" colspan="9"><span id="S4.T1.1.3.1.1" class="ltx_text ltx_font_italic" style="background-color:#FAFAFF;">LLaMA-1</span></td>
</tr>
<tr id="S4.T1.1.4" class="ltx_tr">
<td id="S4.T1.1.4.1" class="ltx_td ltx_align_left ltx_border_t" style="padding:0.5pt 7.2pt;">Vanilla 7B</td>
<td id="S4.T1.1.4.2" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.2pt;">4.70</td>
<td id="S4.T1.1.4.3" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.2pt;">10.77</td>
<td id="S4.T1.1.4.4" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.2pt;">32.10</td>
<td id="S4.T1.1.4.5" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.2pt;">35.30</td>
<td id="S4.T1.1.4.6" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.2pt;">0.67</td>
<td id="S4.T1.1.4.7" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.2pt;">5.55</td>
<td id="S4.T1.1.4.8" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.2pt;">-</td>
<td id="S4.T1.1.4.9" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#ECECEC;padding:0.5pt 7.2pt;"><span id="S4.T1.1.4.9.1" class="ltx_text" style="background-color:#ECECEC;">-</span></td>
</tr>
<tr id="S4.T1.1.5" class="ltx_tr">
<td id="S4.T1.1.5.1" class="ltx_td ltx_align_left" style="padding:0.5pt 7.2pt;">7B-LoRA</td>
<td id="S4.T1.1.5.2" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">17.26</td>
<td id="S4.T1.1.5.3" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">16.93</td>
<td id="S4.T1.1.5.4" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">43.43</td>
<td id="S4.T1.1.5.5" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">38.90</td>
<td id="S4.T1.1.5.6" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">22.91</td>
<td id="S4.T1.1.5.7" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">40.49</td>
<td id="S4.T1.1.5.8" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">9.07</td>
<td id="S4.T1.1.5.9" class="ltx_td ltx_align_center" style="background-color:#ECECEC;padding:0.5pt 7.2pt;"><span id="S4.T1.1.5.9.1" class="ltx_text" style="background-color:#ECECEC;">27.00</span></td>
</tr>
<tr id="S4.T1.1.6" class="ltx_tr">
<td id="S4.T1.1.6.1" class="ltx_td ltx_align_left" style="padding:0.5pt 7.2pt;">    + 13B Param.</td>
<td id="S4.T1.1.6.2" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.6.2.1" class="ltx_text ltx_font_bold">18.73</span></td>
<td id="S4.T1.1.6.3" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.6.3.1" class="ltx_text ltx_font_bold">18.85</span></td>
<td id="S4.T1.1.6.4" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">44.03</td>
<td id="S4.T1.1.6.5" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">39.77</td>
<td id="S4.T1.1.6.6" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">24.51</td>
<td id="S4.T1.1.6.7" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">42.37</td>
<td id="S4.T1.1.6.8" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">9.28</td>
<td id="S4.T1.1.6.9" class="ltx_td ltx_align_center" style="background-color:#ECECEC;padding:0.5pt 7.2pt;"><span id="S4.T1.1.6.9.1" class="ltx_text" style="background-color:#ECECEC;">28.22</span></td>
</tr>
<tr id="S4.T1.1.7" class="ltx_tr">
<td id="S4.T1.1.7.1" class="ltx_td ltx_align_left" style="padding:0.5pt 7.2pt;">    + 30B Param.</td>
<td id="S4.T1.1.7.2" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">18.63</td>
<td id="S4.T1.1.7.3" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">18.52</td>
<td id="S4.T1.1.7.4" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.7.4.1" class="ltx_text ltx_font_bold">45.20</span></td>
<td id="S4.T1.1.7.5" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.7.5.1" class="ltx_text ltx_font_bold">40.60</span></td>
<td id="S4.T1.1.7.6" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.7.6.1" class="ltx_text ltx_font_bold">25.01</span></td>
<td id="S4.T1.1.7.7" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.7.7.1" class="ltx_text ltx_font_bold">43.08</span></td>
<td id="S4.T1.1.7.8" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.7.8.1" class="ltx_text ltx_font_bold">9.40</span></td>
<td id="S4.T1.1.7.9" class="ltx_td ltx_align_center" style="background-color:#ECECEC;padding:0.5pt 7.2pt;"><span id="S4.T1.1.7.9.1" class="ltx_text ltx_font_bold" style="background-color:#ECECEC;">28.63</span></td>
</tr>
<tr id="S4.T1.1.8" class="ltx_tr">
<td id="S4.T1.1.8.1" class="ltx_td ltx_align_left ltx_border_t" style="padding:0.5pt 7.2pt;">Vanilla 13B</td>
<td id="S4.T1.1.8.2" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.2pt;">4.93</td>
<td id="S4.T1.1.8.3" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.2pt;">17.44</td>
<td id="S4.T1.1.8.4" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.2pt;">43.50</td>
<td id="S4.T1.1.8.5" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.2pt;">46.80</td>
<td id="S4.T1.1.8.6" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.2pt;">2.18</td>
<td id="S4.T1.1.8.7" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.2pt;">7.78</td>
<td id="S4.T1.1.8.8" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.2pt;">-</td>
<td id="S4.T1.1.8.9" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#ECECEC;padding:0.5pt 7.2pt;"><span id="S4.T1.1.8.9.1" class="ltx_text" style="background-color:#ECECEC;">-</span></td>
</tr>
<tr id="S4.T1.1.9" class="ltx_tr">
<td id="S4.T1.1.9.1" class="ltx_td ltx_align_left" style="padding:0.5pt 7.2pt;">13B-LoRA</td>
<td id="S4.T1.1.9.2" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">26.18</td>
<td id="S4.T1.1.9.3" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">23.78</td>
<td id="S4.T1.1.9.4" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">50.43</td>
<td id="S4.T1.1.9.5" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">50.03</td>
<td id="S4.T1.1.9.6" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">27.34</td>
<td id="S4.T1.1.9.7" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">45.53</td>
<td id="S4.T1.1.9.8" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">13.91</td>
<td id="S4.T1.1.9.9" class="ltx_td ltx_align_center" style="background-color:#ECECEC;padding:0.5pt 7.2pt;"><span id="S4.T1.1.9.9.1" class="ltx_text" style="background-color:#ECECEC;">33.89</span></td>
</tr>
<tr id="S4.T1.1.10" class="ltx_tr">
<td id="S4.T1.1.10.1" class="ltx_td ltx_align_left" style="padding:0.5pt 7.2pt;">    + 30B Param.</td>
<td id="S4.T1.1.10.2" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.10.2.1" class="ltx_text ltx_font_bold">27.85</span></td>
<td id="S4.T1.1.10.3" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.10.3.1" class="ltx_text ltx_font_bold">27.70</span></td>
<td id="S4.T1.1.10.4" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.10.4.1" class="ltx_text ltx_font_bold">51.30</span></td>
<td id="S4.T1.1.10.5" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.10.5.1" class="ltx_text ltx_font_bold">51.03</span></td>
<td id="S4.T1.1.10.6" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.10.6.1" class="ltx_text ltx_font_bold">27.51</span></td>
<td id="S4.T1.1.10.7" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.10.7.1" class="ltx_text ltx_font_bold">46.09</span></td>
<td id="S4.T1.1.10.8" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.10.8.1" class="ltx_text ltx_font_bold">17.27</span></td>
<td id="S4.T1.1.10.9" class="ltx_td ltx_align_center" style="background-color:#ECECEC;padding:0.5pt 7.2pt;"><span id="S4.T1.1.10.9.1" class="ltx_text ltx_font_bold" style="background-color:#ECECEC;">35.54</span></td>
</tr>
<tr id="S4.T1.1.11" class="ltx_tr">
<td id="S4.T1.1.11.1" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FAFAFF;padding:0.5pt 7.2pt;" colspan="9"><span id="S4.T1.1.11.1.1" class="ltx_text ltx_font_italic" style="background-color:#FAFAFF;">LLaMA-2</span></td>
</tr>
<tr id="S4.T1.1.12" class="ltx_tr">
<td id="S4.T1.1.12.1" class="ltx_td ltx_align_left ltx_border_t" style="padding:0.5pt 7.2pt;">Vanilla 7B</td>
<td id="S4.T1.1.12.2" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.2pt;">3.34</td>
<td id="S4.T1.1.12.3" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.2pt;">15.54</td>
<td id="S4.T1.1.12.4" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.2pt;">41.70</td>
<td id="S4.T1.1.12.5" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.2pt;">45.80</td>
<td id="S4.T1.1.12.6" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.2pt;">0.00</td>
<td id="S4.T1.1.12.7" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.2pt;">4.68</td>
<td id="S4.T1.1.12.8" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.2pt;">-</td>
<td id="S4.T1.1.12.9" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#ECECEC;padding:0.5pt 7.2pt;"><span id="S4.T1.1.12.9.1" class="ltx_text" style="background-color:#ECECEC;">-</span></td>
</tr>
<tr id="S4.T1.1.13" class="ltx_tr">
<td id="S4.T1.1.13.1" class="ltx_td ltx_align_left" style="padding:0.5pt 7.2pt;">Vanilla 13B</td>
<td id="S4.T1.1.13.2" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">6.52</td>
<td id="S4.T1.1.13.3" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">27.82</td>
<td id="S4.T1.1.13.4" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">52.10</td>
<td id="S4.T1.1.13.5" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">55.20</td>
<td id="S4.T1.1.13.6" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">0.00</td>
<td id="S4.T1.1.13.7" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">4.84</td>
<td id="S4.T1.1.13.8" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">-</td>
<td id="S4.T1.1.13.9" class="ltx_td ltx_align_center" style="background-color:#ECECEC;padding:0.5pt 7.2pt;"><span id="S4.T1.1.13.9.1" class="ltx_text" style="background-color:#ECECEC;">-</span></td>
</tr>
<tr id="S4.T1.1.14" class="ltx_tr">
<td id="S4.T1.1.14.1" class="ltx_td ltx_align_left" style="padding:0.5pt 7.2pt;">7B-LoRA</td>
<td id="S4.T1.1.14.2" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">23.38</td>
<td id="S4.T1.1.14.3" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">21.05</td>
<td id="S4.T1.1.14.4" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">47.77</td>
<td id="S4.T1.1.14.5" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.14.5.1" class="ltx_text ltx_font_bold">47.07</span></td>
<td id="S4.T1.1.14.6" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">24.93</td>
<td id="S4.T1.1.14.7" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">41.25</td>
<td id="S4.T1.1.14.8" class="ltx_td ltx_align_center" style="padding:0.5pt 7.2pt;">20.50</td>
<td id="S4.T1.1.14.9" class="ltx_td ltx_align_center" style="background-color:#ECECEC;padding:0.5pt 7.2pt;"><span id="S4.T1.1.14.9.1" class="ltx_text" style="background-color:#ECECEC;">32.28</span></td>
</tr>
<tr id="S4.T1.1.15" class="ltx_tr">
<td id="S4.T1.1.15.1" class="ltx_td ltx_align_left ltx_border_bb" style="padding:0.5pt 7.2pt;">    + 13B Param.</td>
<td id="S4.T1.1.15.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.15.2.1" class="ltx_text ltx_font_bold">25.30</span></td>
<td id="S4.T1.1.15.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.15.3.1" class="ltx_text ltx_font_bold">26.31</span></td>
<td id="S4.T1.1.15.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.15.4.1" class="ltx_text ltx_font_bold">49.37</span></td>
<td id="S4.T1.1.15.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.5pt 7.2pt;">46.53</td>
<td id="S4.T1.1.15.6" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.15.6.1" class="ltx_text ltx_font_bold">26.16</span></td>
<td id="S4.T1.1.15.7" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.15.7.1" class="ltx_text ltx_font_bold">42.98</span></td>
<td id="S4.T1.1.15.8" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.5pt 7.2pt;"><span id="S4.T1.1.15.8.1" class="ltx_text ltx_font_bold">24.64</span></td>
<td id="S4.T1.1.15.9" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#ECECEC;padding:0.5pt 7.2pt;"><span id="S4.T1.1.15.9.1" class="ltx_text ltx_font_bold" style="background-color:#ECECEC;">34.47</span></td>
</tr>
</table>
</figure>
<section id="S4.SS2.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Results for Parametric Knowledge Transfer.</h4>

<div id="S4.SS2.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS2.SSS0.Px1.p1.6" class="ltx_p">Our initial experiments focus on four distinct teacher-student model pairings: LLaMA-1 (13B <math id="S4.SS2.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="\Rightarrow" display="inline"><semantics id="S4.SS2.SSS0.Px1.p1.1.m1.1a"><mo stretchy="false" id="S4.SS2.SSS0.Px1.p1.1.m1.1.1" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.1.cmml">⇒</mo><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p1.1.m1.1b"><ci id="S4.SS2.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.1">⇒</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p1.1.m1.1c">\Rightarrow</annotation></semantics></math> 7B, 30B <math id="S4.SS2.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="\Rightarrow" display="inline"><semantics id="S4.SS2.SSS0.Px1.p1.2.m2.1a"><mo stretchy="false" id="S4.SS2.SSS0.Px1.p1.2.m2.1.1" xref="S4.SS2.SSS0.Px1.p1.2.m2.1.1.cmml">⇒</mo><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p1.2.m2.1b"><ci id="S4.SS2.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S4.SS2.SSS0.Px1.p1.2.m2.1.1">⇒</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p1.2.m2.1c">\Rightarrow</annotation></semantics></math> 7B, 30B <math id="S4.SS2.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="\Rightarrow" display="inline"><semantics id="S4.SS2.SSS0.Px1.p1.3.m3.1a"><mo stretchy="false" id="S4.SS2.SSS0.Px1.p1.3.m3.1.1" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.1.cmml">⇒</mo><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p1.3.m3.1b"><ci id="S4.SS2.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.1">⇒</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p1.3.m3.1c">\Rightarrow</annotation></semantics></math> 13B) and LLaMA-2 (13B <math id="S4.SS2.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="\Rightarrow" display="inline"><semantics id="S4.SS2.SSS0.Px1.p1.4.m4.1a"><mo stretchy="false" id="S4.SS2.SSS0.Px1.p1.4.m4.1.1" xref="S4.SS2.SSS0.Px1.p1.4.m4.1.1.cmml">⇒</mo><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p1.4.m4.1b"><ci id="S4.SS2.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S4.SS2.SSS0.Px1.p1.4.m4.1.1">⇒</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p1.4.m4.1c">\Rightarrow</annotation></semantics></math> 7B). The outcomes are systematically presented in Table <a href="#S4.T1" title="Table 1 ‣ 4.2 Experimental Results ‣ 4 Experiments ‣ Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>. Remarkably, in contrast to the direct fine-tuning approach of LoRA, the student models augmented with parametric knowledge from their respective teacher models exhibit substantial improvements across all four benchmark categories. For instance, the LLaMA-1 30B <math id="S4.SS2.SSS0.Px1.p1.5.m5.1" class="ltx_Math" alttext="\Rightarrow" display="inline"><semantics id="S4.SS2.SSS0.Px1.p1.5.m5.1a"><mo stretchy="false" id="S4.SS2.SSS0.Px1.p1.5.m5.1.1" xref="S4.SS2.SSS0.Px1.p1.5.m5.1.1.cmml">⇒</mo><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p1.5.m5.1b"><ci id="S4.SS2.SSS0.Px1.p1.5.m5.1.1.cmml" xref="S4.SS2.SSS0.Px1.p1.5.m5.1.1">⇒</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p1.5.m5.1c">\Rightarrow</annotation></semantics></math> 7B pairing yields an average performance boost of 6.04% (from 27.00 to 28.63). In a similar vein, the LLaMA-2 13B <math id="S4.SS2.SSS0.Px1.p1.6.m6.1" class="ltx_Math" alttext="\Rightarrow" display="inline"><semantics id="S4.SS2.SSS0.Px1.p1.6.m6.1a"><mo stretchy="false" id="S4.SS2.SSS0.Px1.p1.6.m6.1.1" xref="S4.SS2.SSS0.Px1.p1.6.m6.1.1.cmml">⇒</mo><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p1.6.m6.1b"><ci id="S4.SS2.SSS0.Px1.p1.6.m6.1.1.cmml" xref="S4.SS2.SSS0.Px1.p1.6.m6.1.1">⇒</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p1.6.m6.1c">\Rightarrow</annotation></semantics></math> 7B configuration brings an enhancement of 6.78% (from 32.28 to 34.47).</p>
</div>
<div id="S4.SS2.SSS0.Px1.p2" class="ltx_para ltx_noindent">
<p id="S4.SS2.SSS0.Px1.p2.1" class="ltx_p">Another observation emerges when examining the effects of scaling up the teacher model, specifically transitioning from 13B to 30B. The performance of the student model, LLaMA-1 7B, generally sees an improvement, despite a slight decrement in the GSM benchmark. Beyond the evident performance gains, the overhead introduced by parametric knowledge transfer remains minimal. The only extra commitment involves the teacher model executing inference on a set of 32 seed samples, without any direct participation in the training. Considering both performance and efficiency, parametric knowledge transfer stands out as a practical technique, even as disparities in parameter counts and architectural variances between teacher and student models expand.</p>
</div>
<figure id="S4.T2" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>Transfer experiments with different task-specific extracted parameters. The leftmost column indicates the dataset on which the knowledge extraction is based. The teacher model and student model are LLaMA-2 13B and 7B, respectively.</figcaption>
<table id="S4.T2.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S4.T2.1.1" class="ltx_tr">
<td id="S4.T2.1.1.1" class="ltx_td ltx_align_left ltx_border_tt" style="padding:0.5pt 7.9pt;" rowspan="2"><span id="S4.T2.1.1.1.1" class="ltx_text ltx_font_bold">Models</span></td>
<td id="S4.T2.1.1.2" class="ltx_td ltx_align_center ltx_border_tt" style="padding:0.5pt 7.9pt;" colspan="2"><span id="S4.T2.1.1.2.1" class="ltx_text ltx_font_bold">GSM</span></td>
<td id="S4.T2.1.1.3" class="ltx_td ltx_align_center ltx_border_tt" style="padding:0.5pt 7.9pt;" colspan="2"><span id="S4.T2.1.1.3.1" class="ltx_text ltx_font_bold">MMLU</span></td>
<td id="S4.T2.1.1.4" class="ltx_td ltx_align_center ltx_border_tt" style="padding:0.5pt 7.9pt;" colspan="2"><span id="S4.T2.1.1.4.1" class="ltx_text ltx_font_bold">Super NI</span></td>
<td id="S4.T2.1.1.5" class="ltx_td ltx_align_center ltx_border_tt" style="padding:0.5pt 7.9pt;"><span id="S4.T2.1.1.5.1" class="ltx_text ltx_font_bold">AlpacaFarm</span></td>
<td id="S4.T2.1.1.6" class="ltx_td ltx_align_center ltx_border_tt" style="padding:0.5pt 7.9pt;"><span id="S4.T2.1.1.6.1" class="ltx_text ltx_font_bold">Average</span></td>
</tr>
<tr id="S4.T2.1.2" class="ltx_tr">
<td id="S4.T2.1.2.1" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">0-shot</td>
<td id="S4.T2.1.2.2" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">8-shot</td>
<td id="S4.T2.1.2.3" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">0-shot</td>
<td id="S4.T2.1.2.4" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">5-shot</td>
<td id="S4.T2.1.2.5" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">EM</td>
<td id="S4.T2.1.2.6" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">R-L</td>
<td id="S4.T2.1.2.7" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">Win Rate%</td>
<td id="S4.T2.1.2.8" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">-</td>
</tr>
<tr id="S4.T2.1.3" class="ltx_tr">
<td id="S4.T2.1.3.1" class="ltx_td ltx_align_left ltx_border_t" style="padding:0.5pt 7.9pt;">Vanilla 7B</td>
<td id="S4.T2.1.3.2" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.9pt;">3.34</td>
<td id="S4.T2.1.3.3" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.9pt;">15.54</td>
<td id="S4.T2.1.3.4" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.9pt;">41.70</td>
<td id="S4.T2.1.3.5" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.9pt;">45.80</td>
<td id="S4.T2.1.3.6" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.9pt;">0.00</td>
<td id="S4.T2.1.3.7" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.9pt;">4.68</td>
<td id="S4.T2.1.3.8" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.9pt;">-</td>
<td id="S4.T2.1.3.9" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#ECECEC;padding:0.5pt 7.9pt;"><span id="S4.T2.1.3.9.1" class="ltx_text" style="background-color:#ECECEC;">-</span></td>
</tr>
<tr id="S4.T2.1.4" class="ltx_tr">
<td id="S4.T2.1.4.1" class="ltx_td ltx_align_left" style="padding:0.5pt 7.9pt;">7B-LoRA</td>
<td id="S4.T2.1.4.2" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">23.38</td>
<td id="S4.T2.1.4.3" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">21.05</td>
<td id="S4.T2.1.4.4" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">47.77</td>
<td id="S4.T2.1.4.5" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">47.07</td>
<td id="S4.T2.1.4.6" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">24.93</td>
<td id="S4.T2.1.4.7" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">41.25</td>
<td id="S4.T2.1.4.8" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">20.50</td>
<td id="S4.T2.1.4.9" class="ltx_td ltx_align_center" style="background-color:#ECECEC;padding:0.5pt 7.9pt;"><span id="S4.T2.1.4.9.1" class="ltx_text" style="background-color:#ECECEC;">32.28</span></td>
</tr>
<tr id="S4.T2.1.5" class="ltx_tr">
<td id="S4.T2.1.5.1" class="ltx_td ltx_align_left ltx_border_t" style="padding:0.5pt 7.9pt;">GSM</td>
<td id="S4.T2.1.5.2" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.9pt;"><span id="S4.T2.1.5.2.1" class="ltx_text ltx_font_bold">25.30</span></td>
<td id="S4.T2.1.5.3" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.9pt;"><span id="S4.T2.1.5.3.1" class="ltx_text ltx_font_bold">26.31</span></td>
<td id="S4.T2.1.5.4" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.9pt;">48.40</td>
<td id="S4.T2.1.5.5" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.9pt;">45.97</td>
<td id="S4.T2.1.5.6" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.9pt;">24.45</td>
<td id="S4.T2.1.5.7" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.9pt;">42.11</td>
<td id="S4.T2.1.5.8" class="ltx_td ltx_align_center ltx_border_t" style="padding:0.5pt 7.9pt;">23.68</td>
<td id="S4.T2.1.5.9" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#ECECEC;padding:0.5pt 7.9pt;"><span id="S4.T2.1.5.9.1" class="ltx_text" style="background-color:#ECECEC;">33.75</span></td>
</tr>
<tr id="S4.T2.1.6" class="ltx_tr">
<td id="S4.T2.1.6.1" class="ltx_td ltx_align_left" style="padding:0.5pt 7.9pt;">MMLU</td>
<td id="S4.T2.1.6.2" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">24.11</td>
<td id="S4.T2.1.6.3" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">25.47</td>
<td id="S4.T2.1.6.4" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;"><span id="S4.T2.1.6.4.1" class="ltx_text ltx_font_bold">49.37</span></td>
<td id="S4.T2.1.6.5" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">46.53</td>
<td id="S4.T2.1.6.6" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">25.55</td>
<td id="S4.T2.1.6.7" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">42.55</td>
<td id="S4.T2.1.6.8" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">24.01</td>
<td id="S4.T2.1.6.9" class="ltx_td ltx_align_center" style="background-color:#ECECEC;padding:0.5pt 7.9pt;"><span id="S4.T2.1.6.9.1" class="ltx_text" style="background-color:#ECECEC;">33.94</span></td>
</tr>
<tr id="S4.T2.1.7" class="ltx_tr">
<td id="S4.T2.1.7.1" class="ltx_td ltx_align_left" style="padding:0.5pt 7.9pt;">Super NI</td>
<td id="S4.T2.1.7.2" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">23.78</td>
<td id="S4.T2.1.7.3" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">24.11</td>
<td id="S4.T2.1.7.4" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">48.60</td>
<td id="S4.T2.1.7.5" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">46.70</td>
<td id="S4.T2.1.7.6" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;"><span id="S4.T2.1.7.6.1" class="ltx_text ltx_font_bold">26.16</span></td>
<td id="S4.T2.1.7.7" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;"><span id="S4.T2.1.7.7.1" class="ltx_text ltx_font_bold">42.98</span></td>
<td id="S4.T2.1.7.8" class="ltx_td ltx_align_center" style="padding:0.5pt 7.9pt;">24.31</td>
<td id="S4.T2.1.7.9" class="ltx_td ltx_align_center" style="background-color:#ECECEC;padding:0.5pt 7.9pt;"><span id="S4.T2.1.7.9.1" class="ltx_text" style="background-color:#ECECEC;">33.81</span></td>
</tr>
<tr id="S4.T2.1.8" class="ltx_tr">
<td id="S4.T2.1.8.1" class="ltx_td ltx_align_left ltx_border_bb" style="padding:0.5pt 7.9pt;">LIMA</td>
<td id="S4.T2.1.8.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.5pt 7.9pt;">24.08</td>
<td id="S4.T2.1.8.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.5pt 7.9pt;">25.60</td>
<td id="S4.T2.1.8.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.5pt 7.9pt;">49.03</td>
<td id="S4.T2.1.8.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.5pt 7.9pt;"><span id="S4.T2.1.8.5.1" class="ltx_text ltx_font_bold">47.23</span></td>
<td id="S4.T2.1.8.6" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.5pt 7.9pt;">25.63</td>
<td id="S4.T2.1.8.7" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.5pt 7.9pt;">42.83</td>
<td id="S4.T2.1.8.8" class="ltx_td ltx_align_center ltx_border_bb" style="padding:0.5pt 7.9pt;"><span id="S4.T2.1.8.8.1" class="ltx_text ltx_font_bold">24.64</span></td>
<td id="S4.T2.1.8.9" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#ECECEC;padding:0.5pt 7.9pt;"><span id="S4.T2.1.8.9.1" class="ltx_text ltx_font_bold" style="background-color:#ECECEC;">34.15</span></td>
</tr>
</table>
</figure>
</section>
<section id="S4.SS2.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Transfer Experiments with Task-specific Extracted Parameters.</h4>

<div id="S4.SS2.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS2.SSS0.Px2.p1.1" class="ltx_p">While our results indicate that transferring extracted knowledge from the teacher model positively influences student model performance, the nature of this improvement—whether it is rooted in generalized knowledge or task-specific expertise—warrants deeper exploration. To disentangle this, we conduct experiments wherein extracted parameters, each tailored to a specific task, are integrated into the student model, which is subsequently fine-tuned across all datasets.</p>
</div>
<div id="S4.SS2.SSS0.Px2.p2" class="ltx_para ltx_noindent">
<p id="S4.SS2.SSS0.Px2.p2.1" class="ltx_p">Table <a href="#S4.T2" title="Table 2 ‣ Results for Parametric Knowledge Transfer. ‣ 4.2 Experimental Results ‣ 4 Experiments ‣ Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> offers insights into a prevalent trend: when parameters are extracted from a concrete task, the performance is most significantly amplified for that same task. This is particularly evident in the GSM benchmark. Models equipped with GSM-oriented extracted parameters notably exceed their counterparts—achieving at least a 1.2 increase in 0-shot accuracy—compared to models incorporated with parameters based on alternative datasets. This is likely due to the unique and intricate challenges associated with mathematical reasoning. Additionally, parameters sourced from the LIMA dataset demonstrate remarkable generalizability, presumably owing to their grounding in open-ended dialogues that cover a spectrum of domains and tasks. Overall, these observations highlight the capability of our sensitivity-driven techniques to efficiently target certain types of knowledge, rather than just extracting generalized knowledge.</p>
</div>
</section>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Analysis: Key Factors for Parametric Knowledge Transfer</h3>

<div id="S4.SS3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.p1.1" class="ltx_p">We further analyze the key factors for the process of parametric knowledge transfer as follows.</p>
</div>
<section id="S4.SS3.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Initialization Strategies.</h4>

<figure id="S4.F3" class="ltx_figure ltx_align_floatright"><img src="/html/2310.11451/assets/x3.png" id="S4.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="244" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Comparison of different initialization strategies. The y-axis represents the average score over four datasets. For <math id="S4.F3.8.m1.1" class="ltx_Math" alttext="\bm{W}^{*}=\bm{W}+\bm{BA}" display="inline"><semantics id="S4.F3.8.m1.1b"><mrow id="S4.F3.8.m1.1.1" xref="S4.F3.8.m1.1.1.cmml"><msup id="S4.F3.8.m1.1.1.2" xref="S4.F3.8.m1.1.1.2.cmml"><mi id="S4.F3.8.m1.1.1.2.2" xref="S4.F3.8.m1.1.1.2.2.cmml">𝑾</mi><mo id="S4.F3.8.m1.1.1.2.3" xref="S4.F3.8.m1.1.1.2.3.cmml">∗</mo></msup><mo id="S4.F3.8.m1.1.1.1" xref="S4.F3.8.m1.1.1.1.cmml">=</mo><mrow id="S4.F3.8.m1.1.1.3" xref="S4.F3.8.m1.1.1.3.cmml"><mi id="S4.F3.8.m1.1.1.3.2" xref="S4.F3.8.m1.1.1.3.2.cmml">𝑾</mi><mo id="S4.F3.8.m1.1.1.3.1" xref="S4.F3.8.m1.1.1.3.1.cmml">+</mo><mrow id="S4.F3.8.m1.1.1.3.3" xref="S4.F3.8.m1.1.1.3.3.cmml"><mi id="S4.F3.8.m1.1.1.3.3.2" xref="S4.F3.8.m1.1.1.3.3.2.cmml">𝑩</mi><mo lspace="0em" rspace="0em" id="S4.F3.8.m1.1.1.3.3.1" xref="S4.F3.8.m1.1.1.3.3.1.cmml">​</mo><mi id="S4.F3.8.m1.1.1.3.3.3" xref="S4.F3.8.m1.1.1.3.3.3.cmml">𝑨</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.F3.8.m1.1c"><apply id="S4.F3.8.m1.1.1.cmml" xref="S4.F3.8.m1.1.1"><eq id="S4.F3.8.m1.1.1.1.cmml" xref="S4.F3.8.m1.1.1.1"></eq><apply id="S4.F3.8.m1.1.1.2.cmml" xref="S4.F3.8.m1.1.1.2"><csymbol cd="ambiguous" id="S4.F3.8.m1.1.1.2.1.cmml" xref="S4.F3.8.m1.1.1.2">superscript</csymbol><ci id="S4.F3.8.m1.1.1.2.2.cmml" xref="S4.F3.8.m1.1.1.2.2">𝑾</ci><times id="S4.F3.8.m1.1.1.2.3.cmml" xref="S4.F3.8.m1.1.1.2.3"></times></apply><apply id="S4.F3.8.m1.1.1.3.cmml" xref="S4.F3.8.m1.1.1.3"><plus id="S4.F3.8.m1.1.1.3.1.cmml" xref="S4.F3.8.m1.1.1.3.1"></plus><ci id="S4.F3.8.m1.1.1.3.2.cmml" xref="S4.F3.8.m1.1.1.3.2">𝑾</ci><apply id="S4.F3.8.m1.1.1.3.3.cmml" xref="S4.F3.8.m1.1.1.3.3"><times id="S4.F3.8.m1.1.1.3.3.1.cmml" xref="S4.F3.8.m1.1.1.3.3.1"></times><ci id="S4.F3.8.m1.1.1.3.3.2.cmml" xref="S4.F3.8.m1.1.1.3.3.2">𝑩</ci><ci id="S4.F3.8.m1.1.1.3.3.3.cmml" xref="S4.F3.8.m1.1.1.3.3.3">𝑨</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F3.8.m1.1d">\bm{W}^{*}=\bm{W}+\bm{BA}</annotation></semantics></math>, we employ five methods to initialize <math id="S4.F3.9.m2.1" class="ltx_Math" alttext="\bm{BA}" display="inline"><semantics id="S4.F3.9.m2.1b"><mrow id="S4.F3.9.m2.1.1" xref="S4.F3.9.m2.1.1.cmml"><mi id="S4.F3.9.m2.1.1.2" xref="S4.F3.9.m2.1.1.2.cmml">𝑩</mi><mo lspace="0em" rspace="0em" id="S4.F3.9.m2.1.1.1" xref="S4.F3.9.m2.1.1.1.cmml">​</mo><mi id="S4.F3.9.m2.1.1.3" xref="S4.F3.9.m2.1.1.3.cmml">𝑨</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.F3.9.m2.1c"><apply id="S4.F3.9.m2.1.1.cmml" xref="S4.F3.9.m2.1.1"><times id="S4.F3.9.m2.1.1.1.cmml" xref="S4.F3.9.m2.1.1.1"></times><ci id="S4.F3.9.m2.1.1.2.cmml" xref="S4.F3.9.m2.1.1.2">𝑩</ci><ci id="S4.F3.9.m2.1.1.3.cmml" xref="S4.F3.9.m2.1.1.3">𝑨</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F3.9.m2.1d">\bm{BA}</annotation></semantics></math>, keeping <math id="S4.F3.10.m3.1" class="ltx_Math" alttext="\bm{W}" display="inline"><semantics id="S4.F3.10.m3.1b"><mi id="S4.F3.10.m3.1.1" xref="S4.F3.10.m3.1.1.cmml">𝑾</mi><annotation-xml encoding="MathML-Content" id="S4.F3.10.m3.1c"><ci id="S4.F3.10.m3.1.1.cmml" xref="S4.F3.10.m3.1.1">𝑾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F3.10.m3.1d">\bm{W}</annotation></semantics></math> constant. In the case of <math id="S4.F3.11.m4.1" class="ltx_Math" alttext="\bm{W}^{*}=\bm{W}-\bm{W}_{\text{extract}}+\bm{BA}" display="inline"><semantics id="S4.F3.11.m4.1b"><mrow id="S4.F3.11.m4.1.1" xref="S4.F3.11.m4.1.1.cmml"><msup id="S4.F3.11.m4.1.1.2" xref="S4.F3.11.m4.1.1.2.cmml"><mi id="S4.F3.11.m4.1.1.2.2" xref="S4.F3.11.m4.1.1.2.2.cmml">𝑾</mi><mo id="S4.F3.11.m4.1.1.2.3" xref="S4.F3.11.m4.1.1.2.3.cmml">∗</mo></msup><mo id="S4.F3.11.m4.1.1.1" xref="S4.F3.11.m4.1.1.1.cmml">=</mo><mrow id="S4.F3.11.m4.1.1.3" xref="S4.F3.11.m4.1.1.3.cmml"><mrow id="S4.F3.11.m4.1.1.3.2" xref="S4.F3.11.m4.1.1.3.2.cmml"><mi id="S4.F3.11.m4.1.1.3.2.2" xref="S4.F3.11.m4.1.1.3.2.2.cmml">𝑾</mi><mo id="S4.F3.11.m4.1.1.3.2.1" xref="S4.F3.11.m4.1.1.3.2.1.cmml">−</mo><msub id="S4.F3.11.m4.1.1.3.2.3" xref="S4.F3.11.m4.1.1.3.2.3.cmml"><mi id="S4.F3.11.m4.1.1.3.2.3.2" xref="S4.F3.11.m4.1.1.3.2.3.2.cmml">𝑾</mi><mtext id="S4.F3.11.m4.1.1.3.2.3.3" xref="S4.F3.11.m4.1.1.3.2.3.3a.cmml">extract</mtext></msub></mrow><mo id="S4.F3.11.m4.1.1.3.1" xref="S4.F3.11.m4.1.1.3.1.cmml">+</mo><mrow id="S4.F3.11.m4.1.1.3.3" xref="S4.F3.11.m4.1.1.3.3.cmml"><mi id="S4.F3.11.m4.1.1.3.3.2" xref="S4.F3.11.m4.1.1.3.3.2.cmml">𝑩</mi><mo lspace="0em" rspace="0em" id="S4.F3.11.m4.1.1.3.3.1" xref="S4.F3.11.m4.1.1.3.3.1.cmml">​</mo><mi id="S4.F3.11.m4.1.1.3.3.3" xref="S4.F3.11.m4.1.1.3.3.3.cmml">𝑨</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.F3.11.m4.1c"><apply id="S4.F3.11.m4.1.1.cmml" xref="S4.F3.11.m4.1.1"><eq id="S4.F3.11.m4.1.1.1.cmml" xref="S4.F3.11.m4.1.1.1"></eq><apply id="S4.F3.11.m4.1.1.2.cmml" xref="S4.F3.11.m4.1.1.2"><csymbol cd="ambiguous" id="S4.F3.11.m4.1.1.2.1.cmml" xref="S4.F3.11.m4.1.1.2">superscript</csymbol><ci id="S4.F3.11.m4.1.1.2.2.cmml" xref="S4.F3.11.m4.1.1.2.2">𝑾</ci><times id="S4.F3.11.m4.1.1.2.3.cmml" xref="S4.F3.11.m4.1.1.2.3"></times></apply><apply id="S4.F3.11.m4.1.1.3.cmml" xref="S4.F3.11.m4.1.1.3"><plus id="S4.F3.11.m4.1.1.3.1.cmml" xref="S4.F3.11.m4.1.1.3.1"></plus><apply id="S4.F3.11.m4.1.1.3.2.cmml" xref="S4.F3.11.m4.1.1.3.2"><minus id="S4.F3.11.m4.1.1.3.2.1.cmml" xref="S4.F3.11.m4.1.1.3.2.1"></minus><ci id="S4.F3.11.m4.1.1.3.2.2.cmml" xref="S4.F3.11.m4.1.1.3.2.2">𝑾</ci><apply id="S4.F3.11.m4.1.1.3.2.3.cmml" xref="S4.F3.11.m4.1.1.3.2.3"><csymbol cd="ambiguous" id="S4.F3.11.m4.1.1.3.2.3.1.cmml" xref="S4.F3.11.m4.1.1.3.2.3">subscript</csymbol><ci id="S4.F3.11.m4.1.1.3.2.3.2.cmml" xref="S4.F3.11.m4.1.1.3.2.3.2">𝑾</ci><ci id="S4.F3.11.m4.1.1.3.2.3.3a.cmml" xref="S4.F3.11.m4.1.1.3.2.3.3"><mtext mathsize="70%" id="S4.F3.11.m4.1.1.3.2.3.3.cmml" xref="S4.F3.11.m4.1.1.3.2.3.3">extract</mtext></ci></apply></apply><apply id="S4.F3.11.m4.1.1.3.3.cmml" xref="S4.F3.11.m4.1.1.3.3"><times id="S4.F3.11.m4.1.1.3.3.1.cmml" xref="S4.F3.11.m4.1.1.3.3.1"></times><ci id="S4.F3.11.m4.1.1.3.3.2.cmml" xref="S4.F3.11.m4.1.1.3.3.2">𝑩</ci><ci id="S4.F3.11.m4.1.1.3.3.3.cmml" xref="S4.F3.11.m4.1.1.3.3.3">𝑨</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F3.11.m4.1d">\bm{W}^{*}=\bm{W}-\bm{W}_{\text{extract}}+\bm{BA}</annotation></semantics></math>, these methods correspond to both <math id="S4.F3.12.m5.1" class="ltx_Math" alttext="\bm{W}_{\text{extract}}" display="inline"><semantics id="S4.F3.12.m5.1b"><msub id="S4.F3.12.m5.1.1" xref="S4.F3.12.m5.1.1.cmml"><mi id="S4.F3.12.m5.1.1.2" xref="S4.F3.12.m5.1.1.2.cmml">𝑾</mi><mtext id="S4.F3.12.m5.1.1.3" xref="S4.F3.12.m5.1.1.3a.cmml">extract</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.F3.12.m5.1c"><apply id="S4.F3.12.m5.1.1.cmml" xref="S4.F3.12.m5.1.1"><csymbol cd="ambiguous" id="S4.F3.12.m5.1.1.1.cmml" xref="S4.F3.12.m5.1.1">subscript</csymbol><ci id="S4.F3.12.m5.1.1.2.cmml" xref="S4.F3.12.m5.1.1.2">𝑾</ci><ci id="S4.F3.12.m5.1.1.3a.cmml" xref="S4.F3.12.m5.1.1.3"><mtext mathsize="70%" id="S4.F3.12.m5.1.1.3.cmml" xref="S4.F3.12.m5.1.1.3">extract</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F3.12.m5.1d">\bm{W}_{\text{extract}}</annotation></semantics></math> and <math id="S4.F3.13.m6.1" class="ltx_Math" alttext="\bm{BA}" display="inline"><semantics id="S4.F3.13.m6.1b"><mrow id="S4.F3.13.m6.1.1" xref="S4.F3.13.m6.1.1.cmml"><mi id="S4.F3.13.m6.1.1.2" xref="S4.F3.13.m6.1.1.2.cmml">𝑩</mi><mo lspace="0em" rspace="0em" id="S4.F3.13.m6.1.1.1" xref="S4.F3.13.m6.1.1.1.cmml">​</mo><mi id="S4.F3.13.m6.1.1.3" xref="S4.F3.13.m6.1.1.3.cmml">𝑨</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.F3.13.m6.1c"><apply id="S4.F3.13.m6.1.1.cmml" xref="S4.F3.13.m6.1.1"><times id="S4.F3.13.m6.1.1.1.cmml" xref="S4.F3.13.m6.1.1.1"></times><ci id="S4.F3.13.m6.1.1.2.cmml" xref="S4.F3.13.m6.1.1.2">𝑩</ci><ci id="S4.F3.13.m6.1.1.3.cmml" xref="S4.F3.13.m6.1.1.3">𝑨</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F3.13.m6.1d">\bm{BA}</annotation></semantics></math>, with <math id="S4.F3.14.m7.1" class="ltx_Math" alttext="\bm{W}_{\text{extract}}" display="inline"><semantics id="S4.F3.14.m7.1b"><msub id="S4.F3.14.m7.1.1" xref="S4.F3.14.m7.1.1.cmml"><mi id="S4.F3.14.m7.1.1.2" xref="S4.F3.14.m7.1.1.2.cmml">𝑾</mi><mtext id="S4.F3.14.m7.1.1.3" xref="S4.F3.14.m7.1.1.3a.cmml">extract</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.F3.14.m7.1c"><apply id="S4.F3.14.m7.1.1.cmml" xref="S4.F3.14.m7.1.1"><csymbol cd="ambiguous" id="S4.F3.14.m7.1.1.1.cmml" xref="S4.F3.14.m7.1.1">subscript</csymbol><ci id="S4.F3.14.m7.1.1.2.cmml" xref="S4.F3.14.m7.1.1.2">𝑾</ci><ci id="S4.F3.14.m7.1.1.3a.cmml" xref="S4.F3.14.m7.1.1.3"><mtext mathsize="70%" id="S4.F3.14.m7.1.1.3.cmml" xref="S4.F3.14.m7.1.1.3">extract</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F3.14.m7.1d">\bm{W}_{\text{extract}}</annotation></semantics></math> remaining static during training. The orange dotted line denotes the results of initialization as described in LoRA paper.</figcaption>
</figure>
<div id="S4.SS3.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS0.Px1.p1.2" class="ltx_p">Essentially, for the student model, the injected parameters correspond to the initialization of LoRA module. We compare two initialization methods: the approach described in the LoRA paper, denoted by Equation <a href="#S3.E9" title="In Knowledge Injection with LoRA. ‣ 3.3 Knowledge Injection ‣ 3 Parametric Knowledge Transfer ‣ Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>, and the method adopted in our work, represented by Equation <a href="#S3.E10" title="In Knowledge Injection with LoRA. ‣ 3.3 Knowledge Injection ‣ 3 Parametric Knowledge Transfer ‣ Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a>. Furthermore, we investigate five distinct initializations for LoRA matrix: (1) Gaussian initialization for both <math id="S4.SS3.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="\bm{B}" display="inline"><semantics id="S4.SS3.SSS0.Px1.p1.1.m1.1a"><mi id="S4.SS3.SSS0.Px1.p1.1.m1.1.1" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1.cmml">𝑩</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px1.p1.1.m1.1b"><ci id="S4.SS3.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S4.SS3.SSS0.Px1.p1.1.m1.1.1">𝑩</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px1.p1.1.m1.1c">\bm{B}</annotation></semantics></math> and <math id="S4.SS3.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="\bm{A}" display="inline"><semantics id="S4.SS3.SSS0.Px1.p1.2.m2.1a"><mi id="S4.SS3.SSS0.Px1.p1.2.m2.1.1" xref="S4.SS3.SSS0.Px1.p1.2.m2.1.1.cmml">𝑨</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px1.p1.2.m2.1b"><ci id="S4.SS3.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S4.SS3.SSS0.Px1.p1.2.m2.1.1">𝑨</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px1.p1.2.m2.1c">\bm{A}</annotation></semantics></math>; (2) Sub-matrices extracted randomly from the 13B model; (3) Parameters extracted from the 13B model based on sensitivity; (4) Sub-matrices drawn randomly from the 30B model; (5) Parameters extracted from the 30B model leveraging sensitivity metric.</p>
</div>
<div id="S4.SS3.SSS0.Px1.p2" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS0.Px1.p2.4" class="ltx_p">Using LLaMA-1 7B as the student model, we present our findings in Figure <a href="#S4.F3" title="Figure 3 ‣ Initialization Strategies. ‣ 4.3 Analysis: Key Factors for Parametric Knowledge Transfer ‣ 4 Experiments ‣ Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. Initializing as per the original LoRA paper—but without zeroing out <math id="S4.SS3.SSS0.Px1.p2.1.m1.1" class="ltx_Math" alttext="\rm{B}\rm{A}" display="inline"><semantics id="S4.SS3.SSS0.Px1.p2.1.m1.1a"><mi id="S4.SS3.SSS0.Px1.p2.1.m1.1.1" xref="S4.SS3.SSS0.Px1.p2.1.m1.1.1.cmml">BA</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px1.p2.1.m1.1b"><ci id="S4.SS3.SSS0.Px1.p2.1.m1.1.1.cmml" xref="S4.SS3.SSS0.Px1.p2.1.m1.1.1">BA</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px1.p2.1.m1.1c">\rm{B}\rm{A}</annotation></semantics></math>—leads to a noticeable drop in performance. Recognizing the imperative of leveraging the original model’s weights as a starting point for fine-tuning the LoRA, our initialization strategy in this paper is rooted in Equation <a href="#S3.E10" title="In Knowledge Injection with LoRA. ‣ 3.3 Knowledge Injection ‣ 3 Parametric Knowledge Transfer ‣ Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a>; hence, we keep both <math id="S4.SS3.SSS0.Px1.p2.2.m2.1" class="ltx_Math" alttext="\bm{W}" display="inline"><semantics id="S4.SS3.SSS0.Px1.p2.2.m2.1a"><mi id="S4.SS3.SSS0.Px1.p2.2.m2.1.1" xref="S4.SS3.SSS0.Px1.p2.2.m2.1.1.cmml">𝑾</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px1.p2.2.m2.1b"><ci id="S4.SS3.SSS0.Px1.p2.2.m2.1.1.cmml" xref="S4.SS3.SSS0.Px1.p2.2.m2.1.1">𝑾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px1.p2.2.m2.1c">\bm{W}</annotation></semantics></math> and <math id="S4.SS3.SSS0.Px1.p2.3.m3.1" class="ltx_Math" alttext="\bm{W}_{\text{extract}}" display="inline"><semantics id="S4.SS3.SSS0.Px1.p2.3.m3.1a"><msub id="S4.SS3.SSS0.Px1.p2.3.m3.1.1" xref="S4.SS3.SSS0.Px1.p2.3.m3.1.1.cmml"><mi id="S4.SS3.SSS0.Px1.p2.3.m3.1.1.2" xref="S4.SS3.SSS0.Px1.p2.3.m3.1.1.2.cmml">𝑾</mi><mtext id="S4.SS3.SSS0.Px1.p2.3.m3.1.1.3" xref="S4.SS3.SSS0.Px1.p2.3.m3.1.1.3a.cmml">extract</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px1.p2.3.m3.1b"><apply id="S4.SS3.SSS0.Px1.p2.3.m3.1.1.cmml" xref="S4.SS3.SSS0.Px1.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S4.SS3.SSS0.Px1.p2.3.m3.1.1.1.cmml" xref="S4.SS3.SSS0.Px1.p2.3.m3.1.1">subscript</csymbol><ci id="S4.SS3.SSS0.Px1.p2.3.m3.1.1.2.cmml" xref="S4.SS3.SSS0.Px1.p2.3.m3.1.1.2">𝑾</ci><ci id="S4.SS3.SSS0.Px1.p2.3.m3.1.1.3a.cmml" xref="S4.SS3.SSS0.Px1.p2.3.m3.1.1.3"><mtext mathsize="70%" id="S4.SS3.SSS0.Px1.p2.3.m3.1.1.3.cmml" xref="S4.SS3.SSS0.Px1.p2.3.m3.1.1.3">extract</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px1.p2.3.m3.1c">\bm{W}_{\text{extract}}</annotation></semantics></math> fixed and solely fine-tune <math id="S4.SS3.SSS0.Px1.p2.4.m4.1" class="ltx_Math" alttext="\bm{B}\bm{A}" display="inline"><semantics id="S4.SS3.SSS0.Px1.p2.4.m4.1a"><mrow id="S4.SS3.SSS0.Px1.p2.4.m4.1.1" xref="S4.SS3.SSS0.Px1.p2.4.m4.1.1.cmml"><mi id="S4.SS3.SSS0.Px1.p2.4.m4.1.1.2" xref="S4.SS3.SSS0.Px1.p2.4.m4.1.1.2.cmml">𝑩</mi><mo lspace="0em" rspace="0em" id="S4.SS3.SSS0.Px1.p2.4.m4.1.1.1" xref="S4.SS3.SSS0.Px1.p2.4.m4.1.1.1.cmml">​</mo><mi id="S4.SS3.SSS0.Px1.p2.4.m4.1.1.3" xref="S4.SS3.SSS0.Px1.p2.4.m4.1.1.3.cmml">𝑨</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px1.p2.4.m4.1b"><apply id="S4.SS3.SSS0.Px1.p2.4.m4.1.1.cmml" xref="S4.SS3.SSS0.Px1.p2.4.m4.1.1"><times id="S4.SS3.SSS0.Px1.p2.4.m4.1.1.1.cmml" xref="S4.SS3.SSS0.Px1.p2.4.m4.1.1.1"></times><ci id="S4.SS3.SSS0.Px1.p2.4.m4.1.1.2.cmml" xref="S4.SS3.SSS0.Px1.p2.4.m4.1.1.2">𝑩</ci><ci id="S4.SS3.SSS0.Px1.p2.4.m4.1.1.3.cmml" xref="S4.SS3.SSS0.Px1.p2.4.m4.1.1.3">𝑨</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px1.p2.4.m4.1c">\bm{B}\bm{A}</annotation></semantics></math>. Moreover, our sensitivity-based method consistently outperforms both Gaussian initialization and random parameter extraction from teacher models across varying scales.</p>
</div>
</section>
<section id="S4.SS3.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Number of Seed Samples.</h4>

<figure id="S4.F5.sf1" class="ltx_figure">
</figure>
<div id="S4.SS3.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS0.Px2.p1.4" class="ltx_p">The quantity of seed samples plays a crucial role in determining both the reliability and efficiency of computing sensitivity scores from the teacher model. To delve deeper into its impact, we study how varying numbers of seed samples influence the performance of the student model. As evidenced in Figure <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:fig:transfer</span>, an augmentation in seed samples consistently mitigates variance, whilst the enhancement in performance remains relatively slight. The results demonstrate a tendency to stabilize after the application of 32 seed samples, prompting us to establish this as a hyperparameter in this paper. A further insight is the marked reduction in variance as the student’s scale is escalated (transitioning from 30B <math id="S4.SS3.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="\Rightarrow" display="inline"><semantics id="S4.SS3.SSS0.Px2.p1.1.m1.1a"><mo stretchy="false" id="S4.SS3.SSS0.Px2.p1.1.m1.1.1" xref="S4.SS3.SSS0.Px2.p1.1.m1.1.1.cmml">⇒</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px2.p1.1.m1.1b"><ci id="S4.SS3.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S4.SS3.SSS0.Px2.p1.1.m1.1.1">⇒</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px2.p1.1.m1.1c">\Rightarrow</annotation></semantics></math> 7B to 30B <math id="S4.SS3.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="\Rightarrow" display="inline"><semantics id="S4.SS3.SSS0.Px2.p1.2.m2.1a"><mo stretchy="false" id="S4.SS3.SSS0.Px2.p1.2.m2.1.1" xref="S4.SS3.SSS0.Px2.p1.2.m2.1.1.cmml">⇒</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px2.p1.2.m2.1b"><ci id="S4.SS3.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S4.SS3.SSS0.Px2.p1.2.m2.1.1">⇒</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px2.p1.2.m2.1c">\Rightarrow</annotation></semantics></math> 13B), or as the disparity between the teacher and student models is diminished (transitioning from 30B <math id="S4.SS3.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="\Rightarrow" display="inline"><semantics id="S4.SS3.SSS0.Px2.p1.3.m3.1a"><mo stretchy="false" id="S4.SS3.SSS0.Px2.p1.3.m3.1.1" xref="S4.SS3.SSS0.Px2.p1.3.m3.1.1.cmml">⇒</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px2.p1.3.m3.1b"><ci id="S4.SS3.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S4.SS3.SSS0.Px2.p1.3.m3.1.1">⇒</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px2.p1.3.m3.1c">\Rightarrow</annotation></semantics></math> 7B to 13B <math id="S4.SS3.SSS0.Px2.p1.4.m4.1" class="ltx_Math" alttext="\Rightarrow" display="inline"><semantics id="S4.SS3.SSS0.Px2.p1.4.m4.1a"><mo stretchy="false" id="S4.SS3.SSS0.Px2.p1.4.m4.1.1" xref="S4.SS3.SSS0.Px2.p1.4.m4.1.1.cmml">⇒</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.SSS0.Px2.p1.4.m4.1b"><ci id="S4.SS3.SSS0.Px2.p1.4.m4.1.1.cmml" xref="S4.SS3.SSS0.Px2.p1.4.m4.1.1">⇒</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.SSS0.Px2.p1.4.m4.1c">\Rightarrow</annotation></semantics></math> 7B).</p>
</div>
</section>
<section id="S4.SS3.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Layer Selection Methods.</h4>

<figure id="S4.F5.sf2" class="ltx_figure">
</figure>
<div id="S4.SS3.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS0.Px3.p1.1" class="ltx_p">Owing to the discrepancy in the number of layers between teacher and student models, the selection methodology for these layers potentially influences the final results. We evaluate four strategies: random layer selection, extracting the top or last layers, and a selection based on our sensitivity-centric technique. In the experiments, we consistently map teacher layers to student layers in their inherent sequential order. As Figure <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:fig:layer</span> illustrates, while the layer selection modestly affects student performance, our sensitivity-driven approach excels over the other strategies across both teacher-student model pairings.</p>
</div>
</section>
<section id="S4.SS3.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Origin of Extracted Parameters.</h4>

<div id="S4.SS3.SSS0.Px4.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS0.Px4.p1.1" class="ltx_p">The complex architecture of the Transformer raises inquiries about the most effective module for knowledge transfer. Our explorations involve the Embedding layer, the Feed-Forward Network (FFN), and the Self-Attention layer of the teacher model. As depicted in Figure <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:fig:location</span>, the embedding layer experiences inferior transfer effectiveness, likely due to its lesser parameter quantity. In contrast, the FFN showcases advanced transfer capabilities, intimating that it houses a significant share of the teacher’s knowledge. Optimal results are obtained when transferring knowledge from all available modules.</p>
</div>
</section>
<section id="S4.SS3.SSS0.Px5" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Structure of Extracted Parameters.</h4>

<div id="S4.SS3.SSS0.Px5.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.SSS0.Px5.p1.1" class="ltx_p">The necessity to reduce the parameter matrix’s size for knowledge transfer prompts questions regarding optimal population strategies for this matrix. We undertake a comparison across four methods: random neuron selection from the teacher model, and parameter extraction based on the highest sensitivity at the neuron, row and column, and submatrix levels. Figure <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:fig:structure</span> shows that maintaining the teacher model’s parameter structure significantly benefits student model performance. More precisely, transferring isolated neurons—either randomly or based on sensitivity—yields results comparable to those without knowledge transfer, highlighting the ineffectiveness of such transfers. Preserving the coherence of rows or columns provides a notable improvement, and the preservation of the submatrix structure further augments the performance gains derived from parametric knowledge transfer. This observation underpins our proposed knowledge extraction approach as outlined in Equation <a href="#S3.E5" title="In Layer Selection and Dimensionality Reduction. ‣ 3.2 Knowledge Extraction ‣ 3 Parametric Knowledge Transfer ‣ Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>.</p>
</div>
</section>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.4 </span>Visualization for Parametric Knowledge</h3>

<figure id="S4.F5.sf3" class="ltx_figure">
</figure>
<div id="S4.SS4.p1" class="ltx_para ltx_noindent">
<p id="S4.SS4.p1.1" class="ltx_p">In our exploration, we also attempt to visualize the parametric knowledge intrinsic to different task categories. MMLU is omitted from the set of tasks, given its encompassing knowledge from multiple domains, and we introduce code generation <cite class="ltx_cite ltx_citemacro_citet">Chaudhary (<a href="#bib.bib6" title="" class="ltx_ref">2023</a>)</cite> as an additional task for analysis. LLaMA-1 30B serves as the teacher model, and we base our findings on 32 randomly selected seed samples, illustrating the sensitivity scores layer by layer. During the visualization process, we subject each parameter matrix to min-max normalization, ensuring that sensitivity scores fall within the [0, 1] range. The insights from Figure <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:fig:visualization</span> reveal that the distribution of parametric knowledge across layers varies considerably among tasks. For instance, mathematical reasoning predominantly engages the bottom layer, instruction-driven NLP tasks concentrate on the bottom and middle layers, open-ended conversations are more centered around the middle and upper layers, while code generation appears to draw from all layers. This further emphasizes the efficacy of our sensitivity-based knowledge extraction method in pinpointing task-specific parametric knowledge, thereby aiding the subsequent transfer processes between diverse models.</p>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Conclusion</h2>

<div id="S5.p1" class="ltx_para ltx_noindent">
<p id="S5.p1.1" class="ltx_p">In this paper, we delve into the feasibility of transferring parametric knowledge between LLMs of varying scales, and present a new paradigm, exploring knowledge transfer from a distinct parametric perspective. Through our two-stage framework encompassing knowledge extraction and injection, we perform extensive experiments across four diverse benchmarks, affirming the inherent transferability of model parameters. Furthermore, by meticulously analyzing the key elements influencing parametric knowledge transfer, we aim to shed light on future research in this domain.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Aghajanyan et al. (2021)</span>
<span class="ltx_bibblock">
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer.

</span>
<span class="ltx_bibblock">Intrinsic dimensionality explains the effectiveness of language model fine-tuning.

</span>
<span class="ltx_bibblock">In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), <em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021</em>, pp.  7319–7328. Association for Computational Linguistics, 2021.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2021.acl-long.568</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.18653/v1/2021.acl-long.568" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.18653/v1/2021.acl-long.568</a>.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ainsworth et al. (2023)</span>
<span class="ltx_bibblock">
Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha S. Srinivasa.

</span>
<span class="ltx_bibblock">Git re-basin: Merging models modulo permutation symmetries.

</span>
<span class="ltx_bibblock">In <em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023</em>. OpenReview.net, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/pdf?id=CQsmMYmlP5T" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/pdf?id=CQsmMYmlP5T</a>.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bai et al. (2022)</span>
<span class="ltx_bibblock">
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan.

</span>
<span class="ltx_bibblock">Training a helpful and harmless assistant with reinforcement learning from human feedback.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2204.05862, 2022.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2204.05862</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2204.05862" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2204.05862</a>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al. (2020)</span>
<span class="ltx_bibblock">
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.

</span>
<span class="ltx_bibblock">Language models are few-shot learners.

</span>
<span class="ltx_bibblock">In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), <em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual</em>, 2020.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html</a>.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cao et al. (2021)</span>
<span class="ltx_bibblock">
Nicola De Cao, Wilker Aziz, and Ivan Titov.

</span>
<span class="ltx_bibblock">Editing factual knowledge in language models.

</span>
<span class="ltx_bibblock">In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), <em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021</em>, pp.  6491–6506. Association for Computational Linguistics, 2021.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2021.emnlp-main.522</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.18653/v1/2021.emnlp-main.522" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.18653/v1/2021.emnlp-main.522</a>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chaudhary (2023)</span>
<span class="ltx_bibblock">
Sahil Chaudhary.

</span>
<span class="ltx_bibblock">Code alpaca: An instruction-following llama model for code generation, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://github.com/sahil280114/codealpaca" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/sahil280114/codealpaca</a>.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2022)</span>
<span class="ltx_bibblock">
Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, and Qun Liu.

</span>
<span class="ltx_bibblock">bert2bert: Towards reusable pretrained language models.

</span>
<span class="ltx_bibblock">In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), <em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022</em>, pp.  2134–2148. Association for Computational Linguistics, 2022.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2022.acl-long.151</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.18653/v1/2022.acl-long.151" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.18653/v1/2022.acl-long.151</a>.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2016)</span>
<span class="ltx_bibblock">
Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens.

</span>
<span class="ltx_bibblock">Net2net: Accelerating learning via knowledge transfer.

</span>
<span class="ltx_bibblock">In Yoshua Bengio and Yann LeCun (eds.), <em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings</em>, 2016.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://arxiv.org/abs/1511.05641" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://arxiv.org/abs/1511.05641</a>.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chiang et al. (2023)</span>
<span class="ltx_bibblock">
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.

</span>
<span class="ltx_bibblock">Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.

</span>
<span class="ltx_bibblock">Blog post, March 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://lmsys.org/blog/2023-03-30-vicuna/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://lmsys.org/blog/2023-03-30-vicuna/</a>.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chowdhery et al. (2022)</span>
<span class="ltx_bibblock">
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.

</span>
<span class="ltx_bibblock">Palm: Scaling language modeling with pathways.

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2204.02311, 2022.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2204.02311</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2204.02311" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2204.02311</a>.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cobbe et al. (2021)</span>
<span class="ltx_bibblock">
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.

</span>
<span class="ltx_bibblock">Training verifiers to solve math word problems.

</span>
<span class="ltx_bibblock"><em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2110.14168, 2021.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2110.14168" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2110.14168</a>.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dai et al. (2022)</span>
<span class="ltx_bibblock">
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei.

</span>
<span class="ltx_bibblock">Knowledge neurons in pretrained transformers.

</span>
<span class="ltx_bibblock">In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), <em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022</em>, pp.  8493–8502. Association for Computational Linguistics, 2022.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2022.acl-long.581</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.18653/v1/2022.acl-long.581" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.18653/v1/2022.acl-long.581</a>.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Delétang et al. (2023)</span>
<span class="ltx_bibblock">
Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter, and Joel Veness.

</span>
<span class="ltx_bibblock">Language modeling is compression.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2309.10668, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2309.10668</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2309.10668" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2309.10668</a>.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Devlin et al. (2019)</span>
<span class="ltx_bibblock">
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.

</span>
<span class="ltx_bibblock">BERT: pre-training of deep bidirectional transformers for language understanding.

</span>
<span class="ltx_bibblock">In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), <em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)</em>, pp.  4171–4186. Association for Computational Linguistics, 2019.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/n19-1423</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.18653/v1/n19-1423" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.18653/v1/n19-1423</a>.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dubois et al. (2023)</span>
<span class="ltx_bibblock">
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.

</span>
<span class="ltx_bibblock">Alpacafarm: A simulation framework for methods that learn from human feedback.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2305.14387, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2305.14387</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2305.14387" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2305.14387</a>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Evci et al. (2022)</span>
<span class="ltx_bibblock">
Utku Evci, Bart van Merrienboer, Thomas Unterthiner, Fabian Pedregosa, and Max Vladymyrov.

</span>
<span class="ltx_bibblock">Gradmax: Growing neural networks using gradient information.

</span>
<span class="ltx_bibblock">In <em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022</em>. OpenReview.net, 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=qjN4h_wwUO" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=qjN4h_wwUO</a>.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Geng et al. (2023)</span>
<span class="ltx_bibblock">
Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song.

</span>
<span class="ltx_bibblock">Koala: A dialogue model for academic research.

</span>
<span class="ltx_bibblock">Blog post, April 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://bair.berkeley.edu/blog/2023/04/03/koala/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://bair.berkeley.edu/blog/2023/04/03/koala/</a>.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gesmundo &amp; Maile (2023)</span>
<span class="ltx_bibblock">
Andrea Gesmundo and Kaitlin Maile.

</span>
<span class="ltx_bibblock">Composable function-preserving expansions for transformer architectures.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2308.06103, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2308.06103</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2308.06103" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2308.06103</a>.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gou et al. (2021)</span>
<span class="ltx_bibblock">
Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao.

</span>
<span class="ltx_bibblock">Knowledge distillation: A survey.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">Int. J. Comput. Vis.</em>, 129(6):1789–1819, 2021.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1007/s11263-021-01453-z</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.1007/s11263-021-01453-z" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1007/s11263-021-01453-z</a>.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gu et al. (2021)</span>
<span class="ltx_bibblock">
Xiaotao Gu, Liyuan Liu, Hongkun Yu, Jing Li, Chen Chen, and Jiawei Han.

</span>
<span class="ltx_bibblock">On the transformer growth for progressive BERT training.

</span>
<span class="ltx_bibblock">In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), <em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021</em>, pp.  5174–5180. Association for Computational Linguistics, 2021.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2021.naacl-main.406</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.18653/v1/2021.naacl-main.406" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.18653/v1/2021.naacl-main.406</a>.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. (2021)</span>
<span class="ltx_bibblock">
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.

</span>
<span class="ltx_bibblock">Measuring massive multitask language understanding.

</span>
<span class="ltx_bibblock">In <em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021</em>. OpenReview.net, 2021.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=d7KBjmI3GmQ" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=d7KBjmI3GmQ</a>.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hinton et al. (2015)</span>
<span class="ltx_bibblock">
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean.

</span>
<span class="ltx_bibblock">Distilling the knowledge in a neural network.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/1503.02531, 2015.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://arxiv.org/abs/1503.02531" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://arxiv.org/abs/1503.02531</a>.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Honovich et al. (2023)</span>
<span class="ltx_bibblock">
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick.

</span>
<span class="ltx_bibblock">Unnatural instructions: Tuning language models with (almost) no human labor.

</span>
<span class="ltx_bibblock">In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), <em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023</em>, pp.  14409–14428. Association for Computational Linguistics, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2023.acl-long.806</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.18653/v1/2023.acl-long.806" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.18653/v1/2023.acl-long.806</a>.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hu et al. (2022)</span>
<span class="ltx_bibblock">
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.

</span>
<span class="ltx_bibblock">Lora: Low-rank adaptation of large language models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022</em>. OpenReview.net, 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=nZeVKeeFYf9" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=nZeVKeeFYf9</a>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Huang et al. (2017)</span>
<span class="ltx_bibblock">
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, and Kilian Q. Weinberger.

</span>
<span class="ltx_bibblock">Snapshot ensembles: Train 1, get M for free.

</span>
<span class="ltx_bibblock">In <em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings</em>. OpenReview.net, 2017.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=BJYwwY9ll" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=BJYwwY9ll</a>.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ilharco et al. (2023)</span>
<span class="ltx_bibblock">
Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi.

</span>
<span class="ltx_bibblock">Editing models with task arithmetic.

</span>
<span class="ltx_bibblock">In <em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023</em>. OpenReview.net, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/pdf?id=6t0Kwf8-jrj" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/pdf?id=6t0Kwf8-jrj</a>.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Izmailov et al. (2018)</span>
<span class="ltx_bibblock">
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson.

</span>
<span class="ltx_bibblock">Averaging weights leads to wider optima and better generalization.

</span>
<span class="ltx_bibblock">In Amir Globerson and Ricardo Silva (eds.), <em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018</em>, pp.  876–885. AUAI Press, 2018.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://auai.org/uai2018/proceedings/papers/313.pdf" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://auai.org/uai2018/proceedings/papers/313.pdf</a>.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al. (2020)</span>
<span class="ltx_bibblock">
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig.

</span>
<span class="ltx_bibblock">How can we know what language models know.

</span>
<span class="ltx_bibblock"><em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">Trans. Assoc. Comput. Linguistics</em>, 8:423–438, 2020.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1162/tacl“˙a“˙00324</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.1162/tacl_a_00324" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1162/tacl_a_00324</a>.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jin et al. (2019)</span>
<span class="ltx_bibblock">
Xiao Jin, Baoyun Peng, Yichao Wu, Yu Liu, Jiaheng Liu, Ding Liang, Junjie Yan, and Xiaolin Hu.

</span>
<span class="ltx_bibblock">Knowledge distillation via route constrained optimization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019</em>, pp.  1345–1354. IEEE, 2019.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1109/ICCV.2019.00143</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.1109/ICCV.2019.00143" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1109/ICCV.2019.00143</a>.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Köpf et al. (2023)</span>
<span class="ltx_bibblock">
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick.

</span>
<span class="ltx_bibblock">Openassistant conversations - democratizing large language model alignment.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2304.07327, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2304.07327</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2304.07327" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2304.07327</a>.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lan et al. (2018)</span>
<span class="ltx_bibblock">
Xu Lan, Xiatian Zhu, and Shaogang Gong.

</span>
<span class="ltx_bibblock">Knowledge distillation by on-the-fly native ensemble.

</span>
<span class="ltx_bibblock">In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), <em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada</em>, pp.  7528–7538, 2018.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.neurips.cc/paper/2018/hash/94ef7214c4a90790186e255304f8fd1f-Abstract.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.neurips.cc/paper/2018/hash/94ef7214c4a90790186e255304f8fd1f-Abstract.html</a>.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lee et al. (2019)</span>
<span class="ltx_bibblock">
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr.

</span>
<span class="ltx_bibblock">Snip: single-shot network pruning based on connection sensitivity.

</span>
<span class="ltx_bibblock">In <em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019</em>. OpenReview.net, 2019.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=B1VZqjAcYX" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=B1VZqjAcYX</a>.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2023)</span>
<span class="ltx_bibblock">
Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Xuying Meng, Siqi Fan, Peng Han, Jing Li, Li Du, Bowen Qin, Zheng Zhang, Aixin Sun, and Yequan Wang.

</span>
<span class="ltx_bibblock">FLM-101B: an open LLM and how to train it with $100k budget.

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2309.03852, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2309.03852</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2309.03852" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2309.03852</a>.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liang et al. (2022)</span>
<span class="ltx_bibblock">
Chen Liang, Haoming Jiang, Simiao Zuo, Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Tuo Zhao.

</span>
<span class="ltx_bibblock">No parameters left behind: Sensitivity guided adaptive learning rate for training large transformer models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022</em>. OpenReview.net, 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=cuvga_CiVND" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=cuvga_CiVND</a>.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lubana &amp; Dick (2021)</span>
<span class="ltx_bibblock">
Ekdeep Singh Lubana and Robert P. Dick.

</span>
<span class="ltx_bibblock">A gradient flow framework for analyzing network pruning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021</em>. OpenReview.net, 2021.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=rumv7QmLUue" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=rumv7QmLUue</a>.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Meng et al. (2022)</span>
<span class="ltx_bibblock">
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov.

</span>
<span class="ltx_bibblock">Locating and editing factual associations in GPT.

</span>
<span class="ltx_bibblock">In <em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>, 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://papers.nips.cc/paper_files/paper/2022/hash/6f1d43d5a82a37e89b0665b33bf3a182-Abstract-Conference.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://papers.nips.cc/paper_files/paper/2022/hash/6f1d43d5a82a37e89b0665b33bf3a182-Abstract-Conference.html</a>.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Meng et al. (2023)</span>
<span class="ltx_bibblock">
Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau.

</span>
<span class="ltx_bibblock">Mass-editing memory in a transformer.

</span>
<span class="ltx_bibblock">In <em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023</em>. OpenReview.net, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/pdf?id=MkbcAHIYgyS" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/pdf?id=MkbcAHIYgyS</a>.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mirzadeh et al. (2020)</span>
<span class="ltx_bibblock">
Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh.

</span>
<span class="ltx_bibblock">Improved knowledge distillation via teacher assistant.

</span>
<span class="ltx_bibblock">In <em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020</em>, pp.  5191–5198. AAAI Press, 2020.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1609/aaai.v34i04.5963</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.1609/aaai.v34i04.5963" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1609/aaai.v34i04.5963</a>.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mitchell et al. (2022)</span>
<span class="ltx_bibblock">
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn.

</span>
<span class="ltx_bibblock">Memory-based model editing at scale.

</span>
<span class="ltx_bibblock">In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), <em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA</em>, volume 162 of <em id="bib.bib39.2.2" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning Research</em>, pp.  15817–15831. PMLR, 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.mlr.press/v162/mitchell22a.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.mlr.press/v162/mitchell22a.html</a>.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Molchanov et al. (2017)</span>
<span class="ltx_bibblock">
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz.

</span>
<span class="ltx_bibblock">Pruning convolutional neural networks for resource efficient inference.

</span>
<span class="ltx_bibblock">In <em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings</em>. OpenReview.net, 2017.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=SJGCiw5gl" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=SJGCiw5gl</a>.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mozer &amp; Smolensky (1988)</span>
<span class="ltx_bibblock">
Michael C Mozer and Paul Smolensky.

</span>
<span class="ltx_bibblock">Skeletonization: A technique for trimming the fat from a network via relevance assessment.

</span>
<span class="ltx_bibblock"><em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>, 1, 1988.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.neurips.cc/paper_files/paper/1988/hash/07e1cd7dca89a1678042477183b7ac3f-Abstract.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.neurips.cc/paper_files/paper/1988/hash/07e1cd7dca89a1678042477183b7ac3f-Abstract.html</a>.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">OpenAI (2023)</span>
<span class="ltx_bibblock">
OpenAI.

</span>
<span class="ltx_bibblock">GPT-4 technical report.

</span>
<span class="ltx_bibblock"><em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2303.08774, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2303.08774</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2303.08774" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2303.08774</a>.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ortiz-Jiménez et al. (2023)</span>
<span class="ltx_bibblock">
Guillermo Ortiz-Jiménez, Alessandro Favero, and Pascal Frossard.

</span>
<span class="ltx_bibblock">Task arithmetic in the tangent space: Improved editing of pre-trained models.

</span>
<span class="ltx_bibblock"><em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2305.12827, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2305.12827</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2305.12827" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2305.12827</a>.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Park et al. (2021)</span>
<span class="ltx_bibblock">
Dae Young Park, Moon-Hyun Cha, Changwook Jeong, Daesin Kim, and Bohyung Han.

</span>
<span class="ltx_bibblock">Learning student-friendly teacher networks for knowledge distillation.

</span>
<span class="ltx_bibblock">In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), <em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual</em>, pp.  13292–13303, 2021.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.neurips.cc/paper/2021/hash/6e7d2da6d3953058db75714ac400b584-Abstract.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.neurips.cc/paper/2021/hash/6e7d2da6d3953058db75714ac400b584-Abstract.html</a>.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Peng et al. (2023)</span>
<span class="ltx_bibblock">
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao.

</span>
<span class="ltx_bibblock">Instruction tuning with GPT-4.

</span>
<span class="ltx_bibblock"><em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2304.03277, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2304.03277</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2304.03277" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2304.03277</a>.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Peters et al. (2018)</span>
<span class="ltx_bibblock">
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.

</span>
<span class="ltx_bibblock">Deep contextualized word representations.

</span>
<span class="ltx_bibblock">In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), <em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers)</em>, pp.  2227–2237. Association for Computational Linguistics, 2018.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/n18-1202</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.18653/v1/n18-1202" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.18653/v1/n18-1202</a>.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Petroni et al. (2019)</span>
<span class="ltx_bibblock">
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller.

</span>
<span class="ltx_bibblock">Language models as knowledge bases?

</span>
<span class="ltx_bibblock">In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), <em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019</em>, pp.  2463–2473. Association for Computational Linguistics, 2019.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/D19-1250</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.18653/v1/D19-1250" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.18653/v1/D19-1250</a>.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pham et al. (2021)</span>
<span class="ltx_bibblock">
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V. Le.

</span>
<span class="ltx_bibblock">Meta pseudo labels.

</span>
<span class="ltx_bibblock">In <em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021</em>, pp.  11557–11568. Computer Vision Foundation / IEEE, 2021.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1109/CVPR46437.2021.01139</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openaccess.thecvf.com/content/CVPR2021/html/Pham_Meta_Pseudo_Labels_CVPR_2021_paper.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openaccess.thecvf.com/content/CVPR2021/html/Pham_Meta_Pseudo_Labels_CVPR_2021_paper.html</a>.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Qin et al. (2022)</span>
<span class="ltx_bibblock">
Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou.

</span>
<span class="ltx_bibblock">Knowledge inheritance for pre-trained language models.

</span>
<span class="ltx_bibblock">In Marine Carpuat, Marie-Catherine de Marneffe, and Iván Vladimir Meza Ruíz (eds.), <em id="bib.bib49.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022</em>, pp.  3921–3937. Association for Computational Linguistics, 2022.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2022.naacl-main.288</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.18653/v1/2022.naacl-main.288" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.18653/v1/2022.naacl-main.288</a>.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(50)</span>
<span class="ltx_bibblock">
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.

</span>
<span class="ltx_bibblock">Improving language understanding by generative pre-training.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf</a>.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Roberts et al. (2020)</span>
<span class="ltx_bibblock">
Adam Roberts, Colin Raffel, and Noam Shazeer.

</span>
<span class="ltx_bibblock">How much knowledge can you pack into the parameters of a language model?

</span>
<span class="ltx_bibblock">In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), <em id="bib.bib51.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020</em>, pp.  5418–5426. Association for Computational Linguistics, 2020.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2020.emnlp-main.437</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.18653/v1/2020.emnlp-main.437" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.18653/v1/2020.emnlp-main.437</a>.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sanh et al. (2019)</span>
<span class="ltx_bibblock">
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.

</span>
<span class="ltx_bibblock">Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter.

</span>
<span class="ltx_bibblock"><em id="bib.bib52.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/1910.01108, 2019.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://arxiv.org/abs/1910.01108" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://arxiv.org/abs/1910.01108</a>.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shen et al. (2022)</span>
<span class="ltx_bibblock">
Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew E. Peters, and Iz Beltagy.

</span>
<span class="ltx_bibblock">Staged training for transformer language models.

</span>
<span class="ltx_bibblock">In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), <em id="bib.bib53.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA</em>, volume 162 of <em id="bib.bib53.2.2" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning Research</em>, pp.  19893–19908. PMLR, 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.mlr.press/v162/shen22f.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.mlr.press/v162/shen22f.html</a>.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Stoica et al. (2023)</span>
<span class="ltx_bibblock">
George Stoica, Daniel Bolya, Jakob Bjorner, Taylor Hearn, and Judy Hoffman.

</span>
<span class="ltx_bibblock">Zipit! merging models from different tasks without training.

</span>
<span class="ltx_bibblock"><em id="bib.bib54.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2305.03053, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2305.03053</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2305.03053" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2305.03053</a>.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Taori et al. (2023)</span>
<span class="ltx_bibblock">
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.

</span>
<span class="ltx_bibblock">Stanford alpaca: An instruction-following llama model.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/tatsu-lab/stanford_alpaca" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/tatsu-lab/stanford_alpaca</a>, 2023.

</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023a)</span>
<span class="ltx_bibblock">
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.

</span>
<span class="ltx_bibblock">Llama: Open and efficient foundation language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib56.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2302.13971, 2023a.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2302.13971</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2302.13971" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2302.13971</a>.

</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023b)</span>
<span class="ltx_bibblock">
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov,
and Thomas Scialom.

</span>
<span class="ltx_bibblock">Llama 2: Open foundation and fine-tuned chat models.

</span>
<span class="ltx_bibblock"><em id="bib.bib57.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2307.09288, 2023b.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2307.09288</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2307.09288" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2307.09288</a>.

</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vaswani et al. (2017)</span>
<span class="ltx_bibblock">
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.

</span>
<span class="ltx_bibblock">Attention is all you need.

</span>
<span class="ltx_bibblock">In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), <em id="bib.bib58.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA</em>, pp.  5998–6008, 2017.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html</a>.

</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023a)</span>
<span class="ltx_bibblock">
Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogério Feris, David Daniel Cox, Zhangyang Wang, and Yoon Kim.

</span>
<span class="ltx_bibblock">Learning to grow pretrained models for efficient transformer training.

</span>
<span class="ltx_bibblock">In <em id="bib.bib59.1.1" class="ltx_emph ltx_font_italic">The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023</em>. OpenReview.net, 2023a.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/pdf?id=cDYRS5iZ16f" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/pdf?id=cDYRS5iZ16f</a>.

</span>
</li>
<li id="bib.bib60" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2022)</span>
<span class="ltx_bibblock">
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen.

</span>
<span class="ltx_bibblock">Super-naturalinstructions: Generalization via declarative instructions on 1600+ NLP tasks.

</span>
<span class="ltx_bibblock">In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), <em id="bib.bib60.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022</em>, pp.  5085–5109. Association for Computational Linguistics, 2022.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2022.emnlp-main.340</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.18653/v1/2022.emnlp-main.340" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.18653/v1/2022.emnlp-main.340</a>.

</span>
</li>
<li id="bib.bib61" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023b)</span>
<span class="ltx_bibblock">
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi.

</span>
<span class="ltx_bibblock">How far can camels go? exploring the state of instruction tuning on open resources.

</span>
<span class="ltx_bibblock"><em id="bib.bib61.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2306.04751, 2023b.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2306.04751</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2306.04751" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2306.04751</a>.

</span>
</li>
<li id="bib.bib62" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023c)</span>
<span class="ltx_bibblock">
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.

</span>
<span class="ltx_bibblock">Self-instruct: Aligning language models with self-generated instructions.

</span>
<span class="ltx_bibblock">In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), <em id="bib.bib62.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023</em>, pp.  13484–13508. Association for Computational Linguistics, 2023c.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2023.acl-long.754</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.18653/v1/2023.acl-long.754" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.18653/v1/2023.acl-long.754</a>.

</span>
</li>
<li id="bib.bib63" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. (2022)</span>
<span class="ltx_bibblock">
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou.

</span>
<span class="ltx_bibblock">Chain-of-thought prompting elicits reasoning in large language models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib63.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>, 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://papers.nips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://papers.nips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html</a>.

</span>
</li>
<li id="bib.bib64" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. (2016)</span>
<span class="ltx_bibblock">
Tao Wei, Changhu Wang, Yong Rui, and Chang Wen Chen.

</span>
<span class="ltx_bibblock">Network morphism.

</span>
<span class="ltx_bibblock">In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), <em id="bib.bib64.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016</em>, volume 48 of <em id="bib.bib64.2.2" class="ltx_emph ltx_font_italic">JMLR Workshop and Conference Proceedings</em>, pp.  564–572. JMLR.org, 2016.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://proceedings.mlr.press/v48/wei16.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://proceedings.mlr.press/v48/wei16.html</a>.

</span>
</li>
<li id="bib.bib65" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wu et al. (2023)</span>
<span class="ltx_bibblock">
Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, and Alham Fikri Aji.

</span>
<span class="ltx_bibblock">Lamini-lm: A diverse herd of distilled models from large-scale instructions.

</span>
<span class="ltx_bibblock"><em id="bib.bib65.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2304.14402, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2304.14402</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2304.14402" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2304.14402</a>.

</span>
</li>
<li id="bib.bib66" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al. (2023)</span>
<span class="ltx_bibblock">
Canwen Xu, Daya Guo, Nan Duan, and Julian J. McAuley.

</span>
<span class="ltx_bibblock">Baize: An open-source chat model with parameter-efficient tuning on self-chat data.

</span>
<span class="ltx_bibblock"><em id="bib.bib66.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2304.01196, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2304.01196</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2304.01196" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2304.01196</a>.

</span>
</li>
<li id="bib.bib67" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yao et al. (2023a)</span>
<span class="ltx_bibblock">
Yiqun Yao, Zheng Zhang, Jing Li, and Yequan Wang.

</span>
<span class="ltx_bibblock">2x faster language model pre-training via masked structural growth.

</span>
<span class="ltx_bibblock"><em id="bib.bib67.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2305.02869, 2023a.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2305.02869</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2305.02869" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2305.02869</a>.

</span>
</li>
<li id="bib.bib68" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yao et al. (2023b)</span>
<span class="ltx_bibblock">
Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang.

</span>
<span class="ltx_bibblock">Editing large language models: Problems, methods, and opportunities.

</span>
<span class="ltx_bibblock"><em id="bib.bib68.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2305.13172, 2023b.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2305.13172</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2305.13172" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2305.13172</a>.

</span>
</li>
<li id="bib.bib69" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2023)</span>
<span class="ltx_bibblock">
Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He.

</span>
<span class="ltx_bibblock">Composing parameter-efficient modules with arithmetic operations.

</span>
<span class="ltx_bibblock"><em id="bib.bib69.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2306.14870, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2306.14870</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2306.14870" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2306.14870</a>.

</span>
</li>
<li id="bib.bib70" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2018)</span>
<span class="ltx_bibblock">
Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu.

</span>
<span class="ltx_bibblock">Deep mutual learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib70.1.1" class="ltx_emph ltx_font_italic">2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018</em>, pp.  4320–4328. Computer Vision Foundation / IEEE Computer Society, 2018.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1109/CVPR.2018.00454</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://openaccess.thecvf.com/content_cvpr_2018/html/Zhang_Deep_Mutual_Learning_CVPR_2018_paper.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://openaccess.thecvf.com/content_cvpr_2018/html/Zhang_Deep_Mutual_Learning_CVPR_2018_paper.html</a>.

</span>
</li>
<li id="bib.bib71" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. (2023)</span>
<span class="ltx_bibblock">
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy.

</span>
<span class="ltx_bibblock">LIMA: less is more for alignment.

</span>
<span class="ltx_bibblock"><em id="bib.bib71.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2305.11206, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.48550/arXiv.2305.11206</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2305.11206" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2305.11206</a>.

</span>
</li>
<li id="bib.bib72" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. (2022)</span>
<span class="ltx_bibblock">
Wangchunshu Zhou, Canwen Xu, and Julian J. McAuley.

</span>
<span class="ltx_bibblock">BERT learns to teach: Knowledge distillation with meta learning.

</span>
<span class="ltx_bibblock">In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), <em id="bib.bib72.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022</em>, pp.  7037–7049. Association for Computational Linguistics, 2022.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2022.acl-long.485</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.18653/v1/2022.acl-long.485" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.18653/v1/2022.acl-long.485</a>.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2310.11450" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2310.11451" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2310.11451">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2310.11451" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2310.11452" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Wed Feb 28 00:15:36 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2310.11451
Large Language Models (LLMs)  inherently encode a wealth of knowledge within their parameters through pre-training on extensive corpora. While prior research has delved into operations on these parameters to manipulate the underlying implicit knowledge—encompassing detection, editing, and merging—there remains an ambiguous understanding regarding their transferability across models with varying scales. In this paper, we seek to empirically investigate knowledge transfer from larger to smaller models through a parametric perspective. To achieve this, we employ sensitivity-based techniques to extract and align knowledge-specific parameters between different LLMs. Moreover, the LoRA module is used as the intermediary mechanism for injecting the extracted knowledge into smaller models. Evaluations across four benchmarks validate the efficacy of our proposed method. Our findings highlight the critical factors contributing to the process of parametric knowledge transfer, underscoring the transferability of model parameters across LLMs of different scales. We release code and data athttps://github.com/maszhongming/ParaKnowTransfer.
abstract
Seeking Neural Nuggets: KnowledgeTransfer in Large Language Models from a Parametric Perspective
Abstract
2310.11451
1IntroductionDriven by the advancements of Large Language Models (LLMs) , a transformative wave has reshaped the landscape in multiple areas of Artificial Intelligence, elevating performance across diverse tasks.
From a parametric perspective, the objective of pre-training is to encode substantial amounts of knowledge into model parameters through language modeling on extensive corpora.
In a quest to unravel the intricate workings of LLMs, a multitude of research efforts have been directed toward the detailed exploration and nuanced manipulation of this reservoir of implicit knowledge.Early research efforts sought to detect this parametric knowledge, typically probing the concrete facts by using the “fill-in-the-blank” task under a closed-book setting.
Subsequent studies delved into the feasibility of executing operations on model knowledge, including knowledge editing, a technique designed to modify targeted knowledge while preserving the integrity of the remaining information, and model merging, a strategy that combines multiple models to enhance robustness or facilitate multitasking abilities.
While these investigations exhibited that such parametric knowledge is bothdetectableandeditablewithin a single model, the broader question of whether it istransferableacross different LLMs remains an open and under-explored topic.Knowledge transfer refers to distilling the expertise of larger teacher models into smaller, more manageable counterparts, thereby democratizing access to cutting-edge machine learning capabilities.
As illustrated in Figure1, online and offline distillation currently stand as the primary paradigms.
The former, especially prevalent before the LLM era, capitalizes on teacher models to guide the learning trajectory of student models.
Yet, as LLMs grow in scale, the inherent demand for the teacher model to undergo fine-tuning or participate in student training becomes increasingly cost-prohibitive.
In contrast, offline distillation calls upon the teacher model merely to generate answers to open-ended queries, creating a distilled training dataset for students.
Despite reducing the overhead to thousands of inferences, it completely overlooks the rich knowledge implicitly stored within the teacher’s parameters.In this paper, we empirically investigate knowledge transfer from a distinct parametric perspective, dedicated to selecting static parameters directly from the teacher model and exploring their transferability.
Specifically, we introduce a new parametric knowledge transfer paradigm designed to extract task-specific parameters from the teacher model and subsequently inject them into the student model, thereby enhancing performance on downstream tasks.
Through decoding on a limited set of seed samples (e.g., 32 samples)  with the teacher model, we calculate sensitivity metrics that serve as the basis for knowledge extraction.
Given the discrepancies in the number of layers and dimensions across varied LLM scales, we employ sensitivity-based layer mapping and dimensionality reduction techniques to establish alignment between the teacher and student models.
Lastly, we leverage LoRAas a bridge to inject these extracted parameters into student models, facilitating its fine-tuning on downstream tasks and thus achieving the knowledge transfer process.Figure 1:Different paradigms of knowledge transfer from teacher models to student models. (a)  Online Distillation: utilizing soft logits from the fine-tuned teacher model to guide the training of the student model; (b)  Offline Distillation: generating a distilled dataset that encapsulates the knowledge of the teacher model to fine-tune the student model. (c)  Parametric Knowledge Transfer: extracting knowledge-specific parameters from the vanilla teacher model and injecting them into the student model to enhance its training efficacy.Experimentally, we evaluate the parametric knowledge transfer framework across four benchmark categories: reasoning, professional knowledge, instruction-driven NLP tasks, and open-ended conversation, using various sizes of LLaMA models.
The results indicate that upon transferring the teacher model’s parameters, the student performance demonstrates consistent improvements across all benchmarks, affirming the transferability of parametric knowledge.
Furthermore, our detailed analyses illustrate the underlying factors that contribute to effective parametric knowledge transfer, discovering that the teacher scales, initialization strategies, number of seed samples, and the origin and structure of the extracted parameters all play crucial roles.To summarize, the key contributions of this paper are threefold: (1)  From a distinct perspective, we introduce a parametric knowledge transfer paradigm that encompasses stages of sensitivity-based knowledge extraction and LoRA-driven knowledge injection. (2)  Through comprehensive evaluations, we provide empirical evidence that implicit model knowledge is indeedtransferableacross varying scales of LLMs. (3)  Further enriching our insights into parametric knowledge transfer, we undertake a thorough analysis to pinpoint the pivotal factors that dictate its efficacy.
introduction
Seeking Neural Nuggets: KnowledgeTransfer in Large Language Models from a Parametric Perspective
1Introduction
2310.11451
2Related Work2.1Manipulation of Implicit Model KnowledgeWith the recognition of the vast repository of knowledge embedded in model parameters, ensuing research has sought to execute diverse operations on these parameters, aiming to manipulate the implicit knowledge.
For instance, knowledge editing endeavors to modify or update specific facts by editing the associated parameters, all the while ensuring the broader knowledge base remains untouched.
Another avenue, model merging, combines the weights of two or more models into a unified weight set for greater robustness or multitasking capabilities.
Additionally, a strand of studies probes into performing arithmetic operations on the pre-trained weights of a model or specific modules, thus enabling the model to augment or diminish particular functionalities.
However, these explorations are limited to operations within individual models or the merging of models with identical architectures, without investigating whether implicit knowledge between different scale models can be manipulated and transferred.2.2Inheritance of Model KnowledgeAnother line of research that aligns more closely with our work concerns the operation of model parameters across scales, specifically the concept of model growth.
This refers to accelerating the pre-training of LLMs by incrementally growing and expanding the parameters of a smaller model, using them as an initialization for the larger one.
The majority of existing work is concentrated on devising function-preserving methods, ensuring that the initialized larger model replicates the behaviors of the original smaller model.
Concurrently, several studies adopt data-driven strategies, investigating reverse distillationor mapping learned weights from smaller models to their larger counterparts.
In contrast to this research direction, our emphasis is on the transfer of knowledge from larger teacher to smaller student models, with the aim of exploring not only the efficiency of training, but also the transferability of parametric knowledge across different scenarios.2.3Transfer of Model KnowledgeKnowledge transfer is a research area dedicated to training a smaller student model to mimic the behavior of a larger pre-trained teacher model to achieve similar performance with fewer parameters.
Despite progress in improving the online distillation paradigmand optimizing the efficiency of offline distillation, they both completely ignore the implicit knowledge embedded inherently in the teacher model.
In this paper, we are grounded in a fresh perspective and propose a parametric knowledge transfer framework. It seeks to affirm the transferability of such implicit knowledge by directly extracting parameters from the teacher model and injecting them into the student model. Furthermore, by leveraging the static parameters of the teacher model, our approach naturally holds an efficiency advantage over preceding methodologies.
body
Seeking Neural Nuggets: KnowledgeTransfer in Large Language Models from a Parametric Perspective
2Related Work
2310.11451
3Parametric Knowledge TransferIn this section, we first outline the task formulation for parametric knowledge transfer. Following this, we delve into a detailed description of our proposed method, as depicted in Figure2.3.1Task FormulationThe core objective of parametric knowledge transfer is to enhance a student model by selectively transferring task-specific parametric knowledge from a more knowledgeable teacher model. Given a task, the transfer process begins with a teacher modelendowed with parametersand a student modelcharacterized by parameters.The first step in this procedure involves extraction, where task-relevant parameters are identified from the teacher model and resized to a desired scale based on the student model’s parameter dimensions. This can be expressed as:(1) withencapsulating the logic for both parameter extraction and downscaling.
Following extraction, the next step is the injection of these extracted parameters into the student model:(2) yielding a student model now characterized by the modified parameter set.
Upon completing the knowledge injection, there remains an optional phase wherein the student model fine-tunes the newly incorporated parameterswith respect to the task.Figure 2:Overview of our parametric knowledge transfer framework. Starting with the teacher model, we compute sensitivity metrics using a set of seed samples, which aids in the extraction of task-specific knowledge. Subsequently, the extracted parameter matrices are factorized to initialize the student model’s LoRA module, serving as a bridge for knowledge injection.3.2Knowledge ExtractionIn implementing ourfunction, we employ parameter sensitivity as the foundational metric to guide the extraction process.Sensitivity of the Parameters.Parameter sensitivity serves as a mechanism to measure the variation in the loss upon setting a particular parameter to zero. When this removal elicits a significant shift in the loss, such a parameter is deemed to be of high sensitivity. For a teacher modelwith parameters, whererepresents the total number of parameters, the-th parameter can be expressed as. With gradients of the loss relative torepresented as, the sensitivity of the-th parameter for a specific samplefrom taskis determined as:(3) The rationale behind this sensitivity definition stems from the first-order Taylor expansion ofrelative toat. In essence,provides an approximation for how the loss might change in the absence of:(4) To ascertainfor parameterpertaining to task, we randomly sampleinstances as seed samples for an efficient and representative estimate. Thus, the final formulationfor taskintegrates the cumulative sensitivity over the sampled instances, calculated as.Layer Selection and Dimensionality Reduction.Given that models of varying scales often differ in both the number of layers and their dimensions, we adopt a method of layer selection and dimensionality reduction based on sensitivity scores. Our first step is to assess the layers of the teacher model,, with respect to their relevance to task. Letandrepresent the total number of layers in the teacher and student models, respectively, with. For each layerin, we calculate a layer-specific sensitivity score,, by aggregating the sensitivity scores of all parameters within that layer, represented as:Having computed the sensitivity scores for layerin, we proceed to arrange them in descending order and select the toplayers with the highest scores. To preserve the inherent structure of the teacher model, the chosen layers are subsequently mapped to the student model maintaining their original sequential order.Upon alignment of the layers, the parameter dimensions of each layer in the teacher model typically continue to surpass those of the student model. During this phase of the transfer process, our focus is primarily on all the two-dimensional matrices in the teacher model, which are denoted as.
To preserve the structural information in eachwith the dimensions, we extract the submatrix with the highest cumulative sensitivity to align with the student model’s corresponding matrix with dimensions() .
Formally, this objective is expressed as:(5) By aggregatingacross all layers, we derive the extracted parametersfrom.3.3Knowledge InjectionTo keep the architecture and the number of parameters of the student model unchanged during the knowledge transfer process, we employ the LoRA approach to instantiate thefunction.LoRA Module.LoRA, which stands for Low-Rank Adaptation, is a method designed to optimize parameter efficiency by freezing the pre-trained model weights and inserting trainable rank decomposition matrices into the deep neural network. The guiding intuition is that pre-trained language models possess low “intrinsic dimensions”. This means that even when projected to a smaller subspace, these models can still exhibit efficient learning. Consequently, it can be hypothesized that weight updates during adaptation also exhibit low “intrinsic ranks”. For a given pre-trained weight matrix, it can be updated as:(6) whererepresents the low rank with. The matrixremains constant during this operation, implying that onlyandare updated in the training phase. To ensure that training commences from the original pre-trained weights, eitheroris initialized with zeros.Knowledge Injection with LoRA.The main goal of this step is to integrate knowledge from the teacher model by incorporating extracted parameters into the student’s LoRA module. Initially, SVD is adopted to factorize each matrixininto three constituent matrices as:(7) Here,andare orthogonal matrices containing left and right singular vectors, respectively, whileis a diagonal matrix that hosts the singular values in descending order. To capture the firstranks, we then formulate:(8) The symbolsandrepresent the initialcolumns ofand, respectively, whilecaptures the topsingular values. Subsequently, for each corresponding matrixin the student model, its update can be given as in two ways:(9) (10) whereis initialized using, andis initialized with. During training, bothandremain constant, with only the parameters inbeing updated.
The former is the initialization from LoRA paper, while the latter ensures that training starts from the original weights of.
We provide an empirical comparison of the two initialization strategies in Section4.3.
Here, the introduction of LoRA module aims to effectively utilize the most salient features of the extracted knowledge in a concise and efficient manner.
body
Seeking Neural Nuggets: KnowledgeTransfer in Large Language Models from a Parametric Perspective
3Parametric Knowledge Transfer
2310.11451
4Experiments4.1Experimental SetupFacets of Evaluation.To validate the efficacy of our proposed framework, we conduct evaluations across four distinct benchmark categories:(1) Reasoning:Reasoning stands as a foundational capability for models, particularly when tackling intricate tasks. We leverage the Grade School Math dataset (GSM) to assess the reasoning proficiency of models. The evaluation format requires models, given a math problem, to produce the chain-of-thought processand the final numerical answer.(2) Professional Knowledge:For language models to effectively cater to users’ informational needs, possessing a robust repository of professional knowledge is crucial. We measure this knowledge reservoir using the Massive Multitask Language Understanding dataset (MMLU) . This dataset encompasses questions about 57 subjects, spanning a spectrum of difficulty levels from elementary to advanced professional tiers, all formatted as multiple-choice questions.(3) Instruction-driven NLP Tasks:This set of tasks evaluates a model’s capability in adhering to instructions. Typically, the language model receives both a task definition and an input text, and it must perform the specified classification or generation tasks as directed. Our chosen benchmark for this category is the Super Natural Instructions (Super NI) , a rich dataset comprising 1,616 varied NLP tasks alongside their expert-written instructions.(4) Open-ended Conversation:This represents the primary interface through which models interact with users in real-world scenarios. To evaluate such instructability, we employ AlpacaFarm, which contains 805 instructions including subsets from various evaluations like Self-Instruct, Open Assistant, Anthropic, Vicuna, and Koala. GPT4-32K serves as the evaluator, determining the win rate of the testing model against the outputs generated by Davinci-003.Throughout all evaluations, we adhere to established metrics and prompts, utilizing the evaluation scripts sourced from.Implementation Details.For all our experiments, the larger-scale LLaMA modelserves as the teacher, and its smaller-scale counterpart acts as the student. For the fine-tuning of the student model, we draw a random subset of 1,000 instances from the respective training datasets of each benchmark. In the case of AlpacaFarm, due to the absence of a training set, we utilize LIMA dataas a substitute, which is composed of 1,000 carefully curated open-ended conversations. For each experiment, 32 seed samples are randomly selected from the corresponding training sets. The student model is trained for 3 epochs with a batch size of 64 and a learning rate of 3e-4. Regarding LoRA, we set the rank as 16, and insert LoRA module into the embedding layer, FFN, and self-attention layer in the Transformer architecture. Notably, all results presented in this paper are mean values derived from three separate runs, with each run using a new random set of seed samples.4.2Experimental ResultsTable 1:Results for parametric knowledge transfer. “7B-LoRA + 13B Param.” represents that we extract parameters from the 13B teacher model and transfer them to the 7B student model.ModelsGSMMMLUSuper NIAlpacaFarmAverage0-shot8-shot0-shot5-shotEMR-LWin Rate%-LLaMA-1Vanilla 7B4.7010.7732.1035.300.675.55--7B-LoRA17.2616.9343.4338.9022.9140.499.0727.00+ 13B Param.18.7318.8544.0339.7724.5142.379.2828.22+ 30B Param.18.6318.5245.2040.6025.0143.089.4028.63Vanilla 13B4.9317.4443.5046.802.187.78--13B-LoRA26.1823.7850.4350.0327.3445.5313.9133.89+ 30B Param.27.8527.7051.3051.0327.5146.0917.2735.54LLaMA-2Vanilla 7B3.3415.5441.7045.800.004.68--Vanilla 13B6.5227.8252.1055.200.004.84--7B-LoRA23.3821.0547.7747.0724.9341.2520.5032.28+ 13B Param.25.3026.3149.3746.5326.1642.9824.6434.47Results for Parametric Knowledge Transfer.Our initial experiments focus on four distinct teacher-student model pairings: LLaMA-1 (13B7B, 30B7B, 30B13B)  and LLaMA-2 (13B7B) . The outcomes are systematically presented in Table1. Remarkably, in contrast to the direct fine-tuning approach of LoRA, the student models augmented with parametric knowledge from their respective teacher models exhibit substantial improvements across all four benchmark categories. For instance, the LLaMA-1 30B7B pairing yields an average performance boost of 6.04% (from 27.00 to 28.63) . In a similar vein, the LLaMA-2 13B7B configuration brings an enhancement of 6.78% (from 32.28 to 34.47) .Another observation emerges when examining the effects of scaling up the teacher model, specifically transitioning from 13B to 30B. The performance of the student model, LLaMA-1 7B, generally sees an improvement, despite a slight decrement in the GSM benchmark. Beyond the evident performance gains, the overhead introduced by parametric knowledge transfer remains minimal. The only extra commitment involves the teacher model executing inference on a set of 32 seed samples, without any direct participation in the training. Considering both performance and efficiency, parametric knowledge transfer stands out as a practical technique, even as disparities in parameter counts and architectural variances between teacher and student models expand.Table 2:Transfer experiments with different task-specific extracted parameters. The leftmost column indicates the dataset on which the knowledge extraction is based. The teacher model and student model are LLaMA-2 13B and 7B, respectively.ModelsGSMMMLUSuper NIAlpacaFarmAverage0-shot8-shot0-shot5-shotEMR-LWin Rate%-Vanilla 7B3.3415.5441.7045.800.004.68--7B-LoRA23.3821.0547.7747.0724.9341.2520.5032.28GSM25.3026.3148.4045.9724.4542.1123.6833.75MMLU24.1125.4749.3746.5325.5542.5524.0133.94Super NI23.7824.1148.6046.7026.1642.9824.3133.81LIMA24.0825.6049.0347.2325.6342.8324.6434.15Transfer Experiments with Task-specific Extracted Parameters.While our results indicate that transferring extracted knowledge from the teacher model positively influences student model performance, the nature of this improvement—whether it is rooted in generalized knowledge or task-specific expertise—warrants deeper exploration. To disentangle this, we conduct experiments wherein extracted parameters, each tailored to a specific task, are integrated into the student model, which is subsequently fine-tuned across all datasets.Table2offers insights into a prevalent trend: when parameters are extracted from a concrete task, the performance is most significantly amplified for that same task. This is particularly evident in the GSM benchmark. Models equipped with GSM-oriented extracted parameters notably exceed their counterparts—achieving at least a 1.2 increase in 0-shot accuracy—compared to models incorporated with parameters based on alternative datasets. This is likely due to the unique and intricate challenges associated with mathematical reasoning. Additionally, parameters sourced from the LIMA dataset demonstrate remarkable generalizability, presumably owing to their grounding in open-ended dialogues that cover a spectrum of domains and tasks. Overall, these observations highlight the capability of our sensitivity-driven techniques to efficiently target certain types of knowledge, rather than just extracting generalized knowledge.4.3Analysis: Key Factors for Parametric Knowledge TransferWe further analyze the key factors for the process of parametric knowledge transfer as follows.Initialization Strategies.Figure 3:Comparison of different initialization strategies. The y-axis represents the average score over four datasets. For, we employ five methods to initialize, keepingconstant. In the case of, these methods correspond to bothand, withremaining static during training. The orange dotted line denotes the results of initialization as described in LoRA paper.Essentially, for the student model, the injected parameters correspond to the initialization of LoRA module. We compare two initialization methods: the approach described in the LoRA paper, denoted by Equation9, and the method adopted in our work, represented by Equation10. Furthermore, we investigate five distinct initializations for LoRA matrix: (1)  Gaussian initialization for bothand; (2)  Sub-matrices extracted randomly from the 13B model; (3)  Parameters extracted from the 13B model based on sensitivity; (4)  Sub-matrices drawn randomly from the 30B model; (5)  Parameters extracted from the 30B model leveraging sensitivity metric.Using LLaMA-1 7B as the student model, we present our findings in Figure3. Initializing as per the original LoRA paper—but without zeroing out—leads to a noticeable drop in performance. Recognizing the imperative of leveraging the original model’s weights as a starting point for fine-tuning the LoRA, our initialization strategy in this paper is rooted in Equation10; hence, we keep bothandfixed and solely fine-tune. Moreover, our sensitivity-based method consistently outperforms both Gaussian initialization and random parameter extraction from teacher models across varying scales.Number of Seed Samples.The quantity of seed samples plays a crucial role in determining both the reliability and efficiency of computing sensitivity scores from the teacher model. To delve deeper into its impact, we study how varying numbers of seed samples influence the performance of the student model. As evidenced in FigureLABEL:fig:transfer, an augmentation in seed samples consistently mitigates variance, whilst the enhancement in performance remains relatively slight. The results demonstrate a tendency to stabilize after the application of 32 seed samples, prompting us to establish this as a hyperparameter in this paper. A further insight is the marked reduction in variance as the student’s scale is escalated (transitioning from 30B7B to 30B13B) , or as the disparity between the teacher and student models is diminished (transitioning from 30B7B to 13B7B) .Layer Selection Methods.Owing to the discrepancy in the number of layers between teacher and student models, the selection methodology for these layers potentially influences the final results. We evaluate four strategies: random layer selection, extracting the top or last layers, and a selection based on our sensitivity-centric technique. In the experiments, we consistently map teacher layers to student layers in their inherent sequential order. As FigureLABEL:fig:layerillustrates, while the layer selection modestly affects student performance, our sensitivity-driven approach excels over the other strategies across both teacher-student model pairings.Origin of Extracted Parameters.The complex architecture of the Transformer raises inquiries about the most effective module for knowledge transfer. Our explorations involve the Embedding layer, the Feed-Forward Network (FFN) , and the Self-Attention layer of the teacher model. As depicted in FigureLABEL:fig:location, the embedding layer experiences inferior transfer effectiveness, likely due to its lesser parameter quantity. In contrast, the FFN showcases advanced transfer capabilities, intimating that it houses a significant share of the teacher’s knowledge. Optimal results are obtained when transferring knowledge from all available modules.Structure of Extracted Parameters.The necessity to reduce the parameter matrix’s size for knowledge transfer prompts questions regarding optimal population strategies for this matrix. We undertake a comparison across four methods: random neuron selection from the teacher model, and parameter extraction based on the highest sensitivity at the neuron, row and column, and submatrix levels. FigureLABEL:fig:structureshows that maintaining the teacher model’s parameter structure significantly benefits student model performance. More precisely, transferring isolated neurons—either randomly or based on sensitivity—yields results comparable to those without knowledge transfer, highlighting the ineffectiveness of such transfers. Preserving the coherence of rows or columns provides a notable improvement, and the preservation of the submatrix structure further augments the performance gains derived from parametric knowledge transfer. This observation underpins our proposed knowledge extraction approach as outlined in Equation5.4.4Visualization for Parametric KnowledgeIn our exploration, we also attempt to visualize the parametric knowledge intrinsic to different task categories. MMLU is omitted from the set of tasks, given its encompassing knowledge from multiple domains, and we introduce code generationas an additional task for analysis. LLaMA-1 30B serves as the teacher model, and we base our findings on 32 randomly selected seed samples, illustrating the sensitivity scores layer by layer. During the visualization process, we subject each parameter matrix to min-max normalization, ensuring that sensitivity scores fall within the [0, 1] range. The insights from FigureLABEL:fig:visualizationreveal that the distribution of parametric knowledge across layers varies considerably among tasks. For instance, mathematical reasoning predominantly engages the bottom layer, instruction-driven NLP tasks concentrate on the bottom and middle layers, open-ended conversations are more centered around the middle and upper layers, while code generation appears to draw from all layers. This further emphasizes the efficacy of our sensitivity-based knowledge extraction method in pinpointing task-specific parametric knowledge, thereby aiding the subsequent transfer processes between diverse models.
body
Seeking Neural Nuggets: KnowledgeTransfer in Large Language Models from a Parametric Perspective
4Experiments
2310.11451
5ConclusionIn this paper, we delve into the feasibility of transferring parametric knowledge between LLMs of varying scales, and present a new paradigm, exploring knowledge transfer from a distinct parametric perspective. Through our two-stage framework encompassing knowledge extraction and injection, we perform extensive experiments across four diverse benchmarks, affirming the inherent transferability of model parameters. Furthermore, by meticulously analyzing the key elements influencing parametric knowledge transfer, we aim to shed light on future research in this domain.
conclusion
Seeking Neural Nuggets: KnowledgeTransfer in Large Language Models from a Parametric Perspective
5Conclusion