<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2407.06677] Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules</title><meta property="og:description" content="Is it always necessary to compute tokens from shallow to deep layers in Transformers? The continued success of vanilla Transformers and their variants suggests an undoubted “yes”. In this work, however, we attempt to b…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2407.06677">

<!--Generated on Mon Aug  5 17:22:10 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Mixture-of-Modules: 
<br class="ltx_break">Reinventing Transformers as Dynamic Assemblies of Modules</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Zhuocheng Gong<sup id="id1.1.id1" class="ltx_sup">1</sup><span id="footnotex1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>Equal Contributions.</span></span></span> Ang Lv<sup id="id2.2.id2" class="ltx_sup">2</sup><span id="footnotex2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>Equal Contributions.</span></span></span> Jian Guan<sup id="id3.3.id3" class="ltx_sup">3,4</sup><span id="footnotex3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>Corresponding authors.</span></span></span> Junxi Yan<sup id="id4.4.id4" class="ltx_sup">3</sup> Wei Wu<sup id="id5.5.id5" class="ltx_sup">4</sup> Huishuai Zhang<sup id="id6.6.id6" class="ltx_sup">1</sup>
<br class="ltx_break"><span id="id7.7.id7" class="ltx_text ltx_font_bold">Minlie Huang<sup id="id7.7.id7.1" class="ltx_sup">3</sup> Dongyan Zhao<sup id="id7.7.id7.2" class="ltx_sup">1</sup><span id="footnotex4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note"><span id="footnotex4.1.1.1" class="ltx_text ltx_font_medium">2</span></span>Corresponding authors.</span></span></span> Rui Yan<sup id="id7.7.id7.3" class="ltx_sup">2</sup><span id="footnotex5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note"><span id="footnotex5.1.1.1" class="ltx_text ltx_font_medium">2</span></span>Corresponding authors.</span></span></span>
<br class="ltx_break"><sup id="id7.7.id7.4" class="ltx_sup"><span id="id7.7.id7.4.1" class="ltx_text ltx_font_medium">1</span></sup></span>Peking University  <sup id="id8.8.id8" class="ltx_sup">2</sup>Renmin University  <sup id="id9.9.id9" class="ltx_sup">3</sup>Tsinghua University  <sup id="id10.10.id10" class="ltx_sup">4</sup>Ant Group 
<br class="ltx_break"><span id="id11.11.id11" class="ltx_text ltx_font_typewriter">{gzhch,zhanghuishuai,zhaody}@pku.edu.cn</span>, <span id="id12.12.id12" class="ltx_text ltx_font_typewriter">{anglv,ruiyan}@ruc.edu.cn
<br class="ltx_break">{j-guan19,yanjx21}@mails.tsinghua.edu.cn</span>, <span id="id13.13.id13" class="ltx_text ltx_font_typewriter">aihuang@tsinghua.edu.cn
<br class="ltx_break">congyue.ww@antgroup.com
<br class="ltx_break"></span>
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id14.id1" class="ltx_p">Is it always necessary to compute tokens from shallow to deep layers in Transformers? The continued success of vanilla Transformers and their variants suggests an undoubted “yes”. In this work, however, we attempt to break the depth-ordered convention by proposing a novel architecture dubbed mixture-of-modules (MoM), which is motivated by an intuition that any layer, regardless of its position, can be used to compute a token as long as it possesses the needed processing capabilities.
The construction of MoM starts from a finite set of modules defined by multi-head attention and feed-forward networks, each distinguished by its unique parameterization. Two routers then iteratively select attention modules and feed-forward modules from the set to process a token. The selection dynamically expands the computation graph in the forward pass of the token, culminating in an assembly of modules.
We show that MoM provides not only a unified framework for Transformers and their numerous variants but also a flexible and learnable approach for reducing redundancy in Transformer parameterization.
We pre-train various MoMs using OpenWebText. Empirical results demonstrate that MoMs, of different parameter counts, consistently outperform vanilla transformers on both GLUE and XSUM benchmarks.
More interestingly, with a fixed parameter budget, MoM-large enables an over 38% increase in depth for computation graphs compared to GPT-2-large, resulting in absolute gains of 1.4 on GLUE and 1 on XSUM. On the other hand, MoM-large also enables an over 60% reduction in depth while involving more modules per layer, yielding a 16% reduction in TFLOPs and a 43% decrease in memory usage compared to GPT-2-large, while maintaining comparable performance. <span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>Code is available at <a target="_blank" href="https://github.com/gzhch/MoM" title="" class="ltx_ref ltx_href">https://github.com/gzhch/MoM</a></span></span></span></p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para ltx_noindent">
<p id="S1.p1.1" class="ltx_p">Transformer-based language models <cite class="ltx_cite ltx_citemacro_citep">(Vaswani et al., <a href="#bib.bib44" title="" class="ltx_ref">2017</a>)</cite> have demonstrated remarkable abilities across a wide range of challenging natural language tasks <cite class="ltx_cite ltx_citemacro_citep">(Bubeck et al., <a href="#bib.bib5" title="" class="ltx_ref">2023</a>)</cite>.
In addition, the success of Transformer in natural language processing (NLP) is also inspiring innovations in other fields such as computer vision <cite class="ltx_cite ltx_citemacro_citep">(Peebles &amp; Xie, <a href="#bib.bib33" title="" class="ltx_ref">2023</a>; Agostinelli et al., <a href="#bib.bib1" title="" class="ltx_ref">2023</a>)</cite> and biomedicine <cite class="ltx_cite ltx_citemacro_citep">(Singhal et al., <a href="#bib.bib41" title="" class="ltx_ref">2023</a>; Madani et al., <a href="#bib.bib27" title="" class="ltx_ref">2023</a>)</cite>.
A Transformer architecture typically consists of stacked layers that are identical in structure, whereby layers are organized in the order of depth, using the output of the previous layer as the input for the next. While this design convention has been widely accepted as a matter of course in the Transformer era, we challenge it by reconsidering whether the static and depth-ordered organization can fully unleash the potential of Transformers, given the well-known issues of over-parameterization <cite class="ltx_cite ltx_citemacro_citep">(Zeng et al., <a href="#bib.bib50" title="" class="ltx_ref">2023</a>)</cite> and efficiency <cite class="ltx_cite ltx_citemacro_citep">(Raposo et al., <a href="#bib.bib37" title="" class="ltx_ref">2024</a>)</cite>.</p>
</div>
<div id="S1.p2" class="ltx_para ltx_noindent">
<p id="S1.p2.1" class="ltx_p">Before us, some rudimentary studies have touched on the question–they dissect Transformer into modules such as attention heads and feed-forward networks (<span id="S1.p2.1.1" class="ltx_text ltx_font_typewriter">FFN</span>s) and allow relatively flexible module call order.
For example, Mixture-of-Experts (MoE) <cite class="ltx_cite ltx_citemacro_citep">(Shazeer et al., <a href="#bib.bib40" title="" class="ltx_ref">2017</a>)</cite>) sets up multiple <span id="S1.p2.1.2" class="ltx_text ltx_font_typewriter">FFN</span>s within the same layer and activates a specific subset during inference.
Early-exiting <cite class="ltx_cite ltx_citemacro_citep">(Zhou et al., <a href="#bib.bib51" title="" class="ltx_ref">2020</a>; Xin et al., <a href="#bib.bib48" title="" class="ltx_ref">2020</a>; Schuster et al., <a href="#bib.bib38" title="" class="ltx_ref">2022</a>)</cite> and Mixture-of-Depths (MoD) <cite class="ltx_cite ltx_citemacro_citep">(Raposo et al., <a href="#bib.bib37" title="" class="ltx_ref">2024</a>)</cite>) bypass certain layers when computing each token. On the one hand, these efforts indeed lead to improvements in terms of either efficacy or efficiency through the introduction of dynamic mechanisms into the vanilla structure of Transformers, and thus corroborate our questioning regarding the established convention; on the other hand, they still follow the depth-ordered paradigm (i.e., tokens are passed from shallow layers to deep layers), leaving significant room for better architectures.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2407.06677/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="415" height="138" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>Mixture-of-Modules reinvents Transformers as dynamic assemblies of modules.
In (b), we illustrate the ongoing construction of an MoM model during the forward computation.
The assembly lasts <math id="S1.F1.9.m1.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S1.F1.9.m1.1b"><mi id="S1.F1.9.m1.1.1" xref="S1.F1.9.m1.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S1.F1.9.m1.1c"><ci id="S1.F1.9.m1.1.1.cmml" xref="S1.F1.9.m1.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.9.m1.1d">H</annotation></semantics></math> rounds, with the current illustration showcasing progress in the third round.
For each token, routers select the best <math id="S1.F1.10.m2.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S1.F1.10.m2.1b"><mi id="S1.F1.10.m2.1.1" xref="S1.F1.10.m2.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S1.F1.10.m2.1c"><ci id="S1.F1.10.m2.1.1.cmml" xref="S1.F1.10.m2.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.10.m2.1d">K</annotation></semantics></math> attention modules, denoted as <math id="S1.F1.11.m3.1" class="ltx_Math" alttext="m^{\text{A}}_{k}" display="inline"><semantics id="S1.F1.11.m3.1b"><msubsup id="S1.F1.11.m3.1.1" xref="S1.F1.11.m3.1.1.cmml"><mi id="S1.F1.11.m3.1.1.2.2" xref="S1.F1.11.m3.1.1.2.2.cmml">m</mi><mi id="S1.F1.11.m3.1.1.3" xref="S1.F1.11.m3.1.1.3.cmml">k</mi><mtext id="S1.F1.11.m3.1.1.2.3" xref="S1.F1.11.m3.1.1.2.3a.cmml">A</mtext></msubsup><annotation-xml encoding="MathML-Content" id="S1.F1.11.m3.1c"><apply id="S1.F1.11.m3.1.1.cmml" xref="S1.F1.11.m3.1.1"><csymbol cd="ambiguous" id="S1.F1.11.m3.1.1.1.cmml" xref="S1.F1.11.m3.1.1">subscript</csymbol><apply id="S1.F1.11.m3.1.1.2.cmml" xref="S1.F1.11.m3.1.1"><csymbol cd="ambiguous" id="S1.F1.11.m3.1.1.2.1.cmml" xref="S1.F1.11.m3.1.1">superscript</csymbol><ci id="S1.F1.11.m3.1.1.2.2.cmml" xref="S1.F1.11.m3.1.1.2.2">𝑚</ci><ci id="S1.F1.11.m3.1.1.2.3a.cmml" xref="S1.F1.11.m3.1.1.2.3"><mtext mathsize="70%" id="S1.F1.11.m3.1.1.2.3.cmml" xref="S1.F1.11.m3.1.1.2.3">A</mtext></ci></apply><ci id="S1.F1.11.m3.1.1.3.cmml" xref="S1.F1.11.m3.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.11.m3.1d">m^{\text{A}}_{k}</annotation></semantics></math>, and the best <math id="S1.F1.12.m4.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S1.F1.12.m4.1b"><mi id="S1.F1.12.m4.1.1" xref="S1.F1.12.m4.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S1.F1.12.m4.1c"><ci id="S1.F1.12.m4.1.1.cmml" xref="S1.F1.12.m4.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.12.m4.1d">K</annotation></semantics></math> feed-forward network modules, denoted as <math id="S1.F1.13.m5.1" class="ltx_Math" alttext="m^{\text{F}}_{k}" display="inline"><semantics id="S1.F1.13.m5.1b"><msubsup id="S1.F1.13.m5.1.1" xref="S1.F1.13.m5.1.1.cmml"><mi id="S1.F1.13.m5.1.1.2.2" xref="S1.F1.13.m5.1.1.2.2.cmml">m</mi><mi id="S1.F1.13.m5.1.1.3" xref="S1.F1.13.m5.1.1.3.cmml">k</mi><mtext id="S1.F1.13.m5.1.1.2.3" xref="S1.F1.13.m5.1.1.2.3a.cmml">F</mtext></msubsup><annotation-xml encoding="MathML-Content" id="S1.F1.13.m5.1c"><apply id="S1.F1.13.m5.1.1.cmml" xref="S1.F1.13.m5.1.1"><csymbol cd="ambiguous" id="S1.F1.13.m5.1.1.1.cmml" xref="S1.F1.13.m5.1.1">subscript</csymbol><apply id="S1.F1.13.m5.1.1.2.cmml" xref="S1.F1.13.m5.1.1"><csymbol cd="ambiguous" id="S1.F1.13.m5.1.1.2.1.cmml" xref="S1.F1.13.m5.1.1">superscript</csymbol><ci id="S1.F1.13.m5.1.1.2.2.cmml" xref="S1.F1.13.m5.1.1.2.2">𝑚</ci><ci id="S1.F1.13.m5.1.1.2.3a.cmml" xref="S1.F1.13.m5.1.1.2.3"><mtext mathsize="70%" id="S1.F1.13.m5.1.1.2.3.cmml" xref="S1.F1.13.m5.1.1.2.3">F</mtext></ci></apply><ci id="S1.F1.13.m5.1.1.3.cmml" xref="S1.F1.13.m5.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.13.m5.1d">m^{\text{F}}_{k}</annotation></semantics></math>, from a module set <math id="S1.F1.14.m6.1" class="ltx_Math" alttext="\mathcal{M}" display="inline"><semantics id="S1.F1.14.m6.1b"><mi class="ltx_font_mathcaligraphic" id="S1.F1.14.m6.1.1" xref="S1.F1.14.m6.1.1.cmml">ℳ</mi><annotation-xml encoding="MathML-Content" id="S1.F1.14.m6.1c"><ci id="S1.F1.14.m6.1.1.cmml" xref="S1.F1.14.m6.1.1">ℳ</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.14.m6.1d">\mathcal{M}</annotation></semantics></math> (including “<span id="S1.F1.18.1" class="ltx_text ltx_font_typewriter">SKIP</span>” modules).
These selected modules collectively constitute assembled modules <math id="S1.F1.15.m7.1" class="ltx_Math" alttext="\mathcal{F}^{\text{A}}" display="inline"><semantics id="S1.F1.15.m7.1b"><msup id="S1.F1.15.m7.1.1" xref="S1.F1.15.m7.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S1.F1.15.m7.1.1.2" xref="S1.F1.15.m7.1.1.2.cmml">ℱ</mi><mtext id="S1.F1.15.m7.1.1.3" xref="S1.F1.15.m7.1.1.3a.cmml">A</mtext></msup><annotation-xml encoding="MathML-Content" id="S1.F1.15.m7.1c"><apply id="S1.F1.15.m7.1.1.cmml" xref="S1.F1.15.m7.1.1"><csymbol cd="ambiguous" id="S1.F1.15.m7.1.1.1.cmml" xref="S1.F1.15.m7.1.1">superscript</csymbol><ci id="S1.F1.15.m7.1.1.2.cmml" xref="S1.F1.15.m7.1.1.2">ℱ</ci><ci id="S1.F1.15.m7.1.1.3a.cmml" xref="S1.F1.15.m7.1.1.3"><mtext mathsize="70%" id="S1.F1.15.m7.1.1.3.cmml" xref="S1.F1.15.m7.1.1.3">A</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.15.m7.1d">\mathcal{F}^{\text{A}}</annotation></semantics></math> and <math id="S1.F1.16.m8.1" class="ltx_Math" alttext="\mathcal{F}^{\text{F}}" display="inline"><semantics id="S1.F1.16.m8.1b"><msup id="S1.F1.16.m8.1.1" xref="S1.F1.16.m8.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S1.F1.16.m8.1.1.2" xref="S1.F1.16.m8.1.1.2.cmml">ℱ</mi><mtext id="S1.F1.16.m8.1.1.3" xref="S1.F1.16.m8.1.1.3a.cmml">F</mtext></msup><annotation-xml encoding="MathML-Content" id="S1.F1.16.m8.1c"><apply id="S1.F1.16.m8.1.1.cmml" xref="S1.F1.16.m8.1.1"><csymbol cd="ambiguous" id="S1.F1.16.m8.1.1.1.cmml" xref="S1.F1.16.m8.1.1">superscript</csymbol><ci id="S1.F1.16.m8.1.1.2.cmml" xref="S1.F1.16.m8.1.1.2">ℱ</ci><ci id="S1.F1.16.m8.1.1.3a.cmml" xref="S1.F1.16.m8.1.1.3"><mtext mathsize="70%" id="S1.F1.16.m8.1.1.3.cmml" xref="S1.F1.16.m8.1.1.3">F</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.16.m8.1d">\mathcal{F}^{\text{F}}</annotation></semantics></math>, which are then appended to the existing computation graph. Detailed notations are presented in §<a href="#S3" title="3 Methodology ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>.</figcaption>
</figure>
<div id="S1.p3" class="ltx_para ltx_noindent">
<p id="S1.p3.1" class="ltx_p">In this work, we completely disrupt the traditional practice in the design of Transformers by breaking down the depth-ordered organization. Numerous studies have indicated that knowledge in Transformers is often dispersed across multiple <span id="S1.p3.1.1" class="ltx_text ltx_font_typewriter">FFN</span>s in different layers <cite class="ltx_cite ltx_citemacro_citep">(Geva et al., <a href="#bib.bib16" title="" class="ltx_ref">2021</a>; McGrath et al., <a href="#bib.bib28" title="" class="ltx_ref">2023</a>; Lv et al., <a href="#bib.bib26" title="" class="ltx_ref">2024</a>)</cite>, and many attention heads serve similar or identical functions, such as copying specific token information towards the end position of the input <cite class="ltx_cite ltx_citemacro_citep">(Olsson et al., <a href="#bib.bib32" title="" class="ltx_ref">2022</a>; Wang et al., <a href="#bib.bib46" title="" class="ltx_ref">2023</a>)</cite>.
Encouraged by this evidence, we pose the question of whether the computation of a token can “move” freely across layers, that is the token can be computed by flowing to modules in deeper layers, sticking to modules of the same layer, or even going back to modules in previous layers.
To answer the question, we propose a novel architecture dubbed Mixture-of-Modules (MoM) in which the core idea is to define a neural network as dynamic assemblies of modules derived from vanilla Transformer, as depicted in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.</p>
</div>
<div id="S1.p4" class="ltx_para ltx_noindent">
<p id="S1.p4.1" class="ltx_p">The basis of MoM is a finite set of modules.
Each module is defined by multi-head attention (<span id="S1.p4.1.1" class="ltx_text ltx_font_typewriter">MHA</span>), a feed-forward network (<span id="S1.p4.1.2" class="ltx_text ltx_font_typewriter">FFN</span>) (including Add &amp; Norm), or a specialized module labeled “<span id="S1.p4.1.3" class="ltx_text ltx_font_typewriter">SKIP</span>”.
Each <span id="S1.p4.1.4" class="ltx_text ltx_font_typewriter">MHA</span> or <span id="S1.p4.1.5" class="ltx_text ltx_font_typewriter">FFN</span> module is identical in structure and different in parameterization.
<span id="S1.p4.1.6" class="ltx_text ltx_font_typewriter">SKIP</span> enables skip operations for arbitrary tokens at arbitrary time steps.
Given a token, each time two routers select modules from the set and integrate the modules into the computation graph during the forward pass.
Hence, the whole computation graph of the token is formed as an assembly of modules, and the routers learn to optimize the organization of the modules in the assembly.
We introduce a two-phase approach for training MoM models. In the first phase, we pre-train a vanilla Transformer on a large-scale corpus. Then, in the second phase, we decompose the pre-trained Transformer into modules as a warm-up of MoM, randomly initialize the routers, and further update both the modules and the routers under the mechanism of dynamic assembly. By this means, we can both enhance parameter utilization and accelerate the convergence of the model.</p>
</div>
<div id="S1.p5" class="ltx_para ltx_noindent">
<p id="S1.p5.1" class="ltx_p">MoM has three major advantages over existing Transformer-based architectures: (1) it provides a unified framework for various Transformer variants, incorporating popular methods such as mixture-of-experts, early-exiting, and mixture-of-depths as special cases.
The framework sheds light on architecture design for future works; (2) it brings unprecedented flexibility in forward computation. With the dynamic assembly mechanism, “depth” and “parameter count” are no longer entangled as they are in the conventional sense. One can build powerful architectures by either enlarging the module pool (i.e., increasing parameter count) or increasing the depth (parameter count can be fixed).
Hence, MoM offers a dynamic and learnable approach to reducing redundant parameters in Transformers; and (3) it offers efficient structures that achieve performance comparable to vanilla Transformers but require significantly fewer FLOPs and less memory in forward computation.</p>
</div>
<div id="S1.p6" class="ltx_para ltx_noindent">
<p id="S1.p6.2" class="ltx_p">We pre-train MoM in three sizes–122M (small), 346M (medium), and 774M (large)– using OpenWebText <cite class="ltx_cite ltx_citemacro_citep">(Gokaslan &amp; Cohen, <a href="#bib.bib17" title="" class="ltx_ref">2019</a>)</cite>, and assess their performance with GLUE <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib45" title="" class="ltx_ref">2018a</a>)</cite> and XSUM <cite class="ltx_cite ltx_citemacro_citep">(Narayan et al., <a href="#bib.bib30" title="" class="ltx_ref">2018a</a>)</cite>. Empirical results indicate that (1) MoMs, across all the three parameter counts, consistently outperform vanilla GPT-2 models on both text understanding and generation tasks;
(2) parameters are quite redundant in vanilla Transformers. One can develop an MoM that is at least 30% deeper than a vanilla GPT-2, resulting in at least 1.4 absolute gain in GLUE and at least 1 absolute gain in XSUM. Moreover, one can further remove <math id="S1.p6.1.m1.1" class="ltx_Math" alttext="50" display="inline"><semantics id="S1.p6.1.m1.1a"><mn id="S1.p6.1.m1.1.1" xref="S1.p6.1.m1.1.1.cmml">50</mn><annotation-xml encoding="MathML-Content" id="S1.p6.1.m1.1b"><cn type="integer" id="S1.p6.1.m1.1.1.cmml" xref="S1.p6.1.m1.1.1">50</cn></annotation-xml><annotation encoding="application/x-tex" id="S1.p6.1.m1.1c">50</annotation></semantics></math>% of the <span id="S1.p6.2.1" class="ltx_text ltx_font_typewriter">MHA</span> modules and <math id="S1.p6.2.m2.1" class="ltx_Math" alttext="25" display="inline"><semantics id="S1.p6.2.m2.1a"><mn id="S1.p6.2.m2.1.1" xref="S1.p6.2.m2.1.1.cmml">25</mn><annotation-xml encoding="MathML-Content" id="S1.p6.2.m2.1b"><cn type="integer" id="S1.p6.2.m2.1.1.cmml" xref="S1.p6.2.m2.1.1">25</cn></annotation-xml><annotation encoding="application/x-tex" id="S1.p6.2.m2.1c">25</annotation></semantics></math>% of the <span id="S1.p6.2.2" class="ltx_text ltx_font_typewriter">FFN</span> modules from the module pool of the model, while maintain comparable performance<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>After removing <math id="footnote2.m1.1" class="ltx_Math" alttext="50" display="inline"><semantics id="footnote2.m1.1b"><mn id="footnote2.m1.1.1" xref="footnote2.m1.1.1.cmml">50</mn><annotation-xml encoding="MathML-Content" id="footnote2.m1.1c"><cn type="integer" id="footnote2.m1.1.1.cmml" xref="footnote2.m1.1.1">50</cn></annotation-xml><annotation encoding="application/x-tex" id="footnote2.m1.1d">50</annotation></semantics></math>% of the <span id="footnote2.1" class="ltx_text ltx_font_typewriter">MHA</span> modules and <math id="footnote2.m2.1" class="ltx_Math" alttext="25" display="inline"><semantics id="footnote2.m2.1b"><mn id="footnote2.m2.1.1" xref="footnote2.m2.1.1.cmml">25</mn><annotation-xml encoding="MathML-Content" id="footnote2.m2.1c"><cn type="integer" id="footnote2.m2.1.1.cmml" xref="footnote2.m2.1.1">25</cn></annotation-xml><annotation encoding="application/x-tex" id="footnote2.m2.1d">25</annotation></semantics></math>% of the <span id="footnote2.2" class="ltx_text ltx_font_typewriter">FFN</span> modules, the validation loss increased by <math id="footnote2.m3.1" class="ltx_Math" alttext="0.065" display="inline"><semantics id="footnote2.m3.1b"><mn id="footnote2.m3.1.1" xref="footnote2.m3.1.1.cmml">0.065</mn><annotation-xml encoding="MathML-Content" id="footnote2.m3.1c"><cn type="float" id="footnote2.m3.1.1.cmml" xref="footnote2.m3.1.1">0.065</cn></annotation-xml><annotation encoding="application/x-tex" id="footnote2.m3.1d">0.065</annotation></semantics></math>, the GLUE average score dropped from <math id="footnote2.m4.1" class="ltx_Math" alttext="81.98" display="inline"><semantics id="footnote2.m4.1b"><mn id="footnote2.m4.1.1" xref="footnote2.m4.1.1.cmml">81.98</mn><annotation-xml encoding="MathML-Content" id="footnote2.m4.1c"><cn type="float" id="footnote2.m4.1.1.cmml" xref="footnote2.m4.1.1">81.98</cn></annotation-xml><annotation encoding="application/x-tex" id="footnote2.m4.1d">81.98</annotation></semantics></math> to <math id="footnote2.m5.1" class="ltx_Math" alttext="81.34" display="inline"><semantics id="footnote2.m5.1b"><mn id="footnote2.m5.1.1" xref="footnote2.m5.1.1.cmml">81.34</mn><annotation-xml encoding="MathML-Content" id="footnote2.m5.1c"><cn type="float" id="footnote2.m5.1.1.cmml" xref="footnote2.m5.1.1">81.34</cn></annotation-xml><annotation encoding="application/x-tex" id="footnote2.m5.1d">81.34</annotation></semantics></math>, and the ROUGE average in XSUM dropped from <math id="footnote2.m6.1" class="ltx_Math" alttext="19.34" display="inline"><semantics id="footnote2.m6.1b"><mn id="footnote2.m6.1.1" xref="footnote2.m6.1.1.cmml">19.34</mn><annotation-xml encoding="MathML-Content" id="footnote2.m6.1c"><cn type="float" id="footnote2.m6.1.1.cmml" xref="footnote2.m6.1.1">19.34</cn></annotation-xml><annotation encoding="application/x-tex" id="footnote2.m6.1d">19.34</annotation></semantics></math> to <math id="footnote2.m7.1" class="ltx_Math" alttext="18.55" display="inline"><semantics id="footnote2.m7.1b"><mn id="footnote2.m7.1.1" xref="footnote2.m7.1.1.cmml">18.55</mn><annotation-xml encoding="MathML-Content" id="footnote2.m7.1c"><cn type="float" id="footnote2.m7.1.1.cmml" xref="footnote2.m7.1.1">18.55</cn></annotation-xml><annotation encoding="application/x-tex" id="footnote2.m7.1d">18.55</annotation></semantics></math>.</span></span></span>;
and (3) for those concerned with efficiency, MoM-large can reduce TFLOPs by 16% and memory usage by 42% in forward computation, while maintaining comparable performance to GPT-2-large, via properly increasing the number of modules and compressing the model depth.</p>
</div>
<div id="S1.p7" class="ltx_para ltx_noindent">
<p id="S1.p7.1" class="ltx_p">Our contributions are three-fold: (1) proposal of Mixture-of-Modules to disrupt the depth-ordered convention in Transformer construction, and reinvent Transformers as dynamic assemblies of modules; (2) empirical verification of the efficacy of Mixture-of-Modules on GLUE and XSUM; and (3) a series of new insights into the over-parameterization issue of vanilla Transformers, and their implications for future architecture design. The code for implementing MoM is open-sourced at <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:??</span>.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related works</h2>

<div id="S2.p1" class="ltx_para ltx_noindent">
<p id="S2.p1.1" class="ltx_p">MoM owns a dynamic mechanism of module selection and combination, and thus is related to conditional computation techniques <cite class="ltx_cite ltx_citemacro_citep">(Bengio et al., <a href="#bib.bib4" title="" class="ltx_ref">2013</a>; Davis &amp; Arel, <a href="#bib.bib9" title="" class="ltx_ref">2014</a>; Cho &amp; Bengio, <a href="#bib.bib6" title="" class="ltx_ref">2014</a>)</cite>.
Existing work on conditional computation can be categorized into two groups: <em id="S2.p1.1.1" class="ltx_emph ltx_font_italic">dynamic depth</em> and <em id="S2.p1.1.2" class="ltx_emph ltx_font_italic">dynamic width</em>. In these fields, terms such as gating and routing are used interchangeably, hereafter referred to as “routers” for clarity in presentation.</p>
</div>
<div id="S2.p2" class="ltx_para ltx_noindent">
<p id="S2.p2.1" class="ltx_p">As a typical approach in dynamic depth, <span id="S2.p2.1.1" class="ltx_text ltx_font_italic">Early-exiting</span>  <cite class="ltx_cite ltx_citemacro_citep">(Graves, <a href="#bib.bib18" title="" class="ltx_ref">2016</a>; Figurnov et al., <a href="#bib.bib15" title="" class="ltx_ref">2017</a>; Schuster et al., <a href="#bib.bib38" title="" class="ltx_ref">2022</a>)</cite> accelerates model inference through terminating forward computation at intermediate layers. The decision to exit often relies on confidence-based metrics <cite class="ltx_cite ltx_citemacro_citep">(Elbayad et al., <a href="#bib.bib13" title="" class="ltx_ref">2020</a>; Varshney et al., <a href="#bib.bib43" title="" class="ltx_ref">2023</a>; Xin et al., <a href="#bib.bib48" title="" class="ltx_ref">2020</a>)</cite> or pre-determined strategies <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib24" title="" class="ltx_ref">2020</a>; Corro et al., <a href="#bib.bib8" title="" class="ltx_ref">2023</a>)</cite>. With some degree of generalization, <span id="S2.p2.1.2" class="ltx_text ltx_font_italic">Layer-skip</span> <cite class="ltx_cite ltx_citemacro_citep">(Srivastava et al., <a href="#bib.bib42" title="" class="ltx_ref">2015</a>; Wang et al., <a href="#bib.bib47" title="" class="ltx_ref">2018b</a>; Bapna et al., <a href="#bib.bib3" title="" class="ltx_ref">2020</a>)</cite> represents a more adaptive variant of early-exiting, enabling certain layers to be skipped without terminating the entire forward computation. Existing works mainly facilitate it by training a router <cite class="ltx_cite ltx_citemacro_citep">(Zeng et al., <a href="#bib.bib50" title="" class="ltx_ref">2023</a>; Raposo et al., <a href="#bib.bib37" title="" class="ltx_ref">2024</a>)</cite> or layer pruning <cite class="ltx_cite ltx_citemacro_citep">(Yang et al., <a href="#bib.bib49" title="" class="ltx_ref">2024</a>; Kim et al., <a href="#bib.bib20" title="" class="ltx_ref">2024</a>)</cite>. Finally, if we view parameter copying as a particular way to increase network depth with controlled model size, then some parameter sharing methods <cite class="ltx_cite ltx_citemacro_citep">(Dehghani et al., <a href="#bib.bib12" title="" class="ltx_ref">2019</a>; Lan et al., <a href="#bib.bib21" title="" class="ltx_ref">2020</a>)</cite>, wherein certain modules or layers share parameters, also fall in the dynamic depth group.</p>
</div>
<div id="S2.p3" class="ltx_para ltx_noindent">
<p id="S2.p3.3" class="ltx_p">In terms of dynamic width, <span id="S2.p3.3.1" class="ltx_text ltx_font_italic">Mixture-of-Experts</span> (MoE, <cite class="ltx_cite ltx_citemacro_citep">(Shazeer et al., <a href="#bib.bib40" title="" class="ltx_ref">2017</a>; Lepikhin et al., <a href="#bib.bib23" title="" class="ltx_ref">2021</a>; Fedus et al., <a href="#bib.bib14" title="" class="ltx_ref">2022</a>)</cite>) is a representative method.
An MoE model conceptualizes an FFN module as an “expert” for storing knowledge.
Comprising multiple such experts, an MoE layer replaces the traditional FFN layer within Transformers, aiming for superior performance in handling knowledge-related tasks.
During forward computation, a router network dynamically assigns each token to the top <math id="S2.p3.1.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S2.p3.1.m1.1a"><mi id="S2.p3.1.m1.1.1" xref="S2.p3.1.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S2.p3.1.m1.1b"><ci id="S2.p3.1.m1.1.1.cmml" xref="S2.p3.1.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.1.m1.1c">K</annotation></semantics></math> experts out of a total of <math id="S2.p3.2.m2.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S2.p3.2.m2.1a"><mi id="S2.p3.2.m2.1.1" xref="S2.p3.2.m2.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S2.p3.2.m2.1b"><ci id="S2.p3.2.m2.1.1.cmml" xref="S2.p3.2.m2.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.2.m2.1c">N</annotation></semantics></math> experts, thereby increasing the maximum network width by <math id="S2.p3.3.m3.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S2.p3.3.m3.1a"><mi id="S2.p3.3.m3.1.1" xref="S2.p3.3.m3.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S2.p3.3.m3.1b"><ci id="S2.p3.3.m3.1.1.cmml" xref="S2.p3.3.m3.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.3.m3.1c">K</annotation></semantics></math> times. Other dynamic width methods, such as CODA <cite class="ltx_cite ltx_citemacro_citep">(Lei et al., <a href="#bib.bib22" title="" class="ltx_ref">2024</a>)</cite> and CoLT5 <cite class="ltx_cite ltx_citemacro_citep">(Ainslie et al., <a href="#bib.bib2" title="" class="ltx_ref">2023</a>)</cite>, use similar routing mechanism to select whether a token passes through a heavy or light pathway for not only each FFN layer but also each attention layer.</p>
</div>
<div id="S2.p4" class="ltx_para ltx_noindent">
<p id="S2.p4.1" class="ltx_p">MoM breaks the depth-ordered paradigm followed by existing approaches when performing forward computation. It not only unifies a number of approaches described above but also offers a more flexible and learnable way to achieve conditional computation.</p>
</div>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Methodology</h2>

<div id="S3.p1" class="ltx_para ltx_noindent">
<p id="S3.p1.1" class="ltx_p">The idea of Mixture-of-Modules (MoM) is inspired by the theory presented in “the society of mind” by Marvin Minsky <cite class="ltx_cite ltx_citemacro_citep">(Minsky, <a href="#bib.bib29" title="" class="ltx_ref">1986</a>)</cite>, which explains the true intelligence as certain and very special ways of combinations of simple and modular units (in the book, they are termed “agents”).
In §<a href="#S3.SS1" title="3.1 Mixture-of-Modules (MoM) ‣ 3 Methodology ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a>, we first provide an overview of MoM. Then, we detail the assembly of modules and the routers in §<a href="#S3.SS2" title="3.2 Dynamic assembly of modules ‣ 3 Methodology ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a> and §<a href="#S3.SS3" title="3.3 MoM router (ℛ) ‣ 3 Methodology ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>.
After that, we present the training procedure of MoM in §<a href="#S3.SS4" title="3.4 Training approach ‣ 3 Methodology ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.4</span></a>.
Finally in §<a href="#S3.SS5" title="3.5 MoM as a unified framework ‣ 3 Methodology ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.5</span></a>, we show that MoM unifies various techniques of dynamic computation allocation within Transformers as special cases.</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Mixture-of-Modules (MoM)</h3>

<div id="S3.SS1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS1.p1.7" class="ltx_p">Before delving into the details, we first give a brief description of the workflow of MoM. MoM views the construction of an <math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><mi id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><ci id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">H</annotation></semantics></math>-depth transformer as an <math id="S3.SS1.p1.2.m2.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S3.SS1.p1.2.m2.1a"><mi id="S3.SS1.p1.2.m2.1.1" xref="S3.SS1.p1.2.m2.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.2.m2.1b"><ci id="S3.SS1.p1.2.m2.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.2.m2.1c">H</annotation></semantics></math>-step iterative assembly process. In each assembly step, router <math id="S3.SS1.p1.3.m3.1" class="ltx_Math" alttext="\mathcal{R}" display="inline"><semantics id="S3.SS1.p1.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p1.3.m3.1.1" xref="S3.SS1.p1.3.m3.1.1.cmml">ℛ</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.3.m3.1b"><ci id="S3.SS1.p1.3.m3.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1">ℛ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.3.m3.1c">\mathcal{R}</annotation></semantics></math> dynamically selects <math id="S3.SS1.p1.4.m4.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S3.SS1.p1.4.m4.1a"><mi id="S3.SS1.p1.4.m4.1.1" xref="S3.SS1.p1.4.m4.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.4.m4.1b"><ci id="S3.SS1.p1.4.m4.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.4.m4.1c">K</annotation></semantics></math> modules from a module set <math id="S3.SS1.p1.5.m5.1" class="ltx_Math" alttext="\mathcal{M}" display="inline"><semantics id="S3.SS1.p1.5.m5.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p1.5.m5.1.1" xref="S3.SS1.p1.5.m5.1.1.cmml">ℳ</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.5.m5.1b"><ci id="S3.SS1.p1.5.m5.1.1.cmml" xref="S3.SS1.p1.5.m5.1.1">ℳ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.5.m5.1c">\mathcal{M}</annotation></semantics></math> for each token. Then these selected modules are assembled guided by the assembling function <math id="S3.SS1.p1.6.m6.1" class="ltx_Math" alttext="\phi" display="inline"><semantics id="S3.SS1.p1.6.m6.1a"><mi id="S3.SS1.p1.6.m6.1.1" xref="S3.SS1.p1.6.m6.1.1.cmml">ϕ</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.6.m6.1b"><ci id="S3.SS1.p1.6.m6.1.1.cmml" xref="S3.SS1.p1.6.m6.1.1">italic-ϕ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.6.m6.1c">\phi</annotation></semantics></math>.
Formally, MoM can be defined by a 5-tuple <math id="S3.SS1.p1.7.m7.5" class="ltx_math_unparsed" alttext="&lt;\mathcal{M},\mathcal{R},\phi,K,H&gt;" display="inline"><semantics id="S3.SS1.p1.7.m7.5a"><mrow id="S3.SS1.p1.7.m7.5b"><mo id="S3.SS1.p1.7.m7.5.6">&lt;</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p1.7.m7.1.1">ℳ</mi><mo id="S3.SS1.p1.7.m7.5.7">,</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p1.7.m7.2.2">ℛ</mi><mo id="S3.SS1.p1.7.m7.5.8">,</mo><mi id="S3.SS1.p1.7.m7.3.3">ϕ</mi><mo id="S3.SS1.p1.7.m7.5.9">,</mo><mi id="S3.SS1.p1.7.m7.4.4">K</mi><mo id="S3.SS1.p1.7.m7.5.10">,</mo><mi id="S3.SS1.p1.7.m7.5.5">H</mi><mo id="S3.SS1.p1.7.m7.5.11">&gt;</mo></mrow><annotation encoding="application/x-tex" id="S3.SS1.p1.7.m7.5c">&lt;\mathcal{M},\mathcal{R},\phi,K,H&gt;</annotation></semantics></math>.</p>
</div>
<div id="S3.SS1.p2" class="ltx_para ltx_noindent">
<p id="S3.SS1.p2.6" class="ltx_p"><math id="S3.SS1.p2.1.m1.1" class="ltx_Math" alttext="\mathcal{M}" display="inline"><semantics id="S3.SS1.p2.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p2.1.m1.1.1" xref="S3.SS1.p2.1.m1.1.1.cmml">ℳ</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.1.m1.1b"><ci id="S3.SS1.p2.1.m1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1">ℳ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.1.m1.1c">\mathcal{M}</annotation></semantics></math> is the set that contains all possible modules, where modules are defined as atomic units that could be assembled.
There are two types of modules in a Transformer model, <em id="S3.SS1.p2.6.1" class="ltx_emph ltx_font_italic">i.e.</em>, the multi-head self-attention module (<span id="S3.SS1.p2.6.2" class="ltx_text ltx_font_typewriter">MHA</span>) and the feed-forward network module (<span id="S3.SS1.p2.6.3" class="ltx_text ltx_font_typewriter">FFN</span>), denoted as <math id="S3.SS1.p2.2.m2.1" class="ltx_Math" alttext="m^{\text{A}}" display="inline"><semantics id="S3.SS1.p2.2.m2.1a"><msup id="S3.SS1.p2.2.m2.1.1" xref="S3.SS1.p2.2.m2.1.1.cmml"><mi id="S3.SS1.p2.2.m2.1.1.2" xref="S3.SS1.p2.2.m2.1.1.2.cmml">m</mi><mtext id="S3.SS1.p2.2.m2.1.1.3" xref="S3.SS1.p2.2.m2.1.1.3a.cmml">A</mtext></msup><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.2.m2.1b"><apply id="S3.SS1.p2.2.m2.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.2.m2.1.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1">superscript</csymbol><ci id="S3.SS1.p2.2.m2.1.1.2.cmml" xref="S3.SS1.p2.2.m2.1.1.2">𝑚</ci><ci id="S3.SS1.p2.2.m2.1.1.3a.cmml" xref="S3.SS1.p2.2.m2.1.1.3"><mtext mathsize="70%" id="S3.SS1.p2.2.m2.1.1.3.cmml" xref="S3.SS1.p2.2.m2.1.1.3">A</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.2.m2.1c">m^{\text{A}}</annotation></semantics></math> and <math id="S3.SS1.p2.3.m3.1" class="ltx_Math" alttext="m^{\text{F}}" display="inline"><semantics id="S3.SS1.p2.3.m3.1a"><msup id="S3.SS1.p2.3.m3.1.1" xref="S3.SS1.p2.3.m3.1.1.cmml"><mi id="S3.SS1.p2.3.m3.1.1.2" xref="S3.SS1.p2.3.m3.1.1.2.cmml">m</mi><mtext id="S3.SS1.p2.3.m3.1.1.3" xref="S3.SS1.p2.3.m3.1.1.3a.cmml">F</mtext></msup><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.3.m3.1b"><apply id="S3.SS1.p2.3.m3.1.1.cmml" xref="S3.SS1.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.3.m3.1.1.1.cmml" xref="S3.SS1.p2.3.m3.1.1">superscript</csymbol><ci id="S3.SS1.p2.3.m3.1.1.2.cmml" xref="S3.SS1.p2.3.m3.1.1.2">𝑚</ci><ci id="S3.SS1.p2.3.m3.1.1.3a.cmml" xref="S3.SS1.p2.3.m3.1.1.3"><mtext mathsize="70%" id="S3.SS1.p2.3.m3.1.1.3.cmml" xref="S3.SS1.p2.3.m3.1.1.3">F</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.3.m3.1c">m^{\text{F}}</annotation></semantics></math>, respectively.
In addition, denote the input hidden state of a Transformer layer as <math id="S3.SS1.p2.4.m4.1" class="ltx_Math" alttext="\mathbf{x}\in\mathbb{R}^{d}" display="inline"><semantics id="S3.SS1.p2.4.m4.1a"><mrow id="S3.SS1.p2.4.m4.1.1" xref="S3.SS1.p2.4.m4.1.1.cmml"><mi id="S3.SS1.p2.4.m4.1.1.2" xref="S3.SS1.p2.4.m4.1.1.2.cmml">𝐱</mi><mo id="S3.SS1.p2.4.m4.1.1.1" xref="S3.SS1.p2.4.m4.1.1.1.cmml">∈</mo><msup id="S3.SS1.p2.4.m4.1.1.3" xref="S3.SS1.p2.4.m4.1.1.3.cmml"><mi id="S3.SS1.p2.4.m4.1.1.3.2" xref="S3.SS1.p2.4.m4.1.1.3.2.cmml">ℝ</mi><mi id="S3.SS1.p2.4.m4.1.1.3.3" xref="S3.SS1.p2.4.m4.1.1.3.3.cmml">d</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.4.m4.1b"><apply id="S3.SS1.p2.4.m4.1.1.cmml" xref="S3.SS1.p2.4.m4.1.1"><in id="S3.SS1.p2.4.m4.1.1.1.cmml" xref="S3.SS1.p2.4.m4.1.1.1"></in><ci id="S3.SS1.p2.4.m4.1.1.2.cmml" xref="S3.SS1.p2.4.m4.1.1.2">𝐱</ci><apply id="S3.SS1.p2.4.m4.1.1.3.cmml" xref="S3.SS1.p2.4.m4.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p2.4.m4.1.1.3.1.cmml" xref="S3.SS1.p2.4.m4.1.1.3">superscript</csymbol><ci id="S3.SS1.p2.4.m4.1.1.3.2.cmml" xref="S3.SS1.p2.4.m4.1.1.3.2">ℝ</ci><ci id="S3.SS1.p2.4.m4.1.1.3.3.cmml" xref="S3.SS1.p2.4.m4.1.1.3.3">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.4.m4.1c">\mathbf{x}\in\mathbb{R}^{d}</annotation></semantics></math>, we include a special module <math id="S3.SS1.p2.5.m5.1" class="ltx_Math" alttext="m^{\text{S}}:\mathbf{x}\mapsto\mathbf{x}" display="inline"><semantics id="S3.SS1.p2.5.m5.1a"><mrow id="S3.SS1.p2.5.m5.1.1" xref="S3.SS1.p2.5.m5.1.1.cmml"><msup id="S3.SS1.p2.5.m5.1.1.2" xref="S3.SS1.p2.5.m5.1.1.2.cmml"><mi id="S3.SS1.p2.5.m5.1.1.2.2" xref="S3.SS1.p2.5.m5.1.1.2.2.cmml">m</mi><mtext id="S3.SS1.p2.5.m5.1.1.2.3" xref="S3.SS1.p2.5.m5.1.1.2.3a.cmml">S</mtext></msup><mo lspace="0.278em" rspace="0.278em" id="S3.SS1.p2.5.m5.1.1.1" xref="S3.SS1.p2.5.m5.1.1.1.cmml">:</mo><mrow id="S3.SS1.p2.5.m5.1.1.3" xref="S3.SS1.p2.5.m5.1.1.3.cmml"><mi id="S3.SS1.p2.5.m5.1.1.3.2" xref="S3.SS1.p2.5.m5.1.1.3.2.cmml">𝐱</mi><mo stretchy="false" id="S3.SS1.p2.5.m5.1.1.3.1" xref="S3.SS1.p2.5.m5.1.1.3.1.cmml">↦</mo><mi id="S3.SS1.p2.5.m5.1.1.3.3" xref="S3.SS1.p2.5.m5.1.1.3.3.cmml">𝐱</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.5.m5.1b"><apply id="S3.SS1.p2.5.m5.1.1.cmml" xref="S3.SS1.p2.5.m5.1.1"><ci id="S3.SS1.p2.5.m5.1.1.1.cmml" xref="S3.SS1.p2.5.m5.1.1.1">:</ci><apply id="S3.SS1.p2.5.m5.1.1.2.cmml" xref="S3.SS1.p2.5.m5.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.p2.5.m5.1.1.2.1.cmml" xref="S3.SS1.p2.5.m5.1.1.2">superscript</csymbol><ci id="S3.SS1.p2.5.m5.1.1.2.2.cmml" xref="S3.SS1.p2.5.m5.1.1.2.2">𝑚</ci><ci id="S3.SS1.p2.5.m5.1.1.2.3a.cmml" xref="S3.SS1.p2.5.m5.1.1.2.3"><mtext mathsize="70%" id="S3.SS1.p2.5.m5.1.1.2.3.cmml" xref="S3.SS1.p2.5.m5.1.1.2.3">S</mtext></ci></apply><apply id="S3.SS1.p2.5.m5.1.1.3.cmml" xref="S3.SS1.p2.5.m5.1.1.3"><csymbol cd="latexml" id="S3.SS1.p2.5.m5.1.1.3.1.cmml" xref="S3.SS1.p2.5.m5.1.1.3.1">maps-to</csymbol><ci id="S3.SS1.p2.5.m5.1.1.3.2.cmml" xref="S3.SS1.p2.5.m5.1.1.3.2">𝐱</ci><ci id="S3.SS1.p2.5.m5.1.1.3.3.cmml" xref="S3.SS1.p2.5.m5.1.1.3.3">𝐱</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.5.m5.1c">m^{\text{S}}:\mathbf{x}\mapsto\mathbf{x}</annotation></semantics></math> in <math id="S3.SS1.p2.6.m6.1" class="ltx_Math" alttext="\mathcal{M}" display="inline"><semantics id="S3.SS1.p2.6.m6.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p2.6.m6.1.1" xref="S3.SS1.p2.6.m6.1.1.cmml">ℳ</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.6.m6.1b"><ci id="S3.SS1.p2.6.m6.1.1.cmml" xref="S3.SS1.p2.6.m6.1.1">ℳ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.6.m6.1c">\mathcal{M}</annotation></semantics></math>, which means the absence of an operation applied to the token, allowing for skipping one round of computation.
Therefore, in MoM, we have:</p>
<table id="S3.E1" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S3.E1X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E1X.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle\mathcal{M}=\{m^{\text{A}}_{i}\}_{i=1}^{N_{\text{A}}}\cup\{m^{\text{F}}_{i}\}_{i=1}^{N_{\text{F}}}\cup\{m^{\text{S}}\}," display="inline"><semantics id="S3.E1X.2.1.1.m1.1a"><mrow id="S3.E1X.2.1.1.m1.1.1.1" xref="S3.E1X.2.1.1.m1.1.1.1.1.cmml"><mrow id="S3.E1X.2.1.1.m1.1.1.1.1" xref="S3.E1X.2.1.1.m1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E1X.2.1.1.m1.1.1.1.1.5" xref="S3.E1X.2.1.1.m1.1.1.1.1.5.cmml">ℳ</mi><mo id="S3.E1X.2.1.1.m1.1.1.1.1.4" xref="S3.E1X.2.1.1.m1.1.1.1.1.4.cmml">=</mo><mrow id="S3.E1X.2.1.1.m1.1.1.1.1.3" xref="S3.E1X.2.1.1.m1.1.1.1.1.3.cmml"><msubsup id="S3.E1X.2.1.1.m1.1.1.1.1.1.1" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.cmml"><mrow id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.2" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.2.cmml">{</mo><msubsup id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.2.2" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.2.2.cmml">m</mi><mi id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.3.cmml">i</mi><mtext id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.2.3" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.2.3a.cmml">A</mtext></msubsup><mo stretchy="false" id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.3" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.2.cmml">}</mo></mrow><mrow id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.3" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.3.cmml"><mi id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.3.2" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.3.2.cmml">i</mi><mo id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.3.1" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.3.1.cmml">=</mo><mn id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.3.3" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.3.3.cmml">1</mn></mrow><msub id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.3" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.3.cmml"><mi id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.3.2" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.3.2.cmml">N</mi><mtext id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.3.3" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.3.3a.cmml">A</mtext></msub></msubsup><mo id="S3.E1X.2.1.1.m1.1.1.1.1.3.4" xref="S3.E1X.2.1.1.m1.1.1.1.1.3.4.cmml">∪</mo><msubsup id="S3.E1X.2.1.1.m1.1.1.1.1.2.2" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.cmml"><mrow id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.2.cmml"><mo stretchy="false" id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.2" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.2.cmml">{</mo><msubsup id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1.cmml"><mi id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1.2.2" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1.2.2.cmml">m</mi><mi id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1.3" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1.3.cmml">i</mi><mtext id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1.2.3" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1.2.3a.cmml">F</mtext></msubsup><mo stretchy="false" id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.3" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.2.cmml">}</mo></mrow><mrow id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.3" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.3.cmml"><mi id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.3.2" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.3.2.cmml">i</mi><mo id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.3.1" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.3.1.cmml">=</mo><mn id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.3.3" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.3.3.cmml">1</mn></mrow><msub id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.3" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.3.cmml"><mi id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.3.2" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.3.2.cmml">N</mi><mtext id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.3.3" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.3.3a.cmml">F</mtext></msub></msubsup><mo id="S3.E1X.2.1.1.m1.1.1.1.1.3.4a" xref="S3.E1X.2.1.1.m1.1.1.1.1.3.4.cmml">∪</mo><mrow id="S3.E1X.2.1.1.m1.1.1.1.1.3.3.1" xref="S3.E1X.2.1.1.m1.1.1.1.1.3.3.2.cmml"><mo stretchy="false" id="S3.E1X.2.1.1.m1.1.1.1.1.3.3.1.2" xref="S3.E1X.2.1.1.m1.1.1.1.1.3.3.2.cmml">{</mo><msup id="S3.E1X.2.1.1.m1.1.1.1.1.3.3.1.1" xref="S3.E1X.2.1.1.m1.1.1.1.1.3.3.1.1.cmml"><mi id="S3.E1X.2.1.1.m1.1.1.1.1.3.3.1.1.2" xref="S3.E1X.2.1.1.m1.1.1.1.1.3.3.1.1.2.cmml">m</mi><mtext id="S3.E1X.2.1.1.m1.1.1.1.1.3.3.1.1.3" xref="S3.E1X.2.1.1.m1.1.1.1.1.3.3.1.1.3a.cmml">S</mtext></msup><mo stretchy="false" id="S3.E1X.2.1.1.m1.1.1.1.1.3.3.1.3" xref="S3.E1X.2.1.1.m1.1.1.1.1.3.3.2.cmml">}</mo></mrow></mrow></mrow><mo id="S3.E1X.2.1.1.m1.1.1.1.2" xref="S3.E1X.2.1.1.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E1X.2.1.1.m1.1b"><apply id="S3.E1X.2.1.1.m1.1.1.1.1.cmml" xref="S3.E1X.2.1.1.m1.1.1.1"><eq id="S3.E1X.2.1.1.m1.1.1.1.1.4.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.4"></eq><ci id="S3.E1X.2.1.1.m1.1.1.1.1.5.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.5">ℳ</ci><apply id="S3.E1X.2.1.1.m1.1.1.1.1.3.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.3"><union id="S3.E1X.2.1.1.m1.1.1.1.1.3.4.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.3.4"></union><apply id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.2.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1">superscript</csymbol><apply id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.2.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1">subscript</csymbol><set id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1"><apply id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><apply id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1">superscript</csymbol><ci id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.2.2">𝑚</ci><ci id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.2.3a.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.2.3"><mtext mathsize="70%" id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.2.3">A</mtext></ci></apply><ci id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></set><apply id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.3.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.3"><eq id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.3.1"></eq><ci id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.3.2">𝑖</ci><cn type="integer" id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.1.3.3">1</cn></apply></apply><apply id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.3.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.3.1.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.3.2.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.3.2">𝑁</ci><ci id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.3.3a.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.3.3"><mtext mathsize="50%" id="S3.E1X.2.1.1.m1.1.1.1.1.1.1.3.3.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.1.1.3.3">A</mtext></ci></apply></apply><apply id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.2.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2">superscript</csymbol><apply id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.2.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2">subscript</csymbol><set id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.2.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1"><apply id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1"><csymbol cd="ambiguous" id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1.1.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1">subscript</csymbol><apply id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1.2.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1"><csymbol cd="ambiguous" id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1.2.1.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1">superscript</csymbol><ci id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1.2.2.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1.2.2">𝑚</ci><ci id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1.2.3a.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1.2.3"><mtext mathsize="70%" id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1.2.3.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1.2.3">F</mtext></ci></apply><ci id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1.3.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.1.1.1.3">𝑖</ci></apply></set><apply id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.3.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.3"><eq id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.3.1.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.3.1"></eq><ci id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.3.2.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.3.2">𝑖</ci><cn type="integer" id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.3.3.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.1.3.3">1</cn></apply></apply><apply id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.3.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.3"><csymbol cd="ambiguous" id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.3.1.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.3">subscript</csymbol><ci id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.3.2.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.3.2">𝑁</ci><ci id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.3.3a.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.3.3"><mtext mathsize="50%" id="S3.E1X.2.1.1.m1.1.1.1.1.2.2.3.3.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.2.2.3.3">F</mtext></ci></apply></apply><set id="S3.E1X.2.1.1.m1.1.1.1.1.3.3.2.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.3.3.1"><apply id="S3.E1X.2.1.1.m1.1.1.1.1.3.3.1.1.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.3.3.1.1"><csymbol cd="ambiguous" id="S3.E1X.2.1.1.m1.1.1.1.1.3.3.1.1.1.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.3.3.1.1">superscript</csymbol><ci id="S3.E1X.2.1.1.m1.1.1.1.1.3.3.1.1.2.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.3.3.1.1.2">𝑚</ci><ci id="S3.E1X.2.1.1.m1.1.1.1.1.3.3.1.1.3a.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.3.3.1.1.3"><mtext mathsize="70%" id="S3.E1X.2.1.1.m1.1.1.1.1.3.3.1.1.3.cmml" xref="S3.E1X.2.1.1.m1.1.1.1.1.3.3.1.1.3">S</mtext></ci></apply></set></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E1X.2.1.1.m1.1c">\displaystyle\mathcal{M}=\{m^{\text{A}}_{i}\}_{i=1}^{N_{\text{A}}}\cup\{m^{\text{F}}_{i}\}_{i=1}^{N_{\text{F}}}\cup\{m^{\text{S}}\},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(1)</span></td>
</tr>
</tbody>
</table>
<p id="S3.SS1.p2.8" class="ltx_p">where <math id="S3.SS1.p2.7.m1.1" class="ltx_Math" alttext="N_{\text{A}}" display="inline"><semantics id="S3.SS1.p2.7.m1.1a"><msub id="S3.SS1.p2.7.m1.1.1" xref="S3.SS1.p2.7.m1.1.1.cmml"><mi id="S3.SS1.p2.7.m1.1.1.2" xref="S3.SS1.p2.7.m1.1.1.2.cmml">N</mi><mtext id="S3.SS1.p2.7.m1.1.1.3" xref="S3.SS1.p2.7.m1.1.1.3a.cmml">A</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.7.m1.1b"><apply id="S3.SS1.p2.7.m1.1.1.cmml" xref="S3.SS1.p2.7.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.7.m1.1.1.1.cmml" xref="S3.SS1.p2.7.m1.1.1">subscript</csymbol><ci id="S3.SS1.p2.7.m1.1.1.2.cmml" xref="S3.SS1.p2.7.m1.1.1.2">𝑁</ci><ci id="S3.SS1.p2.7.m1.1.1.3a.cmml" xref="S3.SS1.p2.7.m1.1.1.3"><mtext mathsize="70%" id="S3.SS1.p2.7.m1.1.1.3.cmml" xref="S3.SS1.p2.7.m1.1.1.3">A</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.7.m1.1c">N_{\text{A}}</annotation></semantics></math> are <math id="S3.SS1.p2.8.m2.1" class="ltx_Math" alttext="N_{\text{F}}" display="inline"><semantics id="S3.SS1.p2.8.m2.1a"><msub id="S3.SS1.p2.8.m2.1.1" xref="S3.SS1.p2.8.m2.1.1.cmml"><mi id="S3.SS1.p2.8.m2.1.1.2" xref="S3.SS1.p2.8.m2.1.1.2.cmml">N</mi><mtext id="S3.SS1.p2.8.m2.1.1.3" xref="S3.SS1.p2.8.m2.1.1.3a.cmml">F</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.8.m2.1b"><apply id="S3.SS1.p2.8.m2.1.1.cmml" xref="S3.SS1.p2.8.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p2.8.m2.1.1.1.cmml" xref="S3.SS1.p2.8.m2.1.1">subscript</csymbol><ci id="S3.SS1.p2.8.m2.1.1.2.cmml" xref="S3.SS1.p2.8.m2.1.1.2">𝑁</ci><ci id="S3.SS1.p2.8.m2.1.1.3a.cmml" xref="S3.SS1.p2.8.m2.1.1.3"><mtext mathsize="70%" id="S3.SS1.p2.8.m2.1.1.3.cmml" xref="S3.SS1.p2.8.m2.1.1.3">F</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.8.m2.1c">N_{\text{F}}</annotation></semantics></math> refer to numbers of <span id="S3.SS1.p2.8.1" class="ltx_text ltx_font_typewriter">MHA</span>s and <span id="S3.SS1.p2.8.2" class="ltx_text ltx_font_typewriter">FFN</span>s, respectively.</p>
</div>
<div id="S3.SS1.p3" class="ltx_para ltx_noindent">
<p id="S3.SS1.p3.8" class="ltx_p"><math id="S3.SS1.p3.1.m1.1" class="ltx_Math" alttext="\mathcal{R}" display="inline"><semantics id="S3.SS1.p3.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p3.1.m1.1.1" xref="S3.SS1.p3.1.m1.1.1.cmml">ℛ</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.1.m1.1b"><ci id="S3.SS1.p3.1.m1.1.1.cmml" xref="S3.SS1.p3.1.m1.1.1">ℛ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.1.m1.1c">\mathcal{R}</annotation></semantics></math> is a router responsible for dynamically selecting appropriate modules from <math id="S3.SS1.p3.2.m2.1" class="ltx_Math" alttext="\mathcal{M}" display="inline"><semantics id="S3.SS1.p3.2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p3.2.m2.1.1" xref="S3.SS1.p3.2.m2.1.1.cmml">ℳ</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.2.m2.1b"><ci id="S3.SS1.p3.2.m2.1.1.cmml" xref="S3.SS1.p3.2.m2.1.1">ℳ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.2.m2.1c">\mathcal{M}</annotation></semantics></math> and assembling them into the computation graph.
We use distinct routers for <span id="S3.SS1.p3.8.1" class="ltx_text ltx_font_typewriter">MHA</span>s and <span id="S3.SS1.p3.8.2" class="ltx_text ltx_font_typewriter">FFN</span>s, denoted as <math id="S3.SS1.p3.3.m3.1" class="ltx_Math" alttext="\mathcal{R}^{\text{A}}" display="inline"><semantics id="S3.SS1.p3.3.m3.1a"><msup id="S3.SS1.p3.3.m3.1.1" xref="S3.SS1.p3.3.m3.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p3.3.m3.1.1.2" xref="S3.SS1.p3.3.m3.1.1.2.cmml">ℛ</mi><mtext id="S3.SS1.p3.3.m3.1.1.3" xref="S3.SS1.p3.3.m3.1.1.3a.cmml">A</mtext></msup><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.3.m3.1b"><apply id="S3.SS1.p3.3.m3.1.1.cmml" xref="S3.SS1.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.3.m3.1.1.1.cmml" xref="S3.SS1.p3.3.m3.1.1">superscript</csymbol><ci id="S3.SS1.p3.3.m3.1.1.2.cmml" xref="S3.SS1.p3.3.m3.1.1.2">ℛ</ci><ci id="S3.SS1.p3.3.m3.1.1.3a.cmml" xref="S3.SS1.p3.3.m3.1.1.3"><mtext mathsize="70%" id="S3.SS1.p3.3.m3.1.1.3.cmml" xref="S3.SS1.p3.3.m3.1.1.3">A</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.3.m3.1c">\mathcal{R}^{\text{A}}</annotation></semantics></math> and <math id="S3.SS1.p3.4.m4.1" class="ltx_Math" alttext="\mathcal{R}^{\text{F}}" display="inline"><semantics id="S3.SS1.p3.4.m4.1a"><msup id="S3.SS1.p3.4.m4.1.1" xref="S3.SS1.p3.4.m4.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p3.4.m4.1.1.2" xref="S3.SS1.p3.4.m4.1.1.2.cmml">ℛ</mi><mtext id="S3.SS1.p3.4.m4.1.1.3" xref="S3.SS1.p3.4.m4.1.1.3a.cmml">F</mtext></msup><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.4.m4.1b"><apply id="S3.SS1.p3.4.m4.1.1.cmml" xref="S3.SS1.p3.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.4.m4.1.1.1.cmml" xref="S3.SS1.p3.4.m4.1.1">superscript</csymbol><ci id="S3.SS1.p3.4.m4.1.1.2.cmml" xref="S3.SS1.p3.4.m4.1.1.2">ℛ</ci><ci id="S3.SS1.p3.4.m4.1.1.3a.cmml" xref="S3.SS1.p3.4.m4.1.1.3"><mtext mathsize="70%" id="S3.SS1.p3.4.m4.1.1.3.cmml" xref="S3.SS1.p3.4.m4.1.1.3">F</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.4.m4.1c">\mathcal{R}^{\text{F}}</annotation></semantics></math> respectively.
The output of the router <math id="S3.SS1.p3.5.m5.1" class="ltx_Math" alttext="\mathcal{R}^{\mathcal{X}}" display="inline"><semantics id="S3.SS1.p3.5.m5.1a"><msup id="S3.SS1.p3.5.m5.1.1" xref="S3.SS1.p3.5.m5.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p3.5.m5.1.1.2" xref="S3.SS1.p3.5.m5.1.1.2.cmml">ℛ</mi><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p3.5.m5.1.1.3" xref="S3.SS1.p3.5.m5.1.1.3.cmml">𝒳</mi></msup><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.5.m5.1b"><apply id="S3.SS1.p3.5.m5.1.1.cmml" xref="S3.SS1.p3.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.5.m5.1.1.1.cmml" xref="S3.SS1.p3.5.m5.1.1">superscript</csymbol><ci id="S3.SS1.p3.5.m5.1.1.2.cmml" xref="S3.SS1.p3.5.m5.1.1.2">ℛ</ci><ci id="S3.SS1.p3.5.m5.1.1.3.cmml" xref="S3.SS1.p3.5.m5.1.1.3">𝒳</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.5.m5.1c">\mathcal{R}^{\mathcal{X}}</annotation></semantics></math> is an <math id="S3.SS1.p3.6.m6.1" class="ltx_Math" alttext="(N_{\mathcal{X}}+1)" display="inline"><semantics id="S3.SS1.p3.6.m6.1a"><mrow id="S3.SS1.p3.6.m6.1.1.1" xref="S3.SS1.p3.6.m6.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p3.6.m6.1.1.1.2" xref="S3.SS1.p3.6.m6.1.1.1.1.cmml">(</mo><mrow id="S3.SS1.p3.6.m6.1.1.1.1" xref="S3.SS1.p3.6.m6.1.1.1.1.cmml"><msub id="S3.SS1.p3.6.m6.1.1.1.1.2" xref="S3.SS1.p3.6.m6.1.1.1.1.2.cmml"><mi id="S3.SS1.p3.6.m6.1.1.1.1.2.2" xref="S3.SS1.p3.6.m6.1.1.1.1.2.2.cmml">N</mi><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p3.6.m6.1.1.1.1.2.3" xref="S3.SS1.p3.6.m6.1.1.1.1.2.3.cmml">𝒳</mi></msub><mo id="S3.SS1.p3.6.m6.1.1.1.1.1" xref="S3.SS1.p3.6.m6.1.1.1.1.1.cmml">+</mo><mn id="S3.SS1.p3.6.m6.1.1.1.1.3" xref="S3.SS1.p3.6.m6.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="S3.SS1.p3.6.m6.1.1.1.3" xref="S3.SS1.p3.6.m6.1.1.1.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.6.m6.1b"><apply id="S3.SS1.p3.6.m6.1.1.1.1.cmml" xref="S3.SS1.p3.6.m6.1.1.1"><plus id="S3.SS1.p3.6.m6.1.1.1.1.1.cmml" xref="S3.SS1.p3.6.m6.1.1.1.1.1"></plus><apply id="S3.SS1.p3.6.m6.1.1.1.1.2.cmml" xref="S3.SS1.p3.6.m6.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.p3.6.m6.1.1.1.1.2.1.cmml" xref="S3.SS1.p3.6.m6.1.1.1.1.2">subscript</csymbol><ci id="S3.SS1.p3.6.m6.1.1.1.1.2.2.cmml" xref="S3.SS1.p3.6.m6.1.1.1.1.2.2">𝑁</ci><ci id="S3.SS1.p3.6.m6.1.1.1.1.2.3.cmml" xref="S3.SS1.p3.6.m6.1.1.1.1.2.3">𝒳</ci></apply><cn type="integer" id="S3.SS1.p3.6.m6.1.1.1.1.3.cmml" xref="S3.SS1.p3.6.m6.1.1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.6.m6.1c">(N_{\mathcal{X}}+1)</annotation></semantics></math>-dimensional distribution wherein each item represents the weight assigned to each module <math id="S3.SS1.p3.7.m7.1" class="ltx_Math" alttext="m^{\mathcal{X}}_{i}" display="inline"><semantics id="S3.SS1.p3.7.m7.1a"><msubsup id="S3.SS1.p3.7.m7.1.1" xref="S3.SS1.p3.7.m7.1.1.cmml"><mi id="S3.SS1.p3.7.m7.1.1.2.2" xref="S3.SS1.p3.7.m7.1.1.2.2.cmml">m</mi><mi id="S3.SS1.p3.7.m7.1.1.3" xref="S3.SS1.p3.7.m7.1.1.3.cmml">i</mi><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p3.7.m7.1.1.2.3" xref="S3.SS1.p3.7.m7.1.1.2.3.cmml">𝒳</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.7.m7.1b"><apply id="S3.SS1.p3.7.m7.1.1.cmml" xref="S3.SS1.p3.7.m7.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.7.m7.1.1.1.cmml" xref="S3.SS1.p3.7.m7.1.1">subscript</csymbol><apply id="S3.SS1.p3.7.m7.1.1.2.cmml" xref="S3.SS1.p3.7.m7.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.7.m7.1.1.2.1.cmml" xref="S3.SS1.p3.7.m7.1.1">superscript</csymbol><ci id="S3.SS1.p3.7.m7.1.1.2.2.cmml" xref="S3.SS1.p3.7.m7.1.1.2.2">𝑚</ci><ci id="S3.SS1.p3.7.m7.1.1.2.3.cmml" xref="S3.SS1.p3.7.m7.1.1.2.3">𝒳</ci></apply><ci id="S3.SS1.p3.7.m7.1.1.3.cmml" xref="S3.SS1.p3.7.m7.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.7.m7.1c">m^{\mathcal{X}}_{i}</annotation></semantics></math> as well as <math id="S3.SS1.p3.8.m8.1" class="ltx_Math" alttext="m^{\text{S}}" display="inline"><semantics id="S3.SS1.p3.8.m8.1a"><msup id="S3.SS1.p3.8.m8.1.1" xref="S3.SS1.p3.8.m8.1.1.cmml"><mi id="S3.SS1.p3.8.m8.1.1.2" xref="S3.SS1.p3.8.m8.1.1.2.cmml">m</mi><mtext id="S3.SS1.p3.8.m8.1.1.3" xref="S3.SS1.p3.8.m8.1.1.3a.cmml">S</mtext></msup><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.8.m8.1b"><apply id="S3.SS1.p3.8.m8.1.1.cmml" xref="S3.SS1.p3.8.m8.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.8.m8.1.1.1.cmml" xref="S3.SS1.p3.8.m8.1.1">superscript</csymbol><ci id="S3.SS1.p3.8.m8.1.1.2.cmml" xref="S3.SS1.p3.8.m8.1.1.2">𝑚</ci><ci id="S3.SS1.p3.8.m8.1.1.3a.cmml" xref="S3.SS1.p3.8.m8.1.1.3"><mtext mathsize="70%" id="S3.SS1.p3.8.m8.1.1.3.cmml" xref="S3.SS1.p3.8.m8.1.1.3">S</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.8.m8.1c">m^{\text{S}}</annotation></semantics></math>.
Formally:</p>
<table id="S3.E2" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S3.E2X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E2X.2.1.1.m1.1" class="ltx_math_unparsed" alttext="\displaystyle\mathcal{R}^{\mathcal{X}}:\quad\mathbf{x}\mapsto\mathbf{r}^{\mathcal{X}}," display="inline"><semantics id="S3.E2X.2.1.1.m1.1a"><mrow id="S3.E2X.2.1.1.m1.1b"><msup id="S3.E2X.2.1.1.m1.1.1"><mi class="ltx_font_mathcaligraphic" id="S3.E2X.2.1.1.m1.1.1.2">ℛ</mi><mi class="ltx_font_mathcaligraphic" id="S3.E2X.2.1.1.m1.1.1.3">𝒳</mi></msup><mo lspace="0.278em" id="S3.E2X.2.1.1.m1.1.2">:</mo><mspace width="1.278em" id="S3.E2X.2.1.1.m1.1.3"></mspace><mi id="S3.E2X.2.1.1.m1.1.4">𝐱</mi><mo stretchy="false" id="S3.E2X.2.1.1.m1.1.5">↦</mo><msup id="S3.E2X.2.1.1.m1.1.6"><mi id="S3.E2X.2.1.1.m1.1.6.2">𝐫</mi><mi class="ltx_font_mathcaligraphic" id="S3.E2X.2.1.1.m1.1.6.3">𝒳</mi></msup><mo id="S3.E2X.2.1.1.m1.1.7">,</mo></mrow><annotation encoding="application/x-tex" id="S3.E2X.2.1.1.m1.1c">\displaystyle\mathcal{R}^{\mathcal{X}}:\quad\mathbf{x}\mapsto\mathbf{r}^{\mathcal{X}},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="2" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(2)</span></td>
</tr>
<tr id="S3.E2Xa" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E2Xa.2.1.1.m1.3" class="ltx_Math" alttext="\displaystyle\mathbf{x}\in\mathbb{R}^{d},\quad\mathbf{r}^{\mathcal{X}}\in\mathbb{R}^{N_{\mathcal{X}}+1},\quad\mathcal{X}\in\{\text{A},\text{F}\}." display="inline"><semantics id="S3.E2Xa.2.1.1.m1.3a"><mrow id="S3.E2Xa.2.1.1.m1.3.3.1"><mrow id="S3.E2Xa.2.1.1.m1.3.3.1.1.2" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.3.cmml"><mrow id="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.cmml"><mi id="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.2" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.2.cmml">𝐱</mi><mo id="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.1" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.1.cmml">∈</mo><msup id="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.3" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.3.cmml"><mi id="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.3.2" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.3.2.cmml">ℝ</mi><mi id="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.3.3" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.3.3.cmml">d</mi></msup></mrow><mo rspace="1.167em" id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.3" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.3a.cmml">,</mo><mrow id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.3.cmml"><mrow id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.cmml"><msup id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.2" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.2.cmml"><mi id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.2.2" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.2.2.cmml">𝐫</mi><mi class="ltx_font_mathcaligraphic" id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.2.3" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.2.3.cmml">𝒳</mi></msup><mo id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.1" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.1.cmml">∈</mo><msup id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.cmml"><mi id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.2" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.2.cmml">ℝ</mi><mrow id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.cmml"><msub id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.2" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.2.cmml"><mi id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.2.2" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.2.2.cmml">N</mi><mi class="ltx_font_mathcaligraphic" id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.2.3" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.2.3.cmml">𝒳</mi></msub><mo id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.1" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.1.cmml">+</mo><mn id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.3" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.3.cmml">1</mn></mrow></msup></mrow><mo rspace="1.167em" id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.3" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.3a.cmml">,</mo><mrow id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2.2" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2.2.cmml">𝒳</mi><mo id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2.1" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2.1.cmml">∈</mo><mrow id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2.3.2" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2.3.1.cmml"><mo stretchy="false" id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2.3.2.1" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2.3.1.cmml">{</mo><mtext id="S3.E2Xa.2.1.1.m1.1.1" xref="S3.E2Xa.2.1.1.m1.1.1a.cmml">A</mtext><mo id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2.3.2.2" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2.3.1.cmml">,</mo><mtext id="S3.E2Xa.2.1.1.m1.2.2" xref="S3.E2Xa.2.1.1.m1.2.2a.cmml">F</mtext><mo stretchy="false" id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2.3.2.3" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2.3.1.cmml">}</mo></mrow></mrow></mrow></mrow><mo lspace="0em" id="S3.E2Xa.2.1.1.m1.3.3.1.2">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E2Xa.2.1.1.m1.3b"><apply id="S3.E2Xa.2.1.1.m1.3.3.1.1.3.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2"><csymbol cd="ambiguous" id="S3.E2Xa.2.1.1.m1.3.3.1.1.3a.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.3">formulae-sequence</csymbol><apply id="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1"><in id="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.1.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.1"></in><ci id="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.2.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.2">𝐱</ci><apply id="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.3.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.3.1.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.3">superscript</csymbol><ci id="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.3.2.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.3.2">ℝ</ci><ci id="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.3.3.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.1.1.3.3">𝑑</ci></apply></apply><apply id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.3.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2"><csymbol cd="ambiguous" id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.3a.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.3">formulae-sequence</csymbol><apply id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1"><in id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.1.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.1"></in><apply id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.2.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.2"><csymbol cd="ambiguous" id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.2.1.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.2">superscript</csymbol><ci id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.2.2.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.2.2">𝐫</ci><ci id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.2.3.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.2.3">𝒳</ci></apply><apply id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3"><csymbol cd="ambiguous" id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.1.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3">superscript</csymbol><ci id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.2.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.2">ℝ</ci><apply id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3"><plus id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.1.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.1"></plus><apply id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.2.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.2"><csymbol cd="ambiguous" id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.2.1.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.2">subscript</csymbol><ci id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.2.2.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.2.2">𝑁</ci><ci id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.2.3.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.2.3">𝒳</ci></apply><cn type="integer" id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.3.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.1.1.3.3.3">1</cn></apply></apply></apply><apply id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2"><in id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2.1.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2.1"></in><ci id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2.2.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2.2">𝒳</ci><set id="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2.3.1.cmml" xref="S3.E2Xa.2.1.1.m1.3.3.1.1.2.2.2.2.3.2"><ci id="S3.E2Xa.2.1.1.m1.1.1a.cmml" xref="S3.E2Xa.2.1.1.m1.1.1"><mtext id="S3.E2Xa.2.1.1.m1.1.1.cmml" xref="S3.E2Xa.2.1.1.m1.1.1">A</mtext></ci><ci id="S3.E2Xa.2.1.1.m1.2.2a.cmml" xref="S3.E2Xa.2.1.1.m1.2.2"><mtext id="S3.E2Xa.2.1.1.m1.2.2.cmml" xref="S3.E2Xa.2.1.1.m1.2.2">F</mtext></ci></set></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E2Xa.2.1.1.m1.3c">\displaystyle\mathbf{x}\in\mathbb{R}^{d},\quad\mathbf{r}^{\mathcal{X}}\in\mathbb{R}^{N_{\mathcal{X}}+1},\quad\mathcal{X}\in\{\text{A},\text{F}\}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
</tbody>
</table>
</div>
<div id="S3.SS1.p4" class="ltx_para ltx_noindent">
<p id="S3.SS1.p4.4" class="ltx_p">An MoM model is dynamically assembled step by step. In each step, based on the output of <math id="S3.SS1.p4.1.m1.1" class="ltx_Math" alttext="\mathcal{R}_{\mathcal{X}}" display="inline"><semantics id="S3.SS1.p4.1.m1.1a"><msub id="S3.SS1.p4.1.m1.1.1" xref="S3.SS1.p4.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p4.1.m1.1.1.2" xref="S3.SS1.p4.1.m1.1.1.2.cmml">ℛ</mi><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p4.1.m1.1.1.3" xref="S3.SS1.p4.1.m1.1.1.3.cmml">𝒳</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.1.m1.1b"><apply id="S3.SS1.p4.1.m1.1.1.cmml" xref="S3.SS1.p4.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p4.1.m1.1.1.1.cmml" xref="S3.SS1.p4.1.m1.1.1">subscript</csymbol><ci id="S3.SS1.p4.1.m1.1.1.2.cmml" xref="S3.SS1.p4.1.m1.1.1.2">ℛ</ci><ci id="S3.SS1.p4.1.m1.1.1.3.cmml" xref="S3.SS1.p4.1.m1.1.1.3">𝒳</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.1.m1.1c">\mathcal{R}_{\mathcal{X}}</annotation></semantics></math>, <math id="S3.SS1.p4.2.m2.1" class="ltx_Math" alttext="K_{\mathcal{X}}" display="inline"><semantics id="S3.SS1.p4.2.m2.1a"><msub id="S3.SS1.p4.2.m2.1.1" xref="S3.SS1.p4.2.m2.1.1.cmml"><mi id="S3.SS1.p4.2.m2.1.1.2" xref="S3.SS1.p4.2.m2.1.1.2.cmml">K</mi><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p4.2.m2.1.1.3" xref="S3.SS1.p4.2.m2.1.1.3.cmml">𝒳</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.2.m2.1b"><apply id="S3.SS1.p4.2.m2.1.1.cmml" xref="S3.SS1.p4.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p4.2.m2.1.1.1.cmml" xref="S3.SS1.p4.2.m2.1.1">subscript</csymbol><ci id="S3.SS1.p4.2.m2.1.1.2.cmml" xref="S3.SS1.p4.2.m2.1.1.2">𝐾</ci><ci id="S3.SS1.p4.2.m2.1.1.3.cmml" xref="S3.SS1.p4.2.m2.1.1.3">𝒳</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.2.m2.1c">K_{\mathcal{X}}</annotation></semantics></math> modules are selected from <math id="S3.SS1.p4.3.m3.1" class="ltx_Math" alttext="\mathcal{M}" display="inline"><semantics id="S3.SS1.p4.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p4.3.m3.1.1" xref="S3.SS1.p4.3.m3.1.1.cmml">ℳ</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.3.m3.1b"><ci id="S3.SS1.p4.3.m3.1.1.cmml" xref="S3.SS1.p4.3.m3.1.1">ℳ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.3.m3.1c">\mathcal{M}</annotation></semantics></math> and assembled together. The assembly process lasts <math id="S3.SS1.p4.4.m4.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S3.SS1.p4.4.m4.1a"><mi id="S3.SS1.p4.4.m4.1.1" xref="S3.SS1.p4.4.m4.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.4.m4.1b"><ci id="S3.SS1.p4.4.m4.1.1.cmml" xref="S3.SS1.p4.4.m4.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.4.m4.1c">H</annotation></semantics></math> steps, as detailed in §<a href="#S3.SS2" title="3.2 Dynamic assembly of modules ‣ 3 Methodology ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>.
Further elaboration on these routers, including their architecture and the working pipeline, is deferred to §<a href="#S3.SS3" title="3.3 MoM router (ℛ) ‣ 3 Methodology ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>.
Notably, in MoM, dynamic assembly occurs at the token level, wherein each token is independently and dynamically assigned by routers to appropriate modules for processing.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Dynamic assembly of modules</h3>

<div id="S3.SS2.p1" class="ltx_para ltx_noindent">
<p id="S3.SS2.p1.8" class="ltx_p">We delve into how an MoM model is dynamically assembled.
The construction is an iterative process where in the <math id="S3.SS2.p1.1.m1.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S3.SS2.p1.1.m1.1a"><mi id="S3.SS2.p1.1.m1.1.1" xref="S3.SS2.p1.1.m1.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.1.m1.1b"><ci id="S3.SS2.p1.1.m1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.1.m1.1c">h</annotation></semantics></math>-th step (i.e., the <math id="S3.SS2.p1.2.m2.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S3.SS2.p1.2.m2.1a"><mi id="S3.SS2.p1.2.m2.1.1" xref="S3.SS2.p1.2.m2.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.2.m2.1b"><ci id="S3.SS2.p1.2.m2.1.1.cmml" xref="S3.SS2.p1.2.m2.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.2.m2.1c">h</annotation></semantics></math>-th layer of the model being constructed), we have the input <math id="S3.SS2.p1.3.m3.1" class="ltx_Math" alttext="\mathbf{x}_{h}" display="inline"><semantics id="S3.SS2.p1.3.m3.1a"><msub id="S3.SS2.p1.3.m3.1.1" xref="S3.SS2.p1.3.m3.1.1.cmml"><mi id="S3.SS2.p1.3.m3.1.1.2" xref="S3.SS2.p1.3.m3.1.1.2.cmml">𝐱</mi><mi id="S3.SS2.p1.3.m3.1.1.3" xref="S3.SS2.p1.3.m3.1.1.3.cmml">h</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.3.m3.1b"><apply id="S3.SS2.p1.3.m3.1.1.cmml" xref="S3.SS2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.3.m3.1.1.1.cmml" xref="S3.SS2.p1.3.m3.1.1">subscript</csymbol><ci id="S3.SS2.p1.3.m3.1.1.2.cmml" xref="S3.SS2.p1.3.m3.1.1.2">𝐱</ci><ci id="S3.SS2.p1.3.m3.1.1.3.cmml" xref="S3.SS2.p1.3.m3.1.1.3">ℎ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.3.m3.1c">\mathbf{x}_{h}</annotation></semantics></math>.
The subscript <math id="S3.SS2.p1.4.m4.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S3.SS2.p1.4.m4.1a"><mi id="S3.SS2.p1.4.m4.1.1" xref="S3.SS2.p1.4.m4.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.4.m4.1b"><ci id="S3.SS2.p1.4.m4.1.1.cmml" xref="S3.SS2.p1.4.m4.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.4.m4.1c">h</annotation></semantics></math> is omitted when there’s no ambiguity.
The router <math id="S3.SS2.p1.5.m5.1" class="ltx_Math" alttext="\mathcal{R}" display="inline"><semantics id="S3.SS2.p1.5.m5.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p1.5.m5.1.1" xref="S3.SS2.p1.5.m5.1.1.cmml">ℛ</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.5.m5.1b"><ci id="S3.SS2.p1.5.m5.1.1.cmml" xref="S3.SS2.p1.5.m5.1.1">ℛ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.5.m5.1c">\mathcal{R}</annotation></semantics></math> selects <math id="S3.SS2.p1.6.m6.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S3.SS2.p1.6.m6.1a"><mi id="S3.SS2.p1.6.m6.1.1" xref="S3.SS2.p1.6.m6.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.6.m6.1b"><ci id="S3.SS2.p1.6.m6.1.1.cmml" xref="S3.SS2.p1.6.m6.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.6.m6.1c">K</annotation></semantics></math> modules with the largest routing weight.
We denote the indices of the selected modules as <math id="S3.SS2.p1.7.m7.2" class="ltx_Math" alttext="\mathcal{K}_{\mathcal{X}}=\{i|r_{i}\in\text{TopK}(\mathbf{r}^{\mathcal{X}})\}" display="inline"><semantics id="S3.SS2.p1.7.m7.2a"><mrow id="S3.SS2.p1.7.m7.2.2" xref="S3.SS2.p1.7.m7.2.2.cmml"><msub id="S3.SS2.p1.7.m7.2.2.3" xref="S3.SS2.p1.7.m7.2.2.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p1.7.m7.2.2.3.2" xref="S3.SS2.p1.7.m7.2.2.3.2.cmml">𝒦</mi><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p1.7.m7.2.2.3.3" xref="S3.SS2.p1.7.m7.2.2.3.3.cmml">𝒳</mi></msub><mo id="S3.SS2.p1.7.m7.2.2.2" xref="S3.SS2.p1.7.m7.2.2.2.cmml">=</mo><mrow id="S3.SS2.p1.7.m7.2.2.1.1" xref="S3.SS2.p1.7.m7.2.2.1.2.cmml"><mo stretchy="false" id="S3.SS2.p1.7.m7.2.2.1.1.2" xref="S3.SS2.p1.7.m7.2.2.1.2.1.cmml">{</mo><mi id="S3.SS2.p1.7.m7.1.1" xref="S3.SS2.p1.7.m7.1.1.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.7.m7.2.2.1.1.3" xref="S3.SS2.p1.7.m7.2.2.1.2.1.cmml">|</mo><mrow id="S3.SS2.p1.7.m7.2.2.1.1.1" xref="S3.SS2.p1.7.m7.2.2.1.1.1.cmml"><msub id="S3.SS2.p1.7.m7.2.2.1.1.1.3" xref="S3.SS2.p1.7.m7.2.2.1.1.1.3.cmml"><mi id="S3.SS2.p1.7.m7.2.2.1.1.1.3.2" xref="S3.SS2.p1.7.m7.2.2.1.1.1.3.2.cmml">r</mi><mi id="S3.SS2.p1.7.m7.2.2.1.1.1.3.3" xref="S3.SS2.p1.7.m7.2.2.1.1.1.3.3.cmml">i</mi></msub><mo id="S3.SS2.p1.7.m7.2.2.1.1.1.2" xref="S3.SS2.p1.7.m7.2.2.1.1.1.2.cmml">∈</mo><mrow id="S3.SS2.p1.7.m7.2.2.1.1.1.1" xref="S3.SS2.p1.7.m7.2.2.1.1.1.1.cmml"><mtext id="S3.SS2.p1.7.m7.2.2.1.1.1.1.3" xref="S3.SS2.p1.7.m7.2.2.1.1.1.1.3a.cmml">TopK</mtext><mo lspace="0em" rspace="0em" id="S3.SS2.p1.7.m7.2.2.1.1.1.1.2" xref="S3.SS2.p1.7.m7.2.2.1.1.1.1.2.cmml">​</mo><mrow id="S3.SS2.p1.7.m7.2.2.1.1.1.1.1.1" xref="S3.SS2.p1.7.m7.2.2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p1.7.m7.2.2.1.1.1.1.1.1.2" xref="S3.SS2.p1.7.m7.2.2.1.1.1.1.1.1.1.cmml">(</mo><msup id="S3.SS2.p1.7.m7.2.2.1.1.1.1.1.1.1" xref="S3.SS2.p1.7.m7.2.2.1.1.1.1.1.1.1.cmml"><mi id="S3.SS2.p1.7.m7.2.2.1.1.1.1.1.1.1.2" xref="S3.SS2.p1.7.m7.2.2.1.1.1.1.1.1.1.2.cmml">𝐫</mi><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p1.7.m7.2.2.1.1.1.1.1.1.1.3" xref="S3.SS2.p1.7.m7.2.2.1.1.1.1.1.1.1.3.cmml">𝒳</mi></msup><mo stretchy="false" id="S3.SS2.p1.7.m7.2.2.1.1.1.1.1.1.3" xref="S3.SS2.p1.7.m7.2.2.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S3.SS2.p1.7.m7.2.2.1.1.4" xref="S3.SS2.p1.7.m7.2.2.1.2.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.7.m7.2b"><apply id="S3.SS2.p1.7.m7.2.2.cmml" xref="S3.SS2.p1.7.m7.2.2"><eq id="S3.SS2.p1.7.m7.2.2.2.cmml" xref="S3.SS2.p1.7.m7.2.2.2"></eq><apply id="S3.SS2.p1.7.m7.2.2.3.cmml" xref="S3.SS2.p1.7.m7.2.2.3"><csymbol cd="ambiguous" id="S3.SS2.p1.7.m7.2.2.3.1.cmml" xref="S3.SS2.p1.7.m7.2.2.3">subscript</csymbol><ci id="S3.SS2.p1.7.m7.2.2.3.2.cmml" xref="S3.SS2.p1.7.m7.2.2.3.2">𝒦</ci><ci id="S3.SS2.p1.7.m7.2.2.3.3.cmml" xref="S3.SS2.p1.7.m7.2.2.3.3">𝒳</ci></apply><apply id="S3.SS2.p1.7.m7.2.2.1.2.cmml" xref="S3.SS2.p1.7.m7.2.2.1.1"><csymbol cd="latexml" id="S3.SS2.p1.7.m7.2.2.1.2.1.cmml" xref="S3.SS2.p1.7.m7.2.2.1.1.2">conditional-set</csymbol><ci id="S3.SS2.p1.7.m7.1.1.cmml" xref="S3.SS2.p1.7.m7.1.1">𝑖</ci><apply id="S3.SS2.p1.7.m7.2.2.1.1.1.cmml" xref="S3.SS2.p1.7.m7.2.2.1.1.1"><in id="S3.SS2.p1.7.m7.2.2.1.1.1.2.cmml" xref="S3.SS2.p1.7.m7.2.2.1.1.1.2"></in><apply id="S3.SS2.p1.7.m7.2.2.1.1.1.3.cmml" xref="S3.SS2.p1.7.m7.2.2.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p1.7.m7.2.2.1.1.1.3.1.cmml" xref="S3.SS2.p1.7.m7.2.2.1.1.1.3">subscript</csymbol><ci id="S3.SS2.p1.7.m7.2.2.1.1.1.3.2.cmml" xref="S3.SS2.p1.7.m7.2.2.1.1.1.3.2">𝑟</ci><ci id="S3.SS2.p1.7.m7.2.2.1.1.1.3.3.cmml" xref="S3.SS2.p1.7.m7.2.2.1.1.1.3.3">𝑖</ci></apply><apply id="S3.SS2.p1.7.m7.2.2.1.1.1.1.cmml" xref="S3.SS2.p1.7.m7.2.2.1.1.1.1"><times id="S3.SS2.p1.7.m7.2.2.1.1.1.1.2.cmml" xref="S3.SS2.p1.7.m7.2.2.1.1.1.1.2"></times><ci id="S3.SS2.p1.7.m7.2.2.1.1.1.1.3a.cmml" xref="S3.SS2.p1.7.m7.2.2.1.1.1.1.3"><mtext id="S3.SS2.p1.7.m7.2.2.1.1.1.1.3.cmml" xref="S3.SS2.p1.7.m7.2.2.1.1.1.1.3">TopK</mtext></ci><apply id="S3.SS2.p1.7.m7.2.2.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p1.7.m7.2.2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.7.m7.2.2.1.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p1.7.m7.2.2.1.1.1.1.1.1">superscript</csymbol><ci id="S3.SS2.p1.7.m7.2.2.1.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p1.7.m7.2.2.1.1.1.1.1.1.1.2">𝐫</ci><ci id="S3.SS2.p1.7.m7.2.2.1.1.1.1.1.1.1.3.cmml" xref="S3.SS2.p1.7.m7.2.2.1.1.1.1.1.1.1.3">𝒳</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.7.m7.2c">\mathcal{K}_{\mathcal{X}}=\{i|r_{i}\in\text{TopK}(\mathbf{r}^{\mathcal{X}})\}</annotation></semantics></math>.
Then the selected modules are assembled together through the assembling function <math id="S3.SS2.p1.8.m8.1" class="ltx_Math" alttext="\phi" display="inline"><semantics id="S3.SS2.p1.8.m8.1a"><mi id="S3.SS2.p1.8.m8.1.1" xref="S3.SS2.p1.8.m8.1.1.cmml">ϕ</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.8.m8.1b"><ci id="S3.SS2.p1.8.m8.1.1.cmml" xref="S3.SS2.p1.8.m8.1.1">italic-ϕ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.8.m8.1c">\phi</annotation></semantics></math>. Formally,</p>
<table id="S3.E3" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S3.E3X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E3X.2.1.1.m1.1" class="ltx_math_unparsed" alttext="\displaystyle\phi:\quad&lt;\mathcal{M},\mathcal{R}^{\mathcal{X}},\mathbf{x}_{h}&gt;\ \mapsto\ \mathcal{F}^{\mathcal{X}}_{h},\quad\mathcal{X}\in\{\text{A},\text{F}\}," display="inline"><semantics id="S3.E3X.2.1.1.m1.1a"><mrow id="S3.E3X.2.1.1.m1.1b"><mi id="S3.E3X.2.1.1.m1.1.1">ϕ</mi><mo lspace="0.278em" id="S3.E3X.2.1.1.m1.1.2">:</mo><mspace width="1.278em" id="S3.E3X.2.1.1.m1.1.3"></mspace><mo id="S3.E3X.2.1.1.m1.1.4">&lt;</mo><mi class="ltx_font_mathcaligraphic" id="S3.E3X.2.1.1.m1.1.5">ℳ</mi><mo id="S3.E3X.2.1.1.m1.1.6">,</mo><msup id="S3.E3X.2.1.1.m1.1.7"><mi class="ltx_font_mathcaligraphic" id="S3.E3X.2.1.1.m1.1.7.2">ℛ</mi><mi class="ltx_font_mathcaligraphic" id="S3.E3X.2.1.1.m1.1.7.3">𝒳</mi></msup><mo id="S3.E3X.2.1.1.m1.1.8">,</mo><msub id="S3.E3X.2.1.1.m1.1.9"><mi id="S3.E3X.2.1.1.m1.1.9.2">𝐱</mi><mi id="S3.E3X.2.1.1.m1.1.9.3">h</mi></msub><mo rspace="0.222em" id="S3.E3X.2.1.1.m1.1.10">&gt;</mo><mo rspace="0.778em" stretchy="false" id="S3.E3X.2.1.1.m1.1.11">↦</mo><msubsup id="S3.E3X.2.1.1.m1.1.12"><mi class="ltx_font_mathcaligraphic" id="S3.E3X.2.1.1.m1.1.12.2.2">ℱ</mi><mi id="S3.E3X.2.1.1.m1.1.12.3">h</mi><mi class="ltx_font_mathcaligraphic" id="S3.E3X.2.1.1.m1.1.12.2.3">𝒳</mi></msubsup><mo rspace="1.167em" id="S3.E3X.2.1.1.m1.1.13">,</mo><mi class="ltx_font_mathcaligraphic" id="S3.E3X.2.1.1.m1.1.14">𝒳</mi><mo id="S3.E3X.2.1.1.m1.1.15">∈</mo><mrow id="S3.E3X.2.1.1.m1.1.16"><mo stretchy="false" id="S3.E3X.2.1.1.m1.1.16.1">{</mo><mtext id="S3.E3X.2.1.1.m1.1.16.2">A</mtext><mo id="S3.E3X.2.1.1.m1.1.16.3">,</mo><mtext id="S3.E3X.2.1.1.m1.1.16.4">F</mtext><mo stretchy="false" id="S3.E3X.2.1.1.m1.1.16.5">}</mo></mrow><mo id="S3.E3X.2.1.1.m1.1.17">,</mo></mrow><annotation encoding="application/x-tex" id="S3.E3X.2.1.1.m1.1c">\displaystyle\phi:\quad&lt;\mathcal{M},\mathcal{R}^{\mathcal{X}},\mathbf{x}_{h}&gt;\ \mapsto\ \mathcal{F}^{\mathcal{X}}_{h},\quad\mathcal{X}\in\{\text{A},\text{F}\},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(3)</span></td>
</tr>
</tbody>
</table>
<p id="S3.SS2.p1.14" class="ltx_p">where <math id="S3.SS2.p1.9.m1.1" class="ltx_Math" alttext="\mathcal{F}^{\mathcal{X}}_{h}" display="inline"><semantics id="S3.SS2.p1.9.m1.1a"><msubsup id="S3.SS2.p1.9.m1.1.1" xref="S3.SS2.p1.9.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p1.9.m1.1.1.2.2" xref="S3.SS2.p1.9.m1.1.1.2.2.cmml">ℱ</mi><mi id="S3.SS2.p1.9.m1.1.1.3" xref="S3.SS2.p1.9.m1.1.1.3.cmml">h</mi><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p1.9.m1.1.1.2.3" xref="S3.SS2.p1.9.m1.1.1.2.3.cmml">𝒳</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.9.m1.1b"><apply id="S3.SS2.p1.9.m1.1.1.cmml" xref="S3.SS2.p1.9.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.9.m1.1.1.1.cmml" xref="S3.SS2.p1.9.m1.1.1">subscript</csymbol><apply id="S3.SS2.p1.9.m1.1.1.2.cmml" xref="S3.SS2.p1.9.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.9.m1.1.1.2.1.cmml" xref="S3.SS2.p1.9.m1.1.1">superscript</csymbol><ci id="S3.SS2.p1.9.m1.1.1.2.2.cmml" xref="S3.SS2.p1.9.m1.1.1.2.2">ℱ</ci><ci id="S3.SS2.p1.9.m1.1.1.2.3.cmml" xref="S3.SS2.p1.9.m1.1.1.2.3">𝒳</ci></apply><ci id="S3.SS2.p1.9.m1.1.1.3.cmml" xref="S3.SS2.p1.9.m1.1.1.3">ℎ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.9.m1.1c">\mathcal{F}^{\mathcal{X}}_{h}</annotation></semantics></math> represents the assembled modules.
These assembled modules transform the input <math id="S3.SS2.p1.10.m2.1" class="ltx_Math" alttext="\mathbf{x}_{h}" display="inline"><semantics id="S3.SS2.p1.10.m2.1a"><msub id="S3.SS2.p1.10.m2.1.1" xref="S3.SS2.p1.10.m2.1.1.cmml"><mi id="S3.SS2.p1.10.m2.1.1.2" xref="S3.SS2.p1.10.m2.1.1.2.cmml">𝐱</mi><mi id="S3.SS2.p1.10.m2.1.1.3" xref="S3.SS2.p1.10.m2.1.1.3.cmml">h</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.10.m2.1b"><apply id="S3.SS2.p1.10.m2.1.1.cmml" xref="S3.SS2.p1.10.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.10.m2.1.1.1.cmml" xref="S3.SS2.p1.10.m2.1.1">subscript</csymbol><ci id="S3.SS2.p1.10.m2.1.1.2.cmml" xref="S3.SS2.p1.10.m2.1.1.2">𝐱</ci><ci id="S3.SS2.p1.10.m2.1.1.3.cmml" xref="S3.SS2.p1.10.m2.1.1.3">ℎ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.10.m2.1c">\mathbf{x}_{h}</annotation></semantics></math> into the output <math id="S3.SS2.p1.11.m3.1" class="ltx_Math" alttext="\mathbf{x}_{h+1}" display="inline"><semantics id="S3.SS2.p1.11.m3.1a"><msub id="S3.SS2.p1.11.m3.1.1" xref="S3.SS2.p1.11.m3.1.1.cmml"><mi id="S3.SS2.p1.11.m3.1.1.2" xref="S3.SS2.p1.11.m3.1.1.2.cmml">𝐱</mi><mrow id="S3.SS2.p1.11.m3.1.1.3" xref="S3.SS2.p1.11.m3.1.1.3.cmml"><mi id="S3.SS2.p1.11.m3.1.1.3.2" xref="S3.SS2.p1.11.m3.1.1.3.2.cmml">h</mi><mo id="S3.SS2.p1.11.m3.1.1.3.1" xref="S3.SS2.p1.11.m3.1.1.3.1.cmml">+</mo><mn id="S3.SS2.p1.11.m3.1.1.3.3" xref="S3.SS2.p1.11.m3.1.1.3.3.cmml">1</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.11.m3.1b"><apply id="S3.SS2.p1.11.m3.1.1.cmml" xref="S3.SS2.p1.11.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.11.m3.1.1.1.cmml" xref="S3.SS2.p1.11.m3.1.1">subscript</csymbol><ci id="S3.SS2.p1.11.m3.1.1.2.cmml" xref="S3.SS2.p1.11.m3.1.1.2">𝐱</ci><apply id="S3.SS2.p1.11.m3.1.1.3.cmml" xref="S3.SS2.p1.11.m3.1.1.3"><plus id="S3.SS2.p1.11.m3.1.1.3.1.cmml" xref="S3.SS2.p1.11.m3.1.1.3.1"></plus><ci id="S3.SS2.p1.11.m3.1.1.3.2.cmml" xref="S3.SS2.p1.11.m3.1.1.3.2">ℎ</ci><cn type="integer" id="S3.SS2.p1.11.m3.1.1.3.3.cmml" xref="S3.SS2.p1.11.m3.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.11.m3.1c">\mathbf{x}_{h+1}</annotation></semantics></math>.
We hope the role of the <math id="S3.SS2.p1.12.m4.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S3.SS2.p1.12.m4.1a"><mi id="S3.SS2.p1.12.m4.1.1" xref="S3.SS2.p1.12.m4.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.12.m4.1b"><ci id="S3.SS2.p1.12.m4.1.1.cmml" xref="S3.SS2.p1.12.m4.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.12.m4.1c">h</annotation></semantics></math>-th step of MoM assembly is somewhat akin to the <math id="S3.SS2.p1.13.m5.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S3.SS2.p1.13.m5.1a"><mi id="S3.SS2.p1.13.m5.1.1" xref="S3.SS2.p1.13.m5.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.13.m5.1b"><ci id="S3.SS2.p1.13.m5.1.1.cmml" xref="S3.SS2.p1.13.m5.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.13.m5.1c">h</annotation></semantics></math>-th Transformer block in the conventional sense.
Therefore, we establish two rounds of routing and assembling in each assembly step: one for <span id="S3.SS2.p1.14.1" class="ltx_text ltx_font_typewriter">MHA</span> and the other for <span id="S3.SS2.p1.14.2" class="ltx_text ltx_font_typewriter">FFN</span>.
The forward computation of MoM models at the <math id="S3.SS2.p1.14.m6.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S3.SS2.p1.14.m6.1a"><mi id="S3.SS2.p1.14.m6.1.1" xref="S3.SS2.p1.14.m6.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.14.m6.1b"><ci id="S3.SS2.p1.14.m6.1.1.cmml" xref="S3.SS2.p1.14.m6.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.14.m6.1c">h</annotation></semantics></math>-th step assembly can be represented as:</p>
<table id="S3.E4" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S3.E4X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E4X.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle\mathbf{u}_{h}=\mathcal{F}^{\text{A}}_{h}(\mathbf{x}_{h})+\mathbf{x}_{h}," display="inline"><semantics id="S3.E4X.2.1.1.m1.1a"><mrow id="S3.E4X.2.1.1.m1.1.1.1" xref="S3.E4X.2.1.1.m1.1.1.1.1.cmml"><mrow id="S3.E4X.2.1.1.m1.1.1.1.1" xref="S3.E4X.2.1.1.m1.1.1.1.1.cmml"><msub id="S3.E4X.2.1.1.m1.1.1.1.1.3" xref="S3.E4X.2.1.1.m1.1.1.1.1.3.cmml"><mi id="S3.E4X.2.1.1.m1.1.1.1.1.3.2" xref="S3.E4X.2.1.1.m1.1.1.1.1.3.2.cmml">𝐮</mi><mi id="S3.E4X.2.1.1.m1.1.1.1.1.3.3" xref="S3.E4X.2.1.1.m1.1.1.1.1.3.3.cmml">h</mi></msub><mo id="S3.E4X.2.1.1.m1.1.1.1.1.2" xref="S3.E4X.2.1.1.m1.1.1.1.1.2.cmml">=</mo><mrow id="S3.E4X.2.1.1.m1.1.1.1.1.1" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.cmml"><mrow id="S3.E4X.2.1.1.m1.1.1.1.1.1.1" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.cmml"><msubsup id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3.2.2" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3.2.2.cmml">ℱ</mi><mi id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3.3" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3.3.cmml">h</mi><mtext id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3.2.3" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3.2.3a.cmml">A</mtext></msubsup><mo lspace="0em" rspace="0em" id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.2" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.1.1" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.1.1.2" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.1.1.1" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.1.1.1.2" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.1.1.1.2.cmml">𝐱</mi><mi id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.1.1.1.3" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.1.1.1.3.cmml">h</mi></msub><mo stretchy="false" id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.1.1.3" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.E4X.2.1.1.m1.1.1.1.1.1.2" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.2.cmml">+</mo><msub id="S3.E4X.2.1.1.m1.1.1.1.1.1.3" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.3.cmml"><mi id="S3.E4X.2.1.1.m1.1.1.1.1.1.3.2" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.3.2.cmml">𝐱</mi><mi id="S3.E4X.2.1.1.m1.1.1.1.1.1.3.3" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.3.3.cmml">h</mi></msub></mrow></mrow><mo id="S3.E4X.2.1.1.m1.1.1.1.2" xref="S3.E4X.2.1.1.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E4X.2.1.1.m1.1b"><apply id="S3.E4X.2.1.1.m1.1.1.1.1.cmml" xref="S3.E4X.2.1.1.m1.1.1.1"><eq id="S3.E4X.2.1.1.m1.1.1.1.1.2.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.2"></eq><apply id="S3.E4X.2.1.1.m1.1.1.1.1.3.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4X.2.1.1.m1.1.1.1.1.3.1.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.3">subscript</csymbol><ci id="S3.E4X.2.1.1.m1.1.1.1.1.3.2.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.3.2">𝐮</ci><ci id="S3.E4X.2.1.1.m1.1.1.1.1.3.3.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.3.3">ℎ</ci></apply><apply id="S3.E4X.2.1.1.m1.1.1.1.1.1.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.1"><plus id="S3.E4X.2.1.1.m1.1.1.1.1.1.2.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.2"></plus><apply id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1"><times id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.2.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.2"></times><apply id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3.1.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3">subscript</csymbol><apply id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3.2.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3.2.1.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3">superscript</csymbol><ci id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3.2.2.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3.2.2">ℱ</ci><ci id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3.2.3a.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3.2.3"><mtext mathsize="70%" id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3.2.3.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3.2.3">A</mtext></ci></apply><ci id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3.3.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.3.3">ℎ</ci></apply><apply id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.1.1.1.2">𝐱</ci><ci id="S3.E4X.2.1.1.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.1.1.1.1.3">ℎ</ci></apply></apply><apply id="S3.E4X.2.1.1.m1.1.1.1.1.1.3.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4X.2.1.1.m1.1.1.1.1.1.3.1.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E4X.2.1.1.m1.1.1.1.1.1.3.2.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.3.2">𝐱</ci><ci id="S3.E4X.2.1.1.m1.1.1.1.1.1.3.3.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1.1.3.3">ℎ</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E4X.2.1.1.m1.1c">\displaystyle\mathbf{u}_{h}=\mathcal{F}^{\text{A}}_{h}(\mathbf{x}_{h})+\mathbf{x}_{h},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="2" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(4)</span></td>
</tr>
<tr id="S3.E4Xa" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E4Xa.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle\mathbf{x}_{h+1}=\mathcal{F}^{\text{F}}_{h}(\mathbf{u}_{h})+\mathbf{u}_{h}." display="inline"><semantics id="S3.E4Xa.2.1.1.m1.1a"><mrow id="S3.E4Xa.2.1.1.m1.1.1.1" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.cmml"><mrow id="S3.E4Xa.2.1.1.m1.1.1.1.1" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.cmml"><msub id="S3.E4Xa.2.1.1.m1.1.1.1.1.3" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.3.cmml"><mi id="S3.E4Xa.2.1.1.m1.1.1.1.1.3.2" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.3.2.cmml">𝐱</mi><mrow id="S3.E4Xa.2.1.1.m1.1.1.1.1.3.3" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.3.3.cmml"><mi id="S3.E4Xa.2.1.1.m1.1.1.1.1.3.3.2" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.3.3.2.cmml">h</mi><mo id="S3.E4Xa.2.1.1.m1.1.1.1.1.3.3.1" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.3.3.1.cmml">+</mo><mn id="S3.E4Xa.2.1.1.m1.1.1.1.1.3.3.3" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.3.3.3.cmml">1</mn></mrow></msub><mo id="S3.E4Xa.2.1.1.m1.1.1.1.1.2" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.2.cmml">=</mo><mrow id="S3.E4Xa.2.1.1.m1.1.1.1.1.1" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.cmml"><mrow id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.cmml"><msubsup id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3.2.2" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3.2.2.cmml">ℱ</mi><mi id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3.3" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3.3.cmml">h</mi><mtext id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3.2.3" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3.2.3a.cmml">F</mtext></msubsup><mo lspace="0em" rspace="0em" id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.2" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.1" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.1.2" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.1.1" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.1.1.2" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.1.1.2.cmml">𝐮</mi><mi id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.1.1.3" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.1.1.3.cmml">h</mi></msub><mo stretchy="false" id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.1.3" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.2" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.2.cmml">+</mo><msub id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.3" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.3.cmml"><mi id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.3.2" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.3.2.cmml">𝐮</mi><mi id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.3.3" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.3.3.cmml">h</mi></msub></mrow></mrow><mo lspace="0em" id="S3.E4Xa.2.1.1.m1.1.1.1.2" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E4Xa.2.1.1.m1.1b"><apply id="S3.E4Xa.2.1.1.m1.1.1.1.1.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1"><eq id="S3.E4Xa.2.1.1.m1.1.1.1.1.2.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.2"></eq><apply id="S3.E4Xa.2.1.1.m1.1.1.1.1.3.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4Xa.2.1.1.m1.1.1.1.1.3.1.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.3">subscript</csymbol><ci id="S3.E4Xa.2.1.1.m1.1.1.1.1.3.2.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.3.2">𝐱</ci><apply id="S3.E4Xa.2.1.1.m1.1.1.1.1.3.3.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.3.3"><plus id="S3.E4Xa.2.1.1.m1.1.1.1.1.3.3.1.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.3.3.1"></plus><ci id="S3.E4Xa.2.1.1.m1.1.1.1.1.3.3.2.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.3.3.2">ℎ</ci><cn type="integer" id="S3.E4Xa.2.1.1.m1.1.1.1.1.3.3.3.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.3.3.3">1</cn></apply></apply><apply id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1"><plus id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.2.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.2"></plus><apply id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1"><times id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.2.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.2"></times><apply id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3.1.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3">subscript</csymbol><apply id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3.2.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3.2.1.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3">superscript</csymbol><ci id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3.2.2.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3.2.2">ℱ</ci><ci id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3.2.3a.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3.2.3"><mtext mathsize="70%" id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3.2.3.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3.2.3">F</mtext></ci></apply><ci id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3.3.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3.3">ℎ</ci></apply><apply id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.1.1.2">𝐮</ci><ci id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.1.1.3">ℎ</ci></apply></apply><apply id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.3.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.3.1.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.3.2.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.3.2">𝐮</ci><ci id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.3.3.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.3.3">ℎ</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E4Xa.2.1.1.m1.1c">\displaystyle\mathbf{x}_{h+1}=\mathcal{F}^{\text{F}}_{h}(\mathbf{u}_{h})+\mathbf{u}_{h}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
</tbody>
</table>
<p id="S3.SS2.p1.17" class="ltx_p">We employ Pre-norm in MoM, which normalizes the input before feeding to assembled modules <math id="S3.SS2.p1.15.m1.1" class="ltx_Math" alttext="\mathcal{F}^{\mathcal{X}}" display="inline"><semantics id="S3.SS2.p1.15.m1.1a"><msup id="S3.SS2.p1.15.m1.1.1" xref="S3.SS2.p1.15.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p1.15.m1.1.1.2" xref="S3.SS2.p1.15.m1.1.1.2.cmml">ℱ</mi><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p1.15.m1.1.1.3" xref="S3.SS2.p1.15.m1.1.1.3.cmml">𝒳</mi></msup><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.15.m1.1b"><apply id="S3.SS2.p1.15.m1.1.1.cmml" xref="S3.SS2.p1.15.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.15.m1.1.1.1.cmml" xref="S3.SS2.p1.15.m1.1.1">superscript</csymbol><ci id="S3.SS2.p1.15.m1.1.1.2.cmml" xref="S3.SS2.p1.15.m1.1.1.2">ℱ</ci><ci id="S3.SS2.p1.15.m1.1.1.3.cmml" xref="S3.SS2.p1.15.m1.1.1.3">𝒳</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.15.m1.1c">\mathcal{F}^{\mathcal{X}}</annotation></semantics></math>.
The dynamic assembly process is depicted in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>(b).
We now introduce the detailed formalization for <math id="S3.SS2.p1.16.m2.1" class="ltx_Math" alttext="\mathcal{F}^{\text{A}}" display="inline"><semantics id="S3.SS2.p1.16.m2.1a"><msup id="S3.SS2.p1.16.m2.1.1" xref="S3.SS2.p1.16.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p1.16.m2.1.1.2" xref="S3.SS2.p1.16.m2.1.1.2.cmml">ℱ</mi><mtext id="S3.SS2.p1.16.m2.1.1.3" xref="S3.SS2.p1.16.m2.1.1.3a.cmml">A</mtext></msup><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.16.m2.1b"><apply id="S3.SS2.p1.16.m2.1.1.cmml" xref="S3.SS2.p1.16.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.16.m2.1.1.1.cmml" xref="S3.SS2.p1.16.m2.1.1">superscript</csymbol><ci id="S3.SS2.p1.16.m2.1.1.2.cmml" xref="S3.SS2.p1.16.m2.1.1.2">ℱ</ci><ci id="S3.SS2.p1.16.m2.1.1.3a.cmml" xref="S3.SS2.p1.16.m2.1.1.3"><mtext mathsize="70%" id="S3.SS2.p1.16.m2.1.1.3.cmml" xref="S3.SS2.p1.16.m2.1.1.3">A</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.16.m2.1c">\mathcal{F}^{\text{A}}</annotation></semantics></math> and <math id="S3.SS2.p1.17.m3.1" class="ltx_Math" alttext="\mathcal{F}^{\text{F}}" display="inline"><semantics id="S3.SS2.p1.17.m3.1a"><msup id="S3.SS2.p1.17.m3.1.1" xref="S3.SS2.p1.17.m3.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p1.17.m3.1.1.2" xref="S3.SS2.p1.17.m3.1.1.2.cmml">ℱ</mi><mtext id="S3.SS2.p1.17.m3.1.1.3" xref="S3.SS2.p1.17.m3.1.1.3a.cmml">F</mtext></msup><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.17.m3.1b"><apply id="S3.SS2.p1.17.m3.1.1.cmml" xref="S3.SS2.p1.17.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.17.m3.1.1.1.cmml" xref="S3.SS2.p1.17.m3.1.1">superscript</csymbol><ci id="S3.SS2.p1.17.m3.1.1.2.cmml" xref="S3.SS2.p1.17.m3.1.1.2">ℱ</ci><ci id="S3.SS2.p1.17.m3.1.1.3a.cmml" xref="S3.SS2.p1.17.m3.1.1.3"><mtext mathsize="70%" id="S3.SS2.p1.17.m3.1.1.3.cmml" xref="S3.SS2.p1.17.m3.1.1.3">F</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.17.m3.1c">\mathcal{F}^{\text{F}}</annotation></semantics></math>, respectively.</p>
</div>
<div id="S3.SS2.p2" class="ltx_para ltx_noindent">
<p id="S3.SS2.p2.5" class="ltx_p"><span id="S3.SS2.p2.5.1" class="ltx_text ltx_font_bold">Assembly of attention modules</span> (<math id="S3.SS2.p2.1.m1.1" class="ltx_Math" alttext="\mathcal{F}^{\text{A}}" display="inline"><semantics id="S3.SS2.p2.1.m1.1a"><msup id="S3.SS2.p2.1.m1.1.1" xref="S3.SS2.p2.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p2.1.m1.1.1.2" xref="S3.SS2.p2.1.m1.1.1.2.cmml">ℱ</mi><mtext id="S3.SS2.p2.1.m1.1.1.3" xref="S3.SS2.p2.1.m1.1.1.3a.cmml">A</mtext></msup><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.1.m1.1b"><apply id="S3.SS2.p2.1.m1.1.1.cmml" xref="S3.SS2.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.1.m1.1.1.1.cmml" xref="S3.SS2.p2.1.m1.1.1">superscript</csymbol><ci id="S3.SS2.p2.1.m1.1.1.2.cmml" xref="S3.SS2.p2.1.m1.1.1.2">ℱ</ci><ci id="S3.SS2.p2.1.m1.1.1.3a.cmml" xref="S3.SS2.p2.1.m1.1.1.3"><mtext mathsize="70%" id="S3.SS2.p2.1.m1.1.1.3.cmml" xref="S3.SS2.p2.1.m1.1.1.3">A</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.1.m1.1c">\mathcal{F}^{\text{A}}</annotation></semantics></math>).  We begin by considering the scenario where the <math id="S3.SS2.p2.2.m2.1" class="ltx_Math" alttext="m^{\text{S}}" display="inline"><semantics id="S3.SS2.p2.2.m2.1a"><msup id="S3.SS2.p2.2.m2.1.1" xref="S3.SS2.p2.2.m2.1.1.cmml"><mi id="S3.SS2.p2.2.m2.1.1.2" xref="S3.SS2.p2.2.m2.1.1.2.cmml">m</mi><mtext id="S3.SS2.p2.2.m2.1.1.3" xref="S3.SS2.p2.2.m2.1.1.3a.cmml">S</mtext></msup><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.2.m2.1b"><apply id="S3.SS2.p2.2.m2.1.1.cmml" xref="S3.SS2.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.2.m2.1.1.1.cmml" xref="S3.SS2.p2.2.m2.1.1">superscript</csymbol><ci id="S3.SS2.p2.2.m2.1.1.2.cmml" xref="S3.SS2.p2.2.m2.1.1.2">𝑚</ci><ci id="S3.SS2.p2.2.m2.1.1.3a.cmml" xref="S3.SS2.p2.2.m2.1.1.3"><mtext mathsize="70%" id="S3.SS2.p2.2.m2.1.1.3.cmml" xref="S3.SS2.p2.2.m2.1.1.3">S</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.2.m2.1c">m^{\text{S}}</annotation></semantics></math> module is not selected by routers.
Suppose that an <span id="S3.SS2.p2.5.2" class="ltx_text ltx_font_typewriter">MHA</span> module contains <math id="S3.SS2.p2.3.m3.1" class="ltx_Math" alttext="Z" display="inline"><semantics id="S3.SS2.p2.3.m3.1a"><mi id="S3.SS2.p2.3.m3.1.1" xref="S3.SS2.p2.3.m3.1.1.cmml">Z</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.3.m3.1b"><ci id="S3.SS2.p2.3.m3.1.1.cmml" xref="S3.SS2.p2.3.m3.1.1">𝑍</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.3.m3.1c">Z</annotation></semantics></math> individual heads, then
the assembly of <math id="S3.SS2.p2.4.m4.1" class="ltx_Math" alttext="K_{\text{A}}" display="inline"><semantics id="S3.SS2.p2.4.m4.1a"><msub id="S3.SS2.p2.4.m4.1.1" xref="S3.SS2.p2.4.m4.1.1.cmml"><mi id="S3.SS2.p2.4.m4.1.1.2" xref="S3.SS2.p2.4.m4.1.1.2.cmml">K</mi><mtext id="S3.SS2.p2.4.m4.1.1.3" xref="S3.SS2.p2.4.m4.1.1.3a.cmml">A</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.4.m4.1b"><apply id="S3.SS2.p2.4.m4.1.1.cmml" xref="S3.SS2.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.4.m4.1.1.1.cmml" xref="S3.SS2.p2.4.m4.1.1">subscript</csymbol><ci id="S3.SS2.p2.4.m4.1.1.2.cmml" xref="S3.SS2.p2.4.m4.1.1.2">𝐾</ci><ci id="S3.SS2.p2.4.m4.1.1.3a.cmml" xref="S3.SS2.p2.4.m4.1.1.3"><mtext mathsize="70%" id="S3.SS2.p2.4.m4.1.1.3.cmml" xref="S3.SS2.p2.4.m4.1.1.3">A</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.4.m4.1c">K_{\text{A}}</annotation></semantics></math> <span id="S3.SS2.p2.5.3" class="ltx_text ltx_font_typewriter">MHA</span> modules (i.e., the computation process of <math id="S3.SS2.p2.5.m5.1" class="ltx_Math" alttext="\mathbf{o}=\mathcal{F}^{\text{A}}(\mathbf{x})" display="inline"><semantics id="S3.SS2.p2.5.m5.1a"><mrow id="S3.SS2.p2.5.m5.1.2" xref="S3.SS2.p2.5.m5.1.2.cmml"><mi id="S3.SS2.p2.5.m5.1.2.2" xref="S3.SS2.p2.5.m5.1.2.2.cmml">𝐨</mi><mo id="S3.SS2.p2.5.m5.1.2.1" xref="S3.SS2.p2.5.m5.1.2.1.cmml">=</mo><mrow id="S3.SS2.p2.5.m5.1.2.3" xref="S3.SS2.p2.5.m5.1.2.3.cmml"><msup id="S3.SS2.p2.5.m5.1.2.3.2" xref="S3.SS2.p2.5.m5.1.2.3.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p2.5.m5.1.2.3.2.2" xref="S3.SS2.p2.5.m5.1.2.3.2.2.cmml">ℱ</mi><mtext id="S3.SS2.p2.5.m5.1.2.3.2.3" xref="S3.SS2.p2.5.m5.1.2.3.2.3a.cmml">A</mtext></msup><mo lspace="0em" rspace="0em" id="S3.SS2.p2.5.m5.1.2.3.1" xref="S3.SS2.p2.5.m5.1.2.3.1.cmml">​</mo><mrow id="S3.SS2.p2.5.m5.1.2.3.3.2" xref="S3.SS2.p2.5.m5.1.2.3.cmml"><mo stretchy="false" id="S3.SS2.p2.5.m5.1.2.3.3.2.1" xref="S3.SS2.p2.5.m5.1.2.3.cmml">(</mo><mi id="S3.SS2.p2.5.m5.1.1" xref="S3.SS2.p2.5.m5.1.1.cmml">𝐱</mi><mo stretchy="false" id="S3.SS2.p2.5.m5.1.2.3.3.2.2" xref="S3.SS2.p2.5.m5.1.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.5.m5.1b"><apply id="S3.SS2.p2.5.m5.1.2.cmml" xref="S3.SS2.p2.5.m5.1.2"><eq id="S3.SS2.p2.5.m5.1.2.1.cmml" xref="S3.SS2.p2.5.m5.1.2.1"></eq><ci id="S3.SS2.p2.5.m5.1.2.2.cmml" xref="S3.SS2.p2.5.m5.1.2.2">𝐨</ci><apply id="S3.SS2.p2.5.m5.1.2.3.cmml" xref="S3.SS2.p2.5.m5.1.2.3"><times id="S3.SS2.p2.5.m5.1.2.3.1.cmml" xref="S3.SS2.p2.5.m5.1.2.3.1"></times><apply id="S3.SS2.p2.5.m5.1.2.3.2.cmml" xref="S3.SS2.p2.5.m5.1.2.3.2"><csymbol cd="ambiguous" id="S3.SS2.p2.5.m5.1.2.3.2.1.cmml" xref="S3.SS2.p2.5.m5.1.2.3.2">superscript</csymbol><ci id="S3.SS2.p2.5.m5.1.2.3.2.2.cmml" xref="S3.SS2.p2.5.m5.1.2.3.2.2">ℱ</ci><ci id="S3.SS2.p2.5.m5.1.2.3.2.3a.cmml" xref="S3.SS2.p2.5.m5.1.2.3.2.3"><mtext mathsize="70%" id="S3.SS2.p2.5.m5.1.2.3.2.3.cmml" xref="S3.SS2.p2.5.m5.1.2.3.2.3">A</mtext></ci></apply><ci id="S3.SS2.p2.5.m5.1.1.cmml" xref="S3.SS2.p2.5.m5.1.1">𝐱</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.5.m5.1c">\mathbf{o}=\mathcal{F}^{\text{A}}(\mathbf{x})</annotation></semantics></math>) is defined as:</p>
</div>
<div id="S3.SS2.p3" class="ltx_para ltx_noindent">
<table id="S3.E5" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S3.E5X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E5X.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle\mathbf{o}=\mathbf{a}\sum_{k\in\mathcal{K}_{\text{A}}}r^{\text{A}}_{k}\cdot\mathbf{W}_{k}^{O}," display="inline"><semantics id="S3.E5X.2.1.1.m1.1a"><mrow id="S3.E5X.2.1.1.m1.1.1.1" xref="S3.E5X.2.1.1.m1.1.1.1.1.cmml"><mrow id="S3.E5X.2.1.1.m1.1.1.1.1" xref="S3.E5X.2.1.1.m1.1.1.1.1.cmml"><mi id="S3.E5X.2.1.1.m1.1.1.1.1.2" xref="S3.E5X.2.1.1.m1.1.1.1.1.2.cmml">𝐨</mi><mo id="S3.E5X.2.1.1.m1.1.1.1.1.1" xref="S3.E5X.2.1.1.m1.1.1.1.1.1.cmml">=</mo><mrow id="S3.E5X.2.1.1.m1.1.1.1.1.3" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.cmml"><mi id="S3.E5X.2.1.1.m1.1.1.1.1.3.2" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.2.cmml">𝐚</mi><mo lspace="0em" rspace="0em" id="S3.E5X.2.1.1.m1.1.1.1.1.3.1" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.1.cmml">​</mo><mrow id="S3.E5X.2.1.1.m1.1.1.1.1.3.3" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.cmml"><mstyle displaystyle="true" id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.cmml"><munder id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1a" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.cmml"><mo movablelimits="false" id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.2" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.2.cmml">∑</mo><mrow id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.cmml"><mi id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.2" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.2.cmml">k</mi><mo id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.1" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.1.cmml">∈</mo><msub id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.3" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.3.2" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.3.2.cmml">𝒦</mi><mtext id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.3.3" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.3.3a.cmml">A</mtext></msub></mrow></munder></mstyle><mrow id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.cmml"><msubsup id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2.cmml"><mi id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2.2.2" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2.2.2.cmml">r</mi><mi id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2.3" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2.3.cmml">k</mi><mtext id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2.2.3" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2.2.3a.cmml">A</mtext></msubsup><mo lspace="0.222em" rspace="0.222em" id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.1" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.1.cmml">⋅</mo><msubsup id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3.cmml"><mi id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3.2.2" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3.2.2.cmml">𝐖</mi><mi id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3.2.3" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3.2.3.cmml">k</mi><mi id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3.3" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3.3.cmml">O</mi></msubsup></mrow></mrow></mrow></mrow><mo id="S3.E5X.2.1.1.m1.1.1.1.2" xref="S3.E5X.2.1.1.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E5X.2.1.1.m1.1b"><apply id="S3.E5X.2.1.1.m1.1.1.1.1.cmml" xref="S3.E5X.2.1.1.m1.1.1.1"><eq id="S3.E5X.2.1.1.m1.1.1.1.1.1.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.1"></eq><ci id="S3.E5X.2.1.1.m1.1.1.1.1.2.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.2">𝐨</ci><apply id="S3.E5X.2.1.1.m1.1.1.1.1.3.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3"><times id="S3.E5X.2.1.1.m1.1.1.1.1.3.1.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.1"></times><ci id="S3.E5X.2.1.1.m1.1.1.1.1.3.2.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.2">𝐚</ci><apply id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3"><apply id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1"><csymbol cd="ambiguous" id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.1.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1">subscript</csymbol><sum id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.2.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.2"></sum><apply id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3"><in id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.1.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.1"></in><ci id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.2.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.2">𝑘</ci><apply id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.3.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.3"><csymbol cd="ambiguous" id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.3.1.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.3">subscript</csymbol><ci id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.3.2.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.3.2">𝒦</ci><ci id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.3.3a.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.3.3"><mtext mathsize="50%" id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.3.3.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.1.3.3.3">A</mtext></ci></apply></apply></apply><apply id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2"><ci id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.1.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.1">⋅</ci><apply id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2"><csymbol cd="ambiguous" id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2.1.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2">subscript</csymbol><apply id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2.2.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2"><csymbol cd="ambiguous" id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2.2.1.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2">superscript</csymbol><ci id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2.2.2.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2.2.2">𝑟</ci><ci id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2.2.3a.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2.2.3"><mtext mathsize="70%" id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2.2.3.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2.2.3">A</mtext></ci></apply><ci id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2.3.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.2.3">𝑘</ci></apply><apply id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3"><csymbol cd="ambiguous" id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3.1.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3">superscript</csymbol><apply id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3.2.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3"><csymbol cd="ambiguous" id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3.2.1.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3">subscript</csymbol><ci id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3.2.2.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3.2.2">𝐖</ci><ci id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3.2.3.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3.2.3">𝑘</ci></apply><ci id="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3.3.cmml" xref="S3.E5X.2.1.1.m1.1.1.1.1.3.3.2.3.3">𝑂</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E5X.2.1.1.m1.1c">\displaystyle\mathbf{o}=\mathbf{a}\sum_{k\in\mathcal{K}_{\text{A}}}r^{\text{A}}_{k}\cdot\mathbf{W}_{k}^{O},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="4" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(5)</span></td>
</tr>
<tr id="S3.E5Xa" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E5Xa.2.1.1.m1.4" class="ltx_Math" alttext="\displaystyle\mathbf{r}^{\text{A}}=\mathcal{R}^{\text{A}}(\mathbf{x})=\left(r^{\text{A}}_{1},\ldots,r^{\text{A}}_{k},\ldots,r^{\text{A}}_{K_{A}}\right)," display="inline"><semantics id="S3.E5Xa.2.1.1.m1.4a"><mrow id="S3.E5Xa.2.1.1.m1.4.4.1" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.cmml"><mrow id="S3.E5Xa.2.1.1.m1.4.4.1.1" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.cmml"><msup id="S3.E5Xa.2.1.1.m1.4.4.1.1.5" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.5.cmml"><mi id="S3.E5Xa.2.1.1.m1.4.4.1.1.5.2" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.5.2.cmml">𝐫</mi><mtext id="S3.E5Xa.2.1.1.m1.4.4.1.1.5.3" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.5.3a.cmml">A</mtext></msup><mo id="S3.E5Xa.2.1.1.m1.4.4.1.1.6" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.6.cmml">=</mo><mrow id="S3.E5Xa.2.1.1.m1.4.4.1.1.7" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.7.cmml"><msup id="S3.E5Xa.2.1.1.m1.4.4.1.1.7.2" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.7.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E5Xa.2.1.1.m1.4.4.1.1.7.2.2" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.7.2.2.cmml">ℛ</mi><mtext id="S3.E5Xa.2.1.1.m1.4.4.1.1.7.2.3" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.7.2.3a.cmml">A</mtext></msup><mo lspace="0em" rspace="0em" id="S3.E5Xa.2.1.1.m1.4.4.1.1.7.1" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.7.1.cmml">​</mo><mrow id="S3.E5Xa.2.1.1.m1.4.4.1.1.7.3.2" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.7.cmml"><mo stretchy="false" id="S3.E5Xa.2.1.1.m1.4.4.1.1.7.3.2.1" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.7.cmml">(</mo><mi id="S3.E5Xa.2.1.1.m1.1.1" xref="S3.E5Xa.2.1.1.m1.1.1.cmml">𝐱</mi><mo stretchy="false" id="S3.E5Xa.2.1.1.m1.4.4.1.1.7.3.2.2" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.7.cmml">)</mo></mrow></mrow><mo id="S3.E5Xa.2.1.1.m1.4.4.1.1.8" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.8.cmml">=</mo><mrow id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.4.cmml"><mo id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.4" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.4.cmml">(</mo><msubsup id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.cmml"><mi id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.cmml">r</mi><mn id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.3" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.3.cmml">1</mn><mtext id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.3" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.3a.cmml">A</mtext></msubsup><mo id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.5" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.4.cmml">,</mo><mi mathvariant="normal" id="S3.E5Xa.2.1.1.m1.2.2" xref="S3.E5Xa.2.1.1.m1.2.2.cmml">…</mi><mo id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.6" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.4.cmml">,</mo><msubsup id="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2.cmml"><mi id="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2.2.2" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2.2.2.cmml">r</mi><mi id="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2.3" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2.3.cmml">k</mi><mtext id="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2.2.3" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2.2.3a.cmml">A</mtext></msubsup><mo id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.7" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.4.cmml">,</mo><mi mathvariant="normal" id="S3.E5Xa.2.1.1.m1.3.3" xref="S3.E5Xa.2.1.1.m1.3.3.cmml">…</mi><mo id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.8" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.4.cmml">,</mo><msubsup id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.cmml"><mi id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.2.2" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.2.2.cmml">r</mi><msub id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.3" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.3.cmml"><mi id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.3.2" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.3.2.cmml">K</mi><mi id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.3.3" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.3.3.cmml">A</mi></msub><mtext id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.2.3" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.2.3a.cmml">A</mtext></msubsup><mo id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.9" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.4.cmml">)</mo></mrow></mrow><mo id="S3.E5Xa.2.1.1.m1.4.4.1.2" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E5Xa.2.1.1.m1.4b"><apply id="S3.E5Xa.2.1.1.m1.4.4.1.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1"><and id="S3.E5Xa.2.1.1.m1.4.4.1.1a.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1"></and><apply id="S3.E5Xa.2.1.1.m1.4.4.1.1b.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1"><eq id="S3.E5Xa.2.1.1.m1.4.4.1.1.6.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.6"></eq><apply id="S3.E5Xa.2.1.1.m1.4.4.1.1.5.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.5"><csymbol cd="ambiguous" id="S3.E5Xa.2.1.1.m1.4.4.1.1.5.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.5">superscript</csymbol><ci id="S3.E5Xa.2.1.1.m1.4.4.1.1.5.2.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.5.2">𝐫</ci><ci id="S3.E5Xa.2.1.1.m1.4.4.1.1.5.3a.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.5.3"><mtext mathsize="70%" id="S3.E5Xa.2.1.1.m1.4.4.1.1.5.3.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.5.3">A</mtext></ci></apply><apply id="S3.E5Xa.2.1.1.m1.4.4.1.1.7.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.7"><times id="S3.E5Xa.2.1.1.m1.4.4.1.1.7.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.7.1"></times><apply id="S3.E5Xa.2.1.1.m1.4.4.1.1.7.2.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.7.2"><csymbol cd="ambiguous" id="S3.E5Xa.2.1.1.m1.4.4.1.1.7.2.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.7.2">superscript</csymbol><ci id="S3.E5Xa.2.1.1.m1.4.4.1.1.7.2.2.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.7.2.2">ℛ</ci><ci id="S3.E5Xa.2.1.1.m1.4.4.1.1.7.2.3a.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.7.2.3"><mtext mathsize="70%" id="S3.E5Xa.2.1.1.m1.4.4.1.1.7.2.3.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.7.2.3">A</mtext></ci></apply><ci id="S3.E5Xa.2.1.1.m1.1.1.cmml" xref="S3.E5Xa.2.1.1.m1.1.1">𝐱</ci></apply></apply><apply id="S3.E5Xa.2.1.1.m1.4.4.1.1c.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1"><eq id="S3.E5Xa.2.1.1.m1.4.4.1.1.8.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.8"></eq><share href="#S3.E5Xa.2.1.1.m1.4.4.1.1.7.cmml" id="S3.E5Xa.2.1.1.m1.4.4.1.1d.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1"></share><vector id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.4.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3"><apply id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1">subscript</csymbol><apply id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1">superscript</csymbol><ci id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2">𝑟</ci><ci id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.3a.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.3"><mtext mathsize="70%" id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.3.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.2.3">A</mtext></ci></apply><cn type="integer" id="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.3.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.1.1.1.3">1</cn></apply><ci id="S3.E5Xa.2.1.1.m1.2.2.cmml" xref="S3.E5Xa.2.1.1.m1.2.2">…</ci><apply id="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2"><csymbol cd="ambiguous" id="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2">subscript</csymbol><apply id="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2.2.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2"><csymbol cd="ambiguous" id="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2.2.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2">superscript</csymbol><ci id="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2.2.2.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2.2.2">𝑟</ci><ci id="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2.2.3a.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2.2.3"><mtext mathsize="70%" id="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2.2.3.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2.2.3">A</mtext></ci></apply><ci id="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2.3.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.2.2.2.3">𝑘</ci></apply><ci id="S3.E5Xa.2.1.1.m1.3.3.cmml" xref="S3.E5Xa.2.1.1.m1.3.3">…</ci><apply id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3"><csymbol cd="ambiguous" id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3">subscript</csymbol><apply id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.2.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3"><csymbol cd="ambiguous" id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.2.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3">superscript</csymbol><ci id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.2.2.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.2.2">𝑟</ci><ci id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.2.3a.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.2.3"><mtext mathsize="70%" id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.2.3.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.2.3">A</mtext></ci></apply><apply id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.3.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.3"><csymbol cd="ambiguous" id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.3.1.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.3">subscript</csymbol><ci id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.3.2.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.3.2">𝐾</ci><ci id="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.3.3.cmml" xref="S3.E5Xa.2.1.1.m1.4.4.1.1.3.3.3.3.3">𝐴</ci></apply></apply></vector></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E5Xa.2.1.1.m1.4c">\displaystyle\mathbf{r}^{\text{A}}=\mathcal{R}^{\text{A}}(\mathbf{x})=\left(r^{\text{A}}_{1},\ldots,r^{\text{A}}_{k},\ldots,r^{\text{A}}_{K_{A}}\right),</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
<tr id="S3.E5Xb" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E5Xb.2.1.1.m1.2" class="ltx_math_unparsed" alttext="\displaystyle\mathbf{a}=\left(\mathbf{x}\sum_{k\in\mathcal{K}_{\text{A}}}\mathbf{W}_{k,z}^{V}\right)\cdot" display="inline"><semantics id="S3.E5Xb.2.1.1.m1.2a"><mrow id="S3.E5Xb.2.1.1.m1.2b"><mi id="S3.E5Xb.2.1.1.m1.2.3">𝐚</mi><mo id="S3.E5Xb.2.1.1.m1.2.4">=</mo><mrow id="S3.E5Xb.2.1.1.m1.2.5"><mo id="S3.E5Xb.2.1.1.m1.2.5.1">(</mo><mi id="S3.E5Xb.2.1.1.m1.2.5.2">𝐱</mi><mstyle displaystyle="true" id="S3.E5Xb.2.1.1.m1.2.5.3"><munder id="S3.E5Xb.2.1.1.m1.2.5.3a"><mo movablelimits="false" id="S3.E5Xb.2.1.1.m1.2.5.3.2">∑</mo><mrow id="S3.E5Xb.2.1.1.m1.2.5.3.3"><mi id="S3.E5Xb.2.1.1.m1.2.5.3.3.2">k</mi><mo id="S3.E5Xb.2.1.1.m1.2.5.3.3.1">∈</mo><msub id="S3.E5Xb.2.1.1.m1.2.5.3.3.3"><mi class="ltx_font_mathcaligraphic" id="S3.E5Xb.2.1.1.m1.2.5.3.3.3.2">𝒦</mi><mtext id="S3.E5Xb.2.1.1.m1.2.5.3.3.3.3">A</mtext></msub></mrow></munder></mstyle><msubsup id="S3.E5Xb.2.1.1.m1.2.5.4"><mi id="S3.E5Xb.2.1.1.m1.2.5.4.2.2">𝐖</mi><mrow id="S3.E5Xb.2.1.1.m1.2.2.2.4"><mi id="S3.E5Xb.2.1.1.m1.1.1.1.1">k</mi><mo id="S3.E5Xb.2.1.1.m1.2.2.2.4.1">,</mo><mi id="S3.E5Xb.2.1.1.m1.2.2.2.2">z</mi></mrow><mi id="S3.E5Xb.2.1.1.m1.2.5.4.3">V</mi></msubsup><mo rspace="0.055em" id="S3.E5Xb.2.1.1.m1.2.5.5">)</mo></mrow><mo id="S3.E5Xb.2.1.1.m1.2.6">⋅</mo></mrow><annotation encoding="application/x-tex" id="S3.E5Xb.2.1.1.m1.2c">\displaystyle\mathbf{a}=\left(\mathbf{x}\sum_{k\in\mathcal{K}_{\text{A}}}\mathbf{W}_{k,z}^{V}\right)\cdot</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
<tr id="S3.E5Xc" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E5Xc.2.1.1.m1.7" class="ltx_Math" alttext="\displaystyle\texttt{softmax}\left(\frac{(\mathbf{X}\sum_{k\in\mathcal{K}_{\text{A}}}\mathbf{W}_{k,z}^{Q})(\mathbf{X}\sum_{k\in\mathcal{K}_{\text{A}}}\mathbf{W}_{k,z}^{K})^{\top}}{\sqrt{d_{\text{head}}}}\right)," display="inline"><semantics id="S3.E5Xc.2.1.1.m1.7a"><mrow id="S3.E5Xc.2.1.1.m1.7.7.1" xref="S3.E5Xc.2.1.1.m1.7.7.1.1.cmml"><mrow id="S3.E5Xc.2.1.1.m1.7.7.1.1" xref="S3.E5Xc.2.1.1.m1.7.7.1.1.cmml"><mtext class="ltx_mathvariant_monospace" id="S3.E5Xc.2.1.1.m1.7.7.1.1.2" xref="S3.E5Xc.2.1.1.m1.7.7.1.1.2a.cmml">softmax</mtext><mo lspace="0em" rspace="0em" id="S3.E5Xc.2.1.1.m1.7.7.1.1.1" xref="S3.E5Xc.2.1.1.m1.7.7.1.1.1.cmml">​</mo><mrow id="S3.E5Xc.2.1.1.m1.7.7.1.1.3.2" xref="S3.E5Xc.2.1.1.m1.6.6.cmml"><mo id="S3.E5Xc.2.1.1.m1.7.7.1.1.3.2.1" xref="S3.E5Xc.2.1.1.m1.6.6.cmml">(</mo><mstyle displaystyle="true" id="S3.E5Xc.2.1.1.m1.6.6" xref="S3.E5Xc.2.1.1.m1.6.6.cmml"><mfrac id="S3.E5Xc.2.1.1.m1.6.6a" xref="S3.E5Xc.2.1.1.m1.6.6.cmml"><mrow id="S3.E5Xc.2.1.1.m1.6.6.6" xref="S3.E5Xc.2.1.1.m1.6.6.6.cmml"><mrow id="S3.E5Xc.2.1.1.m1.5.5.5.5.1" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.cmml"><mo stretchy="false" id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.2" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.cmml">(</mo><mrow id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.cmml"><mi id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.2" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.2.cmml">𝐗</mi><mo lspace="0em" rspace="0em" id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.1" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.1.cmml">​</mo><mrow id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.cmml"><msub id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.cmml"><mo id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.2" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.2.cmml">∑</mo><mrow id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.cmml"><mi id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.2" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.2.cmml">k</mi><mo id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.1" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.1.cmml">∈</mo><msub id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.3" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.3.2" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.3.2.cmml">𝒦</mi><mtext id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.3.3" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.3.3a.cmml">A</mtext></msub></mrow></msub><msubsup id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.2" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.2.cmml"><mi id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.2.2.2" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.2.2.2.cmml">𝐖</mi><mrow id="S3.E5Xc.2.1.1.m1.2.2.2.2.2.4" xref="S3.E5Xc.2.1.1.m1.2.2.2.2.2.3.cmml"><mi id="S3.E5Xc.2.1.1.m1.1.1.1.1.1.1" xref="S3.E5Xc.2.1.1.m1.1.1.1.1.1.1.cmml">k</mi><mo id="S3.E5Xc.2.1.1.m1.2.2.2.2.2.4.1" xref="S3.E5Xc.2.1.1.m1.2.2.2.2.2.3.cmml">,</mo><mi id="S3.E5Xc.2.1.1.m1.2.2.2.2.2.2" xref="S3.E5Xc.2.1.1.m1.2.2.2.2.2.2.cmml">z</mi></mrow><mi id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.2.3" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.2.3.cmml">Q</mi></msubsup></mrow></mrow><mo stretchy="false" id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.3" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S3.E5Xc.2.1.1.m1.6.6.6.7" xref="S3.E5Xc.2.1.1.m1.6.6.6.7.cmml">​</mo><msup id="S3.E5Xc.2.1.1.m1.6.6.6.6" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.cmml"><mrow id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.cmml"><mo stretchy="false" id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.2" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.cmml">(</mo><mrow id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.cmml"><mi id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.2" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.2.cmml">𝐗</mi><mo lspace="0em" rspace="0em" id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.1" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.1.cmml">​</mo><mrow id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.cmml"><msub id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.cmml"><mo id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.2" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.2.cmml">∑</mo><mrow id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.cmml"><mi id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.2" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.2.cmml">k</mi><mo id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.1" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.1.cmml">∈</mo><msub id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.3" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.3.2" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.3.2.cmml">𝒦</mi><mtext id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.3.3" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.3.3a.cmml">A</mtext></msub></mrow></msub><msubsup id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.2" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.2.cmml"><mi id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.2.2.2" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.2.2.2.cmml">𝐖</mi><mrow id="S3.E5Xc.2.1.1.m1.4.4.4.4.2.4" xref="S3.E5Xc.2.1.1.m1.4.4.4.4.2.3.cmml"><mi id="S3.E5Xc.2.1.1.m1.3.3.3.3.1.1" xref="S3.E5Xc.2.1.1.m1.3.3.3.3.1.1.cmml">k</mi><mo id="S3.E5Xc.2.1.1.m1.4.4.4.4.2.4.1" xref="S3.E5Xc.2.1.1.m1.4.4.4.4.2.3.cmml">,</mo><mi id="S3.E5Xc.2.1.1.m1.4.4.4.4.2.2" xref="S3.E5Xc.2.1.1.m1.4.4.4.4.2.2.cmml">z</mi></mrow><mi id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.2.3" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.2.3.cmml">K</mi></msubsup></mrow></mrow><mo stretchy="false" id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.3" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.cmml">)</mo></mrow><mo id="S3.E5Xc.2.1.1.m1.6.6.6.6.3" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.3.cmml">⊤</mo></msup></mrow><msqrt id="S3.E5Xc.2.1.1.m1.6.6.8" xref="S3.E5Xc.2.1.1.m1.6.6.8.cmml"><msub id="S3.E5Xc.2.1.1.m1.6.6.8.2" xref="S3.E5Xc.2.1.1.m1.6.6.8.2.cmml"><mi id="S3.E5Xc.2.1.1.m1.6.6.8.2.2" xref="S3.E5Xc.2.1.1.m1.6.6.8.2.2.cmml">d</mi><mtext id="S3.E5Xc.2.1.1.m1.6.6.8.2.3" xref="S3.E5Xc.2.1.1.m1.6.6.8.2.3a.cmml">head</mtext></msub></msqrt></mfrac></mstyle><mo id="S3.E5Xc.2.1.1.m1.7.7.1.1.3.2.2" xref="S3.E5Xc.2.1.1.m1.6.6.cmml">)</mo></mrow></mrow><mo id="S3.E5Xc.2.1.1.m1.7.7.1.2" xref="S3.E5Xc.2.1.1.m1.7.7.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E5Xc.2.1.1.m1.7b"><apply id="S3.E5Xc.2.1.1.m1.7.7.1.1.cmml" xref="S3.E5Xc.2.1.1.m1.7.7.1"><times id="S3.E5Xc.2.1.1.m1.7.7.1.1.1.cmml" xref="S3.E5Xc.2.1.1.m1.7.7.1.1.1"></times><ci id="S3.E5Xc.2.1.1.m1.7.7.1.1.2a.cmml" xref="S3.E5Xc.2.1.1.m1.7.7.1.1.2"><mtext class="ltx_mathvariant_monospace" id="S3.E5Xc.2.1.1.m1.7.7.1.1.2.cmml" xref="S3.E5Xc.2.1.1.m1.7.7.1.1.2">softmax</mtext></ci><apply id="S3.E5Xc.2.1.1.m1.6.6.cmml" xref="S3.E5Xc.2.1.1.m1.7.7.1.1.3.2"><divide id="S3.E5Xc.2.1.1.m1.6.6.7.cmml" xref="S3.E5Xc.2.1.1.m1.7.7.1.1.3.2"></divide><apply id="S3.E5Xc.2.1.1.m1.6.6.6.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6"><times id="S3.E5Xc.2.1.1.m1.6.6.6.7.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.7"></times><apply id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.cmml" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1"><times id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.1.cmml" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.1"></times><ci id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.2.cmml" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.2">𝐗</ci><apply id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.cmml" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3"><apply id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.cmml" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1"><csymbol cd="ambiguous" id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.1.cmml" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1">subscript</csymbol><sum id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.2.cmml" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.2"></sum><apply id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.cmml" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3"><in id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.1.cmml" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.1"></in><ci id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.2.cmml" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.2">𝑘</ci><apply id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.3.cmml" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.3"><csymbol cd="ambiguous" id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.3.1.cmml" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.3">subscript</csymbol><ci id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.3.2.cmml" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.3.2">𝒦</ci><ci id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.3.3a.cmml" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.3.3"><mtext mathsize="50%" id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.3.3.cmml" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.1.3.3.3">A</mtext></ci></apply></apply></apply><apply id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.2.cmml" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.2"><csymbol cd="ambiguous" id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.2.1.cmml" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.2">superscript</csymbol><apply id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.2.2.cmml" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.2"><csymbol cd="ambiguous" id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.2.2.1.cmml" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.2">subscript</csymbol><ci id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.2.2.2.cmml" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.2.2.2">𝐖</ci><list id="S3.E5Xc.2.1.1.m1.2.2.2.2.2.3.cmml" xref="S3.E5Xc.2.1.1.m1.2.2.2.2.2.4"><ci id="S3.E5Xc.2.1.1.m1.1.1.1.1.1.1.cmml" xref="S3.E5Xc.2.1.1.m1.1.1.1.1.1.1">𝑘</ci><ci id="S3.E5Xc.2.1.1.m1.2.2.2.2.2.2.cmml" xref="S3.E5Xc.2.1.1.m1.2.2.2.2.2.2">𝑧</ci></list></apply><ci id="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.2.3.cmml" xref="S3.E5Xc.2.1.1.m1.5.5.5.5.1.1.3.2.3">𝑄</ci></apply></apply></apply><apply id="S3.E5Xc.2.1.1.m1.6.6.6.6.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6"><csymbol cd="ambiguous" id="S3.E5Xc.2.1.1.m1.6.6.6.6.2.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6">superscript</csymbol><apply id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1"><times id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.1.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.1"></times><ci id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.2.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.2">𝐗</ci><apply id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3"><apply id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1"><csymbol cd="ambiguous" id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.1.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1">subscript</csymbol><sum id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.2.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.2"></sum><apply id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3"><in id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.1.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.1"></in><ci id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.2.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.2">𝑘</ci><apply id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.3.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.3"><csymbol cd="ambiguous" id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.3.1.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.3">subscript</csymbol><ci id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.3.2.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.3.2">𝒦</ci><ci id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.3.3a.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.3.3"><mtext mathsize="50%" id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.3.3.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.1.3.3.3">A</mtext></ci></apply></apply></apply><apply id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.2.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.2"><csymbol cd="ambiguous" id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.2.1.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.2">superscript</csymbol><apply id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.2.2.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.2"><csymbol cd="ambiguous" id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.2.2.1.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.2">subscript</csymbol><ci id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.2.2.2.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.2.2.2">𝐖</ci><list id="S3.E5Xc.2.1.1.m1.4.4.4.4.2.3.cmml" xref="S3.E5Xc.2.1.1.m1.4.4.4.4.2.4"><ci id="S3.E5Xc.2.1.1.m1.3.3.3.3.1.1.cmml" xref="S3.E5Xc.2.1.1.m1.3.3.3.3.1.1">𝑘</ci><ci id="S3.E5Xc.2.1.1.m1.4.4.4.4.2.2.cmml" xref="S3.E5Xc.2.1.1.m1.4.4.4.4.2.2">𝑧</ci></list></apply><ci id="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.2.3.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.1.1.1.3.2.3">𝐾</ci></apply></apply></apply><csymbol cd="latexml" id="S3.E5Xc.2.1.1.m1.6.6.6.6.3.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.6.6.3">top</csymbol></apply></apply><apply id="S3.E5Xc.2.1.1.m1.6.6.8.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.8"><root id="S3.E5Xc.2.1.1.m1.6.6.8a.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.8"></root><apply id="S3.E5Xc.2.1.1.m1.6.6.8.2.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.8.2"><csymbol cd="ambiguous" id="S3.E5Xc.2.1.1.m1.6.6.8.2.1.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.8.2">subscript</csymbol><ci id="S3.E5Xc.2.1.1.m1.6.6.8.2.2.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.8.2.2">𝑑</ci><ci id="S3.E5Xc.2.1.1.m1.6.6.8.2.3a.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.8.2.3"><mtext mathsize="70%" id="S3.E5Xc.2.1.1.m1.6.6.8.2.3.cmml" xref="S3.E5Xc.2.1.1.m1.6.6.8.2.3">head</mtext></ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E5Xc.2.1.1.m1.7c">\displaystyle\texttt{softmax}\left(\frac{(\mathbf{X}\sum_{k\in\mathcal{K}_{\text{A}}}\mathbf{W}_{k,z}^{Q})(\mathbf{X}\sum_{k\in\mathcal{K}_{\text{A}}}\mathbf{W}_{k,z}^{K})^{\top}}{\sqrt{d_{\text{head}}}}\right),</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
</tbody>
</table>
</div>
<div id="S3.SS2.p4" class="ltx_para ltx_noindent">
<p id="S3.SS2.p4.7" class="ltx_p">where <math id="S3.SS2.p4.1.m1.1" class="ltx_Math" alttext="\mathbf{X}\in\mathbb{R}^{L\times d}" display="inline"><semantics id="S3.SS2.p4.1.m1.1a"><mrow id="S3.SS2.p4.1.m1.1.1" xref="S3.SS2.p4.1.m1.1.1.cmml"><mi id="S3.SS2.p4.1.m1.1.1.2" xref="S3.SS2.p4.1.m1.1.1.2.cmml">𝐗</mi><mo id="S3.SS2.p4.1.m1.1.1.1" xref="S3.SS2.p4.1.m1.1.1.1.cmml">∈</mo><msup id="S3.SS2.p4.1.m1.1.1.3" xref="S3.SS2.p4.1.m1.1.1.3.cmml"><mi id="S3.SS2.p4.1.m1.1.1.3.2" xref="S3.SS2.p4.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="S3.SS2.p4.1.m1.1.1.3.3" xref="S3.SS2.p4.1.m1.1.1.3.3.cmml"><mi id="S3.SS2.p4.1.m1.1.1.3.3.2" xref="S3.SS2.p4.1.m1.1.1.3.3.2.cmml">L</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.p4.1.m1.1.1.3.3.1" xref="S3.SS2.p4.1.m1.1.1.3.3.1.cmml">×</mo><mi id="S3.SS2.p4.1.m1.1.1.3.3.3" xref="S3.SS2.p4.1.m1.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.1.m1.1b"><apply id="S3.SS2.p4.1.m1.1.1.cmml" xref="S3.SS2.p4.1.m1.1.1"><in id="S3.SS2.p4.1.m1.1.1.1.cmml" xref="S3.SS2.p4.1.m1.1.1.1"></in><ci id="S3.SS2.p4.1.m1.1.1.2.cmml" xref="S3.SS2.p4.1.m1.1.1.2">𝐗</ci><apply id="S3.SS2.p4.1.m1.1.1.3.cmml" xref="S3.SS2.p4.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p4.1.m1.1.1.3.1.cmml" xref="S3.SS2.p4.1.m1.1.1.3">superscript</csymbol><ci id="S3.SS2.p4.1.m1.1.1.3.2.cmml" xref="S3.SS2.p4.1.m1.1.1.3.2">ℝ</ci><apply id="S3.SS2.p4.1.m1.1.1.3.3.cmml" xref="S3.SS2.p4.1.m1.1.1.3.3"><times id="S3.SS2.p4.1.m1.1.1.3.3.1.cmml" xref="S3.SS2.p4.1.m1.1.1.3.3.1"></times><ci id="S3.SS2.p4.1.m1.1.1.3.3.2.cmml" xref="S3.SS2.p4.1.m1.1.1.3.3.2">𝐿</ci><ci id="S3.SS2.p4.1.m1.1.1.3.3.3.cmml" xref="S3.SS2.p4.1.m1.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.1.m1.1c">\mathbf{X}\in\mathbb{R}^{L\times d}</annotation></semantics></math> is the input representation of the sequence, <math id="S3.SS2.p4.2.m2.9" class="ltx_Math" alttext="\mathbf{W}_{k,z}^{Q},\mathbf{W}_{k,z}^{K},\mathbf{W}_{k,z}^{V}\in\mathbb{R}^{d\times d_{\text{head}}}" display="inline"><semantics id="S3.SS2.p4.2.m2.9a"><mrow id="S3.SS2.p4.2.m2.9.9" xref="S3.SS2.p4.2.m2.9.9.cmml"><mrow id="S3.SS2.p4.2.m2.9.9.3.3" xref="S3.SS2.p4.2.m2.9.9.3.4.cmml"><msubsup id="S3.SS2.p4.2.m2.7.7.1.1.1" xref="S3.SS2.p4.2.m2.7.7.1.1.1.cmml"><mi id="S3.SS2.p4.2.m2.7.7.1.1.1.2.2" xref="S3.SS2.p4.2.m2.7.7.1.1.1.2.2.cmml">𝐖</mi><mrow id="S3.SS2.p4.2.m2.2.2.2.4" xref="S3.SS2.p4.2.m2.2.2.2.3.cmml"><mi id="S3.SS2.p4.2.m2.1.1.1.1" xref="S3.SS2.p4.2.m2.1.1.1.1.cmml">k</mi><mo id="S3.SS2.p4.2.m2.2.2.2.4.1" xref="S3.SS2.p4.2.m2.2.2.2.3.cmml">,</mo><mi id="S3.SS2.p4.2.m2.2.2.2.2" xref="S3.SS2.p4.2.m2.2.2.2.2.cmml">z</mi></mrow><mi id="S3.SS2.p4.2.m2.7.7.1.1.1.3" xref="S3.SS2.p4.2.m2.7.7.1.1.1.3.cmml">Q</mi></msubsup><mo id="S3.SS2.p4.2.m2.9.9.3.3.4" xref="S3.SS2.p4.2.m2.9.9.3.4.cmml">,</mo><msubsup id="S3.SS2.p4.2.m2.8.8.2.2.2" xref="S3.SS2.p4.2.m2.8.8.2.2.2.cmml"><mi id="S3.SS2.p4.2.m2.8.8.2.2.2.2.2" xref="S3.SS2.p4.2.m2.8.8.2.2.2.2.2.cmml">𝐖</mi><mrow id="S3.SS2.p4.2.m2.4.4.2.4" xref="S3.SS2.p4.2.m2.4.4.2.3.cmml"><mi id="S3.SS2.p4.2.m2.3.3.1.1" xref="S3.SS2.p4.2.m2.3.3.1.1.cmml">k</mi><mo id="S3.SS2.p4.2.m2.4.4.2.4.1" xref="S3.SS2.p4.2.m2.4.4.2.3.cmml">,</mo><mi id="S3.SS2.p4.2.m2.4.4.2.2" xref="S3.SS2.p4.2.m2.4.4.2.2.cmml">z</mi></mrow><mi id="S3.SS2.p4.2.m2.8.8.2.2.2.3" xref="S3.SS2.p4.2.m2.8.8.2.2.2.3.cmml">K</mi></msubsup><mo id="S3.SS2.p4.2.m2.9.9.3.3.5" xref="S3.SS2.p4.2.m2.9.9.3.4.cmml">,</mo><msubsup id="S3.SS2.p4.2.m2.9.9.3.3.3" xref="S3.SS2.p4.2.m2.9.9.3.3.3.cmml"><mi id="S3.SS2.p4.2.m2.9.9.3.3.3.2.2" xref="S3.SS2.p4.2.m2.9.9.3.3.3.2.2.cmml">𝐖</mi><mrow id="S3.SS2.p4.2.m2.6.6.2.4" xref="S3.SS2.p4.2.m2.6.6.2.3.cmml"><mi id="S3.SS2.p4.2.m2.5.5.1.1" xref="S3.SS2.p4.2.m2.5.5.1.1.cmml">k</mi><mo id="S3.SS2.p4.2.m2.6.6.2.4.1" xref="S3.SS2.p4.2.m2.6.6.2.3.cmml">,</mo><mi id="S3.SS2.p4.2.m2.6.6.2.2" xref="S3.SS2.p4.2.m2.6.6.2.2.cmml">z</mi></mrow><mi id="S3.SS2.p4.2.m2.9.9.3.3.3.3" xref="S3.SS2.p4.2.m2.9.9.3.3.3.3.cmml">V</mi></msubsup></mrow><mo id="S3.SS2.p4.2.m2.9.9.4" xref="S3.SS2.p4.2.m2.9.9.4.cmml">∈</mo><msup id="S3.SS2.p4.2.m2.9.9.5" xref="S3.SS2.p4.2.m2.9.9.5.cmml"><mi id="S3.SS2.p4.2.m2.9.9.5.2" xref="S3.SS2.p4.2.m2.9.9.5.2.cmml">ℝ</mi><mrow id="S3.SS2.p4.2.m2.9.9.5.3" xref="S3.SS2.p4.2.m2.9.9.5.3.cmml"><mi id="S3.SS2.p4.2.m2.9.9.5.3.2" xref="S3.SS2.p4.2.m2.9.9.5.3.2.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.p4.2.m2.9.9.5.3.1" xref="S3.SS2.p4.2.m2.9.9.5.3.1.cmml">×</mo><msub id="S3.SS2.p4.2.m2.9.9.5.3.3" xref="S3.SS2.p4.2.m2.9.9.5.3.3.cmml"><mi id="S3.SS2.p4.2.m2.9.9.5.3.3.2" xref="S3.SS2.p4.2.m2.9.9.5.3.3.2.cmml">d</mi><mtext id="S3.SS2.p4.2.m2.9.9.5.3.3.3" xref="S3.SS2.p4.2.m2.9.9.5.3.3.3a.cmml">head</mtext></msub></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.2.m2.9b"><apply id="S3.SS2.p4.2.m2.9.9.cmml" xref="S3.SS2.p4.2.m2.9.9"><in id="S3.SS2.p4.2.m2.9.9.4.cmml" xref="S3.SS2.p4.2.m2.9.9.4"></in><list id="S3.SS2.p4.2.m2.9.9.3.4.cmml" xref="S3.SS2.p4.2.m2.9.9.3.3"><apply id="S3.SS2.p4.2.m2.7.7.1.1.1.cmml" xref="S3.SS2.p4.2.m2.7.7.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p4.2.m2.7.7.1.1.1.1.cmml" xref="S3.SS2.p4.2.m2.7.7.1.1.1">superscript</csymbol><apply id="S3.SS2.p4.2.m2.7.7.1.1.1.2.cmml" xref="S3.SS2.p4.2.m2.7.7.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p4.2.m2.7.7.1.1.1.2.1.cmml" xref="S3.SS2.p4.2.m2.7.7.1.1.1">subscript</csymbol><ci id="S3.SS2.p4.2.m2.7.7.1.1.1.2.2.cmml" xref="S3.SS2.p4.2.m2.7.7.1.1.1.2.2">𝐖</ci><list id="S3.SS2.p4.2.m2.2.2.2.3.cmml" xref="S3.SS2.p4.2.m2.2.2.2.4"><ci id="S3.SS2.p4.2.m2.1.1.1.1.cmml" xref="S3.SS2.p4.2.m2.1.1.1.1">𝑘</ci><ci id="S3.SS2.p4.2.m2.2.2.2.2.cmml" xref="S3.SS2.p4.2.m2.2.2.2.2">𝑧</ci></list></apply><ci id="S3.SS2.p4.2.m2.7.7.1.1.1.3.cmml" xref="S3.SS2.p4.2.m2.7.7.1.1.1.3">𝑄</ci></apply><apply id="S3.SS2.p4.2.m2.8.8.2.2.2.cmml" xref="S3.SS2.p4.2.m2.8.8.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.p4.2.m2.8.8.2.2.2.1.cmml" xref="S3.SS2.p4.2.m2.8.8.2.2.2">superscript</csymbol><apply id="S3.SS2.p4.2.m2.8.8.2.2.2.2.cmml" xref="S3.SS2.p4.2.m2.8.8.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.p4.2.m2.8.8.2.2.2.2.1.cmml" xref="S3.SS2.p4.2.m2.8.8.2.2.2">subscript</csymbol><ci id="S3.SS2.p4.2.m2.8.8.2.2.2.2.2.cmml" xref="S3.SS2.p4.2.m2.8.8.2.2.2.2.2">𝐖</ci><list id="S3.SS2.p4.2.m2.4.4.2.3.cmml" xref="S3.SS2.p4.2.m2.4.4.2.4"><ci id="S3.SS2.p4.2.m2.3.3.1.1.cmml" xref="S3.SS2.p4.2.m2.3.3.1.1">𝑘</ci><ci id="S3.SS2.p4.2.m2.4.4.2.2.cmml" xref="S3.SS2.p4.2.m2.4.4.2.2">𝑧</ci></list></apply><ci id="S3.SS2.p4.2.m2.8.8.2.2.2.3.cmml" xref="S3.SS2.p4.2.m2.8.8.2.2.2.3">𝐾</ci></apply><apply id="S3.SS2.p4.2.m2.9.9.3.3.3.cmml" xref="S3.SS2.p4.2.m2.9.9.3.3.3"><csymbol cd="ambiguous" id="S3.SS2.p4.2.m2.9.9.3.3.3.1.cmml" xref="S3.SS2.p4.2.m2.9.9.3.3.3">superscript</csymbol><apply id="S3.SS2.p4.2.m2.9.9.3.3.3.2.cmml" xref="S3.SS2.p4.2.m2.9.9.3.3.3"><csymbol cd="ambiguous" id="S3.SS2.p4.2.m2.9.9.3.3.3.2.1.cmml" xref="S3.SS2.p4.2.m2.9.9.3.3.3">subscript</csymbol><ci id="S3.SS2.p4.2.m2.9.9.3.3.3.2.2.cmml" xref="S3.SS2.p4.2.m2.9.9.3.3.3.2.2">𝐖</ci><list id="S3.SS2.p4.2.m2.6.6.2.3.cmml" xref="S3.SS2.p4.2.m2.6.6.2.4"><ci id="S3.SS2.p4.2.m2.5.5.1.1.cmml" xref="S3.SS2.p4.2.m2.5.5.1.1">𝑘</ci><ci id="S3.SS2.p4.2.m2.6.6.2.2.cmml" xref="S3.SS2.p4.2.m2.6.6.2.2">𝑧</ci></list></apply><ci id="S3.SS2.p4.2.m2.9.9.3.3.3.3.cmml" xref="S3.SS2.p4.2.m2.9.9.3.3.3.3">𝑉</ci></apply></list><apply id="S3.SS2.p4.2.m2.9.9.5.cmml" xref="S3.SS2.p4.2.m2.9.9.5"><csymbol cd="ambiguous" id="S3.SS2.p4.2.m2.9.9.5.1.cmml" xref="S3.SS2.p4.2.m2.9.9.5">superscript</csymbol><ci id="S3.SS2.p4.2.m2.9.9.5.2.cmml" xref="S3.SS2.p4.2.m2.9.9.5.2">ℝ</ci><apply id="S3.SS2.p4.2.m2.9.9.5.3.cmml" xref="S3.SS2.p4.2.m2.9.9.5.3"><times id="S3.SS2.p4.2.m2.9.9.5.3.1.cmml" xref="S3.SS2.p4.2.m2.9.9.5.3.1"></times><ci id="S3.SS2.p4.2.m2.9.9.5.3.2.cmml" xref="S3.SS2.p4.2.m2.9.9.5.3.2">𝑑</ci><apply id="S3.SS2.p4.2.m2.9.9.5.3.3.cmml" xref="S3.SS2.p4.2.m2.9.9.5.3.3"><csymbol cd="ambiguous" id="S3.SS2.p4.2.m2.9.9.5.3.3.1.cmml" xref="S3.SS2.p4.2.m2.9.9.5.3.3">subscript</csymbol><ci id="S3.SS2.p4.2.m2.9.9.5.3.3.2.cmml" xref="S3.SS2.p4.2.m2.9.9.5.3.3.2">𝑑</ci><ci id="S3.SS2.p4.2.m2.9.9.5.3.3.3a.cmml" xref="S3.SS2.p4.2.m2.9.9.5.3.3.3"><mtext mathsize="50%" id="S3.SS2.p4.2.m2.9.9.5.3.3.3.cmml" xref="S3.SS2.p4.2.m2.9.9.5.3.3.3">head</mtext></ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.2.m2.9c">\mathbf{W}_{k,z}^{Q},\mathbf{W}_{k,z}^{K},\mathbf{W}_{k,z}^{V}\in\mathbb{R}^{d\times d_{\text{head}}}</annotation></semantics></math>, and <math id="S3.SS2.p4.3.m3.1" class="ltx_Math" alttext="\mathbf{W}_{k}^{O}\in\mathbb{R}^{d_{\text{head}}\times d}" display="inline"><semantics id="S3.SS2.p4.3.m3.1a"><mrow id="S3.SS2.p4.3.m3.1.1" xref="S3.SS2.p4.3.m3.1.1.cmml"><msubsup id="S3.SS2.p4.3.m3.1.1.2" xref="S3.SS2.p4.3.m3.1.1.2.cmml"><mi id="S3.SS2.p4.3.m3.1.1.2.2.2" xref="S3.SS2.p4.3.m3.1.1.2.2.2.cmml">𝐖</mi><mi id="S3.SS2.p4.3.m3.1.1.2.2.3" xref="S3.SS2.p4.3.m3.1.1.2.2.3.cmml">k</mi><mi id="S3.SS2.p4.3.m3.1.1.2.3" xref="S3.SS2.p4.3.m3.1.1.2.3.cmml">O</mi></msubsup><mo id="S3.SS2.p4.3.m3.1.1.1" xref="S3.SS2.p4.3.m3.1.1.1.cmml">∈</mo><msup id="S3.SS2.p4.3.m3.1.1.3" xref="S3.SS2.p4.3.m3.1.1.3.cmml"><mi id="S3.SS2.p4.3.m3.1.1.3.2" xref="S3.SS2.p4.3.m3.1.1.3.2.cmml">ℝ</mi><mrow id="S3.SS2.p4.3.m3.1.1.3.3" xref="S3.SS2.p4.3.m3.1.1.3.3.cmml"><msub id="S3.SS2.p4.3.m3.1.1.3.3.2" xref="S3.SS2.p4.3.m3.1.1.3.3.2.cmml"><mi id="S3.SS2.p4.3.m3.1.1.3.3.2.2" xref="S3.SS2.p4.3.m3.1.1.3.3.2.2.cmml">d</mi><mtext id="S3.SS2.p4.3.m3.1.1.3.3.2.3" xref="S3.SS2.p4.3.m3.1.1.3.3.2.3a.cmml">head</mtext></msub><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.p4.3.m3.1.1.3.3.1" xref="S3.SS2.p4.3.m3.1.1.3.3.1.cmml">×</mo><mi id="S3.SS2.p4.3.m3.1.1.3.3.3" xref="S3.SS2.p4.3.m3.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.3.m3.1b"><apply id="S3.SS2.p4.3.m3.1.1.cmml" xref="S3.SS2.p4.3.m3.1.1"><in id="S3.SS2.p4.3.m3.1.1.1.cmml" xref="S3.SS2.p4.3.m3.1.1.1"></in><apply id="S3.SS2.p4.3.m3.1.1.2.cmml" xref="S3.SS2.p4.3.m3.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p4.3.m3.1.1.2.1.cmml" xref="S3.SS2.p4.3.m3.1.1.2">superscript</csymbol><apply id="S3.SS2.p4.3.m3.1.1.2.2.cmml" xref="S3.SS2.p4.3.m3.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p4.3.m3.1.1.2.2.1.cmml" xref="S3.SS2.p4.3.m3.1.1.2">subscript</csymbol><ci id="S3.SS2.p4.3.m3.1.1.2.2.2.cmml" xref="S3.SS2.p4.3.m3.1.1.2.2.2">𝐖</ci><ci id="S3.SS2.p4.3.m3.1.1.2.2.3.cmml" xref="S3.SS2.p4.3.m3.1.1.2.2.3">𝑘</ci></apply><ci id="S3.SS2.p4.3.m3.1.1.2.3.cmml" xref="S3.SS2.p4.3.m3.1.1.2.3">𝑂</ci></apply><apply id="S3.SS2.p4.3.m3.1.1.3.cmml" xref="S3.SS2.p4.3.m3.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p4.3.m3.1.1.3.1.cmml" xref="S3.SS2.p4.3.m3.1.1.3">superscript</csymbol><ci id="S3.SS2.p4.3.m3.1.1.3.2.cmml" xref="S3.SS2.p4.3.m3.1.1.3.2">ℝ</ci><apply id="S3.SS2.p4.3.m3.1.1.3.3.cmml" xref="S3.SS2.p4.3.m3.1.1.3.3"><times id="S3.SS2.p4.3.m3.1.1.3.3.1.cmml" xref="S3.SS2.p4.3.m3.1.1.3.3.1"></times><apply id="S3.SS2.p4.3.m3.1.1.3.3.2.cmml" xref="S3.SS2.p4.3.m3.1.1.3.3.2"><csymbol cd="ambiguous" id="S3.SS2.p4.3.m3.1.1.3.3.2.1.cmml" xref="S3.SS2.p4.3.m3.1.1.3.3.2">subscript</csymbol><ci id="S3.SS2.p4.3.m3.1.1.3.3.2.2.cmml" xref="S3.SS2.p4.3.m3.1.1.3.3.2.2">𝑑</ci><ci id="S3.SS2.p4.3.m3.1.1.3.3.2.3a.cmml" xref="S3.SS2.p4.3.m3.1.1.3.3.2.3"><mtext mathsize="50%" id="S3.SS2.p4.3.m3.1.1.3.3.2.3.cmml" xref="S3.SS2.p4.3.m3.1.1.3.3.2.3">head</mtext></ci></apply><ci id="S3.SS2.p4.3.m3.1.1.3.3.3.cmml" xref="S3.SS2.p4.3.m3.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.3.m3.1c">\mathbf{W}_{k}^{O}\in\mathbb{R}^{d_{\text{head}}\times d}</annotation></semantics></math> are weight matrices with <math id="S3.SS2.p4.4.m4.1" class="ltx_Math" alttext="d_{\text{head}}=d/Z" display="inline"><semantics id="S3.SS2.p4.4.m4.1a"><mrow id="S3.SS2.p4.4.m4.1.1" xref="S3.SS2.p4.4.m4.1.1.cmml"><msub id="S3.SS2.p4.4.m4.1.1.2" xref="S3.SS2.p4.4.m4.1.1.2.cmml"><mi id="S3.SS2.p4.4.m4.1.1.2.2" xref="S3.SS2.p4.4.m4.1.1.2.2.cmml">d</mi><mtext id="S3.SS2.p4.4.m4.1.1.2.3" xref="S3.SS2.p4.4.m4.1.1.2.3a.cmml">head</mtext></msub><mo id="S3.SS2.p4.4.m4.1.1.1" xref="S3.SS2.p4.4.m4.1.1.1.cmml">=</mo><mrow id="S3.SS2.p4.4.m4.1.1.3" xref="S3.SS2.p4.4.m4.1.1.3.cmml"><mi id="S3.SS2.p4.4.m4.1.1.3.2" xref="S3.SS2.p4.4.m4.1.1.3.2.cmml">d</mi><mo id="S3.SS2.p4.4.m4.1.1.3.1" xref="S3.SS2.p4.4.m4.1.1.3.1.cmml">/</mo><mi id="S3.SS2.p4.4.m4.1.1.3.3" xref="S3.SS2.p4.4.m4.1.1.3.3.cmml">Z</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.4.m4.1b"><apply id="S3.SS2.p4.4.m4.1.1.cmml" xref="S3.SS2.p4.4.m4.1.1"><eq id="S3.SS2.p4.4.m4.1.1.1.cmml" xref="S3.SS2.p4.4.m4.1.1.1"></eq><apply id="S3.SS2.p4.4.m4.1.1.2.cmml" xref="S3.SS2.p4.4.m4.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p4.4.m4.1.1.2.1.cmml" xref="S3.SS2.p4.4.m4.1.1.2">subscript</csymbol><ci id="S3.SS2.p4.4.m4.1.1.2.2.cmml" xref="S3.SS2.p4.4.m4.1.1.2.2">𝑑</ci><ci id="S3.SS2.p4.4.m4.1.1.2.3a.cmml" xref="S3.SS2.p4.4.m4.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.p4.4.m4.1.1.2.3.cmml" xref="S3.SS2.p4.4.m4.1.1.2.3">head</mtext></ci></apply><apply id="S3.SS2.p4.4.m4.1.1.3.cmml" xref="S3.SS2.p4.4.m4.1.1.3"><divide id="S3.SS2.p4.4.m4.1.1.3.1.cmml" xref="S3.SS2.p4.4.m4.1.1.3.1"></divide><ci id="S3.SS2.p4.4.m4.1.1.3.2.cmml" xref="S3.SS2.p4.4.m4.1.1.3.2">𝑑</ci><ci id="S3.SS2.p4.4.m4.1.1.3.3.cmml" xref="S3.SS2.p4.4.m4.1.1.3.3">𝑍</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.4.m4.1c">d_{\text{head}}=d/Z</annotation></semantics></math>. When the <math id="S3.SS2.p4.5.m5.1" class="ltx_Math" alttext="m^{\text{S}}" display="inline"><semantics id="S3.SS2.p4.5.m5.1a"><msup id="S3.SS2.p4.5.m5.1.1" xref="S3.SS2.p4.5.m5.1.1.cmml"><mi id="S3.SS2.p4.5.m5.1.1.2" xref="S3.SS2.p4.5.m5.1.1.2.cmml">m</mi><mtext id="S3.SS2.p4.5.m5.1.1.3" xref="S3.SS2.p4.5.m5.1.1.3a.cmml">S</mtext></msup><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.5.m5.1b"><apply id="S3.SS2.p4.5.m5.1.1.cmml" xref="S3.SS2.p4.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS2.p4.5.m5.1.1.1.cmml" xref="S3.SS2.p4.5.m5.1.1">superscript</csymbol><ci id="S3.SS2.p4.5.m5.1.1.2.cmml" xref="S3.SS2.p4.5.m5.1.1.2">𝑚</ci><ci id="S3.SS2.p4.5.m5.1.1.3a.cmml" xref="S3.SS2.p4.5.m5.1.1.3"><mtext mathsize="70%" id="S3.SS2.p4.5.m5.1.1.3.cmml" xref="S3.SS2.p4.5.m5.1.1.3">S</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.5.m5.1c">m^{\text{S}}</annotation></semantics></math> module is selected, the operation of <math id="S3.SS2.p4.6.m6.1" class="ltx_Math" alttext="\mathcal{F}^{\text{A}}" display="inline"><semantics id="S3.SS2.p4.6.m6.1a"><msup id="S3.SS2.p4.6.m6.1.1" xref="S3.SS2.p4.6.m6.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p4.6.m6.1.1.2" xref="S3.SS2.p4.6.m6.1.1.2.cmml">ℱ</mi><mtext id="S3.SS2.p4.6.m6.1.1.3" xref="S3.SS2.p4.6.m6.1.1.3a.cmml">A</mtext></msup><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.6.m6.1b"><apply id="S3.SS2.p4.6.m6.1.1.cmml" xref="S3.SS2.p4.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS2.p4.6.m6.1.1.1.cmml" xref="S3.SS2.p4.6.m6.1.1">superscript</csymbol><ci id="S3.SS2.p4.6.m6.1.1.2.cmml" xref="S3.SS2.p4.6.m6.1.1.2">ℱ</ci><ci id="S3.SS2.p4.6.m6.1.1.3a.cmml" xref="S3.SS2.p4.6.m6.1.1.3"><mtext mathsize="70%" id="S3.SS2.p4.6.m6.1.1.3.cmml" xref="S3.SS2.p4.6.m6.1.1.3">A</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.6.m6.1c">\mathcal{F}^{\text{A}}</annotation></semantics></math> only involves the remaining <math id="S3.SS2.p4.7.m7.1" class="ltx_Math" alttext="K_{\text{A}}-1" display="inline"><semantics id="S3.SS2.p4.7.m7.1a"><mrow id="S3.SS2.p4.7.m7.1.1" xref="S3.SS2.p4.7.m7.1.1.cmml"><msub id="S3.SS2.p4.7.m7.1.1.2" xref="S3.SS2.p4.7.m7.1.1.2.cmml"><mi id="S3.SS2.p4.7.m7.1.1.2.2" xref="S3.SS2.p4.7.m7.1.1.2.2.cmml">K</mi><mtext id="S3.SS2.p4.7.m7.1.1.2.3" xref="S3.SS2.p4.7.m7.1.1.2.3a.cmml">A</mtext></msub><mo id="S3.SS2.p4.7.m7.1.1.1" xref="S3.SS2.p4.7.m7.1.1.1.cmml">−</mo><mn id="S3.SS2.p4.7.m7.1.1.3" xref="S3.SS2.p4.7.m7.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.7.m7.1b"><apply id="S3.SS2.p4.7.m7.1.1.cmml" xref="S3.SS2.p4.7.m7.1.1"><minus id="S3.SS2.p4.7.m7.1.1.1.cmml" xref="S3.SS2.p4.7.m7.1.1.1"></minus><apply id="S3.SS2.p4.7.m7.1.1.2.cmml" xref="S3.SS2.p4.7.m7.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p4.7.m7.1.1.2.1.cmml" xref="S3.SS2.p4.7.m7.1.1.2">subscript</csymbol><ci id="S3.SS2.p4.7.m7.1.1.2.2.cmml" xref="S3.SS2.p4.7.m7.1.1.2.2">𝐾</ci><ci id="S3.SS2.p4.7.m7.1.1.2.3a.cmml" xref="S3.SS2.p4.7.m7.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.p4.7.m7.1.1.2.3.cmml" xref="S3.SS2.p4.7.m7.1.1.2.3">A</mtext></ci></apply><cn type="integer" id="S3.SS2.p4.7.m7.1.1.3.cmml" xref="S3.SS2.p4.7.m7.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.7.m7.1c">K_{\text{A}}-1</annotation></semantics></math> attention modules.</p>
</div>
<div id="S3.SS2.p5" class="ltx_para ltx_noindent">
<p id="S3.SS2.p5.5" class="ltx_p"><span id="S3.SS2.p5.5.1" class="ltx_text ltx_font_bold">Assembly of feed-forward networks</span> (<math id="S3.SS2.p5.1.m1.1" class="ltx_Math" alttext="\mathcal{F}^{\text{F}}" display="inline"><semantics id="S3.SS2.p5.1.m1.1a"><msup id="S3.SS2.p5.1.m1.1.1" xref="S3.SS2.p5.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p5.1.m1.1.1.2" xref="S3.SS2.p5.1.m1.1.1.2.cmml">ℱ</mi><mtext id="S3.SS2.p5.1.m1.1.1.3" xref="S3.SS2.p5.1.m1.1.1.3a.cmml">F</mtext></msup><annotation-xml encoding="MathML-Content" id="S3.SS2.p5.1.m1.1b"><apply id="S3.SS2.p5.1.m1.1.1.cmml" xref="S3.SS2.p5.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p5.1.m1.1.1.1.cmml" xref="S3.SS2.p5.1.m1.1.1">superscript</csymbol><ci id="S3.SS2.p5.1.m1.1.1.2.cmml" xref="S3.SS2.p5.1.m1.1.1.2">ℱ</ci><ci id="S3.SS2.p5.1.m1.1.1.3a.cmml" xref="S3.SS2.p5.1.m1.1.1.3"><mtext mathsize="70%" id="S3.SS2.p5.1.m1.1.1.3.cmml" xref="S3.SS2.p5.1.m1.1.1.3">F</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p5.1.m1.1c">\mathcal{F}^{\text{F}}</annotation></semantics></math>).  The assembly of <math id="S3.SS2.p5.2.m2.1" class="ltx_Math" alttext="\mathcal{F}^{\text{F}}" display="inline"><semantics id="S3.SS2.p5.2.m2.1a"><msup id="S3.SS2.p5.2.m2.1.1" xref="S3.SS2.p5.2.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p5.2.m2.1.1.2" xref="S3.SS2.p5.2.m2.1.1.2.cmml">ℱ</mi><mtext id="S3.SS2.p5.2.m2.1.1.3" xref="S3.SS2.p5.2.m2.1.1.3a.cmml">F</mtext></msup><annotation-xml encoding="MathML-Content" id="S3.SS2.p5.2.m2.1b"><apply id="S3.SS2.p5.2.m2.1.1.cmml" xref="S3.SS2.p5.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.p5.2.m2.1.1.1.cmml" xref="S3.SS2.p5.2.m2.1.1">superscript</csymbol><ci id="S3.SS2.p5.2.m2.1.1.2.cmml" xref="S3.SS2.p5.2.m2.1.1.2">ℱ</ci><ci id="S3.SS2.p5.2.m2.1.1.3a.cmml" xref="S3.SS2.p5.2.m2.1.1.3"><mtext mathsize="70%" id="S3.SS2.p5.2.m2.1.1.3.cmml" xref="S3.SS2.p5.2.m2.1.1.3">F</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p5.2.m2.1c">\mathcal{F}^{\text{F}}</annotation></semantics></math> is more modular, where the outputs of <math id="S3.SS2.p5.3.m3.1" class="ltx_Math" alttext="K_{\text{F}}" display="inline"><semantics id="S3.SS2.p5.3.m3.1a"><msub id="S3.SS2.p5.3.m3.1.1" xref="S3.SS2.p5.3.m3.1.1.cmml"><mi id="S3.SS2.p5.3.m3.1.1.2" xref="S3.SS2.p5.3.m3.1.1.2.cmml">K</mi><mtext id="S3.SS2.p5.3.m3.1.1.3" xref="S3.SS2.p5.3.m3.1.1.3a.cmml">F</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p5.3.m3.1b"><apply id="S3.SS2.p5.3.m3.1.1.cmml" xref="S3.SS2.p5.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.p5.3.m3.1.1.1.cmml" xref="S3.SS2.p5.3.m3.1.1">subscript</csymbol><ci id="S3.SS2.p5.3.m3.1.1.2.cmml" xref="S3.SS2.p5.3.m3.1.1.2">𝐾</ci><ci id="S3.SS2.p5.3.m3.1.1.3a.cmml" xref="S3.SS2.p5.3.m3.1.1.3"><mtext mathsize="70%" id="S3.SS2.p5.3.m3.1.1.3.cmml" xref="S3.SS2.p5.3.m3.1.1.3">F</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p5.3.m3.1c">K_{\text{F}}</annotation></semantics></math> modules are simply weighted and aggregated.
When the <math id="S3.SS2.p5.4.m4.1" class="ltx_Math" alttext="m^{\text{S}}" display="inline"><semantics id="S3.SS2.p5.4.m4.1a"><msup id="S3.SS2.p5.4.m4.1.1" xref="S3.SS2.p5.4.m4.1.1.cmml"><mi id="S3.SS2.p5.4.m4.1.1.2" xref="S3.SS2.p5.4.m4.1.1.2.cmml">m</mi><mtext id="S3.SS2.p5.4.m4.1.1.3" xref="S3.SS2.p5.4.m4.1.1.3a.cmml">S</mtext></msup><annotation-xml encoding="MathML-Content" id="S3.SS2.p5.4.m4.1b"><apply id="S3.SS2.p5.4.m4.1.1.cmml" xref="S3.SS2.p5.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS2.p5.4.m4.1.1.1.cmml" xref="S3.SS2.p5.4.m4.1.1">superscript</csymbol><ci id="S3.SS2.p5.4.m4.1.1.2.cmml" xref="S3.SS2.p5.4.m4.1.1.2">𝑚</ci><ci id="S3.SS2.p5.4.m4.1.1.3a.cmml" xref="S3.SS2.p5.4.m4.1.1.3"><mtext mathsize="70%" id="S3.SS2.p5.4.m4.1.1.3.cmml" xref="S3.SS2.p5.4.m4.1.1.3">S</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p5.4.m4.1c">m^{\text{S}}</annotation></semantics></math> module is not selected, <math id="S3.SS2.p5.5.m5.1" class="ltx_Math" alttext="\mathcal{F}^{\text{F}}" display="inline"><semantics id="S3.SS2.p5.5.m5.1a"><msup id="S3.SS2.p5.5.m5.1.1" xref="S3.SS2.p5.5.m5.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p5.5.m5.1.1.2" xref="S3.SS2.p5.5.m5.1.1.2.cmml">ℱ</mi><mtext id="S3.SS2.p5.5.m5.1.1.3" xref="S3.SS2.p5.5.m5.1.1.3a.cmml">F</mtext></msup><annotation-xml encoding="MathML-Content" id="S3.SS2.p5.5.m5.1b"><apply id="S3.SS2.p5.5.m5.1.1.cmml" xref="S3.SS2.p5.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS2.p5.5.m5.1.1.1.cmml" xref="S3.SS2.p5.5.m5.1.1">superscript</csymbol><ci id="S3.SS2.p5.5.m5.1.1.2.cmml" xref="S3.SS2.p5.5.m5.1.1.2">ℱ</ci><ci id="S3.SS2.p5.5.m5.1.1.3a.cmml" xref="S3.SS2.p5.5.m5.1.1.3"><mtext mathsize="70%" id="S3.SS2.p5.5.m5.1.1.3.cmml" xref="S3.SS2.p5.5.m5.1.1.3">F</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p5.5.m5.1c">\mathcal{F}^{\text{F}}</annotation></semantics></math> can be formalized as follows:</p>
<table id="S3.E6" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S3.E6X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E6X.2.1.1.m1.3" class="ltx_Math" alttext="\displaystyle\mathcal{F}^{\text{F}}(\mathbf{u}):=\sum_{k\in\mathcal{K}_{\text{F}}}r^{\text{F}}_{k}\cdot m^{\text{F}}_{k}(\mathbf{u})," display="inline"><semantics id="S3.E6X.2.1.1.m1.3a"><mrow id="S3.E6X.2.1.1.m1.3.3.1" xref="S3.E6X.2.1.1.m1.3.3.1.1.cmml"><mrow id="S3.E6X.2.1.1.m1.3.3.1.1" xref="S3.E6X.2.1.1.m1.3.3.1.1.cmml"><mrow id="S3.E6X.2.1.1.m1.3.3.1.1.2" xref="S3.E6X.2.1.1.m1.3.3.1.1.2.cmml"><msup id="S3.E6X.2.1.1.m1.3.3.1.1.2.2" xref="S3.E6X.2.1.1.m1.3.3.1.1.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E6X.2.1.1.m1.3.3.1.1.2.2.2" xref="S3.E6X.2.1.1.m1.3.3.1.1.2.2.2.cmml">ℱ</mi><mtext id="S3.E6X.2.1.1.m1.3.3.1.1.2.2.3" xref="S3.E6X.2.1.1.m1.3.3.1.1.2.2.3a.cmml">F</mtext></msup><mo lspace="0em" rspace="0em" id="S3.E6X.2.1.1.m1.3.3.1.1.2.1" xref="S3.E6X.2.1.1.m1.3.3.1.1.2.1.cmml">​</mo><mrow id="S3.E6X.2.1.1.m1.3.3.1.1.2.3.2" xref="S3.E6X.2.1.1.m1.3.3.1.1.2.cmml"><mo stretchy="false" id="S3.E6X.2.1.1.m1.3.3.1.1.2.3.2.1" xref="S3.E6X.2.1.1.m1.3.3.1.1.2.cmml">(</mo><mi id="S3.E6X.2.1.1.m1.1.1" xref="S3.E6X.2.1.1.m1.1.1.cmml">𝐮</mi><mo rspace="0.278em" stretchy="false" id="S3.E6X.2.1.1.m1.3.3.1.1.2.3.2.2" xref="S3.E6X.2.1.1.m1.3.3.1.1.2.cmml">)</mo></mrow></mrow><mo rspace="0.278em" id="S3.E6X.2.1.1.m1.3.3.1.1.1" xref="S3.E6X.2.1.1.m1.3.3.1.1.1.cmml">:=</mo><mrow id="S3.E6X.2.1.1.m1.3.3.1.1.3" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.cmml"><mstyle displaystyle="true" id="S3.E6X.2.1.1.m1.3.3.1.1.3.1" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.1.cmml"><munder id="S3.E6X.2.1.1.m1.3.3.1.1.3.1a" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.1.cmml"><mo movablelimits="false" id="S3.E6X.2.1.1.m1.3.3.1.1.3.1.2" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.1.2.cmml">∑</mo><mrow id="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.cmml"><mi id="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.2" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.2.cmml">k</mi><mo id="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.1" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.1.cmml">∈</mo><msub id="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.3" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.3.2" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.3.2.cmml">𝒦</mi><mtext id="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.3.3" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.3.3a.cmml">F</mtext></msub></mrow></munder></mstyle><mrow id="S3.E6X.2.1.1.m1.3.3.1.1.3.2" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.cmml"><mrow id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.cmml"><msubsup id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2.cmml"><mi id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2.2.2" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2.2.2.cmml">r</mi><mi id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2.3" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2.3.cmml">k</mi><mtext id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2.2.3" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2.2.3a.cmml">F</mtext></msubsup><mo lspace="0.222em" rspace="0.222em" id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.1" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.1.cmml">⋅</mo><msubsup id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3.cmml"><mi id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3.2.2" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3.2.2.cmml">m</mi><mi id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3.3" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3.3.cmml">k</mi><mtext id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3.2.3" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3.2.3a.cmml">F</mtext></msubsup></mrow><mo lspace="0em" rspace="0em" id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.1" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.1.cmml">​</mo><mrow id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.3.2" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.cmml"><mo stretchy="false" id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.3.2.1" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.cmml">(</mo><mi id="S3.E6X.2.1.1.m1.2.2" xref="S3.E6X.2.1.1.m1.2.2.cmml">𝐮</mi><mo stretchy="false" id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.3.2.2" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.cmml">)</mo></mrow></mrow></mrow></mrow><mo id="S3.E6X.2.1.1.m1.3.3.1.2" xref="S3.E6X.2.1.1.m1.3.3.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E6X.2.1.1.m1.3b"><apply id="S3.E6X.2.1.1.m1.3.3.1.1.cmml" xref="S3.E6X.2.1.1.m1.3.3.1"><csymbol cd="latexml" id="S3.E6X.2.1.1.m1.3.3.1.1.1.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.1">assign</csymbol><apply id="S3.E6X.2.1.1.m1.3.3.1.1.2.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.2"><times id="S3.E6X.2.1.1.m1.3.3.1.1.2.1.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.2.1"></times><apply id="S3.E6X.2.1.1.m1.3.3.1.1.2.2.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.2.2"><csymbol cd="ambiguous" id="S3.E6X.2.1.1.m1.3.3.1.1.2.2.1.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.2.2">superscript</csymbol><ci id="S3.E6X.2.1.1.m1.3.3.1.1.2.2.2.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.2.2.2">ℱ</ci><ci id="S3.E6X.2.1.1.m1.3.3.1.1.2.2.3a.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.2.2.3"><mtext mathsize="70%" id="S3.E6X.2.1.1.m1.3.3.1.1.2.2.3.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.2.2.3">F</mtext></ci></apply><ci id="S3.E6X.2.1.1.m1.1.1.cmml" xref="S3.E6X.2.1.1.m1.1.1">𝐮</ci></apply><apply id="S3.E6X.2.1.1.m1.3.3.1.1.3.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3"><apply id="S3.E6X.2.1.1.m1.3.3.1.1.3.1.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.1"><csymbol cd="ambiguous" id="S3.E6X.2.1.1.m1.3.3.1.1.3.1.1.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.1">subscript</csymbol><sum id="S3.E6X.2.1.1.m1.3.3.1.1.3.1.2.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.1.2"></sum><apply id="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3"><in id="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.1.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.1"></in><ci id="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.2.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.2">𝑘</ci><apply id="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.3.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.3"><csymbol cd="ambiguous" id="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.3.1.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.3">subscript</csymbol><ci id="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.3.2.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.3.2">𝒦</ci><ci id="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.3.3a.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.3.3"><mtext mathsize="50%" id="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.3.3.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.1.3.3.3">F</mtext></ci></apply></apply></apply><apply id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2"><times id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.1.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.1"></times><apply id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2"><ci id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.1.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.1">⋅</ci><apply id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2"><csymbol cd="ambiguous" id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2.1.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2">subscript</csymbol><apply id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2.2.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2"><csymbol cd="ambiguous" id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2.2.1.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2">superscript</csymbol><ci id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2.2.2.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2.2.2">𝑟</ci><ci id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2.2.3a.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2.2.3"><mtext mathsize="70%" id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2.2.3.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2.2.3">F</mtext></ci></apply><ci id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2.3.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.2.3">𝑘</ci></apply><apply id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3"><csymbol cd="ambiguous" id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3.1.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3">subscript</csymbol><apply id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3.2.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3"><csymbol cd="ambiguous" id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3.2.1.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3">superscript</csymbol><ci id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3.2.2.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3.2.2">𝑚</ci><ci id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3.2.3a.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3.2.3"><mtext mathsize="70%" id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3.2.3.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3.2.3">F</mtext></ci></apply><ci id="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3.3.cmml" xref="S3.E6X.2.1.1.m1.3.3.1.1.3.2.2.3.3">𝑘</ci></apply></apply><ci id="S3.E6X.2.1.1.m1.2.2.cmml" xref="S3.E6X.2.1.1.m1.2.2">𝐮</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E6X.2.1.1.m1.3c">\displaystyle\mathcal{F}^{\text{F}}(\mathbf{u}):=\sum_{k\in\mathcal{K}_{\text{F}}}r^{\text{F}}_{k}\cdot m^{\text{F}}_{k}(\mathbf{u}),</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="2" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(6)</span></td>
</tr>
<tr id="S3.E6Xa" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E6Xa.2.1.1.m1.4" class="ltx_Math" alttext="\displaystyle\mathbf{r}^{\text{F}}=\mathcal{R}^{\text{F}}(\mathbf{x})=\left(r^{\text{F}}_{1},\ldots,r^{\text{F}}_{k},\ldots,r^{\text{F}}_{K_{F}}\right)." display="inline"><semantics id="S3.E6Xa.2.1.1.m1.4a"><mrow id="S3.E6Xa.2.1.1.m1.4.4.1" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.cmml"><mrow id="S3.E6Xa.2.1.1.m1.4.4.1.1" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.cmml"><msup id="S3.E6Xa.2.1.1.m1.4.4.1.1.5" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.5.cmml"><mi id="S3.E6Xa.2.1.1.m1.4.4.1.1.5.2" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.5.2.cmml">𝐫</mi><mtext id="S3.E6Xa.2.1.1.m1.4.4.1.1.5.3" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.5.3a.cmml">F</mtext></msup><mo id="S3.E6Xa.2.1.1.m1.4.4.1.1.6" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.6.cmml">=</mo><mrow id="S3.E6Xa.2.1.1.m1.4.4.1.1.7" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.7.cmml"><msup id="S3.E6Xa.2.1.1.m1.4.4.1.1.7.2" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.7.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E6Xa.2.1.1.m1.4.4.1.1.7.2.2" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.7.2.2.cmml">ℛ</mi><mtext id="S3.E6Xa.2.1.1.m1.4.4.1.1.7.2.3" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.7.2.3a.cmml">F</mtext></msup><mo lspace="0em" rspace="0em" id="S3.E6Xa.2.1.1.m1.4.4.1.1.7.1" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.7.1.cmml">​</mo><mrow id="S3.E6Xa.2.1.1.m1.4.4.1.1.7.3.2" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.7.cmml"><mo stretchy="false" id="S3.E6Xa.2.1.1.m1.4.4.1.1.7.3.2.1" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.7.cmml">(</mo><mi id="S3.E6Xa.2.1.1.m1.1.1" xref="S3.E6Xa.2.1.1.m1.1.1.cmml">𝐱</mi><mo stretchy="false" id="S3.E6Xa.2.1.1.m1.4.4.1.1.7.3.2.2" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.7.cmml">)</mo></mrow></mrow><mo id="S3.E6Xa.2.1.1.m1.4.4.1.1.8" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.8.cmml">=</mo><mrow id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.4.cmml"><mo id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.4" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.4.cmml">(</mo><msubsup id="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.cmml"><mi id="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.cmml">r</mi><mn id="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.cmml">1</mn><mtext id="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.3" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.3a.cmml">F</mtext></msubsup><mo id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.5" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.4.cmml">,</mo><mi mathvariant="normal" id="S3.E6Xa.2.1.1.m1.2.2" xref="S3.E6Xa.2.1.1.m1.2.2.cmml">…</mi><mo id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.6" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.4.cmml">,</mo><msubsup id="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2.cmml"><mi id="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2.2.2" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2.2.2.cmml">r</mi><mi id="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2.3" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2.3.cmml">k</mi><mtext id="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2.2.3" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2.2.3a.cmml">F</mtext></msubsup><mo id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.7" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.4.cmml">,</mo><mi mathvariant="normal" id="S3.E6Xa.2.1.1.m1.3.3" xref="S3.E6Xa.2.1.1.m1.3.3.cmml">…</mi><mo id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.8" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.4.cmml">,</mo><msubsup id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.cmml"><mi id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.2.2" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.2.2.cmml">r</mi><msub id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.3" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.3.cmml"><mi id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.3.2" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.3.2.cmml">K</mi><mi id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.3.3" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.3.3.cmml">F</mi></msub><mtext id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.2.3" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.2.3a.cmml">F</mtext></msubsup><mo id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.9" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.4.cmml">)</mo></mrow></mrow><mo lspace="0em" id="S3.E6Xa.2.1.1.m1.4.4.1.2" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E6Xa.2.1.1.m1.4b"><apply id="S3.E6Xa.2.1.1.m1.4.4.1.1.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1"><and id="S3.E6Xa.2.1.1.m1.4.4.1.1a.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1"></and><apply id="S3.E6Xa.2.1.1.m1.4.4.1.1b.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1"><eq id="S3.E6Xa.2.1.1.m1.4.4.1.1.6.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.6"></eq><apply id="S3.E6Xa.2.1.1.m1.4.4.1.1.5.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.5"><csymbol cd="ambiguous" id="S3.E6Xa.2.1.1.m1.4.4.1.1.5.1.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.5">superscript</csymbol><ci id="S3.E6Xa.2.1.1.m1.4.4.1.1.5.2.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.5.2">𝐫</ci><ci id="S3.E6Xa.2.1.1.m1.4.4.1.1.5.3a.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.5.3"><mtext mathsize="70%" id="S3.E6Xa.2.1.1.m1.4.4.1.1.5.3.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.5.3">F</mtext></ci></apply><apply id="S3.E6Xa.2.1.1.m1.4.4.1.1.7.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.7"><times id="S3.E6Xa.2.1.1.m1.4.4.1.1.7.1.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.7.1"></times><apply id="S3.E6Xa.2.1.1.m1.4.4.1.1.7.2.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.7.2"><csymbol cd="ambiguous" id="S3.E6Xa.2.1.1.m1.4.4.1.1.7.2.1.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.7.2">superscript</csymbol><ci id="S3.E6Xa.2.1.1.m1.4.4.1.1.7.2.2.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.7.2.2">ℛ</ci><ci id="S3.E6Xa.2.1.1.m1.4.4.1.1.7.2.3a.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.7.2.3"><mtext mathsize="70%" id="S3.E6Xa.2.1.1.m1.4.4.1.1.7.2.3.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.7.2.3">F</mtext></ci></apply><ci id="S3.E6Xa.2.1.1.m1.1.1.cmml" xref="S3.E6Xa.2.1.1.m1.1.1">𝐱</ci></apply></apply><apply id="S3.E6Xa.2.1.1.m1.4.4.1.1c.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1"><eq id="S3.E6Xa.2.1.1.m1.4.4.1.1.8.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.8"></eq><share href="#S3.E6Xa.2.1.1.m1.4.4.1.1.7.cmml" id="S3.E6Xa.2.1.1.m1.4.4.1.1d.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1"></share><vector id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.4.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3"><apply id="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.1.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1">subscript</csymbol><apply id="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.1.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1">superscript</csymbol><ci id="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.2">𝑟</ci><ci id="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.3a.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.3"><mtext mathsize="70%" id="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.3.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.2.3">F</mtext></ci></apply><cn type="integer" id="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.1.1.1.3">1</cn></apply><ci id="S3.E6Xa.2.1.1.m1.2.2.cmml" xref="S3.E6Xa.2.1.1.m1.2.2">…</ci><apply id="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2"><csymbol cd="ambiguous" id="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2.1.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2">subscript</csymbol><apply id="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2.2.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2"><csymbol cd="ambiguous" id="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2.2.1.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2">superscript</csymbol><ci id="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2.2.2.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2.2.2">𝑟</ci><ci id="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2.2.3a.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2.2.3"><mtext mathsize="70%" id="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2.2.3.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2.2.3">F</mtext></ci></apply><ci id="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2.3.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.2.2.2.3">𝑘</ci></apply><ci id="S3.E6Xa.2.1.1.m1.3.3.cmml" xref="S3.E6Xa.2.1.1.m1.3.3">…</ci><apply id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3"><csymbol cd="ambiguous" id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.1.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3">subscript</csymbol><apply id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.2.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3"><csymbol cd="ambiguous" id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.2.1.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3">superscript</csymbol><ci id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.2.2.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.2.2">𝑟</ci><ci id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.2.3a.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.2.3"><mtext mathsize="70%" id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.2.3.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.2.3">F</mtext></ci></apply><apply id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.3.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.3"><csymbol cd="ambiguous" id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.3.1.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.3">subscript</csymbol><ci id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.3.2.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.3.2">𝐾</ci><ci id="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.3.3.cmml" xref="S3.E6Xa.2.1.1.m1.4.4.1.1.3.3.3.3.3">𝐹</ci></apply></apply></vector></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E6Xa.2.1.1.m1.4c">\displaystyle\mathbf{r}^{\text{F}}=\mathcal{R}^{\text{F}}(\mathbf{x})=\left(r^{\text{F}}_{1},\ldots,r^{\text{F}}_{k},\ldots,r^{\text{F}}_{K_{F}}\right).</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
</tbody>
</table>
<p id="S3.SS2.p5.8" class="ltx_p">When <math id="S3.SS2.p5.6.m1.1" class="ltx_Math" alttext="m^{\text{S}}" display="inline"><semantics id="S3.SS2.p5.6.m1.1a"><msup id="S3.SS2.p5.6.m1.1.1" xref="S3.SS2.p5.6.m1.1.1.cmml"><mi id="S3.SS2.p5.6.m1.1.1.2" xref="S3.SS2.p5.6.m1.1.1.2.cmml">m</mi><mtext id="S3.SS2.p5.6.m1.1.1.3" xref="S3.SS2.p5.6.m1.1.1.3a.cmml">S</mtext></msup><annotation-xml encoding="MathML-Content" id="S3.SS2.p5.6.m1.1b"><apply id="S3.SS2.p5.6.m1.1.1.cmml" xref="S3.SS2.p5.6.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p5.6.m1.1.1.1.cmml" xref="S3.SS2.p5.6.m1.1.1">superscript</csymbol><ci id="S3.SS2.p5.6.m1.1.1.2.cmml" xref="S3.SS2.p5.6.m1.1.1.2">𝑚</ci><ci id="S3.SS2.p5.6.m1.1.1.3a.cmml" xref="S3.SS2.p5.6.m1.1.1.3"><mtext mathsize="70%" id="S3.SS2.p5.6.m1.1.1.3.cmml" xref="S3.SS2.p5.6.m1.1.1.3">S</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p5.6.m1.1c">m^{\text{S}}</annotation></semantics></math> is chosen, likewise, only <math id="S3.SS2.p5.7.m2.1" class="ltx_Math" alttext="K_{\text{F}}-1" display="inline"><semantics id="S3.SS2.p5.7.m2.1a"><mrow id="S3.SS2.p5.7.m2.1.1" xref="S3.SS2.p5.7.m2.1.1.cmml"><msub id="S3.SS2.p5.7.m2.1.1.2" xref="S3.SS2.p5.7.m2.1.1.2.cmml"><mi id="S3.SS2.p5.7.m2.1.1.2.2" xref="S3.SS2.p5.7.m2.1.1.2.2.cmml">K</mi><mtext id="S3.SS2.p5.7.m2.1.1.2.3" xref="S3.SS2.p5.7.m2.1.1.2.3a.cmml">F</mtext></msub><mo id="S3.SS2.p5.7.m2.1.1.1" xref="S3.SS2.p5.7.m2.1.1.1.cmml">−</mo><mn id="S3.SS2.p5.7.m2.1.1.3" xref="S3.SS2.p5.7.m2.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p5.7.m2.1b"><apply id="S3.SS2.p5.7.m2.1.1.cmml" xref="S3.SS2.p5.7.m2.1.1"><minus id="S3.SS2.p5.7.m2.1.1.1.cmml" xref="S3.SS2.p5.7.m2.1.1.1"></minus><apply id="S3.SS2.p5.7.m2.1.1.2.cmml" xref="S3.SS2.p5.7.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p5.7.m2.1.1.2.1.cmml" xref="S3.SS2.p5.7.m2.1.1.2">subscript</csymbol><ci id="S3.SS2.p5.7.m2.1.1.2.2.cmml" xref="S3.SS2.p5.7.m2.1.1.2.2">𝐾</ci><ci id="S3.SS2.p5.7.m2.1.1.2.3a.cmml" xref="S3.SS2.p5.7.m2.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.p5.7.m2.1.1.2.3.cmml" xref="S3.SS2.p5.7.m2.1.1.2.3">F</mtext></ci></apply><cn type="integer" id="S3.SS2.p5.7.m2.1.1.3.cmml" xref="S3.SS2.p5.7.m2.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p5.7.m2.1c">K_{\text{F}}-1</annotation></semantics></math> <span id="S3.SS2.p5.8.1" class="ltx_text ltx_font_typewriter">FFN</span>s form the <math id="S3.SS2.p5.8.m3.1" class="ltx_Math" alttext="\mathcal{F}^{\text{F}}" display="inline"><semantics id="S3.SS2.p5.8.m3.1a"><msup id="S3.SS2.p5.8.m3.1.1" xref="S3.SS2.p5.8.m3.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p5.8.m3.1.1.2" xref="S3.SS2.p5.8.m3.1.1.2.cmml">ℱ</mi><mtext id="S3.SS2.p5.8.m3.1.1.3" xref="S3.SS2.p5.8.m3.1.1.3a.cmml">F</mtext></msup><annotation-xml encoding="MathML-Content" id="S3.SS2.p5.8.m3.1b"><apply id="S3.SS2.p5.8.m3.1.1.cmml" xref="S3.SS2.p5.8.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.p5.8.m3.1.1.1.cmml" xref="S3.SS2.p5.8.m3.1.1">superscript</csymbol><ci id="S3.SS2.p5.8.m3.1.1.2.cmml" xref="S3.SS2.p5.8.m3.1.1.2">ℱ</ci><ci id="S3.SS2.p5.8.m3.1.1.3a.cmml" xref="S3.SS2.p5.8.m3.1.1.3"><mtext mathsize="70%" id="S3.SS2.p5.8.m3.1.1.3.cmml" xref="S3.SS2.p5.8.m3.1.1.3">F</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p5.8.m3.1c">\mathcal{F}^{\text{F}}</annotation></semantics></math>.</p>
</div>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>MoM router (<math id="S3.SS3.1.m1.1" class="ltx_Math" alttext="\mathcal{R}" display="inline"><semantics id="S3.SS3.1.m1.1b"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.1.m1.1.1" xref="S3.SS3.1.m1.1.1.cmml">ℛ</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.1.m1.1c"><ci id="S3.SS3.1.m1.1.1.cmml" xref="S3.SS3.1.m1.1.1">ℛ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.1.m1.1d">\mathcal{R}</annotation></semantics></math>)</h3>

<div id="S3.SS3.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.p1.2" class="ltx_p">In prior approaches, routing occurs as a one-step decision-making process within a layer.
However, in MoM  which possesses a dynamically constructed computation graph, each decision is interdependent with the preceding ones, influencing the entire forward computation.
Consequently, the router in MoM necessitates an awareness of past decisions.
To model such dependency, we employ a gated recurrent unit (GRU, <cite class="ltx_cite ltx_citemacro_citep">(Cho et al., <a href="#bib.bib7" title="" class="ltx_ref">2014</a>)</cite>) as the backbone of routers.
Two routers in MoM are identical in structure.
At each assembly step, the GRU in the <math id="S3.SS3.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{R}^{\mathcal{X}}" display="inline"><semantics id="S3.SS3.p1.1.m1.1a"><msup id="S3.SS3.p1.1.m1.1.1" xref="S3.SS3.p1.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p1.1.m1.1.1.2" xref="S3.SS3.p1.1.m1.1.1.2.cmml">ℛ</mi><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p1.1.m1.1.1.3" xref="S3.SS3.p1.1.m1.1.1.3.cmml">𝒳</mi></msup><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.1.m1.1b"><apply id="S3.SS3.p1.1.m1.1.1.cmml" xref="S3.SS3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p1.1.m1.1.1.1.cmml" xref="S3.SS3.p1.1.m1.1.1">superscript</csymbol><ci id="S3.SS3.p1.1.m1.1.1.2.cmml" xref="S3.SS3.p1.1.m1.1.1.2">ℛ</ci><ci id="S3.SS3.p1.1.m1.1.1.3.cmml" xref="S3.SS3.p1.1.m1.1.1.3">𝒳</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.1.m1.1c">\mathcal{R}^{\mathcal{X}}</annotation></semantics></math> maintains an <math id="S3.SS3.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{s}^{\mathcal{X}}_{h}" display="inline"><semantics id="S3.SS3.p1.2.m2.1a"><msubsup id="S3.SS3.p1.2.m2.1.1" xref="S3.SS3.p1.2.m2.1.1.cmml"><mi id="S3.SS3.p1.2.m2.1.1.2.2" xref="S3.SS3.p1.2.m2.1.1.2.2.cmml">𝐬</mi><mi id="S3.SS3.p1.2.m2.1.1.3" xref="S3.SS3.p1.2.m2.1.1.3.cmml">h</mi><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p1.2.m2.1.1.2.3" xref="S3.SS3.p1.2.m2.1.1.2.3.cmml">𝒳</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.2.m2.1b"><apply id="S3.SS3.p1.2.m2.1.1.cmml" xref="S3.SS3.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS3.p1.2.m2.1.1.1.cmml" xref="S3.SS3.p1.2.m2.1.1">subscript</csymbol><apply id="S3.SS3.p1.2.m2.1.1.2.cmml" xref="S3.SS3.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS3.p1.2.m2.1.1.2.1.cmml" xref="S3.SS3.p1.2.m2.1.1">superscript</csymbol><ci id="S3.SS3.p1.2.m2.1.1.2.2.cmml" xref="S3.SS3.p1.2.m2.1.1.2.2">𝐬</ci><ci id="S3.SS3.p1.2.m2.1.1.2.3.cmml" xref="S3.SS3.p1.2.m2.1.1.2.3">𝒳</ci></apply><ci id="S3.SS3.p1.2.m2.1.1.3.cmml" xref="S3.SS3.p1.2.m2.1.1.3">ℎ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.2.m2.1c">\mathbf{s}^{\mathcal{X}}_{h}</annotation></semantics></math> as the hidden state of the GRU network.
This state is recurrently updated as follows:</p>
<table id="S3.E7" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S3.E7X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E7X.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle\mathbf{s}^{\text{A}}_{h}=\texttt{GRU}^{\text{A}}(\mathbf{x}_{h},\mathbf{s}^{\text{A}}_{h-1})," display="inline"><semantics id="S3.E7X.2.1.1.m1.1a"><mrow id="S3.E7X.2.1.1.m1.1.1.1" xref="S3.E7X.2.1.1.m1.1.1.1.1.cmml"><mrow id="S3.E7X.2.1.1.m1.1.1.1.1" xref="S3.E7X.2.1.1.m1.1.1.1.1.cmml"><msubsup id="S3.E7X.2.1.1.m1.1.1.1.1.4" xref="S3.E7X.2.1.1.m1.1.1.1.1.4.cmml"><mi id="S3.E7X.2.1.1.m1.1.1.1.1.4.2.2" xref="S3.E7X.2.1.1.m1.1.1.1.1.4.2.2.cmml">𝐬</mi><mi id="S3.E7X.2.1.1.m1.1.1.1.1.4.3" xref="S3.E7X.2.1.1.m1.1.1.1.1.4.3.cmml">h</mi><mtext id="S3.E7X.2.1.1.m1.1.1.1.1.4.2.3" xref="S3.E7X.2.1.1.m1.1.1.1.1.4.2.3a.cmml">A</mtext></msubsup><mo id="S3.E7X.2.1.1.m1.1.1.1.1.3" xref="S3.E7X.2.1.1.m1.1.1.1.1.3.cmml">=</mo><mrow id="S3.E7X.2.1.1.m1.1.1.1.1.2" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.cmml"><msup id="S3.E7X.2.1.1.m1.1.1.1.1.2.4" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.4.cmml"><mtext class="ltx_mathvariant_monospace" id="S3.E7X.2.1.1.m1.1.1.1.1.2.4.2" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.4.2a.cmml">GRU</mtext><mtext id="S3.E7X.2.1.1.m1.1.1.1.1.2.4.3" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.4.3a.cmml">A</mtext></msup><mo lspace="0em" rspace="0em" id="S3.E7X.2.1.1.m1.1.1.1.1.2.3" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.3.cmml">​</mo><mrow id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.3.cmml"><mo stretchy="false" id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.3" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.3.cmml">(</mo><msub id="S3.E7X.2.1.1.m1.1.1.1.1.1.1.1.1" xref="S3.E7X.2.1.1.m1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E7X.2.1.1.m1.1.1.1.1.1.1.1.1.2" xref="S3.E7X.2.1.1.m1.1.1.1.1.1.1.1.1.2.cmml">𝐱</mi><mi id="S3.E7X.2.1.1.m1.1.1.1.1.1.1.1.1.3" xref="S3.E7X.2.1.1.m1.1.1.1.1.1.1.1.1.3.cmml">h</mi></msub><mo id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.4" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.3.cmml">,</mo><msubsup id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.cmml"><mi id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.2.2" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.2.2.cmml">𝐬</mi><mrow id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.3" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.3.cmml"><mi id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.3.2" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.3.2.cmml">h</mi><mo id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.3.1" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.3.1.cmml">−</mo><mn id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.3.3" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.3.3.cmml">1</mn></mrow><mtext id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.2.3" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.2.3a.cmml">A</mtext></msubsup><mo stretchy="false" id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.5" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.3.cmml">)</mo></mrow></mrow></mrow><mo id="S3.E7X.2.1.1.m1.1.1.1.2" xref="S3.E7X.2.1.1.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E7X.2.1.1.m1.1b"><apply id="S3.E7X.2.1.1.m1.1.1.1.1.cmml" xref="S3.E7X.2.1.1.m1.1.1.1"><eq id="S3.E7X.2.1.1.m1.1.1.1.1.3.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.3"></eq><apply id="S3.E7X.2.1.1.m1.1.1.1.1.4.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.E7X.2.1.1.m1.1.1.1.1.4.1.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.4">subscript</csymbol><apply id="S3.E7X.2.1.1.m1.1.1.1.1.4.2.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.E7X.2.1.1.m1.1.1.1.1.4.2.1.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.4">superscript</csymbol><ci id="S3.E7X.2.1.1.m1.1.1.1.1.4.2.2.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.4.2.2">𝐬</ci><ci id="S3.E7X.2.1.1.m1.1.1.1.1.4.2.3a.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.4.2.3"><mtext mathsize="70%" id="S3.E7X.2.1.1.m1.1.1.1.1.4.2.3.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.4.2.3">A</mtext></ci></apply><ci id="S3.E7X.2.1.1.m1.1.1.1.1.4.3.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.4.3">ℎ</ci></apply><apply id="S3.E7X.2.1.1.m1.1.1.1.1.2.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.2"><times id="S3.E7X.2.1.1.m1.1.1.1.1.2.3.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.3"></times><apply id="S3.E7X.2.1.1.m1.1.1.1.1.2.4.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.4"><csymbol cd="ambiguous" id="S3.E7X.2.1.1.m1.1.1.1.1.2.4.1.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.4">superscript</csymbol><ci id="S3.E7X.2.1.1.m1.1.1.1.1.2.4.2a.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.4.2"><mtext class="ltx_mathvariant_monospace" id="S3.E7X.2.1.1.m1.1.1.1.1.2.4.2.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.4.2">GRU</mtext></ci><ci id="S3.E7X.2.1.1.m1.1.1.1.1.2.4.3a.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.4.3"><mtext mathsize="70%" id="S3.E7X.2.1.1.m1.1.1.1.1.2.4.3.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.4.3">A</mtext></ci></apply><interval closure="open" id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.3.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2"><apply id="S3.E7X.2.1.1.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E7X.2.1.1.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E7X.2.1.1.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.1.1.1.1.2">𝐱</ci><ci id="S3.E7X.2.1.1.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.1.1.1.1.3">ℎ</ci></apply><apply id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.1.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2">subscript</csymbol><apply id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.2.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.2.1.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2">superscript</csymbol><ci id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.2.2.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.2.2">𝐬</ci><ci id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.2.3a.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.2.3"><mtext mathsize="70%" id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.2.3.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.2.3">A</mtext></ci></apply><apply id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.3.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.3"><minus id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.3.1.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.3.1"></minus><ci id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.3.2.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.3.2">ℎ</ci><cn type="integer" id="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.3.3.cmml" xref="S3.E7X.2.1.1.m1.1.1.1.1.2.2.2.2.3.3">1</cn></apply></apply></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E7X.2.1.1.m1.1c">\displaystyle\mathbf{s}^{\text{A}}_{h}=\texttt{GRU}^{\text{A}}(\mathbf{x}_{h},\mathbf{s}^{\text{A}}_{h-1}),</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="2" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(7)</span></td>
</tr>
<tr id="S3.E7Xa" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E7Xa.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle\mathbf{s}^{\text{F}}_{h}=\texttt{GRU}^{\text{F}}(\mathbf{u}_{h},\mathbf{s}^{\text{F}}_{h-1})." display="inline"><semantics id="S3.E7Xa.2.1.1.m1.1a"><mrow id="S3.E7Xa.2.1.1.m1.1.1.1" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.cmml"><mrow id="S3.E7Xa.2.1.1.m1.1.1.1.1" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.cmml"><msubsup id="S3.E7Xa.2.1.1.m1.1.1.1.1.4" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.4.cmml"><mi id="S3.E7Xa.2.1.1.m1.1.1.1.1.4.2.2" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.4.2.2.cmml">𝐬</mi><mi id="S3.E7Xa.2.1.1.m1.1.1.1.1.4.3" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.4.3.cmml">h</mi><mtext id="S3.E7Xa.2.1.1.m1.1.1.1.1.4.2.3" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.4.2.3a.cmml">F</mtext></msubsup><mo id="S3.E7Xa.2.1.1.m1.1.1.1.1.3" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.3.cmml">=</mo><mrow id="S3.E7Xa.2.1.1.m1.1.1.1.1.2" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.cmml"><msup id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.4" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.4.cmml"><mtext class="ltx_mathvariant_monospace" id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.4.2" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.4.2a.cmml">GRU</mtext><mtext id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.4.3" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.4.3a.cmml">F</mtext></msup><mo lspace="0em" rspace="0em" id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.3" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.3.cmml">​</mo><mrow id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.3.cmml"><mo stretchy="false" id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.3" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.3.cmml">(</mo><msub id="S3.E7Xa.2.1.1.m1.1.1.1.1.1.1.1.1" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E7Xa.2.1.1.m1.1.1.1.1.1.1.1.1.2" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.1.1.1.1.2.cmml">𝐮</mi><mi id="S3.E7Xa.2.1.1.m1.1.1.1.1.1.1.1.1.3" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.1.1.1.1.3.cmml">h</mi></msub><mo id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.4" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.3.cmml">,</mo><msubsup id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.cmml"><mi id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.2.2" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.2.2.cmml">𝐬</mi><mrow id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.3" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.3.cmml"><mi id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.3.2" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.3.2.cmml">h</mi><mo id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.3.1" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.3.1.cmml">−</mo><mn id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.3.3" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.3.3.cmml">1</mn></mrow><mtext id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.2.3" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.2.3a.cmml">F</mtext></msubsup><mo stretchy="false" id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.5" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.3.cmml">)</mo></mrow></mrow></mrow><mo lspace="0em" id="S3.E7Xa.2.1.1.m1.1.1.1.2" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E7Xa.2.1.1.m1.1b"><apply id="S3.E7Xa.2.1.1.m1.1.1.1.1.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1"><eq id="S3.E7Xa.2.1.1.m1.1.1.1.1.3.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.3"></eq><apply id="S3.E7Xa.2.1.1.m1.1.1.1.1.4.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.E7Xa.2.1.1.m1.1.1.1.1.4.1.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.4">subscript</csymbol><apply id="S3.E7Xa.2.1.1.m1.1.1.1.1.4.2.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.E7Xa.2.1.1.m1.1.1.1.1.4.2.1.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.4">superscript</csymbol><ci id="S3.E7Xa.2.1.1.m1.1.1.1.1.4.2.2.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.4.2.2">𝐬</ci><ci id="S3.E7Xa.2.1.1.m1.1.1.1.1.4.2.3a.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.4.2.3"><mtext mathsize="70%" id="S3.E7Xa.2.1.1.m1.1.1.1.1.4.2.3.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.4.2.3">F</mtext></ci></apply><ci id="S3.E7Xa.2.1.1.m1.1.1.1.1.4.3.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.4.3">ℎ</ci></apply><apply id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2"><times id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.3.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.3"></times><apply id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.4.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.4"><csymbol cd="ambiguous" id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.4.1.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.4">superscript</csymbol><ci id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.4.2a.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.4.2"><mtext class="ltx_mathvariant_monospace" id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.4.2.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.4.2">GRU</mtext></ci><ci id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.4.3a.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.4.3"><mtext mathsize="70%" id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.4.3.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.4.3">F</mtext></ci></apply><interval closure="open" id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.3.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2"><apply id="S3.E7Xa.2.1.1.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E7Xa.2.1.1.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E7Xa.2.1.1.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.1.1.1.1.2">𝐮</ci><ci id="S3.E7Xa.2.1.1.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.1.1.1.1.3">ℎ</ci></apply><apply id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.1.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2">subscript</csymbol><apply id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.2.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.2.1.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2">superscript</csymbol><ci id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.2.2.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.2.2">𝐬</ci><ci id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.2.3a.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.2.3"><mtext mathsize="70%" id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.2.3.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.2.3">F</mtext></ci></apply><apply id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.3.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.3"><minus id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.3.1.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.3.1"></minus><ci id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.3.2.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.3.2">ℎ</ci><cn type="integer" id="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.3.3.cmml" xref="S3.E7Xa.2.1.1.m1.1.1.1.1.2.2.2.2.3.3">1</cn></apply></apply></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E7Xa.2.1.1.m1.1c">\displaystyle\mathbf{s}^{\text{F}}_{h}=\texttt{GRU}^{\text{F}}(\mathbf{u}_{h},\mathbf{s}^{\text{F}}_{h-1}).</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
</tbody>
</table>
<p id="S3.SS3.p1.3" class="ltx_p">The weights assigned to each module by <math id="S3.SS3.p1.3.m1.1" class="ltx_Math" alttext="\mathcal{R}^{\mathcal{X}}" display="inline"><semantics id="S3.SS3.p1.3.m1.1a"><msup id="S3.SS3.p1.3.m1.1.1" xref="S3.SS3.p1.3.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p1.3.m1.1.1.2" xref="S3.SS3.p1.3.m1.1.1.2.cmml">ℛ</mi><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p1.3.m1.1.1.3" xref="S3.SS3.p1.3.m1.1.1.3.cmml">𝒳</mi></msup><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.3.m1.1b"><apply id="S3.SS3.p1.3.m1.1.1.cmml" xref="S3.SS3.p1.3.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p1.3.m1.1.1.1.cmml" xref="S3.SS3.p1.3.m1.1.1">superscript</csymbol><ci id="S3.SS3.p1.3.m1.1.1.2.cmml" xref="S3.SS3.p1.3.m1.1.1.2">ℛ</ci><ci id="S3.SS3.p1.3.m1.1.1.3.cmml" xref="S3.SS3.p1.3.m1.1.1.3">𝒳</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.3.m1.1c">\mathcal{R}^{\mathcal{X}}</annotation></semantics></math> are computed as:</p>
<table id="S3.E8" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S3.E8X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E8X.2.1.1.m1.1" class="ltx_Math" alttext="\displaystyle\mathbf{r}^{\mathcal{X}}=\mathbf{W}_{\mathcal{X}}\mathbf{s}^{\mathcal{X}}_{h}," display="inline"><semantics id="S3.E8X.2.1.1.m1.1a"><mrow id="S3.E8X.2.1.1.m1.1.1.1" xref="S3.E8X.2.1.1.m1.1.1.1.1.cmml"><mrow id="S3.E8X.2.1.1.m1.1.1.1.1" xref="S3.E8X.2.1.1.m1.1.1.1.1.cmml"><msup id="S3.E8X.2.1.1.m1.1.1.1.1.2" xref="S3.E8X.2.1.1.m1.1.1.1.1.2.cmml"><mi id="S3.E8X.2.1.1.m1.1.1.1.1.2.2" xref="S3.E8X.2.1.1.m1.1.1.1.1.2.2.cmml">𝐫</mi><mi class="ltx_font_mathcaligraphic" id="S3.E8X.2.1.1.m1.1.1.1.1.2.3" xref="S3.E8X.2.1.1.m1.1.1.1.1.2.3.cmml">𝒳</mi></msup><mo id="S3.E8X.2.1.1.m1.1.1.1.1.1" xref="S3.E8X.2.1.1.m1.1.1.1.1.1.cmml">=</mo><mrow id="S3.E8X.2.1.1.m1.1.1.1.1.3" xref="S3.E8X.2.1.1.m1.1.1.1.1.3.cmml"><msub id="S3.E8X.2.1.1.m1.1.1.1.1.3.2" xref="S3.E8X.2.1.1.m1.1.1.1.1.3.2.cmml"><mi id="S3.E8X.2.1.1.m1.1.1.1.1.3.2.2" xref="S3.E8X.2.1.1.m1.1.1.1.1.3.2.2.cmml">𝐖</mi><mi class="ltx_font_mathcaligraphic" id="S3.E8X.2.1.1.m1.1.1.1.1.3.2.3" xref="S3.E8X.2.1.1.m1.1.1.1.1.3.2.3.cmml">𝒳</mi></msub><mo lspace="0em" rspace="0em" id="S3.E8X.2.1.1.m1.1.1.1.1.3.1" xref="S3.E8X.2.1.1.m1.1.1.1.1.3.1.cmml">​</mo><msubsup id="S3.E8X.2.1.1.m1.1.1.1.1.3.3" xref="S3.E8X.2.1.1.m1.1.1.1.1.3.3.cmml"><mi id="S3.E8X.2.1.1.m1.1.1.1.1.3.3.2.2" xref="S3.E8X.2.1.1.m1.1.1.1.1.3.3.2.2.cmml">𝐬</mi><mi id="S3.E8X.2.1.1.m1.1.1.1.1.3.3.3" xref="S3.E8X.2.1.1.m1.1.1.1.1.3.3.3.cmml">h</mi><mi class="ltx_font_mathcaligraphic" id="S3.E8X.2.1.1.m1.1.1.1.1.3.3.2.3" xref="S3.E8X.2.1.1.m1.1.1.1.1.3.3.2.3.cmml">𝒳</mi></msubsup></mrow></mrow><mo id="S3.E8X.2.1.1.m1.1.1.1.2" xref="S3.E8X.2.1.1.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E8X.2.1.1.m1.1b"><apply id="S3.E8X.2.1.1.m1.1.1.1.1.cmml" xref="S3.E8X.2.1.1.m1.1.1.1"><eq id="S3.E8X.2.1.1.m1.1.1.1.1.1.cmml" xref="S3.E8X.2.1.1.m1.1.1.1.1.1"></eq><apply id="S3.E8X.2.1.1.m1.1.1.1.1.2.cmml" xref="S3.E8X.2.1.1.m1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E8X.2.1.1.m1.1.1.1.1.2.1.cmml" xref="S3.E8X.2.1.1.m1.1.1.1.1.2">superscript</csymbol><ci id="S3.E8X.2.1.1.m1.1.1.1.1.2.2.cmml" xref="S3.E8X.2.1.1.m1.1.1.1.1.2.2">𝐫</ci><ci id="S3.E8X.2.1.1.m1.1.1.1.1.2.3.cmml" xref="S3.E8X.2.1.1.m1.1.1.1.1.2.3">𝒳</ci></apply><apply id="S3.E8X.2.1.1.m1.1.1.1.1.3.cmml" xref="S3.E8X.2.1.1.m1.1.1.1.1.3"><times id="S3.E8X.2.1.1.m1.1.1.1.1.3.1.cmml" xref="S3.E8X.2.1.1.m1.1.1.1.1.3.1"></times><apply id="S3.E8X.2.1.1.m1.1.1.1.1.3.2.cmml" xref="S3.E8X.2.1.1.m1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S3.E8X.2.1.1.m1.1.1.1.1.3.2.1.cmml" xref="S3.E8X.2.1.1.m1.1.1.1.1.3.2">subscript</csymbol><ci id="S3.E8X.2.1.1.m1.1.1.1.1.3.2.2.cmml" xref="S3.E8X.2.1.1.m1.1.1.1.1.3.2.2">𝐖</ci><ci id="S3.E8X.2.1.1.m1.1.1.1.1.3.2.3.cmml" xref="S3.E8X.2.1.1.m1.1.1.1.1.3.2.3">𝒳</ci></apply><apply id="S3.E8X.2.1.1.m1.1.1.1.1.3.3.cmml" xref="S3.E8X.2.1.1.m1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S3.E8X.2.1.1.m1.1.1.1.1.3.3.1.cmml" xref="S3.E8X.2.1.1.m1.1.1.1.1.3.3">subscript</csymbol><apply id="S3.E8X.2.1.1.m1.1.1.1.1.3.3.2.cmml" xref="S3.E8X.2.1.1.m1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S3.E8X.2.1.1.m1.1.1.1.1.3.3.2.1.cmml" xref="S3.E8X.2.1.1.m1.1.1.1.1.3.3">superscript</csymbol><ci id="S3.E8X.2.1.1.m1.1.1.1.1.3.3.2.2.cmml" xref="S3.E8X.2.1.1.m1.1.1.1.1.3.3.2.2">𝐬</ci><ci id="S3.E8X.2.1.1.m1.1.1.1.1.3.3.2.3.cmml" xref="S3.E8X.2.1.1.m1.1.1.1.1.3.3.2.3">𝒳</ci></apply><ci id="S3.E8X.2.1.1.m1.1.1.1.1.3.3.3.cmml" xref="S3.E8X.2.1.1.m1.1.1.1.1.3.3.3">ℎ</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E8X.2.1.1.m1.1c">\displaystyle\mathbf{r}^{\mathcal{X}}=\mathbf{W}_{\mathcal{X}}\mathbf{s}^{\mathcal{X}}_{h},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="2" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(8)</span></td>
</tr>
<tr id="S3.E8Xa" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E8Xa.2.1.1.m1.3" class="ltx_Math" alttext="\displaystyle\mathbf{W}_{\mathcal{X}}\in\mathbb{R}^{(N_{\mathcal{X}}+1)\times d},\quad\mathcal{X}\in\{\text{A, F}\}." display="inline"><semantics id="S3.E8Xa.2.1.1.m1.3a"><mrow id="S3.E8Xa.2.1.1.m1.3.3.1"><mrow id="S3.E8Xa.2.1.1.m1.3.3.1.1.2" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.3.cmml"><mrow id="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.cmml"><msub id="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.2" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.2.cmml"><mi id="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.2.2" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.2.2.cmml">𝐖</mi><mi class="ltx_font_mathcaligraphic" id="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.2.3" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.2.3.cmml">𝒳</mi></msub><mo id="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.1" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.1.cmml">∈</mo><msup id="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.3" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.3.cmml"><mi id="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.3.2" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.3.2.cmml">ℝ</mi><mrow id="S3.E8Xa.2.1.1.m1.1.1.1" xref="S3.E8Xa.2.1.1.m1.1.1.1.cmml"><mrow id="S3.E8Xa.2.1.1.m1.1.1.1.1.1" xref="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E8Xa.2.1.1.m1.1.1.1.1.1.2" xref="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1" xref="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.cmml"><msub id="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.2" xref="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.2.cmml"><mi id="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.2.2" xref="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.2.2.cmml">N</mi><mi class="ltx_font_mathcaligraphic" id="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.2.3" xref="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.2.3.cmml">𝒳</mi></msub><mo id="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.1" xref="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.1.cmml">+</mo><mn id="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.3" xref="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.3.cmml">1</mn></mrow><mo rspace="0.055em" stretchy="false" id="S3.E8Xa.2.1.1.m1.1.1.1.1.1.3" xref="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.cmml">)</mo></mrow><mo rspace="0.222em" id="S3.E8Xa.2.1.1.m1.1.1.1.2" xref="S3.E8Xa.2.1.1.m1.1.1.1.2.cmml">×</mo><mi id="S3.E8Xa.2.1.1.m1.1.1.1.3" xref="S3.E8Xa.2.1.1.m1.1.1.1.3.cmml">d</mi></mrow></msup></mrow><mo rspace="1.167em" id="S3.E8Xa.2.1.1.m1.3.3.1.1.2.3" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.3a.cmml">,</mo><mrow id="S3.E8Xa.2.1.1.m1.3.3.1.1.2.2" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E8Xa.2.1.1.m1.3.3.1.1.2.2.2" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.2.2.2.cmml">𝒳</mi><mo id="S3.E8Xa.2.1.1.m1.3.3.1.1.2.2.1" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.2.2.1.cmml">∈</mo><mrow id="S3.E8Xa.2.1.1.m1.3.3.1.1.2.2.3.2" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.2.2.3.1.cmml"><mo stretchy="false" id="S3.E8Xa.2.1.1.m1.3.3.1.1.2.2.3.2.1" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.2.2.3.1.cmml">{</mo><mtext id="S3.E8Xa.2.1.1.m1.2.2" xref="S3.E8Xa.2.1.1.m1.2.2a.cmml">A, F</mtext><mo stretchy="false" id="S3.E8Xa.2.1.1.m1.3.3.1.1.2.2.3.2.2" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.2.2.3.1.cmml">}</mo></mrow></mrow></mrow><mo lspace="0em" id="S3.E8Xa.2.1.1.m1.3.3.1.2">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E8Xa.2.1.1.m1.3b"><apply id="S3.E8Xa.2.1.1.m1.3.3.1.1.3.cmml" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.2"><csymbol cd="ambiguous" id="S3.E8Xa.2.1.1.m1.3.3.1.1.3a.cmml" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.2.3">formulae-sequence</csymbol><apply id="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.cmml" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1"><in id="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.1.cmml" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.1"></in><apply id="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.2.cmml" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.2.1.cmml" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.2">subscript</csymbol><ci id="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.2.2.cmml" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.2.2">𝐖</ci><ci id="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.2.3.cmml" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.2.3">𝒳</ci></apply><apply id="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.3.cmml" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.3.1.cmml" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.3">superscript</csymbol><ci id="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.3.2.cmml" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.1.1.3.2">ℝ</ci><apply id="S3.E8Xa.2.1.1.m1.1.1.1.cmml" xref="S3.E8Xa.2.1.1.m1.1.1.1"><times id="S3.E8Xa.2.1.1.m1.1.1.1.2.cmml" xref="S3.E8Xa.2.1.1.m1.1.1.1.2"></times><apply id="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.cmml" xref="S3.E8Xa.2.1.1.m1.1.1.1.1.1"><plus id="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.1.cmml" xref="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.1"></plus><apply id="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.2.cmml" xref="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.2.1.cmml" xref="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.2.2.cmml" xref="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.2.2">𝑁</ci><ci id="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.2.3.cmml" xref="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.2.3">𝒳</ci></apply><cn type="integer" id="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.3.cmml" xref="S3.E8Xa.2.1.1.m1.1.1.1.1.1.1.3">1</cn></apply><ci id="S3.E8Xa.2.1.1.m1.1.1.1.3.cmml" xref="S3.E8Xa.2.1.1.m1.1.1.1.3">𝑑</ci></apply></apply></apply><apply id="S3.E8Xa.2.1.1.m1.3.3.1.1.2.2.cmml" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.2.2"><in id="S3.E8Xa.2.1.1.m1.3.3.1.1.2.2.1.cmml" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.2.2.1"></in><ci id="S3.E8Xa.2.1.1.m1.3.3.1.1.2.2.2.cmml" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.2.2.2">𝒳</ci><set id="S3.E8Xa.2.1.1.m1.3.3.1.1.2.2.3.1.cmml" xref="S3.E8Xa.2.1.1.m1.3.3.1.1.2.2.3.2"><ci id="S3.E8Xa.2.1.1.m1.2.2a.cmml" xref="S3.E8Xa.2.1.1.m1.2.2"><mtext id="S3.E8Xa.2.1.1.m1.2.2.cmml" xref="S3.E8Xa.2.1.1.m1.2.2">A, F</mtext></ci></set></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E8Xa.2.1.1.m1.3c">\displaystyle\mathbf{W}_{\mathcal{X}}\in\mathbb{R}^{(N_{\mathcal{X}}+1)\times d},\quad\mathcal{X}\in\{\text{A, F}\}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="S3.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.4 </span>Training approach</h3>

<div id="S3.SS4.p1" class="ltx_para ltx_noindent">
<p id="S3.SS4.p1.1" class="ltx_p">A straightforward approach is to pre-train an MoM model initialized from scratch.
This approach, however, suffers from a degeneration issue, as the learned functions of modules become homogeneous, making router training challenging.
To address the issue, we propose a two-phase training approach. In the first phase, we pre-train a vanilla Transformer where modules acquire distinct functionalities. Then, in the second phase, we initialize the module set <math id="S3.SS4.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{M}" display="inline"><semantics id="S3.SS4.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS4.p1.1.m1.1.1" xref="S3.SS4.p1.1.m1.1.1.cmml">ℳ</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.1.m1.1b"><ci id="S3.SS4.p1.1.m1.1.1.cmml" xref="S3.SS4.p1.1.m1.1.1">ℳ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.1.m1.1c">\mathcal{M}</annotation></semantics></math> with the pre-trained modules and initialize the routers from scratch. Subsequently, we continue training both modules and routers using the same data and objective as in the first phase. Through empirical studies, we find that the two-phase training method improves the specialization of module functionalities and accelerates router convergence.</p>
</div>
</section>
<section id="S3.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.5 </span>MoM as a unified framework</h3>

<div id="S3.SS5.p1" class="ltx_para ltx_noindent">
<p id="S3.SS5.p1.1" class="ltx_p">A compelling property of MoM is that it unifies a wide range of Transformer-based dynamic computation allocation architectures. With specific configurations, layer-skip (e.g., early-exiting, mixture-of-depths, etc.), parameter sharing, and mixture-of-experts can be viewed as special cases.</p>
</div>
<section id="S3.SS5.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Layer-skip.</h4>

<div id="S3.SS5.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS5.SSS0.Px1.p1.1" class="ltx_p">The key idea is to skip layers according to certain criteria which can either be defined heuristically <cite class="ltx_cite ltx_citemacro_cite">Liu et al. (<a href="#bib.bib25" title="" class="ltx_ref">2024</a>)</cite> or learned from data <cite class="ltx_cite ltx_citemacro_cite">Zeng et al. (<a href="#bib.bib50" title="" class="ltx_ref">2023</a>); Raposo et al. (<a href="#bib.bib37" title="" class="ltx_ref">2024</a>)</cite>.
Within the MoM framework, layer-skip can be formulated as a special cluster of assembly functions <math id="S3.SS5.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="\phi" display="inline"><semantics id="S3.SS5.SSS0.Px1.p1.1.m1.1a"><mi id="S3.SS5.SSS0.Px1.p1.1.m1.1.1" xref="S3.SS5.SSS0.Px1.p1.1.m1.1.1.cmml">ϕ</mi><annotation-xml encoding="MathML-Content" id="S3.SS5.SSS0.Px1.p1.1.m1.1b"><ci id="S3.SS5.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S3.SS5.SSS0.Px1.p1.1.m1.1.1">italic-ϕ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.SSS0.Px1.p1.1.m1.1c">\phi</annotation></semantics></math>, namely:</p>
<table id="S3.E9" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S3.E9X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E9X.2.1.1.m1.2" class="ltx_Math" alttext="\displaystyle\phi_{\text{layer-skip}}(\mathcal{M},\mathcal{R}^{\mathcal{X}},\mathbf{x}_{h})=\begin{cases}m_{h}^{\mathcal{X}}&amp;\text{if}\ c_{\text{skip}}(h)=1\\
m^{S}&amp;\text{if}\ c_{\text{skip}}(h)=0\end{cases}," display="inline"><semantics id="S3.E9X.2.1.1.m1.2a"><mrow id="S3.E9X.2.1.1.m1.2.2.1" xref="S3.E9X.2.1.1.m1.2.2.1.1.cmml"><mrow id="S3.E9X.2.1.1.m1.2.2.1.1" xref="S3.E9X.2.1.1.m1.2.2.1.1.cmml"><mrow id="S3.E9X.2.1.1.m1.2.2.1.1.2" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.cmml"><msub id="S3.E9X.2.1.1.m1.2.2.1.1.2.4" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.4.cmml"><mi id="S3.E9X.2.1.1.m1.2.2.1.1.2.4.2" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.4.2.cmml">ϕ</mi><mtext id="S3.E9X.2.1.1.m1.2.2.1.1.2.4.3" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.4.3a.cmml">layer-skip</mtext></msub><mo lspace="0em" rspace="0em" id="S3.E9X.2.1.1.m1.2.2.1.1.2.3" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.3.cmml">​</mo><mrow id="S3.E9X.2.1.1.m1.2.2.1.1.2.2.2" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.2.3.cmml"><mo stretchy="false" id="S3.E9X.2.1.1.m1.2.2.1.1.2.2.2.3" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.2.3.cmml">(</mo><mi class="ltx_font_mathcaligraphic" id="S3.E9X.2.1.1.m1.1.1" xref="S3.E9X.2.1.1.m1.1.1.cmml">ℳ</mi><mo id="S3.E9X.2.1.1.m1.2.2.1.1.2.2.2.4" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.2.3.cmml">,</mo><msup id="S3.E9X.2.1.1.m1.2.2.1.1.1.1.1.1" xref="S3.E9X.2.1.1.m1.2.2.1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E9X.2.1.1.m1.2.2.1.1.1.1.1.1.2" xref="S3.E9X.2.1.1.m1.2.2.1.1.1.1.1.1.2.cmml">ℛ</mi><mi class="ltx_font_mathcaligraphic" id="S3.E9X.2.1.1.m1.2.2.1.1.1.1.1.1.3" xref="S3.E9X.2.1.1.m1.2.2.1.1.1.1.1.1.3.cmml">𝒳</mi></msup><mo id="S3.E9X.2.1.1.m1.2.2.1.1.2.2.2.5" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.2.3.cmml">,</mo><msub id="S3.E9X.2.1.1.m1.2.2.1.1.2.2.2.2" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.2.2.2.cmml"><mi id="S3.E9X.2.1.1.m1.2.2.1.1.2.2.2.2.2" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.2.2.2.2.cmml">𝐱</mi><mi id="S3.E9X.2.1.1.m1.2.2.1.1.2.2.2.2.3" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.2.2.2.3.cmml">h</mi></msub><mo stretchy="false" id="S3.E9X.2.1.1.m1.2.2.1.1.2.2.2.6" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.2.3.cmml">)</mo></mrow></mrow><mo id="S3.E9X.2.1.1.m1.2.2.1.1.3" xref="S3.E9X.2.1.1.m1.2.2.1.1.3.cmml">=</mo><mrow id="S3.E9.m1.4.4.4.4.4.4a" xref="S3.E9X.2.1.1.m1.2.2.1.1.4.1.cmml"><mo id="S3.E9.m1.4.4.4.4.4.4a.5" xref="S3.E9X.2.1.1.m1.2.2.1.1.4.1.1.cmml">{</mo><mtable columnspacing="5pt" rowspacing="0pt" id="S3.E9.m1.4.4.4.4.4.4.4a" xref="S3.E9X.2.1.1.m1.2.2.1.1.4.1.cmml"><mtr id="S3.E9.m1.4.4.4.4.4.4.4aa" xref="S3.E9X.2.1.1.m1.2.2.1.1.4.1.cmml"><mtd class="ltx_align_left" columnalign="left" id="S3.E9.m1.4.4.4.4.4.4.4ab" xref="S3.E9X.2.1.1.m1.2.2.1.1.4.1.cmml"><msubsup id="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf" xref="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf.cmml"><mi id="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf.2.2" xref="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf.2.2.cmml">m</mi><mi id="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf.2.3" xref="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf.2.3.cmml">h</mi><mi class="ltx_font_mathcaligraphic" id="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf.3" xref="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf.3.cmml">𝒳</mi></msubsup></mtd><mtd class="ltx_align_left" columnalign="left" id="S3.E9.m1.4.4.4.4.4.4.4ac" xref="S3.E9X.2.1.1.m1.2.2.1.1.4.1.cmml"><mrow id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.cmml"><mrow id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.cmml"><mtext id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.2" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.2a.cmml">if</mtext><mo lspace="0.500em" rspace="0em" id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.1" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.1.cmml">​</mo><msub id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.3" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.3.cmml"><mi id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.3.2" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.3.2.cmml">c</mi><mtext id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.3.3" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.3.3a.cmml">skip</mtext></msub><mo lspace="0em" rspace="0em" id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.1a" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.1.cmml">​</mo><mrow id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.4.2" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.cmml"><mo stretchy="false" id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.4.2.1" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.cmml">(</mo><mi id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.1" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.1.cmml">h</mi><mo stretchy="false" id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.4.2.2" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.cmml">)</mo></mrow></mrow><mo id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.2" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.2.cmml">=</mo><mn id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.4" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.4.cmml">1</mn></mrow></mtd></mtr><mtr id="S3.E9.m1.4.4.4.4.4.4.4ad" xref="S3.E9X.2.1.1.m1.2.2.1.1.4.1.cmml"><mtd class="ltx_align_left" columnalign="left" id="S3.E9.m1.4.4.4.4.4.4.4ae" xref="S3.E9X.2.1.1.m1.2.2.1.1.4.1.cmml"><msup id="S3.E9.m1.3.3.3.3.3.3.3.3.1.1.mf" xref="S3.E9.m1.3.3.3.3.3.3.3.3.1.1.mf.cmml"><mi id="S3.E9.m1.3.3.3.3.3.3.3.3.1.1.mf.2" xref="S3.E9.m1.3.3.3.3.3.3.3.3.1.1.mf.2.cmml">m</mi><mi id="S3.E9.m1.3.3.3.3.3.3.3.3.1.1.mf.3" xref="S3.E9.m1.3.3.3.3.3.3.3.3.1.1.mf.3.cmml">S</mi></msup></mtd><mtd class="ltx_align_left" columnalign="left" id="S3.E9.m1.4.4.4.4.4.4.4af" xref="S3.E9X.2.1.1.m1.2.2.1.1.4.1.cmml"><mrow id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.cmml"><mrow id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.cmml"><mtext id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.2" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.2a.cmml">if</mtext><mo lspace="0.500em" rspace="0em" id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.1" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.1.cmml">​</mo><msub id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.3" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.3.cmml"><mi id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.3.2" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.3.2.cmml">c</mi><mtext id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.3.3" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.3.3a.cmml">skip</mtext></msub><mo lspace="0em" rspace="0em" id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.1a" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.1.cmml">​</mo><mrow id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.4.2" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.cmml"><mo stretchy="false" id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.4.2.1" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.cmml">(</mo><mi id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.1" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.1.cmml">h</mi><mo stretchy="false" id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.4.2.2" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.cmml">)</mo></mrow></mrow><mo id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.2" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.2.cmml">=</mo><mn id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.4" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.4.cmml">0</mn></mrow></mtd></mtr></mtable></mrow></mrow><mo id="S3.E9X.2.1.1.m1.2.2.1.2" xref="S3.E9X.2.1.1.m1.2.2.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E9X.2.1.1.m1.2b"><apply id="S3.E9X.2.1.1.m1.2.2.1.1.cmml" xref="S3.E9X.2.1.1.m1.2.2.1"><eq id="S3.E9X.2.1.1.m1.2.2.1.1.3.cmml" xref="S3.E9X.2.1.1.m1.2.2.1.1.3"></eq><apply id="S3.E9X.2.1.1.m1.2.2.1.1.2.cmml" xref="S3.E9X.2.1.1.m1.2.2.1.1.2"><times id="S3.E9X.2.1.1.m1.2.2.1.1.2.3.cmml" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.3"></times><apply id="S3.E9X.2.1.1.m1.2.2.1.1.2.4.cmml" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.4"><csymbol cd="ambiguous" id="S3.E9X.2.1.1.m1.2.2.1.1.2.4.1.cmml" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.4">subscript</csymbol><ci id="S3.E9X.2.1.1.m1.2.2.1.1.2.4.2.cmml" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.4.2">italic-ϕ</ci><ci id="S3.E9X.2.1.1.m1.2.2.1.1.2.4.3a.cmml" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.4.3"><mtext mathsize="70%" id="S3.E9X.2.1.1.m1.2.2.1.1.2.4.3.cmml" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.4.3">layer-skip</mtext></ci></apply><vector id="S3.E9X.2.1.1.m1.2.2.1.1.2.2.3.cmml" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.2.2"><ci id="S3.E9X.2.1.1.m1.1.1.cmml" xref="S3.E9X.2.1.1.m1.1.1">ℳ</ci><apply id="S3.E9X.2.1.1.m1.2.2.1.1.1.1.1.1.cmml" xref="S3.E9X.2.1.1.m1.2.2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E9X.2.1.1.m1.2.2.1.1.1.1.1.1.1.cmml" xref="S3.E9X.2.1.1.m1.2.2.1.1.1.1.1.1">superscript</csymbol><ci id="S3.E9X.2.1.1.m1.2.2.1.1.1.1.1.1.2.cmml" xref="S3.E9X.2.1.1.m1.2.2.1.1.1.1.1.1.2">ℛ</ci><ci id="S3.E9X.2.1.1.m1.2.2.1.1.1.1.1.1.3.cmml" xref="S3.E9X.2.1.1.m1.2.2.1.1.1.1.1.1.3">𝒳</ci></apply><apply id="S3.E9X.2.1.1.m1.2.2.1.1.2.2.2.2.cmml" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S3.E9X.2.1.1.m1.2.2.1.1.2.2.2.2.1.cmml" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.2.2.2">subscript</csymbol><ci id="S3.E9X.2.1.1.m1.2.2.1.1.2.2.2.2.2.cmml" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.2.2.2.2">𝐱</ci><ci id="S3.E9X.2.1.1.m1.2.2.1.1.2.2.2.2.3.cmml" xref="S3.E9X.2.1.1.m1.2.2.1.1.2.2.2.2.3">ℎ</ci></apply></vector></apply><apply id="S3.E9X.2.1.1.m1.2.2.1.1.4.1.cmml" xref="S3.E9.m1.4.4.4.4.4.4a"><csymbol cd="latexml" id="S3.E9X.2.1.1.m1.2.2.1.1.4.1.1.cmml" xref="S3.E9.m1.4.4.4.4.4.4a.5">cases</csymbol><apply id="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf.cmml" xref="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf"><csymbol cd="ambiguous" id="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf.1.cmml" xref="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf">superscript</csymbol><apply id="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf.2.cmml" xref="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf"><csymbol cd="ambiguous" id="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf.2.1.cmml" xref="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf">subscript</csymbol><ci id="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf.2.2.cmml" xref="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf.2.2">𝑚</ci><ci id="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf.2.3.cmml" xref="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf.2.3">ℎ</ci></apply><ci id="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf.3.cmml" xref="S3.E9.m1.1.1.1.1.1.1.1.1.1.1.mf.3">𝒳</ci></apply><apply id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.cmml" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf"><eq id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.2.cmml" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.2"></eq><apply id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.cmml" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3"><times id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.1.cmml" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.1"></times><ci id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.2a.cmml" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.2"><mtext id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.2.cmml" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.2">if</mtext></ci><apply id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.3.cmml" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.3"><csymbol cd="ambiguous" id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.3.1.cmml" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.3">subscript</csymbol><ci id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.3.2.cmml" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.3.2">𝑐</ci><ci id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.3.3a.cmml" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.3.3"><mtext mathsize="70%" id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.3.3.cmml" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.3.3.3">skip</mtext></ci></apply><ci id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.1.cmml" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.1">ℎ</ci></apply><cn type="integer" id="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.4.cmml" xref="S3.E9.m1.2.2.2.2.2.2.2.2.2.1.mf.4">1</cn></apply><apply id="S3.E9.m1.3.3.3.3.3.3.3.3.1.1.mf.cmml" xref="S3.E9.m1.3.3.3.3.3.3.3.3.1.1.mf"><csymbol cd="ambiguous" id="S3.E9.m1.3.3.3.3.3.3.3.3.1.1.mf.1.cmml" xref="S3.E9.m1.3.3.3.3.3.3.3.3.1.1.mf">superscript</csymbol><ci id="S3.E9.m1.3.3.3.3.3.3.3.3.1.1.mf.2.cmml" xref="S3.E9.m1.3.3.3.3.3.3.3.3.1.1.mf.2">𝑚</ci><ci id="S3.E9.m1.3.3.3.3.3.3.3.3.1.1.mf.3.cmml" xref="S3.E9.m1.3.3.3.3.3.3.3.3.1.1.mf.3">𝑆</ci></apply><apply id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.cmml" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf"><eq id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.2.cmml" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.2"></eq><apply id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.cmml" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3"><times id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.1.cmml" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.1"></times><ci id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.2a.cmml" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.2"><mtext id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.2.cmml" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.2">if</mtext></ci><apply id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.3.cmml" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.3"><csymbol cd="ambiguous" id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.3.1.cmml" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.3">subscript</csymbol><ci id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.3.2.cmml" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.3.2">𝑐</ci><ci id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.3.3a.cmml" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.3.3"><mtext mathsize="70%" id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.3.3.cmml" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.3.3.3">skip</mtext></ci></apply><ci id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.1.cmml" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.1">ℎ</ci></apply><cn type="integer" id="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.4.cmml" xref="S3.E9.m1.4.4.4.4.4.4.4.4.2.1.mf.4">0</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E9X.2.1.1.m1.2c">\displaystyle\phi_{\text{layer-skip}}(\mathcal{M},\mathcal{R}^{\mathcal{X}},\mathbf{x}_{h})=\begin{cases}m_{h}^{\mathcal{X}}&amp;\text{if}\ c_{\text{skip}}(h)=1\\
m^{S}&amp;\text{if}\ c_{\text{skip}}(h)=0\end{cases},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(9)</span></td>
</tr>
</tbody>
</table>
<p id="S3.SS5.SSS0.Px1.p1.3" class="ltx_p">where <math id="S3.SS5.SSS0.Px1.p1.2.m1.1" class="ltx_Math" alttext="c_{\text{skip}}(\cdot)" display="inline"><semantics id="S3.SS5.SSS0.Px1.p1.2.m1.1a"><mrow id="S3.SS5.SSS0.Px1.p1.2.m1.1.2" xref="S3.SS5.SSS0.Px1.p1.2.m1.1.2.cmml"><msub id="S3.SS5.SSS0.Px1.p1.2.m1.1.2.2" xref="S3.SS5.SSS0.Px1.p1.2.m1.1.2.2.cmml"><mi id="S3.SS5.SSS0.Px1.p1.2.m1.1.2.2.2" xref="S3.SS5.SSS0.Px1.p1.2.m1.1.2.2.2.cmml">c</mi><mtext id="S3.SS5.SSS0.Px1.p1.2.m1.1.2.2.3" xref="S3.SS5.SSS0.Px1.p1.2.m1.1.2.2.3a.cmml">skip</mtext></msub><mo lspace="0em" rspace="0em" id="S3.SS5.SSS0.Px1.p1.2.m1.1.2.1" xref="S3.SS5.SSS0.Px1.p1.2.m1.1.2.1.cmml">​</mo><mrow id="S3.SS5.SSS0.Px1.p1.2.m1.1.2.3.2" xref="S3.SS5.SSS0.Px1.p1.2.m1.1.2.cmml"><mo stretchy="false" id="S3.SS5.SSS0.Px1.p1.2.m1.1.2.3.2.1" xref="S3.SS5.SSS0.Px1.p1.2.m1.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S3.SS5.SSS0.Px1.p1.2.m1.1.1" xref="S3.SS5.SSS0.Px1.p1.2.m1.1.1.cmml">⋅</mo><mo stretchy="false" id="S3.SS5.SSS0.Px1.p1.2.m1.1.2.3.2.2" xref="S3.SS5.SSS0.Px1.p1.2.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS5.SSS0.Px1.p1.2.m1.1b"><apply id="S3.SS5.SSS0.Px1.p1.2.m1.1.2.cmml" xref="S3.SS5.SSS0.Px1.p1.2.m1.1.2"><times id="S3.SS5.SSS0.Px1.p1.2.m1.1.2.1.cmml" xref="S3.SS5.SSS0.Px1.p1.2.m1.1.2.1"></times><apply id="S3.SS5.SSS0.Px1.p1.2.m1.1.2.2.cmml" xref="S3.SS5.SSS0.Px1.p1.2.m1.1.2.2"><csymbol cd="ambiguous" id="S3.SS5.SSS0.Px1.p1.2.m1.1.2.2.1.cmml" xref="S3.SS5.SSS0.Px1.p1.2.m1.1.2.2">subscript</csymbol><ci id="S3.SS5.SSS0.Px1.p1.2.m1.1.2.2.2.cmml" xref="S3.SS5.SSS0.Px1.p1.2.m1.1.2.2.2">𝑐</ci><ci id="S3.SS5.SSS0.Px1.p1.2.m1.1.2.2.3a.cmml" xref="S3.SS5.SSS0.Px1.p1.2.m1.1.2.2.3"><mtext mathsize="70%" id="S3.SS5.SSS0.Px1.p1.2.m1.1.2.2.3.cmml" xref="S3.SS5.SSS0.Px1.p1.2.m1.1.2.2.3">skip</mtext></ci></apply><ci id="S3.SS5.SSS0.Px1.p1.2.m1.1.1.cmml" xref="S3.SS5.SSS0.Px1.p1.2.m1.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.SSS0.Px1.p1.2.m1.1c">c_{\text{skip}}(\cdot)</annotation></semantics></math> is the criterion that decides whether to skip the <math id="S3.SS5.SSS0.Px1.p1.3.m2.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S3.SS5.SSS0.Px1.p1.3.m2.1a"><mi id="S3.SS5.SSS0.Px1.p1.3.m2.1.1" xref="S3.SS5.SSS0.Px1.p1.3.m2.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S3.SS5.SSS0.Px1.p1.3.m2.1b"><ci id="S3.SS5.SSS0.Px1.p1.3.m2.1.1.cmml" xref="S3.SS5.SSS0.Px1.p1.3.m2.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.SSS0.Px1.p1.3.m2.1c">h</annotation></semantics></math>-th layer or not. Note that the technique of early-exiting <cite class="ltx_cite ltx_citemacro_cite">Graves (<a href="#bib.bib18" title="" class="ltx_ref">2016</a>); Figurnov et al. (<a href="#bib.bib15" title="" class="ltx_ref">2017</a>)</cite> can be viewed as a special case of layer-skip, where once a layer is skipped, all subsequent layers will be skipped too.</p>
</div>
</section>
<section id="S3.SS5.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Parameter sharing.</h4>

<div id="S3.SS5.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S3.SS5.SSS0.Px2.p1.3" class="ltx_p">We consider parameter sharing that shares weights across modules and does not involve reparameterization techniques. Under this restriction, the sharing paradigm can be defined as a criterion function <math id="S3.SS5.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="c_{\text{share}}:i\mapsto j\ (j\leq i)" display="inline"><semantics id="S3.SS5.SSS0.Px2.p1.1.m1.1a"><mrow id="S3.SS5.SSS0.Px2.p1.1.m1.1.1" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.cmml"><msub id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.3" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.3.cmml"><mi id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.3.2" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.3.2.cmml">c</mi><mtext id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.3.3" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.3.3a.cmml">share</mtext></msub><mo lspace="0.278em" rspace="0.278em" id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.2" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.2.cmml">:</mo><mrow id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.cmml"><mi id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.3" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.3.cmml">i</mi><mo stretchy="false" id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.2" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.2.cmml">↦</mo><mrow id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.cmml"><mi id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.3" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.3.cmml">j</mi><mo lspace="0.500em" rspace="0em" id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.2" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.2.cmml">​</mo><mrow id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.2" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.cmml"><mi id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.2" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.2.cmml">j</mi><mo id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.1" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.1.cmml">≤</mo><mi id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.3" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.3.cmml">i</mi></mrow><mo stretchy="false" id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.3" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS5.SSS0.Px2.p1.1.m1.1b"><apply id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1"><ci id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.2.cmml" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.2">:</ci><apply id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.3.cmml" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.3.1.cmml" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.3">subscript</csymbol><ci id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.3.2.cmml" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.3.2">𝑐</ci><ci id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.3.3a.cmml" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.3.3"><mtext mathsize="70%" id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.3.3.cmml" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.3.3">share</mtext></ci></apply><apply id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.cmml" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1"><csymbol cd="latexml" id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.2.cmml" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.2">maps-to</csymbol><ci id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.3.cmml" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.3">𝑖</ci><apply id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.cmml" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1"><times id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.2.cmml" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.2"></times><ci id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.3.cmml" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.3">𝑗</ci><apply id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.cmml" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1"><leq id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.1"></leq><ci id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.2">𝑗</ci><ci id="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS5.SSS0.Px2.p1.1.m1.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.SSS0.Px2.p1.1.m1.1c">c_{\text{share}}:i\mapsto j\ (j\leq i)</annotation></semantics></math>, representing using the same weights for module <math id="S3.SS5.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="m_{i}^{\mathcal{X}}" display="inline"><semantics id="S3.SS5.SSS0.Px2.p1.2.m2.1a"><msubsup id="S3.SS5.SSS0.Px2.p1.2.m2.1.1" xref="S3.SS5.SSS0.Px2.p1.2.m2.1.1.cmml"><mi id="S3.SS5.SSS0.Px2.p1.2.m2.1.1.2.2" xref="S3.SS5.SSS0.Px2.p1.2.m2.1.1.2.2.cmml">m</mi><mi id="S3.SS5.SSS0.Px2.p1.2.m2.1.1.2.3" xref="S3.SS5.SSS0.Px2.p1.2.m2.1.1.2.3.cmml">i</mi><mi class="ltx_font_mathcaligraphic" id="S3.SS5.SSS0.Px2.p1.2.m2.1.1.3" xref="S3.SS5.SSS0.Px2.p1.2.m2.1.1.3.cmml">𝒳</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS5.SSS0.Px2.p1.2.m2.1b"><apply id="S3.SS5.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S3.SS5.SSS0.Px2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS5.SSS0.Px2.p1.2.m2.1.1.1.cmml" xref="S3.SS5.SSS0.Px2.p1.2.m2.1.1">superscript</csymbol><apply id="S3.SS5.SSS0.Px2.p1.2.m2.1.1.2.cmml" xref="S3.SS5.SSS0.Px2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS5.SSS0.Px2.p1.2.m2.1.1.2.1.cmml" xref="S3.SS5.SSS0.Px2.p1.2.m2.1.1">subscript</csymbol><ci id="S3.SS5.SSS0.Px2.p1.2.m2.1.1.2.2.cmml" xref="S3.SS5.SSS0.Px2.p1.2.m2.1.1.2.2">𝑚</ci><ci id="S3.SS5.SSS0.Px2.p1.2.m2.1.1.2.3.cmml" xref="S3.SS5.SSS0.Px2.p1.2.m2.1.1.2.3">𝑖</ci></apply><ci id="S3.SS5.SSS0.Px2.p1.2.m2.1.1.3.cmml" xref="S3.SS5.SSS0.Px2.p1.2.m2.1.1.3">𝒳</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.SSS0.Px2.p1.2.m2.1c">m_{i}^{\mathcal{X}}</annotation></semantics></math> and module <math id="S3.SS5.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="m_{j}^{\mathcal{X}}" display="inline"><semantics id="S3.SS5.SSS0.Px2.p1.3.m3.1a"><msubsup id="S3.SS5.SSS0.Px2.p1.3.m3.1.1" xref="S3.SS5.SSS0.Px2.p1.3.m3.1.1.cmml"><mi id="S3.SS5.SSS0.Px2.p1.3.m3.1.1.2.2" xref="S3.SS5.SSS0.Px2.p1.3.m3.1.1.2.2.cmml">m</mi><mi id="S3.SS5.SSS0.Px2.p1.3.m3.1.1.2.3" xref="S3.SS5.SSS0.Px2.p1.3.m3.1.1.2.3.cmml">j</mi><mi class="ltx_font_mathcaligraphic" id="S3.SS5.SSS0.Px2.p1.3.m3.1.1.3" xref="S3.SS5.SSS0.Px2.p1.3.m3.1.1.3.cmml">𝒳</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS5.SSS0.Px2.p1.3.m3.1b"><apply id="S3.SS5.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S3.SS5.SSS0.Px2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS5.SSS0.Px2.p1.3.m3.1.1.1.cmml" xref="S3.SS5.SSS0.Px2.p1.3.m3.1.1">superscript</csymbol><apply id="S3.SS5.SSS0.Px2.p1.3.m3.1.1.2.cmml" xref="S3.SS5.SSS0.Px2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS5.SSS0.Px2.p1.3.m3.1.1.2.1.cmml" xref="S3.SS5.SSS0.Px2.p1.3.m3.1.1">subscript</csymbol><ci id="S3.SS5.SSS0.Px2.p1.3.m3.1.1.2.2.cmml" xref="S3.SS5.SSS0.Px2.p1.3.m3.1.1.2.2">𝑚</ci><ci id="S3.SS5.SSS0.Px2.p1.3.m3.1.1.2.3.cmml" xref="S3.SS5.SSS0.Px2.p1.3.m3.1.1.2.3">𝑗</ci></apply><ci id="S3.SS5.SSS0.Px2.p1.3.m3.1.1.3.cmml" xref="S3.SS5.SSS0.Px2.p1.3.m3.1.1.3">𝒳</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.SSS0.Px2.p1.3.m3.1c">m_{j}^{\mathcal{X}}</annotation></semantics></math>. Within MoM, parameter sharing can be formulated as:</p>
<table id="S3.E10" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S3.E10X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E10X.2.1.1.m1.3" class="ltx_Math" alttext="\displaystyle\phi_{\text{parameter-sharing}}(\mathcal{M},\mathcal{R}^{\mathcal{X}},\mathbf{x}_{h})=m_{c_{\text{share}}(h)}^{\mathcal{X}}." display="inline"><semantics id="S3.E10X.2.1.1.m1.3a"><mrow id="S3.E10X.2.1.1.m1.3.3.1" xref="S3.E10X.2.1.1.m1.3.3.1.1.cmml"><mrow id="S3.E10X.2.1.1.m1.3.3.1.1" xref="S3.E10X.2.1.1.m1.3.3.1.1.cmml"><mrow id="S3.E10X.2.1.1.m1.3.3.1.1.2" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.cmml"><msub id="S3.E10X.2.1.1.m1.3.3.1.1.2.4" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.4.cmml"><mi id="S3.E10X.2.1.1.m1.3.3.1.1.2.4.2" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.4.2.cmml">ϕ</mi><mtext id="S3.E10X.2.1.1.m1.3.3.1.1.2.4.3" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.4.3a.cmml">parameter-sharing</mtext></msub><mo lspace="0em" rspace="0em" id="S3.E10X.2.1.1.m1.3.3.1.1.2.3" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.3.cmml">​</mo><mrow id="S3.E10X.2.1.1.m1.3.3.1.1.2.2.2" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.2.3.cmml"><mo stretchy="false" id="S3.E10X.2.1.1.m1.3.3.1.1.2.2.2.3" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.2.3.cmml">(</mo><mi class="ltx_font_mathcaligraphic" id="S3.E10X.2.1.1.m1.2.2" xref="S3.E10X.2.1.1.m1.2.2.cmml">ℳ</mi><mo id="S3.E10X.2.1.1.m1.3.3.1.1.2.2.2.4" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.2.3.cmml">,</mo><msup id="S3.E10X.2.1.1.m1.3.3.1.1.1.1.1.1" xref="S3.E10X.2.1.1.m1.3.3.1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E10X.2.1.1.m1.3.3.1.1.1.1.1.1.2" xref="S3.E10X.2.1.1.m1.3.3.1.1.1.1.1.1.2.cmml">ℛ</mi><mi class="ltx_font_mathcaligraphic" id="S3.E10X.2.1.1.m1.3.3.1.1.1.1.1.1.3" xref="S3.E10X.2.1.1.m1.3.3.1.1.1.1.1.1.3.cmml">𝒳</mi></msup><mo id="S3.E10X.2.1.1.m1.3.3.1.1.2.2.2.5" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.2.3.cmml">,</mo><msub id="S3.E10X.2.1.1.m1.3.3.1.1.2.2.2.2" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.2.2.2.cmml"><mi id="S3.E10X.2.1.1.m1.3.3.1.1.2.2.2.2.2" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.2.2.2.2.cmml">𝐱</mi><mi id="S3.E10X.2.1.1.m1.3.3.1.1.2.2.2.2.3" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.2.2.2.3.cmml">h</mi></msub><mo stretchy="false" id="S3.E10X.2.1.1.m1.3.3.1.1.2.2.2.6" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.2.3.cmml">)</mo></mrow></mrow><mo id="S3.E10X.2.1.1.m1.3.3.1.1.3" xref="S3.E10X.2.1.1.m1.3.3.1.1.3.cmml">=</mo><msubsup id="S3.E10X.2.1.1.m1.3.3.1.1.4" xref="S3.E10X.2.1.1.m1.3.3.1.1.4.cmml"><mi id="S3.E10X.2.1.1.m1.3.3.1.1.4.2.2" xref="S3.E10X.2.1.1.m1.3.3.1.1.4.2.2.cmml">m</mi><mrow id="S3.E10X.2.1.1.m1.1.1.1" xref="S3.E10X.2.1.1.m1.1.1.1.cmml"><msub id="S3.E10X.2.1.1.m1.1.1.1.3" xref="S3.E10X.2.1.1.m1.1.1.1.3.cmml"><mi id="S3.E10X.2.1.1.m1.1.1.1.3.2" xref="S3.E10X.2.1.1.m1.1.1.1.3.2.cmml">c</mi><mtext id="S3.E10X.2.1.1.m1.1.1.1.3.3" xref="S3.E10X.2.1.1.m1.1.1.1.3.3a.cmml">share</mtext></msub><mo lspace="0em" rspace="0em" id="S3.E10X.2.1.1.m1.1.1.1.2" xref="S3.E10X.2.1.1.m1.1.1.1.2.cmml">​</mo><mrow id="S3.E10X.2.1.1.m1.1.1.1.4.2" xref="S3.E10X.2.1.1.m1.1.1.1.cmml"><mo stretchy="false" id="S3.E10X.2.1.1.m1.1.1.1.4.2.1" xref="S3.E10X.2.1.1.m1.1.1.1.cmml">(</mo><mi id="S3.E10X.2.1.1.m1.1.1.1.1" xref="S3.E10X.2.1.1.m1.1.1.1.1.cmml">h</mi><mo stretchy="false" id="S3.E10X.2.1.1.m1.1.1.1.4.2.2" xref="S3.E10X.2.1.1.m1.1.1.1.cmml">)</mo></mrow></mrow><mi class="ltx_font_mathcaligraphic" id="S3.E10X.2.1.1.m1.3.3.1.1.4.3" xref="S3.E10X.2.1.1.m1.3.3.1.1.4.3.cmml">𝒳</mi></msubsup></mrow><mo lspace="0em" id="S3.E10X.2.1.1.m1.3.3.1.2" xref="S3.E10X.2.1.1.m1.3.3.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E10X.2.1.1.m1.3b"><apply id="S3.E10X.2.1.1.m1.3.3.1.1.cmml" xref="S3.E10X.2.1.1.m1.3.3.1"><eq id="S3.E10X.2.1.1.m1.3.3.1.1.3.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.3"></eq><apply id="S3.E10X.2.1.1.m1.3.3.1.1.2.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.2"><times id="S3.E10X.2.1.1.m1.3.3.1.1.2.3.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.3"></times><apply id="S3.E10X.2.1.1.m1.3.3.1.1.2.4.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.4"><csymbol cd="ambiguous" id="S3.E10X.2.1.1.m1.3.3.1.1.2.4.1.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.4">subscript</csymbol><ci id="S3.E10X.2.1.1.m1.3.3.1.1.2.4.2.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.4.2">italic-ϕ</ci><ci id="S3.E10X.2.1.1.m1.3.3.1.1.2.4.3a.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.4.3"><mtext mathsize="70%" id="S3.E10X.2.1.1.m1.3.3.1.1.2.4.3.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.4.3">parameter-sharing</mtext></ci></apply><vector id="S3.E10X.2.1.1.m1.3.3.1.1.2.2.3.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.2.2"><ci id="S3.E10X.2.1.1.m1.2.2.cmml" xref="S3.E10X.2.1.1.m1.2.2">ℳ</ci><apply id="S3.E10X.2.1.1.m1.3.3.1.1.1.1.1.1.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E10X.2.1.1.m1.3.3.1.1.1.1.1.1.1.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.1.1.1.1">superscript</csymbol><ci id="S3.E10X.2.1.1.m1.3.3.1.1.1.1.1.1.2.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.1.1.1.1.2">ℛ</ci><ci id="S3.E10X.2.1.1.m1.3.3.1.1.1.1.1.1.3.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.1.1.1.1.3">𝒳</ci></apply><apply id="S3.E10X.2.1.1.m1.3.3.1.1.2.2.2.2.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S3.E10X.2.1.1.m1.3.3.1.1.2.2.2.2.1.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.2.2.2">subscript</csymbol><ci id="S3.E10X.2.1.1.m1.3.3.1.1.2.2.2.2.2.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.2.2.2.2">𝐱</ci><ci id="S3.E10X.2.1.1.m1.3.3.1.1.2.2.2.2.3.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.2.2.2.2.3">ℎ</ci></apply></vector></apply><apply id="S3.E10X.2.1.1.m1.3.3.1.1.4.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.4"><csymbol cd="ambiguous" id="S3.E10X.2.1.1.m1.3.3.1.1.4.1.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.4">superscript</csymbol><apply id="S3.E10X.2.1.1.m1.3.3.1.1.4.2.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.4"><csymbol cd="ambiguous" id="S3.E10X.2.1.1.m1.3.3.1.1.4.2.1.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.4">subscript</csymbol><ci id="S3.E10X.2.1.1.m1.3.3.1.1.4.2.2.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.4.2.2">𝑚</ci><apply id="S3.E10X.2.1.1.m1.1.1.1.cmml" xref="S3.E10X.2.1.1.m1.1.1.1"><times id="S3.E10X.2.1.1.m1.1.1.1.2.cmml" xref="S3.E10X.2.1.1.m1.1.1.1.2"></times><apply id="S3.E10X.2.1.1.m1.1.1.1.3.cmml" xref="S3.E10X.2.1.1.m1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E10X.2.1.1.m1.1.1.1.3.1.cmml" xref="S3.E10X.2.1.1.m1.1.1.1.3">subscript</csymbol><ci id="S3.E10X.2.1.1.m1.1.1.1.3.2.cmml" xref="S3.E10X.2.1.1.m1.1.1.1.3.2">𝑐</ci><ci id="S3.E10X.2.1.1.m1.1.1.1.3.3a.cmml" xref="S3.E10X.2.1.1.m1.1.1.1.3.3"><mtext mathsize="50%" id="S3.E10X.2.1.1.m1.1.1.1.3.3.cmml" xref="S3.E10X.2.1.1.m1.1.1.1.3.3">share</mtext></ci></apply><ci id="S3.E10X.2.1.1.m1.1.1.1.1.cmml" xref="S3.E10X.2.1.1.m1.1.1.1.1">ℎ</ci></apply></apply><ci id="S3.E10X.2.1.1.m1.3.3.1.1.4.3.cmml" xref="S3.E10X.2.1.1.m1.3.3.1.1.4.3">𝒳</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E10X.2.1.1.m1.3c">\displaystyle\phi_{\text{parameter-sharing}}(\mathcal{M},\mathcal{R}^{\mathcal{X}},\mathbf{x}_{h})=m_{c_{\text{share}}(h)}^{\mathcal{X}}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(10)</span></td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="S3.SS5.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Mixture-of-Experts.</h4>

<div id="S3.SS5.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S3.SS5.SSS0.Px3.p1.1" class="ltx_p">MoE splits <span id="S3.SS5.SSS0.Px3.p1.1.1" class="ltx_text ltx_font_typewriter">FFN</span> into experts, and the experts are not shared across different layers. In MoE, routing is only performed on <span id="S3.SS5.SSS0.Px3.p1.1.2" class="ltx_text ltx_font_typewriter">FFN</span> modules, thus the computation of <span id="S3.SS5.SSS0.Px3.p1.1.3" class="ltx_text ltx_font_typewriter">MHA</span> is the same as that of a vanilla Transformer. The assembly function for MoE can be written as:</p>
</div>
<div id="S3.SS5.SSS0.Px3.p2" class="ltx_para ltx_noindent">
<table id="S3.E11" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S3.E11X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E11X.2.1.1.m1.2" class="ltx_Math" alttext="\displaystyle\phi_{\text{MoE}}(\mathcal{M},\mathcal{R}^{\mathcal{X}},\mathbf{x}_{h})=\begin{cases}m_{h}^{\text{A}}&amp;\text{if}\ \mathcal{X}=\text{A}\\
\phi_{\text{MoM}}(\mathcal{M}_{h},\mathcal{R}^{\text{F}},\mathbf{x}_{h})&amp;\text{if}\ \mathcal{X}=\text{F}\\
\end{cases}," display="inline"><semantics id="S3.E11X.2.1.1.m1.2a"><mrow id="S3.E11X.2.1.1.m1.2.2.1" xref="S3.E11X.2.1.1.m1.2.2.1.1.cmml"><mrow id="S3.E11X.2.1.1.m1.2.2.1.1" xref="S3.E11X.2.1.1.m1.2.2.1.1.cmml"><mrow id="S3.E11X.2.1.1.m1.2.2.1.1.2" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.cmml"><msub id="S3.E11X.2.1.1.m1.2.2.1.1.2.4" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.4.cmml"><mi id="S3.E11X.2.1.1.m1.2.2.1.1.2.4.2" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.4.2.cmml">ϕ</mi><mtext id="S3.E11X.2.1.1.m1.2.2.1.1.2.4.3" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.4.3a.cmml">MoE</mtext></msub><mo lspace="0em" rspace="0em" id="S3.E11X.2.1.1.m1.2.2.1.1.2.3" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.3.cmml">​</mo><mrow id="S3.E11X.2.1.1.m1.2.2.1.1.2.2.2" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.2.3.cmml"><mo stretchy="false" id="S3.E11X.2.1.1.m1.2.2.1.1.2.2.2.3" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.2.3.cmml">(</mo><mi class="ltx_font_mathcaligraphic" id="S3.E11X.2.1.1.m1.1.1" xref="S3.E11X.2.1.1.m1.1.1.cmml">ℳ</mi><mo id="S3.E11X.2.1.1.m1.2.2.1.1.2.2.2.4" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.2.3.cmml">,</mo><msup id="S3.E11X.2.1.1.m1.2.2.1.1.1.1.1.1" xref="S3.E11X.2.1.1.m1.2.2.1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E11X.2.1.1.m1.2.2.1.1.1.1.1.1.2" xref="S3.E11X.2.1.1.m1.2.2.1.1.1.1.1.1.2.cmml">ℛ</mi><mi class="ltx_font_mathcaligraphic" id="S3.E11X.2.1.1.m1.2.2.1.1.1.1.1.1.3" xref="S3.E11X.2.1.1.m1.2.2.1.1.1.1.1.1.3.cmml">𝒳</mi></msup><mo id="S3.E11X.2.1.1.m1.2.2.1.1.2.2.2.5" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.2.3.cmml">,</mo><msub id="S3.E11X.2.1.1.m1.2.2.1.1.2.2.2.2" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.2.2.2.cmml"><mi id="S3.E11X.2.1.1.m1.2.2.1.1.2.2.2.2.2" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.2.2.2.2.cmml">𝐱</mi><mi id="S3.E11X.2.1.1.m1.2.2.1.1.2.2.2.2.3" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.2.2.2.3.cmml">h</mi></msub><mo stretchy="false" id="S3.E11X.2.1.1.m1.2.2.1.1.2.2.2.6" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.2.3.cmml">)</mo></mrow></mrow><mo id="S3.E11X.2.1.1.m1.2.2.1.1.3" xref="S3.E11X.2.1.1.m1.2.2.1.1.3.cmml">=</mo><mrow id="S3.E11.m1.4.4.4.4.4.4a" xref="S3.E11X.2.1.1.m1.2.2.1.1.4.1.cmml"><mo id="S3.E11.m1.4.4.4.4.4.4a.5" xref="S3.E11X.2.1.1.m1.2.2.1.1.4.1.1.cmml">{</mo><mtable columnspacing="5pt" rowspacing="0pt" id="S3.E11.m1.4.4.4.4.4.4.4a" xref="S3.E11X.2.1.1.m1.2.2.1.1.4.1.cmml"><mtr id="S3.E11.m1.4.4.4.4.4.4.4aa" xref="S3.E11X.2.1.1.m1.2.2.1.1.4.1.cmml"><mtd class="ltx_align_left" columnalign="left" id="S3.E11.m1.4.4.4.4.4.4.4ab" xref="S3.E11X.2.1.1.m1.2.2.1.1.4.1.cmml"><msubsup id="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf" xref="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf.cmml"><mi id="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf.2.2" xref="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf.2.2.cmml">m</mi><mi id="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf.2.3" xref="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf.2.3.cmml">h</mi><mtext id="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf.3" xref="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf.3a.cmml">A</mtext></msubsup></mtd><mtd class="ltx_align_left" columnalign="left" id="S3.E11.m1.4.4.4.4.4.4.4ac" xref="S3.E11X.2.1.1.m1.2.2.1.1.4.1.cmml"><mrow id="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf" xref="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.cmml"><mrow id="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.2" xref="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.2.cmml"><mtext id="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.2.2" xref="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.2.2a.cmml">if</mtext><mo lspace="0.500em" rspace="0em" id="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.2.1" xref="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.2.1.cmml">​</mo><mi class="ltx_font_mathcaligraphic" id="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.2.3" xref="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.2.3.cmml">𝒳</mi></mrow><mo id="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.1" xref="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.1.cmml">=</mo><mtext id="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.3" xref="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.3a.cmml">A</mtext></mrow></mtd></mtr><mtr id="S3.E11.m1.4.4.4.4.4.4.4ad" xref="S3.E11X.2.1.1.m1.2.2.1.1.4.1.cmml"><mtd class="ltx_align_left" columnalign="left" id="S3.E11.m1.4.4.4.4.4.4.4ae" xref="S3.E11X.2.1.1.m1.2.2.1.1.4.1.cmml"><mrow id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.cmml"><msub id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.5" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.5.cmml"><mi id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.5.2" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.5.2.cmml">ϕ</mi><mtext id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.5.3" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.5.3a.cmml">MoM</mtext></msub><mo lspace="0em" rspace="0em" id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.4" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.4.cmml">​</mo><mrow id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.3" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.4.cmml"><mo stretchy="false" id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.3.4" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.4.cmml">(</mo><msub id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.1.1.1" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.1.1.1.2" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.1.1.1.2.cmml">ℳ</mi><mi id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.1.1.1.3" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.1.1.1.3.cmml">h</mi></msub><mo id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.3.5" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.4.cmml">,</mo><msup id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.2.2.2" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.2.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.2.2.2.2" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.2.2.2.2.cmml">ℛ</mi><mtext id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.2.2.2.3" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.2.2.2.3a.cmml">F</mtext></msup><mo id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.3.6" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.4.cmml">,</mo><msub id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.3.3" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.3.3.cmml"><mi id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.3.3.2" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.3.3.2.cmml">𝐱</mi><mi id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.3.3.3" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.3.3.3.cmml">h</mi></msub><mo stretchy="false" id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.3.7" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.4.cmml">)</mo></mrow></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S3.E11.m1.4.4.4.4.4.4.4af" xref="S3.E11X.2.1.1.m1.2.2.1.1.4.1.cmml"><mrow id="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf" xref="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.cmml"><mrow id="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.2" xref="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.2.cmml"><mtext id="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.2.2" xref="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.2.2a.cmml">if</mtext><mo lspace="0.500em" rspace="0em" id="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.2.1" xref="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.2.1.cmml">​</mo><mi class="ltx_font_mathcaligraphic" id="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.2.3" xref="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.2.3.cmml">𝒳</mi></mrow><mo id="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.1" xref="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.1.cmml">=</mo><mtext id="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.3" xref="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.3a.cmml">F</mtext></mrow></mtd></mtr></mtable></mrow></mrow><mo id="S3.E11X.2.1.1.m1.2.2.1.2" xref="S3.E11X.2.1.1.m1.2.2.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E11X.2.1.1.m1.2b"><apply id="S3.E11X.2.1.1.m1.2.2.1.1.cmml" xref="S3.E11X.2.1.1.m1.2.2.1"><eq id="S3.E11X.2.1.1.m1.2.2.1.1.3.cmml" xref="S3.E11X.2.1.1.m1.2.2.1.1.3"></eq><apply id="S3.E11X.2.1.1.m1.2.2.1.1.2.cmml" xref="S3.E11X.2.1.1.m1.2.2.1.1.2"><times id="S3.E11X.2.1.1.m1.2.2.1.1.2.3.cmml" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.3"></times><apply id="S3.E11X.2.1.1.m1.2.2.1.1.2.4.cmml" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.4"><csymbol cd="ambiguous" id="S3.E11X.2.1.1.m1.2.2.1.1.2.4.1.cmml" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.4">subscript</csymbol><ci id="S3.E11X.2.1.1.m1.2.2.1.1.2.4.2.cmml" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.4.2">italic-ϕ</ci><ci id="S3.E11X.2.1.1.m1.2.2.1.1.2.4.3a.cmml" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.4.3"><mtext mathsize="70%" id="S3.E11X.2.1.1.m1.2.2.1.1.2.4.3.cmml" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.4.3">MoE</mtext></ci></apply><vector id="S3.E11X.2.1.1.m1.2.2.1.1.2.2.3.cmml" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.2.2"><ci id="S3.E11X.2.1.1.m1.1.1.cmml" xref="S3.E11X.2.1.1.m1.1.1">ℳ</ci><apply id="S3.E11X.2.1.1.m1.2.2.1.1.1.1.1.1.cmml" xref="S3.E11X.2.1.1.m1.2.2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E11X.2.1.1.m1.2.2.1.1.1.1.1.1.1.cmml" xref="S3.E11X.2.1.1.m1.2.2.1.1.1.1.1.1">superscript</csymbol><ci id="S3.E11X.2.1.1.m1.2.2.1.1.1.1.1.1.2.cmml" xref="S3.E11X.2.1.1.m1.2.2.1.1.1.1.1.1.2">ℛ</ci><ci id="S3.E11X.2.1.1.m1.2.2.1.1.1.1.1.1.3.cmml" xref="S3.E11X.2.1.1.m1.2.2.1.1.1.1.1.1.3">𝒳</ci></apply><apply id="S3.E11X.2.1.1.m1.2.2.1.1.2.2.2.2.cmml" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S3.E11X.2.1.1.m1.2.2.1.1.2.2.2.2.1.cmml" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.2.2.2">subscript</csymbol><ci id="S3.E11X.2.1.1.m1.2.2.1.1.2.2.2.2.2.cmml" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.2.2.2.2">𝐱</ci><ci id="S3.E11X.2.1.1.m1.2.2.1.1.2.2.2.2.3.cmml" xref="S3.E11X.2.1.1.m1.2.2.1.1.2.2.2.2.3">ℎ</ci></apply></vector></apply><apply id="S3.E11X.2.1.1.m1.2.2.1.1.4.1.cmml" xref="S3.E11.m1.4.4.4.4.4.4a"><csymbol cd="latexml" id="S3.E11X.2.1.1.m1.2.2.1.1.4.1.1.cmml" xref="S3.E11.m1.4.4.4.4.4.4a.5">cases</csymbol><apply id="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf.cmml" xref="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf"><csymbol cd="ambiguous" id="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf.1.cmml" xref="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf">superscript</csymbol><apply id="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf.2.cmml" xref="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf"><csymbol cd="ambiguous" id="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf.2.1.cmml" xref="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf">subscript</csymbol><ci id="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf.2.2.cmml" xref="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf.2.2">𝑚</ci><ci id="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf.2.3.cmml" xref="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf.2.3">ℎ</ci></apply><ci id="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf.3a.cmml" xref="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf.3"><mtext mathsize="70%" id="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf.3.cmml" xref="S3.E11.m1.1.1.1.1.1.1.1.1.1.1.mf.3">A</mtext></ci></apply><apply id="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.cmml" xref="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf"><eq id="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.1.cmml" xref="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.1"></eq><apply id="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.2.cmml" xref="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.2"><times id="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.2.1.cmml" xref="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.2.1"></times><ci id="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.2.2a.cmml" xref="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.2.2"><mtext id="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.2.2.cmml" xref="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.2.2">if</mtext></ci><ci id="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.2.3.cmml" xref="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.2.3">𝒳</ci></apply><ci id="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.3a.cmml" xref="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.3"><mtext id="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.3.cmml" xref="S3.E11.m1.2.2.2.2.2.2.2.2.2.1.mf.3">A</mtext></ci></apply><apply id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.cmml" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf"><times id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.4.cmml" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.4"></times><apply id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.5.cmml" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.5"><csymbol cd="ambiguous" id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.5.1.cmml" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.5">subscript</csymbol><ci id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.5.2.cmml" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.5.2">italic-ϕ</ci><ci id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.5.3a.cmml" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.5.3"><mtext mathsize="70%" id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.5.3.cmml" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.5.3">MoM</mtext></ci></apply><vector id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.4.cmml" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.3"><apply id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.1.1.1.cmml" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.1.1.1"><csymbol cd="ambiguous" id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.1.1.1.1.cmml" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.1.1.1">subscript</csymbol><ci id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.1.1.1.2.cmml" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.1.1.1.2">ℳ</ci><ci id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.1.1.1.3.cmml" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.1.1.1.3">ℎ</ci></apply><apply id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.2.2.2.cmml" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.2.2.2"><csymbol cd="ambiguous" id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.2.2.2.1.cmml" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.2.2.2">superscript</csymbol><ci id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.2.2.2.2.cmml" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.2.2.2.2">ℛ</ci><ci id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.2.2.2.3a.cmml" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.2.2.2.3"><mtext mathsize="70%" id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.2.2.2.3.cmml" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.2.2.2.3">F</mtext></ci></apply><apply id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.3.3.cmml" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.3.3"><csymbol cd="ambiguous" id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.3.3.1.cmml" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.3.3">subscript</csymbol><ci id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.3.3.2.cmml" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.3.3.2">𝐱</ci><ci id="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.3.3.3.cmml" xref="S3.E11.m1.3.3.3.3.3.3.3.3.1.1.mf.3.3.3.3">ℎ</ci></apply></vector></apply><apply id="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.cmml" xref="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf"><eq id="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.1.cmml" xref="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.1"></eq><apply id="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.2.cmml" xref="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.2"><times id="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.2.1.cmml" xref="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.2.1"></times><ci id="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.2.2a.cmml" xref="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.2.2"><mtext id="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.2.2.cmml" xref="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.2.2">if</mtext></ci><ci id="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.2.3.cmml" xref="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.2.3">𝒳</ci></apply><ci id="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.3a.cmml" xref="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.3"><mtext id="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.3.cmml" xref="S3.E11.m1.4.4.4.4.4.4.4.4.2.1.mf.3">F</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E11X.2.1.1.m1.2c">\displaystyle\phi_{\text{MoE}}(\mathcal{M},\mathcal{R}^{\mathcal{X}},\mathbf{x}_{h})=\begin{cases}m_{h}^{\text{A}}&amp;\text{if}\ \mathcal{X}=\text{A}\\
\phi_{\text{MoM}}(\mathcal{M}_{h},\mathcal{R}^{\text{F}},\mathbf{x}_{h})&amp;\text{if}\ \mathcal{X}=\text{F}\\
\end{cases},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(11)</span></td>
</tr>
</tbody>
</table>
<p id="S3.SS5.SSS0.Px3.p2.2" class="ltx_p">where <math id="S3.SS5.SSS0.Px3.p2.1.m1.3" class="ltx_Math" alttext="\mathcal{M}_{h}=\{m_{h,i}^{\text{F}}\}_{i=1}^{N_{\text{F}}}" display="inline"><semantics id="S3.SS5.SSS0.Px3.p2.1.m1.3a"><mrow id="S3.SS5.SSS0.Px3.p2.1.m1.3.3" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.cmml"><msub id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.3" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.3.2" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.3.2.cmml">ℳ</mi><mi id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.3.3" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.3.3.cmml">h</mi></msub><mo id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.2" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.2.cmml">=</mo><msubsup id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.cmml"><mrow id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.2" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.2.cmml">{</mo><msubsup id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.1" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.1.cmml"><mi id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.1.2.2" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.1.2.2.cmml">m</mi><mrow id="S3.SS5.SSS0.Px3.p2.1.m1.2.2.2.4" xref="S3.SS5.SSS0.Px3.p2.1.m1.2.2.2.3.cmml"><mi id="S3.SS5.SSS0.Px3.p2.1.m1.1.1.1.1" xref="S3.SS5.SSS0.Px3.p2.1.m1.1.1.1.1.cmml">h</mi><mo id="S3.SS5.SSS0.Px3.p2.1.m1.2.2.2.4.1" xref="S3.SS5.SSS0.Px3.p2.1.m1.2.2.2.3.cmml">,</mo><mi id="S3.SS5.SSS0.Px3.p2.1.m1.2.2.2.2" xref="S3.SS5.SSS0.Px3.p2.1.m1.2.2.2.2.cmml">i</mi></mrow><mtext id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.1.3" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.1.3a.cmml">F</mtext></msubsup><mo stretchy="false" id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.3" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.2.cmml">}</mo></mrow><mrow id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.3" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.3.cmml"><mi id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.3.2" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.3.2.cmml">i</mi><mo id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.3.1" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.3.1.cmml">=</mo><mn id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.3.3" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.3.3.cmml">1</mn></mrow><msub id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.3" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.3.cmml"><mi id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.3.2" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.3.2.cmml">N</mi><mtext id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.3.3" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.3.3a.cmml">F</mtext></msub></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS5.SSS0.Px3.p2.1.m1.3b"><apply id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3"><eq id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.2.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.2"></eq><apply id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.3.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.3"><csymbol cd="ambiguous" id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.3.1.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.3">subscript</csymbol><ci id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.3.2.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.3.2">ℳ</ci><ci id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.3.3.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.3.3">ℎ</ci></apply><apply id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1"><csymbol cd="ambiguous" id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.2.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1">superscript</csymbol><apply id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1"><csymbol cd="ambiguous" id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.2.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1">subscript</csymbol><set id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.2.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1"><apply id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.1.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.1.1.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.1">superscript</csymbol><apply id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.1.2.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.1.2.1.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.1">subscript</csymbol><ci id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.1.2.2.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.1.2.2">𝑚</ci><list id="S3.SS5.SSS0.Px3.p2.1.m1.2.2.2.3.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.2.2.2.4"><ci id="S3.SS5.SSS0.Px3.p2.1.m1.1.1.1.1.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.1.1.1.1">ℎ</ci><ci id="S3.SS5.SSS0.Px3.p2.1.m1.2.2.2.2.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.2.2.2.2">𝑖</ci></list></apply><ci id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.1.3a.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.1.3"><mtext mathsize="70%" id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.1.3.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.1.1.1.3">F</mtext></ci></apply></set><apply id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.3.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.3"><eq id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.3.1.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.3.1"></eq><ci id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.3.2.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.3.2">𝑖</ci><cn type="integer" id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.3.3.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.1.3.3">1</cn></apply></apply><apply id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.3.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.3"><csymbol cd="ambiguous" id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.3.1.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.3">subscript</csymbol><ci id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.3.2.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.3.2">𝑁</ci><ci id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.3.3a.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.3.3"><mtext mathsize="50%" id="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.3.3.cmml" xref="S3.SS5.SSS0.Px3.p2.1.m1.3.3.1.3.3">F</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.SSS0.Px3.p2.1.m1.3c">\mathcal{M}_{h}=\{m_{h,i}^{\text{F}}\}_{i=1}^{N_{\text{F}}}</annotation></semantics></math> is the collection of experts for layer <math id="S3.SS5.SSS0.Px3.p2.2.m2.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S3.SS5.SSS0.Px3.p2.2.m2.1a"><mi id="S3.SS5.SSS0.Px3.p2.2.m2.1.1" xref="S3.SS5.SSS0.Px3.p2.2.m2.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S3.SS5.SSS0.Px3.p2.2.m2.1b"><ci id="S3.SS5.SSS0.Px3.p2.2.m2.1.1.cmml" xref="S3.SS5.SSS0.Px3.p2.2.m2.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.SSS0.Px3.p2.2.m2.1c">h</annotation></semantics></math>.</p>
</div>
<div id="S3.SS5.SSS0.Px3.p3" class="ltx_para ltx_noindent">
<p id="S3.SS5.SSS0.Px3.p3.1" class="ltx_p">Figure <a href="#S3.F2" title="Figure 2 ‣ Mixture-of-Experts. ‣ 3.5 MoM as a unified framework ‣ 3 Methodology ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> illustrates the forward computation process across different methods, offering an intuitive presentation of the versatility and universality in MoM.</p>
</div>
<figure id="S3.F2" class="ltx_figure"><img src="/html/2407.06677/assets/x2.png" id="S3.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="368" height="117" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>
Visualization of forward computation of five models, where each consists of only two layers just for demonstration purposes.
The switch icon symbolizes the selective execution of one (in Layer-skip) or more (in MoE and MoM) subsequent computation pathways.</figcaption>
</figure>
</section>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Experiments</h2>

<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Experimental setup</h3>

<div id="S4.SS1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.p1.1" class="ltx_p">We implement language models with MoM since language modeling is a challenging task requiring both language understanding and generation ability, thereby effectively evaluating MoM and baselines.
Below, we elaborate on the implementation details of MoM, baselines, and evaluation setups.</p>
</div>
<section id="S4.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Implementation details.</h4>

<div id="S4.SS1.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px1.p1.8" class="ltx_p">We conduct experiments on three model scales, which we denote as MoM-small, MoM-medium, and MoM-large, respectively. These models contain 122M, 346M, and 774M parameters, respectively. Detailed configurations can be found in Appendix <a href="#A1" title="Appendix A More implementation details ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>.
The vanilla Transformers used for initializing MoM are official GPT-2 checkpoints downloaded from HuggingFace<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span><a target="_blank" href="https://huggingface.co/openai-community/gpt2" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/openai-community/gpt2</a></span></span></span>.
<math id="S4.SS1.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S4.SS1.SSS0.Px1.p1.1.m1.1a"><mi id="S4.SS1.SSS0.Px1.p1.1.m1.1.1" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p1.1.m1.1b"><ci id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p1.1.m1.1c">K</annotation></semantics></math> and <math id="S4.SS1.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S4.SS1.SSS0.Px1.p1.2.m2.1a"><mi id="S4.SS1.SSS0.Px1.p1.2.m2.1.1" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p1.2.m2.1b"><ci id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p1.2.m2.1c">H</annotation></semantics></math> represent two hyper-parameters of MoM. We denote a configuration where <math id="S4.SS1.SSS0.Px1.p1.3.m3.2" class="ltx_Math" alttext="K=a,H=b" display="inline"><semantics id="S4.SS1.SSS0.Px1.p1.3.m3.2a"><mrow id="S4.SS1.SSS0.Px1.p1.3.m3.2.2.2" xref="S4.SS1.SSS0.Px1.p1.3.m3.2.2.3.cmml"><mrow id="S4.SS1.SSS0.Px1.p1.3.m3.1.1.1.1" xref="S4.SS1.SSS0.Px1.p1.3.m3.1.1.1.1.cmml"><mi id="S4.SS1.SSS0.Px1.p1.3.m3.1.1.1.1.2" xref="S4.SS1.SSS0.Px1.p1.3.m3.1.1.1.1.2.cmml">K</mi><mo id="S4.SS1.SSS0.Px1.p1.3.m3.1.1.1.1.1" xref="S4.SS1.SSS0.Px1.p1.3.m3.1.1.1.1.1.cmml">=</mo><mi id="S4.SS1.SSS0.Px1.p1.3.m3.1.1.1.1.3" xref="S4.SS1.SSS0.Px1.p1.3.m3.1.1.1.1.3.cmml">a</mi></mrow><mo id="S4.SS1.SSS0.Px1.p1.3.m3.2.2.2.3" xref="S4.SS1.SSS0.Px1.p1.3.m3.2.2.3a.cmml">,</mo><mrow id="S4.SS1.SSS0.Px1.p1.3.m3.2.2.2.2" xref="S4.SS1.SSS0.Px1.p1.3.m3.2.2.2.2.cmml"><mi id="S4.SS1.SSS0.Px1.p1.3.m3.2.2.2.2.2" xref="S4.SS1.SSS0.Px1.p1.3.m3.2.2.2.2.2.cmml">H</mi><mo id="S4.SS1.SSS0.Px1.p1.3.m3.2.2.2.2.1" xref="S4.SS1.SSS0.Px1.p1.3.m3.2.2.2.2.1.cmml">=</mo><mi id="S4.SS1.SSS0.Px1.p1.3.m3.2.2.2.2.3" xref="S4.SS1.SSS0.Px1.p1.3.m3.2.2.2.2.3.cmml">b</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p1.3.m3.2b"><apply id="S4.SS1.SSS0.Px1.p1.3.m3.2.2.3.cmml" xref="S4.SS1.SSS0.Px1.p1.3.m3.2.2.2"><csymbol cd="ambiguous" id="S4.SS1.SSS0.Px1.p1.3.m3.2.2.3a.cmml" xref="S4.SS1.SSS0.Px1.p1.3.m3.2.2.2.3">formulae-sequence</csymbol><apply id="S4.SS1.SSS0.Px1.p1.3.m3.1.1.1.1.cmml" xref="S4.SS1.SSS0.Px1.p1.3.m3.1.1.1.1"><eq id="S4.SS1.SSS0.Px1.p1.3.m3.1.1.1.1.1.cmml" xref="S4.SS1.SSS0.Px1.p1.3.m3.1.1.1.1.1"></eq><ci id="S4.SS1.SSS0.Px1.p1.3.m3.1.1.1.1.2.cmml" xref="S4.SS1.SSS0.Px1.p1.3.m3.1.1.1.1.2">𝐾</ci><ci id="S4.SS1.SSS0.Px1.p1.3.m3.1.1.1.1.3.cmml" xref="S4.SS1.SSS0.Px1.p1.3.m3.1.1.1.1.3">𝑎</ci></apply><apply id="S4.SS1.SSS0.Px1.p1.3.m3.2.2.2.2.cmml" xref="S4.SS1.SSS0.Px1.p1.3.m3.2.2.2.2"><eq id="S4.SS1.SSS0.Px1.p1.3.m3.2.2.2.2.1.cmml" xref="S4.SS1.SSS0.Px1.p1.3.m3.2.2.2.2.1"></eq><ci id="S4.SS1.SSS0.Px1.p1.3.m3.2.2.2.2.2.cmml" xref="S4.SS1.SSS0.Px1.p1.3.m3.2.2.2.2.2">𝐻</ci><ci id="S4.SS1.SSS0.Px1.p1.3.m3.2.2.2.2.3.cmml" xref="S4.SS1.SSS0.Px1.p1.3.m3.2.2.2.2.3">𝑏</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p1.3.m3.2c">K=a,H=b</annotation></semantics></math> as K<math id="S4.SS1.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="a" display="inline"><semantics id="S4.SS1.SSS0.Px1.p1.4.m4.1a"><mi id="S4.SS1.SSS0.Px1.p1.4.m4.1.1" xref="S4.SS1.SSS0.Px1.p1.4.m4.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p1.4.m4.1b"><ci id="S4.SS1.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S4.SS1.SSS0.Px1.p1.4.m4.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p1.4.m4.1c">a</annotation></semantics></math>H<math id="S4.SS1.SSS0.Px1.p1.5.m5.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S4.SS1.SSS0.Px1.p1.5.m5.1a"><mi id="S4.SS1.SSS0.Px1.p1.5.m5.1.1" xref="S4.SS1.SSS0.Px1.p1.5.m5.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p1.5.m5.1b"><ci id="S4.SS1.SSS0.Px1.p1.5.m5.1.1.cmml" xref="S4.SS1.SSS0.Px1.p1.5.m5.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p1.5.m5.1c">b</annotation></semantics></math>. If the skip module is included in <math id="S4.SS1.SSS0.Px1.p1.6.m6.1" class="ltx_Math" alttext="\mathcal{M}" display="inline"><semantics id="S4.SS1.SSS0.Px1.p1.6.m6.1a"><mi class="ltx_font_mathcaligraphic" id="S4.SS1.SSS0.Px1.p1.6.m6.1.1" xref="S4.SS1.SSS0.Px1.p1.6.m6.1.1.cmml">ℳ</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p1.6.m6.1b"><ci id="S4.SS1.SSS0.Px1.p1.6.m6.1.1.cmml" xref="S4.SS1.SSS0.Px1.p1.6.m6.1.1">ℳ</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p1.6.m6.1c">\mathcal{M}</annotation></semantics></math>, we append the suffix S to K<math id="S4.SS1.SSS0.Px1.p1.7.m7.1" class="ltx_Math" alttext="a" display="inline"><semantics id="S4.SS1.SSS0.Px1.p1.7.m7.1a"><mi id="S4.SS1.SSS0.Px1.p1.7.m7.1.1" xref="S4.SS1.SSS0.Px1.p1.7.m7.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p1.7.m7.1b"><ci id="S4.SS1.SSS0.Px1.p1.7.m7.1.1.cmml" xref="S4.SS1.SSS0.Px1.p1.7.m7.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p1.7.m7.1c">a</annotation></semantics></math>H<math id="S4.SS1.SSS0.Px1.p1.8.m8.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S4.SS1.SSS0.Px1.p1.8.m8.1a"><mi id="S4.SS1.SSS0.Px1.p1.8.m8.1.1" xref="S4.SS1.SSS0.Px1.p1.8.m8.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p1.8.m8.1b"><ci id="S4.SS1.SSS0.Px1.p1.8.m8.1.1.cmml" xref="S4.SS1.SSS0.Px1.p1.8.m8.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p1.8.m8.1c">b</annotation></semantics></math>.</p>
</div>
<div id="S4.SS1.SSS0.Px1.p2" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px1.p2.9" class="ltx_p">In the two-phase training, we exploit OpenWebText <cite class="ltx_cite ltx_citemacro_citep">(Gokaslan &amp; Cohen, <a href="#bib.bib17" title="" class="ltx_ref">2019</a>)</cite> as the pre-training dataset, and pre-process the data with the same pipeline as nanoGPT <cite class="ltx_cite ltx_citemacro_citep">(karpathy, <a href="#bib.bib19" title="" class="ltx_ref">2023</a>)</cite>.
OpenWebText contains <math id="S4.SS1.SSS0.Px1.p2.1.m1.1" class="ltx_Math" alttext="9" display="inline"><semantics id="S4.SS1.SSS0.Px1.p2.1.m1.1a"><mn id="S4.SS1.SSS0.Px1.p2.1.m1.1.1" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.cmml">9</mn><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p2.1.m1.1b"><cn type="integer" id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1">9</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p2.1.m1.1c">9</annotation></semantics></math> billion tokens after tokenization, from which <math id="S4.SS1.SSS0.Px1.p2.2.m2.1" class="ltx_Math" alttext="4" display="inline"><semantics id="S4.SS1.SSS0.Px1.p2.2.m2.1a"><mn id="S4.SS1.SSS0.Px1.p2.2.m2.1.1" xref="S4.SS1.SSS0.Px1.p2.2.m2.1.1.cmml">4</mn><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p2.2.m2.1b"><cn type="integer" id="S4.SS1.SSS0.Px1.p2.2.m2.1.1.cmml" xref="S4.SS1.SSS0.Px1.p2.2.m2.1.1">4</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p2.2.m2.1c">4</annotation></semantics></math> million tokens are randomly sampled as the validation set.
The training sequence length for every input is <math id="S4.SS1.SSS0.Px1.p2.3.m3.2" class="ltx_Math" alttext="1,024" display="inline"><semantics id="S4.SS1.SSS0.Px1.p2.3.m3.2a"><mrow id="S4.SS1.SSS0.Px1.p2.3.m3.2.3.2" xref="S4.SS1.SSS0.Px1.p2.3.m3.2.3.1.cmml"><mn id="S4.SS1.SSS0.Px1.p2.3.m3.1.1" xref="S4.SS1.SSS0.Px1.p2.3.m3.1.1.cmml">1</mn><mo id="S4.SS1.SSS0.Px1.p2.3.m3.2.3.2.1" xref="S4.SS1.SSS0.Px1.p2.3.m3.2.3.1.cmml">,</mo><mn id="S4.SS1.SSS0.Px1.p2.3.m3.2.2" xref="S4.SS1.SSS0.Px1.p2.3.m3.2.2.cmml">024</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p2.3.m3.2b"><list id="S4.SS1.SSS0.Px1.p2.3.m3.2.3.1.cmml" xref="S4.SS1.SSS0.Px1.p2.3.m3.2.3.2"><cn type="integer" id="S4.SS1.SSS0.Px1.p2.3.m3.1.1.cmml" xref="S4.SS1.SSS0.Px1.p2.3.m3.1.1">1</cn><cn type="integer" id="S4.SS1.SSS0.Px1.p2.3.m3.2.2.cmml" xref="S4.SS1.SSS0.Px1.p2.3.m3.2.2">024</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p2.3.m3.2c">1,024</annotation></semantics></math>.
We set the learning rate to <math id="S4.SS1.SSS0.Px1.p2.4.m4.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S4.SS1.SSS0.Px1.p2.4.m4.1a"><mn id="S4.SS1.SSS0.Px1.p2.4.m4.1.1" xref="S4.SS1.SSS0.Px1.p2.4.m4.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p2.4.m4.1b"><cn type="integer" id="S4.SS1.SSS0.Px1.p2.4.m4.1.1.cmml" xref="S4.SS1.SSS0.Px1.p2.4.m4.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p2.4.m4.1c">1</annotation></semantics></math>e-<math id="S4.SS1.SSS0.Px1.p2.5.m5.1" class="ltx_Math" alttext="3" display="inline"><semantics id="S4.SS1.SSS0.Px1.p2.5.m5.1a"><mn id="S4.SS1.SSS0.Px1.p2.5.m5.1.1" xref="S4.SS1.SSS0.Px1.p2.5.m5.1.1.cmml">3</mn><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p2.5.m5.1b"><cn type="integer" id="S4.SS1.SSS0.Px1.p2.5.m5.1.1.cmml" xref="S4.SS1.SSS0.Px1.p2.5.m5.1.1">3</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p2.5.m5.1c">3</annotation></semantics></math> with a warm-up ratio of 0.1 throughout the two phases, and do not use dropout.
All models are trained on 8 <math id="S4.SS1.SSS0.Px1.p2.6.m6.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.SS1.SSS0.Px1.p2.6.m6.1a"><mo id="S4.SS1.SSS0.Px1.p2.6.m6.1.1" xref="S4.SS1.SSS0.Px1.p2.6.m6.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p2.6.m6.1b"><times id="S4.SS1.SSS0.Px1.p2.6.m6.1.1.cmml" xref="S4.SS1.SSS0.Px1.p2.6.m6.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p2.6.m6.1c">\times</annotation></semantics></math> A100 GPUs with a total batch size of 8 <math id="S4.SS1.SSS0.Px1.p2.7.m7.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.SS1.SSS0.Px1.p2.7.m7.1a"><mo id="S4.SS1.SSS0.Px1.p2.7.m7.1.1" xref="S4.SS1.SSS0.Px1.p2.7.m7.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p2.7.m7.1b"><times id="S4.SS1.SSS0.Px1.p2.7.m7.1.1.cmml" xref="S4.SS1.SSS0.Px1.p2.7.m7.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p2.7.m7.1c">\times</annotation></semantics></math> 64.
Two training phases require <math id="S4.SS1.SSS0.Px1.p2.8.m8.1" class="ltx_Math" alttext="20" display="inline"><semantics id="S4.SS1.SSS0.Px1.p2.8.m8.1a"><mn id="S4.SS1.SSS0.Px1.p2.8.m8.1.1" xref="S4.SS1.SSS0.Px1.p2.8.m8.1.1.cmml">20</mn><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p2.8.m8.1b"><cn type="integer" id="S4.SS1.SSS0.Px1.p2.8.m8.1.1.cmml" xref="S4.SS1.SSS0.Px1.p2.8.m8.1.1">20</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p2.8.m8.1c">20</annotation></semantics></math>k and <math id="S4.SS1.SSS0.Px1.p2.9.m9.1" class="ltx_Math" alttext="10" display="inline"><semantics id="S4.SS1.SSS0.Px1.p2.9.m9.1a"><mn id="S4.SS1.SSS0.Px1.p2.9.m9.1.1" xref="S4.SS1.SSS0.Px1.p2.9.m9.1.1.cmml">10</mn><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p2.9.m9.1b"><cn type="integer" id="S4.SS1.SSS0.Px1.p2.9.m9.1.1.cmml" xref="S4.SS1.SSS0.Px1.p2.9.m9.1.1">10</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p2.9.m9.1c">10</annotation></semantics></math>k optimization steps, respectively.</p>
</div>
<div id="S4.SS1.SSS0.Px1.p3" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px1.p3.4" class="ltx_p">In practice, considering the large search space characterized by <math id="S4.SS1.SSS0.Px1.p3.1.m1.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S4.SS1.SSS0.Px1.p3.1.m1.1a"><mi id="S4.SS1.SSS0.Px1.p3.1.m1.1.1" xref="S4.SS1.SSS0.Px1.p3.1.m1.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p3.1.m1.1b"><ci id="S4.SS1.SSS0.Px1.p3.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px1.p3.1.m1.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p3.1.m1.1c">H</annotation></semantics></math> and <math id="S4.SS1.SSS0.Px1.p3.2.m2.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S4.SS1.SSS0.Px1.p3.2.m2.1a"><mi id="S4.SS1.SSS0.Px1.p3.2.m2.1.1" xref="S4.SS1.SSS0.Px1.p3.2.m2.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p3.2.m2.1b"><ci id="S4.SS1.SSS0.Px1.p3.2.m2.1.1.cmml" xref="S4.SS1.SSS0.Px1.p3.2.m2.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p3.2.m2.1c">K</annotation></semantics></math>, we confine the architecture search space to a practical scale with a “chunking” strategy. An MoM is divided into several chunks. Each chunk is independent and parameterized with identical <math id="S4.SS1.SSS0.Px1.p3.3.m3.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S4.SS1.SSS0.Px1.p3.3.m3.1a"><mi id="S4.SS1.SSS0.Px1.p3.3.m3.1.1" xref="S4.SS1.SSS0.Px1.p3.3.m3.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p3.3.m3.1b"><ci id="S4.SS1.SSS0.Px1.p3.3.m3.1.1.cmml" xref="S4.SS1.SSS0.Px1.p3.3.m3.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p3.3.m3.1c">H</annotation></semantics></math> and <math id="S4.SS1.SSS0.Px1.p3.4.m4.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S4.SS1.SSS0.Px1.p3.4.m4.1a"><mi id="S4.SS1.SSS0.Px1.p3.4.m4.1.1" xref="S4.SS1.SSS0.Px1.p3.4.m4.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p3.4.m4.1b"><ci id="S4.SS1.SSS0.Px1.p3.4.m4.1.1.cmml" xref="S4.SS1.SSS0.Px1.p3.4.m4.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p3.4.m4.1c">K</annotation></semantics></math>.
We present a detailed description and specific configuration in Appendix <a href="#A1" title="Appendix A More implementation details ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>, and an empirical analysis of the efficacy of chunking in Appendix <a href="#A2" title="Appendix B Chunks ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a>.</p>
</div>
<figure id="S4.T1" class="ltx_table">
<div id="S4.T1.12" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:349.6pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-73.4pt,59.2pt) scale(0.747082542682026,0.747082542682026) ;">
<table id="S4.T1.12.12" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T1.12.12.13.1" class="ltx_tr">
<td id="S4.T1.12.12.13.1.1" class="ltx_td ltx_align_left ltx_border_tt"><span id="S4.T1.12.12.13.1.1.1" class="ltx_text ltx_font_bold">Methods</span></td>
<td id="S4.T1.12.12.13.1.2" class="ltx_td ltx_align_left ltx_border_tt">MoM Config</td>
<td id="S4.T1.12.12.13.1.3" class="ltx_td ltx_align_right ltx_border_tt">Parameter</td>
<td id="S4.T1.12.12.13.1.4" class="ltx_td ltx_align_right ltx_border_tt">Computation</td>
<td id="S4.T1.12.12.13.1.5" class="ltx_td ltx_align_right ltx_border_tt">Memory</td>
<td id="S4.T1.12.12.13.1.6" class="ltx_td ltx_align_right ltx_border_tt">Validation</td>
<td id="S4.T1.12.12.13.1.7" class="ltx_td ltx_align_right ltx_border_tt">Validation</td>
<td id="S4.T1.12.12.13.1.8" class="ltx_td ltx_align_right ltx_border_tt">GLUE</td>
<td id="S4.T1.12.12.13.1.9" class="ltx_td ltx_align_right ltx_border_tt">XSUM</td>
</tr>
<tr id="S4.T1.3.3.3" class="ltx_tr">
<td id="S4.T1.1.1.1.1" class="ltx_td ltx_align_left"><math id="S4.T1.1.1.1.1.m1.1" class="ltx_Math" alttext="\phi" display="inline"><semantics id="S4.T1.1.1.1.1.m1.1a"><mi id="S4.T1.1.1.1.1.m1.1.1" xref="S4.T1.1.1.1.1.m1.1.1.cmml">ϕ</mi><annotation-xml encoding="MathML-Content" id="S4.T1.1.1.1.1.m1.1b"><ci id="S4.T1.1.1.1.1.m1.1.1.cmml" xref="S4.T1.1.1.1.1.m1.1.1">italic-ϕ</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.1.1.1.1.m1.1c">\phi</annotation></semantics></math></td>
<td id="S4.T1.3.3.3.3" class="ltx_td ltx_align_left">(K<math id="S4.T1.2.2.2.2.m1.1" class="ltx_Math" alttext="a" display="inline"><semantics id="S4.T1.2.2.2.2.m1.1a"><mi id="S4.T1.2.2.2.2.m1.1.1" xref="S4.T1.2.2.2.2.m1.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="S4.T1.2.2.2.2.m1.1b"><ci id="S4.T1.2.2.2.2.m1.1.1.cmml" xref="S4.T1.2.2.2.2.m1.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.2.2.2.2.m1.1c">a</annotation></semantics></math>H<math id="S4.T1.3.3.3.3.m2.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S4.T1.3.3.3.3.m2.1a"><mi id="S4.T1.3.3.3.3.m2.1.1" xref="S4.T1.3.3.3.3.m2.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S4.T1.3.3.3.3.m2.1b"><ci id="S4.T1.3.3.3.3.m2.1.1.cmml" xref="S4.T1.3.3.3.3.m2.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.3.3.3.3.m2.1c">b</annotation></semantics></math>)</td>
<td id="S4.T1.3.3.3.4" class="ltx_td ltx_align_right">Count</td>
<td id="S4.T1.3.3.3.5" class="ltx_td ltx_align_right">Cost (TFLOPs)</td>
<td id="S4.T1.3.3.3.6" class="ltx_td ltx_align_right">Cost (Gb)</td>
<td id="S4.T1.3.3.3.7" class="ltx_td ltx_align_right">Loss</td>
<td id="S4.T1.3.3.3.8" class="ltx_td ltx_align_right">Perplexity</td>
<td id="S4.T1.3.3.3.9" class="ltx_td ltx_align_right">(Average)</td>
<td id="S4.T1.3.3.3.10" class="ltx_td ltx_align_right">(Average)</td>
</tr>
<tr id="S4.T1.12.12.14.2" class="ltx_tr">
<td id="S4.T1.12.12.14.2.1" class="ltx_td ltx_align_center ltx_border_tt" colspan="9"><span id="S4.T1.12.12.14.2.1.1" class="ltx_text ltx_font_italic">small</span></td>
</tr>
<tr id="S4.T1.12.12.15.3" class="ltx_tr">
<td id="S4.T1.12.12.15.3.1" class="ltx_td ltx_align_left ltx_border_t">GPT2</td>
<td id="S4.T1.12.12.15.3.2" class="ltx_td ltx_align_left ltx_border_t">K1H4</td>
<td id="S4.T1.12.12.15.3.3" class="ltx_td ltx_align_right ltx_border_t">122M</td>
<td id="S4.T1.12.12.15.3.4" class="ltx_td ltx_align_right ltx_border_t">2.92</td>
<td id="S4.T1.12.12.15.3.5" class="ltx_td ltx_align_right ltx_border_t">2.98</td>
<td id="S4.T1.12.12.15.3.6" class="ltx_td ltx_align_right ltx_border_t">3.10</td>
<td id="S4.T1.12.12.15.3.7" class="ltx_td ltx_align_right ltx_border_t">22.22</td>
<td id="S4.T1.12.12.15.3.8" class="ltx_td ltx_align_right ltx_border_t">75.32</td>
<td id="S4.T1.12.12.15.3.9" class="ltx_td ltx_align_right ltx_border_t"><span id="S4.T1.12.12.15.3.9.1" class="ltx_text ltx_framed ltx_framed_underline">14.26</span></td>
</tr>
<tr id="S4.T1.12.12.16.4" class="ltx_tr">
<td id="S4.T1.12.12.16.4.1" class="ltx_td ltx_align_left">MoD</td>
<td id="S4.T1.12.12.16.4.2" class="ltx_td ltx_align_left">K1H4S</td>
<td id="S4.T1.12.12.16.4.3" class="ltx_td ltx_align_right">122M</td>
<td id="S4.T1.12.12.16.4.4" class="ltx_td ltx_align_right">-</td>
<td id="S4.T1.12.12.16.4.5" class="ltx_td ltx_align_right">-</td>
<td id="S4.T1.12.12.16.4.6" class="ltx_td ltx_align_right">3.22</td>
<td id="S4.T1.12.12.16.4.7" class="ltx_td ltx_align_right">25.11</td>
<td id="S4.T1.12.12.16.4.8" class="ltx_td ltx_align_right">72.24</td>
<td id="S4.T1.12.12.16.4.9" class="ltx_td ltx_align_right">9.71</td>
</tr>
<tr id="S4.T1.12.12.17.5" class="ltx_tr">
<td id="S4.T1.12.12.17.5.1" class="ltx_td ltx_align_left">MoE</td>
<td id="S4.T1.12.12.17.5.2" class="ltx_td ltx_align_left">K2H4</td>
<td id="S4.T1.12.12.17.5.3" class="ltx_td ltx_align_right">283M</td>
<td id="S4.T1.12.12.17.5.4" class="ltx_td ltx_align_right">3.81 (<span id="S4.T1.12.12.17.5.4.1" class="ltx_text" style="color:#FF0000;">+30.5%</span>)</td>
<td id="S4.T1.12.12.17.5.5" class="ltx_td ltx_align_right">2.98 (<span id="S4.T1.12.12.17.5.5.1" class="ltx_text" style="color:#FF0000;">+0.0%</span>)</td>
<td id="S4.T1.12.12.17.5.6" class="ltx_td ltx_align_right">3.07</td>
<td id="S4.T1.12.12.17.5.7" class="ltx_td ltx_align_right">21.18</td>
<td id="S4.T1.12.12.17.5.8" class="ltx_td ltx_align_right">77.25</td>
<td id="S4.T1.12.12.17.5.9" class="ltx_td ltx_align_right">14.18</td>
</tr>
<tr id="S4.T1.12.12.18.6" class="ltx_tr">
<td id="S4.T1.12.12.18.6.1" class="ltx_td ltx_align_left">MoE (share)</td>
<td id="S4.T1.12.12.18.6.2" class="ltx_td ltx_align_left">K2H4</td>
<td id="S4.T1.12.12.18.6.3" class="ltx_td ltx_align_right">122M</td>
<td id="S4.T1.12.12.18.6.4" class="ltx_td ltx_align_right">3.81 (<span id="S4.T1.12.12.18.6.4.1" class="ltx_text" style="color:#FF0000;">+30.5%</span>)</td>
<td id="S4.T1.12.12.18.6.5" class="ltx_td ltx_align_right">2.98 (<span id="S4.T1.12.12.18.6.5.1" class="ltx_text" style="color:#FF0000;">+0.0%</span>)</td>
<td id="S4.T1.12.12.18.6.6" class="ltx_td ltx_align_right">3.14</td>
<td id="S4.T1.12.12.18.6.7" class="ltx_td ltx_align_right">23.41</td>
<td id="S4.T1.12.12.18.6.8" class="ltx_td ltx_align_right">75.82</td>
<td id="S4.T1.12.12.18.6.9" class="ltx_td ltx_align_right">14.15</td>
</tr>
<tr id="S4.T1.4.4.4" class="ltx_tr">
<td id="S4.T1.4.4.4.1" class="ltx_td ltx_align_left"><math id="S4.T1.4.4.4.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{E}}" display="inline"><semantics id="S4.T1.4.4.4.1.m1.1a"><msub id="S4.T1.4.4.4.1.m1.1.1" xref="S4.T1.4.4.4.1.m1.1.1.cmml"><mtext id="S4.T1.4.4.4.1.m1.1.1.2" xref="S4.T1.4.4.4.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="S4.T1.4.4.4.1.m1.1.1.3" xref="S4.T1.4.4.4.1.m1.1.1.3a.cmml">E</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.T1.4.4.4.1.m1.1b"><apply id="S4.T1.4.4.4.1.m1.1.1.cmml" xref="S4.T1.4.4.4.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T1.4.4.4.1.m1.1.1.1.cmml" xref="S4.T1.4.4.4.1.m1.1.1">subscript</csymbol><ci id="S4.T1.4.4.4.1.m1.1.1.2a.cmml" xref="S4.T1.4.4.4.1.m1.1.1.2"><mtext id="S4.T1.4.4.4.1.m1.1.1.2.cmml" xref="S4.T1.4.4.4.1.m1.1.1.2">MoM</mtext></ci><ci id="S4.T1.4.4.4.1.m1.1.1.3a.cmml" xref="S4.T1.4.4.4.1.m1.1.1.3"><mtext mathsize="70%" id="S4.T1.4.4.4.1.m1.1.1.3.cmml" xref="S4.T1.4.4.4.1.m1.1.1.3">E</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.4.4.4.1.m1.1c">\text{MoM}_{\text{E}}</annotation></semantics></math></td>
<td id="S4.T1.4.4.4.2" class="ltx_td ltx_align_left">K3H1S</td>
<td id="S4.T1.4.4.4.3" class="ltx_td ltx_align_right">122M</td>
<td id="S4.T1.4.4.4.4" class="ltx_td ltx_align_right">2.45 (<span id="S4.T1.4.4.4.4.1" class="ltx_text" style="color:#0000FF;">-16.1%</span>)</td>
<td id="S4.T1.4.4.4.5" class="ltx_td ltx_align_right">2.45 (<span id="S4.T1.4.4.4.5.1" class="ltx_text" style="color:#0000FF;">-17.8%</span>)</td>
<td id="S4.T1.4.4.4.6" class="ltx_td ltx_align_right">3.16</td>
<td id="S4.T1.4.4.4.7" class="ltx_td ltx_align_right">23.59</td>
<td id="S4.T1.4.4.4.8" class="ltx_td ltx_align_right">75.92</td>
<td id="S4.T1.4.4.4.9" class="ltx_td ltx_align_right">14.17</td>
</tr>
<tr id="S4.T1.5.5.5" class="ltx_tr">
<td id="S4.T1.5.5.5.1" class="ltx_td ltx_align_left"><math id="S4.T1.5.5.5.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{I}}" display="inline"><semantics id="S4.T1.5.5.5.1.m1.1a"><msub id="S4.T1.5.5.5.1.m1.1.1" xref="S4.T1.5.5.5.1.m1.1.1.cmml"><mtext id="S4.T1.5.5.5.1.m1.1.1.2" xref="S4.T1.5.5.5.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="S4.T1.5.5.5.1.m1.1.1.3" xref="S4.T1.5.5.5.1.m1.1.1.3a.cmml">I</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.T1.5.5.5.1.m1.1b"><apply id="S4.T1.5.5.5.1.m1.1.1.cmml" xref="S4.T1.5.5.5.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T1.5.5.5.1.m1.1.1.1.cmml" xref="S4.T1.5.5.5.1.m1.1.1">subscript</csymbol><ci id="S4.T1.5.5.5.1.m1.1.1.2a.cmml" xref="S4.T1.5.5.5.1.m1.1.1.2"><mtext id="S4.T1.5.5.5.1.m1.1.1.2.cmml" xref="S4.T1.5.5.5.1.m1.1.1.2">MoM</mtext></ci><ci id="S4.T1.5.5.5.1.m1.1.1.3a.cmml" xref="S4.T1.5.5.5.1.m1.1.1.3"><mtext mathsize="70%" id="S4.T1.5.5.5.1.m1.1.1.3.cmml" xref="S4.T1.5.5.5.1.m1.1.1.3">I</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.5.5.5.1.m1.1c">\text{MoM}_{\text{I}}</annotation></semantics></math></td>
<td id="S4.T1.5.5.5.2" class="ltx_td ltx_align_left">K3H2S</td>
<td id="S4.T1.5.5.5.3" class="ltx_td ltx_align_right">122M</td>
<td id="S4.T1.5.5.5.4" class="ltx_td ltx_align_right">3.49 (<span id="S4.T1.5.5.5.4.1" class="ltx_text" style="color:#FF0000;">+19.5%</span>)</td>
<td id="S4.T1.5.5.5.5" class="ltx_td ltx_align_right">2.63 (<span id="S4.T1.5.5.5.5.1" class="ltx_text" style="color:#0000FF;">-11.7%</span>)</td>
<td id="S4.T1.5.5.5.6" class="ltx_td ltx_align_right"><span id="S4.T1.5.5.5.6.1" class="ltx_text ltx_framed ltx_framed_underline">3.03</span></td>
<td id="S4.T1.5.5.5.7" class="ltx_td ltx_align_right"><span id="S4.T1.5.5.5.7.1" class="ltx_text ltx_framed ltx_framed_underline">20.79</span></td>
<td id="S4.T1.5.5.5.8" class="ltx_td ltx_align_right"><span id="S4.T1.5.5.5.8.1" class="ltx_text ltx_framed ltx_framed_underline">77.81</span></td>
<td id="S4.T1.5.5.5.9" class="ltx_td ltx_align_right">14.24</td>
</tr>
<tr id="S4.T1.6.6.6" class="ltx_tr">
<td id="S4.T1.6.6.6.1" class="ltx_td ltx_align_left"><math id="S4.T1.6.6.6.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{P}}" display="inline"><semantics id="S4.T1.6.6.6.1.m1.1a"><msub id="S4.T1.6.6.6.1.m1.1.1" xref="S4.T1.6.6.6.1.m1.1.1.cmml"><mtext id="S4.T1.6.6.6.1.m1.1.1.2" xref="S4.T1.6.6.6.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="S4.T1.6.6.6.1.m1.1.1.3" xref="S4.T1.6.6.6.1.m1.1.1.3a.cmml">P</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.T1.6.6.6.1.m1.1b"><apply id="S4.T1.6.6.6.1.m1.1.1.cmml" xref="S4.T1.6.6.6.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T1.6.6.6.1.m1.1.1.1.cmml" xref="S4.T1.6.6.6.1.m1.1.1">subscript</csymbol><ci id="S4.T1.6.6.6.1.m1.1.1.2a.cmml" xref="S4.T1.6.6.6.1.m1.1.1.2"><mtext id="S4.T1.6.6.6.1.m1.1.1.2.cmml" xref="S4.T1.6.6.6.1.m1.1.1.2">MoM</mtext></ci><ci id="S4.T1.6.6.6.1.m1.1.1.3a.cmml" xref="S4.T1.6.6.6.1.m1.1.1.3"><mtext mathsize="70%" id="S4.T1.6.6.6.1.m1.1.1.3.cmml" xref="S4.T1.6.6.6.1.m1.1.1.3">P</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.6.6.6.1.m1.1c">\text{MoM}_{\text{P}}</annotation></semantics></math></td>
<td id="S4.T1.6.6.6.2" class="ltx_td ltx_align_left">K2H6S</td>
<td id="S4.T1.6.6.6.3" class="ltx_td ltx_align_right">122M</td>
<td id="S4.T1.6.6.6.4" class="ltx_td ltx_align_right">5.04 (<span id="S4.T1.6.6.6.4.1" class="ltx_text" style="color:#FF0000;">+72.6%</span>)</td>
<td id="S4.T1.6.6.6.5" class="ltx_td ltx_align_right">3.34 (<span id="S4.T1.6.6.6.5.1" class="ltx_text" style="color:#FF0000;">+12.1%</span>)</td>
<td id="S4.T1.6.6.6.6" class="ltx_td ltx_align_right"><span id="S4.T1.6.6.6.6.1" class="ltx_text ltx_font_bold">2.98</span></td>
<td id="S4.T1.6.6.6.7" class="ltx_td ltx_align_right"><span id="S4.T1.6.6.6.7.1" class="ltx_text ltx_font_bold">19.59</span></td>
<td id="S4.T1.6.6.6.8" class="ltx_td ltx_align_right"><span id="S4.T1.6.6.6.8.1" class="ltx_text ltx_font_bold">78.22</span></td>
<td id="S4.T1.6.6.6.9" class="ltx_td ltx_align_right"><span id="S4.T1.6.6.6.9.1" class="ltx_text ltx_font_bold">15.19</span></td>
</tr>
<tr id="S4.T1.12.12.19.7" class="ltx_tr">
<td id="S4.T1.12.12.19.7.1" class="ltx_td ltx_align_center ltx_border_t" colspan="9"><span id="S4.T1.12.12.19.7.1.1" class="ltx_text ltx_font_italic">medium</span></td>
</tr>
<tr id="S4.T1.12.12.20.8" class="ltx_tr">
<td id="S4.T1.12.12.20.8.1" class="ltx_td ltx_align_left ltx_border_t">GPT2</td>
<td id="S4.T1.12.12.20.8.2" class="ltx_td ltx_align_left ltx_border_t">K1H4</td>
<td id="S4.T1.12.12.20.8.3" class="ltx_td ltx_align_right ltx_border_t">346M</td>
<td id="S4.T1.12.12.20.8.4" class="ltx_td ltx_align_right ltx_border_t">8.28</td>
<td id="S4.T1.12.12.20.8.5" class="ltx_td ltx_align_right ltx_border_t">4.74</td>
<td id="S4.T1.12.12.20.8.6" class="ltx_td ltx_align_right ltx_border_t">2.81</td>
<td id="S4.T1.12.12.20.8.7" class="ltx_td ltx_align_right ltx_border_t">16.69</td>
<td id="S4.T1.12.12.20.8.8" class="ltx_td ltx_align_right ltx_border_t">80.49</td>
<td id="S4.T1.12.12.20.8.9" class="ltx_td ltx_align_right ltx_border_t">18.14</td>
</tr>
<tr id="S4.T1.12.12.21.9" class="ltx_tr">
<td id="S4.T1.12.12.21.9.1" class="ltx_td ltx_align_left">MoD</td>
<td id="S4.T1.12.12.21.9.2" class="ltx_td ltx_align_left">K1H4S</td>
<td id="S4.T1.12.12.21.9.3" class="ltx_td ltx_align_right">346M</td>
<td id="S4.T1.12.12.21.9.4" class="ltx_td ltx_align_right">-</td>
<td id="S4.T1.12.12.21.9.5" class="ltx_td ltx_align_right">-</td>
<td id="S4.T1.12.12.21.9.6" class="ltx_td ltx_align_right">2.99</td>
<td id="S4.T1.12.12.21.9.7" class="ltx_td ltx_align_right">19.82</td>
<td id="S4.T1.12.12.21.9.8" class="ltx_td ltx_align_right">76.17</td>
<td id="S4.T1.12.12.21.9.9" class="ltx_td ltx_align_right">14.81</td>
</tr>
<tr id="S4.T1.12.12.22.10" class="ltx_tr">
<td id="S4.T1.12.12.22.10.1" class="ltx_td ltx_align_left">MoE</td>
<td id="S4.T1.12.12.22.10.2" class="ltx_td ltx_align_left">K2H4</td>
<td id="S4.T1.12.12.22.10.3" class="ltx_td ltx_align_right">921M</td>
<td id="S4.T1.12.12.22.10.4" class="ltx_td ltx_align_right">11.37 (<span id="S4.T1.12.12.22.10.4.1" class="ltx_text" style="color:#FF0000;">+37.3%</span>)</td>
<td id="S4.T1.12.12.22.10.5" class="ltx_td ltx_align_right">4.74 (<span id="S4.T1.12.12.22.10.5.1" class="ltx_text" style="color:#FF0000;">+0.0%</span>)</td>
<td id="S4.T1.12.12.22.10.6" class="ltx_td ltx_align_right">2.80</td>
<td id="S4.T1.12.12.22.10.7" class="ltx_td ltx_align_right">16.53</td>
<td id="S4.T1.12.12.22.10.8" class="ltx_td ltx_align_right">80.47</td>
<td id="S4.T1.12.12.22.10.9" class="ltx_td ltx_align_right">17.75</td>
</tr>
<tr id="S4.T1.12.12.23.11" class="ltx_tr">
<td id="S4.T1.12.12.23.11.1" class="ltx_td ltx_align_left">MoE (share)</td>
<td id="S4.T1.12.12.23.11.2" class="ltx_td ltx_align_left">K2H4</td>
<td id="S4.T1.12.12.23.11.3" class="ltx_td ltx_align_right">346M</td>
<td id="S4.T1.12.12.23.11.4" class="ltx_td ltx_align_right">11.37 (<span id="S4.T1.12.12.23.11.4.1" class="ltx_text" style="color:#FF0000;">+37.3%</span>)</td>
<td id="S4.T1.12.12.23.11.5" class="ltx_td ltx_align_right">4.74 (<span id="S4.T1.12.12.23.11.5.1" class="ltx_text" style="color:#FF0000;">+0.0%</span>)</td>
<td id="S4.T1.12.12.23.11.6" class="ltx_td ltx_align_right">2.82</td>
<td id="S4.T1.12.12.23.11.7" class="ltx_td ltx_align_right">16.81</td>
<td id="S4.T1.12.12.23.11.8" class="ltx_td ltx_align_right">80.35</td>
<td id="S4.T1.12.12.23.11.9" class="ltx_td ltx_align_right">17.59</td>
</tr>
<tr id="S4.T1.7.7.7" class="ltx_tr">
<td id="S4.T1.7.7.7.1" class="ltx_td ltx_align_left"><math id="S4.T1.7.7.7.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{E}}" display="inline"><semantics id="S4.T1.7.7.7.1.m1.1a"><msub id="S4.T1.7.7.7.1.m1.1.1" xref="S4.T1.7.7.7.1.m1.1.1.cmml"><mtext id="S4.T1.7.7.7.1.m1.1.1.2" xref="S4.T1.7.7.7.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="S4.T1.7.7.7.1.m1.1.1.3" xref="S4.T1.7.7.7.1.m1.1.1.3a.cmml">E</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.T1.7.7.7.1.m1.1b"><apply id="S4.T1.7.7.7.1.m1.1.1.cmml" xref="S4.T1.7.7.7.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T1.7.7.7.1.m1.1.1.1.cmml" xref="S4.T1.7.7.7.1.m1.1.1">subscript</csymbol><ci id="S4.T1.7.7.7.1.m1.1.1.2a.cmml" xref="S4.T1.7.7.7.1.m1.1.1.2"><mtext id="S4.T1.7.7.7.1.m1.1.1.2.cmml" xref="S4.T1.7.7.7.1.m1.1.1.2">MoM</mtext></ci><ci id="S4.T1.7.7.7.1.m1.1.1.3a.cmml" xref="S4.T1.7.7.7.1.m1.1.1.3"><mtext mathsize="70%" id="S4.T1.7.7.7.1.m1.1.1.3.cmml" xref="S4.T1.7.7.7.1.m1.1.1.3">E</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.7.7.7.1.m1.1c">\text{MoM}_{\text{E}}</annotation></semantics></math></td>
<td id="S4.T1.7.7.7.2" class="ltx_td ltx_align_left">K3H1S</td>
<td id="S4.T1.7.7.7.3" class="ltx_td ltx_align_right">346M</td>
<td id="S4.T1.7.7.7.4" class="ltx_td ltx_align_right">6.80 (<span id="S4.T1.7.7.7.4.1" class="ltx_text" style="color:#0000FF;">-17.9%</span>)</td>
<td id="S4.T1.7.7.7.5" class="ltx_td ltx_align_right">3.33 (<span id="S4.T1.7.7.7.5.1" class="ltx_text" style="color:#0000FF;">-29.7%</span>)</td>
<td id="S4.T1.7.7.7.6" class="ltx_td ltx_align_right">2.83</td>
<td id="S4.T1.7.7.7.7" class="ltx_td ltx_align_right">16.91</td>
<td id="S4.T1.7.7.7.8" class="ltx_td ltx_align_right">80.41</td>
<td id="S4.T1.7.7.7.9" class="ltx_td ltx_align_right">17.11</td>
</tr>
<tr id="S4.T1.8.8.8" class="ltx_tr">
<td id="S4.T1.8.8.8.1" class="ltx_td ltx_align_left"><math id="S4.T1.8.8.8.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{I}}" display="inline"><semantics id="S4.T1.8.8.8.1.m1.1a"><msub id="S4.T1.8.8.8.1.m1.1.1" xref="S4.T1.8.8.8.1.m1.1.1.cmml"><mtext id="S4.T1.8.8.8.1.m1.1.1.2" xref="S4.T1.8.8.8.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="S4.T1.8.8.8.1.m1.1.1.3" xref="S4.T1.8.8.8.1.m1.1.1.3a.cmml">I</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.T1.8.8.8.1.m1.1b"><apply id="S4.T1.8.8.8.1.m1.1.1.cmml" xref="S4.T1.8.8.8.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T1.8.8.8.1.m1.1.1.1.cmml" xref="S4.T1.8.8.8.1.m1.1.1">subscript</csymbol><ci id="S4.T1.8.8.8.1.m1.1.1.2a.cmml" xref="S4.T1.8.8.8.1.m1.1.1.2"><mtext id="S4.T1.8.8.8.1.m1.1.1.2.cmml" xref="S4.T1.8.8.8.1.m1.1.1.2">MoM</mtext></ci><ci id="S4.T1.8.8.8.1.m1.1.1.3a.cmml" xref="S4.T1.8.8.8.1.m1.1.1.3"><mtext mathsize="70%" id="S4.T1.8.8.8.1.m1.1.1.3.cmml" xref="S4.T1.8.8.8.1.m1.1.1.3">I</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.8.8.8.1.m1.1c">\text{MoM}_{\text{I}}</annotation></semantics></math></td>
<td id="S4.T1.8.8.8.2" class="ltx_td ltx_align_left">K3H2S</td>
<td id="S4.T1.8.8.8.3" class="ltx_td ltx_align_right">346M</td>
<td id="S4.T1.8.8.8.4" class="ltx_td ltx_align_right">10.20 (<span id="S4.T1.8.8.8.4.1" class="ltx_text" style="color:#FF0000;">+23.2%</span>)</td>
<td id="S4.T1.8.8.8.5" class="ltx_td ltx_align_right">3.80 (<span id="S4.T1.8.8.8.5.1" class="ltx_text" style="color:#0000FF;">-19.8%</span>)</td>
<td id="S4.T1.8.8.8.6" class="ltx_td ltx_align_right"><span id="S4.T1.8.8.8.6.1" class="ltx_text ltx_framed ltx_framed_underline">2.77</span></td>
<td id="S4.T1.8.8.8.7" class="ltx_td ltx_align_right"><span id="S4.T1.8.8.8.7.1" class="ltx_text ltx_framed ltx_framed_underline">15.89</span></td>
<td id="S4.T1.8.8.8.8" class="ltx_td ltx_align_right"><span id="S4.T1.8.8.8.8.1" class="ltx_text ltx_framed ltx_framed_underline">81.03</span></td>
<td id="S4.T1.8.8.8.9" class="ltx_td ltx_align_right"><span id="S4.T1.8.8.8.9.1" class="ltx_text ltx_framed ltx_framed_underline">18.66</span></td>
</tr>
<tr id="S4.T1.9.9.9" class="ltx_tr">
<td id="S4.T1.9.9.9.1" class="ltx_td ltx_align_left"><math id="S4.T1.9.9.9.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{P}}" display="inline"><semantics id="S4.T1.9.9.9.1.m1.1a"><msub id="S4.T1.9.9.9.1.m1.1.1" xref="S4.T1.9.9.9.1.m1.1.1.cmml"><mtext id="S4.T1.9.9.9.1.m1.1.1.2" xref="S4.T1.9.9.9.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="S4.T1.9.9.9.1.m1.1.1.3" xref="S4.T1.9.9.9.1.m1.1.1.3a.cmml">P</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.T1.9.9.9.1.m1.1b"><apply id="S4.T1.9.9.9.1.m1.1.1.cmml" xref="S4.T1.9.9.9.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T1.9.9.9.1.m1.1.1.1.cmml" xref="S4.T1.9.9.9.1.m1.1.1">subscript</csymbol><ci id="S4.T1.9.9.9.1.m1.1.1.2a.cmml" xref="S4.T1.9.9.9.1.m1.1.1.2"><mtext id="S4.T1.9.9.9.1.m1.1.1.2.cmml" xref="S4.T1.9.9.9.1.m1.1.1.2">MoM</mtext></ci><ci id="S4.T1.9.9.9.1.m1.1.1.3a.cmml" xref="S4.T1.9.9.9.1.m1.1.1.3"><mtext mathsize="70%" id="S4.T1.9.9.9.1.m1.1.1.3.cmml" xref="S4.T1.9.9.9.1.m1.1.1.3">P</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.9.9.9.1.m1.1c">\text{MoM}_{\text{P}}</annotation></semantics></math></td>
<td id="S4.T1.9.9.9.2" class="ltx_td ltx_align_left">K2H6S</td>
<td id="S4.T1.9.9.9.3" class="ltx_td ltx_align_right">346M</td>
<td id="S4.T1.9.9.9.4" class="ltx_td ltx_align_right">16.23 (<span id="S4.T1.9.9.9.4.1" class="ltx_text" style="color:#FF0000;">+96.0%</span>)</td>
<td id="S4.T1.9.9.9.5" class="ltx_td ltx_align_right">5.69 (<span id="S4.T1.9.9.9.5.1" class="ltx_text" style="color:#FF0000;">+20.0%</span>)</td>
<td id="S4.T1.9.9.9.6" class="ltx_td ltx_align_right"><span id="S4.T1.9.9.9.6.1" class="ltx_text ltx_font_bold">2.72</span></td>
<td id="S4.T1.9.9.9.7" class="ltx_td ltx_align_right"><span id="S4.T1.9.9.9.7.1" class="ltx_text ltx_font_bold">15.18</span></td>
<td id="S4.T1.9.9.9.8" class="ltx_td ltx_align_right"><span id="S4.T1.9.9.9.8.1" class="ltx_text ltx_font_bold">81.93</span></td>
<td id="S4.T1.9.9.9.9" class="ltx_td ltx_align_right"><span id="S4.T1.9.9.9.9.1" class="ltx_text ltx_font_bold">19.30</span></td>
</tr>
<tr id="S4.T1.12.12.24.12" class="ltx_tr">
<td id="S4.T1.12.12.24.12.1" class="ltx_td ltx_align_center ltx_border_t" colspan="9"><span id="S4.T1.12.12.24.12.1.1" class="ltx_text ltx_font_italic">large</span></td>
</tr>
<tr id="S4.T1.12.12.25.13" class="ltx_tr">
<td id="S4.T1.12.12.25.13.1" class="ltx_td ltx_align_left ltx_border_t">GPT2</td>
<td id="S4.T1.12.12.25.13.2" class="ltx_td ltx_align_left ltx_border_t">K1H4</td>
<td id="S4.T1.12.12.25.13.3" class="ltx_td ltx_align_right ltx_border_t">774M</td>
<td id="S4.T1.12.12.25.13.4" class="ltx_td ltx_align_right ltx_border_t">17.76</td>
<td id="S4.T1.12.12.25.13.5" class="ltx_td ltx_align_right ltx_border_t">7.20</td>
<td id="S4.T1.12.12.25.13.6" class="ltx_td ltx_align_right ltx_border_t">2.66</td>
<td id="S4.T1.12.12.25.13.7" class="ltx_td ltx_align_right ltx_border_t">14.33</td>
<td id="S4.T1.12.12.25.13.8" class="ltx_td ltx_align_right ltx_border_t">84.47</td>
<td id="S4.T1.12.12.25.13.9" class="ltx_td ltx_align_right ltx_border_t">20.35</td>
</tr>
<tr id="S4.T1.12.12.26.14" class="ltx_tr">
<td id="S4.T1.12.12.26.14.1" class="ltx_td ltx_align_left">MoD</td>
<td id="S4.T1.12.12.26.14.2" class="ltx_td ltx_align_left">K1H4S</td>
<td id="S4.T1.12.12.26.14.3" class="ltx_td ltx_align_right">774M</td>
<td id="S4.T1.12.12.26.14.4" class="ltx_td ltx_align_right">-</td>
<td id="S4.T1.12.12.26.14.5" class="ltx_td ltx_align_right">-</td>
<td id="S4.T1.12.12.26.14.6" class="ltx_td ltx_align_right">2.81</td>
<td id="S4.T1.12.12.26.14.7" class="ltx_td ltx_align_right">16.62</td>
<td id="S4.T1.12.12.26.14.8" class="ltx_td ltx_align_right">81.49</td>
<td id="S4.T1.12.12.26.14.9" class="ltx_td ltx_align_right">18.62</td>
</tr>
<tr id="S4.T1.12.12.27.15" class="ltx_tr">
<td id="S4.T1.12.12.27.15.1" class="ltx_td ltx_align_left">MoE</td>
<td id="S4.T1.12.12.27.15.2" class="ltx_td ltx_align_left">K2H4</td>
<td id="S4.T1.12.12.27.15.3" class="ltx_td ltx_align_right">2100M</td>
<td id="S4.T1.12.12.27.15.4" class="ltx_td ltx_align_right">25.43 (<span id="S4.T1.12.12.27.15.4.1" class="ltx_text" style="color:#FF0000;">+43.2%</span>)</td>
<td id="S4.T1.12.12.27.15.5" class="ltx_td ltx_align_right">7.20 (<span id="S4.T1.12.12.27.15.5.1" class="ltx_text" style="color:#FF0000;">+0.0%</span>)</td>
<td id="S4.T1.12.12.27.15.6" class="ltx_td ltx_align_right"><span id="S4.T1.12.12.27.15.6.1" class="ltx_text ltx_framed ltx_framed_underline">2.64</span></td>
<td id="S4.T1.12.12.27.15.7" class="ltx_td ltx_align_right">14.17</td>
<td id="S4.T1.12.12.27.15.8" class="ltx_td ltx_align_right">84.43</td>
<td id="S4.T1.12.12.27.15.9" class="ltx_td ltx_align_right">20.63</td>
</tr>
<tr id="S4.T1.12.12.28.16" class="ltx_tr">
<td id="S4.T1.12.12.28.16.1" class="ltx_td ltx_align_left">MoE (share)</td>
<td id="S4.T1.12.12.28.16.2" class="ltx_td ltx_align_left">K2H4</td>
<td id="S4.T1.12.12.28.16.3" class="ltx_td ltx_align_right">774M</td>
<td id="S4.T1.12.12.28.16.4" class="ltx_td ltx_align_right">25.43 (<span id="S4.T1.12.12.28.16.4.1" class="ltx_text" style="color:#FF0000;">+43.2%</span>)</td>
<td id="S4.T1.12.12.28.16.5" class="ltx_td ltx_align_right">7.20 (<span id="S4.T1.12.12.28.16.5.1" class="ltx_text" style="color:#FF0000;">+0.0%</span>)</td>
<td id="S4.T1.12.12.28.16.6" class="ltx_td ltx_align_right">2.65</td>
<td id="S4.T1.12.12.28.16.7" class="ltx_td ltx_align_right">14.22</td>
<td id="S4.T1.12.12.28.16.8" class="ltx_td ltx_align_right">83.83</td>
<td id="S4.T1.12.12.28.16.9" class="ltx_td ltx_align_right">20.39</td>
</tr>
<tr id="S4.T1.10.10.10" class="ltx_tr">
<td id="S4.T1.10.10.10.1" class="ltx_td ltx_align_left"><math id="S4.T1.10.10.10.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{E}}" display="inline"><semantics id="S4.T1.10.10.10.1.m1.1a"><msub id="S4.T1.10.10.10.1.m1.1.1" xref="S4.T1.10.10.10.1.m1.1.1.cmml"><mtext id="S4.T1.10.10.10.1.m1.1.1.2" xref="S4.T1.10.10.10.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="S4.T1.10.10.10.1.m1.1.1.3" xref="S4.T1.10.10.10.1.m1.1.1.3a.cmml">E</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.T1.10.10.10.1.m1.1b"><apply id="S4.T1.10.10.10.1.m1.1.1.cmml" xref="S4.T1.10.10.10.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T1.10.10.10.1.m1.1.1.1.cmml" xref="S4.T1.10.10.10.1.m1.1.1">subscript</csymbol><ci id="S4.T1.10.10.10.1.m1.1.1.2a.cmml" xref="S4.T1.10.10.10.1.m1.1.1.2"><mtext id="S4.T1.10.10.10.1.m1.1.1.2.cmml" xref="S4.T1.10.10.10.1.m1.1.1.2">MoM</mtext></ci><ci id="S4.T1.10.10.10.1.m1.1.1.3a.cmml" xref="S4.T1.10.10.10.1.m1.1.1.3"><mtext mathsize="70%" id="S4.T1.10.10.10.1.m1.1.1.3.cmml" xref="S4.T1.10.10.10.1.m1.1.1.3">E</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.10.10.10.1.m1.1c">\text{MoM}_{\text{E}}</annotation></semantics></math></td>
<td id="S4.T1.10.10.10.2" class="ltx_td ltx_align_left">K3H1S</td>
<td id="S4.T1.10.10.10.3" class="ltx_td ltx_align_right">774M</td>
<td id="S4.T1.10.10.10.4" class="ltx_td ltx_align_right">14.84 (<span id="S4.T1.10.10.10.4.1" class="ltx_text" style="color:#0000FF;">-16.4%</span>)</td>
<td id="S4.T1.10.10.10.5" class="ltx_td ltx_align_right">4.13 (<span id="S4.T1.10.10.10.5.1" class="ltx_text" style="color:#0000FF;">-42.6%</span>)</td>
<td id="S4.T1.10.10.10.6" class="ltx_td ltx_align_right">2.66</td>
<td id="S4.T1.10.10.10.7" class="ltx_td ltx_align_right">14.50</td>
<td id="S4.T1.10.10.10.8" class="ltx_td ltx_align_right">83.39</td>
<td id="S4.T1.10.10.10.9" class="ltx_td ltx_align_right">20.46</td>
</tr>
<tr id="S4.T1.11.11.11" class="ltx_tr">
<td id="S4.T1.11.11.11.1" class="ltx_td ltx_align_left"><math id="S4.T1.11.11.11.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{I}}" display="inline"><semantics id="S4.T1.11.11.11.1.m1.1a"><msub id="S4.T1.11.11.11.1.m1.1.1" xref="S4.T1.11.11.11.1.m1.1.1.cmml"><mtext id="S4.T1.11.11.11.1.m1.1.1.2" xref="S4.T1.11.11.11.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="S4.T1.11.11.11.1.m1.1.1.3" xref="S4.T1.11.11.11.1.m1.1.1.3a.cmml">I</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.T1.11.11.11.1.m1.1b"><apply id="S4.T1.11.11.11.1.m1.1.1.cmml" xref="S4.T1.11.11.11.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T1.11.11.11.1.m1.1.1.1.cmml" xref="S4.T1.11.11.11.1.m1.1.1">subscript</csymbol><ci id="S4.T1.11.11.11.1.m1.1.1.2a.cmml" xref="S4.T1.11.11.11.1.m1.1.1.2"><mtext id="S4.T1.11.11.11.1.m1.1.1.2.cmml" xref="S4.T1.11.11.11.1.m1.1.1.2">MoM</mtext></ci><ci id="S4.T1.11.11.11.1.m1.1.1.3a.cmml" xref="S4.T1.11.11.11.1.m1.1.1.3"><mtext mathsize="70%" id="S4.T1.11.11.11.1.m1.1.1.3.cmml" xref="S4.T1.11.11.11.1.m1.1.1.3">I</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.11.11.11.1.m1.1c">\text{MoM}_{\text{I}}</annotation></semantics></math></td>
<td id="S4.T1.11.11.11.2" class="ltx_td ltx_align_left">K3H2S</td>
<td id="S4.T1.11.11.11.3" class="ltx_td ltx_align_right">774M</td>
<td id="S4.T1.11.11.11.4" class="ltx_td ltx_align_right">20.31 (<span id="S4.T1.11.11.11.4.1" class="ltx_text" style="color:#FF0000;">+14.5%</span>)</td>
<td id="S4.T1.11.11.11.5" class="ltx_td ltx_align_right">5.15 (<span id="S4.T1.11.11.11.5.1" class="ltx_text" style="color:#0000FF;">-28.5%</span>)</td>
<td id="S4.T1.11.11.11.6" class="ltx_td ltx_align_right"><span id="S4.T1.11.11.11.6.1" class="ltx_text ltx_framed ltx_framed_underline">2.64</span></td>
<td id="S4.T1.11.11.11.7" class="ltx_td ltx_align_right"><span id="S4.T1.11.11.11.7.1" class="ltx_text ltx_framed ltx_framed_underline">13.92</span></td>
<td id="S4.T1.11.11.11.8" class="ltx_td ltx_align_right"><span id="S4.T1.11.11.11.8.1" class="ltx_text ltx_framed ltx_framed_underline">84.49</span></td>
<td id="S4.T1.11.11.11.9" class="ltx_td ltx_align_right"><span id="S4.T1.11.11.11.9.1" class="ltx_text ltx_framed ltx_framed_underline">21.73</span></td>
</tr>
<tr id="S4.T1.12.12.12" class="ltx_tr">
<td id="S4.T1.12.12.12.1" class="ltx_td ltx_align_left ltx_border_bb"><math id="S4.T1.12.12.12.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{P}}" display="inline"><semantics id="S4.T1.12.12.12.1.m1.1a"><msub id="S4.T1.12.12.12.1.m1.1.1" xref="S4.T1.12.12.12.1.m1.1.1.cmml"><mtext id="S4.T1.12.12.12.1.m1.1.1.2" xref="S4.T1.12.12.12.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="S4.T1.12.12.12.1.m1.1.1.3" xref="S4.T1.12.12.12.1.m1.1.1.3a.cmml">P</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.T1.12.12.12.1.m1.1b"><apply id="S4.T1.12.12.12.1.m1.1.1.cmml" xref="S4.T1.12.12.12.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T1.12.12.12.1.m1.1.1.1.cmml" xref="S4.T1.12.12.12.1.m1.1.1">subscript</csymbol><ci id="S4.T1.12.12.12.1.m1.1.1.2a.cmml" xref="S4.T1.12.12.12.1.m1.1.1.2"><mtext id="S4.T1.12.12.12.1.m1.1.1.2.cmml" xref="S4.T1.12.12.12.1.m1.1.1.2">MoM</mtext></ci><ci id="S4.T1.12.12.12.1.m1.1.1.3a.cmml" xref="S4.T1.12.12.12.1.m1.1.1.3"><mtext mathsize="70%" id="S4.T1.12.12.12.1.m1.1.1.3.cmml" xref="S4.T1.12.12.12.1.m1.1.1.3">P</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.12.12.12.1.m1.1c">\text{MoM}_{\text{P}}</annotation></semantics></math></td>
<td id="S4.T1.12.12.12.2" class="ltx_td ltx_align_left ltx_border_bb">K2H6S</td>
<td id="S4.T1.12.12.12.3" class="ltx_td ltx_align_right ltx_border_bb">774M</td>
<td id="S4.T1.12.12.12.4" class="ltx_td ltx_align_right ltx_border_bb">36.07 (<span id="S4.T1.12.12.12.4.1" class="ltx_text" style="color:#FF0000;">+103.1%</span>)</td>
<td id="S4.T1.12.12.12.5" class="ltx_td ltx_align_right ltx_border_bb">9.24 (<span id="S4.T1.12.12.12.5.1" class="ltx_text" style="color:#FF0000;">+28.3%</span>)</td>
<td id="S4.T1.12.12.12.6" class="ltx_td ltx_align_right ltx_border_bb"><span id="S4.T1.12.12.12.6.1" class="ltx_text ltx_font_bold">2.60</span></td>
<td id="S4.T1.12.12.12.7" class="ltx_td ltx_align_right ltx_border_bb"><span id="S4.T1.12.12.12.7.1" class="ltx_text ltx_font_bold">13.21</span></td>
<td id="S4.T1.12.12.12.8" class="ltx_td ltx_align_right ltx_border_bb"><span id="S4.T1.12.12.12.8.1" class="ltx_text ltx_font_bold">85.90</span></td>
<td id="S4.T1.12.12.12.9" class="ltx_td ltx_align_right ltx_border_bb"><span id="S4.T1.12.12.12.9.1" class="ltx_text ltx_font_bold">22.36</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>Comprehensive comparison between MoMs and baselines.
We highlight the best results in bold and underline the second-best results.
Appendix <a href="#A3" title="Appendix C Detailed downstream evaluation results ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a> includes the detailed performance on GLUE and XSUM.</figcaption>
</figure>
</section>
<section id="S4.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Baselines.</h4>

<div id="S4.SS1.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px2.p1.1" class="ltx_p">In addition to the vanilla Transformer model <cite class="ltx_cite ltx_citemacro_citep">(Radford et al., <a href="#bib.bib36" title="" class="ltx_ref">2019</a>)</cite>, the following models (or methods) are also employed as baselines: (1) <span id="S4.SS1.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_bold">MoD</span> <cite class="ltx_cite ltx_citemacro_citep">(Raposo et al., <a href="#bib.bib37" title="" class="ltx_ref">2024</a>)</cite>: a layer-skip method proposed recently that dynamically routes around Transformer blocks.<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span>As official code is unavailable until the submission, we follow the paper to implement MoD ourselves.</span></span></span> (2) <span id="S4.SS1.SSS0.Px2.p1.1.2" class="ltx_text ltx_font_bold">MoE</span>: the mixture-of-experts architecture utilized by Mixtral. We implement the model with the open-sourced code.<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span><a target="_blank" href="https://github.com/huggingface/transformers/blob/v4.36.1/src/transformers/models/mixtral/modeling_mixtral.py" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/huggingface/transformers/blob/v4.36.1/src/transformers/models/mixtral/modeling_mixtral.py</a></span></span></span> (3) <span id="S4.SS1.SSS0.Px2.p1.1.3" class="ltx_text ltx_font_bold">MoE (share)</span>: a variant of MoE in which all layers share the same set of experts. We involve this model as a baseline because unlike the standard MoE that has more parameters, MoE-share has the same number of parameters with the vanilla Transformer model, making the comparison more fair. Moreover, it also sheds light on how well the MoE architecture can utilize a fixed budget of modules.</p>
</div>
<div id="S4.SS1.SSS0.Px2.p2" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px2.p2.4" class="ltx_p">Note that all the above-mentioned methods are special cases within the MoM framework with various configurations.
Furthermore, we explore a wide range of MoM instances defined by K<math id="S4.SS1.SSS0.Px2.p2.1.m1.1" class="ltx_Math" alttext="a" display="inline"><semantics id="S4.SS1.SSS0.Px2.p2.1.m1.1a"><mi id="S4.SS1.SSS0.Px2.p2.1.m1.1.1" xref="S4.SS1.SSS0.Px2.p2.1.m1.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p2.1.m1.1b"><ci id="S4.SS1.SSS0.Px2.p2.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p2.1.m1.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p2.1.m1.1c">a</annotation></semantics></math>H<math id="S4.SS1.SSS0.Px2.p2.2.m2.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S4.SS1.SSS0.Px2.p2.2.m2.1a"><mi id="S4.SS1.SSS0.Px2.p2.2.m2.1.1" xref="S4.SS1.SSS0.Px2.p2.2.m2.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p2.2.m2.1b"><ci id="S4.SS1.SSS0.Px2.p2.2.m2.1.1.cmml" xref="S4.SS1.SSS0.Px2.p2.2.m2.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p2.2.m2.1c">b</annotation></semantics></math>S, where <math id="S4.SS1.SSS0.Px2.p2.3.m3.1" class="ltx_Math" alttext="a\leq 4" display="inline"><semantics id="S4.SS1.SSS0.Px2.p2.3.m3.1a"><mrow id="S4.SS1.SSS0.Px2.p2.3.m3.1.1" xref="S4.SS1.SSS0.Px2.p2.3.m3.1.1.cmml"><mi id="S4.SS1.SSS0.Px2.p2.3.m3.1.1.2" xref="S4.SS1.SSS0.Px2.p2.3.m3.1.1.2.cmml">a</mi><mo id="S4.SS1.SSS0.Px2.p2.3.m3.1.1.1" xref="S4.SS1.SSS0.Px2.p2.3.m3.1.1.1.cmml">≤</mo><mn id="S4.SS1.SSS0.Px2.p2.3.m3.1.1.3" xref="S4.SS1.SSS0.Px2.p2.3.m3.1.1.3.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p2.3.m3.1b"><apply id="S4.SS1.SSS0.Px2.p2.3.m3.1.1.cmml" xref="S4.SS1.SSS0.Px2.p2.3.m3.1.1"><leq id="S4.SS1.SSS0.Px2.p2.3.m3.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p2.3.m3.1.1.1"></leq><ci id="S4.SS1.SSS0.Px2.p2.3.m3.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p2.3.m3.1.1.2">𝑎</ci><cn type="integer" id="S4.SS1.SSS0.Px2.p2.3.m3.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p2.3.m3.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p2.3.m3.1c">a\leq 4</annotation></semantics></math> and <math id="S4.SS1.SSS0.Px2.p2.4.m4.1" class="ltx_Math" alttext="b\leq 6" display="inline"><semantics id="S4.SS1.SSS0.Px2.p2.4.m4.1a"><mrow id="S4.SS1.SSS0.Px2.p2.4.m4.1.1" xref="S4.SS1.SSS0.Px2.p2.4.m4.1.1.cmml"><mi id="S4.SS1.SSS0.Px2.p2.4.m4.1.1.2" xref="S4.SS1.SSS0.Px2.p2.4.m4.1.1.2.cmml">b</mi><mo id="S4.SS1.SSS0.Px2.p2.4.m4.1.1.1" xref="S4.SS1.SSS0.Px2.p2.4.m4.1.1.1.cmml">≤</mo><mn id="S4.SS1.SSS0.Px2.p2.4.m4.1.1.3" xref="S4.SS1.SSS0.Px2.p2.4.m4.1.1.3.cmml">6</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p2.4.m4.1b"><apply id="S4.SS1.SSS0.Px2.p2.4.m4.1.1.cmml" xref="S4.SS1.SSS0.Px2.p2.4.m4.1.1"><leq id="S4.SS1.SSS0.Px2.p2.4.m4.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p2.4.m4.1.1.1"></leq><ci id="S4.SS1.SSS0.Px2.p2.4.m4.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p2.4.m4.1.1.2">𝑏</ci><cn type="integer" id="S4.SS1.SSS0.Px2.p2.4.m4.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p2.4.m4.1.1.3">6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p2.4.m4.1c">b\leq 6</annotation></semantics></math>.
We examine all MoM instances within the search space (as detailed in §<a href="#S4.SS3" title="4.3 Insights from hyper-parameter search ‣ 4 Experiments ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.3</span></a>), and spotlight three distinct models for comparison against other baselines:</p>
</div>
<div id="S4.SS1.SSS0.Px2.p3" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px2.p3.4" class="ltx_p"><math id="S4.SS1.SSS0.Px2.p3.1.m1.1" class="ltx_Math" alttext="\bullet" display="inline"><semantics id="S4.SS1.SSS0.Px2.p3.1.m1.1a"><mo id="S4.SS1.SSS0.Px2.p3.1.m1.1.1" xref="S4.SS1.SSS0.Px2.p3.1.m1.1.1.cmml">∙</mo><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p3.1.m1.1b"><ci id="S4.SS1.SSS0.Px2.p3.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.1.m1.1.1">∙</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p3.1.m1.1c">\bullet</annotation></semantics></math> <math id="S4.SS1.SSS0.Px2.p3.2.m2.1" class="ltx_Math" alttext="\text{MoM}_{\text{P}}" display="inline"><semantics id="S4.SS1.SSS0.Px2.p3.2.m2.1a"><msub id="S4.SS1.SSS0.Px2.p3.2.m2.1.1" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.cmml"><mtext class="ltx_mathvariant_bold" id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2a.cmml">MoM</mtext><mtext class="ltx_mathvariant_bold" id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3a.cmml">P</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p3.2.m2.1b"><apply id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1">subscript</csymbol><ci id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2a.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2"><mtext class="ltx_mathvariant_bold" id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2">MoM</mtext></ci><ci id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3a.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3"><mtext class="ltx_mathvariant_bold" mathsize="70%" id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3">P</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p3.2.m2.1c">\text{MoM}_{\text{P}}</annotation></semantics></math> (K2H6S) represents a <span id="S4.SS1.SSS0.Px2.p3.4.1" class="ltx_text ltx_font_bold">P</span>erformant MoM model after tuning <math id="S4.SS1.SSS0.Px2.p3.3.m3.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S4.SS1.SSS0.Px2.p3.3.m3.1a"><mi id="S4.SS1.SSS0.Px2.p3.3.m3.1.1" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p3.3.m3.1b"><ci id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p3.3.m3.1c">K</annotation></semantics></math> and <math id="S4.SS1.SSS0.Px2.p3.4.m4.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S4.SS1.SSS0.Px2.p3.4.m4.1a"><mi id="S4.SS1.SSS0.Px2.p3.4.m4.1.1" xref="S4.SS1.SSS0.Px2.p3.4.m4.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p3.4.m4.1b"><ci id="S4.SS1.SSS0.Px2.p3.4.m4.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.4.m4.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p3.4.m4.1c">H</annotation></semantics></math>.</p>
</div>
<div id="S4.SS1.SSS0.Px2.p4" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px2.p4.2" class="ltx_p"><math id="S4.SS1.SSS0.Px2.p4.1.m1.1" class="ltx_Math" alttext="\bullet" display="inline"><semantics id="S4.SS1.SSS0.Px2.p4.1.m1.1a"><mo id="S4.SS1.SSS0.Px2.p4.1.m1.1.1" xref="S4.SS1.SSS0.Px2.p4.1.m1.1.1.cmml">∙</mo><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p4.1.m1.1b"><ci id="S4.SS1.SSS0.Px2.p4.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p4.1.m1.1.1">∙</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p4.1.m1.1c">\bullet</annotation></semantics></math> <math id="S4.SS1.SSS0.Px2.p4.2.m2.1" class="ltx_Math" alttext="\text{MoM}_{\text{E}}" display="inline"><semantics id="S4.SS1.SSS0.Px2.p4.2.m2.1a"><msub id="S4.SS1.SSS0.Px2.p4.2.m2.1.1" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.cmml"><mtext class="ltx_mathvariant_bold" id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2a.cmml">MoM</mtext><mtext class="ltx_mathvariant_bold" id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.3" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.3a.cmml">E</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p4.2.m2.1b"><apply id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.cmml" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1">subscript</csymbol><ci id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2a.cmml" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2"><mtext class="ltx_mathvariant_bold" id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.2">MoM</mtext></ci><ci id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.3a.cmml" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.3"><mtext class="ltx_mathvariant_bold" mathsize="70%" id="S4.SS1.SSS0.Px2.p4.2.m2.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p4.2.m2.1.1.3">E</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p4.2.m2.1c">\text{MoM}_{\text{E}}</annotation></semantics></math> (K3H1S) significantly enhances <span id="S4.SS1.SSS0.Px2.p4.2.1" class="ltx_text ltx_font_bold">E</span>fficiency compared to vanilla Transformers, while maintaining acceptable performance.</p>
</div>
<div id="S4.SS1.SSS0.Px2.p5" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px2.p5.4" class="ltx_p"><math id="S4.SS1.SSS0.Px2.p5.1.m1.1" class="ltx_Math" alttext="\bullet" display="inline"><semantics id="S4.SS1.SSS0.Px2.p5.1.m1.1a"><mo id="S4.SS1.SSS0.Px2.p5.1.m1.1.1" xref="S4.SS1.SSS0.Px2.p5.1.m1.1.1.cmml">∙</mo><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p5.1.m1.1b"><ci id="S4.SS1.SSS0.Px2.p5.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p5.1.m1.1.1">∙</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p5.1.m1.1c">\bullet</annotation></semantics></math> <math id="S4.SS1.SSS0.Px2.p5.2.m2.1" class="ltx_Math" alttext="\text{MoM}_{\text{I}}" display="inline"><semantics id="S4.SS1.SSS0.Px2.p5.2.m2.1a"><msub id="S4.SS1.SSS0.Px2.p5.2.m2.1.1" xref="S4.SS1.SSS0.Px2.p5.2.m2.1.1.cmml"><mtext class="ltx_mathvariant_bold" id="S4.SS1.SSS0.Px2.p5.2.m2.1.1.2" xref="S4.SS1.SSS0.Px2.p5.2.m2.1.1.2a.cmml">MoM</mtext><mtext class="ltx_mathvariant_bold" id="S4.SS1.SSS0.Px2.p5.2.m2.1.1.3" xref="S4.SS1.SSS0.Px2.p5.2.m2.1.1.3a.cmml">I</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p5.2.m2.1b"><apply id="S4.SS1.SSS0.Px2.p5.2.m2.1.1.cmml" xref="S4.SS1.SSS0.Px2.p5.2.m2.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS0.Px2.p5.2.m2.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p5.2.m2.1.1">subscript</csymbol><ci id="S4.SS1.SSS0.Px2.p5.2.m2.1.1.2a.cmml" xref="S4.SS1.SSS0.Px2.p5.2.m2.1.1.2"><mtext class="ltx_mathvariant_bold" id="S4.SS1.SSS0.Px2.p5.2.m2.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p5.2.m2.1.1.2">MoM</mtext></ci><ci id="S4.SS1.SSS0.Px2.p5.2.m2.1.1.3a.cmml" xref="S4.SS1.SSS0.Px2.p5.2.m2.1.1.3"><mtext class="ltx_mathvariant_bold" mathsize="70%" id="S4.SS1.SSS0.Px2.p5.2.m2.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p5.2.m2.1.1.3">I</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p5.2.m2.1c">\text{MoM}_{\text{I}}</annotation></semantics></math> (K3H2S) serves as a midpoint in configurations between the two preceding models. This model is positioned between <math id="S4.SS1.SSS0.Px2.p5.3.m3.1" class="ltx_Math" alttext="\text{MoM}_{\text{E}}" display="inline"><semantics id="S4.SS1.SSS0.Px2.p5.3.m3.1a"><msub id="S4.SS1.SSS0.Px2.p5.3.m3.1.1" xref="S4.SS1.SSS0.Px2.p5.3.m3.1.1.cmml"><mtext id="S4.SS1.SSS0.Px2.p5.3.m3.1.1.2" xref="S4.SS1.SSS0.Px2.p5.3.m3.1.1.2a.cmml">MoM</mtext><mtext id="S4.SS1.SSS0.Px2.p5.3.m3.1.1.3" xref="S4.SS1.SSS0.Px2.p5.3.m3.1.1.3a.cmml">E</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p5.3.m3.1b"><apply id="S4.SS1.SSS0.Px2.p5.3.m3.1.1.cmml" xref="S4.SS1.SSS0.Px2.p5.3.m3.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS0.Px2.p5.3.m3.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p5.3.m3.1.1">subscript</csymbol><ci id="S4.SS1.SSS0.Px2.p5.3.m3.1.1.2a.cmml" xref="S4.SS1.SSS0.Px2.p5.3.m3.1.1.2"><mtext id="S4.SS1.SSS0.Px2.p5.3.m3.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p5.3.m3.1.1.2">MoM</mtext></ci><ci id="S4.SS1.SSS0.Px2.p5.3.m3.1.1.3a.cmml" xref="S4.SS1.SSS0.Px2.p5.3.m3.1.1.3"><mtext mathsize="70%" id="S4.SS1.SSS0.Px2.p5.3.m3.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p5.3.m3.1.1.3">E</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p5.3.m3.1c">\text{MoM}_{\text{E}}</annotation></semantics></math> and <math id="S4.SS1.SSS0.Px2.p5.4.m4.1" class="ltx_Math" alttext="\text{MoM}_{\text{P}}" display="inline"><semantics id="S4.SS1.SSS0.Px2.p5.4.m4.1a"><msub id="S4.SS1.SSS0.Px2.p5.4.m4.1.1" xref="S4.SS1.SSS0.Px2.p5.4.m4.1.1.cmml"><mtext id="S4.SS1.SSS0.Px2.p5.4.m4.1.1.2" xref="S4.SS1.SSS0.Px2.p5.4.m4.1.1.2a.cmml">MoM</mtext><mtext id="S4.SS1.SSS0.Px2.p5.4.m4.1.1.3" xref="S4.SS1.SSS0.Px2.p5.4.m4.1.1.3a.cmml">P</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p5.4.m4.1b"><apply id="S4.SS1.SSS0.Px2.p5.4.m4.1.1.cmml" xref="S4.SS1.SSS0.Px2.p5.4.m4.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS0.Px2.p5.4.m4.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p5.4.m4.1.1">subscript</csymbol><ci id="S4.SS1.SSS0.Px2.p5.4.m4.1.1.2a.cmml" xref="S4.SS1.SSS0.Px2.p5.4.m4.1.1.2"><mtext id="S4.SS1.SSS0.Px2.p5.4.m4.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p5.4.m4.1.1.2">MoM</mtext></ci><ci id="S4.SS1.SSS0.Px2.p5.4.m4.1.1.3a.cmml" xref="S4.SS1.SSS0.Px2.p5.4.m4.1.1.3"><mtext mathsize="70%" id="S4.SS1.SSS0.Px2.p5.4.m4.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p5.4.m4.1.1.3">P</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p5.4.m4.1c">\text{MoM}_{\text{P}}</annotation></semantics></math>. We aim to highlight performance and efficiency <span id="S4.SS1.SSS0.Px2.p5.4.1" class="ltx_text ltx_font_bold">I</span>nterpolation as feature of MoM through configuration interpolation.</p>
</div>
</section>
<section id="S4.SS1.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Evaluation settings.</h4>

<div id="S4.SS1.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.SSS0.Px3.p1.5" class="ltx_p">We employ GLUE benchmark <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib45" title="" class="ltx_ref">2018a</a>)</cite> to evaluate the language understanding ability and XSUM <cite class="ltx_cite ltx_citemacro_citep">(Narayan et al., <a href="#bib.bib31" title="" class="ltx_ref">2018b</a>)</cite> to evaluate the text generation ability. All models are fine-tuned with a learning rate of <math id="S4.SS1.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="2" display="inline"><semantics id="S4.SS1.SSS0.Px3.p1.1.m1.1a"><mn id="S4.SS1.SSS0.Px3.p1.1.m1.1.1" xref="S4.SS1.SSS0.Px3.p1.1.m1.1.1.cmml">2</mn><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px3.p1.1.m1.1b"><cn type="integer" id="S4.SS1.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px3.p1.1.m1.1.1">2</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px3.p1.1.m1.1c">2</annotation></semantics></math>e-<math id="S4.SS1.SSS0.Px3.p1.2.m2.1" class="ltx_Math" alttext="5" display="inline"><semantics id="S4.SS1.SSS0.Px3.p1.2.m2.1a"><mn id="S4.SS1.SSS0.Px3.p1.2.m2.1.1" xref="S4.SS1.SSS0.Px3.p1.2.m2.1.1.cmml">5</mn><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px3.p1.2.m2.1b"><cn type="integer" id="S4.SS1.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S4.SS1.SSS0.Px3.p1.2.m2.1.1">5</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px3.p1.2.m2.1c">5</annotation></semantics></math>. The sequence is <math id="S4.SS1.SSS0.Px3.p1.3.m3.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S4.SS1.SSS0.Px3.p1.3.m3.1a"><mn id="S4.SS1.SSS0.Px3.p1.3.m3.1.1" xref="S4.SS1.SSS0.Px3.p1.3.m3.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px3.p1.3.m3.1b"><cn type="integer" id="S4.SS1.SSS0.Px3.p1.3.m3.1.1.cmml" xref="S4.SS1.SSS0.Px3.p1.3.m3.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px3.p1.3.m3.1c">128</annotation></semantics></math> for GLUE and <math id="S4.SS1.SSS0.Px3.p1.4.m4.1" class="ltx_Math" alttext="1024" display="inline"><semantics id="S4.SS1.SSS0.Px3.p1.4.m4.1a"><mn id="S4.SS1.SSS0.Px3.p1.4.m4.1.1" xref="S4.SS1.SSS0.Px3.p1.4.m4.1.1.cmml">1024</mn><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px3.p1.4.m4.1b"><cn type="integer" id="S4.SS1.SSS0.Px3.p1.4.m4.1.1.cmml" xref="S4.SS1.SSS0.Px3.p1.4.m4.1.1">1024</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px3.p1.4.m4.1c">1024</annotation></semantics></math> for XSUM. For smaller GLUE sub-datasets (CoLA, STS-B, MRPC, and RTE), we set the batch size to 32 and train for 3 epochs. For larger datasets (MNLI, QNLI, QQP, and SST-2), we utilize a batch size of 64 and perform training for a total of <math id="S4.SS1.SSS0.Px3.p1.5.m5.2" class="ltx_Math" alttext="8,000" display="inline"><semantics id="S4.SS1.SSS0.Px3.p1.5.m5.2a"><mrow id="S4.SS1.SSS0.Px3.p1.5.m5.2.3.2" xref="S4.SS1.SSS0.Px3.p1.5.m5.2.3.1.cmml"><mn id="S4.SS1.SSS0.Px3.p1.5.m5.1.1" xref="S4.SS1.SSS0.Px3.p1.5.m5.1.1.cmml">8</mn><mo id="S4.SS1.SSS0.Px3.p1.5.m5.2.3.2.1" xref="S4.SS1.SSS0.Px3.p1.5.m5.2.3.1.cmml">,</mo><mn id="S4.SS1.SSS0.Px3.p1.5.m5.2.2" xref="S4.SS1.SSS0.Px3.p1.5.m5.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px3.p1.5.m5.2b"><list id="S4.SS1.SSS0.Px3.p1.5.m5.2.3.1.cmml" xref="S4.SS1.SSS0.Px3.p1.5.m5.2.3.2"><cn type="integer" id="S4.SS1.SSS0.Px3.p1.5.m5.1.1.cmml" xref="S4.SS1.SSS0.Px3.p1.5.m5.1.1">8</cn><cn type="integer" id="S4.SS1.SSS0.Px3.p1.5.m5.2.2.cmml" xref="S4.SS1.SSS0.Px3.p1.5.m5.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px3.p1.5.m5.2c">8,000</annotation></semantics></math> gradient steps. For XSUM, we set the batch size to 64 and train for 3 epochs. For efficiency evaluation, we report inference TFLOPs and memory usage.
TFLOPs are calculated using DeepSpeed FLOPs profiler <cite class="ltx_cite ltx_citemacro_citep">(DeepSpeed, <a href="#bib.bib11" title="" class="ltx_ref">2023</a>)</cite> and memory consumption is calculated with PyTorch toolkits <cite class="ltx_cite ltx_citemacro_citep">(pytorch, <a href="#bib.bib34" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
</section>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Main results</h3>

<div id="S4.SS2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS2.p1.1" class="ltx_p">Table <a href="#S4.T1" title="Table 1 ‣ Implementation details. ‣ 4.1 Experimental setup ‣ 4 Experiments ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> reports the evaluation results.
Our analysis yields the following conclusions:</p>
</div>
<section id="S4.SS2.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">MoM unleashes the potential of Transformers and our initial motivation is confirmed.</h4>

<div id="S4.SS2.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS2.SSS0.Px1.p1.4" class="ltx_p">When maintaining the number of parameters, <math id="S4.SS2.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{P}}" display="inline"><semantics id="S4.SS2.SSS0.Px1.p1.1.m1.1a"><msub id="S4.SS2.SSS0.Px1.p1.1.m1.1.1" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.1.cmml"><mtext id="S4.SS2.SSS0.Px1.p1.1.m1.1.1.2" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="S4.SS2.SSS0.Px1.p1.1.m1.1.1.3" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.1.3a.cmml">P</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p1.1.m1.1b"><apply id="S4.SS2.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px1.p1.1.m1.1.1.2a.cmml" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.1.2"><mtext id="S4.SS2.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.1.2">MoM</mtext></ci><ci id="S4.SS2.SSS0.Px1.p1.1.m1.1.1.3a.cmml" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.1.3"><mtext mathsize="70%" id="S4.SS2.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.1.3">P</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p1.1.m1.1c">\text{MoM}_{\text{P}}</annotation></semantics></math> is characterized by the deepest computation graph (<math id="S4.SS2.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S4.SS2.SSS0.Px1.p1.2.m2.1a"><mi id="S4.SS2.SSS0.Px1.p1.2.m2.1.1" xref="S4.SS2.SSS0.Px1.p1.2.m2.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p1.2.m2.1b"><ci id="S4.SS2.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S4.SS2.SSS0.Px1.p1.2.m2.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p1.2.m2.1c">H</annotation></semantics></math>).
Across all model scales, <math id="S4.SS2.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="\text{MoM}_{\text{P}}" display="inline"><semantics id="S4.SS2.SSS0.Px1.p1.3.m3.1a"><msub id="S4.SS2.SSS0.Px1.p1.3.m3.1.1" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.1.cmml"><mtext id="S4.SS2.SSS0.Px1.p1.3.m3.1.1.2" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.1.2a.cmml">MoM</mtext><mtext id="S4.SS2.SSS0.Px1.p1.3.m3.1.1.3" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.1.3a.cmml">P</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p1.3.m3.1b"><apply id="S4.SS2.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px1.p1.3.m3.1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px1.p1.3.m3.1.1.2a.cmml" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.1.2"><mtext id="S4.SS2.SSS0.Px1.p1.3.m3.1.1.2.cmml" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.1.2">MoM</mtext></ci><ci id="S4.SS2.SSS0.Px1.p1.3.m3.1.1.3a.cmml" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.1.3"><mtext mathsize="70%" id="S4.SS2.SSS0.Px1.p1.3.m3.1.1.3.cmml" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.1.3">P</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p1.3.m3.1c">\text{MoM}_{\text{P}}</annotation></semantics></math> consistently outperforms all baselines on both GLUE and XSUM by significant margins.
The enhanced performance of <math id="S4.SS2.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="\text{MoM}_{\text{P}}" display="inline"><semantics id="S4.SS2.SSS0.Px1.p1.4.m4.1a"><msub id="S4.SS2.SSS0.Px1.p1.4.m4.1.1" xref="S4.SS2.SSS0.Px1.p1.4.m4.1.1.cmml"><mtext id="S4.SS2.SSS0.Px1.p1.4.m4.1.1.2" xref="S4.SS2.SSS0.Px1.p1.4.m4.1.1.2a.cmml">MoM</mtext><mtext id="S4.SS2.SSS0.Px1.p1.4.m4.1.1.3" xref="S4.SS2.SSS0.Px1.p1.4.m4.1.1.3a.cmml">P</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p1.4.m4.1b"><apply id="S4.SS2.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S4.SS2.SSS0.Px1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px1.p1.4.m4.1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p1.4.m4.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px1.p1.4.m4.1.1.2a.cmml" xref="S4.SS2.SSS0.Px1.p1.4.m4.1.1.2"><mtext id="S4.SS2.SSS0.Px1.p1.4.m4.1.1.2.cmml" xref="S4.SS2.SSS0.Px1.p1.4.m4.1.1.2">MoM</mtext></ci><ci id="S4.SS2.SSS0.Px1.p1.4.m4.1.1.3a.cmml" xref="S4.SS2.SSS0.Px1.p1.4.m4.1.1.3"><mtext mathsize="70%" id="S4.SS2.SSS0.Px1.p1.4.m4.1.1.3.cmml" xref="S4.SS2.SSS0.Px1.p1.4.m4.1.1.3">P</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p1.4.m4.1c">\text{MoM}_{\text{P}}</annotation></semantics></math> validates our initial motivations: (1) the traditional depth-ordered layer organization is sub-optimal;
(2) improvements can be realized through two key modifications to the computation graph, including <span id="S4.SS2.SSS0.Px1.p1.4.1" class="ltx_text ltx_font_italic">dynamic module organization</span> and <span id="S4.SS2.SSS0.Px1.p1.4.2" class="ltx_text ltx_font_italic">improved parameter utilization</span>.</p>
</div>
<div id="S4.SS2.SSS0.Px1.p2" class="ltx_para ltx_noindent">
<p id="S4.SS2.SSS0.Px1.p2.4" class="ltx_p"><math id="S4.SS2.SSS0.Px1.p2.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{E}}" display="inline"><semantics id="S4.SS2.SSS0.Px1.p2.1.m1.1a"><msub id="S4.SS2.SSS0.Px1.p2.1.m1.1.1" xref="S4.SS2.SSS0.Px1.p2.1.m1.1.1.cmml"><mtext id="S4.SS2.SSS0.Px1.p2.1.m1.1.1.2" xref="S4.SS2.SSS0.Px1.p2.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="S4.SS2.SSS0.Px1.p2.1.m1.1.1.3" xref="S4.SS2.SSS0.Px1.p2.1.m1.1.1.3a.cmml">E</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p2.1.m1.1b"><apply id="S4.SS2.SSS0.Px1.p2.1.m1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px1.p2.1.m1.1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.1.m1.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px1.p2.1.m1.1.1.2a.cmml" xref="S4.SS2.SSS0.Px1.p2.1.m1.1.1.2"><mtext id="S4.SS2.SSS0.Px1.p2.1.m1.1.1.2.cmml" xref="S4.SS2.SSS0.Px1.p2.1.m1.1.1.2">MoM</mtext></ci><ci id="S4.SS2.SSS0.Px1.p2.1.m1.1.1.3a.cmml" xref="S4.SS2.SSS0.Px1.p2.1.m1.1.1.3"><mtext mathsize="70%" id="S4.SS2.SSS0.Px1.p2.1.m1.1.1.3.cmml" xref="S4.SS2.SSS0.Px1.p2.1.m1.1.1.3">E</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p2.1.m1.1c">\text{MoM}_{\text{E}}</annotation></semantics></math> is characterized by its minimum depth (<math id="S4.SS2.SSS0.Px1.p2.2.m2.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S4.SS2.SSS0.Px1.p2.2.m2.1a"><mi id="S4.SS2.SSS0.Px1.p2.2.m2.1.1" xref="S4.SS2.SSS0.Px1.p2.2.m2.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p2.2.m2.1b"><ci id="S4.SS2.SSS0.Px1.p2.2.m2.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.2.m2.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p2.2.m2.1c">H</annotation></semantics></math>).
By strategically selecting appropriate modules at each assembly step, <math id="S4.SS2.SSS0.Px1.p2.3.m3.1" class="ltx_Math" alttext="\text{MoM}_{\text{E}}" display="inline"><semantics id="S4.SS2.SSS0.Px1.p2.3.m3.1a"><msub id="S4.SS2.SSS0.Px1.p2.3.m3.1.1" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.1.cmml"><mtext id="S4.SS2.SSS0.Px1.p2.3.m3.1.1.2" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.1.2a.cmml">MoM</mtext><mtext id="S4.SS2.SSS0.Px1.p2.3.m3.1.1.3" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.1.3a.cmml">E</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p2.3.m3.1b"><apply id="S4.SS2.SSS0.Px1.p2.3.m3.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px1.p2.3.m3.1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px1.p2.3.m3.1.1.2a.cmml" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.1.2"><mtext id="S4.SS2.SSS0.Px1.p2.3.m3.1.1.2.cmml" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.1.2">MoM</mtext></ci><ci id="S4.SS2.SSS0.Px1.p2.3.m3.1.1.3a.cmml" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.1.3"><mtext mathsize="70%" id="S4.SS2.SSS0.Px1.p2.3.m3.1.1.3.cmml" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.1.3">E</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p2.3.m3.1c">\text{MoM}_{\text{E}}</annotation></semantics></math> strives to reduce memory and computation costs while maintaining performance.
Although <math id="S4.SS2.SSS0.Px1.p2.4.m4.1" class="ltx_Math" alttext="\text{MoM}_{\text{E}}" display="inline"><semantics id="S4.SS2.SSS0.Px1.p2.4.m4.1a"><msub id="S4.SS2.SSS0.Px1.p2.4.m4.1.1" xref="S4.SS2.SSS0.Px1.p2.4.m4.1.1.cmml"><mtext id="S4.SS2.SSS0.Px1.p2.4.m4.1.1.2" xref="S4.SS2.SSS0.Px1.p2.4.m4.1.1.2a.cmml">MoM</mtext><mtext id="S4.SS2.SSS0.Px1.p2.4.m4.1.1.3" xref="S4.SS2.SSS0.Px1.p2.4.m4.1.1.3a.cmml">E</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p2.4.m4.1b"><apply id="S4.SS2.SSS0.Px1.p2.4.m4.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px1.p2.4.m4.1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.4.m4.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px1.p2.4.m4.1.1.2a.cmml" xref="S4.SS2.SSS0.Px1.p2.4.m4.1.1.2"><mtext id="S4.SS2.SSS0.Px1.p2.4.m4.1.1.2.cmml" xref="S4.SS2.SSS0.Px1.p2.4.m4.1.1.2">MoM</mtext></ci><ci id="S4.SS2.SSS0.Px1.p2.4.m4.1.1.3a.cmml" xref="S4.SS2.SSS0.Px1.p2.4.m4.1.1.3"><mtext mathsize="70%" id="S4.SS2.SSS0.Px1.p2.4.m4.1.1.3.cmml" xref="S4.SS2.SSS0.Px1.p2.4.m4.1.1.3">E</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p2.4.m4.1c">\text{MoM}_{\text{E}}</annotation></semantics></math> is slightly surpassed by a vanilla Transformer, it outperforms MoD, another efficiency-driven method by large margins.</p>
</div>
<div id="S4.SS2.SSS0.Px1.p3" class="ltx_para ltx_noindent">
<p id="S4.SS2.SSS0.Px1.p3.3" class="ltx_p">Besides, we observe that <math id="S4.SS2.SSS0.Px1.p3.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{I}}" display="inline"><semantics id="S4.SS2.SSS0.Px1.p3.1.m1.1a"><msub id="S4.SS2.SSS0.Px1.p3.1.m1.1.1" xref="S4.SS2.SSS0.Px1.p3.1.m1.1.1.cmml"><mtext id="S4.SS2.SSS0.Px1.p3.1.m1.1.1.2" xref="S4.SS2.SSS0.Px1.p3.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="S4.SS2.SSS0.Px1.p3.1.m1.1.1.3" xref="S4.SS2.SSS0.Px1.p3.1.m1.1.1.3a.cmml">I</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p3.1.m1.1b"><apply id="S4.SS2.SSS0.Px1.p3.1.m1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px1.p3.1.m1.1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p3.1.m1.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px1.p3.1.m1.1.1.2a.cmml" xref="S4.SS2.SSS0.Px1.p3.1.m1.1.1.2"><mtext id="S4.SS2.SSS0.Px1.p3.1.m1.1.1.2.cmml" xref="S4.SS2.SSS0.Px1.p3.1.m1.1.1.2">MoM</mtext></ci><ci id="S4.SS2.SSS0.Px1.p3.1.m1.1.1.3a.cmml" xref="S4.SS2.SSS0.Px1.p3.1.m1.1.1.3"><mtext mathsize="70%" id="S4.SS2.SSS0.Px1.p3.1.m1.1.1.3.cmml" xref="S4.SS2.SSS0.Px1.p3.1.m1.1.1.3">I</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p3.1.m1.1c">\text{MoM}_{\text{I}}</annotation></semantics></math> archives a decent performance by slightly outperforming vanilla GPT-2.
Comparing to vanilla GPT-2, <math id="S4.SS2.SSS0.Px1.p3.2.m2.1" class="ltx_Math" alttext="\text{MoM}_{\text{I}}" display="inline"><semantics id="S4.SS2.SSS0.Px1.p3.2.m2.1a"><msub id="S4.SS2.SSS0.Px1.p3.2.m2.1.1" xref="S4.SS2.SSS0.Px1.p3.2.m2.1.1.cmml"><mtext id="S4.SS2.SSS0.Px1.p3.2.m2.1.1.2" xref="S4.SS2.SSS0.Px1.p3.2.m2.1.1.2a.cmml">MoM</mtext><mtext id="S4.SS2.SSS0.Px1.p3.2.m2.1.1.3" xref="S4.SS2.SSS0.Px1.p3.2.m2.1.1.3a.cmml">I</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p3.2.m2.1b"><apply id="S4.SS2.SSS0.Px1.p3.2.m2.1.1.cmml" xref="S4.SS2.SSS0.Px1.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px1.p3.2.m2.1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p3.2.m2.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px1.p3.2.m2.1.1.2a.cmml" xref="S4.SS2.SSS0.Px1.p3.2.m2.1.1.2"><mtext id="S4.SS2.SSS0.Px1.p3.2.m2.1.1.2.cmml" xref="S4.SS2.SSS0.Px1.p3.2.m2.1.1.2">MoM</mtext></ci><ci id="S4.SS2.SSS0.Px1.p3.2.m2.1.1.3a.cmml" xref="S4.SS2.SSS0.Px1.p3.2.m2.1.1.3"><mtext mathsize="70%" id="S4.SS2.SSS0.Px1.p3.2.m2.1.1.3.cmml" xref="S4.SS2.SSS0.Px1.p3.2.m2.1.1.3">I</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p3.2.m2.1c">\text{MoM}_{\text{I}}</annotation></semantics></math> consumes no more than 25% extra computation but save at least 11.7% memory across all scales, indicating that <math id="S4.SS2.SSS0.Px1.p3.3.m3.1" class="ltx_Math" alttext="\text{MoM}_{\text{I}}" display="inline"><semantics id="S4.SS2.SSS0.Px1.p3.3.m3.1a"><msub id="S4.SS2.SSS0.Px1.p3.3.m3.1.1" xref="S4.SS2.SSS0.Px1.p3.3.m3.1.1.cmml"><mtext id="S4.SS2.SSS0.Px1.p3.3.m3.1.1.2" xref="S4.SS2.SSS0.Px1.p3.3.m3.1.1.2a.cmml">MoM</mtext><mtext id="S4.SS2.SSS0.Px1.p3.3.m3.1.1.3" xref="S4.SS2.SSS0.Px1.p3.3.m3.1.1.3a.cmml">I</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p3.3.m3.1b"><apply id="S4.SS2.SSS0.Px1.p3.3.m3.1.1.cmml" xref="S4.SS2.SSS0.Px1.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px1.p3.3.m3.1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p3.3.m3.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px1.p3.3.m3.1.1.2a.cmml" xref="S4.SS2.SSS0.Px1.p3.3.m3.1.1.2"><mtext id="S4.SS2.SSS0.Px1.p3.3.m3.1.1.2.cmml" xref="S4.SS2.SSS0.Px1.p3.3.m3.1.1.2">MoM</mtext></ci><ci id="S4.SS2.SSS0.Px1.p3.3.m3.1.1.3a.cmml" xref="S4.SS2.SSS0.Px1.p3.3.m3.1.1.3"><mtext mathsize="70%" id="S4.SS2.SSS0.Px1.p3.3.m3.1.1.3.cmml" xref="S4.SS2.SSS0.Px1.p3.3.m3.1.1.3">I</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p3.3.m3.1c">\text{MoM}_{\text{I}}</annotation></semantics></math> achieves a good balance between performance and efficiency.</p>
</div>
<figure id="S4.F3" class="ltx_figure"><img src="/html/2407.06677/assets/x3.png" id="S4.F3.g1" class="ltx_graphics ltx_centering ltx_img_square" width="138" height="138" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>How validation loss varies with respect to <math id="S4.F3.4.m1.1" class="ltx_Math" alttext="N_{\text{A}}" display="inline"><semantics id="S4.F3.4.m1.1b"><msub id="S4.F3.4.m1.1.1" xref="S4.F3.4.m1.1.1.cmml"><mi id="S4.F3.4.m1.1.1.2" xref="S4.F3.4.m1.1.1.2.cmml">N</mi><mtext id="S4.F3.4.m1.1.1.3" xref="S4.F3.4.m1.1.1.3a.cmml">A</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.F3.4.m1.1c"><apply id="S4.F3.4.m1.1.1.cmml" xref="S4.F3.4.m1.1.1"><csymbol cd="ambiguous" id="S4.F3.4.m1.1.1.1.cmml" xref="S4.F3.4.m1.1.1">subscript</csymbol><ci id="S4.F3.4.m1.1.1.2.cmml" xref="S4.F3.4.m1.1.1.2">𝑁</ci><ci id="S4.F3.4.m1.1.1.3a.cmml" xref="S4.F3.4.m1.1.1.3"><mtext mathsize="70%" id="S4.F3.4.m1.1.1.3.cmml" xref="S4.F3.4.m1.1.1.3">A</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F3.4.m1.1d">N_{\text{A}}</annotation></semantics></math> and <math id="S4.F3.5.m2.1" class="ltx_Math" alttext="N_{\text{F}}" display="inline"><semantics id="S4.F3.5.m2.1b"><msub id="S4.F3.5.m2.1.1" xref="S4.F3.5.m2.1.1.cmml"><mi id="S4.F3.5.m2.1.1.2" xref="S4.F3.5.m2.1.1.2.cmml">N</mi><mtext id="S4.F3.5.m2.1.1.3" xref="S4.F3.5.m2.1.1.3a.cmml">F</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.F3.5.m2.1c"><apply id="S4.F3.5.m2.1.1.cmml" xref="S4.F3.5.m2.1.1"><csymbol cd="ambiguous" id="S4.F3.5.m2.1.1.1.cmml" xref="S4.F3.5.m2.1.1">subscript</csymbol><ci id="S4.F3.5.m2.1.1.2.cmml" xref="S4.F3.5.m2.1.1.2">𝑁</ci><ci id="S4.F3.5.m2.1.1.3a.cmml" xref="S4.F3.5.m2.1.1.3"><mtext mathsize="70%" id="S4.F3.5.m2.1.1.3.cmml" xref="S4.F3.5.m2.1.1.3">F</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F3.5.m2.1d">N_{\text{F}}</annotation></semantics></math>, comparing to MoM (medium) with <math id="S4.F3.6.m3.1" class="ltx_Math" alttext="N_{\text{A}}=N_{\text{F}}=4" display="inline"><semantics id="S4.F3.6.m3.1b"><mrow id="S4.F3.6.m3.1.1" xref="S4.F3.6.m3.1.1.cmml"><msub id="S4.F3.6.m3.1.1.2" xref="S4.F3.6.m3.1.1.2.cmml"><mi id="S4.F3.6.m3.1.1.2.2" xref="S4.F3.6.m3.1.1.2.2.cmml">N</mi><mtext id="S4.F3.6.m3.1.1.2.3" xref="S4.F3.6.m3.1.1.2.3a.cmml">A</mtext></msub><mo id="S4.F3.6.m3.1.1.3" xref="S4.F3.6.m3.1.1.3.cmml">=</mo><msub id="S4.F3.6.m3.1.1.4" xref="S4.F3.6.m3.1.1.4.cmml"><mi id="S4.F3.6.m3.1.1.4.2" xref="S4.F3.6.m3.1.1.4.2.cmml">N</mi><mtext id="S4.F3.6.m3.1.1.4.3" xref="S4.F3.6.m3.1.1.4.3a.cmml">F</mtext></msub><mo id="S4.F3.6.m3.1.1.5" xref="S4.F3.6.m3.1.1.5.cmml">=</mo><mn id="S4.F3.6.m3.1.1.6" xref="S4.F3.6.m3.1.1.6.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.F3.6.m3.1c"><apply id="S4.F3.6.m3.1.1.cmml" xref="S4.F3.6.m3.1.1"><and id="S4.F3.6.m3.1.1a.cmml" xref="S4.F3.6.m3.1.1"></and><apply id="S4.F3.6.m3.1.1b.cmml" xref="S4.F3.6.m3.1.1"><eq id="S4.F3.6.m3.1.1.3.cmml" xref="S4.F3.6.m3.1.1.3"></eq><apply id="S4.F3.6.m3.1.1.2.cmml" xref="S4.F3.6.m3.1.1.2"><csymbol cd="ambiguous" id="S4.F3.6.m3.1.1.2.1.cmml" xref="S4.F3.6.m3.1.1.2">subscript</csymbol><ci id="S4.F3.6.m3.1.1.2.2.cmml" xref="S4.F3.6.m3.1.1.2.2">𝑁</ci><ci id="S4.F3.6.m3.1.1.2.3a.cmml" xref="S4.F3.6.m3.1.1.2.3"><mtext mathsize="70%" id="S4.F3.6.m3.1.1.2.3.cmml" xref="S4.F3.6.m3.1.1.2.3">A</mtext></ci></apply><apply id="S4.F3.6.m3.1.1.4.cmml" xref="S4.F3.6.m3.1.1.4"><csymbol cd="ambiguous" id="S4.F3.6.m3.1.1.4.1.cmml" xref="S4.F3.6.m3.1.1.4">subscript</csymbol><ci id="S4.F3.6.m3.1.1.4.2.cmml" xref="S4.F3.6.m3.1.1.4.2">𝑁</ci><ci id="S4.F3.6.m3.1.1.4.3a.cmml" xref="S4.F3.6.m3.1.1.4.3"><mtext mathsize="70%" id="S4.F3.6.m3.1.1.4.3.cmml" xref="S4.F3.6.m3.1.1.4.3">F</mtext></ci></apply></apply><apply id="S4.F3.6.m3.1.1c.cmml" xref="S4.F3.6.m3.1.1"><eq id="S4.F3.6.m3.1.1.5.cmml" xref="S4.F3.6.m3.1.1.5"></eq><share href="#S4.F3.6.m3.1.1.4.cmml" id="S4.F3.6.m3.1.1d.cmml" xref="S4.F3.6.m3.1.1"></share><cn type="integer" id="S4.F3.6.m3.1.1.6.cmml" xref="S4.F3.6.m3.1.1.6">4</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F3.6.m3.1d">N_{\text{A}}=N_{\text{F}}=4</annotation></semantics></math>.</figcaption>
</figure>
</section>
<section id="S4.SS2.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">MoM models provide insights into the over-parameterization issue.</h4>

<div id="S4.SS2.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS2.SSS0.Px2.p1.10" class="ltx_p">With GPT-2-medium as initialization,
we develop a series of MoM models, each defined by different pairs of (<math id="S4.SS2.SSS0.Px2.p1.1.m1.2" class="ltx_Math" alttext="N_{\text{A}},N_{\text{F}}" display="inline"><semantics id="S4.SS2.SSS0.Px2.p1.1.m1.2a"><mrow id="S4.SS2.SSS0.Px2.p1.1.m1.2.2.2" xref="S4.SS2.SSS0.Px2.p1.1.m1.2.2.3.cmml"><msub id="S4.SS2.SSS0.Px2.p1.1.m1.1.1.1.1" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1.1.1.cmml"><mi id="S4.SS2.SSS0.Px2.p1.1.m1.1.1.1.1.2" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1.1.1.2.cmml">N</mi><mtext id="S4.SS2.SSS0.Px2.p1.1.m1.1.1.1.1.3" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1.1.1.3a.cmml">A</mtext></msub><mo id="S4.SS2.SSS0.Px2.p1.1.m1.2.2.2.3" xref="S4.SS2.SSS0.Px2.p1.1.m1.2.2.3.cmml">,</mo><msub id="S4.SS2.SSS0.Px2.p1.1.m1.2.2.2.2" xref="S4.SS2.SSS0.Px2.p1.1.m1.2.2.2.2.cmml"><mi id="S4.SS2.SSS0.Px2.p1.1.m1.2.2.2.2.2" xref="S4.SS2.SSS0.Px2.p1.1.m1.2.2.2.2.2.cmml">N</mi><mtext id="S4.SS2.SSS0.Px2.p1.1.m1.2.2.2.2.3" xref="S4.SS2.SSS0.Px2.p1.1.m1.2.2.2.2.3a.cmml">F</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p1.1.m1.2b"><list id="S4.SS2.SSS0.Px2.p1.1.m1.2.2.3.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.2.2.2"><apply id="S4.SS2.SSS0.Px2.p1.1.m1.1.1.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px2.p1.1.m1.1.1.1.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px2.p1.1.m1.1.1.1.1.2.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1.1.1.2">𝑁</ci><ci id="S4.SS2.SSS0.Px2.p1.1.m1.1.1.1.1.3a.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1.1.1.3"><mtext mathsize="70%" id="S4.SS2.SSS0.Px2.p1.1.m1.1.1.1.1.3.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1.1.1.3">A</mtext></ci></apply><apply id="S4.SS2.SSS0.Px2.p1.1.m1.2.2.2.2.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.2.2.2.2"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px2.p1.1.m1.2.2.2.2.1.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.2.2.2.2">subscript</csymbol><ci id="S4.SS2.SSS0.Px2.p1.1.m1.2.2.2.2.2.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.2.2.2.2.2">𝑁</ci><ci id="S4.SS2.SSS0.Px2.p1.1.m1.2.2.2.2.3a.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.2.2.2.2.3"><mtext mathsize="70%" id="S4.SS2.SSS0.Px2.p1.1.m1.2.2.2.2.3.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.2.2.2.2.3">F</mtext></ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p1.1.m1.2c">N_{\text{A}},N_{\text{F}}</annotation></semantics></math>), with both values not exceeding <math id="S4.SS2.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="4" display="inline"><semantics id="S4.SS2.SSS0.Px2.p1.2.m2.1a"><mn id="S4.SS2.SSS0.Px2.p1.2.m2.1.1" xref="S4.SS2.SSS0.Px2.p1.2.m2.1.1.cmml">4</mn><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p1.2.m2.1b"><cn type="integer" id="S4.SS2.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.2.m2.1.1">4</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p1.2.m2.1c">4</annotation></semantics></math>.
We assess the validation loss increase for each model relative to the benchmark model where <math id="S4.SS2.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="N_{\text{A}}=4" display="inline"><semantics id="S4.SS2.SSS0.Px2.p1.3.m3.1a"><mrow id="S4.SS2.SSS0.Px2.p1.3.m3.1.1" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.cmml"><msub id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.cmml"><mi id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.2" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.2.cmml">N</mi><mtext id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.3" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.3a.cmml">A</mtext></msub><mo id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.1" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.1.cmml">=</mo><mn id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p1.3.m3.1b"><apply id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1"><eq id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.1"></eq><apply id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.1.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2">subscript</csymbol><ci id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.2.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.2">𝑁</ci><ci id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.3a.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.3"><mtext mathsize="70%" id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.3.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.3">A</mtext></ci></apply><cn type="integer" id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p1.3.m3.1c">N_{\text{A}}=4</annotation></semantics></math> and <math id="S4.SS2.SSS0.Px2.p1.4.m4.1" class="ltx_Math" alttext="N_{\text{F}}=4" display="inline"><semantics id="S4.SS2.SSS0.Px2.p1.4.m4.1a"><mrow id="S4.SS2.SSS0.Px2.p1.4.m4.1.1" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.cmml"><msub id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.2" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.2.cmml"><mi id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.2.2" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.2.2.cmml">N</mi><mtext id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.2.3" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.2.3a.cmml">F</mtext></msub><mo id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.1" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.1.cmml">=</mo><mn id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p1.4.m4.1b"><apply id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1"><eq id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.1"></eq><apply id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.2.cmml" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.2"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.2.1.cmml" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.2">subscript</csymbol><ci id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.2.2.cmml" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.2.2">𝑁</ci><ci id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.2.3a.cmml" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.2.3"><mtext mathsize="70%" id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.2.3.cmml" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.2.3">F</mtext></ci></apply><cn type="integer" id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.cmml" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p1.4.m4.1c">N_{\text{F}}=4</annotation></semantics></math>, as illustrated in Figure <a href="#S4.F3" title="Figure 3 ‣ MoM unleashes the potential of Transformers and our initial motivation is confirmed. ‣ 4.2 Main results ‣ 4 Experiments ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>.
In this experiment, we set <math id="S4.SS2.SSS0.Px2.p1.5.m5.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S4.SS2.SSS0.Px2.p1.5.m5.1a"><mi id="S4.SS2.SSS0.Px2.p1.5.m5.1.1" xref="S4.SS2.SSS0.Px2.p1.5.m5.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p1.5.m5.1b"><ci id="S4.SS2.SSS0.Px2.p1.5.m5.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.5.m5.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p1.5.m5.1c">K</annotation></semantics></math> equal to the number of modules to make full use of the module parameters.
Interestingly, the <span id="S4.SS2.SSS0.Px2.p1.10.1" class="ltx_text ltx_font_typewriter">FFN</span> and <span id="S4.SS2.SSS0.Px2.p1.10.2" class="ltx_text ltx_font_typewriter">MHA</span> modules exhibit different degrees of redundancy.
Specifically, when <math id="S4.SS2.SSS0.Px2.p1.6.m6.1" class="ltx_Math" alttext="N_{\text{F}}" display="inline"><semantics id="S4.SS2.SSS0.Px2.p1.6.m6.1a"><msub id="S4.SS2.SSS0.Px2.p1.6.m6.1.1" xref="S4.SS2.SSS0.Px2.p1.6.m6.1.1.cmml"><mi id="S4.SS2.SSS0.Px2.p1.6.m6.1.1.2" xref="S4.SS2.SSS0.Px2.p1.6.m6.1.1.2.cmml">N</mi><mtext id="S4.SS2.SSS0.Px2.p1.6.m6.1.1.3" xref="S4.SS2.SSS0.Px2.p1.6.m6.1.1.3a.cmml">F</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p1.6.m6.1b"><apply id="S4.SS2.SSS0.Px2.p1.6.m6.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px2.p1.6.m6.1.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.6.m6.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px2.p1.6.m6.1.1.2.cmml" xref="S4.SS2.SSS0.Px2.p1.6.m6.1.1.2">𝑁</ci><ci id="S4.SS2.SSS0.Px2.p1.6.m6.1.1.3a.cmml" xref="S4.SS2.SSS0.Px2.p1.6.m6.1.1.3"><mtext mathsize="70%" id="S4.SS2.SSS0.Px2.p1.6.m6.1.1.3.cmml" xref="S4.SS2.SSS0.Px2.p1.6.m6.1.1.3">F</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p1.6.m6.1c">N_{\text{F}}</annotation></semantics></math> is fixed and the number of <span id="S4.SS2.SSS0.Px2.p1.10.3" class="ltx_text ltx_font_typewriter">MHA</span>s is gradually reduced, a significant increase in loss is not observed until <math id="S4.SS2.SSS0.Px2.p1.7.m7.1" class="ltx_Math" alttext="N_{\text{A}}" display="inline"><semantics id="S4.SS2.SSS0.Px2.p1.7.m7.1a"><msub id="S4.SS2.SSS0.Px2.p1.7.m7.1.1" xref="S4.SS2.SSS0.Px2.p1.7.m7.1.1.cmml"><mi id="S4.SS2.SSS0.Px2.p1.7.m7.1.1.2" xref="S4.SS2.SSS0.Px2.p1.7.m7.1.1.2.cmml">N</mi><mtext id="S4.SS2.SSS0.Px2.p1.7.m7.1.1.3" xref="S4.SS2.SSS0.Px2.p1.7.m7.1.1.3a.cmml">A</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p1.7.m7.1b"><apply id="S4.SS2.SSS0.Px2.p1.7.m7.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.7.m7.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px2.p1.7.m7.1.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.7.m7.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px2.p1.7.m7.1.1.2.cmml" xref="S4.SS2.SSS0.Px2.p1.7.m7.1.1.2">𝑁</ci><ci id="S4.SS2.SSS0.Px2.p1.7.m7.1.1.3a.cmml" xref="S4.SS2.SSS0.Px2.p1.7.m7.1.1.3"><mtext mathsize="70%" id="S4.SS2.SSS0.Px2.p1.7.m7.1.1.3.cmml" xref="S4.SS2.SSS0.Px2.p1.7.m7.1.1.3">A</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p1.7.m7.1c">N_{\text{A}}</annotation></semantics></math> is reduced from <math id="S4.SS2.SSS0.Px2.p1.8.m8.1" class="ltx_Math" alttext="2" display="inline"><semantics id="S4.SS2.SSS0.Px2.p1.8.m8.1a"><mn id="S4.SS2.SSS0.Px2.p1.8.m8.1.1" xref="S4.SS2.SSS0.Px2.p1.8.m8.1.1.cmml">2</mn><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p1.8.m8.1b"><cn type="integer" id="S4.SS2.SSS0.Px2.p1.8.m8.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.8.m8.1.1">2</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p1.8.m8.1c">2</annotation></semantics></math> to <math id="S4.SS2.SSS0.Px2.p1.9.m9.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S4.SS2.SSS0.Px2.p1.9.m9.1a"><mn id="S4.SS2.SSS0.Px2.p1.9.m9.1.1" xref="S4.SS2.SSS0.Px2.p1.9.m9.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p1.9.m9.1b"><cn type="integer" id="S4.SS2.SSS0.Px2.p1.9.m9.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.9.m9.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p1.9.m9.1c">1</annotation></semantics></math>, suggesting considerable redundancy in the <span id="S4.SS2.SSS0.Px2.p1.10.4" class="ltx_text ltx_font_typewriter">MHA</span>s of Transformers.
In contrast, when fixing <math id="S4.SS2.SSS0.Px2.p1.10.m10.1" class="ltx_Math" alttext="N_{\text{A}}" display="inline"><semantics id="S4.SS2.SSS0.Px2.p1.10.m10.1a"><msub id="S4.SS2.SSS0.Px2.p1.10.m10.1.1" xref="S4.SS2.SSS0.Px2.p1.10.m10.1.1.cmml"><mi id="S4.SS2.SSS0.Px2.p1.10.m10.1.1.2" xref="S4.SS2.SSS0.Px2.p1.10.m10.1.1.2.cmml">N</mi><mtext id="S4.SS2.SSS0.Px2.p1.10.m10.1.1.3" xref="S4.SS2.SSS0.Px2.p1.10.m10.1.1.3a.cmml">A</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p1.10.m10.1b"><apply id="S4.SS2.SSS0.Px2.p1.10.m10.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.10.m10.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px2.p1.10.m10.1.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.10.m10.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px2.p1.10.m10.1.1.2.cmml" xref="S4.SS2.SSS0.Px2.p1.10.m10.1.1.2">𝑁</ci><ci id="S4.SS2.SSS0.Px2.p1.10.m10.1.1.3a.cmml" xref="S4.SS2.SSS0.Px2.p1.10.m10.1.1.3"><mtext mathsize="70%" id="S4.SS2.SSS0.Px2.p1.10.m10.1.1.3.cmml" xref="S4.SS2.SSS0.Px2.p1.10.m10.1.1.3">A</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p1.10.m10.1c">N_{\text{A}}</annotation></semantics></math> and reducing the number of <span id="S4.SS2.SSS0.Px2.p1.10.5" class="ltx_text ltx_font_typewriter">FFN</span>s gradually, each time of removing an <span id="S4.SS2.SSS0.Px2.p1.10.6" class="ltx_text ltx_font_typewriter">FFN</span> leads to evident loss increase, indicating <span id="S4.SS2.SSS0.Px2.p1.10.7" class="ltx_text ltx_font_typewriter">FFN</span>s are less over-parameterized.
These quantitative findings align with previous research suggesting that the parameterization of attention can be simplified to enhance efficiency while maintaining performance <cite class="ltx_cite ltx_citemacro_citep">(DeepSeek-AI, <a href="#bib.bib10" title="" class="ltx_ref">2024</a>; Shazeer, <a href="#bib.bib39" title="" class="ltx_ref">2019</a>)</cite>.</p>
</div>
</section>
<section id="S4.SS2.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">As the parameter size scales up, MoM models enjoy consistent gains in both performance and efficiency.</h4>

<div id="S4.SS2.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS2.SSS0.Px3.p1.3" class="ltx_p">When we look into the difference across different scales, we observe that (1) the performance gain of MoM is stable; (2) <math id="S4.SS2.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{E}}" display="inline"><semantics id="S4.SS2.SSS0.Px3.p1.1.m1.1a"><msub id="S4.SS2.SSS0.Px3.p1.1.m1.1.1" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.cmml"><mtext id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.3" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.3a.cmml">E</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px3.p1.1.m1.1b"><apply id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2a.cmml" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2"><mtext id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2.cmml" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2">MoM</mtext></ci><ci id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.3a.cmml" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.3"><mtext mathsize="70%" id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.3.cmml" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.3">E</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px3.p1.1.m1.1c">\text{MoM}_{\text{E}}</annotation></semantics></math>-medium and <math id="S4.SS2.SSS0.Px3.p1.2.m2.1" class="ltx_Math" alttext="\text{MoM}_{\text{E}}" display="inline"><semantics id="S4.SS2.SSS0.Px3.p1.2.m2.1a"><msub id="S4.SS2.SSS0.Px3.p1.2.m2.1.1" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.cmml"><mtext id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2a.cmml">MoM</mtext><mtext id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.3" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.3a.cmml">E</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px3.p1.2.m2.1b"><apply id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2a.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2"><mtext id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2">MoM</mtext></ci><ci id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.3a.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.3"><mtext mathsize="70%" id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.3.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.3">E</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px3.p1.2.m2.1c">\text{MoM}_{\text{E}}</annotation></semantics></math>-large exhibit more significant reductions in resource costs comparing to <math id="S4.SS2.SSS0.Px3.p1.3.m3.1" class="ltx_Math" alttext="\text{MoM}_{\text{E}}" display="inline"><semantics id="S4.SS2.SSS0.Px3.p1.3.m3.1a"><msub id="S4.SS2.SSS0.Px3.p1.3.m3.1.1" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1.cmml"><mtext id="S4.SS2.SSS0.Px3.p1.3.m3.1.1.2" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1.2a.cmml">MoM</mtext><mtext id="S4.SS2.SSS0.Px3.p1.3.m3.1.1.3" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1.3a.cmml">E</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px3.p1.3.m3.1b"><apply id="S4.SS2.SSS0.Px3.p1.3.m3.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px3.p1.3.m3.1.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px3.p1.3.m3.1.1.2a.cmml" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1.2"><mtext id="S4.SS2.SSS0.Px3.p1.3.m3.1.1.2.cmml" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1.2">MoM</mtext></ci><ci id="S4.SS2.SSS0.Px3.p1.3.m3.1.1.3a.cmml" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1.3"><mtext mathsize="70%" id="S4.SS2.SSS0.Px3.p1.3.m3.1.1.3.cmml" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1.3">E</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px3.p1.3.m3.1c">\text{MoM}_{\text{E}}</annotation></semantics></math>-small.
These observations across different scales reinforce our previous motivation: Transformers are over-parameterized, which becomes more evident as the model size increases.</p>
</div>
</section>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Insights from hyper-parameter search</h3>

<div id="S4.SS3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.p1.10" class="ltx_p">Figure <a href="#S4.F4" title="Figure 4 ‣ 4.3 Insights from hyper-parameter search ‣ 4 Experiments ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> shows how the validation loss for MoM-small and MoM-medium varies with respect to different settings of <math id="S4.SS3.p1.1.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S4.SS3.p1.1.m1.1a"><mi id="S4.SS3.p1.1.m1.1.1" xref="S4.SS3.p1.1.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.1.m1.1b"><ci id="S4.SS3.p1.1.m1.1.1.cmml" xref="S4.SS3.p1.1.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.1.m1.1c">K</annotation></semantics></math> and <math id="S4.SS3.p1.2.m2.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S4.SS3.p1.2.m2.1a"><mi id="S4.SS3.p1.2.m2.1.1" xref="S4.SS3.p1.2.m2.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.2.m2.1b"><ci id="S4.SS3.p1.2.m2.1.1.cmml" xref="S4.SS3.p1.2.m2.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.2.m2.1c">H</annotation></semantics></math> (<math id="S4.SS3.p1.3.m3.4" class="ltx_Math" alttext="K\in\{1,2,3,4\}" display="inline"><semantics id="S4.SS3.p1.3.m3.4a"><mrow id="S4.SS3.p1.3.m3.4.5" xref="S4.SS3.p1.3.m3.4.5.cmml"><mi id="S4.SS3.p1.3.m3.4.5.2" xref="S4.SS3.p1.3.m3.4.5.2.cmml">K</mi><mo id="S4.SS3.p1.3.m3.4.5.1" xref="S4.SS3.p1.3.m3.4.5.1.cmml">∈</mo><mrow id="S4.SS3.p1.3.m3.4.5.3.2" xref="S4.SS3.p1.3.m3.4.5.3.1.cmml"><mo stretchy="false" id="S4.SS3.p1.3.m3.4.5.3.2.1" xref="S4.SS3.p1.3.m3.4.5.3.1.cmml">{</mo><mn id="S4.SS3.p1.3.m3.1.1" xref="S4.SS3.p1.3.m3.1.1.cmml">1</mn><mo id="S4.SS3.p1.3.m3.4.5.3.2.2" xref="S4.SS3.p1.3.m3.4.5.3.1.cmml">,</mo><mn id="S4.SS3.p1.3.m3.2.2" xref="S4.SS3.p1.3.m3.2.2.cmml">2</mn><mo id="S4.SS3.p1.3.m3.4.5.3.2.3" xref="S4.SS3.p1.3.m3.4.5.3.1.cmml">,</mo><mn id="S4.SS3.p1.3.m3.3.3" xref="S4.SS3.p1.3.m3.3.3.cmml">3</mn><mo id="S4.SS3.p1.3.m3.4.5.3.2.4" xref="S4.SS3.p1.3.m3.4.5.3.1.cmml">,</mo><mn id="S4.SS3.p1.3.m3.4.4" xref="S4.SS3.p1.3.m3.4.4.cmml">4</mn><mo stretchy="false" id="S4.SS3.p1.3.m3.4.5.3.2.5" xref="S4.SS3.p1.3.m3.4.5.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.3.m3.4b"><apply id="S4.SS3.p1.3.m3.4.5.cmml" xref="S4.SS3.p1.3.m3.4.5"><in id="S4.SS3.p1.3.m3.4.5.1.cmml" xref="S4.SS3.p1.3.m3.4.5.1"></in><ci id="S4.SS3.p1.3.m3.4.5.2.cmml" xref="S4.SS3.p1.3.m3.4.5.2">𝐾</ci><set id="S4.SS3.p1.3.m3.4.5.3.1.cmml" xref="S4.SS3.p1.3.m3.4.5.3.2"><cn type="integer" id="S4.SS3.p1.3.m3.1.1.cmml" xref="S4.SS3.p1.3.m3.1.1">1</cn><cn type="integer" id="S4.SS3.p1.3.m3.2.2.cmml" xref="S4.SS3.p1.3.m3.2.2">2</cn><cn type="integer" id="S4.SS3.p1.3.m3.3.3.cmml" xref="S4.SS3.p1.3.m3.3.3">3</cn><cn type="integer" id="S4.SS3.p1.3.m3.4.4.cmml" xref="S4.SS3.p1.3.m3.4.4">4</cn></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.3.m3.4c">K\in\{1,2,3,4\}</annotation></semantics></math>, <math id="S4.SS3.p1.4.m4.6" class="ltx_Math" alttext="H\in\{1,2,3,4,5,6\}" display="inline"><semantics id="S4.SS3.p1.4.m4.6a"><mrow id="S4.SS3.p1.4.m4.6.7" xref="S4.SS3.p1.4.m4.6.7.cmml"><mi id="S4.SS3.p1.4.m4.6.7.2" xref="S4.SS3.p1.4.m4.6.7.2.cmml">H</mi><mo id="S4.SS3.p1.4.m4.6.7.1" xref="S4.SS3.p1.4.m4.6.7.1.cmml">∈</mo><mrow id="S4.SS3.p1.4.m4.6.7.3.2" xref="S4.SS3.p1.4.m4.6.7.3.1.cmml"><mo stretchy="false" id="S4.SS3.p1.4.m4.6.7.3.2.1" xref="S4.SS3.p1.4.m4.6.7.3.1.cmml">{</mo><mn id="S4.SS3.p1.4.m4.1.1" xref="S4.SS3.p1.4.m4.1.1.cmml">1</mn><mo id="S4.SS3.p1.4.m4.6.7.3.2.2" xref="S4.SS3.p1.4.m4.6.7.3.1.cmml">,</mo><mn id="S4.SS3.p1.4.m4.2.2" xref="S4.SS3.p1.4.m4.2.2.cmml">2</mn><mo id="S4.SS3.p1.4.m4.6.7.3.2.3" xref="S4.SS3.p1.4.m4.6.7.3.1.cmml">,</mo><mn id="S4.SS3.p1.4.m4.3.3" xref="S4.SS3.p1.4.m4.3.3.cmml">3</mn><mo id="S4.SS3.p1.4.m4.6.7.3.2.4" xref="S4.SS3.p1.4.m4.6.7.3.1.cmml">,</mo><mn id="S4.SS3.p1.4.m4.4.4" xref="S4.SS3.p1.4.m4.4.4.cmml">4</mn><mo id="S4.SS3.p1.4.m4.6.7.3.2.5" xref="S4.SS3.p1.4.m4.6.7.3.1.cmml">,</mo><mn id="S4.SS3.p1.4.m4.5.5" xref="S4.SS3.p1.4.m4.5.5.cmml">5</mn><mo id="S4.SS3.p1.4.m4.6.7.3.2.6" xref="S4.SS3.p1.4.m4.6.7.3.1.cmml">,</mo><mn id="S4.SS3.p1.4.m4.6.6" xref="S4.SS3.p1.4.m4.6.6.cmml">6</mn><mo stretchy="false" id="S4.SS3.p1.4.m4.6.7.3.2.7" xref="S4.SS3.p1.4.m4.6.7.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.4.m4.6b"><apply id="S4.SS3.p1.4.m4.6.7.cmml" xref="S4.SS3.p1.4.m4.6.7"><in id="S4.SS3.p1.4.m4.6.7.1.cmml" xref="S4.SS3.p1.4.m4.6.7.1"></in><ci id="S4.SS3.p1.4.m4.6.7.2.cmml" xref="S4.SS3.p1.4.m4.6.7.2">𝐻</ci><set id="S4.SS3.p1.4.m4.6.7.3.1.cmml" xref="S4.SS3.p1.4.m4.6.7.3.2"><cn type="integer" id="S4.SS3.p1.4.m4.1.1.cmml" xref="S4.SS3.p1.4.m4.1.1">1</cn><cn type="integer" id="S4.SS3.p1.4.m4.2.2.cmml" xref="S4.SS3.p1.4.m4.2.2">2</cn><cn type="integer" id="S4.SS3.p1.4.m4.3.3.cmml" xref="S4.SS3.p1.4.m4.3.3">3</cn><cn type="integer" id="S4.SS3.p1.4.m4.4.4.cmml" xref="S4.SS3.p1.4.m4.4.4">4</cn><cn type="integer" id="S4.SS3.p1.4.m4.5.5.cmml" xref="S4.SS3.p1.4.m4.5.5">5</cn><cn type="integer" id="S4.SS3.p1.4.m4.6.6.cmml" xref="S4.SS3.p1.4.m4.6.6">6</cn></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.4.m4.6c">H\in\{1,2,3,4,5,6\}</annotation></semantics></math>). From this experiment, we have the following observations and insights: (1) allowing more modules to be assembled at each step (i.e., larger <math id="S4.SS3.p1.5.m5.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S4.SS3.p1.5.m5.1a"><mi id="S4.SS3.p1.5.m5.1.1" xref="S4.SS3.p1.5.m5.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.5.m5.1b"><ci id="S4.SS3.p1.5.m5.1.1.cmml" xref="S4.SS3.p1.5.m5.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.5.m5.1c">K</annotation></semantics></math>) and more rounds of assembling actions (i.e., larger <math id="S4.SS3.p1.6.m6.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S4.SS3.p1.6.m6.1a"><mi id="S4.SS3.p1.6.m6.1.1" xref="S4.SS3.p1.6.m6.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.6.m6.1b"><ci id="S4.SS3.p1.6.m6.1.1.cmml" xref="S4.SS3.p1.6.m6.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.6.m6.1c">H</annotation></semantics></math>) generally leads to better performance, indicating that <span id="S4.SS3.p1.10.1" class="ltx_text ltx_font_italic">Transformer-based models benefit from a larger computation graph even if the parameter size remains the same</span>. However, (2) the benefits of increasing <math id="S4.SS3.p1.7.m7.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S4.SS3.p1.7.m7.1a"><mi id="S4.SS3.p1.7.m7.1.1" xref="S4.SS3.p1.7.m7.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.7.m7.1b"><ci id="S4.SS3.p1.7.m7.1.1.cmml" xref="S4.SS3.p1.7.m7.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.7.m7.1c">K</annotation></semantics></math> and <math id="S4.SS3.p1.8.m8.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S4.SS3.p1.8.m8.1a"><mi id="S4.SS3.p1.8.m8.1.1" xref="S4.SS3.p1.8.m8.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.8.m8.1b"><ci id="S4.SS3.p1.8.m8.1.1.cmml" xref="S4.SS3.p1.8.m8.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.8.m8.1c">H</annotation></semantics></math> become marginal when <math id="S4.SS3.p1.9.m9.1" class="ltx_Math" alttext="K&gt;2" display="inline"><semantics id="S4.SS3.p1.9.m9.1a"><mrow id="S4.SS3.p1.9.m9.1.1" xref="S4.SS3.p1.9.m9.1.1.cmml"><mi id="S4.SS3.p1.9.m9.1.1.2" xref="S4.SS3.p1.9.m9.1.1.2.cmml">K</mi><mo id="S4.SS3.p1.9.m9.1.1.1" xref="S4.SS3.p1.9.m9.1.1.1.cmml">&gt;</mo><mn id="S4.SS3.p1.9.m9.1.1.3" xref="S4.SS3.p1.9.m9.1.1.3.cmml">2</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.9.m9.1b"><apply id="S4.SS3.p1.9.m9.1.1.cmml" xref="S4.SS3.p1.9.m9.1.1"><gt id="S4.SS3.p1.9.m9.1.1.1.cmml" xref="S4.SS3.p1.9.m9.1.1.1"></gt><ci id="S4.SS3.p1.9.m9.1.1.2.cmml" xref="S4.SS3.p1.9.m9.1.1.2">𝐾</ci><cn type="integer" id="S4.SS3.p1.9.m9.1.1.3.cmml" xref="S4.SS3.p1.9.m9.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.9.m9.1c">K&gt;2</annotation></semantics></math> and <math id="S4.SS3.p1.10.m10.1" class="ltx_Math" alttext="H&gt;1" display="inline"><semantics id="S4.SS3.p1.10.m10.1a"><mrow id="S4.SS3.p1.10.m10.1.1" xref="S4.SS3.p1.10.m10.1.1.cmml"><mi id="S4.SS3.p1.10.m10.1.1.2" xref="S4.SS3.p1.10.m10.1.1.2.cmml">H</mi><mo id="S4.SS3.p1.10.m10.1.1.1" xref="S4.SS3.p1.10.m10.1.1.1.cmml">&gt;</mo><mn id="S4.SS3.p1.10.m10.1.1.3" xref="S4.SS3.p1.10.m10.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.10.m10.1b"><apply id="S4.SS3.p1.10.m10.1.1.cmml" xref="S4.SS3.p1.10.m10.1.1"><gt id="S4.SS3.p1.10.m10.1.1.1.cmml" xref="S4.SS3.p1.10.m10.1.1.1"></gt><ci id="S4.SS3.p1.10.m10.1.1.2.cmml" xref="S4.SS3.p1.10.m10.1.1.2">𝐻</ci><cn type="integer" id="S4.SS3.p1.10.m10.1.1.3.cmml" xref="S4.SS3.p1.10.m10.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.10.m10.1c">H&gt;1</annotation></semantics></math>.
Comparing K3H1 to K2H6, we can see that the validation loss is comparable, while K3H1 performs slightly worse on downstream tasks as discussed in §<a href="#S4.SS2" title="4.2 Main results ‣ 4 Experiments ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.2</span></a>. However, K3H2 improves efficiency by flattening the depth, making it a good choice that balances performance and efficiency.
Flattening modules from different depths to the same depth cancels computation dependencies of each other. This characteristic brings an extra benefit because the computation of modules from the same depth can be parallelized.
This technique has been validated and adopted in MoE applications <cite class="ltx_cite ltx_citemacro_citep">(Fedus et al., <a href="#bib.bib14" title="" class="ltx_ref">2022</a>; Lepikhin et al., <a href="#bib.bib23" title="" class="ltx_ref">2021</a>)</cite> (called expert parallelism) and can be easily extended to further accelerate MoM (K3H2).</p>
</div>
<figure id="S4.F4" class="ltx_figure"><img src="/html/2407.06677/assets/x4.png" id="S4.F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="230" height="115" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>Validation loss for MoM-small and MoM-medium under different settings of <math id="S4.F4.3.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S4.F4.3.m1.1b"><mi id="S4.F4.3.m1.1.1" xref="S4.F4.3.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S4.F4.3.m1.1c"><ci id="S4.F4.3.m1.1.1.cmml" xref="S4.F4.3.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F4.3.m1.1d">K</annotation></semantics></math> and <math id="S4.F4.4.m2.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S4.F4.4.m2.1b"><mi id="S4.F4.4.m2.1.1" xref="S4.F4.4.m2.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S4.F4.4.m2.1c"><ci id="S4.F4.4.m2.1.1.cmml" xref="S4.F4.4.m2.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.F4.4.m2.1d">H</annotation></semantics></math>.</figcaption>
</figure>
<figure id="S4.T2" class="ltx_table">
<table id="S4.T2.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T2.1.1.1" class="ltx_tr">
<td id="S4.T2.1.1.1.1" class="ltx_td ltx_border_tt"></td>
<th id="S4.T2.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">Phase 1</th>
<th id="S4.T2.1.1.1.3" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">Phase 2</th>
<th id="S4.T2.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Val. Loss</th>
<th id="S4.T2.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">PPL.</th>
</tr>
<tr id="S4.T2.1.2.2" class="ltx_tr">
<td id="S4.T2.1.2.2.1" class="ltx_td ltx_align_left ltx_border_t">(1)</td>
<td id="S4.T2.1.2.2.2" class="ltx_td ltx_align_left ltx_border_t">MoM</td>
<td id="S4.T2.1.2.2.3" class="ltx_td ltx_align_left ltx_border_t">-</td>
<td id="S4.T2.1.2.2.4" class="ltx_td ltx_align_center ltx_border_t">2.85</td>
<td id="S4.T2.1.2.2.5" class="ltx_td ltx_align_center ltx_border_t">17.26</td>
</tr>
<tr id="S4.T2.1.3.3" class="ltx_tr">
<td id="S4.T2.1.3.3.1" class="ltx_td ltx_align_left">(2)</td>
<td id="S4.T2.1.3.3.2" class="ltx_td ltx_align_left">Vanilla</td>
<td id="S4.T2.1.3.3.3" class="ltx_td ltx_align_left">-</td>
<td id="S4.T2.1.3.3.4" class="ltx_td ltx_align_center">2.81</td>
<td id="S4.T2.1.3.3.5" class="ltx_td ltx_align_center">16.69</td>
</tr>
<tr id="S4.T2.1.4.4" class="ltx_tr">
<td id="S4.T2.1.4.4.1" class="ltx_td ltx_align_left">(3)</td>
<td id="S4.T2.1.4.4.2" class="ltx_td ltx_align_left">MoM</td>
<td id="S4.T2.1.4.4.3" class="ltx_td ltx_align_left">MoM</td>
<td id="S4.T2.1.4.4.4" class="ltx_td ltx_align_center">2.78</td>
<td id="S4.T2.1.4.4.5" class="ltx_td ltx_align_center">16.20</td>
</tr>
<tr id="S4.T2.1.5.5" class="ltx_tr">
<td id="S4.T2.1.5.5.1" class="ltx_td ltx_align_left">(4)</td>
<td id="S4.T2.1.5.5.2" class="ltx_td ltx_align_left">Vanilla</td>
<td id="S4.T2.1.5.5.3" class="ltx_td ltx_align_left">MoM</td>
<td id="S4.T2.1.5.5.4" class="ltx_td ltx_align_center"><span id="S4.T2.1.5.5.4.1" class="ltx_text ltx_font_bold">2.72</span></td>
<td id="S4.T2.1.5.5.5" class="ltx_td ltx_align_center"><span id="S4.T2.1.5.5.5.1" class="ltx_text ltx_font_bold">15.18</span></td>
</tr>
<tr id="S4.T2.1.6.6" class="ltx_tr">
<td id="S4.T2.1.6.6.1" class="ltx_td ltx_align_left ltx_border_bb">(5)</td>
<td id="S4.T2.1.6.6.2" class="ltx_td ltx_align_left ltx_border_bb">Vanilla</td>
<td id="S4.T2.1.6.6.3" class="ltx_td ltx_align_left ltx_border_bb">MoM-same</td>
<td id="S4.T2.1.6.6.4" class="ltx_td ltx_align_center ltx_border_bb">3.13</td>
<td id="S4.T2.1.6.6.5" class="ltx_td ltx_align_center ltx_border_bb">22.87</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>Performance comparison under different training setups. When training MoM from scratch, we set the total gradient steps of phase 1 to 20k. The total steps of phase 2 for all ablations are 10k.</figcaption>
</figure>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.4 </span>Impact of two-phase training</h3>

<div id="S4.SS4.p1" class="ltx_para ltx_noindent">
<p id="S4.SS4.p1.2" class="ltx_p">We investigate how the two-phase training strategy influences model performance using <math id="S4.SS4.p1.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{P}}" display="inline"><semantics id="S4.SS4.p1.1.m1.1a"><msub id="S4.SS4.p1.1.m1.1.1" xref="S4.SS4.p1.1.m1.1.1.cmml"><mtext id="S4.SS4.p1.1.m1.1.1.2" xref="S4.SS4.p1.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="S4.SS4.p1.1.m1.1.1.3" xref="S4.SS4.p1.1.m1.1.1.3a.cmml">P</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS4.p1.1.m1.1b"><apply id="S4.SS4.p1.1.m1.1.1.cmml" xref="S4.SS4.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS4.p1.1.m1.1.1.1.cmml" xref="S4.SS4.p1.1.m1.1.1">subscript</csymbol><ci id="S4.SS4.p1.1.m1.1.1.2a.cmml" xref="S4.SS4.p1.1.m1.1.1.2"><mtext id="S4.SS4.p1.1.m1.1.1.2.cmml" xref="S4.SS4.p1.1.m1.1.1.2">MoM</mtext></ci><ci id="S4.SS4.p1.1.m1.1.1.3a.cmml" xref="S4.SS4.p1.1.m1.1.1.3"><mtext mathsize="70%" id="S4.SS4.p1.1.m1.1.1.3.cmml" xref="S4.SS4.p1.1.m1.1.1.3">P</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p1.1.m1.1c">\text{MoM}_{\text{P}}</annotation></semantics></math>-medium as case studies. Specifically, we consider the following training strategies: (1) training MoM from scratch with 20k steps; (2) training vanilla Transformer with 20k steps; (3) training MoM from scratch with 30k steps; (4) <math id="S4.SS4.p1.2.m2.1" class="ltx_Math" alttext="\text{MoM}_{\text{P}}" display="inline"><semantics id="S4.SS4.p1.2.m2.1a"><msub id="S4.SS4.p1.2.m2.1.1" xref="S4.SS4.p1.2.m2.1.1.cmml"><mtext id="S4.SS4.p1.2.m2.1.1.2" xref="S4.SS4.p1.2.m2.1.1.2a.cmml">MoM</mtext><mtext id="S4.SS4.p1.2.m2.1.1.3" xref="S4.SS4.p1.2.m2.1.1.3a.cmml">P</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS4.p1.2.m2.1b"><apply id="S4.SS4.p1.2.m2.1.1.cmml" xref="S4.SS4.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S4.SS4.p1.2.m2.1.1.1.cmml" xref="S4.SS4.p1.2.m2.1.1">subscript</csymbol><ci id="S4.SS4.p1.2.m2.1.1.2a.cmml" xref="S4.SS4.p1.2.m2.1.1.2"><mtext id="S4.SS4.p1.2.m2.1.1.2.cmml" xref="S4.SS4.p1.2.m2.1.1.2">MoM</mtext></ci><ci id="S4.SS4.p1.2.m2.1.1.3a.cmml" xref="S4.SS4.p1.2.m2.1.1.3"><mtext mathsize="70%" id="S4.SS4.p1.2.m2.1.1.3.cmml" xref="S4.SS4.p1.2.m2.1.1.3">P</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p1.2.m2.1c">\text{MoM}_{\text{P}}</annotation></semantics></math>-medium that is trained with the two-phase strategy; and (5) training MoM with the two-phase strategy but in the second phase, <span id="S4.SS4.p1.2.1" class="ltx_text ltx_font_typewriter">MHA</span> modules and <span id="S4.SS4.p1.2.2" class="ltx_text ltx_font_typewriter">FFN</span> modules are forced to be identical, respectively. Table <a href="#S4.T2" title="Table 2 ‣ 4.3 Insights from hyper-parameter search ‣ 4 Experiments ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> shows the results. First, comparing (1) with (2), we find that the absence of weight initialization compromises the training quality of MoM, making it worse than the vanilla Transformer, underscoring the importance of initializing module weights with a well-trained vanilla Transformer model for MoM. A plausible explanation for the results is that training without warm-up leads to homogeneous modules that hurt the convergence of the routers (as illustrated in Appendix <a href="#A4" title="Appendix D Router analysis ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D</span></a>). The explanation is further justified by comparison between (4) and (5), as when we force all modules to be identical, the performance of MoM also drops dramatically even with the two-phase training strategy. As expected, training with more steps can enhance performance (cf., comparison between (1) and (3)), but the two-phase strategy is still the better choice when we compare (3) with (4).</p>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Conclusions</h2>

<div id="S5.p1" class="ltx_para ltx_noindent">
<p id="S5.p1.1" class="ltx_p">In this work, we propose Mixture-of-Modules (MoM), a novel architecture that reinvents transformers as a collection of individual modules and the dynamic assembly process conducted with these modules. This novel view offers us an opportunity to explore a wide range of different configurations of model architecture and unify a series of transformer variants. With exhaustive experiments, we not only validate the effectiveness of MoM by both significant efficiency and performance gains but also reach new insights about Transformers.</p>
</div>
</section>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Limitations</h2>

<div id="Sx1.p1" class="ltx_para ltx_noindent">
<p id="Sx1.p1.1" class="ltx_p">Our current design of the router still has room for improvement. Unlike MoE wherein the router makes one-time decisions about which experts to select, the router of MoM is responsible for conducting multi-step decision-making. In this scenario, instructing the router to make correct decisions continuously is a hard problem since the decision space grows exponentially with the increase of assembly steps. The current implementation has not considered this question and has not explicitly encouraged or discouraged the router to make some choices, thus, we are not sure whether the learned routing decisions are optimal or not. In the future, we will explore using techniques like reinforcement learning or neural architecture search to design more sophisticated routers.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Agostinelli et al. (2023)</span>
<span class="ltx_bibblock">
Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour, and Christian Frank.

</span>
<span class="ltx_bibblock">Musiclm: Generating music from text, 2023.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ainslie et al. (2023)</span>
<span class="ltx_bibblock">
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, et al.

</span>
<span class="ltx_bibblock">Colt5: Faster long-range transformers with conditional computation.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2303.09752</em>, 2023.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bapna et al. (2020)</span>
<span class="ltx_bibblock">
Ankur Bapna, Naveen Arivazhagan, and Orhan Firat.

</span>
<span class="ltx_bibblock">Controlling computation versus quality for neural sequence models, 2020.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bengio et al. (2013)</span>
<span class="ltx_bibblock">
Yoshua Bengio, Nicholas Léonard, and Aaron Courville.

</span>
<span class="ltx_bibblock">Estimating or propagating gradients through stochastic neurons for conditional computation, 2013.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bubeck et al. (2023)</span>
<span class="ltx_bibblock">
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang.

</span>
<span class="ltx_bibblock">Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cho &amp; Bengio (2014)</span>
<span class="ltx_bibblock">
Kyunghyun Cho and Yoshua Bengio.

</span>
<span class="ltx_bibblock">Exponentially increasing the capacity-to-computation ratio for conditional computation in deep learning, 2014.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cho et al. (2014)</span>
<span class="ltx_bibblock">
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.

</span>
<span class="ltx_bibblock">Learning phrase representations using RNN encoder–decoder for statistical machine translation.

</span>
<span class="ltx_bibblock">In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), <em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)</em>, pp.  1724–1734, Doha, Qatar, October 2014. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.3115/v1/D14-1179</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/D14-1179" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/D14-1179</a>.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Corro et al. (2023)</span>
<span class="ltx_bibblock">
Luciano Del Corro, Allie Del Giorno, Sahaj Agarwal, Bin Yu, Ahmed Awadallah, and Subhabrata Mukherjee.

</span>
<span class="ltx_bibblock">Skipdecode: Autoregressive skip decoding with batching and caching for efficient llm inference, 2023.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Davis &amp; Arel (2014)</span>
<span class="ltx_bibblock">
Andrew Davis and Itamar Arel.

</span>
<span class="ltx_bibblock">Low-rank approximations for conditional feedforward computation in deep neural networks, 2014.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">DeepSeek-AI (2024)</span>
<span class="ltx_bibblock">
DeepSeek-AI.

</span>
<span class="ltx_bibblock">Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">DeepSpeed (2023)</span>
<span class="ltx_bibblock">
DeepSpeed.

</span>
<span class="ltx_bibblock">Deepspeed flops profiler, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://www.deepspeed.ai/tutorials/flops-profiler/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.deepspeed.ai/tutorials/flops-profiler/</a>.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dehghani et al. (2019)</span>
<span class="ltx_bibblock">
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser.

</span>
<span class="ltx_bibblock">Universal transformers.

</span>
<span class="ltx_bibblock">In <em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2019.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=HyzdRiR9Y7" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=HyzdRiR9Y7</a>.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Elbayad et al. (2020)</span>
<span class="ltx_bibblock">
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli.

</span>
<span class="ltx_bibblock">Depth-adaptive transformer, 2020.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fedus et al. (2022)</span>
<span class="ltx_bibblock">
William Fedus, Barret Zoph, and Noam Shazeer.

</span>
<span class="ltx_bibblock">Switch transformers: scaling to trillion parameter models with simple and efficient sparsity.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">J. Mach. Learn. Res.</em>, 23(1), jan 2022.

</span>
<span class="ltx_bibblock">ISSN 1532-4435.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Figurnov et al. (2017)</span>
<span class="ltx_bibblock">
Michael Figurnov, Maxwell D. Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov.

</span>
<span class="ltx_bibblock">Spatially adaptive computation time for residual networks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)</em>, pp.  1790–1799, 2017.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1109/CVPR.2017.194</span>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Geva et al. (2021)</span>
<span class="ltx_bibblock">
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy.

</span>
<span class="ltx_bibblock">Transformer feed-forward layers are key-value memories.

</span>
<span class="ltx_bibblock">In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), <em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing</em>, pp.  5484–5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2021.emnlp-main.446</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2021.emnlp-main.446" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2021.emnlp-main.446</a>.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gokaslan &amp; Cohen (2019)</span>
<span class="ltx_bibblock">
Aaron Gokaslan and Vanya Cohen.

</span>
<span class="ltx_bibblock">Openwebtext corpus.

</span>
<span class="ltx_bibblock"><a target="_blank" href="http://Skylion007.github.io/OpenWebTextCorpus" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://Skylion007.github.io/OpenWebTextCorpus</a>, 2019.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Graves (2016)</span>
<span class="ltx_bibblock">
Alex Graves.

</span>
<span class="ltx_bibblock">Adaptive computation time for recurrent neural networks, 2016.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">karpathy (2023)</span>
<span class="ltx_bibblock">
karpathy.

</span>
<span class="ltx_bibblock">nanogpt, the simplest, fastest repository for training/finetuning medium-sized gpts., 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://github.com/karpathy/nanoGPT" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/karpathy/nanoGPT</a>.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kim et al. (2024)</span>
<span class="ltx_bibblock">
Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kyu Song.

</span>
<span class="ltx_bibblock">Shortened llama: A simple depth pruning for large language models, 2024.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lan et al. (2020)</span>
<span class="ltx_bibblock">
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.

</span>
<span class="ltx_bibblock">Albert: A lite bert for self-supervised learning of language representations.

</span>
<span class="ltx_bibblock">In <em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2020.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=H1eA7AEtvS" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=H1eA7AEtvS</a>.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lei et al. (2024)</span>
<span class="ltx_bibblock">
Tao Lei, Junwen Bai, Siddhartha Brahma, Joshua Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Zhao, Yuexin Wu, Bo Li, et al.

</span>
<span class="ltx_bibblock">Conditional adapters: Parameter-efficient transfer learning with fast inference.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lepikhin et al. (2021)</span>
<span class="ltx_bibblock">
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen.

</span>
<span class="ltx_bibblock">{GS}hard: Scaling giant models with conditional computation and automatic sharding.

</span>
<span class="ltx_bibblock">In <em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2021.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=qrwe7XHTmYb" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=qrwe7XHTmYb</a>.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2020)</span>
<span class="ltx_bibblock">
Yijin Liu, Fandong Meng, Jie Zhou, Yufeng Chen, and Jinan Xu.

</span>
<span class="ltx_bibblock">Faster depth-adaptive transformers.

</span>
<span class="ltx_bibblock">In <em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">AAAI Conference on Artificial Intelligence</em>, 2020.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://api.semanticscholar.org/CorpusID:229283620" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://api.semanticscholar.org/CorpusID:229283620</a>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2024)</span>
<span class="ltx_bibblock">
Yijin Liu, Fandong Meng, and Jie Zhou.

</span>
<span class="ltx_bibblock">Accelerating inference in large language models with a unified layer skipping strategy, 2024.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lv et al. (2024)</span>
<span class="ltx_bibblock">
Ang Lv, Yuhan Chen, Kaiyi Zhang, Yulong Wang, Lifeng Liu, Ji-Rong Wen, Jian Xie, and Rui Yan.

</span>
<span class="ltx_bibblock">Interpreting key mechanisms of factual recall in transformer-based language models, 2024.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Madani et al. (2023)</span>
<span class="ltx_bibblock">
Ali Madani, Ben Krause, Eric R. Greene, Subu Subramanian, Benjamin P. Mohr, James M. Holton, Jose Luis Olmos, Caiming Xiong, Zachary Z. Sun, Richard Socher, James S. Fraser, and Nikhil Naik.

</span>
<span class="ltx_bibblock">Large language models generate functional protein sequences across diverse families.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Nature Biotechnology</em>, 41(8):1099–1106, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1038/s41587-022-01618-2</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.1038/s41587-022-01618-2" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1038/s41587-022-01618-2</a>.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">McGrath et al. (2023)</span>
<span class="ltx_bibblock">
Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg.

</span>
<span class="ltx_bibblock">The hydra effect: Emergent self-repair in language model computations, 2023.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Minsky (1986)</span>
<span class="ltx_bibblock">
M. Minsky.

</span>
<span class="ltx_bibblock"><em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">The Society of Mind</em>.

</span>
<span class="ltx_bibblock">Touchstone book. Simon and Schuster, 1986.

</span>
<span class="ltx_bibblock">ISBN 9780671607401.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://books.google.com.hk/books?id=veVOAAAAMAAJ" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://books.google.com.hk/books?id=veVOAAAAMAAJ</a>.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Narayan et al. (2018a)</span>
<span class="ltx_bibblock">
Shashi Narayan, Shay B Cohen, and Mirella Lapata.

</span>
<span class="ltx_bibblock">Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing</em>, pp.  1797–1807, 2018a.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Narayan et al. (2018b)</span>
<span class="ltx_bibblock">
Shashi Narayan, Shay B. Cohen, and Mirella Lapata.

</span>
<span class="ltx_bibblock">Don’t give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing</em>, Brussels, Belgium, 2018b.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Olsson et al. (2022)</span>
<span class="ltx_bibblock">
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah.

</span>
<span class="ltx_bibblock">In-context learning and induction heads.

</span>
<span class="ltx_bibblock"><em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">Transformer Circuits Thread</em>, 2022.

</span>
<span class="ltx_bibblock">https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Peebles &amp; Xie (2023)</span>
<span class="ltx_bibblock">
William Peebles and Saining Xie.

</span>
<span class="ltx_bibblock">Scalable diffusion models with transformers.

</span>
<span class="ltx_bibblock">In <em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)</em>, pp.  4195–4205, October 2023.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">pytorch (2023)</span>
<span class="ltx_bibblock">
pytorch.

</span>
<span class="ltx_bibblock">Pytorch profiler, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html</a>.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radford et al. (2018)</span>
<span class="ltx_bibblock">
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.

</span>
<span class="ltx_bibblock">Improving language understanding by generative pre-training.

</span>
<span class="ltx_bibblock">2018.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radford et al. (2019)</span>
<span class="ltx_bibblock">
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.

</span>
<span class="ltx_bibblock">Language models are unsupervised multitask learners.

</span>
<span class="ltx_bibblock">2019.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://api.semanticscholar.org/CorpusID:160025533" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://api.semanticscholar.org/CorpusID:160025533</a>.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Raposo et al. (2024)</span>
<span class="ltx_bibblock">
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro.

</span>
<span class="ltx_bibblock">Mixture-of-depths: Dynamically allocating compute in transformer-based language models, 2024.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schuster et al. (2022)</span>
<span class="ltx_bibblock">
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler.

</span>
<span class="ltx_bibblock">Confident adaptive language modeling.

</span>
<span class="ltx_bibblock">In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), <em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, volume 35, pp.  17456–17472. Curran Associates, Inc., 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.neurips.cc/paper_files/paper/2022/file/6fac9e316a4ae75ea244ddcef1982c71-Paper-Conference.pdf" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.neurips.cc/paper_files/paper/2022/file/6fac9e316a4ae75ea244ddcef1982c71-Paper-Conference.pdf</a>.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shazeer (2019)</span>
<span class="ltx_bibblock">
Noam Shazeer.

</span>
<span class="ltx_bibblock">Fast transformer decoding: One write-head is all you need, 2019.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shazeer et al. (2017)</span>
<span class="ltx_bibblock">
Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean.

</span>
<span class="ltx_bibblock">Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.

</span>
<span class="ltx_bibblock">In <em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2017.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=B1ckMDqlg" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=B1ckMDqlg</a>.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Singhal et al. (2023)</span>
<span class="ltx_bibblock">
Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner-Fushman, Blaise Agüera y Arcas, Dale Webster, Greg S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan.

</span>
<span class="ltx_bibblock">Large language models encode clinical knowledge.

</span>
<span class="ltx_bibblock"><em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">Nature</em>, 620(7972):172–180, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1038/s41586-023-06291-2</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.1038/s41586-023-06291-2" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1038/s41586-023-06291-2</a>.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Srivastava et al. (2015)</span>
<span class="ltx_bibblock">
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber.

</span>
<span class="ltx_bibblock">Highway networks, 2015.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Varshney et al. (2023)</span>
<span class="ltx_bibblock">
Neeraj Varshney, Agneet Chatterjee, Mihir Parmar, and Chitta Baral.

</span>
<span class="ltx_bibblock">Accelerating llama inference by enabling intermediate layer decoding via instruction tuning with lite, 2023.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vaswani et al. (2017)</span>
<span class="ltx_bibblock">
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.

</span>
<span class="ltx_bibblock">Attention is all you need.

</span>
<span class="ltx_bibblock">In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), <em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, volume 30. Curran Associates, Inc., 2017.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf</a>.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2018a)</span>
<span class="ltx_bibblock">
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.

</span>
<span class="ltx_bibblock">Glue: A multi-task benchmark and analysis platform for natural language understanding.

</span>
<span class="ltx_bibblock">In <em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2018a.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023)</span>
<span class="ltx_bibblock">
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt.

</span>
<span class="ltx_bibblock">Interpretability in the wild: a circuit for indirect object identification in GPT-2 small.

</span>
<span class="ltx_bibblock">In <em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">The Eleventh International Conference on Learning Representations</em>, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=NpsVSN6o4ul" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=NpsVSN6o4ul</a>.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2018b)</span>
<span class="ltx_bibblock">
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E. Gonzalez.

</span>
<span class="ltx_bibblock">Skipnet: Learning dynamic routing in convolutional networks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIII</em>, pp.  420–436, Berlin, Heidelberg, 2018b. Springer-Verlag.

</span>
<span class="ltx_bibblock">ISBN 978-3-030-01260-1.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1007/978-3-030-01261-8˙25</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.1007/978-3-030-01261-8_25" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1007/978-3-030-01261-8_25</a>.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xin et al. (2020)</span>
<span class="ltx_bibblock">
Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin.

</span>
<span class="ltx_bibblock">DeeBERT: Dynamic early exiting for accelerating BERT inference.

</span>
<span class="ltx_bibblock">In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), <em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics</em>, pp.  2246–2251, Online, July 2020. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2020.acl-main.204</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2020.acl-main.204" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2020.acl-main.204</a>.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yang et al. (2024)</span>
<span class="ltx_bibblock">
Yifei Yang, Zouying Cao, and Hai Zhao.

</span>
<span class="ltx_bibblock">Laco: Large language model pruning via layer collapse.

</span>
<span class="ltx_bibblock"><em id="bib.bib49.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2402.11187, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://api.semanticscholar.org/CorpusID:267751181" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://api.semanticscholar.org/CorpusID:267751181</a>.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zeng et al. (2023)</span>
<span class="ltx_bibblock">
Dewen Zeng, Nan Du, Tao Wang, Yuanzhong Xu, Tao Lei, Zhifeng Chen, and Claire Cui.

</span>
<span class="ltx_bibblock">Learning to skip for language modeling, 2023.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. (2020)</span>
<span class="ltx_bibblock">
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei.

</span>
<span class="ltx_bibblock">Bert loses patience: Fast and robust inference with early exit.

</span>
<span class="ltx_bibblock">In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), <em id="bib.bib51.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, volume 33, pp.  18330–18341. Curran Associates, Inc., 2020.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.neurips.cc/paper_files/paper/2020/file/d4dd111a4fd973394238aca5c05bebe3-Paper.pdf" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.neurips.cc/paper_files/paper/2020/file/d4dd111a4fd973394238aca5c05bebe3-Paper.pdf</a>.

</span>
</li>
</ul>
</section>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>More implementation details</h2>

<div id="A1.p1" class="ltx_para ltx_noindent">
<p id="A1.p1.1" class="ltx_p">Table <a href="#A1.T3" title="Table 3 ‣ Appendix A More implementation details ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> lists the configurations of MoM-small/medium/large.</p>
</div>
<figure id="A1.T3" class="ltx_table">
<table id="A1.T3.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T3.1.1.1" class="ltx_tr">
<th id="A1.T3.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="A1.T3.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T3.1.1.1.2.1" class="ltx_text ltx_font_bold">MoM-small</span></th>
<th id="A1.T3.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T3.1.1.1.3.1" class="ltx_text ltx_font_bold">MoM-medium</span></th>
<th id="A1.T3.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T3.1.1.1.4.1" class="ltx_text ltx_font_bold">MoM-large</span></th>
</tr>
<tr id="A1.T3.1.2.2" class="ltx_tr">
<th id="A1.T3.1.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_t">Initialization model</th>
<th id="A1.T3.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">GPT2-small</th>
<th id="A1.T3.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">GPT2-medium</th>
<th id="A1.T3.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">GPT2-large</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T3.1.3.1" class="ltx_tr">
<th id="A1.T3.1.3.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_row">Hidden size</th>
<td id="A1.T3.1.3.1.2" class="ltx_td ltx_align_center">768</td>
<td id="A1.T3.1.3.1.3" class="ltx_td ltx_align_center">1024</td>
<td id="A1.T3.1.3.1.4" class="ltx_td ltx_align_center">1280</td>
</tr>
<tr id="A1.T3.1.4.2" class="ltx_tr">
<th id="A1.T3.1.4.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row">Total number of <span id="A1.T3.1.4.2.1.1" class="ltx_text ltx_font_typewriter">FFN</span>/<span id="A1.T3.1.4.2.1.2" class="ltx_text ltx_font_typewriter">MHA</span> modules</th>
<td id="A1.T3.1.4.2.2" class="ltx_td ltx_align_center">12</td>
<td id="A1.T3.1.4.2.3" class="ltx_td ltx_align_center">24</td>
<td id="A1.T3.1.4.2.4" class="ltx_td ltx_align_center">36</td>
</tr>
<tr id="A1.T3.1.5.3" class="ltx_tr">
<th id="A1.T3.1.5.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row">Number of attention heads</th>
<td id="A1.T3.1.5.3.2" class="ltx_td ltx_align_center">12</td>
<td id="A1.T3.1.5.3.3" class="ltx_td ltx_align_center">16</td>
<td id="A1.T3.1.5.3.4" class="ltx_td ltx_align_center">20</td>
</tr>
<tr id="A1.T3.1.6.4" class="ltx_tr">
<th id="A1.T3.1.6.4.1" class="ltx_td ltx_align_center ltx_th ltx_th_row">Max sequence length</th>
<td id="A1.T3.1.6.4.2" class="ltx_td ltx_align_center">1024</td>
<td id="A1.T3.1.6.4.3" class="ltx_td ltx_align_center">1024</td>
<td id="A1.T3.1.6.4.4" class="ltx_td ltx_align_center">1024</td>
</tr>
<tr id="A1.T3.1.7.5" class="ltx_tr">
<th id="A1.T3.1.7.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb">Vocabulary size</th>
<td id="A1.T3.1.7.5.2" class="ltx_td ltx_align_center ltx_border_bb">50257</td>
<td id="A1.T3.1.7.5.3" class="ltx_td ltx_align_center ltx_border_bb">50257</td>
<td id="A1.T3.1.7.5.4" class="ltx_td ltx_align_center ltx_border_bb">50257</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>Model configurations for MoM-small/medium/large.</figcaption>
</figure>
<div id="A1.p2" class="ltx_para ltx_noindent">
<p id="A1.p2.6" class="ltx_p">In practice, we segment MoM into equally-sized chunks, each containing <math id="A1.p2.1.m1.1" class="ltx_Math" alttext="4" display="inline"><semantics id="A1.p2.1.m1.1a"><mn id="A1.p2.1.m1.1.1" xref="A1.p2.1.m1.1.1.cmml">4</mn><annotation-xml encoding="MathML-Content" id="A1.p2.1.m1.1b"><cn type="integer" id="A1.p2.1.m1.1.1.cmml" xref="A1.p2.1.m1.1.1">4</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.1.m1.1c">4</annotation></semantics></math> <span id="A1.p2.6.1" class="ltx_text ltx_font_typewriter">MHA</span> modules and <math id="A1.p2.2.m2.1" class="ltx_Math" alttext="4" display="inline"><semantics id="A1.p2.2.m2.1a"><mn id="A1.p2.2.m2.1.1" xref="A1.p2.2.m2.1.1.cmml">4</mn><annotation-xml encoding="MathML-Content" id="A1.p2.2.m2.1b"><cn type="integer" id="A1.p2.2.m2.1.1.cmml" xref="A1.p2.2.m2.1.1">4</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.2.m2.1c">4</annotation></semantics></math> <span id="A1.p2.6.2" class="ltx_text ltx_font_typewriter">FFN</span> modules, namely <math id="A1.p2.3.m3.1" class="ltx_Math" alttext="N=4" display="inline"><semantics id="A1.p2.3.m3.1a"><mrow id="A1.p2.3.m3.1.1" xref="A1.p2.3.m3.1.1.cmml"><mi id="A1.p2.3.m3.1.1.2" xref="A1.p2.3.m3.1.1.2.cmml">N</mi><mo id="A1.p2.3.m3.1.1.1" xref="A1.p2.3.m3.1.1.1.cmml">=</mo><mn id="A1.p2.3.m3.1.1.3" xref="A1.p2.3.m3.1.1.3.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.p2.3.m3.1b"><apply id="A1.p2.3.m3.1.1.cmml" xref="A1.p2.3.m3.1.1"><eq id="A1.p2.3.m3.1.1.1.cmml" xref="A1.p2.3.m3.1.1.1"></eq><ci id="A1.p2.3.m3.1.1.2.cmml" xref="A1.p2.3.m3.1.1.2">𝑁</ci><cn type="integer" id="A1.p2.3.m3.1.1.3.cmml" xref="A1.p2.3.m3.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.3.m3.1c">N=4</annotation></semantics></math>. Within each chunk, we execute the MoM assembly process as presented in §<a href="#S3" title="3 Methodology ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>.
We restrict the search space of each chunk by setting <math id="A1.p2.4.m4.1" class="ltx_Math" alttext="K\leq 4" display="inline"><semantics id="A1.p2.4.m4.1a"><mrow id="A1.p2.4.m4.1.1" xref="A1.p2.4.m4.1.1.cmml"><mi id="A1.p2.4.m4.1.1.2" xref="A1.p2.4.m4.1.1.2.cmml">K</mi><mo id="A1.p2.4.m4.1.1.1" xref="A1.p2.4.m4.1.1.1.cmml">≤</mo><mn id="A1.p2.4.m4.1.1.3" xref="A1.p2.4.m4.1.1.3.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.p2.4.m4.1b"><apply id="A1.p2.4.m4.1.1.cmml" xref="A1.p2.4.m4.1.1"><leq id="A1.p2.4.m4.1.1.1.cmml" xref="A1.p2.4.m4.1.1.1"></leq><ci id="A1.p2.4.m4.1.1.2.cmml" xref="A1.p2.4.m4.1.1.2">𝐾</ci><cn type="integer" id="A1.p2.4.m4.1.1.3.cmml" xref="A1.p2.4.m4.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.4.m4.1c">K\leq 4</annotation></semantics></math> and <math id="A1.p2.5.m5.1" class="ltx_Math" alttext="H\leq 6" display="inline"><semantics id="A1.p2.5.m5.1a"><mrow id="A1.p2.5.m5.1.1" xref="A1.p2.5.m5.1.1.cmml"><mi id="A1.p2.5.m5.1.1.2" xref="A1.p2.5.m5.1.1.2.cmml">H</mi><mo id="A1.p2.5.m5.1.1.1" xref="A1.p2.5.m5.1.1.1.cmml">≤</mo><mn id="A1.p2.5.m5.1.1.3" xref="A1.p2.5.m5.1.1.3.cmml">6</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.p2.5.m5.1b"><apply id="A1.p2.5.m5.1.1.cmml" xref="A1.p2.5.m5.1.1"><leq id="A1.p2.5.m5.1.1.1.cmml" xref="A1.p2.5.m5.1.1.1"></leq><ci id="A1.p2.5.m5.1.1.2.cmml" xref="A1.p2.5.m5.1.1.2">𝐻</ci><cn type="integer" id="A1.p2.5.m5.1.1.3.cmml" xref="A1.p2.5.m5.1.1.3">6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.5.m5.1c">H\leq 6</annotation></semantics></math>, which results in <math id="A1.p2.6.m6.1" class="ltx_Math" alttext="4\times 6=24" display="inline"><semantics id="A1.p2.6.m6.1a"><mrow id="A1.p2.6.m6.1.1" xref="A1.p2.6.m6.1.1.cmml"><mrow id="A1.p2.6.m6.1.1.2" xref="A1.p2.6.m6.1.1.2.cmml"><mn id="A1.p2.6.m6.1.1.2.2" xref="A1.p2.6.m6.1.1.2.2.cmml">4</mn><mo lspace="0.222em" rspace="0.222em" id="A1.p2.6.m6.1.1.2.1" xref="A1.p2.6.m6.1.1.2.1.cmml">×</mo><mn id="A1.p2.6.m6.1.1.2.3" xref="A1.p2.6.m6.1.1.2.3.cmml">6</mn></mrow><mo id="A1.p2.6.m6.1.1.1" xref="A1.p2.6.m6.1.1.1.cmml">=</mo><mn id="A1.p2.6.m6.1.1.3" xref="A1.p2.6.m6.1.1.3.cmml">24</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.p2.6.m6.1b"><apply id="A1.p2.6.m6.1.1.cmml" xref="A1.p2.6.m6.1.1"><eq id="A1.p2.6.m6.1.1.1.cmml" xref="A1.p2.6.m6.1.1.1"></eq><apply id="A1.p2.6.m6.1.1.2.cmml" xref="A1.p2.6.m6.1.1.2"><times id="A1.p2.6.m6.1.1.2.1.cmml" xref="A1.p2.6.m6.1.1.2.1"></times><cn type="integer" id="A1.p2.6.m6.1.1.2.2.cmml" xref="A1.p2.6.m6.1.1.2.2">4</cn><cn type="integer" id="A1.p2.6.m6.1.1.2.3.cmml" xref="A1.p2.6.m6.1.1.2.3">6</cn></apply><cn type="integer" id="A1.p2.6.m6.1.1.3.cmml" xref="A1.p2.6.m6.1.1.3">24</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.6.m6.1c">4\times 6=24</annotation></semantics></math> combinations in total.</p>
</div>
<div id="A1.p3" class="ltx_para ltx_noindent">
<p id="A1.p3.3" class="ltx_p">Then we elaborate on the initialization of chunked MoM. Taking a 8-layer vanilla transformer as an example, the architecture is sliced as: the bottom/top 2 layers remain the same, and modules in the middle 4 layers form a MoM block, wherein we conduct the iterative assembly process. We denote this chunking strategy as <span id="A1.p3.3.1" class="ltx_text ltx_font_typewriter">[1-1-4-1-1]</span> where “<span id="A1.p3.3.2" class="ltx_text ltx_font_typewriter">1</span>” represents the standard Transformer block and “<span id="A1.p3.3.3" class="ltx_text ltx_font_typewriter">4</span>” represents a chunk whose <math id="A1.p3.1.m1.1" class="ltx_Math" alttext="N" display="inline"><semantics id="A1.p3.1.m1.1a"><mi id="A1.p3.1.m1.1.1" xref="A1.p3.1.m1.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="A1.p3.1.m1.1b"><ci id="A1.p3.1.m1.1.1.cmml" xref="A1.p3.1.m1.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p3.1.m1.1c">N</annotation></semantics></math> equals to 4. Empirically, we find this setting to be stable across various choices of K<math id="A1.p3.2.m2.1" class="ltx_Math" alttext="a" display="inline"><semantics id="A1.p3.2.m2.1a"><mi id="A1.p3.2.m2.1.1" xref="A1.p3.2.m2.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="A1.p3.2.m2.1b"><ci id="A1.p3.2.m2.1.1.cmml" xref="A1.p3.2.m2.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p3.2.m2.1c">a</annotation></semantics></math>H<math id="A1.p3.3.m3.1" class="ltx_Math" alttext="b" display="inline"><semantics id="A1.p3.3.m3.1a"><mi id="A1.p3.3.m3.1.1" xref="A1.p3.3.m3.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="A1.p3.3.m3.1b"><ci id="A1.p3.3.m3.1.1.cmml" xref="A1.p3.3.m3.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p3.3.m3.1c">b</annotation></semantics></math>. A detailed experimental analysis of different chunking strategies can be found in Appendix <a href="#A2" title="Appendix B Chunks ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a>. Similarly, for MoM-small, MoM-medium and MoM-large, we use the chunking strategies of <span id="A1.p3.3.4" class="ltx_text ltx_font_typewriter">[1-1-4-1-4-1]</span>, <span id="A1.p3.3.5" class="ltx_text ltx_font_typewriter">[1-1-1-4-1-4-1-4-1-4-1-1]</span>, and <span id="A1.p3.3.6" class="ltx_text ltx_font_typewriter">[1-4-1-4-1-4-1-4-1-4-1-4-1-4-1]</span>, respectively.</p>
</div>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Chunks</h2>

<figure id="A2.T4" class="ltx_table">
<table id="A2.T4.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A2.T4.1.1.1" class="ltx_tr">
<th id="A2.T4.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">Chunking Strategies</th>
<th id="A2.T4.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">MoM Config</th>
<th id="A2.T4.1.1.1.3" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">Val. Loss</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A2.T4.1.2.1" class="ltx_tr">
<td id="A2.T4.1.2.1.1" class="ltx_td ltx_align_left ltx_border_t"><span id="A2.T4.1.2.1.1.1" class="ltx_text ltx_font_typewriter">[1-1-4-1-1]</span></td>
<td id="A2.T4.1.2.1.2" class="ltx_td ltx_align_left ltx_border_t">K1H4S</td>
<td id="A2.T4.1.2.1.3" class="ltx_td ltx_align_left ltx_border_t">3.27</td>
</tr>
<tr id="A2.T4.1.3.2" class="ltx_tr">
<td id="A2.T4.1.3.2.1" class="ltx_td ltx_align_left"><span id="A2.T4.1.3.2.1.1" class="ltx_text ltx_font_typewriter">[1-1-4-1-1]</span></td>
<td id="A2.T4.1.3.2.2" class="ltx_td ltx_align_left">K2H4S</td>
<td id="A2.T4.1.3.2.3" class="ltx_td ltx_align_left">3.22</td>
</tr>
<tr id="A2.T4.1.4.3" class="ltx_tr">
<td id="A2.T4.1.4.3.1" class="ltx_td ltx_align_left"><span id="A2.T4.1.4.3.1.1" class="ltx_text ltx_font_typewriter">[1-6-1]</span></td>
<td id="A2.T4.1.4.3.2" class="ltx_td ltx_align_left">K1H6S</td>
<td id="A2.T4.1.4.3.3" class="ltx_td ltx_align_left">3.45</td>
</tr>
<tr id="A2.T4.1.5.4" class="ltx_tr">
<td id="A2.T4.1.5.4.1" class="ltx_td ltx_align_left"><span id="A2.T4.1.5.4.1.1" class="ltx_text ltx_font_typewriter">[1-6-1]</span></td>
<td id="A2.T4.1.5.4.2" class="ltx_td ltx_align_left">K2H6S</td>
<td id="A2.T4.1.5.4.3" class="ltx_td ltx_align_left">3.21</td>
</tr>
<tr id="A2.T4.1.6.5" class="ltx_tr">
<td id="A2.T4.1.6.5.1" class="ltx_td ltx_align_left"><span id="A2.T4.1.6.5.1.1" class="ltx_text ltx_font_typewriter">[8]</span></td>
<td id="A2.T4.1.6.5.2" class="ltx_td ltx_align_left">K1H8S</td>
<td id="A2.T4.1.6.5.3" class="ltx_td ltx_align_left">4.63</td>
</tr>
<tr id="A2.T4.1.7.6" class="ltx_tr">
<td id="A2.T4.1.7.6.1" class="ltx_td ltx_align_left"><span id="A2.T4.1.7.6.1.1" class="ltx_text ltx_font_typewriter">[8]</span></td>
<td id="A2.T4.1.7.6.2" class="ltx_td ltx_align_left">K2H8S</td>
<td id="A2.T4.1.7.6.3" class="ltx_td ltx_align_left">3.23</td>
</tr>
<tr id="A2.T4.1.8.7" class="ltx_tr">
<td id="A2.T4.1.8.7.1" class="ltx_td ltx_align_left"><span id="A2.T4.1.8.7.1.1" class="ltx_text ltx_font_typewriter">[4-4]</span></td>
<td id="A2.T4.1.8.7.2" class="ltx_td ltx_align_left">K1H4S</td>
<td id="A2.T4.1.8.7.3" class="ltx_td ltx_align_left">5.59</td>
</tr>
<tr id="A2.T4.1.9.8" class="ltx_tr">
<td id="A2.T4.1.9.8.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="A2.T4.1.9.8.1.1" class="ltx_text ltx_font_typewriter">[4-4]</span></td>
<td id="A2.T4.1.9.8.2" class="ltx_td ltx_align_left ltx_border_bb">K2H4S</td>
<td id="A2.T4.1.9.8.3" class="ltx_td ltx_align_left ltx_border_bb">3.22</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>Applying different chunking strategies on an 8-layer MoM. These models follow the same two-phase training procedure and the total training steps of the second phase is 5k.</figcaption>
</figure>
<div id="A2.p1" class="ltx_para ltx_noindent">
<p id="A2.p1.8" class="ltx_p">In this section, we study the impact of different chunking strategies on MoM performance. This experiment is conducted on an 8-layer MoM.
Except for <span id="A2.p1.8.1" class="ltx_text ltx_font_typewriter">[1-1-4-1-1]</span>, we include several alternatives: <span id="A2.p1.8.2" class="ltx_text ltx_font_typewriter">[4-4]</span> (two successive MoM blocks with <math id="A2.p1.1.m1.1" class="ltx_Math" alttext="N=4" display="inline"><semantics id="A2.p1.1.m1.1a"><mrow id="A2.p1.1.m1.1.1" xref="A2.p1.1.m1.1.1.cmml"><mi id="A2.p1.1.m1.1.1.2" xref="A2.p1.1.m1.1.1.2.cmml">N</mi><mo id="A2.p1.1.m1.1.1.1" xref="A2.p1.1.m1.1.1.1.cmml">=</mo><mn id="A2.p1.1.m1.1.1.3" xref="A2.p1.1.m1.1.1.3.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.p1.1.m1.1b"><apply id="A2.p1.1.m1.1.1.cmml" xref="A2.p1.1.m1.1.1"><eq id="A2.p1.1.m1.1.1.1.cmml" xref="A2.p1.1.m1.1.1.1"></eq><ci id="A2.p1.1.m1.1.1.2.cmml" xref="A2.p1.1.m1.1.1.2">𝑁</ci><cn type="integer" id="A2.p1.1.m1.1.1.3.cmml" xref="A2.p1.1.m1.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.1.m1.1c">N=4</annotation></semantics></math>), <span id="A2.p1.8.3" class="ltx_text ltx_font_typewriter">[1-6-1]</span> (the top and bottom one layer are kept unchanged, and modules in the middle 6 layers form an MoM with <math id="A2.p1.2.m2.1" class="ltx_Math" alttext="N=6" display="inline"><semantics id="A2.p1.2.m2.1a"><mrow id="A2.p1.2.m2.1.1" xref="A2.p1.2.m2.1.1.cmml"><mi id="A2.p1.2.m2.1.1.2" xref="A2.p1.2.m2.1.1.2.cmml">N</mi><mo id="A2.p1.2.m2.1.1.1" xref="A2.p1.2.m2.1.1.1.cmml">=</mo><mn id="A2.p1.2.m2.1.1.3" xref="A2.p1.2.m2.1.1.3.cmml">6</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.p1.2.m2.1b"><apply id="A2.p1.2.m2.1.1.cmml" xref="A2.p1.2.m2.1.1"><eq id="A2.p1.2.m2.1.1.1.cmml" xref="A2.p1.2.m2.1.1.1"></eq><ci id="A2.p1.2.m2.1.1.2.cmml" xref="A2.p1.2.m2.1.1.2">𝑁</ci><cn type="integer" id="A2.p1.2.m2.1.1.3.cmml" xref="A2.p1.2.m2.1.1.3">6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.2.m2.1c">N=6</annotation></semantics></math>), and <span id="A2.p1.8.4" class="ltx_text ltx_font_typewriter">[8]</span> (all modules form a big MoM with <math id="A2.p1.3.m3.1" class="ltx_Math" alttext="N=8" display="inline"><semantics id="A2.p1.3.m3.1a"><mrow id="A2.p1.3.m3.1.1" xref="A2.p1.3.m3.1.1.cmml"><mi id="A2.p1.3.m3.1.1.2" xref="A2.p1.3.m3.1.1.2.cmml">N</mi><mo id="A2.p1.3.m3.1.1.1" xref="A2.p1.3.m3.1.1.1.cmml">=</mo><mn id="A2.p1.3.m3.1.1.3" xref="A2.p1.3.m3.1.1.3.cmml">8</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.p1.3.m3.1b"><apply id="A2.p1.3.m3.1.1.cmml" xref="A2.p1.3.m3.1.1"><eq id="A2.p1.3.m3.1.1.1.cmml" xref="A2.p1.3.m3.1.1.1"></eq><ci id="A2.p1.3.m3.1.1.2.cmml" xref="A2.p1.3.m3.1.1.2">𝑁</ci><cn type="integer" id="A2.p1.3.m3.1.1.3.cmml" xref="A2.p1.3.m3.1.1.3">8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.3.m3.1c">N=8</annotation></semantics></math>).
Table <a href="#A2.T4" title="Table 4 ‣ Appendix B Chunks ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> shows the results of different chunking strategies. When <math id="A2.p1.4.m4.1" class="ltx_Math" alttext="K=1" display="inline"><semantics id="A2.p1.4.m4.1a"><mrow id="A2.p1.4.m4.1.1" xref="A2.p1.4.m4.1.1.cmml"><mi id="A2.p1.4.m4.1.1.2" xref="A2.p1.4.m4.1.1.2.cmml">K</mi><mo id="A2.p1.4.m4.1.1.1" xref="A2.p1.4.m4.1.1.1.cmml">=</mo><mn id="A2.p1.4.m4.1.1.3" xref="A2.p1.4.m4.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.p1.4.m4.1b"><apply id="A2.p1.4.m4.1.1.cmml" xref="A2.p1.4.m4.1.1"><eq id="A2.p1.4.m4.1.1.1.cmml" xref="A2.p1.4.m4.1.1.1"></eq><ci id="A2.p1.4.m4.1.1.2.cmml" xref="A2.p1.4.m4.1.1.2">𝐾</ci><cn type="integer" id="A2.p1.4.m4.1.1.3.cmml" xref="A2.p1.4.m4.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.4.m4.1c">K=1</annotation></semantics></math>, strategies other than <span id="A2.p1.8.5" class="ltx_text ltx_font_typewriter">[1-1-4-1-1]</span> exhibit unstable training curves and bad performance.
This is because the routers need to make multi-step decisions in the search space.
A larger search space (the increase of <math id="A2.p1.5.m5.1" class="ltx_Math" alttext="N" display="inline"><semantics id="A2.p1.5.m5.1a"><mi id="A2.p1.5.m5.1.1" xref="A2.p1.5.m5.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="A2.p1.5.m5.1b"><ci id="A2.p1.5.m5.1.1.cmml" xref="A2.p1.5.m5.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.5.m5.1c">N</annotation></semantics></math>) and more assembly steps (the increase of <math id="A2.p1.6.m6.1" class="ltx_Math" alttext="H" display="inline"><semantics id="A2.p1.6.m6.1a"><mi id="A2.p1.6.m6.1.1" xref="A2.p1.6.m6.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="A2.p1.6.m6.1b"><ci id="A2.p1.6.m6.1.1.cmml" xref="A2.p1.6.m6.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.6.m6.1c">H</annotation></semantics></math>) all lead to a harder task for the routers to find the correct path in the search space.
Things are much better when <math id="A2.p1.7.m7.1" class="ltx_Math" alttext="K=2" display="inline"><semantics id="A2.p1.7.m7.1a"><mrow id="A2.p1.7.m7.1.1" xref="A2.p1.7.m7.1.1.cmml"><mi id="A2.p1.7.m7.1.1.2" xref="A2.p1.7.m7.1.1.2.cmml">K</mi><mo id="A2.p1.7.m7.1.1.1" xref="A2.p1.7.m7.1.1.1.cmml">=</mo><mn id="A2.p1.7.m7.1.1.3" xref="A2.p1.7.m7.1.1.3.cmml">2</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.p1.7.m7.1b"><apply id="A2.p1.7.m7.1.1.cmml" xref="A2.p1.7.m7.1.1"><eq id="A2.p1.7.m7.1.1.1.cmml" xref="A2.p1.7.m7.1.1.1"></eq><ci id="A2.p1.7.m7.1.1.2.cmml" xref="A2.p1.7.m7.1.1.2">𝐾</ci><cn type="integer" id="A2.p1.7.m7.1.1.3.cmml" xref="A2.p1.7.m7.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.7.m7.1c">K=2</annotation></semantics></math>, where all strategies converge quite well, indicating that we can develop a full MoM architecture (i.e.,<span id="A2.p1.8.6" class="ltx_text ltx_font_typewriter">[8]</span>) without the chunking strategy. In practice, we still apply the chunking strategy (i.e., <span id="A2.p1.8.7" class="ltx_text ltx_font_typewriter">[1-1-4-1-1]</span>), because the strategy allows us to vary <math id="A2.p1.8.m8.1" class="ltx_Math" alttext="K" display="inline"><semantics id="A2.p1.8.m8.1a"><mi id="A2.p1.8.m8.1.1" xref="A2.p1.8.m8.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="A2.p1.8.m8.1b"><ci id="A2.p1.8.m8.1.1.cmml" xref="A2.p1.8.m8.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.8.m8.1c">K</annotation></semantics></math> in a larger range, and thus we can study MoM with more configurations.
The experiment demonstrates the necessity of manually restricting the search space of MoM so that the decision-making burden for the routers would be relieved.</p>
</div>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Detailed downstream evaluation results</h2>

<div id="A3.p1" class="ltx_para ltx_noindent">
<p id="A3.p1.1" class="ltx_p">Table <a href="#A3.T5" title="Table 5 ‣ Appendix C Detailed downstream evaluation results ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> presents the evaluation results for each sub-task of GLUE across different models and Table <a href="#A3.T6" title="Table 6 ‣ Appendix C Detailed downstream evaluation results ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> presents the results on XSUM with respect to other ROUGE metrics.</p>
</div>
<figure id="A3.T5" class="ltx_table">
<div id="A3.T5.9" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:420.7pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-15.1pt,14.7pt) scale(0.934804183600798,0.934804183600798) ;">
<table id="A3.T5.9.9" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A3.T5.9.9.10.1" class="ltx_tr">
<th id="A3.T5.9.9.10.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Method</th>
<th id="A3.T5.9.9.10.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">SST-2</th>
<th id="A3.T5.9.9.10.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">COLA</th>
<th id="A3.T5.9.9.10.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">MRPC</th>
<th id="A3.T5.9.9.10.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">QQP</th>
<th id="A3.T5.9.9.10.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">QNLI</th>
<th id="A3.T5.9.9.10.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">RTE</th>
<th id="A3.T5.9.9.10.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">MNLI-(m/mm)</th>
<th id="A3.T5.9.9.10.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">STS-B</th>
<th id="A3.T5.9.9.10.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">average</th>
</tr>
<tr id="A3.T5.9.9.11.2" class="ltx_tr">
<th id="A3.T5.9.9.11.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt" colspan="10"><span id="A3.T5.9.9.11.2.1.1" class="ltx_text ltx_font_italic">small</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A3.T5.9.9.12.1" class="ltx_tr">
<th id="A3.T5.9.9.12.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">GPT2</th>
<td id="A3.T5.9.9.12.1.2" class="ltx_td ltx_align_center ltx_border_t">92.09</td>
<td id="A3.T5.9.9.12.1.3" class="ltx_td ltx_align_center ltx_border_t">26.27</td>
<td id="A3.T5.9.9.12.1.4" class="ltx_td ltx_align_center ltx_border_t">85.11</td>
<td id="A3.T5.9.9.12.1.5" class="ltx_td ltx_align_center ltx_border_t">83.09</td>
<td id="A3.T5.9.9.12.1.6" class="ltx_td ltx_align_center ltx_border_t">87.55</td>
<td id="A3.T5.9.9.12.1.7" class="ltx_td ltx_align_center ltx_border_t">62.45</td>
<td id="A3.T5.9.9.12.1.8" class="ltx_td ltx_align_center ltx_border_t">79.90/78.74</td>
<td id="A3.T5.9.9.12.1.9" class="ltx_td ltx_align_center ltx_border_t">82.67</td>
<td id="A3.T5.9.9.12.1.10" class="ltx_td ltx_align_center ltx_border_t">75.32</td>
</tr>
<tr id="A3.T5.9.9.13.2" class="ltx_tr">
<th id="A3.T5.9.9.13.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MoD</th>
<td id="A3.T5.9.9.13.2.2" class="ltx_td ltx_align_center">87.73</td>
<td id="A3.T5.9.9.13.2.3" class="ltx_td ltx_align_center">21.66</td>
<td id="A3.T5.9.9.13.2.4" class="ltx_td ltx_align_center">81.43</td>
<td id="A3.T5.9.9.13.2.5" class="ltx_td ltx_align_center">82.01</td>
<td id="A3.T5.9.9.13.2.6" class="ltx_td ltx_align_center">81.79</td>
<td id="A3.T5.9.9.13.2.7" class="ltx_td ltx_align_center">64.62</td>
<td id="A3.T5.9.9.13.2.8" class="ltx_td ltx_align_center">75.96/75.22</td>
<td id="A3.T5.9.9.13.2.9" class="ltx_td ltx_align_center">79.76</td>
<td id="A3.T5.9.9.13.2.10" class="ltx_td ltx_align_center">72.24</td>
</tr>
<tr id="A3.T5.9.9.14.3" class="ltx_tr">
<th id="A3.T5.9.9.14.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MoE</th>
<td id="A3.T5.9.9.14.3.2" class="ltx_td ltx_align_center">89.68</td>
<td id="A3.T5.9.9.14.3.3" class="ltx_td ltx_align_center">45.87</td>
<td id="A3.T5.9.9.14.3.4" class="ltx_td ltx_align_center">82.05</td>
<td id="A3.T5.9.9.14.3.5" class="ltx_td ltx_align_center">84.09</td>
<td id="A3.T5.9.9.14.3.6" class="ltx_td ltx_align_center">85.94</td>
<td id="A3.T5.9.9.14.3.7" class="ltx_td ltx_align_center">65.52</td>
<td id="A3.T5.9.9.14.3.8" class="ltx_td ltx_align_center">79.46/78.74</td>
<td id="A3.T5.9.9.14.3.9" class="ltx_td ltx_align_center">83.87</td>
<td id="A3.T5.9.9.14.3.10" class="ltx_td ltx_align_center">77.25</td>
</tr>
<tr id="A3.T5.9.9.15.4" class="ltx_tr">
<th id="A3.T5.9.9.15.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MoE (share)</th>
<td id="A3.T5.9.9.15.4.2" class="ltx_td ltx_align_center">89.53</td>
<td id="A3.T5.9.9.15.4.3" class="ltx_td ltx_align_center">37.29</td>
<td id="A3.T5.9.9.15.4.4" class="ltx_td ltx_align_center">82.01</td>
<td id="A3.T5.9.9.15.4.5" class="ltx_td ltx_align_center">83.49</td>
<td id="A3.T5.9.9.15.4.6" class="ltx_td ltx_align_center">85.21</td>
<td id="A3.T5.9.9.15.4.7" class="ltx_td ltx_align_center">64.94</td>
<td id="A3.T5.9.9.15.4.8" class="ltx_td ltx_align_center">78.49/77.99</td>
<td id="A3.T5.9.9.15.4.9" class="ltx_td ltx_align_center">83.43</td>
<td id="A3.T5.9.9.15.4.10" class="ltx_td ltx_align_center">75.82</td>
</tr>
<tr id="A3.T5.1.1.1" class="ltx_tr">
<th id="A3.T5.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><math id="A3.T5.1.1.1.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{E}}" display="inline"><semantics id="A3.T5.1.1.1.1.m1.1a"><msub id="A3.T5.1.1.1.1.m1.1.1" xref="A3.T5.1.1.1.1.m1.1.1.cmml"><mtext id="A3.T5.1.1.1.1.m1.1.1.2" xref="A3.T5.1.1.1.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="A3.T5.1.1.1.1.m1.1.1.3" xref="A3.T5.1.1.1.1.m1.1.1.3a.cmml">E</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.T5.1.1.1.1.m1.1b"><apply id="A3.T5.1.1.1.1.m1.1.1.cmml" xref="A3.T5.1.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="A3.T5.1.1.1.1.m1.1.1.1.cmml" xref="A3.T5.1.1.1.1.m1.1.1">subscript</csymbol><ci id="A3.T5.1.1.1.1.m1.1.1.2a.cmml" xref="A3.T5.1.1.1.1.m1.1.1.2"><mtext id="A3.T5.1.1.1.1.m1.1.1.2.cmml" xref="A3.T5.1.1.1.1.m1.1.1.2">MoM</mtext></ci><ci id="A3.T5.1.1.1.1.m1.1.1.3a.cmml" xref="A3.T5.1.1.1.1.m1.1.1.3"><mtext mathsize="70%" id="A3.T5.1.1.1.1.m1.1.1.3.cmml" xref="A3.T5.1.1.1.1.m1.1.1.3">E</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.T5.1.1.1.1.m1.1c">\text{MoM}_{\text{E}}</annotation></semantics></math></th>
<td id="A3.T5.1.1.1.2" class="ltx_td ltx_align_center">90.83</td>
<td id="A3.T5.1.1.1.3" class="ltx_td ltx_align_center">31.83</td>
<td id="A3.T5.1.1.1.4" class="ltx_td ltx_align_center">82.92</td>
<td id="A3.T5.1.1.1.5" class="ltx_td ltx_align_center">83.49</td>
<td id="A3.T5.1.1.1.6" class="ltx_td ltx_align_center">84.79</td>
<td id="A3.T5.1.1.1.7" class="ltx_td ltx_align_center">69.68</td>
<td id="A3.T5.1.1.1.8" class="ltx_td ltx_align_center">78.52/77.40</td>
<td id="A3.T5.1.1.1.9" class="ltx_td ltx_align_center">83.84</td>
<td id="A3.T5.1.1.1.10" class="ltx_td ltx_align_center">75.92</td>
</tr>
<tr id="A3.T5.2.2.2" class="ltx_tr">
<th id="A3.T5.2.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><math id="A3.T5.2.2.2.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{I}}" display="inline"><semantics id="A3.T5.2.2.2.1.m1.1a"><msub id="A3.T5.2.2.2.1.m1.1.1" xref="A3.T5.2.2.2.1.m1.1.1.cmml"><mtext id="A3.T5.2.2.2.1.m1.1.1.2" xref="A3.T5.2.2.2.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="A3.T5.2.2.2.1.m1.1.1.3" xref="A3.T5.2.2.2.1.m1.1.1.3a.cmml">I</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.T5.2.2.2.1.m1.1b"><apply id="A3.T5.2.2.2.1.m1.1.1.cmml" xref="A3.T5.2.2.2.1.m1.1.1"><csymbol cd="ambiguous" id="A3.T5.2.2.2.1.m1.1.1.1.cmml" xref="A3.T5.2.2.2.1.m1.1.1">subscript</csymbol><ci id="A3.T5.2.2.2.1.m1.1.1.2a.cmml" xref="A3.T5.2.2.2.1.m1.1.1.2"><mtext id="A3.T5.2.2.2.1.m1.1.1.2.cmml" xref="A3.T5.2.2.2.1.m1.1.1.2">MoM</mtext></ci><ci id="A3.T5.2.2.2.1.m1.1.1.3a.cmml" xref="A3.T5.2.2.2.1.m1.1.1.3"><mtext mathsize="70%" id="A3.T5.2.2.2.1.m1.1.1.3.cmml" xref="A3.T5.2.2.2.1.m1.1.1.3">I</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.T5.2.2.2.1.m1.1c">\text{MoM}_{\text{I}}</annotation></semantics></math></th>
<td id="A3.T5.2.2.2.2" class="ltx_td ltx_align_center">90.02</td>
<td id="A3.T5.2.2.2.3" class="ltx_td ltx_align_center">47.21</td>
<td id="A3.T5.2.2.2.4" class="ltx_td ltx_align_center">82.90</td>
<td id="A3.T5.2.2.2.5" class="ltx_td ltx_align_center">84.07</td>
<td id="A3.T5.2.2.2.6" class="ltx_td ltx_align_center">85.34</td>
<td id="A3.T5.2.2.2.7" class="ltx_td ltx_align_center">66.43</td>
<td id="A3.T5.2.2.2.8" class="ltx_td ltx_align_center">79.82/79.72</td>
<td id="A3.T5.2.2.2.9" class="ltx_td ltx_align_center">84.79</td>
<td id="A3.T5.2.2.2.10" class="ltx_td ltx_align_center">77.81</td>
</tr>
<tr id="A3.T5.3.3.3" class="ltx_tr">
<th id="A3.T5.3.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><math id="A3.T5.3.3.3.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{P}}" display="inline"><semantics id="A3.T5.3.3.3.1.m1.1a"><msub id="A3.T5.3.3.3.1.m1.1.1" xref="A3.T5.3.3.3.1.m1.1.1.cmml"><mtext id="A3.T5.3.3.3.1.m1.1.1.2" xref="A3.T5.3.3.3.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="A3.T5.3.3.3.1.m1.1.1.3" xref="A3.T5.3.3.3.1.m1.1.1.3a.cmml">P</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.T5.3.3.3.1.m1.1b"><apply id="A3.T5.3.3.3.1.m1.1.1.cmml" xref="A3.T5.3.3.3.1.m1.1.1"><csymbol cd="ambiguous" id="A3.T5.3.3.3.1.m1.1.1.1.cmml" xref="A3.T5.3.3.3.1.m1.1.1">subscript</csymbol><ci id="A3.T5.3.3.3.1.m1.1.1.2a.cmml" xref="A3.T5.3.3.3.1.m1.1.1.2"><mtext id="A3.T5.3.3.3.1.m1.1.1.2.cmml" xref="A3.T5.3.3.3.1.m1.1.1.2">MoM</mtext></ci><ci id="A3.T5.3.3.3.1.m1.1.1.3a.cmml" xref="A3.T5.3.3.3.1.m1.1.1.3"><mtext mathsize="70%" id="A3.T5.3.3.3.1.m1.1.1.3.cmml" xref="A3.T5.3.3.3.1.m1.1.1.3">P</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.T5.3.3.3.1.m1.1c">\text{MoM}_{\text{P}}</annotation></semantics></math></th>
<td id="A3.T5.3.3.3.2" class="ltx_td ltx_align_center">91.40</td>
<td id="A3.T5.3.3.3.3" class="ltx_td ltx_align_center">46.16</td>
<td id="A3.T5.3.3.3.4" class="ltx_td ltx_align_center">82.68</td>
<td id="A3.T5.3.3.3.5" class="ltx_td ltx_align_center">84.62</td>
<td id="A3.T5.3.3.3.6" class="ltx_td ltx_align_center">86.49</td>
<td id="A3.T5.3.3.3.7" class="ltx_td ltx_align_center">67.51</td>
<td id="A3.T5.3.3.3.8" class="ltx_td ltx_align_center">80.85/80.18</td>
<td id="A3.T5.3.3.3.9" class="ltx_td ltx_align_center">84.06</td>
<td id="A3.T5.3.3.3.10" class="ltx_td ltx_align_center">78.22</td>
</tr>
<tr id="A3.T5.9.9.16.5" class="ltx_tr">
<th id="A3.T5.9.9.16.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" colspan="10"><span id="A3.T5.9.9.16.5.1.1" class="ltx_text ltx_font_italic">medium</span></th>
</tr>
<tr id="A3.T5.9.9.17.6" class="ltx_tr">
<th id="A3.T5.9.9.17.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">GPT2</th>
<td id="A3.T5.9.9.17.6.2" class="ltx_td ltx_align_center ltx_border_t">94.15</td>
<td id="A3.T5.9.9.17.6.3" class="ltx_td ltx_align_center ltx_border_t">48.18</td>
<td id="A3.T5.9.9.17.6.4" class="ltx_td ltx_align_center ltx_border_t">86.00</td>
<td id="A3.T5.9.9.17.6.5" class="ltx_td ltx_align_center ltx_border_t">85.94</td>
<td id="A3.T5.9.9.17.6.6" class="ltx_td ltx_align_center ltx_border_t">90.41</td>
<td id="A3.T5.9.9.17.6.7" class="ltx_td ltx_align_center ltx_border_t">64.98</td>
<td id="A3.T5.9.9.17.6.8" class="ltx_td ltx_align_center ltx_border_t">84.02/83.92</td>
<td id="A3.T5.9.9.17.6.9" class="ltx_td ltx_align_center ltx_border_t">86.77</td>
<td id="A3.T5.9.9.17.6.10" class="ltx_td ltx_align_center ltx_border_t">80.49</td>
</tr>
<tr id="A3.T5.9.9.18.7" class="ltx_tr">
<th id="A3.T5.9.9.18.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MoD</th>
<td id="A3.T5.9.9.18.7.2" class="ltx_td ltx_align_center">89.45</td>
<td id="A3.T5.9.9.18.7.3" class="ltx_td ltx_align_center">37.43</td>
<td id="A3.T5.9.9.18.7.4" class="ltx_td ltx_align_center">84.97</td>
<td id="A3.T5.9.9.18.7.5" class="ltx_td ltx_align_center">84.21</td>
<td id="A3.T5.9.9.18.7.6" class="ltx_td ltx_align_center">84.88</td>
<td id="A3.T5.9.9.18.7.7" class="ltx_td ltx_align_center">64.26</td>
<td id="A3.T5.9.9.18.7.8" class="ltx_td ltx_align_center">78.94/77.94</td>
<td id="A3.T5.9.9.18.7.9" class="ltx_td ltx_align_center">83.47</td>
<td id="A3.T5.9.9.18.7.10" class="ltx_td ltx_align_center">76.17</td>
</tr>
<tr id="A3.T5.9.9.19.8" class="ltx_tr">
<th id="A3.T5.9.9.19.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MoE</th>
<td id="A3.T5.9.9.19.8.2" class="ltx_td ltx_align_center">91.86</td>
<td id="A3.T5.9.9.19.8.3" class="ltx_td ltx_align_center">49.20</td>
<td id="A3.T5.9.9.19.8.4" class="ltx_td ltx_align_center">85.51</td>
<td id="A3.T5.9.9.19.8.5" class="ltx_td ltx_align_center">86.39</td>
<td id="A3.T5.9.9.19.8.6" class="ltx_td ltx_align_center">89.09</td>
<td id="A3.T5.9.9.19.8.7" class="ltx_td ltx_align_center">68.12</td>
<td id="A3.T5.9.9.19.8.8" class="ltx_td ltx_align_center">84.44/83.49</td>
<td id="A3.T5.9.9.19.8.9" class="ltx_td ltx_align_center">86.17</td>
<td id="A3.T5.9.9.19.8.10" class="ltx_td ltx_align_center">80.47</td>
</tr>
<tr id="A3.T5.9.9.20.9" class="ltx_tr">
<th id="A3.T5.9.9.20.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MoE (share)</th>
<td id="A3.T5.9.9.20.9.2" class="ltx_td ltx_align_center">92.78</td>
<td id="A3.T5.9.9.20.9.3" class="ltx_td ltx_align_center">49.44</td>
<td id="A3.T5.9.9.20.9.4" class="ltx_td ltx_align_center">84.56</td>
<td id="A3.T5.9.9.20.9.5" class="ltx_td ltx_align_center">86.40</td>
<td id="A3.T5.9.9.20.9.6" class="ltx_td ltx_align_center">89.35</td>
<td id="A3.T5.9.9.20.9.7" class="ltx_td ltx_align_center">66.79</td>
<td id="A3.T5.9.9.20.9.8" class="ltx_td ltx_align_center">84.00/83.39</td>
<td id="A3.T5.9.9.20.9.9" class="ltx_td ltx_align_center">86.42</td>
<td id="A3.T5.9.9.20.9.10" class="ltx_td ltx_align_center">80.35</td>
</tr>
<tr id="A3.T5.4.4.4" class="ltx_tr">
<th id="A3.T5.4.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><math id="A3.T5.4.4.4.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{E}}" display="inline"><semantics id="A3.T5.4.4.4.1.m1.1a"><msub id="A3.T5.4.4.4.1.m1.1.1" xref="A3.T5.4.4.4.1.m1.1.1.cmml"><mtext id="A3.T5.4.4.4.1.m1.1.1.2" xref="A3.T5.4.4.4.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="A3.T5.4.4.4.1.m1.1.1.3" xref="A3.T5.4.4.4.1.m1.1.1.3a.cmml">E</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.T5.4.4.4.1.m1.1b"><apply id="A3.T5.4.4.4.1.m1.1.1.cmml" xref="A3.T5.4.4.4.1.m1.1.1"><csymbol cd="ambiguous" id="A3.T5.4.4.4.1.m1.1.1.1.cmml" xref="A3.T5.4.4.4.1.m1.1.1">subscript</csymbol><ci id="A3.T5.4.4.4.1.m1.1.1.2a.cmml" xref="A3.T5.4.4.4.1.m1.1.1.2"><mtext id="A3.T5.4.4.4.1.m1.1.1.2.cmml" xref="A3.T5.4.4.4.1.m1.1.1.2">MoM</mtext></ci><ci id="A3.T5.4.4.4.1.m1.1.1.3a.cmml" xref="A3.T5.4.4.4.1.m1.1.1.3"><mtext mathsize="70%" id="A3.T5.4.4.4.1.m1.1.1.3.cmml" xref="A3.T5.4.4.4.1.m1.1.1.3">E</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.T5.4.4.4.1.m1.1c">\text{MoM}_{\text{E}}</annotation></semantics></math></th>
<td id="A3.T5.4.4.4.2" class="ltx_td ltx_align_center">91.40</td>
<td id="A3.T5.4.4.4.3" class="ltx_td ltx_align_center">48.48</td>
<td id="A3.T5.4.4.4.4" class="ltx_td ltx_align_center">87.48</td>
<td id="A3.T5.4.4.4.5" class="ltx_td ltx_align_center">86.92</td>
<td id="A3.T5.4.4.4.6" class="ltx_td ltx_align_center">89.11</td>
<td id="A3.T5.4.4.4.7" class="ltx_td ltx_align_center">67.51</td>
<td id="A3.T5.4.4.4.8" class="ltx_td ltx_align_center">83.73/83.01</td>
<td id="A3.T5.4.4.4.9" class="ltx_td ltx_align_center">86.06</td>
<td id="A3.T5.4.4.4.10" class="ltx_td ltx_align_center">80.41</td>
</tr>
<tr id="A3.T5.5.5.5" class="ltx_tr">
<th id="A3.T5.5.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><math id="A3.T5.5.5.5.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{I}}" display="inline"><semantics id="A3.T5.5.5.5.1.m1.1a"><msub id="A3.T5.5.5.5.1.m1.1.1" xref="A3.T5.5.5.5.1.m1.1.1.cmml"><mtext id="A3.T5.5.5.5.1.m1.1.1.2" xref="A3.T5.5.5.5.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="A3.T5.5.5.5.1.m1.1.1.3" xref="A3.T5.5.5.5.1.m1.1.1.3a.cmml">I</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.T5.5.5.5.1.m1.1b"><apply id="A3.T5.5.5.5.1.m1.1.1.cmml" xref="A3.T5.5.5.5.1.m1.1.1"><csymbol cd="ambiguous" id="A3.T5.5.5.5.1.m1.1.1.1.cmml" xref="A3.T5.5.5.5.1.m1.1.1">subscript</csymbol><ci id="A3.T5.5.5.5.1.m1.1.1.2a.cmml" xref="A3.T5.5.5.5.1.m1.1.1.2"><mtext id="A3.T5.5.5.5.1.m1.1.1.2.cmml" xref="A3.T5.5.5.5.1.m1.1.1.2">MoM</mtext></ci><ci id="A3.T5.5.5.5.1.m1.1.1.3a.cmml" xref="A3.T5.5.5.5.1.m1.1.1.3"><mtext mathsize="70%" id="A3.T5.5.5.5.1.m1.1.1.3.cmml" xref="A3.T5.5.5.5.1.m1.1.1.3">I</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.T5.5.5.5.1.m1.1c">\text{MoM}_{\text{I}}</annotation></semantics></math></th>
<td id="A3.T5.5.5.5.2" class="ltx_td ltx_align_center">92.46</td>
<td id="A3.T5.5.5.5.3" class="ltx_td ltx_align_center">50.19</td>
<td id="A3.T5.5.5.5.4" class="ltx_td ltx_align_center">87.31</td>
<td id="A3.T5.5.5.5.5" class="ltx_td ltx_align_center">86.43</td>
<td id="A3.T5.5.5.5.6" class="ltx_td ltx_align_center">89.35</td>
<td id="A3.T5.5.5.5.7" class="ltx_td ltx_align_center">68.94</td>
<td id="A3.T5.5.5.5.8" class="ltx_td ltx_align_center">84.44/83.46</td>
<td id="A3.T5.5.5.5.9" class="ltx_td ltx_align_center">86.66</td>
<td id="A3.T5.5.5.5.10" class="ltx_td ltx_align_center">81.03</td>
</tr>
<tr id="A3.T5.6.6.6" class="ltx_tr">
<th id="A3.T5.6.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><math id="A3.T5.6.6.6.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{P}}" display="inline"><semantics id="A3.T5.6.6.6.1.m1.1a"><msub id="A3.T5.6.6.6.1.m1.1.1" xref="A3.T5.6.6.6.1.m1.1.1.cmml"><mtext id="A3.T5.6.6.6.1.m1.1.1.2" xref="A3.T5.6.6.6.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="A3.T5.6.6.6.1.m1.1.1.3" xref="A3.T5.6.6.6.1.m1.1.1.3a.cmml">P</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.T5.6.6.6.1.m1.1b"><apply id="A3.T5.6.6.6.1.m1.1.1.cmml" xref="A3.T5.6.6.6.1.m1.1.1"><csymbol cd="ambiguous" id="A3.T5.6.6.6.1.m1.1.1.1.cmml" xref="A3.T5.6.6.6.1.m1.1.1">subscript</csymbol><ci id="A3.T5.6.6.6.1.m1.1.1.2a.cmml" xref="A3.T5.6.6.6.1.m1.1.1.2"><mtext id="A3.T5.6.6.6.1.m1.1.1.2.cmml" xref="A3.T5.6.6.6.1.m1.1.1.2">MoM</mtext></ci><ci id="A3.T5.6.6.6.1.m1.1.1.3a.cmml" xref="A3.T5.6.6.6.1.m1.1.1.3"><mtext mathsize="70%" id="A3.T5.6.6.6.1.m1.1.1.3.cmml" xref="A3.T5.6.6.6.1.m1.1.1.3">P</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.T5.6.6.6.1.m1.1c">\text{MoM}_{\text{P}}</annotation></semantics></math></th>
<td id="A3.T5.6.6.6.2" class="ltx_td ltx_align_center">92.88</td>
<td id="A3.T5.6.6.6.3" class="ltx_td ltx_align_center">53.61</td>
<td id="A3.T5.6.6.6.4" class="ltx_td ltx_align_center">87.64</td>
<td id="A3.T5.6.6.6.5" class="ltx_td ltx_align_center">86.64</td>
<td id="A3.T5.6.6.6.6" class="ltx_td ltx_align_center">89.75</td>
<td id="A3.T5.6.6.6.7" class="ltx_td ltx_align_center">71.06</td>
<td id="A3.T5.6.6.6.8" class="ltx_td ltx_align_center">84.69/83.98</td>
<td id="A3.T5.6.6.6.9" class="ltx_td ltx_align_center">87.14</td>
<td id="A3.T5.6.6.6.10" class="ltx_td ltx_align_center">81.93</td>
</tr>
<tr id="A3.T5.9.9.21.10" class="ltx_tr">
<th id="A3.T5.9.9.21.10.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" colspan="10"><span id="A3.T5.9.9.21.10.1.1" class="ltx_text ltx_font_italic">large</span></th>
</tr>
<tr id="A3.T5.9.9.22.11" class="ltx_tr">
<th id="A3.T5.9.9.22.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">GPT2</th>
<td id="A3.T5.9.9.22.11.2" class="ltx_td ltx_align_center ltx_border_t">94.15</td>
<td id="A3.T5.9.9.22.11.3" class="ltx_td ltx_align_center ltx_border_t">60.04</td>
<td id="A3.T5.9.9.22.11.4" class="ltx_td ltx_align_center ltx_border_t">88.74</td>
<td id="A3.T5.9.9.22.11.5" class="ltx_td ltx_align_center ltx_border_t">87.88</td>
<td id="A3.T5.9.9.22.11.6" class="ltx_td ltx_align_center ltx_border_t">91.89</td>
<td id="A3.T5.9.9.22.11.7" class="ltx_td ltx_align_center ltx_border_t">75.45</td>
<td id="A3.T5.9.9.22.11.8" class="ltx_td ltx_align_center ltx_border_t">86.80/85.98</td>
<td id="A3.T5.9.9.22.11.9" class="ltx_td ltx_align_center ltx_border_t">89.30</td>
<td id="A3.T5.9.9.22.11.10" class="ltx_td ltx_align_center ltx_border_t">84.47</td>
</tr>
<tr id="A3.T5.9.9.23.12" class="ltx_tr">
<th id="A3.T5.9.9.23.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MoD</th>
<td id="A3.T5.9.9.23.12.2" class="ltx_td ltx_align_center">91.51</td>
<td id="A3.T5.9.9.23.12.3" class="ltx_td ltx_align_center">52.86</td>
<td id="A3.T5.9.9.23.12.4" class="ltx_td ltx_align_center">87.29</td>
<td id="A3.T5.9.9.23.12.5" class="ltx_td ltx_align_center">87.20</td>
<td id="A3.T5.9.9.23.12.6" class="ltx_td ltx_align_center">89.57</td>
<td id="A3.T5.9.9.23.12.7" class="ltx_td ltx_align_center">67.87</td>
<td id="A3.T5.9.9.23.12.8" class="ltx_td ltx_align_center">85.07/84.38</td>
<td id="A3.T5.9.9.23.12.9" class="ltx_td ltx_align_center">87.66</td>
<td id="A3.T5.9.9.23.12.10" class="ltx_td ltx_align_center">81.49</td>
</tr>
<tr id="A3.T5.9.9.24.13" class="ltx_tr">
<th id="A3.T5.9.9.24.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MoE</th>
<td id="A3.T5.9.9.24.13.2" class="ltx_td ltx_align_center">94.32</td>
<td id="A3.T5.9.9.24.13.3" class="ltx_td ltx_align_center">61.19</td>
<td id="A3.T5.9.9.24.13.4" class="ltx_td ltx_align_center">88.54</td>
<td id="A3.T5.9.9.24.13.5" class="ltx_td ltx_align_center">88.36</td>
<td id="A3.T5.9.9.24.13.6" class="ltx_td ltx_align_center">92.04</td>
<td id="A3.T5.9.9.24.13.7" class="ltx_td ltx_align_center">71.68</td>
<td id="A3.T5.9.9.24.13.8" class="ltx_td ltx_align_center">87.38/86.94</td>
<td id="A3.T5.9.9.24.13.9" class="ltx_td ltx_align_center">89.45</td>
<td id="A3.T5.9.9.24.13.10" class="ltx_td ltx_align_center">84.43</td>
</tr>
<tr id="A3.T5.9.9.25.14" class="ltx_tr">
<th id="A3.T5.9.9.25.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MoE (share)</th>
<td id="A3.T5.9.9.25.14.2" class="ltx_td ltx_align_center">93.88</td>
<td id="A3.T5.9.9.25.14.3" class="ltx_td ltx_align_center">61.43</td>
<td id="A3.T5.9.9.25.14.4" class="ltx_td ltx_align_center">88.12</td>
<td id="A3.T5.9.9.25.14.5" class="ltx_td ltx_align_center">88.17</td>
<td id="A3.T5.9.9.25.14.6" class="ltx_td ltx_align_center">91.56</td>
<td id="A3.T5.9.9.25.14.7" class="ltx_td ltx_align_center">68.31</td>
<td id="A3.T5.9.9.25.14.8" class="ltx_td ltx_align_center">87.01/86.73</td>
<td id="A3.T5.9.9.25.14.9" class="ltx_td ltx_align_center">89.28</td>
<td id="A3.T5.9.9.25.14.10" class="ltx_td ltx_align_center">83.83</td>
</tr>
<tr id="A3.T5.7.7.7" class="ltx_tr">
<th id="A3.T5.7.7.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><math id="A3.T5.7.7.7.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{E}}" display="inline"><semantics id="A3.T5.7.7.7.1.m1.1a"><msub id="A3.T5.7.7.7.1.m1.1.1" xref="A3.T5.7.7.7.1.m1.1.1.cmml"><mtext id="A3.T5.7.7.7.1.m1.1.1.2" xref="A3.T5.7.7.7.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="A3.T5.7.7.7.1.m1.1.1.3" xref="A3.T5.7.7.7.1.m1.1.1.3a.cmml">E</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.T5.7.7.7.1.m1.1b"><apply id="A3.T5.7.7.7.1.m1.1.1.cmml" xref="A3.T5.7.7.7.1.m1.1.1"><csymbol cd="ambiguous" id="A3.T5.7.7.7.1.m1.1.1.1.cmml" xref="A3.T5.7.7.7.1.m1.1.1">subscript</csymbol><ci id="A3.T5.7.7.7.1.m1.1.1.2a.cmml" xref="A3.T5.7.7.7.1.m1.1.1.2"><mtext id="A3.T5.7.7.7.1.m1.1.1.2.cmml" xref="A3.T5.7.7.7.1.m1.1.1.2">MoM</mtext></ci><ci id="A3.T5.7.7.7.1.m1.1.1.3a.cmml" xref="A3.T5.7.7.7.1.m1.1.1.3"><mtext mathsize="70%" id="A3.T5.7.7.7.1.m1.1.1.3.cmml" xref="A3.T5.7.7.7.1.m1.1.1.3">E</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.T5.7.7.7.1.m1.1c">\text{MoM}_{\text{E}}</annotation></semantics></math></th>
<td id="A3.T5.7.7.7.2" class="ltx_td ltx_align_center">93.64</td>
<td id="A3.T5.7.7.7.3" class="ltx_td ltx_align_center">59.26</td>
<td id="A3.T5.7.7.7.4" class="ltx_td ltx_align_center">88.25</td>
<td id="A3.T5.7.7.7.5" class="ltx_td ltx_align_center">87.59</td>
<td id="A3.T5.7.7.7.6" class="ltx_td ltx_align_center">91.29</td>
<td id="A3.T5.7.7.7.7" class="ltx_td ltx_align_center">72.23</td>
<td id="A3.T5.7.7.7.8" class="ltx_td ltx_align_center">85.20/84.78</td>
<td id="A3.T5.7.7.7.9" class="ltx_td ltx_align_center">88.26</td>
<td id="A3.T5.7.7.7.10" class="ltx_td ltx_align_center">83.39</td>
</tr>
<tr id="A3.T5.8.8.8" class="ltx_tr">
<th id="A3.T5.8.8.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><math id="A3.T5.8.8.8.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{I}}" display="inline"><semantics id="A3.T5.8.8.8.1.m1.1a"><msub id="A3.T5.8.8.8.1.m1.1.1" xref="A3.T5.8.8.8.1.m1.1.1.cmml"><mtext id="A3.T5.8.8.8.1.m1.1.1.2" xref="A3.T5.8.8.8.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="A3.T5.8.8.8.1.m1.1.1.3" xref="A3.T5.8.8.8.1.m1.1.1.3a.cmml">I</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.T5.8.8.8.1.m1.1b"><apply id="A3.T5.8.8.8.1.m1.1.1.cmml" xref="A3.T5.8.8.8.1.m1.1.1"><csymbol cd="ambiguous" id="A3.T5.8.8.8.1.m1.1.1.1.cmml" xref="A3.T5.8.8.8.1.m1.1.1">subscript</csymbol><ci id="A3.T5.8.8.8.1.m1.1.1.2a.cmml" xref="A3.T5.8.8.8.1.m1.1.1.2"><mtext id="A3.T5.8.8.8.1.m1.1.1.2.cmml" xref="A3.T5.8.8.8.1.m1.1.1.2">MoM</mtext></ci><ci id="A3.T5.8.8.8.1.m1.1.1.3a.cmml" xref="A3.T5.8.8.8.1.m1.1.1.3"><mtext mathsize="70%" id="A3.T5.8.8.8.1.m1.1.1.3.cmml" xref="A3.T5.8.8.8.1.m1.1.1.3">I</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.T5.8.8.8.1.m1.1c">\text{MoM}_{\text{I}}</annotation></semantics></math></th>
<td id="A3.T5.8.8.8.2" class="ltx_td ltx_align_center">93.69</td>
<td id="A3.T5.8.8.8.3" class="ltx_td ltx_align_center">62.25</td>
<td id="A3.T5.8.8.8.4" class="ltx_td ltx_align_center">89.19</td>
<td id="A3.T5.8.8.8.5" class="ltx_td ltx_align_center">88.12</td>
<td id="A3.T5.8.8.8.6" class="ltx_td ltx_align_center">92.36</td>
<td id="A3.T5.8.8.8.7" class="ltx_td ltx_align_center">74.98</td>
<td id="A3.T5.8.8.8.8" class="ltx_td ltx_align_center">87.22/86.78</td>
<td id="A3.T5.8.8.8.9" class="ltx_td ltx_align_center">89.90</td>
<td id="A3.T5.8.8.8.10" class="ltx_td ltx_align_center">84.94</td>
</tr>
<tr id="A3.T5.9.9.9" class="ltx_tr">
<th id="A3.T5.9.9.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><math id="A3.T5.9.9.9.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{P}}" display="inline"><semantics id="A3.T5.9.9.9.1.m1.1a"><msub id="A3.T5.9.9.9.1.m1.1.1" xref="A3.T5.9.9.9.1.m1.1.1.cmml"><mtext id="A3.T5.9.9.9.1.m1.1.1.2" xref="A3.T5.9.9.9.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="A3.T5.9.9.9.1.m1.1.1.3" xref="A3.T5.9.9.9.1.m1.1.1.3a.cmml">P</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.T5.9.9.9.1.m1.1b"><apply id="A3.T5.9.9.9.1.m1.1.1.cmml" xref="A3.T5.9.9.9.1.m1.1.1"><csymbol cd="ambiguous" id="A3.T5.9.9.9.1.m1.1.1.1.cmml" xref="A3.T5.9.9.9.1.m1.1.1">subscript</csymbol><ci id="A3.T5.9.9.9.1.m1.1.1.2a.cmml" xref="A3.T5.9.9.9.1.m1.1.1.2"><mtext id="A3.T5.9.9.9.1.m1.1.1.2.cmml" xref="A3.T5.9.9.9.1.m1.1.1.2">MoM</mtext></ci><ci id="A3.T5.9.9.9.1.m1.1.1.3a.cmml" xref="A3.T5.9.9.9.1.m1.1.1.3"><mtext mathsize="70%" id="A3.T5.9.9.9.1.m1.1.1.3.cmml" xref="A3.T5.9.9.9.1.m1.1.1.3">P</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.T5.9.9.9.1.m1.1c">\text{MoM}_{\text{P}}</annotation></semantics></math></th>
<td id="A3.T5.9.9.9.2" class="ltx_td ltx_align_center ltx_border_bb">94.42</td>
<td id="A3.T5.9.9.9.3" class="ltx_td ltx_align_center ltx_border_bb">64.49</td>
<td id="A3.T5.9.9.9.4" class="ltx_td ltx_align_center ltx_border_bb">89.56</td>
<td id="A3.T5.9.9.9.5" class="ltx_td ltx_align_center ltx_border_bb">88.68</td>
<td id="A3.T5.9.9.9.6" class="ltx_td ltx_align_center ltx_border_bb">92.94</td>
<td id="A3.T5.9.9.9.7" class="ltx_td ltx_align_center ltx_border_bb">77.69</td>
<td id="A3.T5.9.9.9.8" class="ltx_td ltx_align_center ltx_border_bb">87.68/86.98</td>
<td id="A3.T5.9.9.9.9" class="ltx_td ltx_align_center ltx_border_bb">90.70</td>
<td id="A3.T5.9.9.9.10" class="ltx_td ltx_align_center ltx_border_bb">85.90</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>Detailed evaluation on the GLUE benchmark. We follow the previous evaluation setting <cite class="ltx_cite ltx_citemacro_cite">Radford et al. (<a href="#bib.bib35" title="" class="ltx_ref">2018</a>)</cite>, for SST-2, QNLI, RTE, and MNLI, we report accuracy as the metric. For MRPC and QQP, we report the F1 score. For STS-b, we report the combined score of Pearson correlation and Spearman correlation.</figcaption>
</figure>
<figure id="A3.T6" class="ltx_table">
<table id="A3.T6.9" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A3.T6.9.10.1" class="ltx_tr">
<th id="A3.T6.9.10.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt">Method</th>
<td id="A3.T6.9.10.1.2" class="ltx_td ltx_align_right ltx_border_tt">ROUGE-1</td>
<td id="A3.T6.9.10.1.3" class="ltx_td ltx_align_right ltx_border_tt">ROUGE-2</td>
<td id="A3.T6.9.10.1.4" class="ltx_td ltx_align_right ltx_border_tt">ROUGE-L</td>
<td id="A3.T6.9.10.1.5" class="ltx_td ltx_align_right ltx_border_tt">ROUGE-AVG</td>
</tr>
<tr id="A3.T6.9.11.2" class="ltx_tr">
<th id="A3.T6.9.11.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_tt" colspan="5"><span id="A3.T6.9.11.2.1.1" class="ltx_text ltx_font_italic">small</span></th>
</tr>
<tr id="A3.T6.9.12.3" class="ltx_tr">
<th id="A3.T6.9.12.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">GPT2</th>
<td id="A3.T6.9.12.3.2" class="ltx_td ltx_align_right ltx_border_t">20.8</td>
<td id="A3.T6.9.12.3.3" class="ltx_td ltx_align_right ltx_border_t">5.05</td>
<td id="A3.T6.9.12.3.4" class="ltx_td ltx_align_right ltx_border_t">16.92</td>
<td id="A3.T6.9.12.3.5" class="ltx_td ltx_align_right ltx_border_t">14.26</td>
</tr>
<tr id="A3.T6.9.13.4" class="ltx_tr">
<th id="A3.T6.9.13.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MoD</th>
<td id="A3.T6.9.13.4.2" class="ltx_td ltx_align_right">14.45</td>
<td id="A3.T6.9.13.4.3" class="ltx_td ltx_align_right">2.89</td>
<td id="A3.T6.9.13.4.4" class="ltx_td ltx_align_right">11.80</td>
<td id="A3.T6.9.13.4.5" class="ltx_td ltx_align_right">9.71</td>
</tr>
<tr id="A3.T6.9.14.5" class="ltx_tr">
<th id="A3.T6.9.14.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MoE</th>
<td id="A3.T6.9.14.5.2" class="ltx_td ltx_align_right">20.56</td>
<td id="A3.T6.9.14.5.3" class="ltx_td ltx_align_right">5.26</td>
<td id="A3.T6.9.14.5.4" class="ltx_td ltx_align_right">16.71</td>
<td id="A3.T6.9.14.5.5" class="ltx_td ltx_align_right">14.18</td>
</tr>
<tr id="A3.T6.9.15.6" class="ltx_tr">
<th id="A3.T6.9.15.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MoE (share)</th>
<td id="A3.T6.9.15.6.2" class="ltx_td ltx_align_right">20.64</td>
<td id="A3.T6.9.15.6.3" class="ltx_td ltx_align_right">5.17</td>
<td id="A3.T6.9.15.6.4" class="ltx_td ltx_align_right">16.64</td>
<td id="A3.T6.9.15.6.5" class="ltx_td ltx_align_right">14.15</td>
</tr>
<tr id="A3.T6.1.1" class="ltx_tr">
<th id="A3.T6.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><math id="A3.T6.1.1.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{E}}" display="inline"><semantics id="A3.T6.1.1.1.m1.1a"><msub id="A3.T6.1.1.1.m1.1.1" xref="A3.T6.1.1.1.m1.1.1.cmml"><mtext id="A3.T6.1.1.1.m1.1.1.2" xref="A3.T6.1.1.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="A3.T6.1.1.1.m1.1.1.3" xref="A3.T6.1.1.1.m1.1.1.3a.cmml">E</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.T6.1.1.1.m1.1b"><apply id="A3.T6.1.1.1.m1.1.1.cmml" xref="A3.T6.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="A3.T6.1.1.1.m1.1.1.1.cmml" xref="A3.T6.1.1.1.m1.1.1">subscript</csymbol><ci id="A3.T6.1.1.1.m1.1.1.2a.cmml" xref="A3.T6.1.1.1.m1.1.1.2"><mtext id="A3.T6.1.1.1.m1.1.1.2.cmml" xref="A3.T6.1.1.1.m1.1.1.2">MoM</mtext></ci><ci id="A3.T6.1.1.1.m1.1.1.3a.cmml" xref="A3.T6.1.1.1.m1.1.1.3"><mtext mathsize="70%" id="A3.T6.1.1.1.m1.1.1.3.cmml" xref="A3.T6.1.1.1.m1.1.1.3">E</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.T6.1.1.1.m1.1c">\text{MoM}_{\text{E}}</annotation></semantics></math></th>
<td id="A3.T6.1.1.2" class="ltx_td ltx_align_right">20.62</td>
<td id="A3.T6.1.1.3" class="ltx_td ltx_align_right">4.98</td>
<td id="A3.T6.1.1.4" class="ltx_td ltx_align_right">16.91</td>
<td id="A3.T6.1.1.5" class="ltx_td ltx_align_right">14.17</td>
</tr>
<tr id="A3.T6.2.2" class="ltx_tr">
<th id="A3.T6.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><math id="A3.T6.2.2.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{I}}" display="inline"><semantics id="A3.T6.2.2.1.m1.1a"><msub id="A3.T6.2.2.1.m1.1.1" xref="A3.T6.2.2.1.m1.1.1.cmml"><mtext id="A3.T6.2.2.1.m1.1.1.2" xref="A3.T6.2.2.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="A3.T6.2.2.1.m1.1.1.3" xref="A3.T6.2.2.1.m1.1.1.3a.cmml">I</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.T6.2.2.1.m1.1b"><apply id="A3.T6.2.2.1.m1.1.1.cmml" xref="A3.T6.2.2.1.m1.1.1"><csymbol cd="ambiguous" id="A3.T6.2.2.1.m1.1.1.1.cmml" xref="A3.T6.2.2.1.m1.1.1">subscript</csymbol><ci id="A3.T6.2.2.1.m1.1.1.2a.cmml" xref="A3.T6.2.2.1.m1.1.1.2"><mtext id="A3.T6.2.2.1.m1.1.1.2.cmml" xref="A3.T6.2.2.1.m1.1.1.2">MoM</mtext></ci><ci id="A3.T6.2.2.1.m1.1.1.3a.cmml" xref="A3.T6.2.2.1.m1.1.1.3"><mtext mathsize="70%" id="A3.T6.2.2.1.m1.1.1.3.cmml" xref="A3.T6.2.2.1.m1.1.1.3">I</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.T6.2.2.1.m1.1c">\text{MoM}_{\text{I}}</annotation></semantics></math></th>
<td id="A3.T6.2.2.2" class="ltx_td ltx_align_right">20.50</td>
<td id="A3.T6.2.2.3" class="ltx_td ltx_align_right">5.31</td>
<td id="A3.T6.2.2.4" class="ltx_td ltx_align_right">16.90</td>
<td id="A3.T6.2.2.5" class="ltx_td ltx_align_right">14.24</td>
</tr>
<tr id="A3.T6.3.3" class="ltx_tr">
<th id="A3.T6.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><math id="A3.T6.3.3.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{P}}" display="inline"><semantics id="A3.T6.3.3.1.m1.1a"><msub id="A3.T6.3.3.1.m1.1.1" xref="A3.T6.3.3.1.m1.1.1.cmml"><mtext id="A3.T6.3.3.1.m1.1.1.2" xref="A3.T6.3.3.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="A3.T6.3.3.1.m1.1.1.3" xref="A3.T6.3.3.1.m1.1.1.3a.cmml">P</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.T6.3.3.1.m1.1b"><apply id="A3.T6.3.3.1.m1.1.1.cmml" xref="A3.T6.3.3.1.m1.1.1"><csymbol cd="ambiguous" id="A3.T6.3.3.1.m1.1.1.1.cmml" xref="A3.T6.3.3.1.m1.1.1">subscript</csymbol><ci id="A3.T6.3.3.1.m1.1.1.2a.cmml" xref="A3.T6.3.3.1.m1.1.1.2"><mtext id="A3.T6.3.3.1.m1.1.1.2.cmml" xref="A3.T6.3.3.1.m1.1.1.2">MoM</mtext></ci><ci id="A3.T6.3.3.1.m1.1.1.3a.cmml" xref="A3.T6.3.3.1.m1.1.1.3"><mtext mathsize="70%" id="A3.T6.3.3.1.m1.1.1.3.cmml" xref="A3.T6.3.3.1.m1.1.1.3">P</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.T6.3.3.1.m1.1c">\text{MoM}_{\text{P}}</annotation></semantics></math></th>
<td id="A3.T6.3.3.2" class="ltx_td ltx_align_right">21.73</td>
<td id="A3.T6.3.3.3" class="ltx_td ltx_align_right">6.14</td>
<td id="A3.T6.3.3.4" class="ltx_td ltx_align_right">17.72</td>
<td id="A3.T6.3.3.5" class="ltx_td ltx_align_right">15.19</td>
</tr>
<tr id="A3.T6.9.16.7" class="ltx_tr">
<th id="A3.T6.9.16.7.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" colspan="5"><span id="A3.T6.9.16.7.1.1" class="ltx_text ltx_font_italic">medium</span></th>
</tr>
<tr id="A3.T6.9.17.8" class="ltx_tr">
<th id="A3.T6.9.17.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">GPT2</th>
<td id="A3.T6.9.17.8.2" class="ltx_td ltx_align_right ltx_border_t">25.07</td>
<td id="A3.T6.9.17.8.3" class="ltx_td ltx_align_right ltx_border_t">8.35</td>
<td id="A3.T6.9.17.8.4" class="ltx_td ltx_align_right ltx_border_t">21.00</td>
<td id="A3.T6.9.17.8.5" class="ltx_td ltx_align_right ltx_border_t">18.14</td>
</tr>
<tr id="A3.T6.9.18.9" class="ltx_tr">
<th id="A3.T6.9.18.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MoD</th>
<td id="A3.T6.9.18.9.2" class="ltx_td ltx_align_right">20.89</td>
<td id="A3.T6.9.18.9.3" class="ltx_td ltx_align_right">6.04</td>
<td id="A3.T6.9.18.9.4" class="ltx_td ltx_align_right">17.52</td>
<td id="A3.T6.9.18.9.5" class="ltx_td ltx_align_right">14.81</td>
</tr>
<tr id="A3.T6.9.19.10" class="ltx_tr">
<th id="A3.T6.9.19.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MoE</th>
<td id="A3.T6.9.19.10.2" class="ltx_td ltx_align_right">24.58</td>
<td id="A3.T6.9.19.10.3" class="ltx_td ltx_align_right">8.20</td>
<td id="A3.T6.9.19.10.4" class="ltx_td ltx_align_right">20.49</td>
<td id="A3.T6.9.19.10.5" class="ltx_td ltx_align_right">17.75</td>
</tr>
<tr id="A3.T6.9.20.11" class="ltx_tr">
<th id="A3.T6.9.20.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MoE (share)</th>
<td id="A3.T6.9.20.11.2" class="ltx_td ltx_align_right">24.36</td>
<td id="A3.T6.9.20.11.3" class="ltx_td ltx_align_right">8.19</td>
<td id="A3.T6.9.20.11.4" class="ltx_td ltx_align_right">20.22</td>
<td id="A3.T6.9.20.11.5" class="ltx_td ltx_align_right">17.59</td>
</tr>
<tr id="A3.T6.4.4" class="ltx_tr">
<th id="A3.T6.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><math id="A3.T6.4.4.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{E}}" display="inline"><semantics id="A3.T6.4.4.1.m1.1a"><msub id="A3.T6.4.4.1.m1.1.1" xref="A3.T6.4.4.1.m1.1.1.cmml"><mtext id="A3.T6.4.4.1.m1.1.1.2" xref="A3.T6.4.4.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="A3.T6.4.4.1.m1.1.1.3" xref="A3.T6.4.4.1.m1.1.1.3a.cmml">E</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.T6.4.4.1.m1.1b"><apply id="A3.T6.4.4.1.m1.1.1.cmml" xref="A3.T6.4.4.1.m1.1.1"><csymbol cd="ambiguous" id="A3.T6.4.4.1.m1.1.1.1.cmml" xref="A3.T6.4.4.1.m1.1.1">subscript</csymbol><ci id="A3.T6.4.4.1.m1.1.1.2a.cmml" xref="A3.T6.4.4.1.m1.1.1.2"><mtext id="A3.T6.4.4.1.m1.1.1.2.cmml" xref="A3.T6.4.4.1.m1.1.1.2">MoM</mtext></ci><ci id="A3.T6.4.4.1.m1.1.1.3a.cmml" xref="A3.T6.4.4.1.m1.1.1.3"><mtext mathsize="70%" id="A3.T6.4.4.1.m1.1.1.3.cmml" xref="A3.T6.4.4.1.m1.1.1.3">E</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.T6.4.4.1.m1.1c">\text{MoM}_{\text{E}}</annotation></semantics></math></th>
<td id="A3.T6.4.4.2" class="ltx_td ltx_align_right">24.56</td>
<td id="A3.T6.4.4.3" class="ltx_td ltx_align_right">7.04</td>
<td id="A3.T6.4.4.4" class="ltx_td ltx_align_right">19.72</td>
<td id="A3.T6.4.4.5" class="ltx_td ltx_align_right">17.11</td>
</tr>
<tr id="A3.T6.5.5" class="ltx_tr">
<th id="A3.T6.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><math id="A3.T6.5.5.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{I}}" display="inline"><semantics id="A3.T6.5.5.1.m1.1a"><msub id="A3.T6.5.5.1.m1.1.1" xref="A3.T6.5.5.1.m1.1.1.cmml"><mtext id="A3.T6.5.5.1.m1.1.1.2" xref="A3.T6.5.5.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="A3.T6.5.5.1.m1.1.1.3" xref="A3.T6.5.5.1.m1.1.1.3a.cmml">I</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.T6.5.5.1.m1.1b"><apply id="A3.T6.5.5.1.m1.1.1.cmml" xref="A3.T6.5.5.1.m1.1.1"><csymbol cd="ambiguous" id="A3.T6.5.5.1.m1.1.1.1.cmml" xref="A3.T6.5.5.1.m1.1.1">subscript</csymbol><ci id="A3.T6.5.5.1.m1.1.1.2a.cmml" xref="A3.T6.5.5.1.m1.1.1.2"><mtext id="A3.T6.5.5.1.m1.1.1.2.cmml" xref="A3.T6.5.5.1.m1.1.1.2">MoM</mtext></ci><ci id="A3.T6.5.5.1.m1.1.1.3a.cmml" xref="A3.T6.5.5.1.m1.1.1.3"><mtext mathsize="70%" id="A3.T6.5.5.1.m1.1.1.3.cmml" xref="A3.T6.5.5.1.m1.1.1.3">I</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.T6.5.5.1.m1.1c">\text{MoM}_{\text{I}}</annotation></semantics></math></th>
<td id="A3.T6.5.5.2" class="ltx_td ltx_align_right">26.29</td>
<td id="A3.T6.5.5.3" class="ltx_td ltx_align_right">8.21</td>
<td id="A3.T6.5.5.4" class="ltx_td ltx_align_right">21.47</td>
<td id="A3.T6.5.5.5" class="ltx_td ltx_align_right">18.66</td>
</tr>
<tr id="A3.T6.6.6" class="ltx_tr">
<th id="A3.T6.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><math id="A3.T6.6.6.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{P}}" display="inline"><semantics id="A3.T6.6.6.1.m1.1a"><msub id="A3.T6.6.6.1.m1.1.1" xref="A3.T6.6.6.1.m1.1.1.cmml"><mtext id="A3.T6.6.6.1.m1.1.1.2" xref="A3.T6.6.6.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="A3.T6.6.6.1.m1.1.1.3" xref="A3.T6.6.6.1.m1.1.1.3a.cmml">P</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.T6.6.6.1.m1.1b"><apply id="A3.T6.6.6.1.m1.1.1.cmml" xref="A3.T6.6.6.1.m1.1.1"><csymbol cd="ambiguous" id="A3.T6.6.6.1.m1.1.1.1.cmml" xref="A3.T6.6.6.1.m1.1.1">subscript</csymbol><ci id="A3.T6.6.6.1.m1.1.1.2a.cmml" xref="A3.T6.6.6.1.m1.1.1.2"><mtext id="A3.T6.6.6.1.m1.1.1.2.cmml" xref="A3.T6.6.6.1.m1.1.1.2">MoM</mtext></ci><ci id="A3.T6.6.6.1.m1.1.1.3a.cmml" xref="A3.T6.6.6.1.m1.1.1.3"><mtext mathsize="70%" id="A3.T6.6.6.1.m1.1.1.3.cmml" xref="A3.T6.6.6.1.m1.1.1.3">P</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.T6.6.6.1.m1.1c">\text{MoM}_{\text{P}}</annotation></semantics></math></th>
<td id="A3.T6.6.6.2" class="ltx_td ltx_align_right">26.61</td>
<td id="A3.T6.6.6.3" class="ltx_td ltx_align_right">9.05</td>
<td id="A3.T6.6.6.4" class="ltx_td ltx_align_right">22.24</td>
<td id="A3.T6.6.6.5" class="ltx_td ltx_align_right">19.30</td>
</tr>
<tr id="A3.T6.9.21.12" class="ltx_tr">
<th id="A3.T6.9.21.12.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" colspan="5"><span id="A3.T6.9.21.12.1.1" class="ltx_text ltx_font_italic">large</span></th>
</tr>
<tr id="A3.T6.9.22.13" class="ltx_tr">
<th id="A3.T6.9.22.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">GPT2</th>
<td id="A3.T6.9.22.13.2" class="ltx_td ltx_align_right ltx_border_t">28.16</td>
<td id="A3.T6.9.22.13.3" class="ltx_td ltx_align_right ltx_border_t">9.92</td>
<td id="A3.T6.9.22.13.4" class="ltx_td ltx_align_right ltx_border_t">22.98</td>
<td id="A3.T6.9.22.13.5" class="ltx_td ltx_align_right ltx_border_t">20.35</td>
</tr>
<tr id="A3.T6.9.23.14" class="ltx_tr">
<th id="A3.T6.9.23.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MoD</th>
<td id="A3.T6.9.23.14.2" class="ltx_td ltx_align_right">26.08</td>
<td id="A3.T6.9.23.14.3" class="ltx_td ltx_align_right">8.38</td>
<td id="A3.T6.9.23.14.4" class="ltx_td ltx_align_right">21.41</td>
<td id="A3.T6.9.23.14.5" class="ltx_td ltx_align_right">18.62</td>
</tr>
<tr id="A3.T6.9.24.15" class="ltx_tr">
<th id="A3.T6.9.24.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MoE</th>
<td id="A3.T6.9.24.15.2" class="ltx_td ltx_align_right">28.54</td>
<td id="A3.T6.9.24.15.3" class="ltx_td ltx_align_right">9.89</td>
<td id="A3.T6.9.24.15.4" class="ltx_td ltx_align_right">23.47</td>
<td id="A3.T6.9.24.15.5" class="ltx_td ltx_align_right">20.63</td>
</tr>
<tr id="A3.T6.9.25.16" class="ltx_tr">
<th id="A3.T6.9.25.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MoE (share)</th>
<td id="A3.T6.9.25.16.2" class="ltx_td ltx_align_right">28.47</td>
<td id="A3.T6.9.25.16.3" class="ltx_td ltx_align_right">9.47</td>
<td id="A3.T6.9.25.16.4" class="ltx_td ltx_align_right">23.23</td>
<td id="A3.T6.9.25.16.5" class="ltx_td ltx_align_right">20.39</td>
</tr>
<tr id="A3.T6.7.7" class="ltx_tr">
<th id="A3.T6.7.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><math id="A3.T6.7.7.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{E}}" display="inline"><semantics id="A3.T6.7.7.1.m1.1a"><msub id="A3.T6.7.7.1.m1.1.1" xref="A3.T6.7.7.1.m1.1.1.cmml"><mtext id="A3.T6.7.7.1.m1.1.1.2" xref="A3.T6.7.7.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="A3.T6.7.7.1.m1.1.1.3" xref="A3.T6.7.7.1.m1.1.1.3a.cmml">E</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.T6.7.7.1.m1.1b"><apply id="A3.T6.7.7.1.m1.1.1.cmml" xref="A3.T6.7.7.1.m1.1.1"><csymbol cd="ambiguous" id="A3.T6.7.7.1.m1.1.1.1.cmml" xref="A3.T6.7.7.1.m1.1.1">subscript</csymbol><ci id="A3.T6.7.7.1.m1.1.1.2a.cmml" xref="A3.T6.7.7.1.m1.1.1.2"><mtext id="A3.T6.7.7.1.m1.1.1.2.cmml" xref="A3.T6.7.7.1.m1.1.1.2">MoM</mtext></ci><ci id="A3.T6.7.7.1.m1.1.1.3a.cmml" xref="A3.T6.7.7.1.m1.1.1.3"><mtext mathsize="70%" id="A3.T6.7.7.1.m1.1.1.3.cmml" xref="A3.T6.7.7.1.m1.1.1.3">E</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.T6.7.7.1.m1.1c">\text{MoM}_{\text{E}}</annotation></semantics></math></th>
<td id="A3.T6.7.7.2" class="ltx_td ltx_align_right">28.48</td>
<td id="A3.T6.7.7.3" class="ltx_td ltx_align_right">9.65</td>
<td id="A3.T6.7.7.4" class="ltx_td ltx_align_right">23.24</td>
<td id="A3.T6.7.7.5" class="ltx_td ltx_align_right">20.46</td>
</tr>
<tr id="A3.T6.8.8" class="ltx_tr">
<th id="A3.T6.8.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><math id="A3.T6.8.8.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{I}}" display="inline"><semantics id="A3.T6.8.8.1.m1.1a"><msub id="A3.T6.8.8.1.m1.1.1" xref="A3.T6.8.8.1.m1.1.1.cmml"><mtext id="A3.T6.8.8.1.m1.1.1.2" xref="A3.T6.8.8.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="A3.T6.8.8.1.m1.1.1.3" xref="A3.T6.8.8.1.m1.1.1.3a.cmml">I</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.T6.8.8.1.m1.1b"><apply id="A3.T6.8.8.1.m1.1.1.cmml" xref="A3.T6.8.8.1.m1.1.1"><csymbol cd="ambiguous" id="A3.T6.8.8.1.m1.1.1.1.cmml" xref="A3.T6.8.8.1.m1.1.1">subscript</csymbol><ci id="A3.T6.8.8.1.m1.1.1.2a.cmml" xref="A3.T6.8.8.1.m1.1.1.2"><mtext id="A3.T6.8.8.1.m1.1.1.2.cmml" xref="A3.T6.8.8.1.m1.1.1.2">MoM</mtext></ci><ci id="A3.T6.8.8.1.m1.1.1.3a.cmml" xref="A3.T6.8.8.1.m1.1.1.3"><mtext mathsize="70%" id="A3.T6.8.8.1.m1.1.1.3.cmml" xref="A3.T6.8.8.1.m1.1.1.3">I</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.T6.8.8.1.m1.1c">\text{MoM}_{\text{I}}</annotation></semantics></math></th>
<td id="A3.T6.8.8.2" class="ltx_td ltx_align_right">30.91</td>
<td id="A3.T6.8.8.3" class="ltx_td ltx_align_right">10.21</td>
<td id="A3.T6.8.8.4" class="ltx_td ltx_align_right">24.08</td>
<td id="A3.T6.8.8.5" class="ltx_td ltx_align_right">21.73</td>
</tr>
<tr id="A3.T6.9.9" class="ltx_tr">
<th id="A3.T6.9.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><math id="A3.T6.9.9.1.m1.1" class="ltx_Math" alttext="\text{MoM}_{\text{P}}" display="inline"><semantics id="A3.T6.9.9.1.m1.1a"><msub id="A3.T6.9.9.1.m1.1.1" xref="A3.T6.9.9.1.m1.1.1.cmml"><mtext id="A3.T6.9.9.1.m1.1.1.2" xref="A3.T6.9.9.1.m1.1.1.2a.cmml">MoM</mtext><mtext id="A3.T6.9.9.1.m1.1.1.3" xref="A3.T6.9.9.1.m1.1.1.3a.cmml">P</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.T6.9.9.1.m1.1b"><apply id="A3.T6.9.9.1.m1.1.1.cmml" xref="A3.T6.9.9.1.m1.1.1"><csymbol cd="ambiguous" id="A3.T6.9.9.1.m1.1.1.1.cmml" xref="A3.T6.9.9.1.m1.1.1">subscript</csymbol><ci id="A3.T6.9.9.1.m1.1.1.2a.cmml" xref="A3.T6.9.9.1.m1.1.1.2"><mtext id="A3.T6.9.9.1.m1.1.1.2.cmml" xref="A3.T6.9.9.1.m1.1.1.2">MoM</mtext></ci><ci id="A3.T6.9.9.1.m1.1.1.3a.cmml" xref="A3.T6.9.9.1.m1.1.1.3"><mtext mathsize="70%" id="A3.T6.9.9.1.m1.1.1.3.cmml" xref="A3.T6.9.9.1.m1.1.1.3">P</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.T6.9.9.1.m1.1c">\text{MoM}_{\text{P}}</annotation></semantics></math></th>
<td id="A3.T6.9.9.2" class="ltx_td ltx_align_right ltx_border_bb">31.38</td>
<td id="A3.T6.9.9.3" class="ltx_td ltx_align_right ltx_border_bb">10.77</td>
<td id="A3.T6.9.9.4" class="ltx_td ltx_align_right ltx_border_bb">24.93</td>
<td id="A3.T6.9.9.5" class="ltx_td ltx_align_right ltx_border_bb">22.36</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 6: </span>Detailed evaluation on the XSUM dataset. ROUGE is employed as the evaluation metrics.</figcaption>
</figure>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Router analysis</h2>

<section id="A4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">D.1 </span>Architecture design</h3>

<figure id="A4.F5" class="ltx_figure"><img src="/html/2407.06677/assets/x5.png" id="A4.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="230" height="172" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>Training curves of MoM-small (K1H4) with {GRU, MLP} routers.</figcaption>
</figure>
<div id="A4.SS1.p1" class="ltx_para ltx_noindent">
<p id="A4.SS1.p1.1" class="ltx_p">We study the implementation of a key component in MoM: the routers.
We substitute the GRU within the router with a simple two-layer MLP, eliminating the interaction among router decision states.
Our exploration of the router’s impact involves two setups: (a) initializing MoM from scratch, and (b) employing the two-phase training approach.
Here are some intriguing results.
As depicted in Figure <a href="#A4.F5" title="Figure 5 ‣ D.1 Architecture design ‣ Appendix D Router analysis ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, when initializing from scratch, both router structures exhibit nearly identical loss curves.
However, under setting (b), training MoM with the MLP router becomes unstable marked by spikes in gradient magnitude throughout the training.
This instability suggests the router’s inability to establish a consistent assembly plan for tokens.
When initializing from scratch, the required capability may not be learned efficiently by these modules, as they often develop homogeneous functionalities that waste the parameters.
Conversely, in setting (b), where modules are initialized with specialized functions, the optimization progresses smoothly and converges quickly.</p>
</div>
</section>
<section id="A4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">D.2 </span>Learned router patterns</h3>

<figure id="A4.F6" class="ltx_figure"><img src="/html/2407.06677/assets/x6.png" id="A4.F6.g1" class="ltx_graphics ltx_centering ltx_img_portrait" width="138" height="246" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>Routing patterns of MoM-medium.</figcaption>
</figure>
<div id="A4.SS2.p1" class="ltx_para ltx_noindent">
<p id="A4.SS2.p1.2" class="ltx_p">We are curious whether the router follows specific patterns when choosing and assembling modules.
We visualize the transition probabilities between modules (Figure <a href="#A4.F6" title="Figure 6 ‣ D.2 Learned router patterns ‣ Appendix D Router analysis ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>) to answer this. The first observation is that the router does not degrade into simply memorizing the original shallow-to-deep order but jumps across modules as expected. For example, a common routing path in Figure <a href="#A4.F6" title="Figure 6 ‣ D.2 Learned router patterns ‣ Appendix D Router analysis ‣ Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> is <math id="A4.SS2.p1.1.m1.6" class="ltx_Math" alttext="(2,2,1,3,4,2)" display="inline"><semantics id="A4.SS2.p1.1.m1.6a"><mrow id="A4.SS2.p1.1.m1.6.7.2" xref="A4.SS2.p1.1.m1.6.7.1.cmml"><mo stretchy="false" id="A4.SS2.p1.1.m1.6.7.2.1" xref="A4.SS2.p1.1.m1.6.7.1.cmml">(</mo><mn id="A4.SS2.p1.1.m1.1.1" xref="A4.SS2.p1.1.m1.1.1.cmml">2</mn><mo id="A4.SS2.p1.1.m1.6.7.2.2" xref="A4.SS2.p1.1.m1.6.7.1.cmml">,</mo><mn id="A4.SS2.p1.1.m1.2.2" xref="A4.SS2.p1.1.m1.2.2.cmml">2</mn><mo id="A4.SS2.p1.1.m1.6.7.2.3" xref="A4.SS2.p1.1.m1.6.7.1.cmml">,</mo><mn id="A4.SS2.p1.1.m1.3.3" xref="A4.SS2.p1.1.m1.3.3.cmml">1</mn><mo id="A4.SS2.p1.1.m1.6.7.2.4" xref="A4.SS2.p1.1.m1.6.7.1.cmml">,</mo><mn id="A4.SS2.p1.1.m1.4.4" xref="A4.SS2.p1.1.m1.4.4.cmml">3</mn><mo id="A4.SS2.p1.1.m1.6.7.2.5" xref="A4.SS2.p1.1.m1.6.7.1.cmml">,</mo><mn id="A4.SS2.p1.1.m1.5.5" xref="A4.SS2.p1.1.m1.5.5.cmml">4</mn><mo id="A4.SS2.p1.1.m1.6.7.2.6" xref="A4.SS2.p1.1.m1.6.7.1.cmml">,</mo><mn id="A4.SS2.p1.1.m1.6.6" xref="A4.SS2.p1.1.m1.6.6.cmml">2</mn><mo stretchy="false" id="A4.SS2.p1.1.m1.6.7.2.7" xref="A4.SS2.p1.1.m1.6.7.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.SS2.p1.1.m1.6b"><vector id="A4.SS2.p1.1.m1.6.7.1.cmml" xref="A4.SS2.p1.1.m1.6.7.2"><cn type="integer" id="A4.SS2.p1.1.m1.1.1.cmml" xref="A4.SS2.p1.1.m1.1.1">2</cn><cn type="integer" id="A4.SS2.p1.1.m1.2.2.cmml" xref="A4.SS2.p1.1.m1.2.2">2</cn><cn type="integer" id="A4.SS2.p1.1.m1.3.3.cmml" xref="A4.SS2.p1.1.m1.3.3">1</cn><cn type="integer" id="A4.SS2.p1.1.m1.4.4.cmml" xref="A4.SS2.p1.1.m1.4.4">3</cn><cn type="integer" id="A4.SS2.p1.1.m1.5.5.cmml" xref="A4.SS2.p1.1.m1.5.5">4</cn><cn type="integer" id="A4.SS2.p1.1.m1.6.6.cmml" xref="A4.SS2.p1.1.m1.6.6">2</cn></vector></annotation-xml><annotation encoding="application/x-tex" id="A4.SS2.p1.1.m1.6c">(2,2,1,3,4,2)</annotation></semantics></math> for <span id="A4.SS2.p1.2.1" class="ltx_text ltx_font_typewriter">FFN</span> modules and <math id="A4.SS2.p1.2.m2.6" class="ltx_Math" alttext="(3,1,3,2,S,4)" display="inline"><semantics id="A4.SS2.p1.2.m2.6a"><mrow id="A4.SS2.p1.2.m2.6.7.2" xref="A4.SS2.p1.2.m2.6.7.1.cmml"><mo stretchy="false" id="A4.SS2.p1.2.m2.6.7.2.1" xref="A4.SS2.p1.2.m2.6.7.1.cmml">(</mo><mn id="A4.SS2.p1.2.m2.1.1" xref="A4.SS2.p1.2.m2.1.1.cmml">3</mn><mo id="A4.SS2.p1.2.m2.6.7.2.2" xref="A4.SS2.p1.2.m2.6.7.1.cmml">,</mo><mn id="A4.SS2.p1.2.m2.2.2" xref="A4.SS2.p1.2.m2.2.2.cmml">1</mn><mo id="A4.SS2.p1.2.m2.6.7.2.3" xref="A4.SS2.p1.2.m2.6.7.1.cmml">,</mo><mn id="A4.SS2.p1.2.m2.3.3" xref="A4.SS2.p1.2.m2.3.3.cmml">3</mn><mo id="A4.SS2.p1.2.m2.6.7.2.4" xref="A4.SS2.p1.2.m2.6.7.1.cmml">,</mo><mn id="A4.SS2.p1.2.m2.4.4" xref="A4.SS2.p1.2.m2.4.4.cmml">2</mn><mo id="A4.SS2.p1.2.m2.6.7.2.5" xref="A4.SS2.p1.2.m2.6.7.1.cmml">,</mo><mi id="A4.SS2.p1.2.m2.5.5" xref="A4.SS2.p1.2.m2.5.5.cmml">S</mi><mo id="A4.SS2.p1.2.m2.6.7.2.6" xref="A4.SS2.p1.2.m2.6.7.1.cmml">,</mo><mn id="A4.SS2.p1.2.m2.6.6" xref="A4.SS2.p1.2.m2.6.6.cmml">4</mn><mo stretchy="false" id="A4.SS2.p1.2.m2.6.7.2.7" xref="A4.SS2.p1.2.m2.6.7.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.SS2.p1.2.m2.6b"><vector id="A4.SS2.p1.2.m2.6.7.1.cmml" xref="A4.SS2.p1.2.m2.6.7.2"><cn type="integer" id="A4.SS2.p1.2.m2.1.1.cmml" xref="A4.SS2.p1.2.m2.1.1">3</cn><cn type="integer" id="A4.SS2.p1.2.m2.2.2.cmml" xref="A4.SS2.p1.2.m2.2.2">1</cn><cn type="integer" id="A4.SS2.p1.2.m2.3.3.cmml" xref="A4.SS2.p1.2.m2.3.3">3</cn><cn type="integer" id="A4.SS2.p1.2.m2.4.4.cmml" xref="A4.SS2.p1.2.m2.4.4">2</cn><ci id="A4.SS2.p1.2.m2.5.5.cmml" xref="A4.SS2.p1.2.m2.5.5">𝑆</ci><cn type="integer" id="A4.SS2.p1.2.m2.6.6.cmml" xref="A4.SS2.p1.2.m2.6.6">4</cn></vector></annotation-xml><annotation encoding="application/x-tex" id="A4.SS2.p1.2.m2.6c">(3,1,3,2,S,4)</annotation></semantics></math> for <span id="A4.SS2.p1.2.2" class="ltx_text ltx_font_typewriter">MHA</span> modules (number represents the module index).
Another observation is that the loads of different modules are imbalanced.
In some cases, specific modules are hardly used. Unlike MoE, which uses an auxiliary loss to balance the loads across different experts <cite class="ltx_cite ltx_citemacro_citep">(Fedus et al., <a href="#bib.bib14" title="" class="ltx_ref">2022</a>)</cite>, we do not see a positive effect by adding a balance loss to MoM.
Adding regularization alleviates the imbalance issue at the cost of performance degradation (by increasing validation perplexity by 1.8 points).
We posit an intuitive explanation: within the language model framework, tasks that can be decomposed into numerous sub-tasks may exhibit various levels of difficulty.
Consequently, some sub-tasks necessitate more engagement of modules with specific processing capabilities, thus contributing to the observed imbalance.</p>
</div>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2407.06676" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2407.06677" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2407.06677">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2407.06677" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2407.06678" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Mon Aug  5 17:22:10 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2407.06677
Is it always necessary to compute tokens from shallow to deep layers in Transformers? The continued success of vanilla Transformers and their variants suggests an undoubted “yes”. In this work, however, we attempt to break the depth-ordered convention by proposing a novel architecture dubbed mixture-of-modules (MoM) , which is motivated by an intuition that any layer, regardless of its position, can be used to compute a token as long as it possesses the needed processing capabilities.
The construction of MoM starts from a finite set of modules defined by multi-head attention and feed-forward networks, each distinguished by its unique parameterization. Two routers then iteratively select attention modules and feed-forward modules from the set to process a token. The selection dynamically expands the computation graph in the forward pass of the token, culminating in an assembly of modules.
We show that MoM provides not only a unified framework for Transformers and their numerous variants but also a flexible and learnable approach for reducing redundancy in Transformer parameterization.
We pre-train various MoMs using OpenWebText. Empirical results demonstrate that MoMs, of different parameter counts, consistently outperform vanilla transformers on both GLUE and XSUM benchmarks.
More interestingly, with a fixed parameter budget, MoM-large enables an over 38% increase in depth for computation graphs compared to GPT-2-large, resulting in absolute gains of 1.4 on GLUE and 1 on XSUM. On the other hand, MoM-large also enables an over 60% reduction in depth while involving more modules per layer, yielding a 16% reduction in TFLOPs and a 43% decrease in memory usage compared to GPT-2-large, while maintaining comparable performance.111Code is available athttps://github.com/gzhch/MoM
abstract
Mixture-of-Modules:Reinventing Transformers as Dynamic Assemblies of Modules
Abstract
2407.06677
1IntroductionTransformer-based language modelshave demonstrated remarkable abilities across a wide range of challenging natural language tasks.
In addition, the success of Transformer in natural language processing (NLP)  is also inspiring innovations in other fields such as computer visionand biomedicine.
A Transformer architecture typically consists of stacked layers that are identical in structure, whereby layers are organized in the order of depth, using the output of the previous layer as the input for the next. While this design convention has been widely accepted as a matter of course in the Transformer era, we challenge it by reconsidering whether the static and depth-ordered organization can fully unleash the potential of Transformers, given the well-known issues of over-parameterizationand efficiency.Before us, some rudimentary studies have touched on the question–they dissect Transformer into modules such as attention heads and feed-forward networks (FFNs)  and allow relatively flexible module call order.
For example, Mixture-of-Experts (MoE) )  sets up multipleFFNs within the same layer and activates a specific subset during inference.
Early-exitingand Mixture-of-Depths (MoD) )  bypass certain layers when computing each token. On the one hand, these efforts indeed lead to improvements in terms of either efficacy or efficiency through the introduction of dynamic mechanisms into the vanilla structure of Transformers, and thus corroborate our questioning regarding the established convention; on the other hand, they still follow the depth-ordered paradigm (i.e., tokens are passed from shallow layers to deep layers) , leaving significant room for better architectures.Figure 1:Mixture-of-Modules reinvents Transformers as dynamic assemblies of modules.
In (b) , we illustrate the ongoing construction of an MoM model during the forward computation.
The assembly lastsrounds, with the current illustration showcasing progress in the third round.
For each token, routers select the bestattention modules, denoted as, and the bestfeed-forward network modules, denoted as, from a module set(including “SKIP” modules) .
These selected modules collectively constitute assembled modulesand, which are then appended to the existing computation graph. Detailed notations are presented in §3.In this work, we completely disrupt the traditional practice in the design of Transformers by breaking down the depth-ordered organization. Numerous studies have indicated that knowledge in Transformers is often dispersed across multipleFFNs in different layers, and many attention heads serve similar or identical functions, such as copying specific token information towards the end position of the input.
Encouraged by this evidence, we pose the question of whether the computation of a token can “move” freely across layers, that is the token can be computed by flowing to modules in deeper layers, sticking to modules of the same layer, or even going back to modules in previous layers.
To answer the question, we propose a novel architecture dubbed Mixture-of-Modules (MoM)  in which the core idea is to define a neural network as dynamic assemblies of modules derived from vanilla Transformer, as depicted in Figure1.The basis of MoM is a finite set of modules.
Each module is defined by multi-head attention (MHA) , a feed-forward network (FFN)  (including Add & Norm) , or a specialized module labeled “SKIP”.
EachMHAorFFNmodule is identical in structure and different in parameterization.SKIPenables skip operations for arbitrary tokens at arbitrary time steps.
Given a token, each time two routers select modules from the set and integrate the modules into the computation graph during the forward pass.
Hence, the whole computation graph of the token is formed as an assembly of modules, and the routers learn to optimize the organization of the modules in the assembly.
We introduce a two-phase approach for training MoM models. In the first phase, we pre-train a vanilla Transformer on a large-scale corpus. Then, in the second phase, we decompose the pre-trained Transformer into modules as a warm-up of MoM, randomly initialize the routers, and further update both the modules and the routers under the mechanism of dynamic assembly. By this means, we can both enhance parameter utilization and accelerate the convergence of the model.MoM has three major advantages over existing Transformer-based architectures: (1)  it provides a unified framework for various Transformer variants, incorporating popular methods such as mixture-of-experts, early-exiting, and mixture-of-depths as special cases.
The framework sheds light on architecture design for future works; (2)  it brings unprecedented flexibility in forward computation. With the dynamic assembly mechanism, “depth” and “parameter count” are no longer entangled as they are in the conventional sense. One can build powerful architectures by either enlarging the module pool (i.e., increasing parameter count)  or increasing the depth (parameter count can be fixed) .
Hence, MoM offers a dynamic and learnable approach to reducing redundant parameters in Transformers; and (3)  it offers efficient structures that achieve performance comparable to vanilla Transformers but require significantly fewer FLOPs and less memory in forward computation.We pre-train MoM in three sizes–122M (small) , 346M (medium) , and 774M (large) – using OpenWebText, and assess their performance with GLUEand XSUM. Empirical results indicate that (1)  MoMs, across all the three parameter counts, consistently outperform vanilla GPT-2 models on both text understanding and generation tasks;
(2)  parameters are quite redundant in vanilla Transformers. One can develop an MoM that is at least 30% deeper than a vanilla GPT-2, resulting in at least 1.4 absolute gain in GLUE and at least 1 absolute gain in XSUM. Moreover, one can further remove% of theMHAmodules and% of theFFNmodules from the module pool of the model, while maintain comparable performance222After removing% of theMHAmodules and% of theFFNmodules, the validation loss increased by, the GLUE average score dropped fromto, and the ROUGE average in XSUM dropped fromto.;
and (3)  for those concerned with efficiency, MoM-large can reduce TFLOPs by 16% and memory usage by 42% in forward computation, while maintaining comparable performance to GPT-2-large, via properly increasing the number of modules and compressing the model depth.Our contributions are three-fold: (1)  proposal of Mixture-of-Modules to disrupt the depth-ordered convention in Transformer construction, and reinvent Transformers as dynamic assemblies of modules; (2)  empirical verification of the efficacy of Mixture-of-Modules on GLUE and XSUM; and (3)  a series of new insights into the over-parameterization issue of vanilla Transformers, and their implications for future architecture design. The code for implementing MoM is open-sourced atLABEL:??.
introduction
Mixture-of-Modules:Reinventing Transformers as Dynamic Assemblies of Modules
1Introduction
2407.06677
2Related worksMoM owns a dynamic mechanism of module selection and combination, and thus is related to conditional computation techniques.
Existing work on conditional computation can be categorized into two groups:dynamic depthanddynamic width. In these fields, terms such as gating and routing are used interchangeably, hereafter referred to as “routers” for clarity in presentation.As a typical approach in dynamic depth,Early-exitingaccelerates model inference through terminating forward computation at intermediate layers. The decision to exit often relies on confidence-based metricsor pre-determined strategies. With some degree of generalization,Layer-skiprepresents a more adaptive variant of early-exiting, enabling certain layers to be skipped without terminating the entire forward computation. Existing works mainly facilitate it by training a routeror layer pruning. Finally, if we view parameter copying as a particular way to increase network depth with controlled model size, then some parameter sharing methods, wherein certain modules or layers share parameters, also fall in the dynamic depth group.In terms of dynamic width,Mixture-of-Experts(MoE,)  is a representative method.
An MoE model conceptualizes an FFN module as an “expert” for storing knowledge.
Comprising multiple such experts, an MoE layer replaces the traditional FFN layer within Transformers, aiming for superior performance in handling knowledge-related tasks.
During forward computation, a router network dynamically assigns each token to the topexperts out of a total ofexperts, thereby increasing the maximum network width bytimes. Other dynamic width methods, such as CODAand CoLT5, use similar routing mechanism to select whether a token passes through a heavy or light pathway for not only each FFN layer but also each attention layer.MoM breaks the depth-ordered paradigm followed by existing approaches when performing forward computation. It not only unifies a number of approaches described above but also offers a more flexible and learnable way to achieve conditional computation.
body
Mixture-of-Modules:Reinventing Transformers as Dynamic Assemblies of Modules
2Related works
2407.06677
3MethodologyThe idea of Mixture-of-Modules (MoM)  is inspired by the theory presented in “the society of mind” by Marvin Minsky, which explains the true intelligence as certain and very special ways of combinations of simple and modular units (in the book, they are termed “agents”) .
In §3.1, we first provide an overview of MoM. Then, we detail the assembly of modules and the routers in §3.2and §3.3.
After that, we present the training procedure of MoM in §3.4.
Finally in §3.5, we show that MoM unifies various techniques of dynamic computation allocation within Transformers as special cases.3.1Mixture-of-Modules (MoM) Before delving into the details, we first give a brief description of the workflow of MoM. MoM views the construction of an-depth transformer as an-step iterative assembly process. In each assembly step, routerdynamically selectsmodules from a module setfor each token. Then these selected modules are assembled guided by the assembling function.
Formally, MoM can be defined by a 5-tuple.is the set that contains all possible modules, where modules are defined as atomic units that could be assembled.
There are two types of modules in a Transformer model,i.e., the multi-head self-attention module (MHA)  and the feed-forward network module (FFN) , denoted asand, respectively.
In addition, denote the input hidden state of a Transformer layer as, we include a special modulein, which means the absence of an operation applied to the token, allowing for skipping one round of computation.
Therefore, in MoM, we have:(1) wherearerefer to numbers ofMHAs andFFNs, respectively.is a router responsible for dynamically selecting appropriate modules fromand assembling them into the computation graph.
We use distinct routers forMHAs andFFNs, denoted asandrespectively.
The output of the routeris an-dimensional distribution wherein each item represents the weight assigned to each moduleas well as.
Formally:(2) An MoM model is dynamically assembled step by step. In each step, based on the output of,modules are selected fromand assembled together. The assembly process lastssteps, as detailed in §3.2.
Further elaboration on these routers, including their architecture and the working pipeline, is deferred to §3.3.
Notably, in MoM, dynamic assembly occurs at the token level, wherein each token is independently and dynamically assigned by routers to appropriate modules for processing.3.2Dynamic assembly of modulesWe delve into how an MoM model is dynamically assembled.
The construction is an iterative process where in the-th step (i.e., the-th layer of the model being constructed) , we have the input.
The subscriptis omitted when there’s no ambiguity.
The routerselectsmodules with the largest routing weight.
We denote the indices of the selected modules as.
Then the selected modules are assembled together through the assembling function. Formally,(3) whererepresents the assembled modules.
These assembled modules transform the inputinto the output.
We hope the role of the-th step of MoM assembly is somewhat akin to the-th Transformer block in the conventional sense.
Therefore, we establish two rounds of routing and assembling in each assembly step: one forMHAand the other forFFN.
The forward computation of MoM models at the-th step assembly can be represented as:(4) We employ Pre-norm in MoM, which normalizes the input before feeding to assembled modules.
The dynamic assembly process is depicted in Figure1(b) .
We now introduce the detailed formalization forand, respectively.Assembly of attention modules() .  We begin by considering the scenario where themodule is not selected by routers.
Suppose that anMHAmodule containsindividual heads, then
the assembly ofMHAmodules (i.e., the computation process of)  is defined as:(5) whereis the input representation of the sequence,, andare weight matrices with. When themodule is selected, the operation ofonly involves the remainingattention modules.Assembly of feed-forward networks() .  The assembly ofis more modular, where the outputs ofmodules are simply weighted and aggregated.
When themodule is not selected,can be formalized as follows:(6) Whenis chosen, likewise, onlyFFNs form the.3.3MoM router () In prior approaches, routing occurs as a one-step decision-making process within a layer.
However, in MoM  which possesses a dynamically constructed computation graph, each decision is interdependent with the preceding ones, influencing the entire forward computation.
Consequently, the router in MoM necessitates an awareness of past decisions.
To model such dependency, we employ a gated recurrent unit (GRU,)  as the backbone of routers.
Two routers in MoM are identical in structure.
At each assembly step, the GRU in themaintains anas the hidden state of the GRU network.
This state is recurrently updated as follows:(7) The weights assigned to each module byare computed as:(8) 3.4Training approachA straightforward approach is to pre-train an MoM model initialized from scratch.
This approach, however, suffers from a degeneration issue, as the learned functions of modules become homogeneous, making router training challenging.
To address the issue, we propose a two-phase training approach. In the first phase, we pre-train a vanilla Transformer where modules acquire distinct functionalities. Then, in the second phase, we initialize the module setwith the pre-trained modules and initialize the routers from scratch. Subsequently, we continue training both modules and routers using the same data and objective as in the first phase. Through empirical studies, we find that the two-phase training method improves the specialization of module functionalities and accelerates router convergence.3.5MoM as a unified frameworkA compelling property of MoM is that it unifies a wide range of Transformer-based dynamic computation allocation architectures. With specific configurations, layer-skip (e.g., early-exiting, mixture-of-depths, etc.) , parameter sharing, and mixture-of-experts can be viewed as special cases.Layer-skip.The key idea is to skip layers according to certain criteria which can either be defined heuristicallyor learned from data.
Within the MoM framework, layer-skip can be formulated as a special cluster of assembly functions, namely:(9) whereis the criterion that decides whether to skip the-th layer or not. Note that the technique of early-exitingcan be viewed as a special case of layer-skip, where once a layer is skipped, all subsequent layers will be skipped too.Parameter sharing.We consider parameter sharing that shares weights across modules and does not involve reparameterization techniques. Under this restriction, the sharing paradigm can be defined as a criterion function, representing using the same weights for moduleand module. Within MoM, parameter sharing can be formulated as:(10) Mixture-of-Experts.MoE splitsFFNinto experts, and the experts are not shared across different layers. In MoE, routing is only performed onFFNmodules, thus the computation ofMHAis the same as that of a vanilla Transformer. The assembly function for MoE can be written as:(11) whereis the collection of experts for layer.Figure2illustrates the forward computation process across different methods, offering an intuitive presentation of the versatility and universality in MoM.Figure 2:Visualization of forward computation of five models, where each consists of only two layers just for demonstration purposes.
The switch icon symbolizes the selective execution of one (in Layer-skip)  or more (in MoE and MoM)  subsequent computation pathways.
body
Mixture-of-Modules:Reinventing Transformers as Dynamic Assemblies of Modules
3Methodology
2407.06677
4Experiments4.1Experimental setupWe implement language models with MoM since language modeling is a challenging task requiring both language understanding and generation ability, thereby effectively evaluating MoM and baselines.
Below, we elaborate on the implementation details of MoM, baselines, and evaluation setups.Implementation details.We conduct experiments on three model scales, which we denote as MoM-small, MoM-medium, and MoM-large, respectively. These models contain 122M, 346M, and 774M parameters, respectively. Detailed configurations can be found in AppendixA.
The vanilla Transformers used for initializing MoM are official GPT-2 checkpoints downloaded from HuggingFace333https://huggingface.co/openai-community/gpt2.andrepresent two hyper-parameters of MoM. We denote a configuration whereas KH. If the skip module is included in, we append the suffix S to KH.In the two-phase training, we exploit OpenWebTextas the pre-training dataset, and pre-process the data with the same pipeline as nanoGPT.
OpenWebText containsbillion tokens after tokenization, from whichmillion tokens are randomly sampled as the validation set.
The training sequence length for every input is.
We set the learning rate toe-with a warm-up ratio of 0.1 throughout the two phases, and do not use dropout.
All models are trained on 8A100 GPUs with a total batch size of 864.
Two training phases requirek andk optimization steps, respectively.In practice, considering the large search space characterized byand, we confine the architecture search space to a practical scale with a “chunking” strategy. An MoM is divided into several chunks. Each chunk is independent and parameterized with identicaland.
We present a detailed description and specific configuration in AppendixA, and an empirical analysis of the efficacy of chunking in AppendixB.MethodsMoM ConfigParameterComputationMemoryValidationValidationGLUEXSUM(KH) CountCost (TFLOPs) Cost (Gb) LossPerplexity(Average) (Average) smallGPT2K1H4122M2.922.983.1022.2275.3214.26MoDK1H4S122M--3.2225.1172.249.71MoEK2H4283M3.81 (+30.5%) 2.98 (+0.0%) 3.0721.1877.2514.18MoE (share) K2H4122M3.81 (+30.5%) 2.98 (+0.0%) 3.1423.4175.8214.15K3H1S122M2.45 (-16.1%) 2.45 (-17.8%) 3.1623.5975.9214.17K3H2S122M3.49 (+19.5%) 2.63 (-11.7%) 3.0320.7977.8114.24K2H6S122M5.04 (+72.6%) 3.34 (+12.1%) 2.9819.5978.2215.19mediumGPT2K1H4346M8.284.742.8116.6980.4918.14MoDK1H4S346M--2.9919.8276.1714.81MoEK2H4921M11.37 (+37.3%) 4.74 (+0.0%) 2.8016.5380.4717.75MoE (share) K2H4346M11.37 (+37.3%) 4.74 (+0.0%) 2.8216.8180.3517.59K3H1S346M6.80 (-17.9%) 3.33 (-29.7%) 2.8316.9180.4117.11K3H2S346M10.20 (+23.2%) 3.80 (-19.8%) 2.7715.8981.0318.66K2H6S346M16.23 (+96.0%) 5.69 (+20.0%) 2.7215.1881.9319.30largeGPT2K1H4774M17.767.202.6614.3384.4720.35MoDK1H4S774M--2.8116.6281.4918.62MoEK2H42100M25.43 (+43.2%) 7.20 (+0.0%) 2.6414.1784.4320.63MoE (share) K2H4774M25.43 (+43.2%) 7.20 (+0.0%) 2.6514.2283.8320.39K3H1S774M14.84 (-16.4%) 4.13 (-42.6%) 2.6614.5083.3920.46K3H2S774M20.31 (+14.5%) 5.15 (-28.5%) 2.6413.9284.4921.73K2H6S774M36.07 (+103.1%) 9.24 (+28.3%) 2.6013.2185.9022.36Table 1:Comprehensive comparison between MoMs and baselines.
We highlight the best results in bold and underline the second-best results.
AppendixCincludes the detailed performance on GLUE and XSUM.Baselines.In addition to the vanilla Transformer model, the following models (or methods)  are also employed as baselines: (1) MoD: a layer-skip method proposed recently that dynamically routes around Transformer blocks.444As official code is unavailable until the submission, we follow the paper to implement MoD ourselves.(2) MoE: the mixture-of-experts architecture utilized by Mixtral. We implement the model with the open-sourced code.555https://github.com/huggingface/transformers/blob/v4.36.1/src/transformers/models/mixtral/modeling_mixtral.py(3) MoE (share) : a variant of MoE in which all layers share the same set of experts. We involve this model as a baseline because unlike the standard MoE that has more parameters, MoE-share has the same number of parameters with the vanilla Transformer model, making the comparison more fair. Moreover, it also sheds light on how well the MoE architecture can utilize a fixed budget of modules.Note that all the above-mentioned methods are special cases within the MoM framework with various configurations.
Furthermore, we explore a wide range of MoM instances defined by KHS, whereand.
We examine all MoM instances within the search space (as detailed in §4.3) , and spotlight three distinct models for comparison against other baselines:(K2H6S)  represents aPerformant MoM model after tuningand.(K3H1S)  significantly enhancesEfficiency compared to vanilla Transformers, while maintaining acceptable performance.(K3H2S)  serves as a midpoint in configurations between the two preceding models. This model is positioned betweenand. We aim to highlight performance and efficiencyInterpolation as feature of MoM through configuration interpolation.Evaluation settings.We employ GLUE benchmarkto evaluate the language understanding ability and XSUMto evaluate the text generation ability. All models are fine-tuned with a learning rate ofe-. The sequence isfor GLUE andfor XSUM. For smaller GLUE sub-datasets (CoLA, STS-B, MRPC, and RTE) , we set the batch size to 32 and train for 3 epochs. For larger datasets (MNLI, QNLI, QQP, and SST-2) , we utilize a batch size of 64 and perform training for a total ofgradient steps. For XSUM, we set the batch size to 64 and train for 3 epochs. For efficiency evaluation, we report inference TFLOPs and memory usage.
TFLOPs are calculated using DeepSpeed FLOPs profilerand memory consumption is calculated with PyTorch toolkits.4.2Main resultsTable1reports the evaluation results.
Our analysis yields the following conclusions:MoM unleashes the potential of Transformers and our initial motivation is confirmed.When maintaining the number of parameters,is characterized by the deepest computation graph () .
Across all model scales,consistently outperforms all baselines on both GLUE and XSUM by significant margins.
The enhanced performance ofvalidates our initial motivations: (1)  the traditional depth-ordered layer organization is sub-optimal;
(2)  improvements can be realized through two key modifications to the computation graph, includingdynamic module organizationandimproved parameter utilization.is characterized by its minimum depth () .
By strategically selecting appropriate modules at each assembly step,strives to reduce memory and computation costs while maintaining performance.
Althoughis slightly surpassed by a vanilla Transformer, it outperforms MoD, another efficiency-driven method by large margins.Besides, we observe thatarchives a decent performance by slightly outperforming vanilla GPT-2.
Comparing to vanilla GPT-2,consumes no more than 25% extra computation but save at least 11.7% memory across all scales, indicating thatachieves a good balance between performance and efficiency.Figure 3:How validation loss varies with respect toand, comparing to MoM (medium)  with.MoM models provide insights into the over-parameterization issue.With GPT-2-medium as initialization,
we develop a series of MoM models, each defined by different pairs of () , with both values not exceeding.
We assess the validation loss increase for each model relative to the benchmark model whereand, as illustrated in Figure3.
In this experiment, we setequal to the number of modules to make full use of the module parameters.
Interestingly, theFFNandMHAmodules exhibit different degrees of redundancy.
Specifically, whenis fixed and the number ofMHAs is gradually reduced, a significant increase in loss is not observed untilis reduced fromto, suggesting considerable redundancy in theMHAs of Transformers.
In contrast, when fixingand reducing the number ofFFNs gradually, each time of removing anFFNleads to evident loss increase, indicatingFFNs are less over-parameterized.
These quantitative findings align with previous research suggesting that the parameterization of attention can be simplified to enhance efficiency while maintaining performance.As the parameter size scales up, MoM models enjoy consistent gains in both performance and efficiency.When we look into the difference across different scales, we observe that (1)  the performance gain of MoM is stable; (2) -medium and-large exhibit more significant reductions in resource costs comparing to-small.
These observations across different scales reinforce our previous motivation: Transformers are over-parameterized, which becomes more evident as the model size increases.4.3Insights from hyper-parameter searchFigure4shows how the validation loss for MoM-small and MoM-medium varies with respect to different settings ofand(,) . From this experiment, we have the following observations and insights: (1)  allowing more modules to be assembled at each step (i.e., larger)  and more rounds of assembling actions (i.e., larger)  generally leads to better performance, indicating thatTransformer-based models benefit from a larger computation graph even if the parameter size remains the same. However, (2)  the benefits of increasingandbecome marginal whenand.
Comparing K3H1 to K2H6, we can see that the validation loss is comparable, while K3H1 performs slightly worse on downstream tasks as discussed in §4.2. However, K3H2 improves efficiency by flattening the depth, making it a good choice that balances performance and efficiency.
Flattening modules from different depths to the same depth cancels computation dependencies of each other. This characteristic brings an extra benefit because the computation of modules from the same depth can be parallelized.
This technique has been validated and adopted in MoE applications(called expert parallelism)  and can be easily extended to further accelerate MoM (K3H2) .Figure 4:Validation loss for MoM-small and MoM-medium under different settings ofand.Phase 1Phase 2Val. LossPPL.(1) MoM-2.8517.26(2) Vanilla-2.8116.69(3) MoMMoM2.7816.20(4) VanillaMoM2.7215.18(5) VanillaMoM-same3.1322.87Table 2:Performance comparison under different training setups. When training MoM from scratch, we set the total gradient steps of phase 1 to 20k. The total steps of phase 2 for all ablations are 10k.4.4Impact of two-phase trainingWe investigate how the two-phase training strategy influences model performance using-medium as case studies. Specifically, we consider the following training strategies: (1)  training MoM from scratch with 20k steps; (2)  training vanilla Transformer with 20k steps; (3)  training MoM from scratch with 30k steps; (4) -medium that is trained with the two-phase strategy; and (5)  training MoM with the two-phase strategy but in the second phase,MHAmodules andFFNmodules are forced to be identical, respectively. Table2shows the results. First, comparing (1)  with (2) , we find that the absence of weight initialization compromises the training quality of MoM, making it worse than the vanilla Transformer, underscoring the importance of initializing module weights with a well-trained vanilla Transformer model for MoM. A plausible explanation for the results is that training without warm-up leads to homogeneous modules that hurt the convergence of the routers (as illustrated in AppendixD) . The explanation is further justified by comparison between (4)  and (5) , as when we force all modules to be identical, the performance of MoM also drops dramatically even with the two-phase training strategy. As expected, training with more steps can enhance performance (cf., comparison between (1)  and (3) ) , but the two-phase strategy is still the better choice when we compare (3)  with (4) .
body
Mixture-of-Modules:Reinventing Transformers as Dynamic Assemblies of Modules
4Experiments
2407.06677
5ConclusionsIn this work, we propose Mixture-of-Modules (MoM) , a novel architecture that reinvents transformers as a collection of individual modules and the dynamic assembly process conducted with these modules. This novel view offers us an opportunity to explore a wide range of different configurations of model architecture and unify a series of transformer variants. With exhaustive experiments, we not only validate the effectiveness of MoM by both significant efficiency and performance gains but also reach new insights about Transformers.
body
Mixture-of-Modules:Reinventing Transformers as Dynamic Assemblies of Modules
5Conclusions
2407.06677
LimitationsOur current design of the router still has room for improvement. Unlike MoE wherein the router makes one-time decisions about which experts to select, the router of MoM is responsible for conducting multi-step decision-making. In this scenario, instructing the router to make correct decisions continuously is a hard problem since the decision space grows exponentially with the increase of assembly steps. The current implementation has not considered this question and has not explicitly encouraged or discouraged the router to make some choices, thus, we are not sure whether the learned routing decisions are optimal or not. In the future, we will explore using techniques like reinforcement learning or neural architecture search to design more sophisticated routers.
conclusion
Mixture-of-Modules:Reinventing Transformers as Dynamic Assemblies of Modules
Limitations