<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2407.04153] Mixture of A Million Experts</title><meta property="og:description" content="The feedforward (FFW) layers in standard transformer architectures incur a linear increase in computational costs and activation memory as the hidden layer width grows. Sparse mixture-of-experts (MoE) architectures hav…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Mixture of A Million Experts">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Mixture of A Million Experts">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2407.04153">

<!--Generated on Mon Aug  5 13:03:08 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Mixture of A Million Experts</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname"><span id="id1.1.id1" class="ltx_text ltx_font_bold">Xu Owen He  </span><span id="id2.2.id2" class="ltx_text ltx_font_italic" style="font-size:90%;">hexu@google.com 
<br class="ltx_break">Google DeepMind</span>
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id3.id1" class="ltx_p">The feedforward (FFW) layers in standard transformer architectures incur a linear increase in computational costs and activation memory as the hidden layer width grows. Sparse mixture-of-experts (MoE) architectures have emerged as a viable approach to address this issue by decoupling model size from computational cost. The recent discovery of the fine-grained MoE scaling law shows that higher granularity leads to better performance. However, existing MoE models are limited to a small number of experts due to computational and optimization challenges. This paper introduces PEER (parameter efficient expert retrieval), a novel layer design that utilizes the product key technique for sparse retrieval from a vast pool of tiny experts (over a million). Experiments on language modeling tasks demonstrate that PEER layers outperform dense FFWs and coarse-grained MoEs in terms of performance-compute trade-off. By enabling efficient utilization of a massive number of experts, PEER unlocks the potential for further scaling of transformer models while maintaining computational efficiency.</p>
</div>
<figure id="S0.F1" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S0.F1.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.04153/assets/figs/isoflops6e18.png" id="S0.F1.sf1.g1" class="ltx_graphics ltx_img_square" width="318" height="308" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S0.F1.sf1.4.1.1" class="ltx_text" style="font-size:80%;">(a)</span> </span><math id="S0.F1.sf1.2.m1.1" class="ltx_Math" alttext="6e18" display="inline"><semantics id="S0.F1.sf1.2.m1.1b"><mrow id="S0.F1.sf1.2.m1.1.1" xref="S0.F1.sf1.2.m1.1.1.cmml"><mn mathsize="80%" id="S0.F1.sf1.2.m1.1.1.2" xref="S0.F1.sf1.2.m1.1.1.2.cmml">6</mn><mo lspace="0em" rspace="0em" id="S0.F1.sf1.2.m1.1.1.1" xref="S0.F1.sf1.2.m1.1.1.1.cmml">​</mo><mi mathsize="80%" id="S0.F1.sf1.2.m1.1.1.3" xref="S0.F1.sf1.2.m1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S0.F1.sf1.2.m1.1.1.1b" xref="S0.F1.sf1.2.m1.1.1.1.cmml">​</mo><mn mathsize="80%" id="S0.F1.sf1.2.m1.1.1.4" xref="S0.F1.sf1.2.m1.1.1.4.cmml">18</mn></mrow><annotation-xml encoding="MathML-Content" id="S0.F1.sf1.2.m1.1c"><apply id="S0.F1.sf1.2.m1.1.1.cmml" xref="S0.F1.sf1.2.m1.1.1"><times id="S0.F1.sf1.2.m1.1.1.1.cmml" xref="S0.F1.sf1.2.m1.1.1.1"></times><cn type="integer" id="S0.F1.sf1.2.m1.1.1.2.cmml" xref="S0.F1.sf1.2.m1.1.1.2">6</cn><ci id="S0.F1.sf1.2.m1.1.1.3.cmml" xref="S0.F1.sf1.2.m1.1.1.3">𝑒</ci><cn type="integer" id="S0.F1.sf1.2.m1.1.1.4.cmml" xref="S0.F1.sf1.2.m1.1.1.4">18</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S0.F1.sf1.2.m1.1d">6e18</annotation></semantics></math><span id="S0.F1.sf1.5.2" class="ltx_text" style="font-size:80%;"> FLOPs</span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S0.F1.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.04153/assets/figs/isoflops2e19.png" id="S0.F1.sf2.g1" class="ltx_graphics ltx_img_square" width="318" height="308" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S0.F1.sf2.4.1.1" class="ltx_text" style="font-size:80%;">(b)</span> </span><math id="S0.F1.sf2.2.m1.1" class="ltx_Math" alttext="2e19" display="inline"><semantics id="S0.F1.sf2.2.m1.1b"><mrow id="S0.F1.sf2.2.m1.1.1" xref="S0.F1.sf2.2.m1.1.1.cmml"><mn mathsize="80%" id="S0.F1.sf2.2.m1.1.1.2" xref="S0.F1.sf2.2.m1.1.1.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S0.F1.sf2.2.m1.1.1.1" xref="S0.F1.sf2.2.m1.1.1.1.cmml">​</mo><mi mathsize="80%" id="S0.F1.sf2.2.m1.1.1.3" xref="S0.F1.sf2.2.m1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S0.F1.sf2.2.m1.1.1.1b" xref="S0.F1.sf2.2.m1.1.1.1.cmml">​</mo><mn mathsize="80%" id="S0.F1.sf2.2.m1.1.1.4" xref="S0.F1.sf2.2.m1.1.1.4.cmml">19</mn></mrow><annotation-xml encoding="MathML-Content" id="S0.F1.sf2.2.m1.1c"><apply id="S0.F1.sf2.2.m1.1.1.cmml" xref="S0.F1.sf2.2.m1.1.1"><times id="S0.F1.sf2.2.m1.1.1.1.cmml" xref="S0.F1.sf2.2.m1.1.1.1"></times><cn type="integer" id="S0.F1.sf2.2.m1.1.1.2.cmml" xref="S0.F1.sf2.2.m1.1.1.2">2</cn><ci id="S0.F1.sf2.2.m1.1.1.3.cmml" xref="S0.F1.sf2.2.m1.1.1.3">𝑒</ci><cn type="integer" id="S0.F1.sf2.2.m1.1.1.4.cmml" xref="S0.F1.sf2.2.m1.1.1.4">19</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S0.F1.sf2.2.m1.1d">2e19</annotation></semantics></math><span id="S0.F1.sf2.5.2" class="ltx_text" style="font-size:80%;"> FLOPs</span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>Isoflop comparison on the C4 dataset between PEER and other baselines with two different FLOP budgets (<math id="S0.F1.4.m1.1" class="ltx_Math" alttext="6e18" display="inline"><semantics id="S0.F1.4.m1.1b"><mrow id="S0.F1.4.m1.1.1" xref="S0.F1.4.m1.1.1.cmml"><mn id="S0.F1.4.m1.1.1.2" xref="S0.F1.4.m1.1.1.2.cmml">6</mn><mo lspace="0em" rspace="0em" id="S0.F1.4.m1.1.1.1" xref="S0.F1.4.m1.1.1.1.cmml">​</mo><mi id="S0.F1.4.m1.1.1.3" xref="S0.F1.4.m1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S0.F1.4.m1.1.1.1b" xref="S0.F1.4.m1.1.1.1.cmml">​</mo><mn id="S0.F1.4.m1.1.1.4" xref="S0.F1.4.m1.1.1.4.cmml">18</mn></mrow><annotation-xml encoding="MathML-Content" id="S0.F1.4.m1.1c"><apply id="S0.F1.4.m1.1.1.cmml" xref="S0.F1.4.m1.1.1"><times id="S0.F1.4.m1.1.1.1.cmml" xref="S0.F1.4.m1.1.1.1"></times><cn type="integer" id="S0.F1.4.m1.1.1.2.cmml" xref="S0.F1.4.m1.1.1.2">6</cn><ci id="S0.F1.4.m1.1.1.3.cmml" xref="S0.F1.4.m1.1.1.3">𝑒</ci><cn type="integer" id="S0.F1.4.m1.1.1.4.cmml" xref="S0.F1.4.m1.1.1.4">18</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S0.F1.4.m1.1d">6e18</annotation></semantics></math> and <math id="S0.F1.5.m2.1" class="ltx_Math" alttext="2e19" display="inline"><semantics id="S0.F1.5.m2.1b"><mrow id="S0.F1.5.m2.1.1" xref="S0.F1.5.m2.1.1.cmml"><mn id="S0.F1.5.m2.1.1.2" xref="S0.F1.5.m2.1.1.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S0.F1.5.m2.1.1.1" xref="S0.F1.5.m2.1.1.1.cmml">​</mo><mi id="S0.F1.5.m2.1.1.3" xref="S0.F1.5.m2.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S0.F1.5.m2.1.1.1b" xref="S0.F1.5.m2.1.1.1.cmml">​</mo><mn id="S0.F1.5.m2.1.1.4" xref="S0.F1.5.m2.1.1.4.cmml">19</mn></mrow><annotation-xml encoding="MathML-Content" id="S0.F1.5.m2.1c"><apply id="S0.F1.5.m2.1.1.cmml" xref="S0.F1.5.m2.1.1"><times id="S0.F1.5.m2.1.1.1.cmml" xref="S0.F1.5.m2.1.1.1"></times><cn type="integer" id="S0.F1.5.m2.1.1.2.cmml" xref="S0.F1.5.m2.1.1.2">2</cn><ci id="S0.F1.5.m2.1.1.3.cmml" xref="S0.F1.5.m2.1.1.3">𝑒</ci><cn type="integer" id="S0.F1.5.m2.1.1.4.cmml" xref="S0.F1.5.m2.1.1.4">19</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S0.F1.5.m2.1d">2e19</annotation></semantics></math> FLOPs). The <math id="S0.F1.6.m3.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S0.F1.6.m3.1b"><mi id="S0.F1.6.m3.1.1" xref="S0.F1.6.m3.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S0.F1.6.m3.1c"><ci id="S0.F1.6.m3.1.1.cmml" xref="S0.F1.6.m3.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S0.F1.6.m3.1d">x</annotation></semantics></math> axis is in log scale.</figcaption>
</figure>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para ltx_noindent">
<p id="S1.p1.1" class="ltx_p">The past few years have seen the power of scaling <cite class="ltx_cite ltx_citemacro_citep">(Kaplan et al., <a href="#bib.bib23" title="" class="ltx_ref">2020</a>; Hoffmann et al., <a href="#bib.bib21" title="" class="ltx_ref">2022</a>)</cite>: increasing the number of parameters, amount of training data, or the computational budget has proven to be a reliable way to improve model performance. Notably, feedforward (FFW) layers, responsible for storing factual knowledge <cite class="ltx_cite ltx_citemacro_citep">(Geva et al., <a href="#bib.bib19" title="" class="ltx_ref">2021</a>; Dai et al., <a href="#bib.bib11" title="" class="ltx_ref">2022</a>)</cite>, account for two-thirds of the total parameters in a transformer. However, one drawback of these dense FFWs is that their computational footprint (FLOPs and device memory consumption) is linearly proportional to their parameter count.</p>
</div>
<div id="S1.p2" class="ltx_para ltx_noindent">
<p id="S1.p2.1" class="ltx_p">To break the coupling between computational cost and parameter count, many recent works <cite class="ltx_cite ltx_citemacro_citep">(Shazeer et al., <a href="#bib.bib37" title="" class="ltx_ref">2017</a>; Lepikhin et al., <a href="#bib.bib29" title="" class="ltx_ref">2020</a>; Fedus et al., <a href="#bib.bib16" title="" class="ltx_ref">2022</a>; Zhou et al., <a href="#bib.bib41" title="" class="ltx_ref">2022</a>)</cite> have adopted the Mixture-of-Experts (MoE) architecture, which uses a set of sparsely activated expert modules (often FFWs) in place of a single dense FFW. <cite class="ltx_cite ltx_citemacro_citet">Clark et al. (<a href="#bib.bib8" title="" class="ltx_ref">2022</a>)</cite> studied the scaling law of MoE language models and showed that increasing the number of experts is an effective way to improve performance without increasing the inference cost. However, their experiments showed that the efficiency gains provided by MoEs plateau after a certain model size is reached. More recently, <cite class="ltx_cite ltx_citemacro_citet">Krajewski et al. (<a href="#bib.bib25" title="" class="ltx_ref">2024</a>)</cite> discovered that this plateau was caused by using a fixed number of training tokens. When the number of training tokens is compute-optimal, MoEs consistently outperform dense models in terms of FLOP efficiency. Moreover, they introduced granularity (the number of active experts) as a new scaling axis and empirically showed that using higher granularity improves performance. Extrapolating this fine-grained MoE scaling law suggests that continued improvement of model capacity will ultimately lead to a large model with high granularity, corresponding to an architecture of an immense number of tiny experts.</p>
</div>
<div id="S1.p3" class="ltx_para ltx_noindent">
<p id="S1.p3.1" class="ltx_p">Beyond efficient scaling, another reason to have a vast number of experts is lifelong learning, where MoE has emerged as a promising approach <cite class="ltx_cite ltx_citemacro_citep">(Aljundi et al., <a href="#bib.bib1" title="" class="ltx_ref">2017</a>; Chen et al., <a href="#bib.bib7" title="" class="ltx_ref">2023</a>; Yu et al., <a href="#bib.bib39" title="" class="ltx_ref">2024</a>; Li et al., <a href="#bib.bib30" title="" class="ltx_ref">2024</a>)</cite>. For instance, <cite class="ltx_cite ltx_citemacro_citet">Chen et al. (<a href="#bib.bib7" title="" class="ltx_ref">2023</a>)</cite> showed that, by simply adding new experts and regularizing them properly, MoE models can adapt to continuous data streams. Freezing old experts and updating only new ones prevents catastrophic forgetting and maintains plasticity by design. In lifelong learning settings, the data stream can be indefinitely long or never-ending <cite class="ltx_cite ltx_citemacro_citep">(Mitchell et al., <a href="#bib.bib32" title="" class="ltx_ref">2018</a>)</cite>, necessitating an expanding pool of experts.</p>
</div>
<div id="S1.p4" class="ltx_para ltx_noindent">
<p id="S1.p4.1" class="ltx_p">Although both efficient scaling and lifelong learning require MoE designs capable of handling a vast number of experts, to the best of our knowledge, the only architecture supporting more than ten thousands of experts is the Mixture of Word Experts (MoWE) <cite class="ltx_cite ltx_citemacro_citep">(dos Santos et al., <a href="#bib.bib14" title="" class="ltx_ref">2023</a>)</cite>. However, MoWE is language-specific and uses a fixed routing scheme. Theoretical and empirical evidence <cite class="ltx_cite ltx_citemacro_citep">(Clark et al., <a href="#bib.bib8" title="" class="ltx_ref">2022</a>; Dikkala et al., <a href="#bib.bib13" title="" class="ltx_ref">2023</a>)</cite> highlights the advantages of learned routers over non-trainable ones. Thus, an MoE design with a learned router scalable to over a million experts remains an open area for exploration.</p>
</div>
<div id="S1.p5" class="ltx_para ltx_noindent">
<p id="S1.p5.1" class="ltx_p">This work introduces the Parameter Efficient Expert Retrieval (PEER) architecture, leveraging product key retrieval <cite class="ltx_cite ltx_citemacro_citep">(Lample et al., <a href="#bib.bib27" title="" class="ltx_ref">2019</a>)</cite> for efficient routing to an extremely large number of experts, decoupling computational cost from parameter count. This design demonstrates a superior compute-performance trade-off in our experiments, positioning it as a competitive alternative to dense FFW layers for scaling foundation models. The main contributions of this work are:</p>
<ul id="S1.I1" class="ltx_itemize">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p"><span id="S1.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">Exploration of Extreme MoE Setting:</span> Deviating from the focus on a small number of large experts in previous MoE research, this work investigates the under-explored case of numerous tiny experts.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p"><span id="S1.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">Learned Index Structure for Routing:</span> Demonstrating for the first time that a learned index structure <cite class="ltx_cite ltx_citemacro_citep">(Kraska et al., <a href="#bib.bib26" title="" class="ltx_ref">2018</a>)</cite> can efficiently route to over a million experts.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.1" class="ltx_p"><span id="S1.I1.i3.p1.1.1" class="ltx_text ltx_font_bold">New Layer Design:</span> Combining product key routing with single-neuron experts, we introduce the PEER layer that expands layer capacity without significant computational overheads. Empirical results demonstrate its superior efficiency compared to dense FFW, coarse-grained MoEs and Product Key Memory (PKM) layers.</p>
</div>
</li>
<li id="S1.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i4.p1" class="ltx_para ltx_noindent">
<p id="S1.I1.i4.p1.1" class="ltx_p"><span id="S1.I1.i4.p1.1.1" class="ltx_text ltx_font_bold">Comprehensive Ablation Studies:</span> We investigate the impact of different design choices of PEER such as number of experts, active parameters, number of heads and query batch normalization on language modeling tasks.</p>
</div>
</li>
</ul>
</div>
<figure id="S1.F2" class="ltx_figure"><img src="/html/2407.04153/assets/x1.png" id="S1.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="438" height="161" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span><span id="S1.F2.14.1" class="ltx_text ltx_font_bold">Illustration of the PEER layer.</span> A PEER layer can be inserted in the middle of a transformer backbone or can be used to replace FFW layers. Given the state vector <math id="S1.F2.7.m1.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S1.F2.7.m1.1b"><mi id="S1.F2.7.m1.1.1" xref="S1.F2.7.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S1.F2.7.m1.1c"><ci id="S1.F2.7.m1.1.1.cmml" xref="S1.F2.7.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.F2.7.m1.1d">x</annotation></semantics></math> from the previous layer, a query network <math id="S1.F2.8.m2.1" class="ltx_Math" alttext="q" display="inline"><semantics id="S1.F2.8.m2.1b"><mi id="S1.F2.8.m2.1.1" xref="S1.F2.8.m2.1.1.cmml">q</mi><annotation-xml encoding="MathML-Content" id="S1.F2.8.m2.1c"><ci id="S1.F2.8.m2.1.1.cmml" xref="S1.F2.8.m2.1.1">𝑞</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.F2.8.m2.1d">q</annotation></semantics></math> maps it to a query vector <math id="S1.F2.9.m3.1" class="ltx_Math" alttext="q(x)" display="inline"><semantics id="S1.F2.9.m3.1b"><mrow id="S1.F2.9.m3.1.2" xref="S1.F2.9.m3.1.2.cmml"><mi id="S1.F2.9.m3.1.2.2" xref="S1.F2.9.m3.1.2.2.cmml">q</mi><mo lspace="0em" rspace="0em" id="S1.F2.9.m3.1.2.1" xref="S1.F2.9.m3.1.2.1.cmml">​</mo><mrow id="S1.F2.9.m3.1.2.3.2" xref="S1.F2.9.m3.1.2.cmml"><mo stretchy="false" id="S1.F2.9.m3.1.2.3.2.1" xref="S1.F2.9.m3.1.2.cmml">(</mo><mi id="S1.F2.9.m3.1.1" xref="S1.F2.9.m3.1.1.cmml">x</mi><mo stretchy="false" id="S1.F2.9.m3.1.2.3.2.2" xref="S1.F2.9.m3.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.F2.9.m3.1c"><apply id="S1.F2.9.m3.1.2.cmml" xref="S1.F2.9.m3.1.2"><times id="S1.F2.9.m3.1.2.1.cmml" xref="S1.F2.9.m3.1.2.1"></times><ci id="S1.F2.9.m3.1.2.2.cmml" xref="S1.F2.9.m3.1.2.2">𝑞</ci><ci id="S1.F2.9.m3.1.1.cmml" xref="S1.F2.9.m3.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F2.9.m3.1d">q(x)</annotation></semantics></math>, which is then compared with the product keys to compute the router scores and to retrieve the top <math id="S1.F2.10.m4.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S1.F2.10.m4.1b"><mi id="S1.F2.10.m4.1.1" xref="S1.F2.10.m4.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S1.F2.10.m4.1c"><ci id="S1.F2.10.m4.1.1.cmml" xref="S1.F2.10.m4.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S1.F2.10.m4.1d">k</annotation></semantics></math> experts <math id="S1.F2.11.m5.3" class="ltx_Math" alttext="e_{1},...,e_{k}" display="inline"><semantics id="S1.F2.11.m5.3b"><mrow id="S1.F2.11.m5.3.3.2" xref="S1.F2.11.m5.3.3.3.cmml"><msub id="S1.F2.11.m5.2.2.1.1" xref="S1.F2.11.m5.2.2.1.1.cmml"><mi id="S1.F2.11.m5.2.2.1.1.2" xref="S1.F2.11.m5.2.2.1.1.2.cmml">e</mi><mn id="S1.F2.11.m5.2.2.1.1.3" xref="S1.F2.11.m5.2.2.1.1.3.cmml">1</mn></msub><mo id="S1.F2.11.m5.3.3.2.3" xref="S1.F2.11.m5.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S1.F2.11.m5.1.1" xref="S1.F2.11.m5.1.1.cmml">…</mi><mo id="S1.F2.11.m5.3.3.2.4" xref="S1.F2.11.m5.3.3.3.cmml">,</mo><msub id="S1.F2.11.m5.3.3.2.2" xref="S1.F2.11.m5.3.3.2.2.cmml"><mi id="S1.F2.11.m5.3.3.2.2.2" xref="S1.F2.11.m5.3.3.2.2.2.cmml">e</mi><mi id="S1.F2.11.m5.3.3.2.2.3" xref="S1.F2.11.m5.3.3.2.2.3.cmml">k</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S1.F2.11.m5.3c"><list id="S1.F2.11.m5.3.3.3.cmml" xref="S1.F2.11.m5.3.3.2"><apply id="S1.F2.11.m5.2.2.1.1.cmml" xref="S1.F2.11.m5.2.2.1.1"><csymbol cd="ambiguous" id="S1.F2.11.m5.2.2.1.1.1.cmml" xref="S1.F2.11.m5.2.2.1.1">subscript</csymbol><ci id="S1.F2.11.m5.2.2.1.1.2.cmml" xref="S1.F2.11.m5.2.2.1.1.2">𝑒</ci><cn type="integer" id="S1.F2.11.m5.2.2.1.1.3.cmml" xref="S1.F2.11.m5.2.2.1.1.3">1</cn></apply><ci id="S1.F2.11.m5.1.1.cmml" xref="S1.F2.11.m5.1.1">…</ci><apply id="S1.F2.11.m5.3.3.2.2.cmml" xref="S1.F2.11.m5.3.3.2.2"><csymbol cd="ambiguous" id="S1.F2.11.m5.3.3.2.2.1.cmml" xref="S1.F2.11.m5.3.3.2.2">subscript</csymbol><ci id="S1.F2.11.m5.3.3.2.2.2.cmml" xref="S1.F2.11.m5.3.3.2.2.2">𝑒</ci><ci id="S1.F2.11.m5.3.3.2.2.3.cmml" xref="S1.F2.11.m5.3.3.2.2.3">𝑘</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S1.F2.11.m5.3d">e_{1},...,e_{k}</annotation></semantics></math>. After the retrieved experts make their predictions <math id="S1.F2.12.m6.1" class="ltx_Math" alttext="e_{i}(x)" display="inline"><semantics id="S1.F2.12.m6.1b"><mrow id="S1.F2.12.m6.1.2" xref="S1.F2.12.m6.1.2.cmml"><msub id="S1.F2.12.m6.1.2.2" xref="S1.F2.12.m6.1.2.2.cmml"><mi id="S1.F2.12.m6.1.2.2.2" xref="S1.F2.12.m6.1.2.2.2.cmml">e</mi><mi id="S1.F2.12.m6.1.2.2.3" xref="S1.F2.12.m6.1.2.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S1.F2.12.m6.1.2.1" xref="S1.F2.12.m6.1.2.1.cmml">​</mo><mrow id="S1.F2.12.m6.1.2.3.2" xref="S1.F2.12.m6.1.2.cmml"><mo stretchy="false" id="S1.F2.12.m6.1.2.3.2.1" xref="S1.F2.12.m6.1.2.cmml">(</mo><mi id="S1.F2.12.m6.1.1" xref="S1.F2.12.m6.1.1.cmml">x</mi><mo stretchy="false" id="S1.F2.12.m6.1.2.3.2.2" xref="S1.F2.12.m6.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.F2.12.m6.1c"><apply id="S1.F2.12.m6.1.2.cmml" xref="S1.F2.12.m6.1.2"><times id="S1.F2.12.m6.1.2.1.cmml" xref="S1.F2.12.m6.1.2.1"></times><apply id="S1.F2.12.m6.1.2.2.cmml" xref="S1.F2.12.m6.1.2.2"><csymbol cd="ambiguous" id="S1.F2.12.m6.1.2.2.1.cmml" xref="S1.F2.12.m6.1.2.2">subscript</csymbol><ci id="S1.F2.12.m6.1.2.2.2.cmml" xref="S1.F2.12.m6.1.2.2.2">𝑒</ci><ci id="S1.F2.12.m6.1.2.2.3.cmml" xref="S1.F2.12.m6.1.2.2.3">𝑖</ci></apply><ci id="S1.F2.12.m6.1.1.cmml" xref="S1.F2.12.m6.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F2.12.m6.1d">e_{i}(x)</annotation></semantics></math>, their outputs are linearly combined using the softmax-normalized router scores as weights.</figcaption>
</figure>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Method</h2>

<div id="S2.p1" class="ltx_para ltx_noindent">
<p id="S2.p1.1" class="ltx_p">In this section, we introduce the Parameter Efficient Expert Retrieval (PEER) layer, which is a Mixture of Experts architecture using product keys <cite class="ltx_cite ltx_citemacro_citep">(Lample et al., <a href="#bib.bib27" title="" class="ltx_ref">2019</a>)</cite> in the router and single-neuron MLPs as experts. Fig. <a href="#S1.F2" title="Figure 2 ‣ 1 Introduction ‣ Mixture of A Million Experts" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> illustrates the computational process within a PEER layer.</p>
</div>
<section id="S2.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">PEER Overview</h4>

<div id="S2.SS0.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px1.p1.14" class="ltx_p">Formally, a PEER layer is a function <math id="S2.SS0.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="f:\mathbb{R}^{n}\to\mathbb{R}^{m}" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.1.m1.1a"><mrow id="S2.SS0.SSS0.Px1.p1.1.m1.1.1" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.cmml"><mi id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.2" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.2.cmml">f</mi><mo lspace="0.278em" rspace="0.278em" id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.cmml">:</mo><mrow id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.cmml"><msup id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2.cmml"><mi id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2.2" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2.2.cmml">ℝ</mi><mi id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2.3" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2.3.cmml">n</mi></msup><mo stretchy="false" id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.1" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.1.cmml">→</mo><msup id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.cmml"><mi id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.2" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.2.cmml">ℝ</mi><mi id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.3" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.3.cmml">m</mi></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.1.m1.1b"><apply id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1"><ci id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1">:</ci><ci id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.2">𝑓</ci><apply id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3"><ci id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.1">→</ci><apply id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2.1.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2">superscript</csymbol><ci id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2.2.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2.2">ℝ</ci><ci id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2.3.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2.3">𝑛</ci></apply><apply id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.1.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3">superscript</csymbol><ci id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.2.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.2">ℝ</ci><ci id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.3.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.3">𝑚</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.1.m1.1c">f:\mathbb{R}^{n}\to\mathbb{R}^{m}</annotation></semantics></math> that consists of three parts: a pool of <math id="S2.SS0.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.2.m2.1a"><mi id="S2.SS0.SSS0.Px1.p1.2.m2.1.1" xref="S2.SS0.SSS0.Px1.p1.2.m2.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.2.m2.1b"><ci id="S2.SS0.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.2.m2.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.2.m2.1c">N</annotation></semantics></math> experts <math id="S2.SS0.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="\mathbb{E}:=\{e_{i}\}_{i=1}^{N}" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.3.m3.1a"><mrow id="S2.SS0.SSS0.Px1.p1.3.m3.1.1" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.cmml"><mi id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.cmml">𝔼</mi><mo lspace="0.278em" rspace="0.278em" id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.2" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.2.cmml">:=</mo><msubsup id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.cmml"><mrow id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.1" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.1.2" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.2.cmml">{</mo><msub id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.1.1" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.1.1.cmml"><mi id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.1.1.2" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.1.1.2.cmml">e</mi><mi id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.1.1.3" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.1.3" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.2.cmml">}</mo></mrow><mrow id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.3" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.3.2" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.3.2.cmml">i</mi><mo id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.3.1" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.3.1.cmml">=</mo><mn id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.3.3" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.3.3.cmml">1</mn></mrow><mi id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.3" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.3.cmml">N</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.3.m3.1b"><apply id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1"><csymbol cd="latexml" id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.2">assign</csymbol><ci id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3">𝔼</ci><apply id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1">superscript</csymbol><apply id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1">subscript</csymbol><set id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.1"><apply id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.1.1.2">𝑒</ci><ci id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.1.1.3">𝑖</ci></apply></set><apply id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.3"><eq id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.3.1"></eq><ci id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.3.2">𝑖</ci><cn type="integer" id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.3.3">1</cn></apply></apply><ci id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.3">𝑁</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.3.m3.1c">\mathbb{E}:=\{e_{i}\}_{i=1}^{N}</annotation></semantics></math>, where each expert <math id="S2.SS0.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="e_{i}:\mathbb{R}^{n}\to\mathbb{R}^{m}" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.4.m4.1a"><mrow id="S2.SS0.SSS0.Px1.p1.4.m4.1.1" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.cmml"><msub id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2.cmml"><mi id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2.2" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2.2.cmml">e</mi><mi id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2.3" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2.3.cmml">i</mi></msub><mo lspace="0.278em" rspace="0.278em" id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.1" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.1.cmml">:</mo><mrow id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.cmml"><msup id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.2" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.2.cmml"><mi id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.2.2" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.2.2.cmml">ℝ</mi><mi id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.2.3" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.2.3.cmml">n</mi></msup><mo stretchy="false" id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.1" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.1.cmml">→</mo><msup id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.3" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.3.cmml"><mi id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.3.2" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.3.2.cmml">ℝ</mi><mi id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.3.3" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.3.3.cmml">m</mi></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.4.m4.1b"><apply id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1"><ci id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.1">:</ci><apply id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2.1.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2">subscript</csymbol><ci id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2.2.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2.2">𝑒</ci><ci id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2.3.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2.3">𝑖</ci></apply><apply id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3"><ci id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.1">→</ci><apply id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.2.1.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.2">superscript</csymbol><ci id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.2.2.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.2.2">ℝ</ci><ci id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.2.3.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.2.3">𝑛</ci></apply><apply id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.3.1.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.3">superscript</csymbol><ci id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.3.2.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.3.2">ℝ</ci><ci id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.3.3.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.3.3">𝑚</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.4.m4.1c">e_{i}:\mathbb{R}^{n}\to\mathbb{R}^{m}</annotation></semantics></math> shares the same signature as <math id="S2.SS0.SSS0.Px1.p1.5.m5.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.5.m5.1a"><mi id="S2.SS0.SSS0.Px1.p1.5.m5.1.1" xref="S2.SS0.SSS0.Px1.p1.5.m5.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.5.m5.1b"><ci id="S2.SS0.SSS0.Px1.p1.5.m5.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.5.m5.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.5.m5.1c">f</annotation></semantics></math>, a corresponding set of <math id="S2.SS0.SSS0.Px1.p1.6.m6.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.6.m6.1a"><mi id="S2.SS0.SSS0.Px1.p1.6.m6.1.1" xref="S2.SS0.SSS0.Px1.p1.6.m6.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.6.m6.1b"><ci id="S2.SS0.SSS0.Px1.p1.6.m6.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.6.m6.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.6.m6.1c">N</annotation></semantics></math> product keys <math id="S2.SS0.SSS0.Px1.p1.7.m7.1" class="ltx_Math" alttext="\mathbb{K}:=\{k_{i}\}_{i=1}^{N}\subset\mathbb{R}^{d}" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.7.m7.1a"><mrow id="S2.SS0.SSS0.Px1.p1.7.m7.1.1" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.cmml"><mi id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.3" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.3.cmml">𝕂</mi><mo lspace="0.278em" rspace="0.278em" id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.4" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.4.cmml">:=</mo><msubsup id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.cmml"><mrow id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.2" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.2.cmml">{</mo><msub id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.cmml"><mi id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2.cmml">k</mi><mi id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.3" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.3" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.2.cmml">}</mo></mrow><mrow id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.2" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.2.cmml">i</mi><mo id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.1" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.1.cmml">=</mo><mn id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.3" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.3.cmml">1</mn></mrow><mi id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.3" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.3.cmml">N</mi></msubsup><mo id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.5" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.5.cmml">⊂</mo><msup id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.6" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.6.cmml"><mi id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.6.2" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.6.2.cmml">ℝ</mi><mi id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.6.3" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.6.3.cmml">d</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.7.m7.1b"><apply id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1"><and id="S2.SS0.SSS0.Px1.p1.7.m7.1.1a.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1"></and><apply id="S2.SS0.SSS0.Px1.p1.7.m7.1.1b.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1"><csymbol cd="latexml" id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.4.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.4">assign</csymbol><ci id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.3">𝕂</ci><apply id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1">superscript</csymbol><apply id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1">subscript</csymbol><set id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1"><apply id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2">𝑘</ci><ci id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.3">𝑖</ci></apply></set><apply id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3"><eq id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.1"></eq><ci id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.2">𝑖</ci><cn type="integer" id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.3">1</cn></apply></apply><ci id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.3">𝑁</ci></apply></apply><apply id="S2.SS0.SSS0.Px1.p1.7.m7.1.1c.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1"><subset id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.5.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.5"></subset><share href="#S2.SS0.SSS0.Px1.p1.7.m7.1.1.1.cmml" id="S2.SS0.SSS0.Px1.p1.7.m7.1.1d.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1"></share><apply id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.6.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.6"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.6.1.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.6">superscript</csymbol><ci id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.6.2.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.6.2">ℝ</ci><ci id="S2.SS0.SSS0.Px1.p1.7.m7.1.1.6.3.cmml" xref="S2.SS0.SSS0.Px1.p1.7.m7.1.1.6.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.7.m7.1c">\mathbb{K}:=\{k_{i}\}_{i=1}^{N}\subset\mathbb{R}^{d}</annotation></semantics></math>, and a query network <math id="S2.SS0.SSS0.Px1.p1.8.m8.1" class="ltx_Math" alttext="q:\mathbb{R}^{n}\to\mathbb{R}^{d}" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.8.m8.1a"><mrow id="S2.SS0.SSS0.Px1.p1.8.m8.1.1" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.cmml"><mi id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.2" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.2.cmml">q</mi><mo lspace="0.278em" rspace="0.278em" id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.1" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.1.cmml">:</mo><mrow id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.cmml"><msup id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.2" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.2.cmml"><mi id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.2.2" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.2.2.cmml">ℝ</mi><mi id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.2.3" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.2.3.cmml">n</mi></msup><mo stretchy="false" id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.1" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.1.cmml">→</mo><msup id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.3" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.3.cmml"><mi id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.3.2" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.3.2.cmml">ℝ</mi><mi id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.3.3" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.3.3.cmml">d</mi></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.8.m8.1b"><apply id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1"><ci id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.1">:</ci><ci id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.2">𝑞</ci><apply id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3"><ci id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.1">→</ci><apply id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.2.1.cmml" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.2">superscript</csymbol><ci id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.2.2.cmml" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.2.2">ℝ</ci><ci id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.2.3.cmml" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.2.3">𝑛</ci></apply><apply id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.3.1.cmml" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.3">superscript</csymbol><ci id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.3.2.cmml" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.3.2">ℝ</ci><ci id="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.3.3.cmml" xref="S2.SS0.SSS0.Px1.p1.8.m8.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.8.m8.1c">q:\mathbb{R}^{n}\to\mathbb{R}^{d}</annotation></semantics></math> that maps the input vector <math id="S2.SS0.SSS0.Px1.p1.9.m9.1" class="ltx_Math" alttext="x\in\mathbb{R}^{n}" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.9.m9.1a"><mrow id="S2.SS0.SSS0.Px1.p1.9.m9.1.1" xref="S2.SS0.SSS0.Px1.p1.9.m9.1.1.cmml"><mi id="S2.SS0.SSS0.Px1.p1.9.m9.1.1.2" xref="S2.SS0.SSS0.Px1.p1.9.m9.1.1.2.cmml">x</mi><mo id="S2.SS0.SSS0.Px1.p1.9.m9.1.1.1" xref="S2.SS0.SSS0.Px1.p1.9.m9.1.1.1.cmml">∈</mo><msup id="S2.SS0.SSS0.Px1.p1.9.m9.1.1.3" xref="S2.SS0.SSS0.Px1.p1.9.m9.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px1.p1.9.m9.1.1.3.2" xref="S2.SS0.SSS0.Px1.p1.9.m9.1.1.3.2.cmml">ℝ</mi><mi id="S2.SS0.SSS0.Px1.p1.9.m9.1.1.3.3" xref="S2.SS0.SSS0.Px1.p1.9.m9.1.1.3.3.cmml">n</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.9.m9.1b"><apply id="S2.SS0.SSS0.Px1.p1.9.m9.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.9.m9.1.1"><in id="S2.SS0.SSS0.Px1.p1.9.m9.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.9.m9.1.1.1"></in><ci id="S2.SS0.SSS0.Px1.p1.9.m9.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.9.m9.1.1.2">𝑥</ci><apply id="S2.SS0.SSS0.Px1.p1.9.m9.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.9.m9.1.1.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.9.m9.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px1.p1.9.m9.1.1.3">superscript</csymbol><ci id="S2.SS0.SSS0.Px1.p1.9.m9.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px1.p1.9.m9.1.1.3.2">ℝ</ci><ci id="S2.SS0.SSS0.Px1.p1.9.m9.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px1.p1.9.m9.1.1.3.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.9.m9.1c">x\in\mathbb{R}^{n}</annotation></semantics></math> to a query vector <math id="S2.SS0.SSS0.Px1.p1.10.m10.1" class="ltx_Math" alttext="q(x)" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.10.m10.1a"><mrow id="S2.SS0.SSS0.Px1.p1.10.m10.1.2" xref="S2.SS0.SSS0.Px1.p1.10.m10.1.2.cmml"><mi id="S2.SS0.SSS0.Px1.p1.10.m10.1.2.2" xref="S2.SS0.SSS0.Px1.p1.10.m10.1.2.2.cmml">q</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px1.p1.10.m10.1.2.1" xref="S2.SS0.SSS0.Px1.p1.10.m10.1.2.1.cmml">​</mo><mrow id="S2.SS0.SSS0.Px1.p1.10.m10.1.2.3.2" xref="S2.SS0.SSS0.Px1.p1.10.m10.1.2.cmml"><mo stretchy="false" id="S2.SS0.SSS0.Px1.p1.10.m10.1.2.3.2.1" xref="S2.SS0.SSS0.Px1.p1.10.m10.1.2.cmml">(</mo><mi id="S2.SS0.SSS0.Px1.p1.10.m10.1.1" xref="S2.SS0.SSS0.Px1.p1.10.m10.1.1.cmml">x</mi><mo stretchy="false" id="S2.SS0.SSS0.Px1.p1.10.m10.1.2.3.2.2" xref="S2.SS0.SSS0.Px1.p1.10.m10.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.10.m10.1b"><apply id="S2.SS0.SSS0.Px1.p1.10.m10.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.10.m10.1.2"><times id="S2.SS0.SSS0.Px1.p1.10.m10.1.2.1.cmml" xref="S2.SS0.SSS0.Px1.p1.10.m10.1.2.1"></times><ci id="S2.SS0.SSS0.Px1.p1.10.m10.1.2.2.cmml" xref="S2.SS0.SSS0.Px1.p1.10.m10.1.2.2">𝑞</ci><ci id="S2.SS0.SSS0.Px1.p1.10.m10.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.10.m10.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.10.m10.1c">q(x)</annotation></semantics></math>.
Let <math id="S2.SS0.SSS0.Px1.p1.11.m11.1" class="ltx_Math" alttext="\mathcal{T}_{k}" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.11.m11.1a"><msub id="S2.SS0.SSS0.Px1.p1.11.m11.1.1" xref="S2.SS0.SSS0.Px1.p1.11.m11.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.SS0.SSS0.Px1.p1.11.m11.1.1.2" xref="S2.SS0.SSS0.Px1.p1.11.m11.1.1.2.cmml">𝒯</mi><mi id="S2.SS0.SSS0.Px1.p1.11.m11.1.1.3" xref="S2.SS0.SSS0.Px1.p1.11.m11.1.1.3.cmml">k</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.11.m11.1b"><apply id="S2.SS0.SSS0.Px1.p1.11.m11.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.11.m11.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.11.m11.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.11.m11.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px1.p1.11.m11.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.11.m11.1.1.2">𝒯</ci><ci id="S2.SS0.SSS0.Px1.p1.11.m11.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.11.m11.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.11.m11.1c">\mathcal{T}_{k}</annotation></semantics></math> denote the top-k operator. Given an input <math id="S2.SS0.SSS0.Px1.p1.12.m12.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.12.m12.1a"><mi id="S2.SS0.SSS0.Px1.p1.12.m12.1.1" xref="S2.SS0.SSS0.Px1.p1.12.m12.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.12.m12.1b"><ci id="S2.SS0.SSS0.Px1.p1.12.m12.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.12.m12.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.12.m12.1c">x</annotation></semantics></math>, we first retrieve a subset of <math id="S2.SS0.SSS0.Px1.p1.13.m13.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.13.m13.1a"><mi id="S2.SS0.SSS0.Px1.p1.13.m13.1.1" xref="S2.SS0.SSS0.Px1.p1.13.m13.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.13.m13.1b"><ci id="S2.SS0.SSS0.Px1.p1.13.m13.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.13.m13.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.13.m13.1c">k</annotation></semantics></math> experts whose corresponding product keys have the highest inner products with the query <math id="S2.SS0.SSS0.Px1.p1.14.m14.1" class="ltx_Math" alttext="q(x)" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.14.m14.1a"><mrow id="S2.SS0.SSS0.Px1.p1.14.m14.1.2" xref="S2.SS0.SSS0.Px1.p1.14.m14.1.2.cmml"><mi id="S2.SS0.SSS0.Px1.p1.14.m14.1.2.2" xref="S2.SS0.SSS0.Px1.p1.14.m14.1.2.2.cmml">q</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px1.p1.14.m14.1.2.1" xref="S2.SS0.SSS0.Px1.p1.14.m14.1.2.1.cmml">​</mo><mrow id="S2.SS0.SSS0.Px1.p1.14.m14.1.2.3.2" xref="S2.SS0.SSS0.Px1.p1.14.m14.1.2.cmml"><mo stretchy="false" id="S2.SS0.SSS0.Px1.p1.14.m14.1.2.3.2.1" xref="S2.SS0.SSS0.Px1.p1.14.m14.1.2.cmml">(</mo><mi id="S2.SS0.SSS0.Px1.p1.14.m14.1.1" xref="S2.SS0.SSS0.Px1.p1.14.m14.1.1.cmml">x</mi><mo stretchy="false" id="S2.SS0.SSS0.Px1.p1.14.m14.1.2.3.2.2" xref="S2.SS0.SSS0.Px1.p1.14.m14.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.14.m14.1b"><apply id="S2.SS0.SSS0.Px1.p1.14.m14.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.14.m14.1.2"><times id="S2.SS0.SSS0.Px1.p1.14.m14.1.2.1.cmml" xref="S2.SS0.SSS0.Px1.p1.14.m14.1.2.1"></times><ci id="S2.SS0.SSS0.Px1.p1.14.m14.1.2.2.cmml" xref="S2.SS0.SSS0.Px1.p1.14.m14.1.2.2">𝑞</ci><ci id="S2.SS0.SSS0.Px1.p1.14.m14.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.14.m14.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.14.m14.1c">q(x)</annotation></semantics></math>.</p>
<table id="Sx1.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S2.E1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.E1.m1.1" class="ltx_Math" alttext="\displaystyle\mathbb{I}" display="inline"><semantics id="S2.E1.m1.1a"><mi id="S2.E1.m1.1.1" xref="S2.E1.m1.1.1.cmml">𝕀</mi><annotation-xml encoding="MathML-Content" id="S2.E1.m1.1b"><ci id="S2.E1.m1.1.1.cmml" xref="S2.E1.m1.1.1">𝕀</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.E1.m1.1c">\displaystyle\mathbb{I}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.E1.m2.2" class="ltx_Math" alttext="\displaystyle=\mathcal{T}_{k}\left(\{q(x)^{T}k_{i}\}_{i=1}^{N}\right)" display="inline"><semantics id="S2.E1.m2.2a"><mrow id="S2.E1.m2.2.2" xref="S2.E1.m2.2.2.cmml"><mi id="S2.E1.m2.2.2.3" xref="S2.E1.m2.2.2.3.cmml"></mi><mo id="S2.E1.m2.2.2.2" xref="S2.E1.m2.2.2.2.cmml">=</mo><mrow id="S2.E1.m2.2.2.1" xref="S2.E1.m2.2.2.1.cmml"><msub id="S2.E1.m2.2.2.1.3" xref="S2.E1.m2.2.2.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.E1.m2.2.2.1.3.2" xref="S2.E1.m2.2.2.1.3.2.cmml">𝒯</mi><mi id="S2.E1.m2.2.2.1.3.3" xref="S2.E1.m2.2.2.1.3.3.cmml">k</mi></msub><mo lspace="0em" rspace="0em" id="S2.E1.m2.2.2.1.2" xref="S2.E1.m2.2.2.1.2.cmml">​</mo><mrow id="S2.E1.m2.2.2.1.1.1" xref="S2.E1.m2.2.2.1.1.1.1.cmml"><mo id="S2.E1.m2.2.2.1.1.1.2" xref="S2.E1.m2.2.2.1.1.1.1.cmml">(</mo><msubsup id="S2.E1.m2.2.2.1.1.1.1" xref="S2.E1.m2.2.2.1.1.1.1.cmml"><mrow id="S2.E1.m2.2.2.1.1.1.1.1.1.1" xref="S2.E1.m2.2.2.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S2.E1.m2.2.2.1.1.1.1.1.1.1.2" xref="S2.E1.m2.2.2.1.1.1.1.1.1.2.cmml">{</mo><mrow id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.cmml"><mi id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.2" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.2.cmml">q</mi><mo lspace="0em" rspace="0em" id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.1" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.1.cmml">​</mo><msup id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.3" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.3.cmml"><mrow id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.3.2.2" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.3.cmml"><mo stretchy="false" id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.3.2.2.1" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.3.cmml">(</mo><mi id="S2.E1.m2.1.1" xref="S2.E1.m2.1.1.cmml">x</mi><mo stretchy="false" id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.3.2.2.2" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.3.cmml">)</mo></mrow><mi id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.3.3" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.3.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.1a" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.1.cmml">​</mo><msub id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.4" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.4.cmml"><mi id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.4.2" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.4.2.cmml">k</mi><mi id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.4.3" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.4.3.cmml">i</mi></msub></mrow><mo stretchy="false" id="S2.E1.m2.2.2.1.1.1.1.1.1.1.3" xref="S2.E1.m2.2.2.1.1.1.1.1.1.2.cmml">}</mo></mrow><mrow id="S2.E1.m2.2.2.1.1.1.1.1.3" xref="S2.E1.m2.2.2.1.1.1.1.1.3.cmml"><mi id="S2.E1.m2.2.2.1.1.1.1.1.3.2" xref="S2.E1.m2.2.2.1.1.1.1.1.3.2.cmml">i</mi><mo id="S2.E1.m2.2.2.1.1.1.1.1.3.1" xref="S2.E1.m2.2.2.1.1.1.1.1.3.1.cmml">=</mo><mn id="S2.E1.m2.2.2.1.1.1.1.1.3.3" xref="S2.E1.m2.2.2.1.1.1.1.1.3.3.cmml">1</mn></mrow><mi id="S2.E1.m2.2.2.1.1.1.1.3" xref="S2.E1.m2.2.2.1.1.1.1.3.cmml">N</mi></msubsup><mo id="S2.E1.m2.2.2.1.1.1.3" xref="S2.E1.m2.2.2.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E1.m2.2b"><apply id="S2.E1.m2.2.2.cmml" xref="S2.E1.m2.2.2"><eq id="S2.E1.m2.2.2.2.cmml" xref="S2.E1.m2.2.2.2"></eq><csymbol cd="latexml" id="S2.E1.m2.2.2.3.cmml" xref="S2.E1.m2.2.2.3">absent</csymbol><apply id="S2.E1.m2.2.2.1.cmml" xref="S2.E1.m2.2.2.1"><times id="S2.E1.m2.2.2.1.2.cmml" xref="S2.E1.m2.2.2.1.2"></times><apply id="S2.E1.m2.2.2.1.3.cmml" xref="S2.E1.m2.2.2.1.3"><csymbol cd="ambiguous" id="S2.E1.m2.2.2.1.3.1.cmml" xref="S2.E1.m2.2.2.1.3">subscript</csymbol><ci id="S2.E1.m2.2.2.1.3.2.cmml" xref="S2.E1.m2.2.2.1.3.2">𝒯</ci><ci id="S2.E1.m2.2.2.1.3.3.cmml" xref="S2.E1.m2.2.2.1.3.3">𝑘</ci></apply><apply id="S2.E1.m2.2.2.1.1.1.1.cmml" xref="S2.E1.m2.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.E1.m2.2.2.1.1.1.1.2.cmml" xref="S2.E1.m2.2.2.1.1.1">superscript</csymbol><apply id="S2.E1.m2.2.2.1.1.1.1.1.cmml" xref="S2.E1.m2.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.E1.m2.2.2.1.1.1.1.1.2.cmml" xref="S2.E1.m2.2.2.1.1.1">subscript</csymbol><set id="S2.E1.m2.2.2.1.1.1.1.1.1.2.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1"><apply id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1"><times id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.1"></times><ci id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.2">𝑞</ci><apply id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.3.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.3.1.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.3">superscript</csymbol><ci id="S2.E1.m2.1.1.cmml" xref="S2.E1.m2.1.1">𝑥</ci><ci id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.3.3.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.3.3">𝑇</ci></apply><apply id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.4.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.4.1.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.4">subscript</csymbol><ci id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.4.2.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.4.2">𝑘</ci><ci id="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.4.3.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.1.1.1.4.3">𝑖</ci></apply></apply></set><apply id="S2.E1.m2.2.2.1.1.1.1.1.3.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.3"><eq id="S2.E1.m2.2.2.1.1.1.1.1.3.1.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.3.1"></eq><ci id="S2.E1.m2.2.2.1.1.1.1.1.3.2.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.3.2">𝑖</ci><cn type="integer" id="S2.E1.m2.2.2.1.1.1.1.1.3.3.cmml" xref="S2.E1.m2.2.2.1.1.1.1.1.3.3">1</cn></apply></apply><ci id="S2.E1.m2.2.2.1.1.1.1.3.cmml" xref="S2.E1.m2.2.2.1.1.1.1.3">𝑁</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1.m2.2c">\displaystyle=\mathcal{T}_{k}\left(\{q(x)^{T}k_{i}\}_{i=1}^{N}\right)</annotation></semantics></math></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><span id="S2.E1.m3.1.1.1a" class="ltx_text ltx_markedasmath"># Retrieve top <math id="S2.E1.m3.1.1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S2.E1.m3.1.1.1.m1.1a"><mi id="S2.E1.m3.1.1.1.m1.1.1" xref="S2.E1.m3.1.1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S2.E1.m3.1.1.1.m1.1b"><ci id="S2.E1.m3.1.1.1.m1.1.1.cmml" xref="S2.E1.m3.1.1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.E1.m3.1.1.1.m1.1c">k</annotation></semantics></math> experts</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<p id="S2.SS0.SSS0.Px1.p1.15" class="ltx_p">Then we apply nonlinear activations (such as softmax or sigmoid) to the query-key inner products of these top <math id="S2.SS0.SSS0.Px1.p1.15.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.15.m1.1a"><mi id="S2.SS0.SSS0.Px1.p1.15.m1.1.1" xref="S2.SS0.SSS0.Px1.p1.15.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.15.m1.1b"><ci id="S2.SS0.SSS0.Px1.p1.15.m1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.15.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.15.m1.1c">k</annotation></semantics></math> experts to obtain the router scores.</p>
<table id="Sx1.EGx2" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S2.E2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.E2.m1.1" class="ltx_Math" alttext="\displaystyle g_{i}(x)" display="inline"><semantics id="S2.E2.m1.1a"><mrow id="S2.E2.m1.1.2" xref="S2.E2.m1.1.2.cmml"><msub id="S2.E2.m1.1.2.2" xref="S2.E2.m1.1.2.2.cmml"><mi id="S2.E2.m1.1.2.2.2" xref="S2.E2.m1.1.2.2.2.cmml">g</mi><mi id="S2.E2.m1.1.2.2.3" xref="S2.E2.m1.1.2.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S2.E2.m1.1.2.1" xref="S2.E2.m1.1.2.1.cmml">​</mo><mrow id="S2.E2.m1.1.2.3.2" xref="S2.E2.m1.1.2.cmml"><mo stretchy="false" id="S2.E2.m1.1.2.3.2.1" xref="S2.E2.m1.1.2.cmml">(</mo><mi id="S2.E2.m1.1.1" xref="S2.E2.m1.1.1.cmml">x</mi><mo stretchy="false" id="S2.E2.m1.1.2.3.2.2" xref="S2.E2.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E2.m1.1b"><apply id="S2.E2.m1.1.2.cmml" xref="S2.E2.m1.1.2"><times id="S2.E2.m1.1.2.1.cmml" xref="S2.E2.m1.1.2.1"></times><apply id="S2.E2.m1.1.2.2.cmml" xref="S2.E2.m1.1.2.2"><csymbol cd="ambiguous" id="S2.E2.m1.1.2.2.1.cmml" xref="S2.E2.m1.1.2.2">subscript</csymbol><ci id="S2.E2.m1.1.2.2.2.cmml" xref="S2.E2.m1.1.2.2.2">𝑔</ci><ci id="S2.E2.m1.1.2.2.3.cmml" xref="S2.E2.m1.1.2.2.3">𝑖</ci></apply><ci id="S2.E2.m1.1.1.cmml" xref="S2.E2.m1.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E2.m1.1c">\displaystyle g_{i}(x)</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.E2.m2.2" class="ltx_Math" alttext="\displaystyle=s(q(x)^{T}k_{i})" display="inline"><semantics id="S2.E2.m2.2a"><mrow id="S2.E2.m2.2.2" xref="S2.E2.m2.2.2.cmml"><mi id="S2.E2.m2.2.2.3" xref="S2.E2.m2.2.2.3.cmml"></mi><mo id="S2.E2.m2.2.2.2" xref="S2.E2.m2.2.2.2.cmml">=</mo><mrow id="S2.E2.m2.2.2.1" xref="S2.E2.m2.2.2.1.cmml"><mi id="S2.E2.m2.2.2.1.3" xref="S2.E2.m2.2.2.1.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="S2.E2.m2.2.2.1.2" xref="S2.E2.m2.2.2.1.2.cmml">​</mo><mrow id="S2.E2.m2.2.2.1.1.1" xref="S2.E2.m2.2.2.1.1.1.1.cmml"><mo stretchy="false" id="S2.E2.m2.2.2.1.1.1.2" xref="S2.E2.m2.2.2.1.1.1.1.cmml">(</mo><mrow id="S2.E2.m2.2.2.1.1.1.1" xref="S2.E2.m2.2.2.1.1.1.1.cmml"><mi id="S2.E2.m2.2.2.1.1.1.1.2" xref="S2.E2.m2.2.2.1.1.1.1.2.cmml">q</mi><mo lspace="0em" rspace="0em" id="S2.E2.m2.2.2.1.1.1.1.1" xref="S2.E2.m2.2.2.1.1.1.1.1.cmml">​</mo><msup id="S2.E2.m2.2.2.1.1.1.1.3" xref="S2.E2.m2.2.2.1.1.1.1.3.cmml"><mrow id="S2.E2.m2.2.2.1.1.1.1.3.2.2" xref="S2.E2.m2.2.2.1.1.1.1.3.cmml"><mo stretchy="false" id="S2.E2.m2.2.2.1.1.1.1.3.2.2.1" xref="S2.E2.m2.2.2.1.1.1.1.3.cmml">(</mo><mi id="S2.E2.m2.1.1" xref="S2.E2.m2.1.1.cmml">x</mi><mo stretchy="false" id="S2.E2.m2.2.2.1.1.1.1.3.2.2.2" xref="S2.E2.m2.2.2.1.1.1.1.3.cmml">)</mo></mrow><mi id="S2.E2.m2.2.2.1.1.1.1.3.3" xref="S2.E2.m2.2.2.1.1.1.1.3.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="S2.E2.m2.2.2.1.1.1.1.1a" xref="S2.E2.m2.2.2.1.1.1.1.1.cmml">​</mo><msub id="S2.E2.m2.2.2.1.1.1.1.4" xref="S2.E2.m2.2.2.1.1.1.1.4.cmml"><mi id="S2.E2.m2.2.2.1.1.1.1.4.2" xref="S2.E2.m2.2.2.1.1.1.1.4.2.cmml">k</mi><mi id="S2.E2.m2.2.2.1.1.1.1.4.3" xref="S2.E2.m2.2.2.1.1.1.1.4.3.cmml">i</mi></msub></mrow><mo stretchy="false" id="S2.E2.m2.2.2.1.1.1.3" xref="S2.E2.m2.2.2.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E2.m2.2b"><apply id="S2.E2.m2.2.2.cmml" xref="S2.E2.m2.2.2"><eq id="S2.E2.m2.2.2.2.cmml" xref="S2.E2.m2.2.2.2"></eq><csymbol cd="latexml" id="S2.E2.m2.2.2.3.cmml" xref="S2.E2.m2.2.2.3">absent</csymbol><apply id="S2.E2.m2.2.2.1.cmml" xref="S2.E2.m2.2.2.1"><times id="S2.E2.m2.2.2.1.2.cmml" xref="S2.E2.m2.2.2.1.2"></times><ci id="S2.E2.m2.2.2.1.3.cmml" xref="S2.E2.m2.2.2.1.3">𝑠</ci><apply id="S2.E2.m2.2.2.1.1.1.1.cmml" xref="S2.E2.m2.2.2.1.1.1"><times id="S2.E2.m2.2.2.1.1.1.1.1.cmml" xref="S2.E2.m2.2.2.1.1.1.1.1"></times><ci id="S2.E2.m2.2.2.1.1.1.1.2.cmml" xref="S2.E2.m2.2.2.1.1.1.1.2">𝑞</ci><apply id="S2.E2.m2.2.2.1.1.1.1.3.cmml" xref="S2.E2.m2.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E2.m2.2.2.1.1.1.1.3.1.cmml" xref="S2.E2.m2.2.2.1.1.1.1.3">superscript</csymbol><ci id="S2.E2.m2.1.1.cmml" xref="S2.E2.m2.1.1">𝑥</ci><ci id="S2.E2.m2.2.2.1.1.1.1.3.3.cmml" xref="S2.E2.m2.2.2.1.1.1.1.3.3">𝑇</ci></apply><apply id="S2.E2.m2.2.2.1.1.1.1.4.cmml" xref="S2.E2.m2.2.2.1.1.1.1.4"><csymbol cd="ambiguous" id="S2.E2.m2.2.2.1.1.1.1.4.1.cmml" xref="S2.E2.m2.2.2.1.1.1.1.4">subscript</csymbol><ci id="S2.E2.m2.2.2.1.1.1.1.4.2.cmml" xref="S2.E2.m2.2.2.1.1.1.1.4.2">𝑘</ci><ci id="S2.E2.m2.2.2.1.1.1.1.4.3.cmml" xref="S2.E2.m2.2.2.1.1.1.1.4.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E2.m2.2c">\displaystyle=s(q(x)^{T}k_{i})</annotation></semantics></math></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><span id="S2.E2.6.2.1.1" class="ltx_text ltx_markedasmath"># Compute router scores</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
<p id="S2.SS0.SSS0.Px1.p1.16" class="ltx_p">Finally, we compute the output by linearly combining the expert outputs weighted by the router scores.</p>
<table id="Sx1.EGx3" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S2.E3"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.E3.m1.1" class="ltx_Math" alttext="\displaystyle f(x)" display="inline"><semantics id="S2.E3.m1.1a"><mrow id="S2.E3.m1.1.2" xref="S2.E3.m1.1.2.cmml"><mi id="S2.E3.m1.1.2.2" xref="S2.E3.m1.1.2.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S2.E3.m1.1.2.1" xref="S2.E3.m1.1.2.1.cmml">​</mo><mrow id="S2.E3.m1.1.2.3.2" xref="S2.E3.m1.1.2.cmml"><mo stretchy="false" id="S2.E3.m1.1.2.3.2.1" xref="S2.E3.m1.1.2.cmml">(</mo><mi id="S2.E3.m1.1.1" xref="S2.E3.m1.1.1.cmml">x</mi><mo stretchy="false" id="S2.E3.m1.1.2.3.2.2" xref="S2.E3.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E3.m1.1b"><apply id="S2.E3.m1.1.2.cmml" xref="S2.E3.m1.1.2"><times id="S2.E3.m1.1.2.1.cmml" xref="S2.E3.m1.1.2.1"></times><ci id="S2.E3.m1.1.2.2.cmml" xref="S2.E3.m1.1.2.2">𝑓</ci><ci id="S2.E3.m1.1.1.cmml" xref="S2.E3.m1.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E3.m1.1c">\displaystyle f(x)</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.E3.m2.2" class="ltx_Math" alttext="\displaystyle=\sum\nolimits_{i\in\mathbb{I}}g_{i}(x)e_{i}(x)" display="inline"><semantics id="S2.E3.m2.2a"><mrow id="S2.E3.m2.2.3" xref="S2.E3.m2.2.3.cmml"><mi id="S2.E3.m2.2.3.2" xref="S2.E3.m2.2.3.2.cmml"></mi><mo id="S2.E3.m2.2.3.1" xref="S2.E3.m2.2.3.1.cmml">=</mo><mrow id="S2.E3.m2.2.3.3" xref="S2.E3.m2.2.3.3.cmml"><mstyle displaystyle="true" id="S2.E3.m2.2.3.3.1" xref="S2.E3.m2.2.3.3.1.cmml"><msub id="S2.E3.m2.2.3.3.1a" xref="S2.E3.m2.2.3.3.1.cmml"><mo id="S2.E3.m2.2.3.3.1.2" xref="S2.E3.m2.2.3.3.1.2.cmml">∑</mo><mrow id="S2.E3.m2.2.3.3.1.3" xref="S2.E3.m2.2.3.3.1.3.cmml"><mi id="S2.E3.m2.2.3.3.1.3.2" xref="S2.E3.m2.2.3.3.1.3.2.cmml">i</mi><mo id="S2.E3.m2.2.3.3.1.3.1" xref="S2.E3.m2.2.3.3.1.3.1.cmml">∈</mo><mi id="S2.E3.m2.2.3.3.1.3.3" xref="S2.E3.m2.2.3.3.1.3.3.cmml">𝕀</mi></mrow></msub></mstyle><mrow id="S2.E3.m2.2.3.3.2" xref="S2.E3.m2.2.3.3.2.cmml"><msub id="S2.E3.m2.2.3.3.2.2" xref="S2.E3.m2.2.3.3.2.2.cmml"><mi id="S2.E3.m2.2.3.3.2.2.2" xref="S2.E3.m2.2.3.3.2.2.2.cmml">g</mi><mi id="S2.E3.m2.2.3.3.2.2.3" xref="S2.E3.m2.2.3.3.2.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S2.E3.m2.2.3.3.2.1" xref="S2.E3.m2.2.3.3.2.1.cmml">​</mo><mrow id="S2.E3.m2.2.3.3.2.3.2" xref="S2.E3.m2.2.3.3.2.cmml"><mo stretchy="false" id="S2.E3.m2.2.3.3.2.3.2.1" xref="S2.E3.m2.2.3.3.2.cmml">(</mo><mi id="S2.E3.m2.1.1" xref="S2.E3.m2.1.1.cmml">x</mi><mo stretchy="false" id="S2.E3.m2.2.3.3.2.3.2.2" xref="S2.E3.m2.2.3.3.2.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S2.E3.m2.2.3.3.2.1a" xref="S2.E3.m2.2.3.3.2.1.cmml">​</mo><msub id="S2.E3.m2.2.3.3.2.4" xref="S2.E3.m2.2.3.3.2.4.cmml"><mi id="S2.E3.m2.2.3.3.2.4.2" xref="S2.E3.m2.2.3.3.2.4.2.cmml">e</mi><mi id="S2.E3.m2.2.3.3.2.4.3" xref="S2.E3.m2.2.3.3.2.4.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S2.E3.m2.2.3.3.2.1b" xref="S2.E3.m2.2.3.3.2.1.cmml">​</mo><mrow id="S2.E3.m2.2.3.3.2.5.2" xref="S2.E3.m2.2.3.3.2.cmml"><mo stretchy="false" id="S2.E3.m2.2.3.3.2.5.2.1" xref="S2.E3.m2.2.3.3.2.cmml">(</mo><mi id="S2.E3.m2.2.2" xref="S2.E3.m2.2.2.cmml">x</mi><mo stretchy="false" id="S2.E3.m2.2.3.3.2.5.2.2" xref="S2.E3.m2.2.3.3.2.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E3.m2.2b"><apply id="S2.E3.m2.2.3.cmml" xref="S2.E3.m2.2.3"><eq id="S2.E3.m2.2.3.1.cmml" xref="S2.E3.m2.2.3.1"></eq><csymbol cd="latexml" id="S2.E3.m2.2.3.2.cmml" xref="S2.E3.m2.2.3.2">absent</csymbol><apply id="S2.E3.m2.2.3.3.cmml" xref="S2.E3.m2.2.3.3"><apply id="S2.E3.m2.2.3.3.1.cmml" xref="S2.E3.m2.2.3.3.1"><csymbol cd="ambiguous" id="S2.E3.m2.2.3.3.1.1.cmml" xref="S2.E3.m2.2.3.3.1">subscript</csymbol><sum id="S2.E3.m2.2.3.3.1.2.cmml" xref="S2.E3.m2.2.3.3.1.2"></sum><apply id="S2.E3.m2.2.3.3.1.3.cmml" xref="S2.E3.m2.2.3.3.1.3"><in id="S2.E3.m2.2.3.3.1.3.1.cmml" xref="S2.E3.m2.2.3.3.1.3.1"></in><ci id="S2.E3.m2.2.3.3.1.3.2.cmml" xref="S2.E3.m2.2.3.3.1.3.2">𝑖</ci><ci id="S2.E3.m2.2.3.3.1.3.3.cmml" xref="S2.E3.m2.2.3.3.1.3.3">𝕀</ci></apply></apply><apply id="S2.E3.m2.2.3.3.2.cmml" xref="S2.E3.m2.2.3.3.2"><times id="S2.E3.m2.2.3.3.2.1.cmml" xref="S2.E3.m2.2.3.3.2.1"></times><apply id="S2.E3.m2.2.3.3.2.2.cmml" xref="S2.E3.m2.2.3.3.2.2"><csymbol cd="ambiguous" id="S2.E3.m2.2.3.3.2.2.1.cmml" xref="S2.E3.m2.2.3.3.2.2">subscript</csymbol><ci id="S2.E3.m2.2.3.3.2.2.2.cmml" xref="S2.E3.m2.2.3.3.2.2.2">𝑔</ci><ci id="S2.E3.m2.2.3.3.2.2.3.cmml" xref="S2.E3.m2.2.3.3.2.2.3">𝑖</ci></apply><ci id="S2.E3.m2.1.1.cmml" xref="S2.E3.m2.1.1">𝑥</ci><apply id="S2.E3.m2.2.3.3.2.4.cmml" xref="S2.E3.m2.2.3.3.2.4"><csymbol cd="ambiguous" id="S2.E3.m2.2.3.3.2.4.1.cmml" xref="S2.E3.m2.2.3.3.2.4">subscript</csymbol><ci id="S2.E3.m2.2.3.3.2.4.2.cmml" xref="S2.E3.m2.2.3.3.2.4.2">𝑒</ci><ci id="S2.E3.m2.2.3.3.2.4.3.cmml" xref="S2.E3.m2.2.3.3.2.4.3">𝑖</ci></apply><ci id="S2.E3.m2.2.2.cmml" xref="S2.E3.m2.2.2">𝑥</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E3.m2.2c">\displaystyle=\sum\nolimits_{i\in\mathbb{I}}g_{i}(x)e_{i}(x)</annotation></semantics></math></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><span id="S2.E3.6.2.1.1" class="ltx_text ltx_markedasmath"># Aggregate expert outputs</span></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
</div>
</section>
<section id="S2.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Product Key Retrieval</h4>

<div id="S2.SS0.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px2.p1.7" class="ltx_p">Since we intend to use a very large number of experts (<math id="S2.SS0.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="N\geq 10^{6}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p1.1.m1.1a"><mrow id="S2.SS0.SSS0.Px2.p1.1.m1.1.1" xref="S2.SS0.SSS0.Px2.p1.1.m1.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p1.1.m1.1.1.2" xref="S2.SS0.SSS0.Px2.p1.1.m1.1.1.2.cmml">N</mi><mo id="S2.SS0.SSS0.Px2.p1.1.m1.1.1.1" xref="S2.SS0.SSS0.Px2.p1.1.m1.1.1.1.cmml">≥</mo><msup id="S2.SS0.SSS0.Px2.p1.1.m1.1.1.3" xref="S2.SS0.SSS0.Px2.p1.1.m1.1.1.3.cmml"><mn id="S2.SS0.SSS0.Px2.p1.1.m1.1.1.3.2" xref="S2.SS0.SSS0.Px2.p1.1.m1.1.1.3.2.cmml">10</mn><mn id="S2.SS0.SSS0.Px2.p1.1.m1.1.1.3.3" xref="S2.SS0.SSS0.Px2.p1.1.m1.1.1.3.3.cmml">6</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p1.1.m1.1b"><apply id="S2.SS0.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.1.m1.1.1"><geq id="S2.SS0.SSS0.Px2.p1.1.m1.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.1.m1.1.1.1"></geq><ci id="S2.SS0.SSS0.Px2.p1.1.m1.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p1.1.m1.1.1.2">𝑁</ci><apply id="S2.SS0.SSS0.Px2.p1.1.m1.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p1.1.m1.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px2.p1.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="S2.SS0.SSS0.Px2.p1.1.m1.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px2.p1.1.m1.1.1.3.2">10</cn><cn type="integer" id="S2.SS0.SSS0.Px2.p1.1.m1.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px2.p1.1.m1.1.1.3.3">6</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p1.1.m1.1c">N\geq 10^{6}</annotation></semantics></math>), naively computing the top <math id="S2.SS0.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S2.SS0.SSS0.Px2.p1.2.m2.1a"><mi id="S2.SS0.SSS0.Px2.p1.2.m2.1.1" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p1.2.m2.1b"><ci id="S2.SS0.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p1.2.m2.1c">k</annotation></semantics></math> indices in Eq. <a href="#S2.E1" title="In PEER Overview ‣ 2 Method ‣ Mixture of A Million Experts" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> can be very expensive. Hence we apply the product key retrieval technique here. Instead of using <math id="S2.SS0.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S2.SS0.SSS0.Px2.p1.3.m3.1a"><mi id="S2.SS0.SSS0.Px2.p1.3.m3.1.1" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p1.3.m3.1b"><ci id="S2.SS0.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.3.m3.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p1.3.m3.1c">N</annotation></semantics></math> independent <math id="S2.SS0.SSS0.Px2.p1.4.m4.1" class="ltx_Math" alttext="d" display="inline"><semantics id="S2.SS0.SSS0.Px2.p1.4.m4.1a"><mi id="S2.SS0.SSS0.Px2.p1.4.m4.1.1" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1.cmml">d</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p1.4.m4.1b"><ci id="S2.SS0.SSS0.Px2.p1.4.m4.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.4.m4.1.1">𝑑</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p1.4.m4.1c">d</annotation></semantics></math>-dimensional vectors as our keys <math id="S2.SS0.SSS0.Px2.p1.5.m5.1" class="ltx_Math" alttext="k_{i}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p1.5.m5.1a"><msub id="S2.SS0.SSS0.Px2.p1.5.m5.1.1" xref="S2.SS0.SSS0.Px2.p1.5.m5.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p1.5.m5.1.1.2" xref="S2.SS0.SSS0.Px2.p1.5.m5.1.1.2.cmml">k</mi><mi id="S2.SS0.SSS0.Px2.p1.5.m5.1.1.3" xref="S2.SS0.SSS0.Px2.p1.5.m5.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p1.5.m5.1b"><apply id="S2.SS0.SSS0.Px2.p1.5.m5.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p1.5.m5.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.5.m5.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px2.p1.5.m5.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p1.5.m5.1.1.2">𝑘</ci><ci id="S2.SS0.SSS0.Px2.p1.5.m5.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p1.5.m5.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p1.5.m5.1c">k_{i}</annotation></semantics></math>, we create them by concatenating vectors from two independent sets of <math id="S2.SS0.SSS0.Px2.p1.6.m6.1" class="ltx_Math" alttext="\frac{d}{2}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p1.6.m6.1a"><mfrac id="S2.SS0.SSS0.Px2.p1.6.m6.1.1" xref="S2.SS0.SSS0.Px2.p1.6.m6.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p1.6.m6.1.1.2" xref="S2.SS0.SSS0.Px2.p1.6.m6.1.1.2.cmml">d</mi><mn id="S2.SS0.SSS0.Px2.p1.6.m6.1.1.3" xref="S2.SS0.SSS0.Px2.p1.6.m6.1.1.3.cmml">2</mn></mfrac><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p1.6.m6.1b"><apply id="S2.SS0.SSS0.Px2.p1.6.m6.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.6.m6.1.1"><divide id="S2.SS0.SSS0.Px2.p1.6.m6.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.6.m6.1.1"></divide><ci id="S2.SS0.SSS0.Px2.p1.6.m6.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p1.6.m6.1.1.2">𝑑</ci><cn type="integer" id="S2.SS0.SSS0.Px2.p1.6.m6.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p1.6.m6.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p1.6.m6.1c">\frac{d}{2}</annotation></semantics></math>-dimensional sub-keys <math id="S2.SS0.SSS0.Px2.p1.7.m7.2" class="ltx_Math" alttext="\mathbb{C},\mathbb{C}^{\prime}\subset\mathbb{R}^{\frac{d}{2}}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p1.7.m7.2a"><mrow id="S2.SS0.SSS0.Px2.p1.7.m7.2.2" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.cmml"><mrow id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.1.1" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.1.2.cmml"><mi id="S2.SS0.SSS0.Px2.p1.7.m7.1.1" xref="S2.SS0.SSS0.Px2.p1.7.m7.1.1.cmml">ℂ</mi><mo id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.1.1.2" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.1.2.cmml">,</mo><msup id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.1.1.1" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.1.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.1.1.1.2" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.1.1.1.2.cmml">ℂ</mi><mo id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.1.1.1.3" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.1.1.1.3.cmml">′</mo></msup></mrow><mo id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.2" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.2.cmml">⊂</mo><msup id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3.cmml"><mi id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3.2" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3.2.cmml">ℝ</mi><mfrac id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3.3" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3.3.cmml"><mi id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3.3.2" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3.3.2.cmml">d</mi><mn id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3.3.3" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3.3.3.cmml">2</mn></mfrac></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p1.7.m7.2b"><apply id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.cmml" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2"><subset id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.2.cmml" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.2"></subset><list id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.1.2.cmml" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.1.1"><ci id="S2.SS0.SSS0.Px2.p1.7.m7.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.7.m7.1.1">ℂ</ci><apply id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.1.1.1">superscript</csymbol><ci id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.1.1.1.2">ℂ</ci><ci id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.1.1.1.3">′</ci></apply></list><apply id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3.cmml" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3.1.cmml" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3">superscript</csymbol><ci id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3.2.cmml" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3.2">ℝ</ci><apply id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3.3.cmml" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3.3"><divide id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3.3.1.cmml" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3.3"></divide><ci id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3.3.2.cmml" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3.3.2">𝑑</ci><cn type="integer" id="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3.3.3.cmml" xref="S2.SS0.SSS0.Px2.p1.7.m7.2.2.3.3.3">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p1.7.m7.2c">\mathbb{C},\mathbb{C}^{\prime}\subset\mathbb{R}^{\frac{d}{2}}</annotation></semantics></math>:</p>
<table id="S2.E4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E4.m1.2" class="ltx_Math" alttext="\mathbb{K}=\{\begin{bmatrix}c\\
c^{\prime}\end{bmatrix}|c\in\mathbb{C},c^{\prime}\in\mathbb{C}^{\prime}\}" display="block"><semantics id="S2.E4.m1.2a"><mrow id="S2.E4.m1.2.2" xref="S2.E4.m1.2.2.cmml"><mi id="S2.E4.m1.2.2.3" xref="S2.E4.m1.2.2.3.cmml">𝕂</mi><mo id="S2.E4.m1.2.2.2" xref="S2.E4.m1.2.2.2.cmml">=</mo><mrow id="S2.E4.m1.2.2.1.1" xref="S2.E4.m1.2.2.1.2.cmml"><mo stretchy="false" id="S2.E4.m1.2.2.1.1.2" xref="S2.E4.m1.2.2.1.2.1.cmml">{</mo><mrow id="S2.E4.m1.1.1.3" xref="S2.E4.m1.1.1.2.cmml"><mo id="S2.E4.m1.1.1.3.1" xref="S2.E4.m1.1.1.2.1.cmml">[</mo><mtable displaystyle="true" rowspacing="0pt" id="S2.E4.m1.1.1.1.1" xref="S2.E4.m1.1.1.1.1.cmml"><mtr id="S2.E4.m1.1.1.1.1a" xref="S2.E4.m1.1.1.1.1.cmml"><mtd id="S2.E4.m1.1.1.1.1b" xref="S2.E4.m1.1.1.1.1.cmml"><mi id="S2.E4.m1.1.1.1.1.1.1.1" xref="S2.E4.m1.1.1.1.1.1.1.1.cmml">c</mi></mtd></mtr><mtr id="S2.E4.m1.1.1.1.1c" xref="S2.E4.m1.1.1.1.1.cmml"><mtd id="S2.E4.m1.1.1.1.1d" xref="S2.E4.m1.1.1.1.1.cmml"><msup id="S2.E4.m1.1.1.1.1.2.1.1" xref="S2.E4.m1.1.1.1.1.2.1.1.cmml"><mi id="S2.E4.m1.1.1.1.1.2.1.1.2" xref="S2.E4.m1.1.1.1.1.2.1.1.2.cmml">c</mi><mo id="S2.E4.m1.1.1.1.1.2.1.1.3" xref="S2.E4.m1.1.1.1.1.2.1.1.3.cmml">′</mo></msup></mtd></mtr></mtable><mo id="S2.E4.m1.1.1.3.2" xref="S2.E4.m1.1.1.2.1.cmml">]</mo></mrow><mo lspace="0em" rspace="0em" id="S2.E4.m1.2.2.1.1.3" xref="S2.E4.m1.2.2.1.2.1.cmml">|</mo><mrow id="S2.E4.m1.2.2.1.1.1.2" xref="S2.E4.m1.2.2.1.1.1.3.cmml"><mrow id="S2.E4.m1.2.2.1.1.1.1.1" xref="S2.E4.m1.2.2.1.1.1.1.1.cmml"><mi id="S2.E4.m1.2.2.1.1.1.1.1.2" xref="S2.E4.m1.2.2.1.1.1.1.1.2.cmml">c</mi><mo id="S2.E4.m1.2.2.1.1.1.1.1.1" xref="S2.E4.m1.2.2.1.1.1.1.1.1.cmml">∈</mo><mi id="S2.E4.m1.2.2.1.1.1.1.1.3" xref="S2.E4.m1.2.2.1.1.1.1.1.3.cmml">ℂ</mi></mrow><mo id="S2.E4.m1.2.2.1.1.1.2.3" xref="S2.E4.m1.2.2.1.1.1.3a.cmml">,</mo><mrow id="S2.E4.m1.2.2.1.1.1.2.2" xref="S2.E4.m1.2.2.1.1.1.2.2.cmml"><msup id="S2.E4.m1.2.2.1.1.1.2.2.2" xref="S2.E4.m1.2.2.1.1.1.2.2.2.cmml"><mi id="S2.E4.m1.2.2.1.1.1.2.2.2.2" xref="S2.E4.m1.2.2.1.1.1.2.2.2.2.cmml">c</mi><mo id="S2.E4.m1.2.2.1.1.1.2.2.2.3" xref="S2.E4.m1.2.2.1.1.1.2.2.2.3.cmml">′</mo></msup><mo id="S2.E4.m1.2.2.1.1.1.2.2.1" xref="S2.E4.m1.2.2.1.1.1.2.2.1.cmml">∈</mo><msup id="S2.E4.m1.2.2.1.1.1.2.2.3" xref="S2.E4.m1.2.2.1.1.1.2.2.3.cmml"><mi id="S2.E4.m1.2.2.1.1.1.2.2.3.2" xref="S2.E4.m1.2.2.1.1.1.2.2.3.2.cmml">ℂ</mi><mo id="S2.E4.m1.2.2.1.1.1.2.2.3.3" xref="S2.E4.m1.2.2.1.1.1.2.2.3.3.cmml">′</mo></msup></mrow></mrow><mo stretchy="false" id="S2.E4.m1.2.2.1.1.4" xref="S2.E4.m1.2.2.1.2.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E4.m1.2b"><apply id="S2.E4.m1.2.2.cmml" xref="S2.E4.m1.2.2"><eq id="S2.E4.m1.2.2.2.cmml" xref="S2.E4.m1.2.2.2"></eq><ci id="S2.E4.m1.2.2.3.cmml" xref="S2.E4.m1.2.2.3">𝕂</ci><apply id="S2.E4.m1.2.2.1.2.cmml" xref="S2.E4.m1.2.2.1.1"><csymbol cd="latexml" id="S2.E4.m1.2.2.1.2.1.cmml" xref="S2.E4.m1.2.2.1.1.2">conditional-set</csymbol><apply id="S2.E4.m1.1.1.2.cmml" xref="S2.E4.m1.1.1.3"><csymbol cd="latexml" id="S2.E4.m1.1.1.2.1.cmml" xref="S2.E4.m1.1.1.3.1">matrix</csymbol><matrix id="S2.E4.m1.1.1.1.1.cmml" xref="S2.E4.m1.1.1.1.1"><matrixrow id="S2.E4.m1.1.1.1.1a.cmml" xref="S2.E4.m1.1.1.1.1"><ci id="S2.E4.m1.1.1.1.1.1.1.1.cmml" xref="S2.E4.m1.1.1.1.1.1.1.1">𝑐</ci></matrixrow><matrixrow id="S2.E4.m1.1.1.1.1b.cmml" xref="S2.E4.m1.1.1.1.1"><apply id="S2.E4.m1.1.1.1.1.2.1.1.cmml" xref="S2.E4.m1.1.1.1.1.2.1.1"><csymbol cd="ambiguous" id="S2.E4.m1.1.1.1.1.2.1.1.1.cmml" xref="S2.E4.m1.1.1.1.1.2.1.1">superscript</csymbol><ci id="S2.E4.m1.1.1.1.1.2.1.1.2.cmml" xref="S2.E4.m1.1.1.1.1.2.1.1.2">𝑐</ci><ci id="S2.E4.m1.1.1.1.1.2.1.1.3.cmml" xref="S2.E4.m1.1.1.1.1.2.1.1.3">′</ci></apply></matrixrow></matrix></apply><apply id="S2.E4.m1.2.2.1.1.1.3.cmml" xref="S2.E4.m1.2.2.1.1.1.2"><csymbol cd="ambiguous" id="S2.E4.m1.2.2.1.1.1.3a.cmml" xref="S2.E4.m1.2.2.1.1.1.2.3">formulae-sequence</csymbol><apply id="S2.E4.m1.2.2.1.1.1.1.1.cmml" xref="S2.E4.m1.2.2.1.1.1.1.1"><in id="S2.E4.m1.2.2.1.1.1.1.1.1.cmml" xref="S2.E4.m1.2.2.1.1.1.1.1.1"></in><ci id="S2.E4.m1.2.2.1.1.1.1.1.2.cmml" xref="S2.E4.m1.2.2.1.1.1.1.1.2">𝑐</ci><ci id="S2.E4.m1.2.2.1.1.1.1.1.3.cmml" xref="S2.E4.m1.2.2.1.1.1.1.1.3">ℂ</ci></apply><apply id="S2.E4.m1.2.2.1.1.1.2.2.cmml" xref="S2.E4.m1.2.2.1.1.1.2.2"><in id="S2.E4.m1.2.2.1.1.1.2.2.1.cmml" xref="S2.E4.m1.2.2.1.1.1.2.2.1"></in><apply id="S2.E4.m1.2.2.1.1.1.2.2.2.cmml" xref="S2.E4.m1.2.2.1.1.1.2.2.2"><csymbol cd="ambiguous" id="S2.E4.m1.2.2.1.1.1.2.2.2.1.cmml" xref="S2.E4.m1.2.2.1.1.1.2.2.2">superscript</csymbol><ci id="S2.E4.m1.2.2.1.1.1.2.2.2.2.cmml" xref="S2.E4.m1.2.2.1.1.1.2.2.2.2">𝑐</ci><ci id="S2.E4.m1.2.2.1.1.1.2.2.2.3.cmml" xref="S2.E4.m1.2.2.1.1.1.2.2.2.3">′</ci></apply><apply id="S2.E4.m1.2.2.1.1.1.2.2.3.cmml" xref="S2.E4.m1.2.2.1.1.1.2.2.3"><csymbol cd="ambiguous" id="S2.E4.m1.2.2.1.1.1.2.2.3.1.cmml" xref="S2.E4.m1.2.2.1.1.1.2.2.3">superscript</csymbol><ci id="S2.E4.m1.2.2.1.1.1.2.2.3.2.cmml" xref="S2.E4.m1.2.2.1.1.1.2.2.3.2">ℂ</ci><ci id="S2.E4.m1.2.2.1.1.1.2.2.3.3.cmml" xref="S2.E4.m1.2.2.1.1.1.2.2.3.3">′</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E4.m1.2c">\mathbb{K}=\{\begin{bmatrix}c\\
c^{\prime}\end{bmatrix}|c\in\mathbb{C},c^{\prime}\in\mathbb{C}^{\prime}\}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
<p id="S2.SS0.SSS0.Px2.p1.13" class="ltx_p">Note that here <math id="S2.SS0.SSS0.Px2.p1.8.m1.2" class="ltx_Math" alttext="\mathbb{C},\mathbb{C}^{\prime}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p1.8.m1.2a"><mrow id="S2.SS0.SSS0.Px2.p1.8.m1.2.2.1" xref="S2.SS0.SSS0.Px2.p1.8.m1.2.2.2.cmml"><mi id="S2.SS0.SSS0.Px2.p1.8.m1.1.1" xref="S2.SS0.SSS0.Px2.p1.8.m1.1.1.cmml">ℂ</mi><mo id="S2.SS0.SSS0.Px2.p1.8.m1.2.2.1.2" xref="S2.SS0.SSS0.Px2.p1.8.m1.2.2.2.cmml">,</mo><msup id="S2.SS0.SSS0.Px2.p1.8.m1.2.2.1.1" xref="S2.SS0.SSS0.Px2.p1.8.m1.2.2.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p1.8.m1.2.2.1.1.2" xref="S2.SS0.SSS0.Px2.p1.8.m1.2.2.1.1.2.cmml">ℂ</mi><mo id="S2.SS0.SSS0.Px2.p1.8.m1.2.2.1.1.3" xref="S2.SS0.SSS0.Px2.p1.8.m1.2.2.1.1.3.cmml">′</mo></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p1.8.m1.2b"><list id="S2.SS0.SSS0.Px2.p1.8.m1.2.2.2.cmml" xref="S2.SS0.SSS0.Px2.p1.8.m1.2.2.1"><ci id="S2.SS0.SSS0.Px2.p1.8.m1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.8.m1.1.1">ℂ</ci><apply id="S2.SS0.SSS0.Px2.p1.8.m1.2.2.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.8.m1.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p1.8.m1.2.2.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.8.m1.2.2.1.1">superscript</csymbol><ci id="S2.SS0.SSS0.Px2.p1.8.m1.2.2.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p1.8.m1.2.2.1.1.2">ℂ</ci><ci id="S2.SS0.SSS0.Px2.p1.8.m1.2.2.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p1.8.m1.2.2.1.1.3">′</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p1.8.m1.2c">\mathbb{C},\mathbb{C}^{\prime}</annotation></semantics></math> have cardinality <math id="S2.SS0.SSS0.Px2.p1.9.m2.1" class="ltx_Math" alttext="\sqrt{N}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p1.9.m2.1a"><msqrt id="S2.SS0.SSS0.Px2.p1.9.m2.1.1" xref="S2.SS0.SSS0.Px2.p1.9.m2.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p1.9.m2.1.1.2" xref="S2.SS0.SSS0.Px2.p1.9.m2.1.1.2.cmml">N</mi></msqrt><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p1.9.m2.1b"><apply id="S2.SS0.SSS0.Px2.p1.9.m2.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.9.m2.1.1"><root id="S2.SS0.SSS0.Px2.p1.9.m2.1.1a.cmml" xref="S2.SS0.SSS0.Px2.p1.9.m2.1.1"></root><ci id="S2.SS0.SSS0.Px2.p1.9.m2.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p1.9.m2.1.1.2">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p1.9.m2.1c">\sqrt{N}</annotation></semantics></math> and <math id="S2.SS0.SSS0.Px2.p1.10.m3.2" class="ltx_Math" alttext="c,c^{\prime}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p1.10.m3.2a"><mrow id="S2.SS0.SSS0.Px2.p1.10.m3.2.2.1" xref="S2.SS0.SSS0.Px2.p1.10.m3.2.2.2.cmml"><mi id="S2.SS0.SSS0.Px2.p1.10.m3.1.1" xref="S2.SS0.SSS0.Px2.p1.10.m3.1.1.cmml">c</mi><mo id="S2.SS0.SSS0.Px2.p1.10.m3.2.2.1.2" xref="S2.SS0.SSS0.Px2.p1.10.m3.2.2.2.cmml">,</mo><msup id="S2.SS0.SSS0.Px2.p1.10.m3.2.2.1.1" xref="S2.SS0.SSS0.Px2.p1.10.m3.2.2.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p1.10.m3.2.2.1.1.2" xref="S2.SS0.SSS0.Px2.p1.10.m3.2.2.1.1.2.cmml">c</mi><mo id="S2.SS0.SSS0.Px2.p1.10.m3.2.2.1.1.3" xref="S2.SS0.SSS0.Px2.p1.10.m3.2.2.1.1.3.cmml">′</mo></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p1.10.m3.2b"><list id="S2.SS0.SSS0.Px2.p1.10.m3.2.2.2.cmml" xref="S2.SS0.SSS0.Px2.p1.10.m3.2.2.1"><ci id="S2.SS0.SSS0.Px2.p1.10.m3.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.10.m3.1.1">𝑐</ci><apply id="S2.SS0.SSS0.Px2.p1.10.m3.2.2.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.10.m3.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p1.10.m3.2.2.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.10.m3.2.2.1.1">superscript</csymbol><ci id="S2.SS0.SSS0.Px2.p1.10.m3.2.2.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p1.10.m3.2.2.1.1.2">𝑐</ci><ci id="S2.SS0.SSS0.Px2.p1.10.m3.2.2.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p1.10.m3.2.2.1.1.3">′</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p1.10.m3.2c">c,c^{\prime}</annotation></semantics></math> have dimensionality <math id="S2.SS0.SSS0.Px2.p1.11.m4.1" class="ltx_Math" alttext="\frac{d}{2}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p1.11.m4.1a"><mfrac id="S2.SS0.SSS0.Px2.p1.11.m4.1.1" xref="S2.SS0.SSS0.Px2.p1.11.m4.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p1.11.m4.1.1.2" xref="S2.SS0.SSS0.Px2.p1.11.m4.1.1.2.cmml">d</mi><mn id="S2.SS0.SSS0.Px2.p1.11.m4.1.1.3" xref="S2.SS0.SSS0.Px2.p1.11.m4.1.1.3.cmml">2</mn></mfrac><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p1.11.m4.1b"><apply id="S2.SS0.SSS0.Px2.p1.11.m4.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.11.m4.1.1"><divide id="S2.SS0.SSS0.Px2.p1.11.m4.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.11.m4.1.1"></divide><ci id="S2.SS0.SSS0.Px2.p1.11.m4.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p1.11.m4.1.1.2">𝑑</ci><cn type="integer" id="S2.SS0.SSS0.Px2.p1.11.m4.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p1.11.m4.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p1.11.m4.1c">\frac{d}{2}</annotation></semantics></math>. So in practice, we choose <math id="S2.SS0.SSS0.Px2.p1.12.m5.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S2.SS0.SSS0.Px2.p1.12.m5.1a"><mi id="S2.SS0.SSS0.Px2.p1.12.m5.1.1" xref="S2.SS0.SSS0.Px2.p1.12.m5.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p1.12.m5.1b"><ci id="S2.SS0.SSS0.Px2.p1.12.m5.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.12.m5.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p1.12.m5.1c">N</annotation></semantics></math> to be a perfect square and <math id="S2.SS0.SSS0.Px2.p1.13.m6.1" class="ltx_Math" alttext="d" display="inline"><semantics id="S2.SS0.SSS0.Px2.p1.13.m6.1a"><mi id="S2.SS0.SSS0.Px2.p1.13.m6.1.1" xref="S2.SS0.SSS0.Px2.p1.13.m6.1.1.cmml">d</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p1.13.m6.1b"><ci id="S2.SS0.SSS0.Px2.p1.13.m6.1.1.cmml" xref="S2.SS0.SSS0.Px2.p1.13.m6.1.1">𝑑</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p1.13.m6.1c">d</annotation></semantics></math> to be an even number.</p>
</div>
<div id="S2.SS0.SSS0.Px2.p2" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px2.p2.8" class="ltx_p">This Cartesian product structure of <math id="S2.SS0.SSS0.Px2.p2.1.m1.1" class="ltx_Math" alttext="\mathbb{K}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p2.1.m1.1a"><mi id="S2.SS0.SSS0.Px2.p2.1.m1.1.1" xref="S2.SS0.SSS0.Px2.p2.1.m1.1.1.cmml">𝕂</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p2.1.m1.1b"><ci id="S2.SS0.SSS0.Px2.p2.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.1.m1.1.1">𝕂</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p2.1.m1.1c">\mathbb{K}</annotation></semantics></math> allows us to find the top <math id="S2.SS0.SSS0.Px2.p2.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S2.SS0.SSS0.Px2.p2.2.m2.1a"><mi id="S2.SS0.SSS0.Px2.p2.2.m2.1.1" xref="S2.SS0.SSS0.Px2.p2.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p2.2.m2.1b"><ci id="S2.SS0.SSS0.Px2.p2.2.m2.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p2.2.m2.1c">k</annotation></semantics></math> experts efficiently. Instead of comparing <math id="S2.SS0.SSS0.Px2.p2.3.m3.1" class="ltx_Math" alttext="q(x)" display="inline"><semantics id="S2.SS0.SSS0.Px2.p2.3.m3.1a"><mrow id="S2.SS0.SSS0.Px2.p2.3.m3.1.2" xref="S2.SS0.SSS0.Px2.p2.3.m3.1.2.cmml"><mi id="S2.SS0.SSS0.Px2.p2.3.m3.1.2.2" xref="S2.SS0.SSS0.Px2.p2.3.m3.1.2.2.cmml">q</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p2.3.m3.1.2.1" xref="S2.SS0.SSS0.Px2.p2.3.m3.1.2.1.cmml">​</mo><mrow id="S2.SS0.SSS0.Px2.p2.3.m3.1.2.3.2" xref="S2.SS0.SSS0.Px2.p2.3.m3.1.2.cmml"><mo stretchy="false" id="S2.SS0.SSS0.Px2.p2.3.m3.1.2.3.2.1" xref="S2.SS0.SSS0.Px2.p2.3.m3.1.2.cmml">(</mo><mi id="S2.SS0.SSS0.Px2.p2.3.m3.1.1" xref="S2.SS0.SSS0.Px2.p2.3.m3.1.1.cmml">x</mi><mo stretchy="false" id="S2.SS0.SSS0.Px2.p2.3.m3.1.2.3.2.2" xref="S2.SS0.SSS0.Px2.p2.3.m3.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p2.3.m3.1b"><apply id="S2.SS0.SSS0.Px2.p2.3.m3.1.2.cmml" xref="S2.SS0.SSS0.Px2.p2.3.m3.1.2"><times id="S2.SS0.SSS0.Px2.p2.3.m3.1.2.1.cmml" xref="S2.SS0.SSS0.Px2.p2.3.m3.1.2.1"></times><ci id="S2.SS0.SSS0.Px2.p2.3.m3.1.2.2.cmml" xref="S2.SS0.SSS0.Px2.p2.3.m3.1.2.2">𝑞</ci><ci id="S2.SS0.SSS0.Px2.p2.3.m3.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.3.m3.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p2.3.m3.1c">q(x)</annotation></semantics></math> to all <math id="S2.SS0.SSS0.Px2.p2.4.m4.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S2.SS0.SSS0.Px2.p2.4.m4.1a"><mi id="S2.SS0.SSS0.Px2.p2.4.m4.1.1" xref="S2.SS0.SSS0.Px2.p2.4.m4.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p2.4.m4.1b"><ci id="S2.SS0.SSS0.Px2.p2.4.m4.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.4.m4.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p2.4.m4.1c">N</annotation></semantics></math> keys in <math id="S2.SS0.SSS0.Px2.p2.5.m5.1" class="ltx_Math" alttext="\mathbb{K}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p2.5.m5.1a"><mi id="S2.SS0.SSS0.Px2.p2.5.m5.1.1" xref="S2.SS0.SSS0.Px2.p2.5.m5.1.1.cmml">𝕂</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p2.5.m5.1b"><ci id="S2.SS0.SSS0.Px2.p2.5.m5.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.5.m5.1.1">𝕂</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p2.5.m5.1c">\mathbb{K}</annotation></semantics></math> and selecting the top k matches, we can split the query vector <math id="S2.SS0.SSS0.Px2.p2.6.m6.1" class="ltx_Math" alttext="q(x)" display="inline"><semantics id="S2.SS0.SSS0.Px2.p2.6.m6.1a"><mrow id="S2.SS0.SSS0.Px2.p2.6.m6.1.2" xref="S2.SS0.SSS0.Px2.p2.6.m6.1.2.cmml"><mi id="S2.SS0.SSS0.Px2.p2.6.m6.1.2.2" xref="S2.SS0.SSS0.Px2.p2.6.m6.1.2.2.cmml">q</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p2.6.m6.1.2.1" xref="S2.SS0.SSS0.Px2.p2.6.m6.1.2.1.cmml">​</mo><mrow id="S2.SS0.SSS0.Px2.p2.6.m6.1.2.3.2" xref="S2.SS0.SSS0.Px2.p2.6.m6.1.2.cmml"><mo stretchy="false" id="S2.SS0.SSS0.Px2.p2.6.m6.1.2.3.2.1" xref="S2.SS0.SSS0.Px2.p2.6.m6.1.2.cmml">(</mo><mi id="S2.SS0.SSS0.Px2.p2.6.m6.1.1" xref="S2.SS0.SSS0.Px2.p2.6.m6.1.1.cmml">x</mi><mo stretchy="false" id="S2.SS0.SSS0.Px2.p2.6.m6.1.2.3.2.2" xref="S2.SS0.SSS0.Px2.p2.6.m6.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p2.6.m6.1b"><apply id="S2.SS0.SSS0.Px2.p2.6.m6.1.2.cmml" xref="S2.SS0.SSS0.Px2.p2.6.m6.1.2"><times id="S2.SS0.SSS0.Px2.p2.6.m6.1.2.1.cmml" xref="S2.SS0.SSS0.Px2.p2.6.m6.1.2.1"></times><ci id="S2.SS0.SSS0.Px2.p2.6.m6.1.2.2.cmml" xref="S2.SS0.SSS0.Px2.p2.6.m6.1.2.2">𝑞</ci><ci id="S2.SS0.SSS0.Px2.p2.6.m6.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.6.m6.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p2.6.m6.1c">q(x)</annotation></semantics></math> into two sub-queries <math id="S2.SS0.SSS0.Px2.p2.7.m7.1" class="ltx_Math" alttext="q_{1}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p2.7.m7.1a"><msub id="S2.SS0.SSS0.Px2.p2.7.m7.1.1" xref="S2.SS0.SSS0.Px2.p2.7.m7.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p2.7.m7.1.1.2" xref="S2.SS0.SSS0.Px2.p2.7.m7.1.1.2.cmml">q</mi><mn id="S2.SS0.SSS0.Px2.p2.7.m7.1.1.3" xref="S2.SS0.SSS0.Px2.p2.7.m7.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p2.7.m7.1b"><apply id="S2.SS0.SSS0.Px2.p2.7.m7.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.7.m7.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p2.7.m7.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.7.m7.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px2.p2.7.m7.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p2.7.m7.1.1.2">𝑞</ci><cn type="integer" id="S2.SS0.SSS0.Px2.p2.7.m7.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p2.7.m7.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p2.7.m7.1c">q_{1}</annotation></semantics></math> and <math id="S2.SS0.SSS0.Px2.p2.8.m8.1" class="ltx_Math" alttext="q_{2}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p2.8.m8.1a"><msub id="S2.SS0.SSS0.Px2.p2.8.m8.1.1" xref="S2.SS0.SSS0.Px2.p2.8.m8.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p2.8.m8.1.1.2" xref="S2.SS0.SSS0.Px2.p2.8.m8.1.1.2.cmml">q</mi><mn id="S2.SS0.SSS0.Px2.p2.8.m8.1.1.3" xref="S2.SS0.SSS0.Px2.p2.8.m8.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p2.8.m8.1b"><apply id="S2.SS0.SSS0.Px2.p2.8.m8.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.8.m8.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p2.8.m8.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.8.m8.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px2.p2.8.m8.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p2.8.m8.1.1.2">𝑞</ci><cn type="integer" id="S2.SS0.SSS0.Px2.p2.8.m8.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p2.8.m8.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p2.8.m8.1c">q_{2}</annotation></semantics></math> and apply the top k operations to the inner products between the sub-queries and sub-keys respectively:</p>
<table id="S2.E5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E5.m1.2" class="ltx_Math" alttext="\mathbb{I}_{\mathbb{C}}=\mathcal{T}_{k}\left((q_{1}^{T}c_{i})\right),\quad\quad\mathbb{I}_{\mathbb{C}^{\prime}}=\mathcal{T}_{k}\left((q_{2}^{T}c_{j}^{\prime})\right)\\
" display="block"><semantics id="S2.E5.m1.2a"><mrow id="S2.E5.m1.2.2.2" xref="S2.E5.m1.2.2.3.cmml"><mrow id="S2.E5.m1.1.1.1.1" xref="S2.E5.m1.1.1.1.1.cmml"><msub id="S2.E5.m1.1.1.1.1.3" xref="S2.E5.m1.1.1.1.1.3.cmml"><mi id="S2.E5.m1.1.1.1.1.3.2" xref="S2.E5.m1.1.1.1.1.3.2.cmml">𝕀</mi><mi id="S2.E5.m1.1.1.1.1.3.3" xref="S2.E5.m1.1.1.1.1.3.3.cmml">ℂ</mi></msub><mo id="S2.E5.m1.1.1.1.1.2" xref="S2.E5.m1.1.1.1.1.2.cmml">=</mo><mrow id="S2.E5.m1.1.1.1.1.1" xref="S2.E5.m1.1.1.1.1.1.cmml"><msub id="S2.E5.m1.1.1.1.1.1.3" xref="S2.E5.m1.1.1.1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.E5.m1.1.1.1.1.1.3.2" xref="S2.E5.m1.1.1.1.1.1.3.2.cmml">𝒯</mi><mi id="S2.E5.m1.1.1.1.1.1.3.3" xref="S2.E5.m1.1.1.1.1.1.3.3.cmml">k</mi></msub><mo lspace="0em" rspace="0em" id="S2.E5.m1.1.1.1.1.1.2" xref="S2.E5.m1.1.1.1.1.1.2.cmml">​</mo><mrow id="S2.E5.m1.1.1.1.1.1.1.1" xref="S2.E5.m1.1.1.1.1.1.cmml"><mo id="S2.E5.m1.1.1.1.1.1.1.1.2" xref="S2.E5.m1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E5.m1.1.1.1.1.1.1.1.1.1" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E5.m1.1.1.1.1.1.1.1.1.1.2" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.cmml"><msubsup id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2.2.2" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2.2.2.cmml">q</mi><mn id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2.2.3" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2.2.3.cmml">1</mn><mi id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2.3" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.1" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">​</mo><msub id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.3" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.3.2" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.3.2.cmml">c</mi><mi id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.3.3" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.3.3.cmml">i</mi></msub></mrow><mo stretchy="false" id="S2.E5.m1.1.1.1.1.1.1.1.1.1.3" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo id="S2.E5.m1.1.1.1.1.1.1.1.3" xref="S2.E5.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo rspace="2.167em" id="S2.E5.m1.2.2.2.3" xref="S2.E5.m1.2.2.3a.cmml">,</mo><mrow id="S2.E5.m1.2.2.2.2" xref="S2.E5.m1.2.2.2.2.cmml"><msub id="S2.E5.m1.2.2.2.2.3" xref="S2.E5.m1.2.2.2.2.3.cmml"><mi id="S2.E5.m1.2.2.2.2.3.2" xref="S2.E5.m1.2.2.2.2.3.2.cmml">𝕀</mi><msup id="S2.E5.m1.2.2.2.2.3.3" xref="S2.E5.m1.2.2.2.2.3.3.cmml"><mi id="S2.E5.m1.2.2.2.2.3.3.2" xref="S2.E5.m1.2.2.2.2.3.3.2.cmml">ℂ</mi><mo id="S2.E5.m1.2.2.2.2.3.3.3" xref="S2.E5.m1.2.2.2.2.3.3.3.cmml">′</mo></msup></msub><mo id="S2.E5.m1.2.2.2.2.2" xref="S2.E5.m1.2.2.2.2.2.cmml">=</mo><mrow id="S2.E5.m1.2.2.2.2.1" xref="S2.E5.m1.2.2.2.2.1.cmml"><msub id="S2.E5.m1.2.2.2.2.1.3" xref="S2.E5.m1.2.2.2.2.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.E5.m1.2.2.2.2.1.3.2" xref="S2.E5.m1.2.2.2.2.1.3.2.cmml">𝒯</mi><mi id="S2.E5.m1.2.2.2.2.1.3.3" xref="S2.E5.m1.2.2.2.2.1.3.3.cmml">k</mi></msub><mo lspace="0em" rspace="0em" id="S2.E5.m1.2.2.2.2.1.2" xref="S2.E5.m1.2.2.2.2.1.2.cmml">​</mo><mrow id="S2.E5.m1.2.2.2.2.1.1.1" xref="S2.E5.m1.2.2.2.2.1.cmml"><mo id="S2.E5.m1.2.2.2.2.1.1.1.2" xref="S2.E5.m1.2.2.2.2.1.cmml">(</mo><mrow id="S2.E5.m1.2.2.2.2.1.1.1.1.1" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E5.m1.2.2.2.2.1.1.1.1.1.2" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.cmml"><msubsup id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2.cmml"><mi id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2.2.2" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2.2.2.cmml">q</mi><mn id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2.2.3" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2.2.3.cmml">2</mn><mi id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2.3" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.1" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.1.cmml">​</mo><msubsup id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3.cmml"><mi id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3.2.2" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3.2.2.cmml">c</mi><mi id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3.2.3" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3.2.3.cmml">j</mi><mo id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3.3" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3.3.cmml">′</mo></msubsup></mrow><mo stretchy="false" id="S2.E5.m1.2.2.2.2.1.1.1.1.1.3" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.cmml">)</mo></mrow><mo id="S2.E5.m1.2.2.2.2.1.1.1.3" xref="S2.E5.m1.2.2.2.2.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E5.m1.2b"><apply id="S2.E5.m1.2.2.3.cmml" xref="S2.E5.m1.2.2.2"><csymbol cd="ambiguous" id="S2.E5.m1.2.2.3a.cmml" xref="S2.E5.m1.2.2.2.3">formulae-sequence</csymbol><apply id="S2.E5.m1.1.1.1.1.cmml" xref="S2.E5.m1.1.1.1.1"><eq id="S2.E5.m1.1.1.1.1.2.cmml" xref="S2.E5.m1.1.1.1.1.2"></eq><apply id="S2.E5.m1.1.1.1.1.3.cmml" xref="S2.E5.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E5.m1.1.1.1.1.3.1.cmml" xref="S2.E5.m1.1.1.1.1.3">subscript</csymbol><ci id="S2.E5.m1.1.1.1.1.3.2.cmml" xref="S2.E5.m1.1.1.1.1.3.2">𝕀</ci><ci id="S2.E5.m1.1.1.1.1.3.3.cmml" xref="S2.E5.m1.1.1.1.1.3.3">ℂ</ci></apply><apply id="S2.E5.m1.1.1.1.1.1.cmml" xref="S2.E5.m1.1.1.1.1.1"><times id="S2.E5.m1.1.1.1.1.1.2.cmml" xref="S2.E5.m1.1.1.1.1.1.2"></times><apply id="S2.E5.m1.1.1.1.1.1.3.cmml" xref="S2.E5.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E5.m1.1.1.1.1.1.3.1.cmml" xref="S2.E5.m1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.E5.m1.1.1.1.1.1.3.2.cmml" xref="S2.E5.m1.1.1.1.1.1.3.2">𝒯</ci><ci id="S2.E5.m1.1.1.1.1.1.3.3.cmml" xref="S2.E5.m1.1.1.1.1.1.3.3">𝑘</ci></apply><apply id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1"><times id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.1"></times><apply id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2">superscript</csymbol><apply id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2.2.1.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2.2.2.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2.2.2">𝑞</ci><cn type="integer" id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2.2.3.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2.2.3">1</cn></apply><ci id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.2.3">𝑇</ci></apply><apply id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.3.2">𝑐</ci><ci id="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.1.3.3">𝑖</ci></apply></apply></apply></apply><apply id="S2.E5.m1.2.2.2.2.cmml" xref="S2.E5.m1.2.2.2.2"><eq id="S2.E5.m1.2.2.2.2.2.cmml" xref="S2.E5.m1.2.2.2.2.2"></eq><apply id="S2.E5.m1.2.2.2.2.3.cmml" xref="S2.E5.m1.2.2.2.2.3"><csymbol cd="ambiguous" id="S2.E5.m1.2.2.2.2.3.1.cmml" xref="S2.E5.m1.2.2.2.2.3">subscript</csymbol><ci id="S2.E5.m1.2.2.2.2.3.2.cmml" xref="S2.E5.m1.2.2.2.2.3.2">𝕀</ci><apply id="S2.E5.m1.2.2.2.2.3.3.cmml" xref="S2.E5.m1.2.2.2.2.3.3"><csymbol cd="ambiguous" id="S2.E5.m1.2.2.2.2.3.3.1.cmml" xref="S2.E5.m1.2.2.2.2.3.3">superscript</csymbol><ci id="S2.E5.m1.2.2.2.2.3.3.2.cmml" xref="S2.E5.m1.2.2.2.2.3.3.2">ℂ</ci><ci id="S2.E5.m1.2.2.2.2.3.3.3.cmml" xref="S2.E5.m1.2.2.2.2.3.3.3">′</ci></apply></apply><apply id="S2.E5.m1.2.2.2.2.1.cmml" xref="S2.E5.m1.2.2.2.2.1"><times id="S2.E5.m1.2.2.2.2.1.2.cmml" xref="S2.E5.m1.2.2.2.2.1.2"></times><apply id="S2.E5.m1.2.2.2.2.1.3.cmml" xref="S2.E5.m1.2.2.2.2.1.3"><csymbol cd="ambiguous" id="S2.E5.m1.2.2.2.2.1.3.1.cmml" xref="S2.E5.m1.2.2.2.2.1.3">subscript</csymbol><ci id="S2.E5.m1.2.2.2.2.1.3.2.cmml" xref="S2.E5.m1.2.2.2.2.1.3.2">𝒯</ci><ci id="S2.E5.m1.2.2.2.2.1.3.3.cmml" xref="S2.E5.m1.2.2.2.2.1.3.3">𝑘</ci></apply><apply id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.cmml" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1"><times id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.1.cmml" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.1"></times><apply id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2.cmml" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2.1.cmml" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2">superscript</csymbol><apply id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2.2.cmml" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2.2.1.cmml" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2">subscript</csymbol><ci id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2.2.2.cmml" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2.2.2">𝑞</ci><cn type="integer" id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2.2.3.cmml" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2.2.3">2</cn></apply><ci id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2.3.cmml" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.2.3">𝑇</ci></apply><apply id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3.cmml" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3.1.cmml" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3">superscript</csymbol><apply id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3.2.cmml" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3.2.1.cmml" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3.2.2.cmml" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3.2.2">𝑐</ci><ci id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3.2.3.cmml" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3.2.3">𝑗</ci></apply><ci id="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3.3.cmml" xref="S2.E5.m1.2.2.2.2.1.1.1.1.1.1.3.3">′</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E5.m1.2c">\mathbb{I}_{\mathbb{C}}=\mathcal{T}_{k}\left((q_{1}^{T}c_{i})\right),\quad\quad\mathbb{I}_{\mathbb{C}^{\prime}}=\mathcal{T}_{k}\left((q_{2}^{T}c_{j}^{\prime})\right)\\
</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(5)</span></td>
</tr></tbody>
</table>
<p id="S2.SS0.SSS0.Px2.p2.19" class="ltx_p">This results in a set of <math id="S2.SS0.SSS0.Px2.p2.9.m1.1" class="ltx_Math" alttext="k^{2}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p2.9.m1.1a"><msup id="S2.SS0.SSS0.Px2.p2.9.m1.1.1" xref="S2.SS0.SSS0.Px2.p2.9.m1.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p2.9.m1.1.1.2" xref="S2.SS0.SSS0.Px2.p2.9.m1.1.1.2.cmml">k</mi><mn id="S2.SS0.SSS0.Px2.p2.9.m1.1.1.3" xref="S2.SS0.SSS0.Px2.p2.9.m1.1.1.3.cmml">2</mn></msup><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p2.9.m1.1b"><apply id="S2.SS0.SSS0.Px2.p2.9.m1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.9.m1.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p2.9.m1.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.9.m1.1.1">superscript</csymbol><ci id="S2.SS0.SSS0.Px2.p2.9.m1.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p2.9.m1.1.1.2">𝑘</ci><cn type="integer" id="S2.SS0.SSS0.Px2.p2.9.m1.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p2.9.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p2.9.m1.1c">k^{2}</annotation></semantics></math> candidate keys <math id="S2.SS0.SSS0.Px2.p2.10.m2.2" class="ltx_Math" alttext="\mathbb{K}^{\prime}:=\{\begin{bmatrix}c_{i}\\
c_{j}\end{bmatrix}|i\in\mathbb{I}_{\mathbb{C}},j\in\mathbb{I}_{\mathbb{C}}^{\prime}\}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p2.10.m2.2a"><mrow id="S2.SS0.SSS0.Px2.p2.10.m2.2.2" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.cmml"><msup id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.3" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.3.cmml"><mi id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.3.2" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.3.2.cmml">𝕂</mi><mo id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.3.3" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.3.3.cmml">′</mo></msup><mo lspace="0.278em" rspace="0.278em" id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.2" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.2.cmml">:=</mo><mrow id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.2.cmml"><mo stretchy="false" id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.2" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.2.1.cmml">{</mo><mrow id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.3" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.2.cmml"><mo id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.3.1" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.2.1.cmml">[</mo><mtable rowspacing="0pt" id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.cmml"><mtr id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1a" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.cmml"><mtd id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1b" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.cmml"><msub id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.1.1.1" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.1.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.1.1.1.2" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.1.1.1.2.cmml">c</mi><mi id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.1.1.1.3" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.1.1.1.3.cmml">i</mi></msub></mtd></mtr><mtr id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1c" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.cmml"><mtd id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1d" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.cmml"><msub id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.2.1.1" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.2.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.2.1.1.2" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.2.1.1.2.cmml">c</mi><mi id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.2.1.1.3" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.2.1.1.3.cmml">j</mi></msub></mtd></mtr></mtable><mo id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.3.2" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.2.1.cmml">]</mo></mrow><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.3" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.2.1.cmml">|</mo><mrow id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.3.cmml"><mrow id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.2" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.2.cmml">i</mi><mo id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.1" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.1.cmml">∈</mo><msub id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.3" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.3.2" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.3.2.cmml">𝕀</mi><mi id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.3.3" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.3.3.cmml">ℂ</mi></msub></mrow><mo id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.3" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.3a.cmml">,</mo><mrow id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.cmml"><mi id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.2" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.2.cmml">j</mi><mo id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.1" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.1.cmml">∈</mo><msubsup id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3.cmml"><mi id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3.2.2" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3.2.2.cmml">𝕀</mi><mi id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3.2.3" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3.2.3.cmml">ℂ</mi><mo id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3.3" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3.3.cmml">′</mo></msubsup></mrow></mrow><mo stretchy="false" id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.4" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.2.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p2.10.m2.2b"><apply id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2"><csymbol cd="latexml" id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.2.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.2">assign</csymbol><apply id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.3.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.3.1.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.3">superscript</csymbol><ci id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.3.2.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.3.2">𝕂</ci><ci id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.3.3.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.3.3">′</ci></apply><apply id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.2.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1"><csymbol cd="latexml" id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.2.1.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.2">conditional-set</csymbol><apply id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.3"><csymbol cd="latexml" id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.2.1.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.3.1">matrix</csymbol><matrix id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1"><matrixrow id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1a.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1"><apply id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.1.1.1.2">𝑐</ci><ci id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.1.1.1.3">𝑖</ci></apply></matrixrow><matrixrow id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1b.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1"><apply id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.2.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.2.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.2.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.2.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.2.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.2.1.1.2">𝑐</ci><ci id="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.2.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.1.1.1.1.2.1.1.3">𝑗</ci></apply></matrixrow></matrix></apply><apply id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.3a.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.3">formulae-sequence</csymbol><apply id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1"><in id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.1"></in><ci id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.2">𝑖</ci><apply id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.3.2">𝕀</ci><ci id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.1.1.3.3">ℂ</ci></apply></apply><apply id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2"><in id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.1.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.1"></in><ci id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.2.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.2">𝑗</ci><apply id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3.1.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3">superscript</csymbol><apply id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3.2.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3.2.1.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3.2.2.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3.2.2">𝕀</ci><ci id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3.2.3.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3.2.3">ℂ</ci></apply><ci id="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3.3.cmml" xref="S2.SS0.SSS0.Px2.p2.10.m2.2.2.1.1.1.2.2.3.3">′</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p2.10.m2.2c">\mathbb{K}^{\prime}:=\{\begin{bmatrix}c_{i}\\
c_{j}\end{bmatrix}|i\in\mathbb{I}_{\mathbb{C}},j\in\mathbb{I}_{\mathbb{C}}^{\prime}\}</annotation></semantics></math>, and it is mathematically guaranteed that the <math id="S2.SS0.SSS0.Px2.p2.11.m3.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S2.SS0.SSS0.Px2.p2.11.m3.1a"><mi id="S2.SS0.SSS0.Px2.p2.11.m3.1.1" xref="S2.SS0.SSS0.Px2.p2.11.m3.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p2.11.m3.1b"><ci id="S2.SS0.SSS0.Px2.p2.11.m3.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.11.m3.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p2.11.m3.1c">k</annotation></semantics></math> most similar keys to <math id="S2.SS0.SSS0.Px2.p2.12.m4.1" class="ltx_Math" alttext="q(x)" display="inline"><semantics id="S2.SS0.SSS0.Px2.p2.12.m4.1a"><mrow id="S2.SS0.SSS0.Px2.p2.12.m4.1.2" xref="S2.SS0.SSS0.Px2.p2.12.m4.1.2.cmml"><mi id="S2.SS0.SSS0.Px2.p2.12.m4.1.2.2" xref="S2.SS0.SSS0.Px2.p2.12.m4.1.2.2.cmml">q</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p2.12.m4.1.2.1" xref="S2.SS0.SSS0.Px2.p2.12.m4.1.2.1.cmml">​</mo><mrow id="S2.SS0.SSS0.Px2.p2.12.m4.1.2.3.2" xref="S2.SS0.SSS0.Px2.p2.12.m4.1.2.cmml"><mo stretchy="false" id="S2.SS0.SSS0.Px2.p2.12.m4.1.2.3.2.1" xref="S2.SS0.SSS0.Px2.p2.12.m4.1.2.cmml">(</mo><mi id="S2.SS0.SSS0.Px2.p2.12.m4.1.1" xref="S2.SS0.SSS0.Px2.p2.12.m4.1.1.cmml">x</mi><mo stretchy="false" id="S2.SS0.SSS0.Px2.p2.12.m4.1.2.3.2.2" xref="S2.SS0.SSS0.Px2.p2.12.m4.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p2.12.m4.1b"><apply id="S2.SS0.SSS0.Px2.p2.12.m4.1.2.cmml" xref="S2.SS0.SSS0.Px2.p2.12.m4.1.2"><times id="S2.SS0.SSS0.Px2.p2.12.m4.1.2.1.cmml" xref="S2.SS0.SSS0.Px2.p2.12.m4.1.2.1"></times><ci id="S2.SS0.SSS0.Px2.p2.12.m4.1.2.2.cmml" xref="S2.SS0.SSS0.Px2.p2.12.m4.1.2.2">𝑞</ci><ci id="S2.SS0.SSS0.Px2.p2.12.m4.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.12.m4.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p2.12.m4.1c">q(x)</annotation></semantics></math> from <math id="S2.SS0.SSS0.Px2.p2.13.m5.1" class="ltx_Math" alttext="\mathbb{K}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p2.13.m5.1a"><mi id="S2.SS0.SSS0.Px2.p2.13.m5.1.1" xref="S2.SS0.SSS0.Px2.p2.13.m5.1.1.cmml">𝕂</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p2.13.m5.1b"><ci id="S2.SS0.SSS0.Px2.p2.13.m5.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.13.m5.1.1">𝕂</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p2.13.m5.1c">\mathbb{K}</annotation></semantics></math> are in this candidate set. Moreover, the inner product between the candidate key and <math id="S2.SS0.SSS0.Px2.p2.14.m6.1" class="ltx_Math" alttext="q(x)" display="inline"><semantics id="S2.SS0.SSS0.Px2.p2.14.m6.1a"><mrow id="S2.SS0.SSS0.Px2.p2.14.m6.1.2" xref="S2.SS0.SSS0.Px2.p2.14.m6.1.2.cmml"><mi id="S2.SS0.SSS0.Px2.p2.14.m6.1.2.2" xref="S2.SS0.SSS0.Px2.p2.14.m6.1.2.2.cmml">q</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p2.14.m6.1.2.1" xref="S2.SS0.SSS0.Px2.p2.14.m6.1.2.1.cmml">​</mo><mrow id="S2.SS0.SSS0.Px2.p2.14.m6.1.2.3.2" xref="S2.SS0.SSS0.Px2.p2.14.m6.1.2.cmml"><mo stretchy="false" id="S2.SS0.SSS0.Px2.p2.14.m6.1.2.3.2.1" xref="S2.SS0.SSS0.Px2.p2.14.m6.1.2.cmml">(</mo><mi id="S2.SS0.SSS0.Px2.p2.14.m6.1.1" xref="S2.SS0.SSS0.Px2.p2.14.m6.1.1.cmml">x</mi><mo stretchy="false" id="S2.SS0.SSS0.Px2.p2.14.m6.1.2.3.2.2" xref="S2.SS0.SSS0.Px2.p2.14.m6.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p2.14.m6.1b"><apply id="S2.SS0.SSS0.Px2.p2.14.m6.1.2.cmml" xref="S2.SS0.SSS0.Px2.p2.14.m6.1.2"><times id="S2.SS0.SSS0.Px2.p2.14.m6.1.2.1.cmml" xref="S2.SS0.SSS0.Px2.p2.14.m6.1.2.1"></times><ci id="S2.SS0.SSS0.Px2.p2.14.m6.1.2.2.cmml" xref="S2.SS0.SSS0.Px2.p2.14.m6.1.2.2">𝑞</ci><ci id="S2.SS0.SSS0.Px2.p2.14.m6.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.14.m6.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p2.14.m6.1c">q(x)</annotation></semantics></math> is simply the sum of inner products between the sub-keys and sub-queries: <math id="S2.SS0.SSS0.Px2.p2.15.m7.2" class="ltx_Math" alttext="q(x)^{T}\begin{bmatrix}c_{i}\\
c_{j}\end{bmatrix}=q_{1}^{T}c_{i}+q_{2}^{T}c_{j}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p2.15.m7.2a"><mrow id="S2.SS0.SSS0.Px2.p2.15.m7.2.3" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.cmml"><mrow id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.cmml"><mi id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.2" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.2.cmml">q</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.1" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.1.cmml">​</mo><msup id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.3" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.3.cmml"><mrow id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.3.2.2" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.3.cmml"><mo stretchy="false" id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.3.2.2.1" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.3.cmml">(</mo><mi id="S2.SS0.SSS0.Px2.p2.15.m7.2.2" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.2.cmml">x</mi><mo stretchy="false" id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.3.2.2.2" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.3.cmml">)</mo></mrow><mi id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.3.3" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.3.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.1a" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.1.cmml">​</mo><mrow id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.3" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.2.cmml"><mo id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.3.1" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.2.1.cmml">[</mo><mtable rowspacing="0pt" id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.cmml"><mtr id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1a" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.cmml"><mtd id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1b" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.cmml"><msub id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.1.1.1" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.1.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.1.1.1.2" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.1.1.1.2.cmml">c</mi><mi id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.1.1.1.3" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.1.1.1.3.cmml">i</mi></msub></mtd></mtr><mtr id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1c" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.cmml"><mtd id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1d" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.cmml"><msub id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.2.1.1" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.2.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.2.1.1.2" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.2.1.1.2.cmml">c</mi><mi id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.2.1.1.3" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.2.1.1.3.cmml">j</mi></msub></mtd></mtr></mtable><mo id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.3.2" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.1" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.1.cmml">=</mo><mrow id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.cmml"><mrow id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.cmml"><msubsup id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2.cmml"><mi id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2.2.2" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2.2.2.cmml">q</mi><mn id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2.2.3" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2.2.3.cmml">1</mn><mi id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2.3" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.1" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.1.cmml">​</mo><msub id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.3" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.3.cmml"><mi id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.3.2" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.3.2.cmml">c</mi><mi id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.3.3" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.3.3.cmml">i</mi></msub></mrow><mo id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.1" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.1.cmml">+</mo><mrow id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.cmml"><msubsup id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2.cmml"><mi id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2.2.2" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2.2.2.cmml">q</mi><mn id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2.2.3" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2.2.3.cmml">2</mn><mi id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2.3" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.1" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.1.cmml">​</mo><msub id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.3" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.3.cmml"><mi id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.3.2" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.3.2.cmml">c</mi><mi id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.3.3" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.3.3.cmml">j</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p2.15.m7.2b"><apply id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3"><eq id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.1.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.1"></eq><apply id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2"><times id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.1.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.1"></times><ci id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.2.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.2">𝑞</ci><apply id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.3.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.3.1.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.3">superscript</csymbol><ci id="S2.SS0.SSS0.Px2.p2.15.m7.2.2.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.2">𝑥</ci><ci id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.3.3.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.2.3.3">𝑇</ci></apply><apply id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.3"><csymbol cd="latexml" id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.2.1.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.3.1">matrix</csymbol><matrix id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1"><matrixrow id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1a.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1"><apply id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.1.1.1.2">𝑐</ci><ci id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.1.1.1.3">𝑖</ci></apply></matrixrow><matrixrow id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1b.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1"><apply id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.2.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.2.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.2.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.2.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.2.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.2.1.1.2">𝑐</ci><ci id="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.2.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.1.1.1.1.2.1.1.3">𝑗</ci></apply></matrixrow></matrix></apply></apply><apply id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3"><plus id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.1.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.1"></plus><apply id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2"><times id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.1.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.1"></times><apply id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2.1.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2">superscript</csymbol><apply id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2.2.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2.2.1.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2">subscript</csymbol><ci id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2.2.2.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2.2.2">𝑞</ci><cn type="integer" id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2.2.3.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2.2.3">1</cn></apply><ci id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2.3.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.2.3">𝑇</ci></apply><apply id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.3.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.3.1.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.3.2.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.3.2">𝑐</ci><ci id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.3.3.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.2.3.3">𝑖</ci></apply></apply><apply id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3"><times id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.1.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.1"></times><apply id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2.1.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2">superscript</csymbol><apply id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2.2.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2.2.1.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2">subscript</csymbol><ci id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2.2.2.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2.2.2">𝑞</ci><cn type="integer" id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2.2.3.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2.2.3">2</cn></apply><ci id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2.3.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.2.3">𝑇</ci></apply><apply id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.3.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.3.1.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.3.2.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.3.2">𝑐</ci><ci id="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.3.3.cmml" xref="S2.SS0.SSS0.Px2.p2.15.m7.2.3.3.3.3.3">𝑗</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p2.15.m7.2c">q(x)^{T}\begin{bmatrix}c_{i}\\
c_{j}\end{bmatrix}=q_{1}^{T}c_{i}+q_{2}^{T}c_{j}</annotation></semantics></math>. Hence we can apply the top-k operator again to these <math id="S2.SS0.SSS0.Px2.p2.16.m8.1" class="ltx_Math" alttext="k^{2}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p2.16.m8.1a"><msup id="S2.SS0.SSS0.Px2.p2.16.m8.1.1" xref="S2.SS0.SSS0.Px2.p2.16.m8.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p2.16.m8.1.1.2" xref="S2.SS0.SSS0.Px2.p2.16.m8.1.1.2.cmml">k</mi><mn id="S2.SS0.SSS0.Px2.p2.16.m8.1.1.3" xref="S2.SS0.SSS0.Px2.p2.16.m8.1.1.3.cmml">2</mn></msup><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p2.16.m8.1b"><apply id="S2.SS0.SSS0.Px2.p2.16.m8.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.16.m8.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p2.16.m8.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.16.m8.1.1">superscript</csymbol><ci id="S2.SS0.SSS0.Px2.p2.16.m8.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p2.16.m8.1.1.2">𝑘</ci><cn type="integer" id="S2.SS0.SSS0.Px2.p2.16.m8.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p2.16.m8.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p2.16.m8.1c">k^{2}</annotation></semantics></math> inner products to get the top k matching keys from the original set of product keys <math id="S2.SS0.SSS0.Px2.p2.17.m9.1" class="ltx_Math" alttext="\mathbb{K}" display="inline"><semantics id="S2.SS0.SSS0.Px2.p2.17.m9.1a"><mi id="S2.SS0.SSS0.Px2.p2.17.m9.1.1" xref="S2.SS0.SSS0.Px2.p2.17.m9.1.1.cmml">𝕂</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p2.17.m9.1b"><ci id="S2.SS0.SSS0.Px2.p2.17.m9.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.17.m9.1.1">𝕂</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p2.17.m9.1c">\mathbb{K}</annotation></semantics></math>. As explained in <cite class="ltx_cite ltx_citemacro_cite">Lample et al. (<a href="#bib.bib27" title="" class="ltx_ref">2019</a>)</cite>. This reduces the complexity of top k expert retrieval in Eq. <a href="#S2.E1" title="In PEER Overview ‣ 2 Method ‣ Mixture of A Million Experts" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> from <math id="S2.SS0.SSS0.Px2.p2.18.m10.1" class="ltx_Math" alttext="O(Nd)" display="inline"><semantics id="S2.SS0.SSS0.Px2.p2.18.m10.1a"><mrow id="S2.SS0.SSS0.Px2.p2.18.m10.1.1" xref="S2.SS0.SSS0.Px2.p2.18.m10.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p2.18.m10.1.1.3" xref="S2.SS0.SSS0.Px2.p2.18.m10.1.1.3.cmml">O</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p2.18.m10.1.1.2" xref="S2.SS0.SSS0.Px2.p2.18.m10.1.1.2.cmml">​</mo><mrow id="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1" xref="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1.2" xref="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1.1.cmml">(</mo><mrow id="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1.1" xref="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1.1.2" xref="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1.1.2.cmml">N</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1.1.1" xref="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1.1.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1.1.3" xref="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1.1.3.cmml">d</mi></mrow><mo stretchy="false" id="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1.3" xref="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p2.18.m10.1b"><apply id="S2.SS0.SSS0.Px2.p2.18.m10.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.18.m10.1.1"><times id="S2.SS0.SSS0.Px2.p2.18.m10.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p2.18.m10.1.1.2"></times><ci id="S2.SS0.SSS0.Px2.p2.18.m10.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p2.18.m10.1.1.3">𝑂</ci><apply id="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1"><times id="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1.1.1"></times><ci id="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1.1.2">𝑁</ci><ci id="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p2.18.m10.1.1.1.1.1.3">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p2.18.m10.1c">O(Nd)</annotation></semantics></math> as done naively by exhaustive search to <math id="S2.SS0.SSS0.Px2.p2.19.m11.1" class="ltx_Math" alttext="O((\sqrt{N}+k^{2})d)" display="inline"><semantics id="S2.SS0.SSS0.Px2.p2.19.m11.1a"><mrow id="S2.SS0.SSS0.Px2.p2.19.m11.1.1" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.cmml"><mi id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.3" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.3.cmml">O</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.2" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.2.cmml">​</mo><mrow id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.2" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.cmml">(</mo><mrow id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.cmml"><mrow id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.2" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.cmml"><msqrt id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.2" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.2.cmml"><mi id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.2.2" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.2.2.cmml">N</mi></msqrt><mo id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.1" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.1.cmml">+</mo><msup id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.3" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.3.2" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.3.2.cmml">k</mi><mn id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.3.3" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.3.3.cmml">2</mn></msup></mrow><mo stretchy="false" id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.3" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.2" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.2.cmml">​</mo><mi id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.3" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.3.cmml">d</mi></mrow><mo stretchy="false" id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.3" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px2.p2.19.m11.1b"><apply id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1"><times id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.2"></times><ci id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.3">𝑂</ci><apply id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1"><times id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.2"></times><apply id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1"><plus id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.1"></plus><apply id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.2"><root id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.2a.cmml" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.2"></root><ci id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.2.2.cmml" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.2.2">𝑁</ci></apply><apply id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.3">superscript</csymbol><ci id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.3.2">𝑘</ci><cn type="integer" id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.1.1.1.3.3">2</cn></apply></apply><ci id="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px2.p2.19.m11.1.1.1.1.1.3">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px2.p2.19.m11.1c">O((\sqrt{N}+k^{2})d)</annotation></semantics></math>.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Parameter Efficient Experts and Multi-Head Retrieval</h4>

<div id="S2.SS0.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px3.p1.1" class="ltx_p">Unlike other MoE architectures, which often set the hidden layer of each expert to the same size as other FFW layers, in PEER, every expert <math id="S2.SS0.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="e_{i}" display="inline"><semantics id="S2.SS0.SSS0.Px3.p1.1.m1.1a"><msub id="S2.SS0.SSS0.Px3.p1.1.m1.1.1" xref="S2.SS0.SSS0.Px3.p1.1.m1.1.1.cmml"><mi id="S2.SS0.SSS0.Px3.p1.1.m1.1.1.2" xref="S2.SS0.SSS0.Px3.p1.1.m1.1.1.2.cmml">e</mi><mi id="S2.SS0.SSS0.Px3.p1.1.m1.1.1.3" xref="S2.SS0.SSS0.Px3.p1.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px3.p1.1.m1.1b"><apply id="S2.SS0.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px3.p1.1.m1.1.1.1.cmml" xref="S2.SS0.SSS0.Px3.p1.1.m1.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px3.p1.1.m1.1.1.2.cmml" xref="S2.SS0.SSS0.Px3.p1.1.m1.1.1.2">𝑒</ci><ci id="S2.SS0.SSS0.Px3.p1.1.m1.1.1.3.cmml" xref="S2.SS0.SSS0.Px3.p1.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px3.p1.1.m1.1c">e_{i}</annotation></semantics></math> is a singleton MLP, in other words, it has only one hidden layer with a single neuron:</p>
<table id="S2.E6" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E6.m1.2" class="ltx_Math" alttext="e_{i}(x):=\sigma(u_{i}^{T}x)v_{i}" display="block"><semantics id="S2.E6.m1.2a"><mrow id="S2.E6.m1.2.2" xref="S2.E6.m1.2.2.cmml"><mrow id="S2.E6.m1.2.2.3" xref="S2.E6.m1.2.2.3.cmml"><msub id="S2.E6.m1.2.2.3.2" xref="S2.E6.m1.2.2.3.2.cmml"><mi id="S2.E6.m1.2.2.3.2.2" xref="S2.E6.m1.2.2.3.2.2.cmml">e</mi><mi id="S2.E6.m1.2.2.3.2.3" xref="S2.E6.m1.2.2.3.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S2.E6.m1.2.2.3.1" xref="S2.E6.m1.2.2.3.1.cmml">​</mo><mrow id="S2.E6.m1.2.2.3.3.2" xref="S2.E6.m1.2.2.3.cmml"><mo stretchy="false" id="S2.E6.m1.2.2.3.3.2.1" xref="S2.E6.m1.2.2.3.cmml">(</mo><mi id="S2.E6.m1.1.1" xref="S2.E6.m1.1.1.cmml">x</mi><mo rspace="0.278em" stretchy="false" id="S2.E6.m1.2.2.3.3.2.2" xref="S2.E6.m1.2.2.3.cmml">)</mo></mrow></mrow><mo rspace="0.278em" id="S2.E6.m1.2.2.2" xref="S2.E6.m1.2.2.2.cmml">:=</mo><mrow id="S2.E6.m1.2.2.1" xref="S2.E6.m1.2.2.1.cmml"><mi id="S2.E6.m1.2.2.1.3" xref="S2.E6.m1.2.2.1.3.cmml">σ</mi><mo lspace="0em" rspace="0em" id="S2.E6.m1.2.2.1.2" xref="S2.E6.m1.2.2.1.2.cmml">​</mo><mrow id="S2.E6.m1.2.2.1.1.1" xref="S2.E6.m1.2.2.1.1.1.1.cmml"><mo stretchy="false" id="S2.E6.m1.2.2.1.1.1.2" xref="S2.E6.m1.2.2.1.1.1.1.cmml">(</mo><mrow id="S2.E6.m1.2.2.1.1.1.1" xref="S2.E6.m1.2.2.1.1.1.1.cmml"><msubsup id="S2.E6.m1.2.2.1.1.1.1.2" xref="S2.E6.m1.2.2.1.1.1.1.2.cmml"><mi id="S2.E6.m1.2.2.1.1.1.1.2.2.2" xref="S2.E6.m1.2.2.1.1.1.1.2.2.2.cmml">u</mi><mi id="S2.E6.m1.2.2.1.1.1.1.2.2.3" xref="S2.E6.m1.2.2.1.1.1.1.2.2.3.cmml">i</mi><mi id="S2.E6.m1.2.2.1.1.1.1.2.3" xref="S2.E6.m1.2.2.1.1.1.1.2.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="S2.E6.m1.2.2.1.1.1.1.1" xref="S2.E6.m1.2.2.1.1.1.1.1.cmml">​</mo><mi id="S2.E6.m1.2.2.1.1.1.1.3" xref="S2.E6.m1.2.2.1.1.1.1.3.cmml">x</mi></mrow><mo stretchy="false" id="S2.E6.m1.2.2.1.1.1.3" xref="S2.E6.m1.2.2.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S2.E6.m1.2.2.1.2a" xref="S2.E6.m1.2.2.1.2.cmml">​</mo><msub id="S2.E6.m1.2.2.1.4" xref="S2.E6.m1.2.2.1.4.cmml"><mi id="S2.E6.m1.2.2.1.4.2" xref="S2.E6.m1.2.2.1.4.2.cmml">v</mi><mi id="S2.E6.m1.2.2.1.4.3" xref="S2.E6.m1.2.2.1.4.3.cmml">i</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E6.m1.2b"><apply id="S2.E6.m1.2.2.cmml" xref="S2.E6.m1.2.2"><csymbol cd="latexml" id="S2.E6.m1.2.2.2.cmml" xref="S2.E6.m1.2.2.2">assign</csymbol><apply id="S2.E6.m1.2.2.3.cmml" xref="S2.E6.m1.2.2.3"><times id="S2.E6.m1.2.2.3.1.cmml" xref="S2.E6.m1.2.2.3.1"></times><apply id="S2.E6.m1.2.2.3.2.cmml" xref="S2.E6.m1.2.2.3.2"><csymbol cd="ambiguous" id="S2.E6.m1.2.2.3.2.1.cmml" xref="S2.E6.m1.2.2.3.2">subscript</csymbol><ci id="S2.E6.m1.2.2.3.2.2.cmml" xref="S2.E6.m1.2.2.3.2.2">𝑒</ci><ci id="S2.E6.m1.2.2.3.2.3.cmml" xref="S2.E6.m1.2.2.3.2.3">𝑖</ci></apply><ci id="S2.E6.m1.1.1.cmml" xref="S2.E6.m1.1.1">𝑥</ci></apply><apply id="S2.E6.m1.2.2.1.cmml" xref="S2.E6.m1.2.2.1"><times id="S2.E6.m1.2.2.1.2.cmml" xref="S2.E6.m1.2.2.1.2"></times><ci id="S2.E6.m1.2.2.1.3.cmml" xref="S2.E6.m1.2.2.1.3">𝜎</ci><apply id="S2.E6.m1.2.2.1.1.1.1.cmml" xref="S2.E6.m1.2.2.1.1.1"><times id="S2.E6.m1.2.2.1.1.1.1.1.cmml" xref="S2.E6.m1.2.2.1.1.1.1.1"></times><apply id="S2.E6.m1.2.2.1.1.1.1.2.cmml" xref="S2.E6.m1.2.2.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E6.m1.2.2.1.1.1.1.2.1.cmml" xref="S2.E6.m1.2.2.1.1.1.1.2">superscript</csymbol><apply id="S2.E6.m1.2.2.1.1.1.1.2.2.cmml" xref="S2.E6.m1.2.2.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E6.m1.2.2.1.1.1.1.2.2.1.cmml" xref="S2.E6.m1.2.2.1.1.1.1.2">subscript</csymbol><ci id="S2.E6.m1.2.2.1.1.1.1.2.2.2.cmml" xref="S2.E6.m1.2.2.1.1.1.1.2.2.2">𝑢</ci><ci id="S2.E6.m1.2.2.1.1.1.1.2.2.3.cmml" xref="S2.E6.m1.2.2.1.1.1.1.2.2.3">𝑖</ci></apply><ci id="S2.E6.m1.2.2.1.1.1.1.2.3.cmml" xref="S2.E6.m1.2.2.1.1.1.1.2.3">𝑇</ci></apply><ci id="S2.E6.m1.2.2.1.1.1.1.3.cmml" xref="S2.E6.m1.2.2.1.1.1.1.3">𝑥</ci></apply><apply id="S2.E6.m1.2.2.1.4.cmml" xref="S2.E6.m1.2.2.1.4"><csymbol cd="ambiguous" id="S2.E6.m1.2.2.1.4.1.cmml" xref="S2.E6.m1.2.2.1.4">subscript</csymbol><ci id="S2.E6.m1.2.2.1.4.2.cmml" xref="S2.E6.m1.2.2.1.4.2">𝑣</ci><ci id="S2.E6.m1.2.2.1.4.3.cmml" xref="S2.E6.m1.2.2.1.4.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E6.m1.2c">e_{i}(x):=\sigma(u_{i}^{T}x)v_{i}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(6)</span></td>
</tr></tbody>
</table>
<p id="S2.SS0.SSS0.Px3.p1.4" class="ltx_p">where <math id="S2.SS0.SSS0.Px3.p1.2.m1.2" class="ltx_Math" alttext="v_{i},u_{i}" display="inline"><semantics id="S2.SS0.SSS0.Px3.p1.2.m1.2a"><mrow id="S2.SS0.SSS0.Px3.p1.2.m1.2.2.2" xref="S2.SS0.SSS0.Px3.p1.2.m1.2.2.3.cmml"><msub id="S2.SS0.SSS0.Px3.p1.2.m1.1.1.1.1" xref="S2.SS0.SSS0.Px3.p1.2.m1.1.1.1.1.cmml"><mi id="S2.SS0.SSS0.Px3.p1.2.m1.1.1.1.1.2" xref="S2.SS0.SSS0.Px3.p1.2.m1.1.1.1.1.2.cmml">v</mi><mi id="S2.SS0.SSS0.Px3.p1.2.m1.1.1.1.1.3" xref="S2.SS0.SSS0.Px3.p1.2.m1.1.1.1.1.3.cmml">i</mi></msub><mo id="S2.SS0.SSS0.Px3.p1.2.m1.2.2.2.3" xref="S2.SS0.SSS0.Px3.p1.2.m1.2.2.3.cmml">,</mo><msub id="S2.SS0.SSS0.Px3.p1.2.m1.2.2.2.2" xref="S2.SS0.SSS0.Px3.p1.2.m1.2.2.2.2.cmml"><mi id="S2.SS0.SSS0.Px3.p1.2.m1.2.2.2.2.2" xref="S2.SS0.SSS0.Px3.p1.2.m1.2.2.2.2.2.cmml">u</mi><mi id="S2.SS0.SSS0.Px3.p1.2.m1.2.2.2.2.3" xref="S2.SS0.SSS0.Px3.p1.2.m1.2.2.2.2.3.cmml">i</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px3.p1.2.m1.2b"><list id="S2.SS0.SSS0.Px3.p1.2.m1.2.2.3.cmml" xref="S2.SS0.SSS0.Px3.p1.2.m1.2.2.2"><apply id="S2.SS0.SSS0.Px3.p1.2.m1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px3.p1.2.m1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px3.p1.2.m1.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px3.p1.2.m1.1.1.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px3.p1.2.m1.1.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px3.p1.2.m1.1.1.1.1.2">𝑣</ci><ci id="S2.SS0.SSS0.Px3.p1.2.m1.1.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px3.p1.2.m1.1.1.1.1.3">𝑖</ci></apply><apply id="S2.SS0.SSS0.Px3.p1.2.m1.2.2.2.2.cmml" xref="S2.SS0.SSS0.Px3.p1.2.m1.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px3.p1.2.m1.2.2.2.2.1.cmml" xref="S2.SS0.SSS0.Px3.p1.2.m1.2.2.2.2">subscript</csymbol><ci id="S2.SS0.SSS0.Px3.p1.2.m1.2.2.2.2.2.cmml" xref="S2.SS0.SSS0.Px3.p1.2.m1.2.2.2.2.2">𝑢</ci><ci id="S2.SS0.SSS0.Px3.p1.2.m1.2.2.2.2.3.cmml" xref="S2.SS0.SSS0.Px3.p1.2.m1.2.2.2.2.3">𝑖</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px3.p1.2.m1.2c">v_{i},u_{i}</annotation></semantics></math> are not matrices but vectors with the same dimension as <math id="S2.SS0.SSS0.Px3.p1.3.m2.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S2.SS0.SSS0.Px3.p1.3.m2.1a"><mi id="S2.SS0.SSS0.Px3.p1.3.m2.1.1" xref="S2.SS0.SSS0.Px3.p1.3.m2.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px3.p1.3.m2.1b"><ci id="S2.SS0.SSS0.Px3.p1.3.m2.1.1.cmml" xref="S2.SS0.SSS0.Px3.p1.3.m2.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px3.p1.3.m2.1c">x</annotation></semantics></math>, and <math id="S2.SS0.SSS0.Px3.p1.4.m3.1" class="ltx_Math" alttext="\sigma" display="inline"><semantics id="S2.SS0.SSS0.Px3.p1.4.m3.1a"><mi id="S2.SS0.SSS0.Px3.p1.4.m3.1.1" xref="S2.SS0.SSS0.Px3.p1.4.m3.1.1.cmml">σ</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px3.p1.4.m3.1b"><ci id="S2.SS0.SSS0.Px3.p1.4.m3.1.1.cmml" xref="S2.SS0.SSS0.Px3.p1.4.m3.1.1">𝜎</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px3.p1.4.m3.1c">\sigma</annotation></semantics></math> is a nonlinear activation function such as ReLU or GELU. We omit bias terms here for brevity.</p>
</div>
<div id="S2.SS0.SSS0.Px3.p2" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px3.p2.3" class="ltx_p">Instead of varying the size of individual experts, we adjust the expressiveness of a PEER layer by using multi-head retrieval, similar to the multi-head attention mechanism in transformers and the multi-head memory in PKMs. In particular, we use <math id="S2.SS0.SSS0.Px3.p2.1.m1.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S2.SS0.SSS0.Px3.p2.1.m1.1a"><mi id="S2.SS0.SSS0.Px3.p2.1.m1.1.1" xref="S2.SS0.SSS0.Px3.p2.1.m1.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px3.p2.1.m1.1b"><ci id="S2.SS0.SSS0.Px3.p2.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px3.p2.1.m1.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px3.p2.1.m1.1c">h</annotation></semantics></math> independent query networks instead of one, each computes its own query and retrieves a separate set of <math id="S2.SS0.SSS0.Px3.p2.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S2.SS0.SSS0.Px3.p2.2.m2.1a"><mi id="S2.SS0.SSS0.Px3.p2.2.m2.1.1" xref="S2.SS0.SSS0.Px3.p2.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px3.p2.2.m2.1b"><ci id="S2.SS0.SSS0.Px3.p2.2.m2.1.1.cmml" xref="S2.SS0.SSS0.Px3.p2.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px3.p2.2.m2.1c">k</annotation></semantics></math> experts. However, different heads share the same pool of experts with the same set of product keys. The outputs of these <math id="S2.SS0.SSS0.Px3.p2.3.m3.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S2.SS0.SSS0.Px3.p2.3.m3.1a"><mi id="S2.SS0.SSS0.Px3.p2.3.m3.1.1" xref="S2.SS0.SSS0.Px3.p2.3.m3.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px3.p2.3.m3.1b"><ci id="S2.SS0.SSS0.Px3.p2.3.m3.1.1.cmml" xref="S2.SS0.SSS0.Px3.p2.3.m3.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px3.p2.3.m3.1c">h</annotation></semantics></math> heads are simply summed up:</p>
<table id="S2.E7" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E7.m1.4" class="ltx_Math" alttext="f(x):=\sum_{i=1}^{h}f^{i}(x)=\sum_{i=1}^{h}\sum_{j\in\mathbb{I}^{i}}g_{j}(x)e_{j}(x)" display="block"><semantics id="S2.E7.m1.4a"><mrow id="S2.E7.m1.4.5" xref="S2.E7.m1.4.5.cmml"><mrow id="S2.E7.m1.4.5.2" xref="S2.E7.m1.4.5.2.cmml"><mi id="S2.E7.m1.4.5.2.2" xref="S2.E7.m1.4.5.2.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S2.E7.m1.4.5.2.1" xref="S2.E7.m1.4.5.2.1.cmml">​</mo><mrow id="S2.E7.m1.4.5.2.3.2" xref="S2.E7.m1.4.5.2.cmml"><mo stretchy="false" id="S2.E7.m1.4.5.2.3.2.1" xref="S2.E7.m1.4.5.2.cmml">(</mo><mi id="S2.E7.m1.1.1" xref="S2.E7.m1.1.1.cmml">x</mi><mo rspace="0.278em" stretchy="false" id="S2.E7.m1.4.5.2.3.2.2" xref="S2.E7.m1.4.5.2.cmml">)</mo></mrow></mrow><mo rspace="0.111em" id="S2.E7.m1.4.5.3" xref="S2.E7.m1.4.5.3.cmml">:=</mo><mrow id="S2.E7.m1.4.5.4" xref="S2.E7.m1.4.5.4.cmml"><munderover id="S2.E7.m1.4.5.4.1" xref="S2.E7.m1.4.5.4.1.cmml"><mo movablelimits="false" id="S2.E7.m1.4.5.4.1.2.2" xref="S2.E7.m1.4.5.4.1.2.2.cmml">∑</mo><mrow id="S2.E7.m1.4.5.4.1.2.3" xref="S2.E7.m1.4.5.4.1.2.3.cmml"><mi id="S2.E7.m1.4.5.4.1.2.3.2" xref="S2.E7.m1.4.5.4.1.2.3.2.cmml">i</mi><mo id="S2.E7.m1.4.5.4.1.2.3.1" xref="S2.E7.m1.4.5.4.1.2.3.1.cmml">=</mo><mn id="S2.E7.m1.4.5.4.1.2.3.3" xref="S2.E7.m1.4.5.4.1.2.3.3.cmml">1</mn></mrow><mi id="S2.E7.m1.4.5.4.1.3" xref="S2.E7.m1.4.5.4.1.3.cmml">h</mi></munderover><mrow id="S2.E7.m1.4.5.4.2" xref="S2.E7.m1.4.5.4.2.cmml"><msup id="S2.E7.m1.4.5.4.2.2" xref="S2.E7.m1.4.5.4.2.2.cmml"><mi id="S2.E7.m1.4.5.4.2.2.2" xref="S2.E7.m1.4.5.4.2.2.2.cmml">f</mi><mi id="S2.E7.m1.4.5.4.2.2.3" xref="S2.E7.m1.4.5.4.2.2.3.cmml">i</mi></msup><mo lspace="0em" rspace="0em" id="S2.E7.m1.4.5.4.2.1" xref="S2.E7.m1.4.5.4.2.1.cmml">​</mo><mrow id="S2.E7.m1.4.5.4.2.3.2" xref="S2.E7.m1.4.5.4.2.cmml"><mo stretchy="false" id="S2.E7.m1.4.5.4.2.3.2.1" xref="S2.E7.m1.4.5.4.2.cmml">(</mo><mi id="S2.E7.m1.2.2" xref="S2.E7.m1.2.2.cmml">x</mi><mo stretchy="false" id="S2.E7.m1.4.5.4.2.3.2.2" xref="S2.E7.m1.4.5.4.2.cmml">)</mo></mrow></mrow></mrow><mo rspace="0.111em" id="S2.E7.m1.4.5.5" xref="S2.E7.m1.4.5.5.cmml">=</mo><mrow id="S2.E7.m1.4.5.6" xref="S2.E7.m1.4.5.6.cmml"><munderover id="S2.E7.m1.4.5.6.1" xref="S2.E7.m1.4.5.6.1.cmml"><mo movablelimits="false" rspace="0em" id="S2.E7.m1.4.5.6.1.2.2" xref="S2.E7.m1.4.5.6.1.2.2.cmml">∑</mo><mrow id="S2.E7.m1.4.5.6.1.2.3" xref="S2.E7.m1.4.5.6.1.2.3.cmml"><mi id="S2.E7.m1.4.5.6.1.2.3.2" xref="S2.E7.m1.4.5.6.1.2.3.2.cmml">i</mi><mo id="S2.E7.m1.4.5.6.1.2.3.1" xref="S2.E7.m1.4.5.6.1.2.3.1.cmml">=</mo><mn id="S2.E7.m1.4.5.6.1.2.3.3" xref="S2.E7.m1.4.5.6.1.2.3.3.cmml">1</mn></mrow><mi id="S2.E7.m1.4.5.6.1.3" xref="S2.E7.m1.4.5.6.1.3.cmml">h</mi></munderover><mrow id="S2.E7.m1.4.5.6.2" xref="S2.E7.m1.4.5.6.2.cmml"><munder id="S2.E7.m1.4.5.6.2.1" xref="S2.E7.m1.4.5.6.2.1.cmml"><mo movablelimits="false" id="S2.E7.m1.4.5.6.2.1.2" xref="S2.E7.m1.4.5.6.2.1.2.cmml">∑</mo><mrow id="S2.E7.m1.4.5.6.2.1.3" xref="S2.E7.m1.4.5.6.2.1.3.cmml"><mi id="S2.E7.m1.4.5.6.2.1.3.2" xref="S2.E7.m1.4.5.6.2.1.3.2.cmml">j</mi><mo id="S2.E7.m1.4.5.6.2.1.3.1" xref="S2.E7.m1.4.5.6.2.1.3.1.cmml">∈</mo><msup id="S2.E7.m1.4.5.6.2.1.3.3" xref="S2.E7.m1.4.5.6.2.1.3.3.cmml"><mi id="S2.E7.m1.4.5.6.2.1.3.3.2" xref="S2.E7.m1.4.5.6.2.1.3.3.2.cmml">𝕀</mi><mi id="S2.E7.m1.4.5.6.2.1.3.3.3" xref="S2.E7.m1.4.5.6.2.1.3.3.3.cmml">i</mi></msup></mrow></munder><mrow id="S2.E7.m1.4.5.6.2.2" xref="S2.E7.m1.4.5.6.2.2.cmml"><msub id="S2.E7.m1.4.5.6.2.2.2" xref="S2.E7.m1.4.5.6.2.2.2.cmml"><mi id="S2.E7.m1.4.5.6.2.2.2.2" xref="S2.E7.m1.4.5.6.2.2.2.2.cmml">g</mi><mi id="S2.E7.m1.4.5.6.2.2.2.3" xref="S2.E7.m1.4.5.6.2.2.2.3.cmml">j</mi></msub><mo lspace="0em" rspace="0em" id="S2.E7.m1.4.5.6.2.2.1" xref="S2.E7.m1.4.5.6.2.2.1.cmml">​</mo><mrow id="S2.E7.m1.4.5.6.2.2.3.2" xref="S2.E7.m1.4.5.6.2.2.cmml"><mo stretchy="false" id="S2.E7.m1.4.5.6.2.2.3.2.1" xref="S2.E7.m1.4.5.6.2.2.cmml">(</mo><mi id="S2.E7.m1.3.3" xref="S2.E7.m1.3.3.cmml">x</mi><mo stretchy="false" id="S2.E7.m1.4.5.6.2.2.3.2.2" xref="S2.E7.m1.4.5.6.2.2.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S2.E7.m1.4.5.6.2.2.1a" xref="S2.E7.m1.4.5.6.2.2.1.cmml">​</mo><msub id="S2.E7.m1.4.5.6.2.2.4" xref="S2.E7.m1.4.5.6.2.2.4.cmml"><mi id="S2.E7.m1.4.5.6.2.2.4.2" xref="S2.E7.m1.4.5.6.2.2.4.2.cmml">e</mi><mi id="S2.E7.m1.4.5.6.2.2.4.3" xref="S2.E7.m1.4.5.6.2.2.4.3.cmml">j</mi></msub><mo lspace="0em" rspace="0em" id="S2.E7.m1.4.5.6.2.2.1b" xref="S2.E7.m1.4.5.6.2.2.1.cmml">​</mo><mrow id="S2.E7.m1.4.5.6.2.2.5.2" xref="S2.E7.m1.4.5.6.2.2.cmml"><mo stretchy="false" id="S2.E7.m1.4.5.6.2.2.5.2.1" xref="S2.E7.m1.4.5.6.2.2.cmml">(</mo><mi id="S2.E7.m1.4.4" xref="S2.E7.m1.4.4.cmml">x</mi><mo stretchy="false" id="S2.E7.m1.4.5.6.2.2.5.2.2" xref="S2.E7.m1.4.5.6.2.2.cmml">)</mo></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.E7.m1.4b"><apply id="S2.E7.m1.4.5.cmml" xref="S2.E7.m1.4.5"><and id="S2.E7.m1.4.5a.cmml" xref="S2.E7.m1.4.5"></and><apply id="S2.E7.m1.4.5b.cmml" xref="S2.E7.m1.4.5"><csymbol cd="latexml" id="S2.E7.m1.4.5.3.cmml" xref="S2.E7.m1.4.5.3">assign</csymbol><apply id="S2.E7.m1.4.5.2.cmml" xref="S2.E7.m1.4.5.2"><times id="S2.E7.m1.4.5.2.1.cmml" xref="S2.E7.m1.4.5.2.1"></times><ci id="S2.E7.m1.4.5.2.2.cmml" xref="S2.E7.m1.4.5.2.2">𝑓</ci><ci id="S2.E7.m1.1.1.cmml" xref="S2.E7.m1.1.1">𝑥</ci></apply><apply id="S2.E7.m1.4.5.4.cmml" xref="S2.E7.m1.4.5.4"><apply id="S2.E7.m1.4.5.4.1.cmml" xref="S2.E7.m1.4.5.4.1"><csymbol cd="ambiguous" id="S2.E7.m1.4.5.4.1.1.cmml" xref="S2.E7.m1.4.5.4.1">superscript</csymbol><apply id="S2.E7.m1.4.5.4.1.2.cmml" xref="S2.E7.m1.4.5.4.1"><csymbol cd="ambiguous" id="S2.E7.m1.4.5.4.1.2.1.cmml" xref="S2.E7.m1.4.5.4.1">subscript</csymbol><sum id="S2.E7.m1.4.5.4.1.2.2.cmml" xref="S2.E7.m1.4.5.4.1.2.2"></sum><apply id="S2.E7.m1.4.5.4.1.2.3.cmml" xref="S2.E7.m1.4.5.4.1.2.3"><eq id="S2.E7.m1.4.5.4.1.2.3.1.cmml" xref="S2.E7.m1.4.5.4.1.2.3.1"></eq><ci id="S2.E7.m1.4.5.4.1.2.3.2.cmml" xref="S2.E7.m1.4.5.4.1.2.3.2">𝑖</ci><cn type="integer" id="S2.E7.m1.4.5.4.1.2.3.3.cmml" xref="S2.E7.m1.4.5.4.1.2.3.3">1</cn></apply></apply><ci id="S2.E7.m1.4.5.4.1.3.cmml" xref="S2.E7.m1.4.5.4.1.3">ℎ</ci></apply><apply id="S2.E7.m1.4.5.4.2.cmml" xref="S2.E7.m1.4.5.4.2"><times id="S2.E7.m1.4.5.4.2.1.cmml" xref="S2.E7.m1.4.5.4.2.1"></times><apply id="S2.E7.m1.4.5.4.2.2.cmml" xref="S2.E7.m1.4.5.4.2.2"><csymbol cd="ambiguous" id="S2.E7.m1.4.5.4.2.2.1.cmml" xref="S2.E7.m1.4.5.4.2.2">superscript</csymbol><ci id="S2.E7.m1.4.5.4.2.2.2.cmml" xref="S2.E7.m1.4.5.4.2.2.2">𝑓</ci><ci id="S2.E7.m1.4.5.4.2.2.3.cmml" xref="S2.E7.m1.4.5.4.2.2.3">𝑖</ci></apply><ci id="S2.E7.m1.2.2.cmml" xref="S2.E7.m1.2.2">𝑥</ci></apply></apply></apply><apply id="S2.E7.m1.4.5c.cmml" xref="S2.E7.m1.4.5"><eq id="S2.E7.m1.4.5.5.cmml" xref="S2.E7.m1.4.5.5"></eq><share href="#S2.E7.m1.4.5.4.cmml" id="S2.E7.m1.4.5d.cmml" xref="S2.E7.m1.4.5"></share><apply id="S2.E7.m1.4.5.6.cmml" xref="S2.E7.m1.4.5.6"><apply id="S2.E7.m1.4.5.6.1.cmml" xref="S2.E7.m1.4.5.6.1"><csymbol cd="ambiguous" id="S2.E7.m1.4.5.6.1.1.cmml" xref="S2.E7.m1.4.5.6.1">superscript</csymbol><apply id="S2.E7.m1.4.5.6.1.2.cmml" xref="S2.E7.m1.4.5.6.1"><csymbol cd="ambiguous" id="S2.E7.m1.4.5.6.1.2.1.cmml" xref="S2.E7.m1.4.5.6.1">subscript</csymbol><sum id="S2.E7.m1.4.5.6.1.2.2.cmml" xref="S2.E7.m1.4.5.6.1.2.2"></sum><apply id="S2.E7.m1.4.5.6.1.2.3.cmml" xref="S2.E7.m1.4.5.6.1.2.3"><eq id="S2.E7.m1.4.5.6.1.2.3.1.cmml" xref="S2.E7.m1.4.5.6.1.2.3.1"></eq><ci id="S2.E7.m1.4.5.6.1.2.3.2.cmml" xref="S2.E7.m1.4.5.6.1.2.3.2">𝑖</ci><cn type="integer" id="S2.E7.m1.4.5.6.1.2.3.3.cmml" xref="S2.E7.m1.4.5.6.1.2.3.3">1</cn></apply></apply><ci id="S2.E7.m1.4.5.6.1.3.cmml" xref="S2.E7.m1.4.5.6.1.3">ℎ</ci></apply><apply id="S2.E7.m1.4.5.6.2.cmml" xref="S2.E7.m1.4.5.6.2"><apply id="S2.E7.m1.4.5.6.2.1.cmml" xref="S2.E7.m1.4.5.6.2.1"><csymbol cd="ambiguous" id="S2.E7.m1.4.5.6.2.1.1.cmml" xref="S2.E7.m1.4.5.6.2.1">subscript</csymbol><sum id="S2.E7.m1.4.5.6.2.1.2.cmml" xref="S2.E7.m1.4.5.6.2.1.2"></sum><apply id="S2.E7.m1.4.5.6.2.1.3.cmml" xref="S2.E7.m1.4.5.6.2.1.3"><in id="S2.E7.m1.4.5.6.2.1.3.1.cmml" xref="S2.E7.m1.4.5.6.2.1.3.1"></in><ci id="S2.E7.m1.4.5.6.2.1.3.2.cmml" xref="S2.E7.m1.4.5.6.2.1.3.2">𝑗</ci><apply id="S2.E7.m1.4.5.6.2.1.3.3.cmml" xref="S2.E7.m1.4.5.6.2.1.3.3"><csymbol cd="ambiguous" id="S2.E7.m1.4.5.6.2.1.3.3.1.cmml" xref="S2.E7.m1.4.5.6.2.1.3.3">superscript</csymbol><ci id="S2.E7.m1.4.5.6.2.1.3.3.2.cmml" xref="S2.E7.m1.4.5.6.2.1.3.3.2">𝕀</ci><ci id="S2.E7.m1.4.5.6.2.1.3.3.3.cmml" xref="S2.E7.m1.4.5.6.2.1.3.3.3">𝑖</ci></apply></apply></apply><apply id="S2.E7.m1.4.5.6.2.2.cmml" xref="S2.E7.m1.4.5.6.2.2"><times id="S2.E7.m1.4.5.6.2.2.1.cmml" xref="S2.E7.m1.4.5.6.2.2.1"></times><apply id="S2.E7.m1.4.5.6.2.2.2.cmml" xref="S2.E7.m1.4.5.6.2.2.2"><csymbol cd="ambiguous" id="S2.E7.m1.4.5.6.2.2.2.1.cmml" xref="S2.E7.m1.4.5.6.2.2.2">subscript</csymbol><ci id="S2.E7.m1.4.5.6.2.2.2.2.cmml" xref="S2.E7.m1.4.5.6.2.2.2.2">𝑔</ci><ci id="S2.E7.m1.4.5.6.2.2.2.3.cmml" xref="S2.E7.m1.4.5.6.2.2.2.3">𝑗</ci></apply><ci id="S2.E7.m1.3.3.cmml" xref="S2.E7.m1.3.3">𝑥</ci><apply id="S2.E7.m1.4.5.6.2.2.4.cmml" xref="S2.E7.m1.4.5.6.2.2.4"><csymbol cd="ambiguous" id="S2.E7.m1.4.5.6.2.2.4.1.cmml" xref="S2.E7.m1.4.5.6.2.2.4">subscript</csymbol><ci id="S2.E7.m1.4.5.6.2.2.4.2.cmml" xref="S2.E7.m1.4.5.6.2.2.4.2">𝑒</ci><ci id="S2.E7.m1.4.5.6.2.2.4.3.cmml" xref="S2.E7.m1.4.5.6.2.2.4.3">𝑗</ci></apply><ci id="S2.E7.m1.4.4.cmml" xref="S2.E7.m1.4.4">𝑥</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E7.m1.4c">f(x):=\sum_{i=1}^{h}f^{i}(x)=\sum_{i=1}^{h}\sum_{j\in\mathbb{I}^{i}}g_{j}(x)e_{j}(x)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(7)</span></td>
</tr></tbody>
</table>
<p id="S2.SS0.SSS0.Px3.p2.6" class="ltx_p">One can verify that when only one expert is retrieved (<math id="S2.SS0.SSS0.Px3.p2.4.m1.1" class="ltx_Math" alttext="k=1" display="inline"><semantics id="S2.SS0.SSS0.Px3.p2.4.m1.1a"><mrow id="S2.SS0.SSS0.Px3.p2.4.m1.1.1" xref="S2.SS0.SSS0.Px3.p2.4.m1.1.1.cmml"><mi id="S2.SS0.SSS0.Px3.p2.4.m1.1.1.2" xref="S2.SS0.SSS0.Px3.p2.4.m1.1.1.2.cmml">k</mi><mo id="S2.SS0.SSS0.Px3.p2.4.m1.1.1.1" xref="S2.SS0.SSS0.Px3.p2.4.m1.1.1.1.cmml">=</mo><mn id="S2.SS0.SSS0.Px3.p2.4.m1.1.1.3" xref="S2.SS0.SSS0.Px3.p2.4.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px3.p2.4.m1.1b"><apply id="S2.SS0.SSS0.Px3.p2.4.m1.1.1.cmml" xref="S2.SS0.SSS0.Px3.p2.4.m1.1.1"><eq id="S2.SS0.SSS0.Px3.p2.4.m1.1.1.1.cmml" xref="S2.SS0.SSS0.Px3.p2.4.m1.1.1.1"></eq><ci id="S2.SS0.SSS0.Px3.p2.4.m1.1.1.2.cmml" xref="S2.SS0.SSS0.Px3.p2.4.m1.1.1.2">𝑘</ci><cn type="integer" id="S2.SS0.SSS0.Px3.p2.4.m1.1.1.3.cmml" xref="S2.SS0.SSS0.Px3.p2.4.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px3.p2.4.m1.1c">k=1</annotation></semantics></math>) per head, using a PEER layer with <math id="S2.SS0.SSS0.Px3.p2.5.m2.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S2.SS0.SSS0.Px3.p2.5.m2.1a"><mi id="S2.SS0.SSS0.Px3.p2.5.m2.1.1" xref="S2.SS0.SSS0.Px3.p2.5.m2.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px3.p2.5.m2.1b"><ci id="S2.SS0.SSS0.Px3.p2.5.m2.1.1.cmml" xref="S2.SS0.SSS0.Px3.p2.5.m2.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px3.p2.5.m2.1c">h</annotation></semantics></math> heads is the same as using one expert with <math id="S2.SS0.SSS0.Px3.p2.6.m3.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S2.SS0.SSS0.Px3.p2.6.m3.1a"><mi id="S2.SS0.SSS0.Px3.p2.6.m3.1.1" xref="S2.SS0.SSS0.Px3.p2.6.m3.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px3.p2.6.m3.1b"><ci id="S2.SS0.SSS0.Px3.p2.6.m3.1.1.cmml" xref="S2.SS0.SSS0.Px3.p2.6.m3.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px3.p2.6.m3.1c">h</annotation></semantics></math> hidden neurons:</p>
<table id="S2.E8" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E8.m1.3" class="ltx_Math" alttext="f(x)=\sum_{i=1}^{h}e^{i}(x)=\sum_{i=1}^{h}\sigma(u_{i}^{T}x)v_{i}=V\sigma(W^{T}x);" display="block"><semantics id="S2.E8.m1.3a"><mrow id="S2.E8.m1.3.3.1" xref="S2.E8.m1.3.3.1.1.cmml"><mrow id="S2.E8.m1.3.3.1.1" xref="S2.E8.m1.3.3.1.1.cmml"><mrow id="S2.E8.m1.3.3.1.1.4" xref="S2.E8.m1.3.3.1.1.4.cmml"><mi id="S2.E8.m1.3.3.1.1.4.2" xref="S2.E8.m1.3.3.1.1.4.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S2.E8.m1.3.3.1.1.4.1" xref="S2.E8.m1.3.3.1.1.4.1.cmml">​</mo><mrow id="S2.E8.m1.3.3.1.1.4.3.2" xref="S2.E8.m1.3.3.1.1.4.cmml"><mo stretchy="false" id="S2.E8.m1.3.3.1.1.4.3.2.1" xref="S2.E8.m1.3.3.1.1.4.cmml">(</mo><mi id="S2.E8.m1.1.1" xref="S2.E8.m1.1.1.cmml">x</mi><mo stretchy="false" id="S2.E8.m1.3.3.1.1.4.3.2.2" xref="S2.E8.m1.3.3.1.1.4.cmml">)</mo></mrow></mrow><mo rspace="0.111em" id="S2.E8.m1.3.3.1.1.5" xref="S2.E8.m1.3.3.1.1.5.cmml">=</mo><mrow id="S2.E8.m1.3.3.1.1.6" xref="S2.E8.m1.3.3.1.1.6.cmml"><munderover id="S2.E8.m1.3.3.1.1.6.1" xref="S2.E8.m1.3.3.1.1.6.1.cmml"><mo movablelimits="false" id="S2.E8.m1.3.3.1.1.6.1.2.2" xref="S2.E8.m1.3.3.1.1.6.1.2.2.cmml">∑</mo><mrow id="S2.E8.m1.3.3.1.1.6.1.2.3" xref="S2.E8.m1.3.3.1.1.6.1.2.3.cmml"><mi id="S2.E8.m1.3.3.1.1.6.1.2.3.2" xref="S2.E8.m1.3.3.1.1.6.1.2.3.2.cmml">i</mi><mo id="S2.E8.m1.3.3.1.1.6.1.2.3.1" xref="S2.E8.m1.3.3.1.1.6.1.2.3.1.cmml">=</mo><mn id="S2.E8.m1.3.3.1.1.6.1.2.3.3" xref="S2.E8.m1.3.3.1.1.6.1.2.3.3.cmml">1</mn></mrow><mi id="S2.E8.m1.3.3.1.1.6.1.3" xref="S2.E8.m1.3.3.1.1.6.1.3.cmml">h</mi></munderover><mrow id="S2.E8.m1.3.3.1.1.6.2" xref="S2.E8.m1.3.3.1.1.6.2.cmml"><msup id="S2.E8.m1.3.3.1.1.6.2.2" xref="S2.E8.m1.3.3.1.1.6.2.2.cmml"><mi id="S2.E8.m1.3.3.1.1.6.2.2.2" xref="S2.E8.m1.3.3.1.1.6.2.2.2.cmml">e</mi><mi id="S2.E8.m1.3.3.1.1.6.2.2.3" xref="S2.E8.m1.3.3.1.1.6.2.2.3.cmml">i</mi></msup><mo lspace="0em" rspace="0em" id="S2.E8.m1.3.3.1.1.6.2.1" xref="S2.E8.m1.3.3.1.1.6.2.1.cmml">​</mo><mrow id="S2.E8.m1.3.3.1.1.6.2.3.2" xref="S2.E8.m1.3.3.1.1.6.2.cmml"><mo stretchy="false" id="S2.E8.m1.3.3.1.1.6.2.3.2.1" xref="S2.E8.m1.3.3.1.1.6.2.cmml">(</mo><mi id="S2.E8.m1.2.2" xref="S2.E8.m1.2.2.cmml">x</mi><mo stretchy="false" id="S2.E8.m1.3.3.1.1.6.2.3.2.2" xref="S2.E8.m1.3.3.1.1.6.2.cmml">)</mo></mrow></mrow></mrow><mo rspace="0.111em" id="S2.E8.m1.3.3.1.1.7" xref="S2.E8.m1.3.3.1.1.7.cmml">=</mo><mrow id="S2.E8.m1.3.3.1.1.1" xref="S2.E8.m1.3.3.1.1.1.cmml"><munderover id="S2.E8.m1.3.3.1.1.1.2" xref="S2.E8.m1.3.3.1.1.1.2.cmml"><mo movablelimits="false" id="S2.E8.m1.3.3.1.1.1.2.2.2" xref="S2.E8.m1.3.3.1.1.1.2.2.2.cmml">∑</mo><mrow id="S2.E8.m1.3.3.1.1.1.2.2.3" xref="S2.E8.m1.3.3.1.1.1.2.2.3.cmml"><mi id="S2.E8.m1.3.3.1.1.1.2.2.3.2" xref="S2.E8.m1.3.3.1.1.1.2.2.3.2.cmml">i</mi><mo id="S2.E8.m1.3.3.1.1.1.2.2.3.1" xref="S2.E8.m1.3.3.1.1.1.2.2.3.1.cmml">=</mo><mn id="S2.E8.m1.3.3.1.1.1.2.2.3.3" xref="S2.E8.m1.3.3.1.1.1.2.2.3.3.cmml">1</mn></mrow><mi id="S2.E8.m1.3.3.1.1.1.2.3" xref="S2.E8.m1.3.3.1.1.1.2.3.cmml">h</mi></munderover><mrow id="S2.E8.m1.3.3.1.1.1.1" xref="S2.E8.m1.3.3.1.1.1.1.cmml"><mi id="S2.E8.m1.3.3.1.1.1.1.3" xref="S2.E8.m1.3.3.1.1.1.1.3.cmml">σ</mi><mo lspace="0em" rspace="0em" id="S2.E8.m1.3.3.1.1.1.1.2" xref="S2.E8.m1.3.3.1.1.1.1.2.cmml">​</mo><mrow id="S2.E8.m1.3.3.1.1.1.1.1.1" xref="S2.E8.m1.3.3.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E8.m1.3.3.1.1.1.1.1.1.2" xref="S2.E8.m1.3.3.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E8.m1.3.3.1.1.1.1.1.1.1" xref="S2.E8.m1.3.3.1.1.1.1.1.1.1.cmml"><msubsup id="S2.E8.m1.3.3.1.1.1.1.1.1.1.2" xref="S2.E8.m1.3.3.1.1.1.1.1.1.1.2.cmml"><mi id="S2.E8.m1.3.3.1.1.1.1.1.1.1.2.2.2" xref="S2.E8.m1.3.3.1.1.1.1.1.1.1.2.2.2.cmml">u</mi><mi id="S2.E8.m1.3.3.1.1.1.1.1.1.1.2.2.3" xref="S2.E8.m1.3.3.1.1.1.1.1.1.1.2.2.3.cmml">i</mi><mi id="S2.E8.m1.3.3.1.1.1.1.1.1.1.2.3" xref="S2.E8.m1.3.3.1.1.1.1.1.1.1.2.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="S2.E8.m1.3.3.1.1.1.1.1.1.1.1" xref="S2.E8.m1.3.3.1.1.1.1.1.1.1.1.cmml">​</mo><mi id="S2.E8.m1.3.3.1.1.1.1.1.1.1.3" xref="S2.E8.m1.3.3.1.1.1.1.1.1.1.3.cmml">x</mi></mrow><mo stretchy="false" id="S2.E8.m1.3.3.1.1.1.1.1.1.3" xref="S2.E8.m1.3.3.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S2.E8.m1.3.3.1.1.1.1.2a" xref="S2.E8.m1.3.3.1.1.1.1.2.cmml">​</mo><msub id="S2.E8.m1.3.3.1.1.1.1.4" xref="S2.E8.m1.3.3.1.1.1.1.4.cmml"><mi id="S2.E8.m1.3.3.1.1.1.1.4.2" xref="S2.E8.m1.3.3.1.1.1.1.4.2.cmml">v</mi><mi id="S2.E8.m1.3.3.1.1.1.1.4.3" xref="S2.E8.m1.3.3.1.1.1.1.4.3.cmml">i</mi></msub></mrow></mrow><mo id="S2.E8.m1.3.3.1.1.8" xref="S2.E8.m1.3.3.1.1.8.cmml">=</mo><mrow id="S2.E8.m1.3.3.1.1.2" xref="S2.E8.m1.3.3.1.1.2.cmml"><mi id="S2.E8.m1.3.3.1.1.2.3" xref="S2.E8.m1.3.3.1.1.2.3.cmml">V</mi><mo lspace="0em" rspace="0em" id="S2.E8.m1.3.3.1.1.2.2" xref="S2.E8.m1.3.3.1.1.2.2.cmml">​</mo><mi id="S2.E8.m1.3.3.1.1.2.4" xref="S2.E8.m1.3.3.1.1.2.4.cmml">σ</mi><mo lspace="0em" rspace="0em" id="S2.E8.m1.3.3.1.1.2.2a" xref="S2.E8.m1.3.3.1.1.2.2.cmml">​</mo><mrow id="S2.E8.m1.3.3.1.1.2.1.1" xref="S2.E8.m1.3.3.1.1.2.1.1.1.cmml"><mo stretchy="false" id="S2.E8.m1.3.3.1.1.2.1.1.2" xref="S2.E8.m1.3.3.1.1.2.1.1.1.cmml">(</mo><mrow id="S2.E8.m1.3.3.1.1.2.1.1.1" xref="S2.E8.m1.3.3.1.1.2.1.1.1.cmml"><msup id="S2.E8.m1.3.3.1.1.2.1.1.1.2" xref="S2.E8.m1.3.3.1.1.2.1.1.1.2.cmml"><mi id="S2.E8.m1.3.3.1.1.2.1.1.1.2.2" xref="S2.E8.m1.3.3.1.1.2.1.1.1.2.2.cmml">W</mi><mi id="S2.E8.m1.3.3.1.1.2.1.1.1.2.3" xref="S2.E8.m1.3.3.1.1.2.1.1.1.2.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="S2.E8.m1.3.3.1.1.2.1.1.1.1" xref="S2.E8.m1.3.3.1.1.2.1.1.1.1.cmml">​</mo><mi id="S2.E8.m1.3.3.1.1.2.1.1.1.3" xref="S2.E8.m1.3.3.1.1.2.1.1.1.3.cmml">x</mi></mrow><mo stretchy="false" id="S2.E8.m1.3.3.1.1.2.1.1.3" xref="S2.E8.m1.3.3.1.1.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo id="S2.E8.m1.3.3.1.2" xref="S2.E8.m1.3.3.1.1.cmml">;</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E8.m1.3b"><apply id="S2.E8.m1.3.3.1.1.cmml" xref="S2.E8.m1.3.3.1"><and id="S2.E8.m1.3.3.1.1a.cmml" xref="S2.E8.m1.3.3.1"></and><apply id="S2.E8.m1.3.3.1.1b.cmml" xref="S2.E8.m1.3.3.1"><eq id="S2.E8.m1.3.3.1.1.5.cmml" xref="S2.E8.m1.3.3.1.1.5"></eq><apply id="S2.E8.m1.3.3.1.1.4.cmml" xref="S2.E8.m1.3.3.1.1.4"><times id="S2.E8.m1.3.3.1.1.4.1.cmml" xref="S2.E8.m1.3.3.1.1.4.1"></times><ci id="S2.E8.m1.3.3.1.1.4.2.cmml" xref="S2.E8.m1.3.3.1.1.4.2">𝑓</ci><ci id="S2.E8.m1.1.1.cmml" xref="S2.E8.m1.1.1">𝑥</ci></apply><apply id="S2.E8.m1.3.3.1.1.6.cmml" xref="S2.E8.m1.3.3.1.1.6"><apply id="S2.E8.m1.3.3.1.1.6.1.cmml" xref="S2.E8.m1.3.3.1.1.6.1"><csymbol cd="ambiguous" id="S2.E8.m1.3.3.1.1.6.1.1.cmml" xref="S2.E8.m1.3.3.1.1.6.1">superscript</csymbol><apply id="S2.E8.m1.3.3.1.1.6.1.2.cmml" xref="S2.E8.m1.3.3.1.1.6.1"><csymbol cd="ambiguous" id="S2.E8.m1.3.3.1.1.6.1.2.1.cmml" xref="S2.E8.m1.3.3.1.1.6.1">subscript</csymbol><sum id="S2.E8.m1.3.3.1.1.6.1.2.2.cmml" xref="S2.E8.m1.3.3.1.1.6.1.2.2"></sum><apply id="S2.E8.m1.3.3.1.1.6.1.2.3.cmml" xref="S2.E8.m1.3.3.1.1.6.1.2.3"><eq id="S2.E8.m1.3.3.1.1.6.1.2.3.1.cmml" xref="S2.E8.m1.3.3.1.1.6.1.2.3.1"></eq><ci id="S2.E8.m1.3.3.1.1.6.1.2.3.2.cmml" xref="S2.E8.m1.3.3.1.1.6.1.2.3.2">𝑖</ci><cn type="integer" id="S2.E8.m1.3.3.1.1.6.1.2.3.3.cmml" xref="S2.E8.m1.3.3.1.1.6.1.2.3.3">1</cn></apply></apply><ci id="S2.E8.m1.3.3.1.1.6.1.3.cmml" xref="S2.E8.m1.3.3.1.1.6.1.3">ℎ</ci></apply><apply id="S2.E8.m1.3.3.1.1.6.2.cmml" xref="S2.E8.m1.3.3.1.1.6.2"><times id="S2.E8.m1.3.3.1.1.6.2.1.cmml" xref="S2.E8.m1.3.3.1.1.6.2.1"></times><apply id="S2.E8.m1.3.3.1.1.6.2.2.cmml" xref="S2.E8.m1.3.3.1.1.6.2.2"><csymbol cd="ambiguous" id="S2.E8.m1.3.3.1.1.6.2.2.1.cmml" xref="S2.E8.m1.3.3.1.1.6.2.2">superscript</csymbol><ci id="S2.E8.m1.3.3.1.1.6.2.2.2.cmml" xref="S2.E8.m1.3.3.1.1.6.2.2.2">𝑒</ci><ci id="S2.E8.m1.3.3.1.1.6.2.2.3.cmml" xref="S2.E8.m1.3.3.1.1.6.2.2.3">𝑖</ci></apply><ci id="S2.E8.m1.2.2.cmml" xref="S2.E8.m1.2.2">𝑥</ci></apply></apply></apply><apply id="S2.E8.m1.3.3.1.1c.cmml" xref="S2.E8.m1.3.3.1"><eq id="S2.E8.m1.3.3.1.1.7.cmml" xref="S2.E8.m1.3.3.1.1.7"></eq><share href="#S2.E8.m1.3.3.1.1.6.cmml" id="S2.E8.m1.3.3.1.1d.cmml" xref="S2.E8.m1.3.3.1"></share><apply id="S2.E8.m1.3.3.1.1.1.cmml" xref="S2.E8.m1.3.3.1.1.1"><apply id="S2.E8.m1.3.3.1.1.1.2.cmml" xref="S2.E8.m1.3.3.1.1.1.2"><csymbol cd="ambiguous" id="S2.E8.m1.3.3.1.1.1.2.1.cmml" xref="S2.E8.m1.3.3.1.1.1.2">superscript</csymbol><apply id="S2.E8.m1.3.3.1.1.1.2.2.cmml" xref="S2.E8.m1.3.3.1.1.1.2"><csymbol cd="ambiguous" id="S2.E8.m1.3.3.1.1.1.2.2.1.cmml" xref="S2.E8.m1.3.3.1.1.1.2">subscript</csymbol><sum id="S2.E8.m1.3.3.1.1.1.2.2.2.cmml" xref="S2.E8.m1.3.3.1.1.1.2.2.2"></sum><apply id="S2.E8.m1.3.3.1.1.1.2.2.3.cmml" xref="S2.E8.m1.3.3.1.1.1.2.2.3"><eq id="S2.E8.m1.3.3.1.1.1.2.2.3.1.cmml" xref="S2.E8.m1.3.3.1.1.1.2.2.3.1"></eq><ci id="S2.E8.m1.3.3.1.1.1.2.2.3.2.cmml" xref="S2.E8.m1.3.3.1.1.1.2.2.3.2">𝑖</ci><cn type="integer" id="S2.E8.m1.3.3.1.1.1.2.2.3.3.cmml" xref="S2.E8.m1.3.3.1.1.1.2.2.3.3">1</cn></apply></apply><ci id="S2.E8.m1.3.3.1.1.1.2.3.cmml" xref="S2.E8.m1.3.3.1.1.1.2.3">ℎ</ci></apply><apply id="S2.E8.m1.3.3.1.1.1.1.cmml" xref="S2.E8.m1.3.3.1.1.1.1"><times id="S2.E8.m1.3.3.1.1.1.1.2.cmml" xref="S2.E8.m1.3.3.1.1.1.1.2"></times><ci id="S2.E8.m1.3.3.1.1.1.1.3.cmml" xref="S2.E8.m1.3.3.1.1.1.1.3">𝜎</ci><apply id="S2.E8.m1.3.3.1.1.1.1.1.1.1.cmml" xref="S2.E8.m1.3.3.1.1.1.1.1.1"><times id="S2.E8.m1.3.3.1.1.1.1.1.1.1.1.cmml" xref="S2.E8.m1.3.3.1.1.1.1.1.1.1.1"></times><apply id="S2.E8.m1.3.3.1.1.1.1.1.1.1.2.cmml" xref="S2.E8.m1.3.3.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E8.m1.3.3.1.1.1.1.1.1.1.2.1.cmml" xref="S2.E8.m1.3.3.1.1.1.1.1.1.1.2">superscript</csymbol><apply id="S2.E8.m1.3.3.1.1.1.1.1.1.1.2.2.cmml" xref="S2.E8.m1.3.3.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E8.m1.3.3.1.1.1.1.1.1.1.2.2.1.cmml" xref="S2.E8.m1.3.3.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S2.E8.m1.3.3.1.1.1.1.1.1.1.2.2.2.cmml" xref="S2.E8.m1.3.3.1.1.1.1.1.1.1.2.2.2">𝑢</ci><ci id="S2.E8.m1.3.3.1.1.1.1.1.1.1.2.2.3.cmml" xref="S2.E8.m1.3.3.1.1.1.1.1.1.1.2.2.3">𝑖</ci></apply><ci id="S2.E8.m1.3.3.1.1.1.1.1.1.1.2.3.cmml" xref="S2.E8.m1.3.3.1.1.1.1.1.1.1.2.3">𝑇</ci></apply><ci id="S2.E8.m1.3.3.1.1.1.1.1.1.1.3.cmml" xref="S2.E8.m1.3.3.1.1.1.1.1.1.1.3">𝑥</ci></apply><apply id="S2.E8.m1.3.3.1.1.1.1.4.cmml" xref="S2.E8.m1.3.3.1.1.1.1.4"><csymbol cd="ambiguous" id="S2.E8.m1.3.3.1.1.1.1.4.1.cmml" xref="S2.E8.m1.3.3.1.1.1.1.4">subscript</csymbol><ci id="S2.E8.m1.3.3.1.1.1.1.4.2.cmml" xref="S2.E8.m1.3.3.1.1.1.1.4.2">𝑣</ci><ci id="S2.E8.m1.3.3.1.1.1.1.4.3.cmml" xref="S2.E8.m1.3.3.1.1.1.1.4.3">𝑖</ci></apply></apply></apply></apply><apply id="S2.E8.m1.3.3.1.1e.cmml" xref="S2.E8.m1.3.3.1"><eq id="S2.E8.m1.3.3.1.1.8.cmml" xref="S2.E8.m1.3.3.1.1.8"></eq><share href="#S2.E8.m1.3.3.1.1.1.cmml" id="S2.E8.m1.3.3.1.1f.cmml" xref="S2.E8.m1.3.3.1"></share><apply id="S2.E8.m1.3.3.1.1.2.cmml" xref="S2.E8.m1.3.3.1.1.2"><times id="S2.E8.m1.3.3.1.1.2.2.cmml" xref="S2.E8.m1.3.3.1.1.2.2"></times><ci id="S2.E8.m1.3.3.1.1.2.3.cmml" xref="S2.E8.m1.3.3.1.1.2.3">𝑉</ci><ci id="S2.E8.m1.3.3.1.1.2.4.cmml" xref="S2.E8.m1.3.3.1.1.2.4">𝜎</ci><apply id="S2.E8.m1.3.3.1.1.2.1.1.1.cmml" xref="S2.E8.m1.3.3.1.1.2.1.1"><times id="S2.E8.m1.3.3.1.1.2.1.1.1.1.cmml" xref="S2.E8.m1.3.3.1.1.2.1.1.1.1"></times><apply id="S2.E8.m1.3.3.1.1.2.1.1.1.2.cmml" xref="S2.E8.m1.3.3.1.1.2.1.1.1.2"><csymbol cd="ambiguous" id="S2.E8.m1.3.3.1.1.2.1.1.1.2.1.cmml" xref="S2.E8.m1.3.3.1.1.2.1.1.1.2">superscript</csymbol><ci id="S2.E8.m1.3.3.1.1.2.1.1.1.2.2.cmml" xref="S2.E8.m1.3.3.1.1.2.1.1.1.2.2">𝑊</ci><ci id="S2.E8.m1.3.3.1.1.2.1.1.1.2.3.cmml" xref="S2.E8.m1.3.3.1.1.2.1.1.1.2.3">𝑇</ci></apply><ci id="S2.E8.m1.3.3.1.1.2.1.1.1.3.cmml" xref="S2.E8.m1.3.3.1.1.2.1.1.1.3">𝑥</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E8.m1.3c">f(x)=\sum_{i=1}^{h}e^{i}(x)=\sum_{i=1}^{h}\sigma(u_{i}^{T}x)v_{i}=V\sigma(W^{T}x);</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(8)</span></td>
</tr></tbody>
</table>
<p id="S2.SS0.SSS0.Px3.p2.9" class="ltx_p">where <math id="S2.SS0.SSS0.Px3.p2.7.m1.4" class="ltx_Math" alttext="W=[u_{1},\cdots,u_{h}],V=[v_{1},\cdots,v_{h}]" display="inline"><semantics id="S2.SS0.SSS0.Px3.p2.7.m1.4a"><mrow id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.3.cmml"><mrow id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.cmml"><mi id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.4" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.4.cmml">W</mi><mo id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.3" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.3.cmml">=</mo><mrow id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.2" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.3.cmml"><mo stretchy="false" id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.2.3" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.3.cmml">[</mo><msub id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.1.1.1" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.1.1.1.cmml"><mi id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.1.1.1.2" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.1.1.1.2.cmml">u</mi><mn id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.1.1.1.3" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.1.1.1.3.cmml">1</mn></msub><mo id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.2.4" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS0.SSS0.Px3.p2.7.m1.1.1" xref="S2.SS0.SSS0.Px3.p2.7.m1.1.1.cmml">⋯</mi><mo id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.2.5" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.3.cmml">,</mo><msub id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.2.2" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.2.2.cmml"><mi id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.2.2.2" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.2.2.2.cmml">u</mi><mi id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.2.2.3" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.2.2.3.cmml">h</mi></msub><mo stretchy="false" id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.2.6" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.3.cmml">]</mo></mrow></mrow><mo id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.3" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.3a.cmml">,</mo><mrow id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.cmml"><mi id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.4" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.4.cmml">V</mi><mo id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.3" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.3.cmml">=</mo><mrow id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.2" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.3.cmml"><mo stretchy="false" id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.2.3" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.3.cmml">[</mo><msub id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.1.1.1" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.1.1.1.cmml"><mi id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.1.1.1.2" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.1.1.1.2.cmml">v</mi><mn id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.1.1.1.3" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.1.1.1.3.cmml">1</mn></msub><mo id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.2.4" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS0.SSS0.Px3.p2.7.m1.2.2" xref="S2.SS0.SSS0.Px3.p2.7.m1.2.2.cmml">⋯</mi><mo id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.2.5" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.3.cmml">,</mo><msub id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.2.2" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.2.2.cmml"><mi id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.2.2.2" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.2.2.2.cmml">v</mi><mi id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.2.2.3" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.2.2.3.cmml">h</mi></msub><mo stretchy="false" id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.2.6" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.3.cmml">]</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px3.p2.7.m1.4b"><apply id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.3.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.3a.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.3">formulae-sequence</csymbol><apply id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1"><eq id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.3.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.3"></eq><ci id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.4.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.4">𝑊</ci><list id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.3.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.2"><apply id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.1.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.1.1.1.2">𝑢</ci><cn type="integer" id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.1.1.1.3">1</cn></apply><ci id="S2.SS0.SSS0.Px3.p2.7.m1.1.1.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.1.1">⋯</ci><apply id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.2.2.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.2.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.2.2.1.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.2.2">subscript</csymbol><ci id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.2.2.2.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.2.2.2">𝑢</ci><ci id="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.2.2.3.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.3.3.1.1.2.2.2.3">ℎ</ci></apply></list></apply><apply id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2"><eq id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.3.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.3"></eq><ci id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.4.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.4">𝑉</ci><list id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.3.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.2"><apply id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.1.1.1.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.1.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.1.1.1.2">𝑣</ci><cn type="integer" id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.1.1.1.3">1</cn></apply><ci id="S2.SS0.SSS0.Px3.p2.7.m1.2.2.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.2.2">⋯</ci><apply id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.2.2.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.2.2.1.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.2.2">subscript</csymbol><ci id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.2.2.2.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.2.2.2">𝑣</ci><ci id="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.2.2.3.cmml" xref="S2.SS0.SSS0.Px3.p2.7.m1.4.4.2.2.2.2.2.3">ℎ</ci></apply></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px3.p2.7.m1.4c">W=[u_{1},\cdots,u_{h}],V=[v_{1},\cdots,v_{h}]</annotation></semantics></math>. In other words, PEER dynamically assembles an MLP with <math id="S2.SS0.SSS0.Px3.p2.8.m2.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S2.SS0.SSS0.Px3.p2.8.m2.1a"><mi id="S2.SS0.SSS0.Px3.p2.8.m2.1.1" xref="S2.SS0.SSS0.Px3.p2.8.m2.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px3.p2.8.m2.1b"><ci id="S2.SS0.SSS0.Px3.p2.8.m2.1.1.cmml" xref="S2.SS0.SSS0.Px3.p2.8.m2.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px3.p2.8.m2.1c">h</annotation></semantics></math> neurons by aggregating <math id="S2.SS0.SSS0.Px3.p2.9.m3.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S2.SS0.SSS0.Px3.p2.9.m3.1a"><mi id="S2.SS0.SSS0.Px3.p2.9.m3.1.1" xref="S2.SS0.SSS0.Px3.p2.9.m3.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px3.p2.9.m3.1b"><ci id="S2.SS0.SSS0.Px3.p2.9.m3.1.1.cmml" xref="S2.SS0.SSS0.Px3.p2.9.m3.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px3.p2.9.m3.1c">h</annotation></semantics></math> singleton MLPs retrieved from a shared repository. Compared to existing MoE approaches that use MLPs with multiple hidden neurons as experts, this design allows shared hidden neurons among experts, enhancing knowledge transfer and parameter efficiency.</p>
</div>
<div id="S2.SS0.SSS0.Px3.p3" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px3.p3.1" class="ltx_p">Algorithm <a href="#LST1" title="Algorithm 1 ‣ Why A Large Number of Small Experts? ‣ 2 Method ‣ Mixture of A Million Experts" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> shows a simplified implementation of the PEER forward pass, storing parameter-efficient expert weights in embedding layers and combining them with einsum operations. This implementation can be easily extended to experts of the GLU variants <cite class="ltx_cite ltx_citemacro_citep">(Shazeer, <a href="#bib.bib36" title="" class="ltx_ref">2020</a>)</cite> by adding additional linear gating weights. In practice, an efficient implementation may require specialized hardware kernels to accelerate embedding lookup and fusion with the einsum operations.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Why A Large Number of Small Experts?</h4>

<div id="S2.SS0.SSS0.Px4.p1" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px4.p1.3" class="ltx_p">Given an MoE layer, we can characterize it by three hyperparameters: the total number of parameters <math id="S2.SS0.SSS0.Px4.p1.1.m1.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S2.SS0.SSS0.Px4.p1.1.m1.1a"><mi id="S2.SS0.SSS0.Px4.p1.1.m1.1.1" xref="S2.SS0.SSS0.Px4.p1.1.m1.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p1.1.m1.1b"><ci id="S2.SS0.SSS0.Px4.p1.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p1.1.m1.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p1.1.m1.1c">P</annotation></semantics></math>, the number of active parameters per token <math id="S2.SS0.SSS0.Px4.p1.2.m2.1" class="ltx_Math" alttext="P_{\text{active}}" display="inline"><semantics id="S2.SS0.SSS0.Px4.p1.2.m2.1a"><msub id="S2.SS0.SSS0.Px4.p1.2.m2.1.1" xref="S2.SS0.SSS0.Px4.p1.2.m2.1.1.cmml"><mi id="S2.SS0.SSS0.Px4.p1.2.m2.1.1.2" xref="S2.SS0.SSS0.Px4.p1.2.m2.1.1.2.cmml">P</mi><mtext id="S2.SS0.SSS0.Px4.p1.2.m2.1.1.3" xref="S2.SS0.SSS0.Px4.p1.2.m2.1.1.3a.cmml">active</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p1.2.m2.1b"><apply id="S2.SS0.SSS0.Px4.p1.2.m2.1.1.cmml" xref="S2.SS0.SSS0.Px4.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p1.2.m2.1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p1.2.m2.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p1.2.m2.1.1.2.cmml" xref="S2.SS0.SSS0.Px4.p1.2.m2.1.1.2">𝑃</ci><ci id="S2.SS0.SSS0.Px4.p1.2.m2.1.1.3a.cmml" xref="S2.SS0.SSS0.Px4.p1.2.m2.1.1.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p1.2.m2.1.1.3.cmml" xref="S2.SS0.SSS0.Px4.p1.2.m2.1.1.3">active</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p1.2.m2.1c">P_{\text{active}}</annotation></semantics></math> and the size of a single expert <math id="S2.SS0.SSS0.Px4.p1.3.m3.1" class="ltx_Math" alttext="P_{\text{expert}}" display="inline"><semantics id="S2.SS0.SSS0.Px4.p1.3.m3.1a"><msub id="S2.SS0.SSS0.Px4.p1.3.m3.1.1" xref="S2.SS0.SSS0.Px4.p1.3.m3.1.1.cmml"><mi id="S2.SS0.SSS0.Px4.p1.3.m3.1.1.2" xref="S2.SS0.SSS0.Px4.p1.3.m3.1.1.2.cmml">P</mi><mtext id="S2.SS0.SSS0.Px4.p1.3.m3.1.1.3" xref="S2.SS0.SSS0.Px4.p1.3.m3.1.1.3a.cmml">expert</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p1.3.m3.1b"><apply id="S2.SS0.SSS0.Px4.p1.3.m3.1.1.cmml" xref="S2.SS0.SSS0.Px4.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p1.3.m3.1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p1.3.m3.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p1.3.m3.1.1.2.cmml" xref="S2.SS0.SSS0.Px4.p1.3.m3.1.1.2">𝑃</ci><ci id="S2.SS0.SSS0.Px4.p1.3.m3.1.1.3a.cmml" xref="S2.SS0.SSS0.Px4.p1.3.m3.1.1.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p1.3.m3.1.1.3.cmml" xref="S2.SS0.SSS0.Px4.p1.3.m3.1.1.3">expert</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p1.3.m3.1c">P_{\text{expert}}</annotation></semantics></math>. <cite class="ltx_cite ltx_citemacro_citet">Krajewski et al. (<a href="#bib.bib25" title="" class="ltx_ref">2024</a>)</cite> showed that the scaling law of MoE models has the following form:</p>
<table id="S2.E9" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E9.m1.4" class="ltx_Math" alttext="\mathcal{L}(P,D,G)=c+(\frac{g}{G^{\gamma}}+a)\frac{1}{P^{\alpha}}+\frac{b}{D^{\beta}}," display="block"><semantics id="S2.E9.m1.4a"><mrow id="S2.E9.m1.4.4.1" xref="S2.E9.m1.4.4.1.1.cmml"><mrow id="S2.E9.m1.4.4.1.1" xref="S2.E9.m1.4.4.1.1.cmml"><mrow id="S2.E9.m1.4.4.1.1.3" xref="S2.E9.m1.4.4.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S2.E9.m1.4.4.1.1.3.2" xref="S2.E9.m1.4.4.1.1.3.2.cmml">ℒ</mi><mo lspace="0em" rspace="0em" id="S2.E9.m1.4.4.1.1.3.1" xref="S2.E9.m1.4.4.1.1.3.1.cmml">​</mo><mrow id="S2.E9.m1.4.4.1.1.3.3.2" xref="S2.E9.m1.4.4.1.1.3.3.1.cmml"><mo stretchy="false" id="S2.E9.m1.4.4.1.1.3.3.2.1" xref="S2.E9.m1.4.4.1.1.3.3.1.cmml">(</mo><mi id="S2.E9.m1.1.1" xref="S2.E9.m1.1.1.cmml">P</mi><mo id="S2.E9.m1.4.4.1.1.3.3.2.2" xref="S2.E9.m1.4.4.1.1.3.3.1.cmml">,</mo><mi id="S2.E9.m1.2.2" xref="S2.E9.m1.2.2.cmml">D</mi><mo id="S2.E9.m1.4.4.1.1.3.3.2.3" xref="S2.E9.m1.4.4.1.1.3.3.1.cmml">,</mo><mi id="S2.E9.m1.3.3" xref="S2.E9.m1.3.3.cmml">G</mi><mo stretchy="false" id="S2.E9.m1.4.4.1.1.3.3.2.4" xref="S2.E9.m1.4.4.1.1.3.3.1.cmml">)</mo></mrow></mrow><mo id="S2.E9.m1.4.4.1.1.2" xref="S2.E9.m1.4.4.1.1.2.cmml">=</mo><mrow id="S2.E9.m1.4.4.1.1.1" xref="S2.E9.m1.4.4.1.1.1.cmml"><mi id="S2.E9.m1.4.4.1.1.1.3" xref="S2.E9.m1.4.4.1.1.1.3.cmml">c</mi><mo id="S2.E9.m1.4.4.1.1.1.2" xref="S2.E9.m1.4.4.1.1.1.2.cmml">+</mo><mrow id="S2.E9.m1.4.4.1.1.1.1" xref="S2.E9.m1.4.4.1.1.1.1.cmml"><mrow id="S2.E9.m1.4.4.1.1.1.1.1.1" xref="S2.E9.m1.4.4.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E9.m1.4.4.1.1.1.1.1.1.2" xref="S2.E9.m1.4.4.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E9.m1.4.4.1.1.1.1.1.1.1" xref="S2.E9.m1.4.4.1.1.1.1.1.1.1.cmml"><mfrac id="S2.E9.m1.4.4.1.1.1.1.1.1.1.2" xref="S2.E9.m1.4.4.1.1.1.1.1.1.1.2.cmml"><mi id="S2.E9.m1.4.4.1.1.1.1.1.1.1.2.2" xref="S2.E9.m1.4.4.1.1.1.1.1.1.1.2.2.cmml">g</mi><msup id="S2.E9.m1.4.4.1.1.1.1.1.1.1.2.3" xref="S2.E9.m1.4.4.1.1.1.1.1.1.1.2.3.cmml"><mi id="S2.E9.m1.4.4.1.1.1.1.1.1.1.2.3.2" xref="S2.E9.m1.4.4.1.1.1.1.1.1.1.2.3.2.cmml">G</mi><mi id="S2.E9.m1.4.4.1.1.1.1.1.1.1.2.3.3" xref="S2.E9.m1.4.4.1.1.1.1.1.1.1.2.3.3.cmml">γ</mi></msup></mfrac><mo id="S2.E9.m1.4.4.1.1.1.1.1.1.1.1" xref="S2.E9.m1.4.4.1.1.1.1.1.1.1.1.cmml">+</mo><mi id="S2.E9.m1.4.4.1.1.1.1.1.1.1.3" xref="S2.E9.m1.4.4.1.1.1.1.1.1.1.3.cmml">a</mi></mrow><mo stretchy="false" id="S2.E9.m1.4.4.1.1.1.1.1.1.3" xref="S2.E9.m1.4.4.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S2.E9.m1.4.4.1.1.1.1.2" xref="S2.E9.m1.4.4.1.1.1.1.2.cmml">​</mo><mfrac id="S2.E9.m1.4.4.1.1.1.1.3" xref="S2.E9.m1.4.4.1.1.1.1.3.cmml"><mn id="S2.E9.m1.4.4.1.1.1.1.3.2" xref="S2.E9.m1.4.4.1.1.1.1.3.2.cmml">1</mn><msup id="S2.E9.m1.4.4.1.1.1.1.3.3" xref="S2.E9.m1.4.4.1.1.1.1.3.3.cmml"><mi id="S2.E9.m1.4.4.1.1.1.1.3.3.2" xref="S2.E9.m1.4.4.1.1.1.1.3.3.2.cmml">P</mi><mi id="S2.E9.m1.4.4.1.1.1.1.3.3.3" xref="S2.E9.m1.4.4.1.1.1.1.3.3.3.cmml">α</mi></msup></mfrac></mrow><mo id="S2.E9.m1.4.4.1.1.1.2a" xref="S2.E9.m1.4.4.1.1.1.2.cmml">+</mo><mfrac id="S2.E9.m1.4.4.1.1.1.4" xref="S2.E9.m1.4.4.1.1.1.4.cmml"><mi id="S2.E9.m1.4.4.1.1.1.4.2" xref="S2.E9.m1.4.4.1.1.1.4.2.cmml">b</mi><msup id="S2.E9.m1.4.4.1.1.1.4.3" xref="S2.E9.m1.4.4.1.1.1.4.3.cmml"><mi id="S2.E9.m1.4.4.1.1.1.4.3.2" xref="S2.E9.m1.4.4.1.1.1.4.3.2.cmml">D</mi><mi id="S2.E9.m1.4.4.1.1.1.4.3.3" xref="S2.E9.m1.4.4.1.1.1.4.3.3.cmml">β</mi></msup></mfrac></mrow></mrow><mo id="S2.E9.m1.4.4.1.2" xref="S2.E9.m1.4.4.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E9.m1.4b"><apply id="S2.E9.m1.4.4.1.1.cmml" xref="S2.E9.m1.4.4.1"><eq id="S2.E9.m1.4.4.1.1.2.cmml" xref="S2.E9.m1.4.4.1.1.2"></eq><apply id="S2.E9.m1.4.4.1.1.3.cmml" xref="S2.E9.m1.4.4.1.1.3"><times id="S2.E9.m1.4.4.1.1.3.1.cmml" xref="S2.E9.m1.4.4.1.1.3.1"></times><ci id="S2.E9.m1.4.4.1.1.3.2.cmml" xref="S2.E9.m1.4.4.1.1.3.2">ℒ</ci><vector id="S2.E9.m1.4.4.1.1.3.3.1.cmml" xref="S2.E9.m1.4.4.1.1.3.3.2"><ci id="S2.E9.m1.1.1.cmml" xref="S2.E9.m1.1.1">𝑃</ci><ci id="S2.E9.m1.2.2.cmml" xref="S2.E9.m1.2.2">𝐷</ci><ci id="S2.E9.m1.3.3.cmml" xref="S2.E9.m1.3.3">𝐺</ci></vector></apply><apply id="S2.E9.m1.4.4.1.1.1.cmml" xref="S2.E9.m1.4.4.1.1.1"><plus id="S2.E9.m1.4.4.1.1.1.2.cmml" xref="S2.E9.m1.4.4.1.1.1.2"></plus><ci id="S2.E9.m1.4.4.1.1.1.3.cmml" xref="S2.E9.m1.4.4.1.1.1.3">𝑐</ci><apply id="S2.E9.m1.4.4.1.1.1.1.cmml" xref="S2.E9.m1.4.4.1.1.1.1"><times id="S2.E9.m1.4.4.1.1.1.1.2.cmml" xref="S2.E9.m1.4.4.1.1.1.1.2"></times><apply id="S2.E9.m1.4.4.1.1.1.1.1.1.1.cmml" xref="S2.E9.m1.4.4.1.1.1.1.1.1"><plus id="S2.E9.m1.4.4.1.1.1.1.1.1.1.1.cmml" xref="S2.E9.m1.4.4.1.1.1.1.1.1.1.1"></plus><apply id="S2.E9.m1.4.4.1.1.1.1.1.1.1.2.cmml" xref="S2.E9.m1.4.4.1.1.1.1.1.1.1.2"><divide id="S2.E9.m1.4.4.1.1.1.1.1.1.1.2.1.cmml" xref="S2.E9.m1.4.4.1.1.1.1.1.1.1.2"></divide><ci id="S2.E9.m1.4.4.1.1.1.1.1.1.1.2.2.cmml" xref="S2.E9.m1.4.4.1.1.1.1.1.1.1.2.2">𝑔</ci><apply id="S2.E9.m1.4.4.1.1.1.1.1.1.1.2.3.cmml" xref="S2.E9.m1.4.4.1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S2.E9.m1.4.4.1.1.1.1.1.1.1.2.3.1.cmml" xref="S2.E9.m1.4.4.1.1.1.1.1.1.1.2.3">superscript</csymbol><ci id="S2.E9.m1.4.4.1.1.1.1.1.1.1.2.3.2.cmml" xref="S2.E9.m1.4.4.1.1.1.1.1.1.1.2.3.2">𝐺</ci><ci id="S2.E9.m1.4.4.1.1.1.1.1.1.1.2.3.3.cmml" xref="S2.E9.m1.4.4.1.1.1.1.1.1.1.2.3.3">𝛾</ci></apply></apply><ci id="S2.E9.m1.4.4.1.1.1.1.1.1.1.3.cmml" xref="S2.E9.m1.4.4.1.1.1.1.1.1.1.3">𝑎</ci></apply><apply id="S2.E9.m1.4.4.1.1.1.1.3.cmml" xref="S2.E9.m1.4.4.1.1.1.1.3"><divide id="S2.E9.m1.4.4.1.1.1.1.3.1.cmml" xref="S2.E9.m1.4.4.1.1.1.1.3"></divide><cn type="integer" id="S2.E9.m1.4.4.1.1.1.1.3.2.cmml" xref="S2.E9.m1.4.4.1.1.1.1.3.2">1</cn><apply id="S2.E9.m1.4.4.1.1.1.1.3.3.cmml" xref="S2.E9.m1.4.4.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S2.E9.m1.4.4.1.1.1.1.3.3.1.cmml" xref="S2.E9.m1.4.4.1.1.1.1.3.3">superscript</csymbol><ci id="S2.E9.m1.4.4.1.1.1.1.3.3.2.cmml" xref="S2.E9.m1.4.4.1.1.1.1.3.3.2">𝑃</ci><ci id="S2.E9.m1.4.4.1.1.1.1.3.3.3.cmml" xref="S2.E9.m1.4.4.1.1.1.1.3.3.3">𝛼</ci></apply></apply></apply><apply id="S2.E9.m1.4.4.1.1.1.4.cmml" xref="S2.E9.m1.4.4.1.1.1.4"><divide id="S2.E9.m1.4.4.1.1.1.4.1.cmml" xref="S2.E9.m1.4.4.1.1.1.4"></divide><ci id="S2.E9.m1.4.4.1.1.1.4.2.cmml" xref="S2.E9.m1.4.4.1.1.1.4.2">𝑏</ci><apply id="S2.E9.m1.4.4.1.1.1.4.3.cmml" xref="S2.E9.m1.4.4.1.1.1.4.3"><csymbol cd="ambiguous" id="S2.E9.m1.4.4.1.1.1.4.3.1.cmml" xref="S2.E9.m1.4.4.1.1.1.4.3">superscript</csymbol><ci id="S2.E9.m1.4.4.1.1.1.4.3.2.cmml" xref="S2.E9.m1.4.4.1.1.1.4.3.2">𝐷</ci><ci id="S2.E9.m1.4.4.1.1.1.4.3.3.cmml" xref="S2.E9.m1.4.4.1.1.1.4.3.3">𝛽</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E9.m1.4c">\mathcal{L}(P,D,G)=c+(\frac{g}{G^{\gamma}}+a)\frac{1}{P^{\alpha}}+\frac{b}{D^{\beta}},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(9)</span></td>
</tr></tbody>
</table>
<p id="S2.SS0.SSS0.Px4.p1.7" class="ltx_p">where <math id="S2.SS0.SSS0.Px4.p1.4.m1.1" class="ltx_Math" alttext="\mathcal{L}" display="inline"><semantics id="S2.SS0.SSS0.Px4.p1.4.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S2.SS0.SSS0.Px4.p1.4.m1.1.1" xref="S2.SS0.SSS0.Px4.p1.4.m1.1.1.cmml">ℒ</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p1.4.m1.1b"><ci id="S2.SS0.SSS0.Px4.p1.4.m1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p1.4.m1.1.1">ℒ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p1.4.m1.1c">\mathcal{L}</annotation></semantics></math> is the final test loss, <math id="S2.SS0.SSS0.Px4.p1.5.m2.6" class="ltx_Math" alttext="a,b,g,\gamma,\alpha,\beta" display="inline"><semantics id="S2.SS0.SSS0.Px4.p1.5.m2.6a"><mrow id="S2.SS0.SSS0.Px4.p1.5.m2.6.7.2" xref="S2.SS0.SSS0.Px4.p1.5.m2.6.7.1.cmml"><mi id="S2.SS0.SSS0.Px4.p1.5.m2.1.1" xref="S2.SS0.SSS0.Px4.p1.5.m2.1.1.cmml">a</mi><mo id="S2.SS0.SSS0.Px4.p1.5.m2.6.7.2.1" xref="S2.SS0.SSS0.Px4.p1.5.m2.6.7.1.cmml">,</mo><mi id="S2.SS0.SSS0.Px4.p1.5.m2.2.2" xref="S2.SS0.SSS0.Px4.p1.5.m2.2.2.cmml">b</mi><mo id="S2.SS0.SSS0.Px4.p1.5.m2.6.7.2.2" xref="S2.SS0.SSS0.Px4.p1.5.m2.6.7.1.cmml">,</mo><mi id="S2.SS0.SSS0.Px4.p1.5.m2.3.3" xref="S2.SS0.SSS0.Px4.p1.5.m2.3.3.cmml">g</mi><mo id="S2.SS0.SSS0.Px4.p1.5.m2.6.7.2.3" xref="S2.SS0.SSS0.Px4.p1.5.m2.6.7.1.cmml">,</mo><mi id="S2.SS0.SSS0.Px4.p1.5.m2.4.4" xref="S2.SS0.SSS0.Px4.p1.5.m2.4.4.cmml">γ</mi><mo id="S2.SS0.SSS0.Px4.p1.5.m2.6.7.2.4" xref="S2.SS0.SSS0.Px4.p1.5.m2.6.7.1.cmml">,</mo><mi id="S2.SS0.SSS0.Px4.p1.5.m2.5.5" xref="S2.SS0.SSS0.Px4.p1.5.m2.5.5.cmml">α</mi><mo id="S2.SS0.SSS0.Px4.p1.5.m2.6.7.2.5" xref="S2.SS0.SSS0.Px4.p1.5.m2.6.7.1.cmml">,</mo><mi id="S2.SS0.SSS0.Px4.p1.5.m2.6.6" xref="S2.SS0.SSS0.Px4.p1.5.m2.6.6.cmml">β</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p1.5.m2.6b"><list id="S2.SS0.SSS0.Px4.p1.5.m2.6.7.1.cmml" xref="S2.SS0.SSS0.Px4.p1.5.m2.6.7.2"><ci id="S2.SS0.SSS0.Px4.p1.5.m2.1.1.cmml" xref="S2.SS0.SSS0.Px4.p1.5.m2.1.1">𝑎</ci><ci id="S2.SS0.SSS0.Px4.p1.5.m2.2.2.cmml" xref="S2.SS0.SSS0.Px4.p1.5.m2.2.2">𝑏</ci><ci id="S2.SS0.SSS0.Px4.p1.5.m2.3.3.cmml" xref="S2.SS0.SSS0.Px4.p1.5.m2.3.3">𝑔</ci><ci id="S2.SS0.SSS0.Px4.p1.5.m2.4.4.cmml" xref="S2.SS0.SSS0.Px4.p1.5.m2.4.4">𝛾</ci><ci id="S2.SS0.SSS0.Px4.p1.5.m2.5.5.cmml" xref="S2.SS0.SSS0.Px4.p1.5.m2.5.5">𝛼</ci><ci id="S2.SS0.SSS0.Px4.p1.5.m2.6.6.cmml" xref="S2.SS0.SSS0.Px4.p1.5.m2.6.6">𝛽</ci></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p1.5.m2.6c">a,b,g,\gamma,\alpha,\beta</annotation></semantics></math> are constants, <math id="S2.SS0.SSS0.Px4.p1.6.m3.1" class="ltx_Math" alttext="D" display="inline"><semantics id="S2.SS0.SSS0.Px4.p1.6.m3.1a"><mi id="S2.SS0.SSS0.Px4.p1.6.m3.1.1" xref="S2.SS0.SSS0.Px4.p1.6.m3.1.1.cmml">D</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p1.6.m3.1b"><ci id="S2.SS0.SSS0.Px4.p1.6.m3.1.1.cmml" xref="S2.SS0.SSS0.Px4.p1.6.m3.1.1">𝐷</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p1.6.m3.1c">D</annotation></semantics></math> is the total number of training tokens and the granularity <math id="S2.SS0.SSS0.Px4.p1.7.m4.1" class="ltx_Math" alttext="G" display="inline"><semantics id="S2.SS0.SSS0.Px4.p1.7.m4.1a"><mi id="S2.SS0.SSS0.Px4.p1.7.m4.1.1" xref="S2.SS0.SSS0.Px4.p1.7.m4.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p1.7.m4.1b"><ci id="S2.SS0.SSS0.Px4.p1.7.m4.1.1.cmml" xref="S2.SS0.SSS0.Px4.p1.7.m4.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p1.7.m4.1c">G</annotation></semantics></math> is the number of active experts:</p>
<table id="S2.E10" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E10.m1.1" class="ltx_Math" alttext="G:=\frac{P_{\text{active}}}{P_{\text{expert}}}" display="block"><semantics id="S2.E10.m1.1a"><mrow id="S2.E10.m1.1.1" xref="S2.E10.m1.1.1.cmml"><mi id="S2.E10.m1.1.1.2" xref="S2.E10.m1.1.1.2.cmml">G</mi><mo lspace="0.278em" rspace="0.278em" id="S2.E10.m1.1.1.1" xref="S2.E10.m1.1.1.1.cmml">:=</mo><mfrac id="S2.E10.m1.1.1.3" xref="S2.E10.m1.1.1.3.cmml"><msub id="S2.E10.m1.1.1.3.2" xref="S2.E10.m1.1.1.3.2.cmml"><mi id="S2.E10.m1.1.1.3.2.2" xref="S2.E10.m1.1.1.3.2.2.cmml">P</mi><mtext id="S2.E10.m1.1.1.3.2.3" xref="S2.E10.m1.1.1.3.2.3a.cmml">active</mtext></msub><msub id="S2.E10.m1.1.1.3.3" xref="S2.E10.m1.1.1.3.3.cmml"><mi id="S2.E10.m1.1.1.3.3.2" xref="S2.E10.m1.1.1.3.3.2.cmml">P</mi><mtext id="S2.E10.m1.1.1.3.3.3" xref="S2.E10.m1.1.1.3.3.3a.cmml">expert</mtext></msub></mfrac></mrow><annotation-xml encoding="MathML-Content" id="S2.E10.m1.1b"><apply id="S2.E10.m1.1.1.cmml" xref="S2.E10.m1.1.1"><csymbol cd="latexml" id="S2.E10.m1.1.1.1.cmml" xref="S2.E10.m1.1.1.1">assign</csymbol><ci id="S2.E10.m1.1.1.2.cmml" xref="S2.E10.m1.1.1.2">𝐺</ci><apply id="S2.E10.m1.1.1.3.cmml" xref="S2.E10.m1.1.1.3"><divide id="S2.E10.m1.1.1.3.1.cmml" xref="S2.E10.m1.1.1.3"></divide><apply id="S2.E10.m1.1.1.3.2.cmml" xref="S2.E10.m1.1.1.3.2"><csymbol cd="ambiguous" id="S2.E10.m1.1.1.3.2.1.cmml" xref="S2.E10.m1.1.1.3.2">subscript</csymbol><ci id="S2.E10.m1.1.1.3.2.2.cmml" xref="S2.E10.m1.1.1.3.2.2">𝑃</ci><ci id="S2.E10.m1.1.1.3.2.3a.cmml" xref="S2.E10.m1.1.1.3.2.3"><mtext mathsize="70%" id="S2.E10.m1.1.1.3.2.3.cmml" xref="S2.E10.m1.1.1.3.2.3">active</mtext></ci></apply><apply id="S2.E10.m1.1.1.3.3.cmml" xref="S2.E10.m1.1.1.3.3"><csymbol cd="ambiguous" id="S2.E10.m1.1.1.3.3.1.cmml" xref="S2.E10.m1.1.1.3.3">subscript</csymbol><ci id="S2.E10.m1.1.1.3.3.2.cmml" xref="S2.E10.m1.1.1.3.3.2">𝑃</ci><ci id="S2.E10.m1.1.1.3.3.3a.cmml" xref="S2.E10.m1.1.1.3.3.3"><mtext mathsize="70%" id="S2.E10.m1.1.1.3.3.3.cmml" xref="S2.E10.m1.1.1.3.3.3">expert</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E10.m1.1c">G:=\frac{P_{\text{active}}}{P_{\text{expert}}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(10)</span></td>
</tr></tbody>
</table>
<p id="S2.SS0.SSS0.Px4.p1.11" class="ltx_p">In order to improve model performance, we need to scale up <math id="S2.SS0.SSS0.Px4.p1.8.m1.3" class="ltx_Math" alttext="P,D,G" display="inline"><semantics id="S2.SS0.SSS0.Px4.p1.8.m1.3a"><mrow id="S2.SS0.SSS0.Px4.p1.8.m1.3.4.2" xref="S2.SS0.SSS0.Px4.p1.8.m1.3.4.1.cmml"><mi id="S2.SS0.SSS0.Px4.p1.8.m1.1.1" xref="S2.SS0.SSS0.Px4.p1.8.m1.1.1.cmml">P</mi><mo id="S2.SS0.SSS0.Px4.p1.8.m1.3.4.2.1" xref="S2.SS0.SSS0.Px4.p1.8.m1.3.4.1.cmml">,</mo><mi id="S2.SS0.SSS0.Px4.p1.8.m1.2.2" xref="S2.SS0.SSS0.Px4.p1.8.m1.2.2.cmml">D</mi><mo id="S2.SS0.SSS0.Px4.p1.8.m1.3.4.2.2" xref="S2.SS0.SSS0.Px4.p1.8.m1.3.4.1.cmml">,</mo><mi id="S2.SS0.SSS0.Px4.p1.8.m1.3.3" xref="S2.SS0.SSS0.Px4.p1.8.m1.3.3.cmml">G</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p1.8.m1.3b"><list id="S2.SS0.SSS0.Px4.p1.8.m1.3.4.1.cmml" xref="S2.SS0.SSS0.Px4.p1.8.m1.3.4.2"><ci id="S2.SS0.SSS0.Px4.p1.8.m1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p1.8.m1.1.1">𝑃</ci><ci id="S2.SS0.SSS0.Px4.p1.8.m1.2.2.cmml" xref="S2.SS0.SSS0.Px4.p1.8.m1.2.2">𝐷</ci><ci id="S2.SS0.SSS0.Px4.p1.8.m1.3.3.cmml" xref="S2.SS0.SSS0.Px4.p1.8.m1.3.3">𝐺</ci></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p1.8.m1.3c">P,D,G</annotation></semantics></math>. On the other hand, it is essential to limit <math id="S2.SS0.SSS0.Px4.p1.9.m2.1" class="ltx_Math" alttext="P_{\text{active}}" display="inline"><semantics id="S2.SS0.SSS0.Px4.p1.9.m2.1a"><msub id="S2.SS0.SSS0.Px4.p1.9.m2.1.1" xref="S2.SS0.SSS0.Px4.p1.9.m2.1.1.cmml"><mi id="S2.SS0.SSS0.Px4.p1.9.m2.1.1.2" xref="S2.SS0.SSS0.Px4.p1.9.m2.1.1.2.cmml">P</mi><mtext id="S2.SS0.SSS0.Px4.p1.9.m2.1.1.3" xref="S2.SS0.SSS0.Px4.p1.9.m2.1.1.3a.cmml">active</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p1.9.m2.1b"><apply id="S2.SS0.SSS0.Px4.p1.9.m2.1.1.cmml" xref="S2.SS0.SSS0.Px4.p1.9.m2.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p1.9.m2.1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p1.9.m2.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p1.9.m2.1.1.2.cmml" xref="S2.SS0.SSS0.Px4.p1.9.m2.1.1.2">𝑃</ci><ci id="S2.SS0.SSS0.Px4.p1.9.m2.1.1.3a.cmml" xref="S2.SS0.SSS0.Px4.p1.9.m2.1.1.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p1.9.m2.1.1.3.cmml" xref="S2.SS0.SSS0.Px4.p1.9.m2.1.1.3">active</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p1.9.m2.1c">P_{\text{active}}</annotation></semantics></math> because the computational and memory costs are primarily determined by the active parameters during training and inference. Notably, the memory footprint corresponding to <math id="S2.SS0.SSS0.Px4.p1.10.m3.1" class="ltx_Math" alttext="P_{\text{active}}" display="inline"><semantics id="S2.SS0.SSS0.Px4.p1.10.m3.1a"><msub id="S2.SS0.SSS0.Px4.p1.10.m3.1.1" xref="S2.SS0.SSS0.Px4.p1.10.m3.1.1.cmml"><mi id="S2.SS0.SSS0.Px4.p1.10.m3.1.1.2" xref="S2.SS0.SSS0.Px4.p1.10.m3.1.1.2.cmml">P</mi><mtext id="S2.SS0.SSS0.Px4.p1.10.m3.1.1.3" xref="S2.SS0.SSS0.Px4.p1.10.m3.1.1.3a.cmml">active</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p1.10.m3.1b"><apply id="S2.SS0.SSS0.Px4.p1.10.m3.1.1.cmml" xref="S2.SS0.SSS0.Px4.p1.10.m3.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p1.10.m3.1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p1.10.m3.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p1.10.m3.1.1.2.cmml" xref="S2.SS0.SSS0.Px4.p1.10.m3.1.1.2">𝑃</ci><ci id="S2.SS0.SSS0.Px4.p1.10.m3.1.1.3a.cmml" xref="S2.SS0.SSS0.Px4.p1.10.m3.1.1.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p1.10.m3.1.1.3.cmml" xref="S2.SS0.SSS0.Px4.p1.10.m3.1.1.3">active</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p1.10.m3.1c">P_{\text{active}}</annotation></semantics></math> has to be multiplied by the number of tokens in a batch, while the memory cost of <math id="S2.SS0.SSS0.Px4.p1.11.m4.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S2.SS0.SSS0.Px4.p1.11.m4.1a"><mi id="S2.SS0.SSS0.Px4.p1.11.m4.1.1" xref="S2.SS0.SSS0.Px4.p1.11.m4.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p1.11.m4.1b"><ci id="S2.SS0.SSS0.Px4.p1.11.m4.1.1.cmml" xref="S2.SS0.SSS0.Px4.p1.11.m4.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p1.11.m4.1c">P</annotation></semantics></math> is independent of the batch size and sequence length because only one copy of the model needs to be stored.</p>
</div>
<div id="S2.SS0.SSS0.Px4.p2" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px4.p2.6" class="ltx_p">As a result, we want to increase <math id="S2.SS0.SSS0.Px4.p2.1.m1.2" class="ltx_Math" alttext="P,G" display="inline"><semantics id="S2.SS0.SSS0.Px4.p2.1.m1.2a"><mrow id="S2.SS0.SSS0.Px4.p2.1.m1.2.3.2" xref="S2.SS0.SSS0.Px4.p2.1.m1.2.3.1.cmml"><mi id="S2.SS0.SSS0.Px4.p2.1.m1.1.1" xref="S2.SS0.SSS0.Px4.p2.1.m1.1.1.cmml">P</mi><mo id="S2.SS0.SSS0.Px4.p2.1.m1.2.3.2.1" xref="S2.SS0.SSS0.Px4.p2.1.m1.2.3.1.cmml">,</mo><mi id="S2.SS0.SSS0.Px4.p2.1.m1.2.2" xref="S2.SS0.SSS0.Px4.p2.1.m1.2.2.cmml">G</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p2.1.m1.2b"><list id="S2.SS0.SSS0.Px4.p2.1.m1.2.3.1.cmml" xref="S2.SS0.SSS0.Px4.p2.1.m1.2.3.2"><ci id="S2.SS0.SSS0.Px4.p2.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p2.1.m1.1.1">𝑃</ci><ci id="S2.SS0.SSS0.Px4.p2.1.m1.2.2.cmml" xref="S2.SS0.SSS0.Px4.p2.1.m1.2.2">𝐺</ci></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p2.1.m1.2c">P,G</annotation></semantics></math> but not <math id="S2.SS0.SSS0.Px4.p2.2.m2.1" class="ltx_Math" alttext="P_{\text{active}}" display="inline"><semantics id="S2.SS0.SSS0.Px4.p2.2.m2.1a"><msub id="S2.SS0.SSS0.Px4.p2.2.m2.1.1" xref="S2.SS0.SSS0.Px4.p2.2.m2.1.1.cmml"><mi id="S2.SS0.SSS0.Px4.p2.2.m2.1.1.2" xref="S2.SS0.SSS0.Px4.p2.2.m2.1.1.2.cmml">P</mi><mtext id="S2.SS0.SSS0.Px4.p2.2.m2.1.1.3" xref="S2.SS0.SSS0.Px4.p2.2.m2.1.1.3a.cmml">active</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p2.2.m2.1b"><apply id="S2.SS0.SSS0.Px4.p2.2.m2.1.1.cmml" xref="S2.SS0.SSS0.Px4.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p2.2.m2.1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p2.2.m2.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p2.2.m2.1.1.2.cmml" xref="S2.SS0.SSS0.Px4.p2.2.m2.1.1.2">𝑃</ci><ci id="S2.SS0.SSS0.Px4.p2.2.m2.1.1.3a.cmml" xref="S2.SS0.SSS0.Px4.p2.2.m2.1.1.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p2.2.m2.1.1.3.cmml" xref="S2.SS0.SSS0.Px4.p2.2.m2.1.1.3">active</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p2.2.m2.1c">P_{\text{active}}</annotation></semantics></math>. Since the expert size <math id="S2.SS0.SSS0.Px4.p2.3.m3.1" class="ltx_Math" alttext="P_{\text{expert}}=P_{\text{active}}/G" display="inline"><semantics id="S2.SS0.SSS0.Px4.p2.3.m3.1a"><mrow id="S2.SS0.SSS0.Px4.p2.3.m3.1.1" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.cmml"><msub id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.2" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.2.cmml"><mi id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.2.2" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.2.2.cmml">P</mi><mtext id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.2.3" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.2.3a.cmml">expert</mtext></msub><mo id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.1" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.1.cmml">=</mo><mrow id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.cmml"><msub id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.2" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.2.cmml"><mi id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.2.2" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.2.2.cmml">P</mi><mtext id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.2.3" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.2.3a.cmml">active</mtext></msub><mo id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.1" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.1.cmml">/</mo><mi id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.3" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.3.cmml">G</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p2.3.m3.1b"><apply id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.cmml" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1"><eq id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.1"></eq><apply id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.2.cmml" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.2.1.cmml" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.2">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.2.2.cmml" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.2.2">𝑃</ci><ci id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.2.3a.cmml" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.2.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.2.3.cmml" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.2.3">expert</mtext></ci></apply><apply id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.cmml" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3"><divide id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.1"></divide><apply id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.2.1.cmml" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.2">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.2.2.cmml" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.2.2">𝑃</ci><ci id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.2.3a.cmml" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.2.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.2.3.cmml" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.2.3">active</mtext></ci></apply><ci id="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px4.p2.3.m3.1.1.3.3">𝐺</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p2.3.m3.1c">P_{\text{expert}}=P_{\text{active}}/G</annotation></semantics></math> and the number of experts <math id="S2.SS0.SSS0.Px4.p2.4.m4.1" class="ltx_Math" alttext="N=P/P_{\text{expert}}=P\cdot G/P_{\text{active}}" display="inline"><semantics id="S2.SS0.SSS0.Px4.p2.4.m4.1a"><mrow id="S2.SS0.SSS0.Px4.p2.4.m4.1.1" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.cmml"><mi id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.2" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.2.cmml">N</mi><mo id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.3" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.3.cmml">=</mo><mrow id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.cmml"><mi id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.2" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.2.cmml">P</mi><mo id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.1" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.1.cmml">/</mo><msub id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.3" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.3.cmml"><mi id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.3.2" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.3.2.cmml">P</mi><mtext id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.3.3" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.3.3a.cmml">expert</mtext></msub></mrow><mo id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.5" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.5.cmml">=</mo><mrow id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.cmml"><mrow id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.2" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.2.cmml"><mi id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.2.2" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.2.2.cmml">P</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.2.1" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.2.1.cmml">⋅</mo><mi id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.2.3" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.2.3.cmml">G</mi></mrow><mo id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.1" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.1.cmml">/</mo><msub id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.3" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.3.cmml"><mi id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.3.2" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.3.2.cmml">P</mi><mtext id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.3.3" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.3.3a.cmml">active</mtext></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p2.4.m4.1b"><apply id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1"><and id="S2.SS0.SSS0.Px4.p2.4.m4.1.1a.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1"></and><apply id="S2.SS0.SSS0.Px4.p2.4.m4.1.1b.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1"><eq id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.3.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.3"></eq><ci id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.2.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.2">𝑁</ci><apply id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4"><divide id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.1.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.1"></divide><ci id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.2.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.2">𝑃</ci><apply id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.3.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.3.1.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.3.2.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.3.2">𝑃</ci><ci id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.3.3a.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.3.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.3.3.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.3.3">expert</mtext></ci></apply></apply></apply><apply id="S2.SS0.SSS0.Px4.p2.4.m4.1.1c.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1"><eq id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.5.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.5"></eq><share href="#S2.SS0.SSS0.Px4.p2.4.m4.1.1.4.cmml" id="S2.SS0.SSS0.Px4.p2.4.m4.1.1d.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1"></share><apply id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6"><divide id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.1.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.1"></divide><apply id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.2.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.2"><ci id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.2.1.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.2.1">⋅</ci><ci id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.2.2.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.2.2">𝑃</ci><ci id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.2.3.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.2.3">𝐺</ci></apply><apply id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.3.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.3.1.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.3.2.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.3.2">𝑃</ci><ci id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.3.3a.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.3.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.3.3.cmml" xref="S2.SS0.SSS0.Px4.p2.4.m4.1.1.6.3.3">active</mtext></ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p2.4.m4.1c">N=P/P_{\text{expert}}=P\cdot G/P_{\text{active}}</annotation></semantics></math>, this implies that we should decrease the size of each expert, <math id="S2.SS0.SSS0.Px4.p2.5.m5.1" class="ltx_Math" alttext="P_{\text{expert}}" display="inline"><semantics id="S2.SS0.SSS0.Px4.p2.5.m5.1a"><msub id="S2.SS0.SSS0.Px4.p2.5.m5.1.1" xref="S2.SS0.SSS0.Px4.p2.5.m5.1.1.cmml"><mi id="S2.SS0.SSS0.Px4.p2.5.m5.1.1.2" xref="S2.SS0.SSS0.Px4.p2.5.m5.1.1.2.cmml">P</mi><mtext id="S2.SS0.SSS0.Px4.p2.5.m5.1.1.3" xref="S2.SS0.SSS0.Px4.p2.5.m5.1.1.3a.cmml">expert</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p2.5.m5.1b"><apply id="S2.SS0.SSS0.Px4.p2.5.m5.1.1.cmml" xref="S2.SS0.SSS0.Px4.p2.5.m5.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p2.5.m5.1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p2.5.m5.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p2.5.m5.1.1.2.cmml" xref="S2.SS0.SSS0.Px4.p2.5.m5.1.1.2">𝑃</ci><ci id="S2.SS0.SSS0.Px4.p2.5.m5.1.1.3a.cmml" xref="S2.SS0.SSS0.Px4.p2.5.m5.1.1.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p2.5.m5.1.1.3.cmml" xref="S2.SS0.SSS0.Px4.p2.5.m5.1.1.3">expert</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p2.5.m5.1c">P_{\text{expert}}</annotation></semantics></math>, and increase the number of experts <math id="S2.SS0.SSS0.Px4.p2.6.m6.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S2.SS0.SSS0.Px4.p2.6.m6.1a"><mi id="S2.SS0.SSS0.Px4.p2.6.m6.1.1" xref="S2.SS0.SSS0.Px4.p2.6.m6.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p2.6.m6.1b"><ci id="S2.SS0.SSS0.Px4.p2.6.m6.1.1.cmml" xref="S2.SS0.SSS0.Px4.p2.6.m6.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p2.6.m6.1c">N</annotation></semantics></math>. Hence we need a large number of small experts.</p>
</div>
<div id="S2.SS0.SSS0.Px4.p3" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px4.p3.5" class="ltx_p">In general, for experts that are MLPs with a single hidden layer. <math id="S2.SS0.SSS0.Px4.p3.1.m1.1" class="ltx_Math" alttext="P_{\text{expert}}=(2d_{\text{model}}+1)d_{\text{expert}}" display="inline"><semantics id="S2.SS0.SSS0.Px4.p3.1.m1.1a"><mrow id="S2.SS0.SSS0.Px4.p3.1.m1.1.1" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.cmml"><msub id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.3" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.3.2" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.3.2.cmml">P</mi><mtext id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.3.3" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.3.3a.cmml">expert</mtext></msub><mo id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.2" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.2.cmml">=</mo><mrow id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.cmml"><mrow id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.2" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.cmml"><mrow id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.cmml"><mn id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.2" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.1" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.1.cmml">​</mo><msub id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.3" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.3.cmml"><mi id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.3.2" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.3.2.cmml">d</mi><mtext id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.3.3" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.3.3a.cmml">model</mtext></msub></mrow><mo id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.1" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.1.cmml">+</mo><mn id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.3" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.3" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.2" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.2.cmml">​</mo><msub id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.3" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.3.2" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.3.2.cmml">d</mi><mtext id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.3.3" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.3.3a.cmml">expert</mtext></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p3.1.m1.1b"><apply id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1"><eq id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.2.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.2"></eq><apply id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.3.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.3.2">𝑃</ci><ci id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.3.3a.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.3.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.3.3">expert</mtext></ci></apply><apply id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1"><times id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.2"></times><apply id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1"><plus id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.1"></plus><apply id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2"><times id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.1.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.1"></times><cn type="integer" id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.2.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.2">2</cn><apply id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.3.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.3.1.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.3.2.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.3.2">𝑑</ci><ci id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.3.3a.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.3.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.3.3.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.2.3.3">model</mtext></ci></apply></apply><cn type="integer" id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.1.1.1.3">1</cn></apply><apply id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.3.2">𝑑</ci><ci id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.3.3a.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.3.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px4.p3.1.m1.1.1.1.3.3">expert</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p3.1.m1.1c">P_{\text{expert}}=(2d_{\text{model}}+1)d_{\text{expert}}</annotation></semantics></math> and <math id="S2.SS0.SSS0.Px4.p3.2.m2.1" class="ltx_Math" alttext="P_{\text{active}}=(2d_{\text{model}}+1)d_{\text{active}}" display="inline"><semantics id="S2.SS0.SSS0.Px4.p3.2.m2.1a"><mrow id="S2.SS0.SSS0.Px4.p3.2.m2.1.1" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.cmml"><msub id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.3" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.3.2" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.3.2.cmml">P</mi><mtext id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.3.3" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.3.3a.cmml">active</mtext></msub><mo id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.2" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.2.cmml">=</mo><mrow id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.cmml"><mrow id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.2" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.cmml"><mrow id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.cmml"><mn id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.2" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.1" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.1.cmml">​</mo><msub id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.3" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.3.cmml"><mi id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.3.2" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.3.2.cmml">d</mi><mtext id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.3.3" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.3.3a.cmml">model</mtext></msub></mrow><mo id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.1" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.1.cmml">+</mo><mn id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.3" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.3" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.2" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.2.cmml">​</mo><msub id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.3" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.3.2" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.3.2.cmml">d</mi><mtext id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.3.3" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.3.3a.cmml">active</mtext></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p3.2.m2.1b"><apply id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1"><eq id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.2.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.2"></eq><apply id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.3.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.3.2">𝑃</ci><ci id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.3.3a.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.3.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.3.3">active</mtext></ci></apply><apply id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1"><times id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.2"></times><apply id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1"><plus id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.1"></plus><apply id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2"><times id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.1.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.1"></times><cn type="integer" id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.2.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.2">2</cn><apply id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.3.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.3.1.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.3.2.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.3.2">𝑑</ci><ci id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.3.3a.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.3.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.3.3.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.2.3.3">model</mtext></ci></apply></apply><cn type="integer" id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.1.1.1.3">1</cn></apply><apply id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.3.2">𝑑</ci><ci id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.3.3a.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.3.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px4.p3.2.m2.1.1.1.3.3">active</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p3.2.m2.1c">P_{\text{active}}=(2d_{\text{model}}+1)d_{\text{active}}</annotation></semantics></math>, where <math id="S2.SS0.SSS0.Px4.p3.3.m3.1" class="ltx_Math" alttext="d_{\text{model}}" display="inline"><semantics id="S2.SS0.SSS0.Px4.p3.3.m3.1a"><msub id="S2.SS0.SSS0.Px4.p3.3.m3.1.1" xref="S2.SS0.SSS0.Px4.p3.3.m3.1.1.cmml"><mi id="S2.SS0.SSS0.Px4.p3.3.m3.1.1.2" xref="S2.SS0.SSS0.Px4.p3.3.m3.1.1.2.cmml">d</mi><mtext id="S2.SS0.SSS0.Px4.p3.3.m3.1.1.3" xref="S2.SS0.SSS0.Px4.p3.3.m3.1.1.3a.cmml">model</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p3.3.m3.1b"><apply id="S2.SS0.SSS0.Px4.p3.3.m3.1.1.cmml" xref="S2.SS0.SSS0.Px4.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p3.3.m3.1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p3.3.m3.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p3.3.m3.1.1.2.cmml" xref="S2.SS0.SSS0.Px4.p3.3.m3.1.1.2">𝑑</ci><ci id="S2.SS0.SSS0.Px4.p3.3.m3.1.1.3a.cmml" xref="S2.SS0.SSS0.Px4.p3.3.m3.1.1.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p3.3.m3.1.1.3.cmml" xref="S2.SS0.SSS0.Px4.p3.3.m3.1.1.3">model</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p3.3.m3.1c">d_{\text{model}}</annotation></semantics></math>, <math id="S2.SS0.SSS0.Px4.p3.4.m4.1" class="ltx_Math" alttext="d_{\text{expert}}" display="inline"><semantics id="S2.SS0.SSS0.Px4.p3.4.m4.1a"><msub id="S2.SS0.SSS0.Px4.p3.4.m4.1.1" xref="S2.SS0.SSS0.Px4.p3.4.m4.1.1.cmml"><mi id="S2.SS0.SSS0.Px4.p3.4.m4.1.1.2" xref="S2.SS0.SSS0.Px4.p3.4.m4.1.1.2.cmml">d</mi><mtext id="S2.SS0.SSS0.Px4.p3.4.m4.1.1.3" xref="S2.SS0.SSS0.Px4.p3.4.m4.1.1.3a.cmml">expert</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p3.4.m4.1b"><apply id="S2.SS0.SSS0.Px4.p3.4.m4.1.1.cmml" xref="S2.SS0.SSS0.Px4.p3.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p3.4.m4.1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p3.4.m4.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p3.4.m4.1.1.2.cmml" xref="S2.SS0.SSS0.Px4.p3.4.m4.1.1.2">𝑑</ci><ci id="S2.SS0.SSS0.Px4.p3.4.m4.1.1.3a.cmml" xref="S2.SS0.SSS0.Px4.p3.4.m4.1.1.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p3.4.m4.1.1.3.cmml" xref="S2.SS0.SSS0.Px4.p3.4.m4.1.1.3">expert</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p3.4.m4.1c">d_{\text{expert}}</annotation></semantics></math> and <math id="S2.SS0.SSS0.Px4.p3.5.m5.1" class="ltx_Math" alttext="d_{\text{active}}" display="inline"><semantics id="S2.SS0.SSS0.Px4.p3.5.m5.1a"><msub id="S2.SS0.SSS0.Px4.p3.5.m5.1.1" xref="S2.SS0.SSS0.Px4.p3.5.m5.1.1.cmml"><mi id="S2.SS0.SSS0.Px4.p3.5.m5.1.1.2" xref="S2.SS0.SSS0.Px4.p3.5.m5.1.1.2.cmml">d</mi><mtext id="S2.SS0.SSS0.Px4.p3.5.m5.1.1.3" xref="S2.SS0.SSS0.Px4.p3.5.m5.1.1.3a.cmml">active</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p3.5.m5.1b"><apply id="S2.SS0.SSS0.Px4.p3.5.m5.1.1.cmml" xref="S2.SS0.SSS0.Px4.p3.5.m5.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p3.5.m5.1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p3.5.m5.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p3.5.m5.1.1.2.cmml" xref="S2.SS0.SSS0.Px4.p3.5.m5.1.1.2">𝑑</ci><ci id="S2.SS0.SSS0.Px4.p3.5.m5.1.1.3a.cmml" xref="S2.SS0.SSS0.Px4.p3.5.m5.1.1.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p3.5.m5.1.1.3.cmml" xref="S2.SS0.SSS0.Px4.p3.5.m5.1.1.3">active</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p3.5.m5.1c">d_{\text{active}}</annotation></semantics></math> are the hidden dimension of the transformer, the number of hidden neurons used in one expert and the total number of hidden neurons activated per token, respectively.</p>
</div>
<div id="S2.SS0.SSS0.Px4.p4" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px4.p4.3" class="ltx_p">In the case of PEER, we use the smallest expert size possible by setting <math id="S2.SS0.SSS0.Px4.p4.1.m1.1" class="ltx_Math" alttext="d_{\text{expert}}=1" display="inline"><semantics id="S2.SS0.SSS0.Px4.p4.1.m1.1a"><mrow id="S2.SS0.SSS0.Px4.p4.1.m1.1.1" xref="S2.SS0.SSS0.Px4.p4.1.m1.1.1.cmml"><msub id="S2.SS0.SSS0.Px4.p4.1.m1.1.1.2" xref="S2.SS0.SSS0.Px4.p4.1.m1.1.1.2.cmml"><mi id="S2.SS0.SSS0.Px4.p4.1.m1.1.1.2.2" xref="S2.SS0.SSS0.Px4.p4.1.m1.1.1.2.2.cmml">d</mi><mtext id="S2.SS0.SSS0.Px4.p4.1.m1.1.1.2.3" xref="S2.SS0.SSS0.Px4.p4.1.m1.1.1.2.3a.cmml">expert</mtext></msub><mo id="S2.SS0.SSS0.Px4.p4.1.m1.1.1.1" xref="S2.SS0.SSS0.Px4.p4.1.m1.1.1.1.cmml">=</mo><mn id="S2.SS0.SSS0.Px4.p4.1.m1.1.1.3" xref="S2.SS0.SSS0.Px4.p4.1.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p4.1.m1.1b"><apply id="S2.SS0.SSS0.Px4.p4.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p4.1.m1.1.1"><eq id="S2.SS0.SSS0.Px4.p4.1.m1.1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p4.1.m1.1.1.1"></eq><apply id="S2.SS0.SSS0.Px4.p4.1.m1.1.1.2.cmml" xref="S2.SS0.SSS0.Px4.p4.1.m1.1.1.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p4.1.m1.1.1.2.1.cmml" xref="S2.SS0.SSS0.Px4.p4.1.m1.1.1.2">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p4.1.m1.1.1.2.2.cmml" xref="S2.SS0.SSS0.Px4.p4.1.m1.1.1.2.2">𝑑</ci><ci id="S2.SS0.SSS0.Px4.p4.1.m1.1.1.2.3a.cmml" xref="S2.SS0.SSS0.Px4.p4.1.m1.1.1.2.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p4.1.m1.1.1.2.3.cmml" xref="S2.SS0.SSS0.Px4.p4.1.m1.1.1.2.3">expert</mtext></ci></apply><cn type="integer" id="S2.SS0.SSS0.Px4.p4.1.m1.1.1.3.cmml" xref="S2.SS0.SSS0.Px4.p4.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p4.1.m1.1c">d_{\text{expert}}=1</annotation></semantics></math>, and the number of activated neurons is the number of retrieval heads multiplied by the number of experts retrieved per head: <math id="S2.SS0.SSS0.Px4.p4.2.m2.1" class="ltx_Math" alttext="d_{\text{active}}=hk" display="inline"><semantics id="S2.SS0.SSS0.Px4.p4.2.m2.1a"><mrow id="S2.SS0.SSS0.Px4.p4.2.m2.1.1" xref="S2.SS0.SSS0.Px4.p4.2.m2.1.1.cmml"><msub id="S2.SS0.SSS0.Px4.p4.2.m2.1.1.2" xref="S2.SS0.SSS0.Px4.p4.2.m2.1.1.2.cmml"><mi id="S2.SS0.SSS0.Px4.p4.2.m2.1.1.2.2" xref="S2.SS0.SSS0.Px4.p4.2.m2.1.1.2.2.cmml">d</mi><mtext id="S2.SS0.SSS0.Px4.p4.2.m2.1.1.2.3" xref="S2.SS0.SSS0.Px4.p4.2.m2.1.1.2.3a.cmml">active</mtext></msub><mo id="S2.SS0.SSS0.Px4.p4.2.m2.1.1.1" xref="S2.SS0.SSS0.Px4.p4.2.m2.1.1.1.cmml">=</mo><mrow id="S2.SS0.SSS0.Px4.p4.2.m2.1.1.3" xref="S2.SS0.SSS0.Px4.p4.2.m2.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px4.p4.2.m2.1.1.3.2" xref="S2.SS0.SSS0.Px4.p4.2.m2.1.1.3.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px4.p4.2.m2.1.1.3.1" xref="S2.SS0.SSS0.Px4.p4.2.m2.1.1.3.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px4.p4.2.m2.1.1.3.3" xref="S2.SS0.SSS0.Px4.p4.2.m2.1.1.3.3.cmml">k</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p4.2.m2.1b"><apply id="S2.SS0.SSS0.Px4.p4.2.m2.1.1.cmml" xref="S2.SS0.SSS0.Px4.p4.2.m2.1.1"><eq id="S2.SS0.SSS0.Px4.p4.2.m2.1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p4.2.m2.1.1.1"></eq><apply id="S2.SS0.SSS0.Px4.p4.2.m2.1.1.2.cmml" xref="S2.SS0.SSS0.Px4.p4.2.m2.1.1.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p4.2.m2.1.1.2.1.cmml" xref="S2.SS0.SSS0.Px4.p4.2.m2.1.1.2">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p4.2.m2.1.1.2.2.cmml" xref="S2.SS0.SSS0.Px4.p4.2.m2.1.1.2.2">𝑑</ci><ci id="S2.SS0.SSS0.Px4.p4.2.m2.1.1.2.3a.cmml" xref="S2.SS0.SSS0.Px4.p4.2.m2.1.1.2.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p4.2.m2.1.1.2.3.cmml" xref="S2.SS0.SSS0.Px4.p4.2.m2.1.1.2.3">active</mtext></ci></apply><apply id="S2.SS0.SSS0.Px4.p4.2.m2.1.1.3.cmml" xref="S2.SS0.SSS0.Px4.p4.2.m2.1.1.3"><times id="S2.SS0.SSS0.Px4.p4.2.m2.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px4.p4.2.m2.1.1.3.1"></times><ci id="S2.SS0.SSS0.Px4.p4.2.m2.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px4.p4.2.m2.1.1.3.2">ℎ</ci><ci id="S2.SS0.SSS0.Px4.p4.2.m2.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px4.p4.2.m2.1.1.3.3">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p4.2.m2.1c">d_{\text{active}}=hk</annotation></semantics></math>. Consequently, the granularity of PEER is always <math id="S2.SS0.SSS0.Px4.p4.3.m3.1" class="ltx_Math" alttext="G=P_{\text{active}}/P_{\text{expert}}=d_{\text{active}}/d_{\text{expert}}=hk" display="inline"><semantics id="S2.SS0.SSS0.Px4.p4.3.m3.1a"><mrow id="S2.SS0.SSS0.Px4.p4.3.m3.1.1" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.cmml"><mi id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.2" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.2.cmml">G</mi><mo id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.3" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.3.cmml">=</mo><mrow id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.cmml"><msub id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.2" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.2.cmml"><mi id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.2.2" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.2.2.cmml">P</mi><mtext id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.2.3" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.2.3a.cmml">active</mtext></msub><mo id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.1" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.1.cmml">/</mo><msub id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.3" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.3.cmml"><mi id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.3.2" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.3.2.cmml">P</mi><mtext id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.3.3" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.3.3a.cmml">expert</mtext></msub></mrow><mo id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.5" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.5.cmml">=</mo><mrow id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.cmml"><msub id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.2" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.2.cmml"><mi id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.2.2" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.2.2.cmml">d</mi><mtext id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.2.3" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.2.3a.cmml">active</mtext></msub><mo id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.1" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.1.cmml">/</mo><msub id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.3" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.3.cmml"><mi id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.3.2" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.3.2.cmml">d</mi><mtext id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.3.3" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.3.3a.cmml">expert</mtext></msub></mrow><mo id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.7" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.7.cmml">=</mo><mrow id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.8" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.8.cmml"><mi id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.8.2" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.8.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.8.1" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.8.1.cmml">​</mo><mi id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.8.3" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.8.3.cmml">k</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p4.3.m3.1b"><apply id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1"><and id="S2.SS0.SSS0.Px4.p4.3.m3.1.1a.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1"></and><apply id="S2.SS0.SSS0.Px4.p4.3.m3.1.1b.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1"><eq id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.3.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.3"></eq><ci id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.2.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.2">𝐺</ci><apply id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4"><divide id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.1.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.1"></divide><apply id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.2.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.2.1.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.2">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.2.2.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.2.2">𝑃</ci><ci id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.2.3a.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.2.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.2.3.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.2.3">active</mtext></ci></apply><apply id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.3.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.3.1.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.3.2.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.3.2">𝑃</ci><ci id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.3.3a.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.3.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.3.3.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.3.3">expert</mtext></ci></apply></apply></apply><apply id="S2.SS0.SSS0.Px4.p4.3.m3.1.1c.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1"><eq id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.5.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.5"></eq><share href="#S2.SS0.SSS0.Px4.p4.3.m3.1.1.4.cmml" id="S2.SS0.SSS0.Px4.p4.3.m3.1.1d.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1"></share><apply id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6"><divide id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.1.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.1"></divide><apply id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.2.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.2.1.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.2">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.2.2.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.2.2">𝑑</ci><ci id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.2.3a.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.2.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.2.3.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.2.3">active</mtext></ci></apply><apply id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.3.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.3.1.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.3.2.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.3.2">𝑑</ci><ci id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.3.3a.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.3.3"><mtext mathsize="70%" id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.3.3.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.3.3">expert</mtext></ci></apply></apply></apply><apply id="S2.SS0.SSS0.Px4.p4.3.m3.1.1e.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1"><eq id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.7.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.7"></eq><share href="#S2.SS0.SSS0.Px4.p4.3.m3.1.1.6.cmml" id="S2.SS0.SSS0.Px4.p4.3.m3.1.1f.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1"></share><apply id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.8.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.8"><times id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.8.1.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.8.1"></times><ci id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.8.2.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.8.2">ℎ</ci><ci id="S2.SS0.SSS0.Px4.p4.3.m3.1.1.8.3.cmml" xref="S2.SS0.SSS0.Px4.p4.3.m3.1.1.8.3">𝑘</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p4.3.m3.1c">G=P_{\text{active}}/P_{\text{expert}}=d_{\text{active}}/d_{\text{expert}}=hk</annotation></semantics></math>.</p>
</div>
<figure id="LST1" class="ltx_float ltx_lstlisting">
<div id="LST1.1" class="ltx_listing ltx_lst_language_Python ltx_lst_numbers_left ltx_lstlisting ltx_listing" style="background-color:#F2F2EB;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,ZGVmIHBlZXJfZm9yd2FyZChzZWxmLCB4KToKICAgICMgRW1iZWRkaW5nIGxheWVycyBzdG9yaW5nIHRoZSBkb3duL3VwIHByb2plY3Rpb24gd2VpZ2h0cyBvZiBhbGwgZXhwZXJ0cwogICAgc2VsZi53X2Rvd25fZW1iZWQgPSBubi5FbWJlZChudW1fZW1iZWRkaW5ncz1zZWxmLm5fZXhwZXJ0cywgZmVhdHVyZXM9c2VsZi5kX21vZGVsKQogICAgc2VsZi53X3VwX2VtYmVkID0gbm4uRW1iZWQobnVtX2VtYmVkZGluZ3M9c2VsZi5uX2V4cGVydHMsIGZlYXR1cmVzPXNlbGYuZF9tb2RlbCkKCiAgICAjIFJldHJpZXZlIHRoZSB3ZWlnaHRzIG9mIHRoZSB0b3AgbWF0Y2hpbmcgZXhwZXJ0cyB1c2luZyBwcm9kdWN0IGtleXMKICAgICMgaW5kaWNlcyBhbmQgc2NvcmVzIGhhdmUgdGhlIHNoYXBlICdidGhrJywgd2hlcmUgaCBpcyB0aGUgbnVtYmVyIG9mIGhlYWRzCiAgICBpbmRpY2VzLCBzY29yZXMgPSBzZWxmLmdldF9pbmRpY2VzKHNlbGYucXVlcnlfcHJvaih4KSwgc2VsZi5zdWJfa2V5cywgdG9wX2s9c2VsZi5rKQogICAgd19kb3duID0gc2VsZi53X2Rvd25fZW1iZWQoaW5kaWNlcykKICAgIHdfdXAgPSBzZWxmLndfdXBfZW1iZWQoaW5kaWNlcykKCiAgICAjIENvbXB1dGUgd2VpZ2h0ZWQgYXZlcmFnZSBvZiBleHBlcnQgb3V0cHV0cwogICAgeCA9IGpucC5laW5zdW0oJ2J0ZCwgYnRoa2QtPmJ0aGsnLCB4LCB3X2Rvd24pCiAgICB4ID0gc2VsZi5hY3RpdmF0aW9uKHgpCiAgICB4ID0geCAqIG5uLnNvZnRtYXgoc2NvcmVzKQogICAgeCA9IGpucC5laW5zdW0oJ2J0aGssIGJ0aGtkLT5idGQnLCB4LCB3X3VwKQogICAgcmV0dXJuIHg=" download="">⬇</a></div>
<div id="lstnumberx1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">1</span><span id="lstnumberx1.1" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="font-size:80%;color:#FF00FF;">def</span><span id="lstnumberx1.2" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx1.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">peer_forward</span><span id="lstnumberx1.4" class="ltx_text ltx_font_typewriter" style="font-size:80%;">(</span><span id="lstnumberx1.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">self</span><span id="lstnumberx1.6" class="ltx_text ltx_font_typewriter" style="font-size:80%;">,</span><span id="lstnumberx1.7" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx1.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">x</span><span id="lstnumberx1.9" class="ltx_text ltx_font_typewriter" style="font-size:80%;">):</span>
</div>
<div id="lstnumberx2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">2</span><span id="lstnumberx2.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;">    </span><span id="lstnumberx2.2" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="font-size:80%;color:#009900;">#<span id="lstnumberx2.2.1" class="ltx_text ltx_lst_space"> </span>Embedding<span id="lstnumberx2.2.2" class="ltx_text ltx_lst_space"> </span>layers<span id="lstnumberx2.2.3" class="ltx_text ltx_lst_space"> </span>storing<span id="lstnumberx2.2.4" class="ltx_text ltx_lst_space"> </span>the<span id="lstnumberx2.2.5" class="ltx_text ltx_lst_space"> </span>down/up<span id="lstnumberx2.2.6" class="ltx_text ltx_lst_space"> </span>projection<span id="lstnumberx2.2.7" class="ltx_text ltx_lst_space"> </span>weights<span id="lstnumberx2.2.8" class="ltx_text ltx_lst_space"> </span>of<span id="lstnumberx2.2.9" class="ltx_text ltx_lst_space"> </span>all<span id="lstnumberx2.2.10" class="ltx_text ltx_lst_space"> </span>experts</span>
</div>
<div id="lstnumberx3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">3</span><span id="lstnumberx3.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;">    </span><span id="lstnumberx3.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">self</span><span id="lstnumberx3.3" class="ltx_text ltx_font_typewriter" style="font-size:80%;">.</span><span id="lstnumberx3.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">w_down_embed</span><span id="lstnumberx3.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx3.6" class="ltx_text ltx_font_typewriter" style="font-size:80%;">=</span><span id="lstnumberx3.7" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx3.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">nn</span><span id="lstnumberx3.9" class="ltx_text ltx_font_typewriter" style="font-size:80%;">.</span><span id="lstnumberx3.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">Embed</span><span id="lstnumberx3.11" class="ltx_text ltx_font_typewriter" style="font-size:80%;">(</span><span id="lstnumberx3.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">num_embeddings</span><span id="lstnumberx3.13" class="ltx_text ltx_font_typewriter" style="font-size:80%;">=</span><span id="lstnumberx3.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">self</span><span id="lstnumberx3.15" class="ltx_text ltx_font_typewriter" style="font-size:80%;">.</span><span id="lstnumberx3.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">n_experts</span><span id="lstnumberx3.17" class="ltx_text ltx_font_typewriter" style="font-size:80%;">,</span><span id="lstnumberx3.18" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx3.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">features</span><span id="lstnumberx3.20" class="ltx_text ltx_font_typewriter" style="font-size:80%;">=</span><span id="lstnumberx3.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">self</span><span id="lstnumberx3.22" class="ltx_text ltx_font_typewriter" style="font-size:80%;">.</span><span id="lstnumberx3.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">d_model</span><span id="lstnumberx3.24" class="ltx_text ltx_font_typewriter" style="font-size:80%;">)</span>
</div>
<div id="lstnumberx4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">4</span><span id="lstnumberx4.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;">    </span><span id="lstnumberx4.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">self</span><span id="lstnumberx4.3" class="ltx_text ltx_font_typewriter" style="font-size:80%;">.</span><span id="lstnumberx4.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">w_up_embed</span><span id="lstnumberx4.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx4.6" class="ltx_text ltx_font_typewriter" style="font-size:80%;">=</span><span id="lstnumberx4.7" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx4.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">nn</span><span id="lstnumberx4.9" class="ltx_text ltx_font_typewriter" style="font-size:80%;">.</span><span id="lstnumberx4.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">Embed</span><span id="lstnumberx4.11" class="ltx_text ltx_font_typewriter" style="font-size:80%;">(</span><span id="lstnumberx4.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">num_embeddings</span><span id="lstnumberx4.13" class="ltx_text ltx_font_typewriter" style="font-size:80%;">=</span><span id="lstnumberx4.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">self</span><span id="lstnumberx4.15" class="ltx_text ltx_font_typewriter" style="font-size:80%;">.</span><span id="lstnumberx4.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">n_experts</span><span id="lstnumberx4.17" class="ltx_text ltx_font_typewriter" style="font-size:80%;">,</span><span id="lstnumberx4.18" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx4.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">features</span><span id="lstnumberx4.20" class="ltx_text ltx_font_typewriter" style="font-size:80%;">=</span><span id="lstnumberx4.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">self</span><span id="lstnumberx4.22" class="ltx_text ltx_font_typewriter" style="font-size:80%;">.</span><span id="lstnumberx4.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">d_model</span><span id="lstnumberx4.24" class="ltx_text ltx_font_typewriter" style="font-size:80%;">)</span>
</div>
<div id="lstnumberx5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">5</span>
</div>
<div id="lstnumberx6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">6</span><span id="lstnumberx6.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;">    </span><span id="lstnumberx6.2" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="font-size:80%;color:#009900;">#<span id="lstnumberx6.2.1" class="ltx_text ltx_lst_space"> </span>Retrieve<span id="lstnumberx6.2.2" class="ltx_text ltx_lst_space"> </span>the<span id="lstnumberx6.2.3" class="ltx_text ltx_lst_space"> </span>weights<span id="lstnumberx6.2.4" class="ltx_text ltx_lst_space"> </span>of<span id="lstnumberx6.2.5" class="ltx_text ltx_lst_space"> </span>the<span id="lstnumberx6.2.6" class="ltx_text ltx_lst_space"> </span>top<span id="lstnumberx6.2.7" class="ltx_text ltx_lst_space"> </span>matching<span id="lstnumberx6.2.8" class="ltx_text ltx_lst_space"> </span>experts<span id="lstnumberx6.2.9" class="ltx_text ltx_lst_space"> </span>using<span id="lstnumberx6.2.10" class="ltx_text ltx_lst_space"> </span>product<span id="lstnumberx6.2.11" class="ltx_text ltx_lst_space"> </span>keys</span>
</div>
<div id="lstnumberx7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">7</span><span id="lstnumberx7.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;">    </span><span id="lstnumberx7.2" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="font-size:80%;color:#009900;">#<span id="lstnumberx7.2.1" class="ltx_text ltx_lst_space"> </span>indices<span id="lstnumberx7.2.2" class="ltx_text ltx_lst_space"> </span>and<span id="lstnumberx7.2.3" class="ltx_text ltx_lst_space"> </span>scores<span id="lstnumberx7.2.4" class="ltx_text ltx_lst_space"> </span>have<span id="lstnumberx7.2.5" class="ltx_text ltx_lst_space"> </span>the<span id="lstnumberx7.2.6" class="ltx_text ltx_lst_space"> </span>shape<span id="lstnumberx7.2.7" class="ltx_text ltx_lst_space"> </span>’bthk’,<span id="lstnumberx7.2.8" class="ltx_text ltx_lst_space"> </span>where<span id="lstnumberx7.2.9" class="ltx_text ltx_lst_space"> </span>h<span id="lstnumberx7.2.10" class="ltx_text ltx_lst_space"> </span>is<span id="lstnumberx7.2.11" class="ltx_text ltx_lst_space"> </span>the<span id="lstnumberx7.2.12" class="ltx_text ltx_lst_space"> </span>number<span id="lstnumberx7.2.13" class="ltx_text ltx_lst_space"> </span>of<span id="lstnumberx7.2.14" class="ltx_text ltx_lst_space"> </span>heads</span>
</div>
<div id="lstnumberx8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">8</span><span id="lstnumberx8.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;">    </span><span id="lstnumberx8.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">indices</span><span id="lstnumberx8.3" class="ltx_text ltx_font_typewriter" style="font-size:80%;">,</span><span id="lstnumberx8.4" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx8.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">scores</span><span id="lstnumberx8.6" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx8.7" class="ltx_text ltx_font_typewriter" style="font-size:80%;">=</span><span id="lstnumberx8.8" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx8.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">self</span><span id="lstnumberx8.10" class="ltx_text ltx_font_typewriter" style="font-size:80%;">.</span><span id="lstnumberx8.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">get_indices</span><span id="lstnumberx8.12" class="ltx_text ltx_font_typewriter" style="font-size:80%;">(</span><span id="lstnumberx8.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">self</span><span id="lstnumberx8.14" class="ltx_text ltx_font_typewriter" style="font-size:80%;">.</span><span id="lstnumberx8.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">query_proj</span><span id="lstnumberx8.16" class="ltx_text ltx_font_typewriter" style="font-size:80%;">(</span><span id="lstnumberx8.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">x</span><span id="lstnumberx8.18" class="ltx_text ltx_font_typewriter" style="font-size:80%;">),</span><span id="lstnumberx8.19" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx8.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">self</span><span id="lstnumberx8.21" class="ltx_text ltx_font_typewriter" style="font-size:80%;">.</span><span id="lstnumberx8.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">sub_keys</span><span id="lstnumberx8.23" class="ltx_text ltx_font_typewriter" style="font-size:80%;">,</span><span id="lstnumberx8.24" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx8.25" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">top_k</span><span id="lstnumberx8.26" class="ltx_text ltx_font_typewriter" style="font-size:80%;">=</span><span id="lstnumberx8.27" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">self</span><span id="lstnumberx8.28" class="ltx_text ltx_font_typewriter" style="font-size:80%;">.</span><span id="lstnumberx8.29" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">k</span><span id="lstnumberx8.30" class="ltx_text ltx_font_typewriter" style="font-size:80%;">)</span>
</div>
<div id="lstnumberx9" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">9</span><span id="lstnumberx9.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;">    </span><span id="lstnumberx9.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">w_down</span><span id="lstnumberx9.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx9.4" class="ltx_text ltx_font_typewriter" style="font-size:80%;">=</span><span id="lstnumberx9.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx9.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">self</span><span id="lstnumberx9.7" class="ltx_text ltx_font_typewriter" style="font-size:80%;">.</span><span id="lstnumberx9.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">w_down_embed</span><span id="lstnumberx9.9" class="ltx_text ltx_font_typewriter" style="font-size:80%;">(</span><span id="lstnumberx9.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">indices</span><span id="lstnumberx9.11" class="ltx_text ltx_font_typewriter" style="font-size:80%;">)</span>
</div>
<div id="lstnumberx10" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">10</span><span id="lstnumberx10.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;">    </span><span id="lstnumberx10.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">w_up</span><span id="lstnumberx10.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx10.4" class="ltx_text ltx_font_typewriter" style="font-size:80%;">=</span><span id="lstnumberx10.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx10.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">self</span><span id="lstnumberx10.7" class="ltx_text ltx_font_typewriter" style="font-size:80%;">.</span><span id="lstnumberx10.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">w_up_embed</span><span id="lstnumberx10.9" class="ltx_text ltx_font_typewriter" style="font-size:80%;">(</span><span id="lstnumberx10.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">indices</span><span id="lstnumberx10.11" class="ltx_text ltx_font_typewriter" style="font-size:80%;">)</span>
</div>
<div id="lstnumberx11" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">11</span>
</div>
<div id="lstnumberx12" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">12</span><span id="lstnumberx12.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;">    </span><span id="lstnumberx12.2" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="font-size:80%;color:#009900;">#<span id="lstnumberx12.2.1" class="ltx_text ltx_lst_space"> </span>Compute<span id="lstnumberx12.2.2" class="ltx_text ltx_lst_space"> </span>weighted<span id="lstnumberx12.2.3" class="ltx_text ltx_lst_space"> </span>average<span id="lstnumberx12.2.4" class="ltx_text ltx_lst_space"> </span>of<span id="lstnumberx12.2.5" class="ltx_text ltx_lst_space"> </span>expert<span id="lstnumberx12.2.6" class="ltx_text ltx_lst_space"> </span>outputs</span>
</div>
<div id="lstnumberx13" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">13</span><span id="lstnumberx13.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;">    </span><span id="lstnumberx13.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">x</span><span id="lstnumberx13.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx13.4" class="ltx_text ltx_font_typewriter" style="font-size:80%;">=</span><span id="lstnumberx13.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx13.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">jnp</span><span id="lstnumberx13.7" class="ltx_text ltx_font_typewriter" style="font-size:80%;">.</span><span id="lstnumberx13.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">einsum</span><span id="lstnumberx13.9" class="ltx_text ltx_font_typewriter" style="font-size:80%;">(</span><span id="lstnumberx13.10" class="ltx_text ltx_lst_string ltx_font_typewriter" style="font-size:80%;color:#9400D1;">’btd,<span id="lstnumberx13.10.1" class="ltx_text ltx_lst_space"> </span>bthkd-&gt;bthk’</span><span id="lstnumberx13.11" class="ltx_text ltx_font_typewriter" style="font-size:80%;">,</span><span id="lstnumberx13.12" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx13.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">x</span><span id="lstnumberx13.14" class="ltx_text ltx_font_typewriter" style="font-size:80%;">,</span><span id="lstnumberx13.15" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx13.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">w_down</span><span id="lstnumberx13.17" class="ltx_text ltx_font_typewriter" style="font-size:80%;">)</span>
</div>
<div id="lstnumberx14" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">14</span><span id="lstnumberx14.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;">    </span><span id="lstnumberx14.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">x</span><span id="lstnumberx14.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx14.4" class="ltx_text ltx_font_typewriter" style="font-size:80%;">=</span><span id="lstnumberx14.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx14.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">self</span><span id="lstnumberx14.7" class="ltx_text ltx_font_typewriter" style="font-size:80%;">.</span><span id="lstnumberx14.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">activation</span><span id="lstnumberx14.9" class="ltx_text ltx_font_typewriter" style="font-size:80%;">(</span><span id="lstnumberx14.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">x</span><span id="lstnumberx14.11" class="ltx_text ltx_font_typewriter" style="font-size:80%;">)</span>
</div>
<div id="lstnumberx15" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">15</span><span id="lstnumberx15.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;">    </span><span id="lstnumberx15.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">x</span><span id="lstnumberx15.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx15.4" class="ltx_text ltx_font_typewriter" style="font-size:80%;">=</span><span id="lstnumberx15.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx15.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">x</span><span id="lstnumberx15.7" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx15.8" class="ltx_text ltx_font_typewriter" style="font-size:80%;">*</span><span id="lstnumberx15.9" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx15.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">nn</span><span id="lstnumberx15.11" class="ltx_text ltx_font_typewriter" style="font-size:80%;">.</span><span id="lstnumberx15.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">softmax</span><span id="lstnumberx15.13" class="ltx_text ltx_font_typewriter" style="font-size:80%;">(</span><span id="lstnumberx15.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">scores</span><span id="lstnumberx15.15" class="ltx_text ltx_font_typewriter" style="font-size:80%;">)</span>
</div>
<div id="lstnumberx16" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">16</span><span id="lstnumberx16.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;">    </span><span id="lstnumberx16.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">x</span><span id="lstnumberx16.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx16.4" class="ltx_text ltx_font_typewriter" style="font-size:80%;">=</span><span id="lstnumberx16.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx16.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">jnp</span><span id="lstnumberx16.7" class="ltx_text ltx_font_typewriter" style="font-size:80%;">.</span><span id="lstnumberx16.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">einsum</span><span id="lstnumberx16.9" class="ltx_text ltx_font_typewriter" style="font-size:80%;">(</span><span id="lstnumberx16.10" class="ltx_text ltx_lst_string ltx_font_typewriter" style="font-size:80%;color:#9400D1;">’bthk,<span id="lstnumberx16.10.1" class="ltx_text ltx_lst_space"> </span>bthkd-&gt;btd’</span><span id="lstnumberx16.11" class="ltx_text ltx_font_typewriter" style="font-size:80%;">,</span><span id="lstnumberx16.12" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx16.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">x</span><span id="lstnumberx16.14" class="ltx_text ltx_font_typewriter" style="font-size:80%;">,</span><span id="lstnumberx16.15" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx16.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">w_up</span><span id="lstnumberx16.17" class="ltx_text ltx_font_typewriter" style="font-size:80%;">)</span>
</div>
<div id="lstnumberx17" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">17</span><span id="lstnumberx17.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;">    </span><span id="lstnumberx17.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="font-size:80%;color:#FF00FF;">return</span><span id="lstnumberx17.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:80%;"> </span><span id="lstnumberx17.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:80%;">x</span>
</div>
</div>
<figcaption class="ltx_caption" style="background-color:#F2F2EB;"><span class="ltx_tag ltx_tag_float">Algorithm 1: </span>Pseudo code implementation of a PEER layer forward pass. An example implementation of the get_indices and query_proj functions in Pytorch can be found in <cite class="ltx_cite ltx_citemacro_citet">Lample et al. (<a href="#bib.bib28" title="" class="ltx_ref">2021</a>)</cite></figcaption>
</figure>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Experiments</h2>

<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Pretraining isoFLOP Analysis</h3>

<div id="S3.SS1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS1.p1.2" class="ltx_p">We compare PEER with various baselines using isoFLOP analysis <cite class="ltx_cite ltx_citemacro_citep">(Borgeaud et al., <a href="#bib.bib6" title="" class="ltx_ref">2022b</a>)</cite>. We chose a fixed FLOP budget (<math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="6e18" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><mrow id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml"><mn id="S3.SS1.p1.1.m1.1.1.2" xref="S3.SS1.p1.1.m1.1.1.2.cmml">6</mn><mo lspace="0em" rspace="0em" id="S3.SS1.p1.1.m1.1.1.1" xref="S3.SS1.p1.1.m1.1.1.1.cmml">​</mo><mi id="S3.SS1.p1.1.m1.1.1.3" xref="S3.SS1.p1.1.m1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.1.m1.1.1.1a" xref="S3.SS1.p1.1.m1.1.1.1.cmml">​</mo><mn id="S3.SS1.p1.1.m1.1.1.4" xref="S3.SS1.p1.1.m1.1.1.4.cmml">18</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><apply id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1"><times id="S3.SS1.p1.1.m1.1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1.1"></times><cn type="integer" id="S3.SS1.p1.1.m1.1.1.2.cmml" xref="S3.SS1.p1.1.m1.1.1.2">6</cn><ci id="S3.SS1.p1.1.m1.1.1.3.cmml" xref="S3.SS1.p1.1.m1.1.1.3">𝑒</ci><cn type="integer" id="S3.SS1.p1.1.m1.1.1.4.cmml" xref="S3.SS1.p1.1.m1.1.1.4">18</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">6e18</annotation></semantics></math> and <math id="S3.SS1.p1.2.m2.1" class="ltx_Math" alttext="2e19" display="inline"><semantics id="S3.SS1.p1.2.m2.1a"><mrow id="S3.SS1.p1.2.m2.1.1" xref="S3.SS1.p1.2.m2.1.1.cmml"><mn id="S3.SS1.p1.2.m2.1.1.2" xref="S3.SS1.p1.2.m2.1.1.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S3.SS1.p1.2.m2.1.1.1" xref="S3.SS1.p1.2.m2.1.1.1.cmml">​</mo><mi id="S3.SS1.p1.2.m2.1.1.3" xref="S3.SS1.p1.2.m2.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.2.m2.1.1.1a" xref="S3.SS1.p1.2.m2.1.1.1.cmml">​</mo><mn id="S3.SS1.p1.2.m2.1.1.4" xref="S3.SS1.p1.2.m2.1.1.4.cmml">19</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.2.m2.1b"><apply id="S3.SS1.p1.2.m2.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1"><times id="S3.SS1.p1.2.m2.1.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1.1"></times><cn type="integer" id="S3.SS1.p1.2.m2.1.1.2.cmml" xref="S3.SS1.p1.2.m2.1.1.2">2</cn><ci id="S3.SS1.p1.2.m2.1.1.3.cmml" xref="S3.SS1.p1.2.m2.1.1.3">𝑒</ci><cn type="integer" id="S3.SS1.p1.2.m2.1.1.4.cmml" xref="S3.SS1.p1.2.m2.1.1.4">19</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.2.m2.1c">2e19</annotation></semantics></math>) and jointly varied the model size and the number of training tokens from the C4 dataset <cite class="ltx_cite ltx_citemacro_citep">(Raffel et al., <a href="#bib.bib34" title="" class="ltx_ref">2020</a>)</cite> to obtain isoFLOP curves. Each point on an isoFLOP curve has the same computational cost, and we plot them in terms of their model size and final validation perplexity on C4.</p>
</div>
<div id="S3.SS1.p2" class="ltx_para ltx_noindent">
<p id="S3.SS1.p2.1" class="ltx_p">For the dense baselines, we varied their size by changing the number of layers, attention heads and model dimensions. For MoE, PKM and PEER methods, we took each of the dense models considered and replaced the FFW layer in the middle block (e.g. in a 12 block transformer, we replace the FFN in block 6) by a layer of MoE, PKM and PEER, respectively.</p>
</div>
<div id="S3.SS1.p3" class="ltx_para ltx_noindent">
<p id="S3.SS1.p3.1" class="ltx_p">In MoE, we used the expert-choice <cite class="ltx_cite ltx_citemacro_citep">(Zhou et al., <a href="#bib.bib41" title="" class="ltx_ref">2022</a>)</cite> routing algorithm, which effectively addresses the expert load imbalance issue and generally outperforms token-choice MoEs (see Section <a href="#S4" title="4 Related Works ‣ Mixture of A Million Experts" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> for a review and comparison of these approaches). Each expert has the same size as the original MLPs in the corresponding dense model, and we use <math id="S3.SS1.p3.1.m1.1" class="ltx_Math" alttext="128" display="inline"><semantics id="S3.SS1.p3.1.m1.1a"><mn id="S3.SS1.p3.1.m1.1.1" xref="S3.SS1.p3.1.m1.1.1.cmml">128</mn><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.1.m1.1b"><cn type="integer" id="S3.SS1.p3.1.m1.1.1.cmml" xref="S3.SS1.p3.1.m1.1.1">128</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.1.m1.1c">128</annotation></semantics></math> experts to cover the same range of model sizes as our PEER models. This type of MoE represents standard coarse-grained MoE approaches, which consist of a small number of large experts.</p>
</div>
<div id="S3.SS1.p4" class="ltx_para ltx_noindent">
<p id="S3.SS1.p4.3" class="ltx_p">In PKM, we used <math id="S3.SS1.p4.1.m1.1" class="ltx_Math" alttext="1024^{2}" display="inline"><semantics id="S3.SS1.p4.1.m1.1a"><msup id="S3.SS1.p4.1.m1.1.1" xref="S3.SS1.p4.1.m1.1.1.cmml"><mn id="S3.SS1.p4.1.m1.1.1.2" xref="S3.SS1.p4.1.m1.1.1.2.cmml">1024</mn><mn id="S3.SS1.p4.1.m1.1.1.3" xref="S3.SS1.p4.1.m1.1.1.3.cmml">2</mn></msup><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.1.m1.1b"><apply id="S3.SS1.p4.1.m1.1.1.cmml" xref="S3.SS1.p4.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p4.1.m1.1.1.1.cmml" xref="S3.SS1.p4.1.m1.1.1">superscript</csymbol><cn type="integer" id="S3.SS1.p4.1.m1.1.1.2.cmml" xref="S3.SS1.p4.1.m1.1.1.2">1024</cn><cn type="integer" id="S3.SS1.p4.1.m1.1.1.3.cmml" xref="S3.SS1.p4.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.1.m1.1c">1024^{2}</annotation></semantics></math> memories with <math id="S3.SS1.p4.2.m2.1" class="ltx_Math" alttext="h=8" display="inline"><semantics id="S3.SS1.p4.2.m2.1a"><mrow id="S3.SS1.p4.2.m2.1.1" xref="S3.SS1.p4.2.m2.1.1.cmml"><mi id="S3.SS1.p4.2.m2.1.1.2" xref="S3.SS1.p4.2.m2.1.1.2.cmml">h</mi><mo id="S3.SS1.p4.2.m2.1.1.1" xref="S3.SS1.p4.2.m2.1.1.1.cmml">=</mo><mn id="S3.SS1.p4.2.m2.1.1.3" xref="S3.SS1.p4.2.m2.1.1.3.cmml">8</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.2.m2.1b"><apply id="S3.SS1.p4.2.m2.1.1.cmml" xref="S3.SS1.p4.2.m2.1.1"><eq id="S3.SS1.p4.2.m2.1.1.1.cmml" xref="S3.SS1.p4.2.m2.1.1.1"></eq><ci id="S3.SS1.p4.2.m2.1.1.2.cmml" xref="S3.SS1.p4.2.m2.1.1.2">ℎ</ci><cn type="integer" id="S3.SS1.p4.2.m2.1.1.3.cmml" xref="S3.SS1.p4.2.m2.1.1.3">8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.2.m2.1c">h=8</annotation></semantics></math> heads and top <math id="S3.SS1.p4.3.m3.1" class="ltx_Math" alttext="k=32" display="inline"><semantics id="S3.SS1.p4.3.m3.1a"><mrow id="S3.SS1.p4.3.m3.1.1" xref="S3.SS1.p4.3.m3.1.1.cmml"><mi id="S3.SS1.p4.3.m3.1.1.2" xref="S3.SS1.p4.3.m3.1.1.2.cmml">k</mi><mo id="S3.SS1.p4.3.m3.1.1.1" xref="S3.SS1.p4.3.m3.1.1.1.cmml">=</mo><mn id="S3.SS1.p4.3.m3.1.1.3" xref="S3.SS1.p4.3.m3.1.1.3.cmml">32</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.3.m3.1b"><apply id="S3.SS1.p4.3.m3.1.1.cmml" xref="S3.SS1.p4.3.m3.1.1"><eq id="S3.SS1.p4.3.m3.1.1.1.cmml" xref="S3.SS1.p4.3.m3.1.1.1"></eq><ci id="S3.SS1.p4.3.m3.1.1.2.cmml" xref="S3.SS1.p4.3.m3.1.1.2">𝑘</ci><cn type="integer" id="S3.SS1.p4.3.m3.1.1.3.cmml" xref="S3.SS1.p4.3.m3.1.1.3">32</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.3.m3.1c">k=32</annotation></semantics></math> memories were selected per head. We also applied query batch normalization, as recommended in the original PKM paper <cite class="ltx_cite ltx_citemacro_citep">(Lample et al., <a href="#bib.bib27" title="" class="ltx_ref">2019</a>)</cite>, to enhance memory usage.</p>
</div>
<div id="S3.SS1.p5" class="ltx_para ltx_noindent">
<p id="S3.SS1.p5.3" class="ltx_p">In PEER, we used <math id="S3.SS1.p5.1.m1.1" class="ltx_Math" alttext="1024^{2}" display="inline"><semantics id="S3.SS1.p5.1.m1.1a"><msup id="S3.SS1.p5.1.m1.1.1" xref="S3.SS1.p5.1.m1.1.1.cmml"><mn id="S3.SS1.p5.1.m1.1.1.2" xref="S3.SS1.p5.1.m1.1.1.2.cmml">1024</mn><mn id="S3.SS1.p5.1.m1.1.1.3" xref="S3.SS1.p5.1.m1.1.1.3.cmml">2</mn></msup><annotation-xml encoding="MathML-Content" id="S3.SS1.p5.1.m1.1b"><apply id="S3.SS1.p5.1.m1.1.1.cmml" xref="S3.SS1.p5.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p5.1.m1.1.1.1.cmml" xref="S3.SS1.p5.1.m1.1.1">superscript</csymbol><cn type="integer" id="S3.SS1.p5.1.m1.1.1.2.cmml" xref="S3.SS1.p5.1.m1.1.1.2">1024</cn><cn type="integer" id="S3.SS1.p5.1.m1.1.1.3.cmml" xref="S3.SS1.p5.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p5.1.m1.1c">1024^{2}</annotation></semantics></math> experts with <math id="S3.SS1.p5.2.m2.1" class="ltx_Math" alttext="h=8" display="inline"><semantics id="S3.SS1.p5.2.m2.1a"><mrow id="S3.SS1.p5.2.m2.1.1" xref="S3.SS1.p5.2.m2.1.1.cmml"><mi id="S3.SS1.p5.2.m2.1.1.2" xref="S3.SS1.p5.2.m2.1.1.2.cmml">h</mi><mo id="S3.SS1.p5.2.m2.1.1.1" xref="S3.SS1.p5.2.m2.1.1.1.cmml">=</mo><mn id="S3.SS1.p5.2.m2.1.1.3" xref="S3.SS1.p5.2.m2.1.1.3.cmml">8</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p5.2.m2.1b"><apply id="S3.SS1.p5.2.m2.1.1.cmml" xref="S3.SS1.p5.2.m2.1.1"><eq id="S3.SS1.p5.2.m2.1.1.1.cmml" xref="S3.SS1.p5.2.m2.1.1.1"></eq><ci id="S3.SS1.p5.2.m2.1.1.2.cmml" xref="S3.SS1.p5.2.m2.1.1.2">ℎ</ci><cn type="integer" id="S3.SS1.p5.2.m2.1.1.3.cmml" xref="S3.SS1.p5.2.m2.1.1.3">8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p5.2.m2.1c">h=8</annotation></semantics></math> heads and top <math id="S3.SS1.p5.3.m3.1" class="ltx_Math" alttext="k=16" display="inline"><semantics id="S3.SS1.p5.3.m3.1a"><mrow id="S3.SS1.p5.3.m3.1.1" xref="S3.SS1.p5.3.m3.1.1.cmml"><mi id="S3.SS1.p5.3.m3.1.1.2" xref="S3.SS1.p5.3.m3.1.1.2.cmml">k</mi><mo id="S3.SS1.p5.3.m3.1.1.1" xref="S3.SS1.p5.3.m3.1.1.1.cmml">=</mo><mn id="S3.SS1.p5.3.m3.1.1.3" xref="S3.SS1.p5.3.m3.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p5.3.m3.1b"><apply id="S3.SS1.p5.3.m3.1.1.cmml" xref="S3.SS1.p5.3.m3.1.1"><eq id="S3.SS1.p5.3.m3.1.1.1.cmml" xref="S3.SS1.p5.3.m3.1.1.1"></eq><ci id="S3.SS1.p5.3.m3.1.1.2.cmml" xref="S3.SS1.p5.3.m3.1.1.2">𝑘</ci><cn type="integer" id="S3.SS1.p5.3.m3.1.1.3.cmml" xref="S3.SS1.p5.3.m3.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p5.3.m3.1c">k=16</annotation></semantics></math> experts per head. By default, we also enabled query BatchNorm to increase expert usage. Ablation studies in subsection <a href="#S3.SS3" title="3.3 Ablations ‣ 3 Experiments ‣ Mixture of A Million Experts" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a> investigate the effect of these hyperparameters. Unlike the expert-choice MoE baseline, PEER represents a fine-grained approach where a large number of small experts are employed.</p>
</div>
<div id="S3.SS1.p6" class="ltx_para ltx_noindent">
<p id="S3.SS1.p6.2" class="ltx_p">Across all model sizes and methods, we maintained a consistent batch size (128) and sequence length (2048). We calculated the number of training steps by dividing the total compute budget by the FLOPs per training step. Fig. <a href="#S0.F1" title="Figure 1 ‣ Mixture of A Million Experts" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> presents the isoFLOP profiles. Compared to the dense FFW baseline, the sparse alternatives shift the isoFLOP curves downward and to right because they introduce a larger number of total parameters <math id="S3.SS1.p6.1.m1.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S3.SS1.p6.1.m1.1a"><mi id="S3.SS1.p6.1.m1.1.1" xref="S3.SS1.p6.1.m1.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p6.1.m1.1b"><ci id="S3.SS1.p6.1.m1.1.1.cmml" xref="S3.SS1.p6.1.m1.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p6.1.m1.1c">P</annotation></semantics></math> but utilize a smaller or equal number of active parameters <math id="S3.SS1.p6.2.m2.1" class="ltx_Math" alttext="P_{\text{active}}" display="inline"><semantics id="S3.SS1.p6.2.m2.1a"><msub id="S3.SS1.p6.2.m2.1.1" xref="S3.SS1.p6.2.m2.1.1.cmml"><mi id="S3.SS1.p6.2.m2.1.1.2" xref="S3.SS1.p6.2.m2.1.1.2.cmml">P</mi><mtext id="S3.SS1.p6.2.m2.1.1.3" xref="S3.SS1.p6.2.m2.1.1.3a.cmml">active</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p6.2.m2.1b"><apply id="S3.SS1.p6.2.m2.1.1.cmml" xref="S3.SS1.p6.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p6.2.m2.1.1.1.cmml" xref="S3.SS1.p6.2.m2.1.1">subscript</csymbol><ci id="S3.SS1.p6.2.m2.1.1.2.cmml" xref="S3.SS1.p6.2.m2.1.1.2">𝑃</ci><ci id="S3.SS1.p6.2.m2.1.1.3a.cmml" xref="S3.SS1.p6.2.m2.1.1.3"><mtext mathsize="70%" id="S3.SS1.p6.2.m2.1.1.3.cmml" xref="S3.SS1.p6.2.m2.1.1.3">active</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p6.2.m2.1c">P_{\text{active}}</annotation></semantics></math>. Given the same compute budget, a PEER model achieves the lowest compute-optimal perplexity.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Evaluation on Language Modeling Datasets</h3>

<div id="S3.SS2.p1" class="ltx_para ltx_noindent">
<p id="S3.SS2.p1.1" class="ltx_p">After determining the compute-optimal model for each method based on the isoFLOP curves, we evaluated the performance of these pretrained models on several popular language modeling datasets, including Curation Corpus <cite class="ltx_cite ltx_citemacro_citep">(Curation, <a href="#bib.bib10" title="" class="ltx_ref">2020</a>)</cite>, Lambada <cite class="ltx_cite ltx_citemacro_citep">(Paperno et al., <a href="#bib.bib33" title="" class="ltx_ref">2016</a>)</cite>, the Pile <cite class="ltx_cite ltx_citemacro_citep">(Gao et al., <a href="#bib.bib17" title="" class="ltx_ref">2020</a>)</cite>, Wikitext <cite class="ltx_cite ltx_citemacro_citep">(Merity et al., <a href="#bib.bib31" title="" class="ltx_ref">2016</a>)</cite> and the pretraining dataset C4. Table <a href="#S3.T1" title="Table 1 ‣ 3.2 Evaluation on Language Modeling Datasets ‣ 3 Experiments ‣ Mixture of A Million Experts" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> presents a summary of the evaluation results. We grouped the models based on their FLOP budgets used during training.</p>
</div>
<figure id="S3.T1" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 1: </span>Perplexities of the compute-optimal models of each method on language modeling datasets.
</figcaption>
<table id="S3.T1.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_bottom">
<thead class="ltx_thead">
<tr id="S3.T1.1.1.1" class="ltx_tr">
<th id="S3.T1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><span id="S3.T1.1.1.1.1.1" class="ltx_text" style="font-size:90%;">Method</span></th>
<th id="S3.T1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T1.1.1.1.2.1" class="ltx_text" style="font-size:90%;">Curation</span></th>
<th id="S3.T1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T1.1.1.1.3.1" class="ltx_text" style="font-size:90%;">Lambada</span></th>
<th id="S3.T1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T1.1.1.1.4.1" class="ltx_text" style="font-size:90%;">Pile</span></th>
<th id="S3.T1.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T1.1.1.1.5.1" class="ltx_text" style="font-size:90%;">Wikitext</span></th>
<th id="S3.T1.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T1.1.1.1.6.1" class="ltx_text" style="font-size:90%;">C4</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S3.T1.1.2.1" class="ltx_tr">
<th id="S3.T1.1.2.1.1" class="ltx_td ltx_th ltx_th_row"></th>
<th id="S3.T1.1.2.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S3.T1.1.2.1.2.1" class="ltx_text" style="font-size:90%;">Corpus</span></th>
<th id="S3.T1.1.2.1.3" class="ltx_td ltx_th ltx_th_column"></th>
<th id="S3.T1.1.2.1.4" class="ltx_td ltx_th ltx_th_column"></th>
<th id="S3.T1.1.2.1.5" class="ltx_td ltx_th ltx_th_column"></th>
<td id="S3.T1.1.2.1.6" class="ltx_td"></td>
</tr>
<tr id="S3.T1.1.3.2" class="ltx_tr">
<th id="S3.T1.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t"><span id="S3.T1.1.3.2.1.1" class="ltx_text" style="font-size:90%;">Dense (6e18)</span></th>
<td id="S3.T1.1.3.2.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T1.1.3.2.2.1" class="ltx_text" style="font-size:90%;">23.26</span></td>
<td id="S3.T1.1.3.2.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T1.1.3.2.3.1" class="ltx_text" style="font-size:90%;">21.95</span></td>
<td id="S3.T1.1.3.2.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T1.1.3.2.4.1" class="ltx_text" style="font-size:90%;">24.55</span></td>
<td id="S3.T1.1.3.2.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T1.1.3.2.5.1" class="ltx_text" style="font-size:90%;">29.14</span></td>
<td id="S3.T1.1.3.2.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T1.1.3.2.6.1" class="ltx_text" style="font-size:90%;">23.84</span></td>
</tr>
<tr id="S3.T1.1.4.3" class="ltx_tr">
<th id="S3.T1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S3.T1.1.4.3.1.1" class="ltx_text" style="font-size:90%;">MoE (6e18)</span></th>
<td id="S3.T1.1.4.3.2" class="ltx_td ltx_align_center"><span id="S3.T1.1.4.3.2.1" class="ltx_text" style="font-size:90%;">20.98</span></td>
<td id="S3.T1.1.4.3.3" class="ltx_td ltx_align_center"><span id="S3.T1.1.4.3.3.1" class="ltx_text" style="font-size:90%;">19.09</span></td>
<td id="S3.T1.1.4.3.4" class="ltx_td ltx_align_center"><span id="S3.T1.1.4.3.4.1" class="ltx_text" style="font-size:90%;">23.26</span></td>
<td id="S3.T1.1.4.3.5" class="ltx_td ltx_align_center"><span id="S3.T1.1.4.3.5.1" class="ltx_text" style="font-size:90%;">26.10</span></td>
<td id="S3.T1.1.4.3.6" class="ltx_td ltx_align_center"><span id="S3.T1.1.4.3.6.1" class="ltx_text" style="font-size:90%;">21.41</span></td>
</tr>
<tr id="S3.T1.1.5.4" class="ltx_tr">
<th id="S3.T1.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S3.T1.1.5.4.1.1" class="ltx_text" style="font-size:90%;">PKM (6e18)</span></th>
<td id="S3.T1.1.5.4.2" class="ltx_td ltx_align_center"><span id="S3.T1.1.5.4.2.1" class="ltx_text" style="font-size:90%;">21.80</span></td>
<td id="S3.T1.1.5.4.3" class="ltx_td ltx_align_center"><span id="S3.T1.1.5.4.3.1" class="ltx_text" style="font-size:90%;">19.39</span></td>
<td id="S3.T1.1.5.4.4" class="ltx_td ltx_align_center"><span id="S3.T1.1.5.4.4.1" class="ltx_text" style="font-size:90%;">20.49</span></td>
<td id="S3.T1.1.5.4.5" class="ltx_td ltx_align_center"><span id="S3.T1.1.5.4.5.1" class="ltx_text" style="font-size:90%;">27.09</span></td>
<td id="S3.T1.1.5.4.6" class="ltx_td ltx_align_center"><span id="S3.T1.1.5.4.6.1" class="ltx_text" style="font-size:90%;">21.92</span></td>
</tr>
<tr id="S3.T1.1.6.5" class="ltx_tr">
<th id="S3.T1.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S3.T1.1.6.5.1.1" class="ltx_text" style="font-size:90%;">PEER (6e18)</span></th>
<td id="S3.T1.1.6.5.2" class="ltx_td ltx_align_center"><span id="S3.T1.1.6.5.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">20.68</span></td>
<td id="S3.T1.1.6.5.3" class="ltx_td ltx_align_center"><span id="S3.T1.1.6.5.3.1" class="ltx_text ltx_font_bold" style="font-size:90%;">17.65</span></td>
<td id="S3.T1.1.6.5.4" class="ltx_td ltx_align_center"><span id="S3.T1.1.6.5.4.1" class="ltx_text ltx_font_bold" style="font-size:90%;">19.01</span></td>
<td id="S3.T1.1.6.5.5" class="ltx_td ltx_align_center"><span id="S3.T1.1.6.5.5.1" class="ltx_text ltx_font_bold" style="font-size:90%;">25.48</span></td>
<td id="S3.T1.1.6.5.6" class="ltx_td ltx_align_center"><span id="S3.T1.1.6.5.6.1" class="ltx_text ltx_font_bold" style="font-size:90%;">20.63</span></td>
</tr>
<tr id="S3.T1.1.7.6" class="ltx_tr">
<th id="S3.T1.1.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t"><span id="S3.T1.1.7.6.1.1" class="ltx_text" style="font-size:90%;">Dense (2e19)</span></th>
<td id="S3.T1.1.7.6.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T1.1.7.6.2.1" class="ltx_text" style="font-size:90%;">17.70</span></td>
<td id="S3.T1.1.7.6.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T1.1.7.6.3.1" class="ltx_text" style="font-size:90%;">12.28</span></td>
<td id="S3.T1.1.7.6.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T1.1.7.6.4.1" class="ltx_text" style="font-size:90%;">18.19</span></td>
<td id="S3.T1.1.7.6.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T1.1.7.6.5.1" class="ltx_text" style="font-size:90%;">21.21</span></td>
<td id="S3.T1.1.7.6.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T1.1.7.6.6.1" class="ltx_text" style="font-size:90%;">18.31</span></td>
</tr>
<tr id="S3.T1.1.8.7" class="ltx_tr">
<th id="S3.T1.1.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S3.T1.1.8.7.1.1" class="ltx_text" style="font-size:90%;">MoE (2e19)</span></th>
<td id="S3.T1.1.8.7.2" class="ltx_td ltx_align_center"><span id="S3.T1.1.8.7.2.1" class="ltx_text" style="font-size:90%;">16.88</span></td>
<td id="S3.T1.1.8.7.3" class="ltx_td ltx_align_center"><span id="S3.T1.1.8.7.3.1" class="ltx_text" style="font-size:90%;">12.97</span></td>
<td id="S3.T1.1.8.7.4" class="ltx_td ltx_align_center"><span id="S3.T1.1.8.7.4.1" class="ltx_text" style="font-size:90%;">17.41</span></td>
<td id="S3.T1.1.8.7.5" class="ltx_td ltx_align_center"><span id="S3.T1.1.8.7.5.1" class="ltx_text" style="font-size:90%;">20.28</span></td>
<td id="S3.T1.1.8.7.6" class="ltx_td ltx_align_center"><span id="S3.T1.1.8.7.6.1" class="ltx_text" style="font-size:90%;">17.12</span></td>
</tr>
<tr id="S3.T1.1.9.8" class="ltx_tr">
<th id="S3.T1.1.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S3.T1.1.9.8.1.1" class="ltx_text" style="font-size:90%;">PKM (2e19)</span></th>
<td id="S3.T1.1.9.8.2" class="ltx_td ltx_align_center"><span id="S3.T1.1.9.8.2.1" class="ltx_text" style="font-size:90%;">17.03</span></td>
<td id="S3.T1.1.9.8.3" class="ltx_td ltx_align_center"><span id="S3.T1.1.9.8.3.1" class="ltx_text" style="font-size:90%;">11.18</span></td>
<td id="S3.T1.1.9.8.4" class="ltx_td ltx_align_center"><span id="S3.T1.1.9.8.4.1" class="ltx_text" style="font-size:90%;">16.34</span></td>
<td id="S3.T1.1.9.8.5" class="ltx_td ltx_align_center"><span id="S3.T1.1.9.8.5.1" class="ltx_text" style="font-size:90%;">20.26</span></td>
<td id="S3.T1.1.9.8.6" class="ltx_td ltx_align_center"><span id="S3.T1.1.9.8.6.1" class="ltx_text" style="font-size:90%;">17.36</span></td>
</tr>
<tr id="S3.T1.1.10.9" class="ltx_tr">
<th id="S3.T1.1.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="S3.T1.1.10.9.1.1" class="ltx_text" style="font-size:90%;">PEER (2e19)</span></th>
<td id="S3.T1.1.10.9.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T1.1.10.9.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">16.34</span></td>
<td id="S3.T1.1.10.9.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T1.1.10.9.3.1" class="ltx_text ltx_font_bold" style="font-size:90%;">10.33</span></td>
<td id="S3.T1.1.10.9.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T1.1.10.9.4.1" class="ltx_text ltx_font_bold" style="font-size:90%;">14.99</span></td>
<td id="S3.T1.1.10.9.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T1.1.10.9.5.1" class="ltx_text ltx_font_bold" style="font-size:90%;">19.09</span></td>
<td id="S3.T1.1.10.9.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T1.1.10.9.6.1" class="ltx_text ltx_font_bold" style="font-size:90%;">16.45</span></td>
</tr>
</tbody>
</table>
</figure>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Ablations</h3>

<section id="S3.SS3.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Varying the Number of Total Experts</h4>

<div id="S3.SS3.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS0.Px1.p1.6" class="ltx_p">The models in the isoFLOP plot depicted in Fig. <a href="#S0.F1" title="Figure 1 ‣ Mixture of A Million Experts" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> all have over a million (<math id="S3.SS3.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="1024^{2}" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.1.m1.1a"><msup id="S3.SS3.SSS0.Px1.p1.1.m1.1.1" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.cmml"><mn id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.cmml">1024</mn><mn id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.cmml">2</mn></msup><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.1.m1.1b"><apply id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1">superscript</csymbol><cn type="integer" id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.2">1024</cn><cn type="integer" id="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.1.m1.1c">1024^{2}</annotation></semantics></math>) experts. Here we conduct an ablation study on the effect of the number of experts <math id="S3.SS3.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.2.m2.1a"><mi id="S3.SS3.SSS0.Px1.p1.2.m2.1.1" xref="S3.SS3.SSS0.Px1.p1.2.m2.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.2.m2.1b"><ci id="S3.SS3.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.2.m2.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.2.m2.1c">N</annotation></semantics></math>, which determines the total parameter count <math id="S3.SS3.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="P" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.3.m3.1a"><mi id="S3.SS3.SSS0.Px1.p1.3.m3.1.1" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1.cmml">P</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.3.m3.1b"><ci id="S3.SS3.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.3.m3.1.1">𝑃</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.3.m3.1c">P</annotation></semantics></math> in Eq. <a href="#S2.E9" title="In Why A Large Number of Small Experts? ‣ 2 Method ‣ Mixture of A Million Experts" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>. We selected the model at the isoFLOP-optimal position and vary the number of experts (<math id="S3.SS3.SSS0.Px1.p1.4.m4.4" class="ltx_Math" alttext="N=128^{2},256^{2},512^{2},1024^{2}" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.4.m4.4a"><mrow id="S3.SS3.SSS0.Px1.p1.4.m4.4.4" xref="S3.SS3.SSS0.Px1.p1.4.m4.4.4.cmml"><mi id="S3.SS3.SSS0.Px1.p1.4.m4.4.4.6" xref="S3.SS3.SSS0.Px1.p1.4.m4.4.4.6.cmml">N</mi><mo id="S3.SS3.SSS0.Px1.p1.4.m4.4.4.5" xref="S3.SS3.SSS0.Px1.p1.4.m4.4.4.5.cmml">=</mo><mrow id="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.4" xref="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.5.cmml"><msup id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.cmml"><mn id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.2" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.2.cmml">128</mn><mn id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.3" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.3.cmml">2</mn></msup><mo id="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.4.5" xref="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.5.cmml">,</mo><msup id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.cmml"><mn id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.2" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.2.cmml">256</mn><mn id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.3" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.3.cmml">2</mn></msup><mo id="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.4.6" xref="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.5.cmml">,</mo><msup id="S3.SS3.SSS0.Px1.p1.4.m4.3.3.3.3.3" xref="S3.SS3.SSS0.Px1.p1.4.m4.3.3.3.3.3.cmml"><mn id="S3.SS3.SSS0.Px1.p1.4.m4.3.3.3.3.3.2" xref="S3.SS3.SSS0.Px1.p1.4.m4.3.3.3.3.3.2.cmml">512</mn><mn id="S3.SS3.SSS0.Px1.p1.4.m4.3.3.3.3.3.3" xref="S3.SS3.SSS0.Px1.p1.4.m4.3.3.3.3.3.3.cmml">2</mn></msup><mo id="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.4.7" xref="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.5.cmml">,</mo><msup id="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.4.4" xref="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.4.4.cmml"><mn id="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.4.4.2" xref="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.4.4.2.cmml">1024</mn><mn id="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.4.4.3" xref="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.4.4.3.cmml">2</mn></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.4.m4.4b"><apply id="S3.SS3.SSS0.Px1.p1.4.m4.4.4.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.4.4"><eq id="S3.SS3.SSS0.Px1.p1.4.m4.4.4.5.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.4.4.5"></eq><ci id="S3.SS3.SSS0.Px1.p1.4.m4.4.4.6.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.4.4.6">𝑁</ci><list id="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.5.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.4"><apply id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1">superscript</csymbol><cn type="integer" id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.2">128</cn><cn type="integer" id="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.1.1.1.1.1.3">2</cn></apply><apply id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.1.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2">superscript</csymbol><cn type="integer" id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.2.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.2">256</cn><cn type="integer" id="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.3.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.2.2.2.2.2.3">2</cn></apply><apply id="S3.SS3.SSS0.Px1.p1.4.m4.3.3.3.3.3.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.3.3.3.3.3"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.4.m4.3.3.3.3.3.1.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.3.3.3.3.3">superscript</csymbol><cn type="integer" id="S3.SS3.SSS0.Px1.p1.4.m4.3.3.3.3.3.2.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.3.3.3.3.3.2">512</cn><cn type="integer" id="S3.SS3.SSS0.Px1.p1.4.m4.3.3.3.3.3.3.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.3.3.3.3.3.3">2</cn></apply><apply id="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.4.4.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.4.4"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.4.4.1.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.4.4">superscript</csymbol><cn type="integer" id="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.4.4.2.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.4.4.2">1024</cn><cn type="integer" id="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.4.4.3.cmml" xref="S3.SS3.SSS0.Px1.p1.4.m4.4.4.4.4.4.3">2</cn></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.4.m4.4c">N=128^{2},256^{2},512^{2},1024^{2}</annotation></semantics></math>) in the PEER layer while keeping the number of active experts constant (<math id="S3.SS3.SSS0.Px1.p1.5.m5.2" class="ltx_Math" alttext="h=8,k=16" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.5.m5.2a"><mrow id="S3.SS3.SSS0.Px1.p1.5.m5.2.2.2" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.2.3.cmml"><mrow id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.1.1" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1.1.1.cmml"><mi id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.1.1.2" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1.1.1.2.cmml">h</mi><mo id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.1.1.1" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1.1.1.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.1.1.3" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1.1.1.3.cmml">8</mn></mrow><mo id="S3.SS3.SSS0.Px1.p1.5.m5.2.2.2.3" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.2.3a.cmml">,</mo><mrow id="S3.SS3.SSS0.Px1.p1.5.m5.2.2.2.2" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.2.2.2.cmml"><mi id="S3.SS3.SSS0.Px1.p1.5.m5.2.2.2.2.2" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.2.2.2.2.cmml">k</mi><mo id="S3.SS3.SSS0.Px1.p1.5.m5.2.2.2.2.1" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.2.2.2.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px1.p1.5.m5.2.2.2.2.3" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.2.2.2.3.cmml">16</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.5.m5.2b"><apply id="S3.SS3.SSS0.Px1.p1.5.m5.2.2.3.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.2.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.5.m5.2.2.3a.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.2.2.3">formulae-sequence</csymbol><apply id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1.1.1"><eq id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1.1.1.1"></eq><ci id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1.1.1.2">ℎ</ci><cn type="integer" id="S3.SS3.SSS0.Px1.p1.5.m5.1.1.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.1.1.1.1.3">8</cn></apply><apply id="S3.SS3.SSS0.Px1.p1.5.m5.2.2.2.2.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.2.2.2"><eq id="S3.SS3.SSS0.Px1.p1.5.m5.2.2.2.2.1.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.2.2.2.1"></eq><ci id="S3.SS3.SSS0.Px1.p1.5.m5.2.2.2.2.2.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.2.2.2.2">𝑘</ci><cn type="integer" id="S3.SS3.SSS0.Px1.p1.5.m5.2.2.2.2.3.cmml" xref="S3.SS3.SSS0.Px1.p1.5.m5.2.2.2.2.3">16</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.5.m5.2c">h=8,k=16</annotation></semantics></math>). The results are shown in Fig. <a href="#S3.F3" title="Figure 3 ‣ Varying the Number of Total Experts ‣ 3.3 Ablations ‣ 3 Experiments ‣ Mixture of A Million Experts" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> (a). As can be seen, the isoFLOP curve interpolates between the PEER model with <math id="S3.SS3.SSS0.Px1.p1.6.m6.1" class="ltx_Math" alttext="1024^{2}" display="inline"><semantics id="S3.SS3.SSS0.Px1.p1.6.m6.1a"><msup id="S3.SS3.SSS0.Px1.p1.6.m6.1.1" xref="S3.SS3.SSS0.Px1.p1.6.m6.1.1.cmml"><mn id="S3.SS3.SSS0.Px1.p1.6.m6.1.1.2" xref="S3.SS3.SSS0.Px1.p1.6.m6.1.1.2.cmml">1024</mn><mn id="S3.SS3.SSS0.Px1.p1.6.m6.1.1.3" xref="S3.SS3.SSS0.Px1.p1.6.m6.1.1.3.cmml">2</mn></msup><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px1.p1.6.m6.1b"><apply id="S3.SS3.SSS0.Px1.p1.6.m6.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px1.p1.6.m6.1.1.1.cmml" xref="S3.SS3.SSS0.Px1.p1.6.m6.1.1">superscript</csymbol><cn type="integer" id="S3.SS3.SSS0.Px1.p1.6.m6.1.1.2.cmml" xref="S3.SS3.SSS0.Px1.p1.6.m6.1.1.2">1024</cn><cn type="integer" id="S3.SS3.SSS0.Px1.p1.6.m6.1.1.3.cmml" xref="S3.SS3.SSS0.Px1.p1.6.m6.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px1.p1.6.m6.1c">1024^{2}</annotation></semantics></math> experts and the corresponding dense backbone without replacing the FFW layer in the middle block by a PEER layer. This demonstrates that simply increasing the number experts can improve model performance.</p>
</div>
<figure id="S3.F3" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S3.F3.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.04153/assets/figs/ablation_N.png" id="S3.F3.sf1.g1" class="ltx_graphics ltx_img_square" width="311" height="313" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S3.F3.sf1.2.1.1" class="ltx_text" style="font-size:80%;">(a)</span> </span><span id="S3.F3.sf1.3.2" class="ltx_text" style="font-size:80%;">Varying Total Expert Num</span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S3.F3.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.04153/assets/figs/ablation_hk.png" id="S3.F3.sf2.g1" class="ltx_graphics ltx_img_square" width="324" height="311" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S3.F3.sf2.2.1.1" class="ltx_text" style="font-size:80%;">(b)</span> </span><span id="S3.F3.sf2.3.2" class="ltx_text" style="font-size:80%;">Varying Active Expert Num</span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>We conduct two ablation studies using the same PEER model configuration. In (a), we vary the total number of experts <math id="S3.F3.7.m1.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S3.F3.7.m1.1b"><mi id="S3.F3.7.m1.1.1" xref="S3.F3.7.m1.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S3.F3.7.m1.1c"><ci id="S3.F3.7.m1.1.1.cmml" xref="S3.F3.7.m1.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.7.m1.1d">N</annotation></semantics></math> while keeping the same number of active experts <math id="S3.F3.8.m2.1" class="ltx_Math" alttext="hk=128" display="inline"><semantics id="S3.F3.8.m2.1b"><mrow id="S3.F3.8.m2.1.1" xref="S3.F3.8.m2.1.1.cmml"><mrow id="S3.F3.8.m2.1.1.2" xref="S3.F3.8.m2.1.1.2.cmml"><mi id="S3.F3.8.m2.1.1.2.2" xref="S3.F3.8.m2.1.1.2.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S3.F3.8.m2.1.1.2.1" xref="S3.F3.8.m2.1.1.2.1.cmml">​</mo><mi id="S3.F3.8.m2.1.1.2.3" xref="S3.F3.8.m2.1.1.2.3.cmml">k</mi></mrow><mo id="S3.F3.8.m2.1.1.1" xref="S3.F3.8.m2.1.1.1.cmml">=</mo><mn id="S3.F3.8.m2.1.1.3" xref="S3.F3.8.m2.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.F3.8.m2.1c"><apply id="S3.F3.8.m2.1.1.cmml" xref="S3.F3.8.m2.1.1"><eq id="S3.F3.8.m2.1.1.1.cmml" xref="S3.F3.8.m2.1.1.1"></eq><apply id="S3.F3.8.m2.1.1.2.cmml" xref="S3.F3.8.m2.1.1.2"><times id="S3.F3.8.m2.1.1.2.1.cmml" xref="S3.F3.8.m2.1.1.2.1"></times><ci id="S3.F3.8.m2.1.1.2.2.cmml" xref="S3.F3.8.m2.1.1.2.2">ℎ</ci><ci id="S3.F3.8.m2.1.1.2.3.cmml" xref="S3.F3.8.m2.1.1.2.3">𝑘</ci></apply><cn type="integer" id="S3.F3.8.m2.1.1.3.cmml" xref="S3.F3.8.m2.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.8.m2.1d">hk=128</annotation></semantics></math>. In (b), we vary the number of active experts <math id="S3.F3.9.m3.1" class="ltx_Math" alttext="G=hk" display="inline"><semantics id="S3.F3.9.m3.1b"><mrow id="S3.F3.9.m3.1.1" xref="S3.F3.9.m3.1.1.cmml"><mi id="S3.F3.9.m3.1.1.2" xref="S3.F3.9.m3.1.1.2.cmml">G</mi><mo id="S3.F3.9.m3.1.1.1" xref="S3.F3.9.m3.1.1.1.cmml">=</mo><mrow id="S3.F3.9.m3.1.1.3" xref="S3.F3.9.m3.1.1.3.cmml"><mi id="S3.F3.9.m3.1.1.3.2" xref="S3.F3.9.m3.1.1.3.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S3.F3.9.m3.1.1.3.1" xref="S3.F3.9.m3.1.1.3.1.cmml">​</mo><mi id="S3.F3.9.m3.1.1.3.3" xref="S3.F3.9.m3.1.1.3.3.cmml">k</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.F3.9.m3.1c"><apply id="S3.F3.9.m3.1.1.cmml" xref="S3.F3.9.m3.1.1"><eq id="S3.F3.9.m3.1.1.1.cmml" xref="S3.F3.9.m3.1.1.1"></eq><ci id="S3.F3.9.m3.1.1.2.cmml" xref="S3.F3.9.m3.1.1.2">𝐺</ci><apply id="S3.F3.9.m3.1.1.3.cmml" xref="S3.F3.9.m3.1.1.3"><times id="S3.F3.9.m3.1.1.3.1.cmml" xref="S3.F3.9.m3.1.1.3.1"></times><ci id="S3.F3.9.m3.1.1.3.2.cmml" xref="S3.F3.9.m3.1.1.3.2">ℎ</ci><ci id="S3.F3.9.m3.1.1.3.3.cmml" xref="S3.F3.9.m3.1.1.3.3">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.9.m3.1d">G=hk</annotation></semantics></math> by jointly changing <math id="S3.F3.10.m4.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S3.F3.10.m4.1b"><mi id="S3.F3.10.m4.1.1" xref="S3.F3.10.m4.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S3.F3.10.m4.1c"><ci id="S3.F3.10.m4.1.1.cmml" xref="S3.F3.10.m4.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.10.m4.1d">h</annotation></semantics></math> and <math id="S3.F3.11.m5.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.F3.11.m5.1b"><mi id="S3.F3.11.m5.1.1" xref="S3.F3.11.m5.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.F3.11.m5.1c"><ci id="S3.F3.11.m5.1.1.cmml" xref="S3.F3.11.m5.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.11.m5.1d">k</annotation></semantics></math> while keeping the total number of experts at <math id="S3.F3.12.m6.1" class="ltx_Math" alttext="N=1024^{2}" display="inline"><semantics id="S3.F3.12.m6.1b"><mrow id="S3.F3.12.m6.1.1" xref="S3.F3.12.m6.1.1.cmml"><mi id="S3.F3.12.m6.1.1.2" xref="S3.F3.12.m6.1.1.2.cmml">N</mi><mo id="S3.F3.12.m6.1.1.1" xref="S3.F3.12.m6.1.1.1.cmml">=</mo><msup id="S3.F3.12.m6.1.1.3" xref="S3.F3.12.m6.1.1.3.cmml"><mn id="S3.F3.12.m6.1.1.3.2" xref="S3.F3.12.m6.1.1.3.2.cmml">1024</mn><mn id="S3.F3.12.m6.1.1.3.3" xref="S3.F3.12.m6.1.1.3.3.cmml">2</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.F3.12.m6.1c"><apply id="S3.F3.12.m6.1.1.cmml" xref="S3.F3.12.m6.1.1"><eq id="S3.F3.12.m6.1.1.1.cmml" xref="S3.F3.12.m6.1.1.1"></eq><ci id="S3.F3.12.m6.1.1.2.cmml" xref="S3.F3.12.m6.1.1.2">𝑁</ci><apply id="S3.F3.12.m6.1.1.3.cmml" xref="S3.F3.12.m6.1.1.3"><csymbol cd="ambiguous" id="S3.F3.12.m6.1.1.3.1.cmml" xref="S3.F3.12.m6.1.1.3">superscript</csymbol><cn type="integer" id="S3.F3.12.m6.1.1.3.2.cmml" xref="S3.F3.12.m6.1.1.3.2">1024</cn><cn type="integer" id="S3.F3.12.m6.1.1.3.3.cmml" xref="S3.F3.12.m6.1.1.3.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.12.m6.1d">N=1024^{2}</annotation></semantics></math>.</figcaption>
</figure>
</section>
<section id="S3.SS3.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Varying the Number of Active Experts</h4>

<div id="S3.SS3.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS0.Px2.p1.8" class="ltx_p">We also conducted an ablation study on the effect of the number of active experts <math id="S3.SS3.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="hk" display="inline"><semantics id="S3.SS3.SSS0.Px2.p1.1.m1.1a"><mrow id="S3.SS3.SSS0.Px2.p1.1.m1.1.1" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.2" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.1" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.1.cmml">​</mo><mi id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.3" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.3.cmml">k</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p1.1.m1.1b"><apply id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1"><times id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.1"></times><ci id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.2">ℎ</ci><ci id="S3.SS3.SSS0.Px2.p1.1.m1.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p1.1.m1.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p1.1.m1.1c">hk</annotation></semantics></math>, which equals the granularity <math id="S3.SS3.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="G" display="inline"><semantics id="S3.SS3.SSS0.Px2.p1.2.m2.1a"><mi id="S3.SS3.SSS0.Px2.p1.2.m2.1.1" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p1.2.m2.1b"><ci id="S3.SS3.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.2.m2.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p1.2.m2.1c">G</annotation></semantics></math> in Eq. <a href="#S2.E9" title="In Why A Large Number of Small Experts? ‣ 2 Method ‣ Mixture of A Million Experts" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>. We systematically varied the number of active experts (<math id="S3.SS3.SSS0.Px2.p1.3.m3.5" class="ltx_Math" alttext="hk=32,64,128,256,512" display="inline"><semantics id="S3.SS3.SSS0.Px2.p1.3.m3.5a"><mrow id="S3.SS3.SSS0.Px2.p1.3.m3.5.6" xref="S3.SS3.SSS0.Px2.p1.3.m3.5.6.cmml"><mrow id="S3.SS3.SSS0.Px2.p1.3.m3.5.6.2" xref="S3.SS3.SSS0.Px2.p1.3.m3.5.6.2.cmml"><mi id="S3.SS3.SSS0.Px2.p1.3.m3.5.6.2.2" xref="S3.SS3.SSS0.Px2.p1.3.m3.5.6.2.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S3.SS3.SSS0.Px2.p1.3.m3.5.6.2.1" xref="S3.SS3.SSS0.Px2.p1.3.m3.5.6.2.1.cmml">​</mo><mi id="S3.SS3.SSS0.Px2.p1.3.m3.5.6.2.3" xref="S3.SS3.SSS0.Px2.p1.3.m3.5.6.2.3.cmml">k</mi></mrow><mo id="S3.SS3.SSS0.Px2.p1.3.m3.5.6.1" xref="S3.SS3.SSS0.Px2.p1.3.m3.5.6.1.cmml">=</mo><mrow id="S3.SS3.SSS0.Px2.p1.3.m3.5.6.3.2" xref="S3.SS3.SSS0.Px2.p1.3.m3.5.6.3.1.cmml"><mn id="S3.SS3.SSS0.Px2.p1.3.m3.1.1" xref="S3.SS3.SSS0.Px2.p1.3.m3.1.1.cmml">32</mn><mo id="S3.SS3.SSS0.Px2.p1.3.m3.5.6.3.2.1" xref="S3.SS3.SSS0.Px2.p1.3.m3.5.6.3.1.cmml">,</mo><mn id="S3.SS3.SSS0.Px2.p1.3.m3.2.2" xref="S3.SS3.SSS0.Px2.p1.3.m3.2.2.cmml">64</mn><mo id="S3.SS3.SSS0.Px2.p1.3.m3.5.6.3.2.2" xref="S3.SS3.SSS0.Px2.p1.3.m3.5.6.3.1.cmml">,</mo><mn id="S3.SS3.SSS0.Px2.p1.3.m3.3.3" xref="S3.SS3.SSS0.Px2.p1.3.m3.3.3.cmml">128</mn><mo id="S3.SS3.SSS0.Px2.p1.3.m3.5.6.3.2.3" xref="S3.SS3.SSS0.Px2.p1.3.m3.5.6.3.1.cmml">,</mo><mn id="S3.SS3.SSS0.Px2.p1.3.m3.4.4" xref="S3.SS3.SSS0.Px2.p1.3.m3.4.4.cmml">256</mn><mo id="S3.SS3.SSS0.Px2.p1.3.m3.5.6.3.2.4" xref="S3.SS3.SSS0.Px2.p1.3.m3.5.6.3.1.cmml">,</mo><mn id="S3.SS3.SSS0.Px2.p1.3.m3.5.5" xref="S3.SS3.SSS0.Px2.p1.3.m3.5.5.cmml">512</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p1.3.m3.5b"><apply id="S3.SS3.SSS0.Px2.p1.3.m3.5.6.cmml" xref="S3.SS3.SSS0.Px2.p1.3.m3.5.6"><eq id="S3.SS3.SSS0.Px2.p1.3.m3.5.6.1.cmml" xref="S3.SS3.SSS0.Px2.p1.3.m3.5.6.1"></eq><apply id="S3.SS3.SSS0.Px2.p1.3.m3.5.6.2.cmml" xref="S3.SS3.SSS0.Px2.p1.3.m3.5.6.2"><times id="S3.SS3.SSS0.Px2.p1.3.m3.5.6.2.1.cmml" xref="S3.SS3.SSS0.Px2.p1.3.m3.5.6.2.1"></times><ci id="S3.SS3.SSS0.Px2.p1.3.m3.5.6.2.2.cmml" xref="S3.SS3.SSS0.Px2.p1.3.m3.5.6.2.2">ℎ</ci><ci id="S3.SS3.SSS0.Px2.p1.3.m3.5.6.2.3.cmml" xref="S3.SS3.SSS0.Px2.p1.3.m3.5.6.2.3">𝑘</ci></apply><list id="S3.SS3.SSS0.Px2.p1.3.m3.5.6.3.1.cmml" xref="S3.SS3.SSS0.Px2.p1.3.m3.5.6.3.2"><cn type="integer" id="S3.SS3.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.3.m3.1.1">32</cn><cn type="integer" id="S3.SS3.SSS0.Px2.p1.3.m3.2.2.cmml" xref="S3.SS3.SSS0.Px2.p1.3.m3.2.2">64</cn><cn type="integer" id="S3.SS3.SSS0.Px2.p1.3.m3.3.3.cmml" xref="S3.SS3.SSS0.Px2.p1.3.m3.3.3">128</cn><cn type="integer" id="S3.SS3.SSS0.Px2.p1.3.m3.4.4.cmml" xref="S3.SS3.SSS0.Px2.p1.3.m3.4.4">256</cn><cn type="integer" id="S3.SS3.SSS0.Px2.p1.3.m3.5.5.cmml" xref="S3.SS3.SSS0.Px2.p1.3.m3.5.5">512</cn></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p1.3.m3.5c">hk=32,64,128,256,512</annotation></semantics></math>) while keeping the number of total experts constant (<math id="S3.SS3.SSS0.Px2.p1.4.m4.1" class="ltx_Math" alttext="N=1024^{2}" display="inline"><semantics id="S3.SS3.SSS0.Px2.p1.4.m4.1a"><mrow id="S3.SS3.SSS0.Px2.p1.4.m4.1.1" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.2" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.2.cmml">N</mi><mo id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.1" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.1.cmml">=</mo><msup id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.3" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.3.cmml"><mn id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.3.2" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.3.2.cmml">1024</mn><mn id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.3.3" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.3.3.cmml">2</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p1.4.m4.1b"><apply id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1"><eq id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.1"></eq><ci id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.2">𝑁</ci><apply id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.3.1.cmml" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.3">superscript</csymbol><cn type="integer" id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.3.2.cmml" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.3.2">1024</cn><cn type="integer" id="S3.SS3.SSS0.Px2.p1.4.m4.1.1.3.3.cmml" xref="S3.SS3.SSS0.Px2.p1.4.m4.1.1.3.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p1.4.m4.1c">N=1024^{2}</annotation></semantics></math>). Furthermore, for a given <math id="S3.SS3.SSS0.Px2.p1.5.m5.1" class="ltx_Math" alttext="hk" display="inline"><semantics id="S3.SS3.SSS0.Px2.p1.5.m5.1a"><mrow id="S3.SS3.SSS0.Px2.p1.5.m5.1.1" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p1.5.m5.1.1.2" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S3.SS3.SSS0.Px2.p1.5.m5.1.1.1" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1.1.cmml">​</mo><mi id="S3.SS3.SSS0.Px2.p1.5.m5.1.1.3" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1.3.cmml">k</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p1.5.m5.1b"><apply id="S3.SS3.SSS0.Px2.p1.5.m5.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1"><times id="S3.SS3.SSS0.Px2.p1.5.m5.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1.1"></times><ci id="S3.SS3.SSS0.Px2.p1.5.m5.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1.2">ℎ</ci><ci id="S3.SS3.SSS0.Px2.p1.5.m5.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p1.5.m5.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p1.5.m5.1c">hk</annotation></semantics></math>, we jointly varied <math id="S3.SS3.SSS0.Px2.p1.6.m6.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S3.SS3.SSS0.Px2.p1.6.m6.1a"><mi id="S3.SS3.SSS0.Px2.p1.6.m6.1.1" xref="S3.SS3.SSS0.Px2.p1.6.m6.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p1.6.m6.1b"><ci id="S3.SS3.SSS0.Px2.p1.6.m6.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.6.m6.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p1.6.m6.1c">h</annotation></semantics></math> and <math id="S3.SS3.SSS0.Px2.p1.7.m7.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS3.SSS0.Px2.p1.7.m7.1a"><mi id="S3.SS3.SSS0.Px2.p1.7.m7.1.1" xref="S3.SS3.SSS0.Px2.p1.7.m7.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p1.7.m7.1b"><ci id="S3.SS3.SSS0.Px2.p1.7.m7.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.7.m7.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p1.7.m7.1c">k</annotation></semantics></math> to identify the optimal composition. The resulting isoFLOP curves, plotted over the number of heads (<math id="S3.SS3.SSS0.Px2.p1.8.m8.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S3.SS3.SSS0.Px2.p1.8.m8.1a"><mi id="S3.SS3.SSS0.Px2.p1.8.m8.1.1" xref="S3.SS3.SSS0.Px2.p1.8.m8.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p1.8.m8.1b"><ci id="S3.SS3.SSS0.Px2.p1.8.m8.1.1.cmml" xref="S3.SS3.SSS0.Px2.p1.8.m8.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p1.8.m8.1c">h</annotation></semantics></math>), are shown in Fig. <a href="#S3.F3" title="Figure 3 ‣ Varying the Number of Total Experts ‣ 3.3 Ablations ‣ 3 Experiments ‣ Mixture of A Million Experts" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> (b).</p>
</div>
<div id="S3.SS3.SSS0.Px2.p2" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS0.Px2.p2.4" class="ltx_p">The results indicate that, within the range of values considered, higher <math id="S3.SS3.SSS0.Px2.p2.1.m1.1" class="ltx_Math" alttext="hk" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.1.m1.1a"><mrow id="S3.SS3.SSS0.Px2.p2.1.m1.1.1" xref="S3.SS3.SSS0.Px2.p2.1.m1.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p2.1.m1.1.1.2" xref="S3.SS3.SSS0.Px2.p2.1.m1.1.1.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S3.SS3.SSS0.Px2.p2.1.m1.1.1.1" xref="S3.SS3.SSS0.Px2.p2.1.m1.1.1.1.cmml">​</mo><mi id="S3.SS3.SSS0.Px2.p2.1.m1.1.1.3" xref="S3.SS3.SSS0.Px2.p2.1.m1.1.1.3.cmml">k</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.1.m1.1b"><apply id="S3.SS3.SSS0.Px2.p2.1.m1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.1.m1.1.1"><times id="S3.SS3.SSS0.Px2.p2.1.m1.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.1.m1.1.1.1"></times><ci id="S3.SS3.SSS0.Px2.p2.1.m1.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p2.1.m1.1.1.2">ℎ</ci><ci id="S3.SS3.SSS0.Px2.p2.1.m1.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p2.1.m1.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.1.m1.1c">hk</annotation></semantics></math> generally leads to improved performance. Notably, the optimal <math id="S3.SS3.SSS0.Px2.p2.2.m2.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.2.m2.1a"><mi id="S3.SS3.SSS0.Px2.p2.2.m2.1.1" xref="S3.SS3.SSS0.Px2.p2.2.m2.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.2.m2.1b"><ci id="S3.SS3.SSS0.Px2.p2.2.m2.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.2.m2.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.2.m2.1c">h</annotation></semantics></math> increases as <math id="S3.SS3.SSS0.Px2.p2.3.m3.1" class="ltx_Math" alttext="hk" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.3.m3.1a"><mrow id="S3.SS3.SSS0.Px2.p2.3.m3.1.1" xref="S3.SS3.SSS0.Px2.p2.3.m3.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p2.3.m3.1.1.2" xref="S3.SS3.SSS0.Px2.p2.3.m3.1.1.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S3.SS3.SSS0.Px2.p2.3.m3.1.1.1" xref="S3.SS3.SSS0.Px2.p2.3.m3.1.1.1.cmml">​</mo><mi id="S3.SS3.SSS0.Px2.p2.3.m3.1.1.3" xref="S3.SS3.SSS0.Px2.p2.3.m3.1.1.3.cmml">k</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.3.m3.1b"><apply id="S3.SS3.SSS0.Px2.p2.3.m3.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.3.m3.1.1"><times id="S3.SS3.SSS0.Px2.p2.3.m3.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.3.m3.1.1.1"></times><ci id="S3.SS3.SSS0.Px2.p2.3.m3.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p2.3.m3.1.1.2">ℎ</ci><ci id="S3.SS3.SSS0.Px2.p2.3.m3.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p2.3.m3.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.3.m3.1c">hk</annotation></semantics></math> increases. However, the performance gradually saturates, and increasing the number of active experts also increases device memory consumption and may necessitate additional accelerator devices. Thus in practice, the appropriate <math id="S3.SS3.SSS0.Px2.p2.4.m4.1" class="ltx_Math" alttext="hk" display="inline"><semantics id="S3.SS3.SSS0.Px2.p2.4.m4.1a"><mrow id="S3.SS3.SSS0.Px2.p2.4.m4.1.1" xref="S3.SS3.SSS0.Px2.p2.4.m4.1.1.cmml"><mi id="S3.SS3.SSS0.Px2.p2.4.m4.1.1.2" xref="S3.SS3.SSS0.Px2.p2.4.m4.1.1.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S3.SS3.SSS0.Px2.p2.4.m4.1.1.1" xref="S3.SS3.SSS0.Px2.p2.4.m4.1.1.1.cmml">​</mo><mi id="S3.SS3.SSS0.Px2.p2.4.m4.1.1.3" xref="S3.SS3.SSS0.Px2.p2.4.m4.1.1.3.cmml">k</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px2.p2.4.m4.1b"><apply id="S3.SS3.SSS0.Px2.p2.4.m4.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.4.m4.1.1"><times id="S3.SS3.SSS0.Px2.p2.4.m4.1.1.1.cmml" xref="S3.SS3.SSS0.Px2.p2.4.m4.1.1.1"></times><ci id="S3.SS3.SSS0.Px2.p2.4.m4.1.1.2.cmml" xref="S3.SS3.SSS0.Px2.p2.4.m4.1.1.2">ℎ</ci><ci id="S3.SS3.SSS0.Px2.p2.4.m4.1.1.3.cmml" xref="S3.SS3.SSS0.Px2.p2.4.m4.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px2.p2.4.m4.1c">hk</annotation></semantics></math> values should be selected based on the trade-off between performance, device number and computational resource requirements.</p>
</div>
<figure id="S3.T2" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 2: </span><span id="S3.T2.28.1" class="ltx_text ltx_font_bold">KL and expert usage for different memory sizes, with and without query BN.</span> Similar to the findings in PKM, using query BN results in a more balanced usage of the experts.
</figcaption>
<table id="S3.T2.26" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_bottom">
<thead class="ltx_thead">
<tr id="S3.T2.1.1" class="ltx_tr">
<th id="S3.T2.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt">
<span id="S3.T2.1.1.1.1" class="ltx_text" style="font-size:90%;">Expert num </span><math id="S3.T2.1.1.1.m1.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S3.T2.1.1.1.m1.1a"><mi mathsize="90%" id="S3.T2.1.1.1.m1.1.1" xref="S3.T2.1.1.1.m1.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S3.T2.1.1.1.m1.1b"><ci id="S3.T2.1.1.1.m1.1.1.cmml" xref="S3.T2.1.1.1.m1.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.1.1.1.m1.1c">N</annotation></semantics></math>
</th>
<th id="S3.T2.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" colspan="2"><span id="S3.T2.1.1.2.1" class="ltx_text" style="font-size:90%;">16k</span></th>
<th id="S3.T2.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" colspan="2"><span id="S3.T2.1.1.3.1" class="ltx_text" style="font-size:90%;">65k</span></th>
<th id="S3.T2.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" colspan="2"><span id="S3.T2.1.1.4.1" class="ltx_text" style="font-size:90%;">262k</span></th>
<th id="S3.T2.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2"><span id="S3.T2.1.1.5.1" class="ltx_text" style="font-size:90%;">1M</span></th>
</tr>
<tr id="S3.T2.26.27.1" class="ltx_tr">
<th id="S3.T2.26.27.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_r"><span id="S3.T2.26.27.1.1.1" class="ltx_text" style="font-size:90%;">BatchNorm</span></th>
<th id="S3.T2.26.27.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S3.T2.26.27.1.2.1" class="ltx_text" style="font-size:90%;">No</span></th>
<th id="S3.T2.26.27.1.3" class="ltx_td ltx_nopad_l ltx_align_center ltx_th ltx_th_column ltx_border_r"><span id="S3.T2.26.27.1.3.1" class="ltx_text" style="font-size:90%;">Yes</span></th>
<th id="S3.T2.26.27.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S3.T2.26.27.1.4.1" class="ltx_text" style="font-size:90%;">No</span></th>
<th id="S3.T2.26.27.1.5" class="ltx_td ltx_nopad_l ltx_align_center ltx_th ltx_th_column ltx_border_r"><span id="S3.T2.26.27.1.5.1" class="ltx_text" style="font-size:90%;">Yes</span></th>
<th id="S3.T2.26.27.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S3.T2.26.27.1.6.1" class="ltx_text" style="font-size:90%;">No</span></th>
<th id="S3.T2.26.27.1.7" class="ltx_td ltx_nopad_l ltx_align_center ltx_th ltx_th_column ltx_border_r"><span id="S3.T2.26.27.1.7.1" class="ltx_text" style="font-size:90%;">Yes</span></th>
<th id="S3.T2.26.27.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S3.T2.26.27.1.8.1" class="ltx_text" style="font-size:90%;">No</span></th>
<th id="S3.T2.26.27.1.9" class="ltx_td ltx_nopad_l ltx_align_center ltx_th ltx_th_column"><span id="S3.T2.26.27.1.9.1" class="ltx_text" style="font-size:90%;">Yes</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S3.T2.9.9" class="ltx_tr">
<th id="S3.T2.9.9.9" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t"><span id="S3.T2.9.9.9.1" class="ltx_text" style="font-size:90%;">Perplexity</span></th>
<td id="S3.T2.2.2.1" class="ltx_td ltx_align_center ltx_border_t"><math id="S3.T2.2.2.1.m1.1" class="ltx_Math" alttext="23.4658" display="inline"><semantics id="S3.T2.2.2.1.m1.1a"><mn mathsize="90%" id="S3.T2.2.2.1.m1.1.1" xref="S3.T2.2.2.1.m1.1.1.cmml">23.4658</mn><annotation-xml encoding="MathML-Content" id="S3.T2.2.2.1.m1.1b"><cn type="float" id="S3.T2.2.2.1.m1.1.1.cmml" xref="S3.T2.2.2.1.m1.1.1">23.4658</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.2.2.1.m1.1c">23.4658</annotation></semantics></math></td>
<td id="S3.T2.3.3.2" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_r ltx_border_t"><math id="S3.T2.3.3.2.m1.1" class="ltx_Math" alttext="23.4690" display="inline"><semantics id="S3.T2.3.3.2.m1.1a"><mn mathsize="90%" id="S3.T2.3.3.2.m1.1.1" xref="S3.T2.3.3.2.m1.1.1.cmml">23.4690</mn><annotation-xml encoding="MathML-Content" id="S3.T2.3.3.2.m1.1b"><cn type="float" id="S3.T2.3.3.2.m1.1.1.cmml" xref="S3.T2.3.3.2.m1.1.1">23.4690</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.3.3.2.m1.1c">23.4690</annotation></semantics></math></td>
<td id="S3.T2.4.4.3" class="ltx_td ltx_align_center ltx_border_t"><math id="S3.T2.4.4.3.m1.1" class="ltx_Math" alttext="22.6121" display="inline"><semantics id="S3.T2.4.4.3.m1.1a"><mn mathsize="90%" id="S3.T2.4.4.3.m1.1.1" xref="S3.T2.4.4.3.m1.1.1.cmml">22.6121</mn><annotation-xml encoding="MathML-Content" id="S3.T2.4.4.3.m1.1b"><cn type="float" id="S3.T2.4.4.3.m1.1.1.cmml" xref="S3.T2.4.4.3.m1.1.1">22.6121</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.4.4.3.m1.1c">22.6121</annotation></semantics></math></td>
<td id="S3.T2.5.5.4" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_r ltx_border_t"><math id="S3.T2.5.5.4.m1.1" class="ltx_Math" alttext="22.5543" display="inline"><semantics id="S3.T2.5.5.4.m1.1a"><mn mathsize="90%" id="S3.T2.5.5.4.m1.1.1" xref="S3.T2.5.5.4.m1.1.1.cmml">22.5543</mn><annotation-xml encoding="MathML-Content" id="S3.T2.5.5.4.m1.1b"><cn type="float" id="S3.T2.5.5.4.m1.1.1.cmml" xref="S3.T2.5.5.4.m1.1.1">22.5543</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.5.5.4.m1.1c">22.5543</annotation></semantics></math></td>
<td id="S3.T2.6.6.5" class="ltx_td ltx_align_center ltx_border_t"><math id="S3.T2.6.6.5.m1.1" class="ltx_Math" alttext="21.5409" display="inline"><semantics id="S3.T2.6.6.5.m1.1a"><mn mathsize="90%" id="S3.T2.6.6.5.m1.1.1" xref="S3.T2.6.6.5.m1.1.1.cmml">21.5409</mn><annotation-xml encoding="MathML-Content" id="S3.T2.6.6.5.m1.1b"><cn type="float" id="S3.T2.6.6.5.m1.1.1.cmml" xref="S3.T2.6.6.5.m1.1.1">21.5409</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.6.6.5.m1.1c">21.5409</annotation></semantics></math></td>
<td id="S3.T2.7.7.6" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_r ltx_border_t"><math id="S3.T2.7.7.6.m1.1" class="ltx_Math" alttext="21.4696" display="inline"><semantics id="S3.T2.7.7.6.m1.1a"><mn mathsize="90%" id="S3.T2.7.7.6.m1.1.1" xref="S3.T2.7.7.6.m1.1.1.cmml">21.4696</mn><annotation-xml encoding="MathML-Content" id="S3.T2.7.7.6.m1.1b"><cn type="float" id="S3.T2.7.7.6.m1.1.1.cmml" xref="S3.T2.7.7.6.m1.1.1">21.4696</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.7.7.6.m1.1c">21.4696</annotation></semantics></math></td>
<td id="S3.T2.8.8.7" class="ltx_td ltx_align_center ltx_border_t"><math id="S3.T2.8.8.7.m1.1" class="ltx_Math" alttext="20.7307" display="inline"><semantics id="S3.T2.8.8.7.m1.1a"><mn mathsize="90%" id="S3.T2.8.8.7.m1.1.1" xref="S3.T2.8.8.7.m1.1.1.cmml">20.7307</mn><annotation-xml encoding="MathML-Content" id="S3.T2.8.8.7.m1.1b"><cn type="float" id="S3.T2.8.8.7.m1.1.1.cmml" xref="S3.T2.8.8.7.m1.1.1">20.7307</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.8.8.7.m1.1c">20.7307</annotation></semantics></math></td>
<td id="S3.T2.9.9.8" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_t"><math id="S3.T2.9.9.8.m1.1" class="ltx_Math" alttext="20.6449" display="inline"><semantics id="S3.T2.9.9.8.m1.1a"><mn mathsize="90%" id="S3.T2.9.9.8.m1.1.1" xref="S3.T2.9.9.8.m1.1.1.cmml">20.6449</mn><annotation-xml encoding="MathML-Content" id="S3.T2.9.9.8.m1.1b"><cn type="float" id="S3.T2.9.9.8.m1.1.1.cmml" xref="S3.T2.9.9.8.m1.1.1">20.6449</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.9.9.8.m1.1c">20.6449</annotation></semantics></math></td>
</tr>
<tr id="S3.T2.17.17" class="ltx_tr">
<th id="S3.T2.17.17.9" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r"><span id="S3.T2.17.17.9.1" class="ltx_text" style="font-size:90%;">Expert Usage (%)</span></th>
<td id="S3.T2.10.10.1" class="ltx_td ltx_align_center"><math id="S3.T2.10.10.1.m1.1" class="ltx_Math" alttext="100" display="inline"><semantics id="S3.T2.10.10.1.m1.1a"><mn mathsize="90%" id="S3.T2.10.10.1.m1.1.1" xref="S3.T2.10.10.1.m1.1.1.cmml">100</mn><annotation-xml encoding="MathML-Content" id="S3.T2.10.10.1.m1.1b"><cn type="integer" id="S3.T2.10.10.1.m1.1.1.cmml" xref="S3.T2.10.10.1.m1.1.1">100</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.10.10.1.m1.1c">100</annotation></semantics></math></td>
<td id="S3.T2.11.11.2" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_r"><math id="S3.T2.11.11.2.m1.1" class="ltx_Math" alttext="100" display="inline"><semantics id="S3.T2.11.11.2.m1.1a"><mn mathsize="90%" id="S3.T2.11.11.2.m1.1.1" xref="S3.T2.11.11.2.m1.1.1.cmml">100</mn><annotation-xml encoding="MathML-Content" id="S3.T2.11.11.2.m1.1b"><cn type="integer" id="S3.T2.11.11.2.m1.1.1.cmml" xref="S3.T2.11.11.2.m1.1.1">100</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.11.11.2.m1.1c">100</annotation></semantics></math></td>
<td id="S3.T2.12.12.3" class="ltx_td ltx_align_center"><math id="S3.T2.12.12.3.m1.1" class="ltx_Math" alttext="100" display="inline"><semantics id="S3.T2.12.12.3.m1.1a"><mn mathsize="90%" id="S3.T2.12.12.3.m1.1.1" xref="S3.T2.12.12.3.m1.1.1.cmml">100</mn><annotation-xml encoding="MathML-Content" id="S3.T2.12.12.3.m1.1b"><cn type="integer" id="S3.T2.12.12.3.m1.1.1.cmml" xref="S3.T2.12.12.3.m1.1.1">100</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.12.12.3.m1.1c">100</annotation></semantics></math></td>
<td id="S3.T2.13.13.4" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_r"><math id="S3.T2.13.13.4.m1.1" class="ltx_Math" alttext="100" display="inline"><semantics id="S3.T2.13.13.4.m1.1a"><mn mathsize="90%" id="S3.T2.13.13.4.m1.1.1" xref="S3.T2.13.13.4.m1.1.1.cmml">100</mn><annotation-xml encoding="MathML-Content" id="S3.T2.13.13.4.m1.1b"><cn type="integer" id="S3.T2.13.13.4.m1.1.1.cmml" xref="S3.T2.13.13.4.m1.1.1">100</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.13.13.4.m1.1c">100</annotation></semantics></math></td>
<td id="S3.T2.14.14.5" class="ltx_td ltx_align_center"><math id="S3.T2.14.14.5.m1.1" class="ltx_Math" alttext="99.9996" display="inline"><semantics id="S3.T2.14.14.5.m1.1a"><mn mathsize="90%" id="S3.T2.14.14.5.m1.1.1" xref="S3.T2.14.14.5.m1.1.1.cmml">99.9996</mn><annotation-xml encoding="MathML-Content" id="S3.T2.14.14.5.m1.1b"><cn type="float" id="S3.T2.14.14.5.m1.1.1.cmml" xref="S3.T2.14.14.5.m1.1.1">99.9996</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.14.14.5.m1.1c">99.9996</annotation></semantics></math></td>
<td id="S3.T2.15.15.6" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_r"><math id="S3.T2.15.15.6.m1.1" class="ltx_Math" alttext="100" display="inline"><semantics id="S3.T2.15.15.6.m1.1a"><mn mathsize="90%" id="S3.T2.15.15.6.m1.1.1" xref="S3.T2.15.15.6.m1.1.1.cmml">100</mn><annotation-xml encoding="MathML-Content" id="S3.T2.15.15.6.m1.1b"><cn type="integer" id="S3.T2.15.15.6.m1.1.1.cmml" xref="S3.T2.15.15.6.m1.1.1">100</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.15.15.6.m1.1c">100</annotation></semantics></math></td>
<td id="S3.T2.16.16.7" class="ltx_td ltx_align_center"><math id="S3.T2.16.16.7.m1.1" class="ltx_Math" alttext="99.7930" display="inline"><semantics id="S3.T2.16.16.7.m1.1a"><mn mathsize="90%" id="S3.T2.16.16.7.m1.1.1" xref="S3.T2.16.16.7.m1.1.1.cmml">99.7930</mn><annotation-xml encoding="MathML-Content" id="S3.T2.16.16.7.m1.1b"><cn type="float" id="S3.T2.16.16.7.m1.1.1.cmml" xref="S3.T2.16.16.7.m1.1.1">99.7930</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.16.16.7.m1.1c">99.7930</annotation></semantics></math></td>
<td id="S3.T2.17.17.8" class="ltx_td ltx_nopad_l ltx_align_center"><math id="S3.T2.17.17.8.m1.1" class="ltx_Math" alttext="99.9754" display="inline"><semantics id="S3.T2.17.17.8.m1.1a"><mn mathsize="90%" id="S3.T2.17.17.8.m1.1.1" xref="S3.T2.17.17.8.m1.1.1.cmml">99.9754</mn><annotation-xml encoding="MathML-Content" id="S3.T2.17.17.8.m1.1b"><cn type="float" id="S3.T2.17.17.8.m1.1.1.cmml" xref="S3.T2.17.17.8.m1.1.1">99.9754</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.17.17.8.m1.1c">99.9754</annotation></semantics></math></td>
</tr>
<tr id="S3.T2.26.26" class="ltx_tr">
<th id="S3.T2.18.18.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r">
<span id="S3.T2.18.18.1.1" class="ltx_text" style="font-size:90%;">Unevenness (</span><math id="S3.T2.18.18.1.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="S3.T2.18.18.1.m1.1a"><mo mathsize="90%" stretchy="false" id="S3.T2.18.18.1.m1.1.1" xref="S3.T2.18.18.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S3.T2.18.18.1.m1.1b"><ci id="S3.T2.18.18.1.m1.1.1.cmml" xref="S3.T2.18.18.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.18.18.1.m1.1c">\downarrow</annotation></semantics></math><span id="S3.T2.18.18.1.2" class="ltx_text" style="font-size:90%;">)</span>
</th>
<td id="S3.T2.19.19.2" class="ltx_td ltx_align_center ltx_border_bb"><math id="S3.T2.19.19.2.m1.1" class="ltx_Math" alttext="0.4489" display="inline"><semantics id="S3.T2.19.19.2.m1.1a"><mn mathsize="90%" id="S3.T2.19.19.2.m1.1.1" xref="S3.T2.19.19.2.m1.1.1.cmml">0.4489</mn><annotation-xml encoding="MathML-Content" id="S3.T2.19.19.2.m1.1b"><cn type="float" id="S3.T2.19.19.2.m1.1.1.cmml" xref="S3.T2.19.19.2.m1.1.1">0.4489</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.19.19.2.m1.1c">0.4489</annotation></semantics></math></td>
<td id="S3.T2.20.20.3" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_bb ltx_border_r"><math id="S3.T2.20.20.3.m1.1" class="ltx_Math" alttext="0.2954" display="inline"><semantics id="S3.T2.20.20.3.m1.1a"><mn mathsize="90%" id="S3.T2.20.20.3.m1.1.1" xref="S3.T2.20.20.3.m1.1.1.cmml">0.2954</mn><annotation-xml encoding="MathML-Content" id="S3.T2.20.20.3.m1.1b"><cn type="float" id="S3.T2.20.20.3.m1.1.1.cmml" xref="S3.T2.20.20.3.m1.1.1">0.2954</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.20.20.3.m1.1c">0.2954</annotation></semantics></math></td>
<td id="S3.T2.21.21.4" class="ltx_td ltx_align_center ltx_border_bb"><math id="S3.T2.21.21.4.m1.1" class="ltx_Math" alttext="0.6325" display="inline"><semantics id="S3.T2.21.21.4.m1.1a"><mn mathsize="90%" id="S3.T2.21.21.4.m1.1.1" xref="S3.T2.21.21.4.m1.1.1.cmml">0.6325</mn><annotation-xml encoding="MathML-Content" id="S3.T2.21.21.4.m1.1b"><cn type="float" id="S3.T2.21.21.4.m1.1.1.cmml" xref="S3.T2.21.21.4.m1.1.1">0.6325</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.21.21.4.m1.1c">0.6325</annotation></semantics></math></td>
<td id="S3.T2.22.22.5" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_bb ltx_border_r"><math id="S3.T2.22.22.5.m1.1" class="ltx_Math" alttext="0.4420" display="inline"><semantics id="S3.T2.22.22.5.m1.1a"><mn mathsize="90%" id="S3.T2.22.22.5.m1.1.1" xref="S3.T2.22.22.5.m1.1.1.cmml">0.4420</mn><annotation-xml encoding="MathML-Content" id="S3.T2.22.22.5.m1.1b"><cn type="float" id="S3.T2.22.22.5.m1.1.1.cmml" xref="S3.T2.22.22.5.m1.1.1">0.4420</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.22.22.5.m1.1c">0.4420</annotation></semantics></math></td>
<td id="S3.T2.23.23.6" class="ltx_td ltx_align_center ltx_border_bb"><math id="S3.T2.23.23.6.m1.1" class="ltx_Math" alttext="0.9707" display="inline"><semantics id="S3.T2.23.23.6.m1.1a"><mn mathsize="90%" id="S3.T2.23.23.6.m1.1.1" xref="S3.T2.23.23.6.m1.1.1.cmml">0.9707</mn><annotation-xml encoding="MathML-Content" id="S3.T2.23.23.6.m1.1b"><cn type="float" id="S3.T2.23.23.6.m1.1.1.cmml" xref="S3.T2.23.23.6.m1.1.1">0.9707</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.23.23.6.m1.1c">0.9707</annotation></semantics></math></td>
<td id="S3.T2.24.24.7" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_bb ltx_border_r"><math id="S3.T2.24.24.7.m1.1" class="ltx_Math" alttext="0.6552" display="inline"><semantics id="S3.T2.24.24.7.m1.1a"><mn mathsize="90%" id="S3.T2.24.24.7.m1.1.1" xref="S3.T2.24.24.7.m1.1.1.cmml">0.6552</mn><annotation-xml encoding="MathML-Content" id="S3.T2.24.24.7.m1.1b"><cn type="float" id="S3.T2.24.24.7.m1.1.1.cmml" xref="S3.T2.24.24.7.m1.1.1">0.6552</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.24.24.7.m1.1c">0.6552</annotation></semantics></math></td>
<td id="S3.T2.25.25.8" class="ltx_td ltx_align_center ltx_border_bb"><math id="S3.T2.25.25.8.m1.1" class="ltx_Math" alttext="1.5244" display="inline"><semantics id="S3.T2.25.25.8.m1.1a"><mn mathsize="90%" id="S3.T2.25.25.8.m1.1.1" xref="S3.T2.25.25.8.m1.1.1.cmml">1.5244</mn><annotation-xml encoding="MathML-Content" id="S3.T2.25.25.8.m1.1b"><cn type="float" id="S3.T2.25.25.8.m1.1.1.cmml" xref="S3.T2.25.25.8.m1.1.1">1.5244</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.25.25.8.m1.1c">1.5244</annotation></semantics></math></td>
<td id="S3.T2.26.26.9" class="ltx_td ltx_nopad_l ltx_align_center ltx_border_bb"><math id="S3.T2.26.26.9.m1.1" class="ltx_Math" alttext="1.0588" display="inline"><semantics id="S3.T2.26.26.9.m1.1a"><mn mathsize="90%" id="S3.T2.26.26.9.m1.1.1" xref="S3.T2.26.26.9.m1.1.1.cmml">1.0588</mn><annotation-xml encoding="MathML-Content" id="S3.T2.26.26.9.m1.1b"><cn type="float" id="S3.T2.26.26.9.m1.1.1.cmml" xref="S3.T2.26.26.9.m1.1.1">1.0588</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.T2.26.26.9.m1.1c">1.0588</annotation></semantics></math></td>
</tr>
</tbody>
</table>
</figure>
</section>
<section id="S3.SS3.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Expert Usage and Query Batch Normalization</h4>

<div id="S3.SS3.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS0.Px3.p1.8" class="ltx_p">Given the presence of over a million experts in the PEER layer, it is natural to inquire how many of these experts are actually selected during inference and whether their usage is evenly distributed. To analyze this, we kept an accumulated router score, denoted as <math id="S3.SS3.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="z^{\prime}_{i}=\sum_{x}g_{i}(x)" display="inline"><semantics id="S3.SS3.SSS0.Px3.p1.1.m1.1a"><mrow id="S3.SS3.SSS0.Px3.p1.1.m1.1.2" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.cmml"><msubsup id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2.cmml"><mi id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2.2.2" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2.2.2.cmml">z</mi><mi id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2.3" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2.3.cmml">i</mi><mo id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2.2.3" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2.2.3.cmml">′</mo></msubsup><mo rspace="0.111em" id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.1" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.1.cmml">=</mo><mrow id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.cmml"><msub id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.1" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.1.cmml"><mo id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.1.2" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.1.2.cmml">∑</mo><mi id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.1.3" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.1.3.cmml">x</mi></msub><mrow id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.cmml"><msub id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.2" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.2.cmml"><mi id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.2.2" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.2.2.cmml">g</mi><mi id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.2.3" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.1" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.1.cmml">​</mo><mrow id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.3.2" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.cmml"><mo stretchy="false" id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.3.2.1" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.cmml">(</mo><mi id="S3.SS3.SSS0.Px3.p1.1.m1.1.1" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.1.cmml">x</mi><mo stretchy="false" id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.3.2.2" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px3.p1.1.m1.1b"><apply id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.cmml" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2"><eq id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.1.cmml" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.1"></eq><apply id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2.cmml" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2.1.cmml" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2">subscript</csymbol><apply id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2.2.cmml" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2.2.1.cmml" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2">superscript</csymbol><ci id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2.2.2.cmml" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2.2.2">𝑧</ci><ci id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2.2.3.cmml" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2.2.3">′</ci></apply><ci id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2.3.cmml" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.2.3">𝑖</ci></apply><apply id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.cmml" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3"><apply id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.1.cmml" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.1.1.cmml" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.1">subscript</csymbol><sum id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.1.2.cmml" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.1.2"></sum><ci id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.1.3.cmml" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.1.3">𝑥</ci></apply><apply id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.cmml" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2"><times id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.1.cmml" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.1"></times><apply id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.2.cmml" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.2.1.cmml" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.2.2.cmml" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.2.2">𝑔</ci><ci id="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.2.3.cmml" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.2.3.2.2.3">𝑖</ci></apply><ci id="S3.SS3.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S3.SS3.SSS0.Px3.p1.1.m1.1.1">𝑥</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px3.p1.1.m1.1c">z^{\prime}_{i}=\sum_{x}g_{i}(x)</annotation></semantics></math> for each expert <math id="S3.SS3.SSS0.Px3.p1.2.m2.1" class="ltx_Math" alttext="e_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px3.p1.2.m2.1a"><msub id="S3.SS3.SSS0.Px3.p1.2.m2.1.1" xref="S3.SS3.SSS0.Px3.p1.2.m2.1.1.cmml"><mi id="S3.SS3.SSS0.Px3.p1.2.m2.1.1.2" xref="S3.SS3.SSS0.Px3.p1.2.m2.1.1.2.cmml">e</mi><mi id="S3.SS3.SSS0.Px3.p1.2.m2.1.1.3" xref="S3.SS3.SSS0.Px3.p1.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px3.p1.2.m2.1b"><apply id="S3.SS3.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S3.SS3.SSS0.Px3.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px3.p1.2.m2.1.1.1.cmml" xref="S3.SS3.SSS0.Px3.p1.2.m2.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px3.p1.2.m2.1.1.2.cmml" xref="S3.SS3.SSS0.Px3.p1.2.m2.1.1.2">𝑒</ci><ci id="S3.SS3.SSS0.Px3.p1.2.m2.1.1.3.cmml" xref="S3.SS3.SSS0.Px3.p1.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px3.p1.2.m2.1c">e_{i}</annotation></semantics></math> across all tokens <math id="S3.SS3.SSS0.Px3.p1.3.m3.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S3.SS3.SSS0.Px3.p1.3.m3.1a"><mi id="S3.SS3.SSS0.Px3.p1.3.m3.1.1" xref="S3.SS3.SSS0.Px3.p1.3.m3.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px3.p1.3.m3.1b"><ci id="S3.SS3.SSS0.Px3.p1.3.m3.1.1.cmml" xref="S3.SS3.SSS0.Px3.p1.3.m3.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px3.p1.3.m3.1c">x</annotation></semantics></math> within the C4 validation set. Here <math id="S3.SS3.SSS0.Px3.p1.4.m4.1" class="ltx_Math" alttext="g_{i}(x)" display="inline"><semantics id="S3.SS3.SSS0.Px3.p1.4.m4.1a"><mrow id="S3.SS3.SSS0.Px3.p1.4.m4.1.2" xref="S3.SS3.SSS0.Px3.p1.4.m4.1.2.cmml"><msub id="S3.SS3.SSS0.Px3.p1.4.m4.1.2.2" xref="S3.SS3.SSS0.Px3.p1.4.m4.1.2.2.cmml"><mi id="S3.SS3.SSS0.Px3.p1.4.m4.1.2.2.2" xref="S3.SS3.SSS0.Px3.p1.4.m4.1.2.2.2.cmml">g</mi><mi id="S3.SS3.SSS0.Px3.p1.4.m4.1.2.2.3" xref="S3.SS3.SSS0.Px3.p1.4.m4.1.2.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS3.SSS0.Px3.p1.4.m4.1.2.1" xref="S3.SS3.SSS0.Px3.p1.4.m4.1.2.1.cmml">​</mo><mrow id="S3.SS3.SSS0.Px3.p1.4.m4.1.2.3.2" xref="S3.SS3.SSS0.Px3.p1.4.m4.1.2.cmml"><mo stretchy="false" id="S3.SS3.SSS0.Px3.p1.4.m4.1.2.3.2.1" xref="S3.SS3.SSS0.Px3.p1.4.m4.1.2.cmml">(</mo><mi id="S3.SS3.SSS0.Px3.p1.4.m4.1.1" xref="S3.SS3.SSS0.Px3.p1.4.m4.1.1.cmml">x</mi><mo stretchy="false" id="S3.SS3.SSS0.Px3.p1.4.m4.1.2.3.2.2" xref="S3.SS3.SSS0.Px3.p1.4.m4.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px3.p1.4.m4.1b"><apply id="S3.SS3.SSS0.Px3.p1.4.m4.1.2.cmml" xref="S3.SS3.SSS0.Px3.p1.4.m4.1.2"><times id="S3.SS3.SSS0.Px3.p1.4.m4.1.2.1.cmml" xref="S3.SS3.SSS0.Px3.p1.4.m4.1.2.1"></times><apply id="S3.SS3.SSS0.Px3.p1.4.m4.1.2.2.cmml" xref="S3.SS3.SSS0.Px3.p1.4.m4.1.2.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px3.p1.4.m4.1.2.2.1.cmml" xref="S3.SS3.SSS0.Px3.p1.4.m4.1.2.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px3.p1.4.m4.1.2.2.2.cmml" xref="S3.SS3.SSS0.Px3.p1.4.m4.1.2.2.2">𝑔</ci><ci id="S3.SS3.SSS0.Px3.p1.4.m4.1.2.2.3.cmml" xref="S3.SS3.SSS0.Px3.p1.4.m4.1.2.2.3">𝑖</ci></apply><ci id="S3.SS3.SSS0.Px3.p1.4.m4.1.1.cmml" xref="S3.SS3.SSS0.Px3.p1.4.m4.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px3.p1.4.m4.1c">g_{i}(x)</annotation></semantics></math> is the router score used to aggregate the expert output when token <math id="S3.SS3.SSS0.Px3.p1.5.m5.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S3.SS3.SSS0.Px3.p1.5.m5.1a"><mi id="S3.SS3.SSS0.Px3.p1.5.m5.1.1" xref="S3.SS3.SSS0.Px3.p1.5.m5.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px3.p1.5.m5.1b"><ci id="S3.SS3.SSS0.Px3.p1.5.m5.1.1.cmml" xref="S3.SS3.SSS0.Px3.p1.5.m5.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px3.p1.5.m5.1c">x</annotation></semantics></math> is given as input, with <math id="S3.SS3.SSS0.Px3.p1.6.m6.1" class="ltx_Math" alttext="g_{i}(x)=0" display="inline"><semantics id="S3.SS3.SSS0.Px3.p1.6.m6.1a"><mrow id="S3.SS3.SSS0.Px3.p1.6.m6.1.2" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.2.cmml"><mrow id="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.cmml"><msub id="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.2" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.2.cmml"><mi id="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.2.2" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.2.2.cmml">g</mi><mi id="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.2.3" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.1" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.1.cmml">​</mo><mrow id="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.3.2" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.cmml"><mo stretchy="false" id="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.3.2.1" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.cmml">(</mo><mi id="S3.SS3.SSS0.Px3.p1.6.m6.1.1" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.1.cmml">x</mi><mo stretchy="false" id="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.3.2.2" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.cmml">)</mo></mrow></mrow><mo id="S3.SS3.SSS0.Px3.p1.6.m6.1.2.1" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.2.1.cmml">=</mo><mn id="S3.SS3.SSS0.Px3.p1.6.m6.1.2.3" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.2.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px3.p1.6.m6.1b"><apply id="S3.SS3.SSS0.Px3.p1.6.m6.1.2.cmml" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.2"><eq id="S3.SS3.SSS0.Px3.p1.6.m6.1.2.1.cmml" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.2.1"></eq><apply id="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.cmml" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2"><times id="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.1.cmml" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.1"></times><apply id="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.2.cmml" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.2"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.2.1.cmml" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.2">subscript</csymbol><ci id="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.2.2.cmml" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.2.2">𝑔</ci><ci id="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.2.3.cmml" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.2.2.2.3">𝑖</ci></apply><ci id="S3.SS3.SSS0.Px3.p1.6.m6.1.1.cmml" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.1">𝑥</ci></apply><cn type="integer" id="S3.SS3.SSS0.Px3.p1.6.m6.1.2.3.cmml" xref="S3.SS3.SSS0.Px3.p1.6.m6.1.2.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px3.p1.6.m6.1c">g_{i}(x)=0</annotation></semantics></math> if expert <math id="S3.SS3.SSS0.Px3.p1.7.m7.1" class="ltx_Math" alttext="e_{i}" display="inline"><semantics id="S3.SS3.SSS0.Px3.p1.7.m7.1a"><msub id="S3.SS3.SSS0.Px3.p1.7.m7.1.1" xref="S3.SS3.SSS0.Px3.p1.7.m7.1.1.cmml"><mi id="S3.SS3.SSS0.Px3.p1.7.m7.1.1.2" xref="S3.SS3.SSS0.Px3.p1.7.m7.1.1.2.cmml">e</mi><mi id="S3.SS3.SSS0.Px3.p1.7.m7.1.1.3" xref="S3.SS3.SSS0.Px3.p1.7.m7.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px3.p1.7.m7.1b"><apply id="S3.SS3.SSS0.Px3.p1.7.m7.1.1.cmml" xref="S3.SS3.SSS0.Px3.p1.7.m7.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px3.p1.7.m7.1.1.1.cmml" xref="S3.SS3.SSS0.Px3.p1.7.m7.1.1">subscript</csymbol><ci id="S3.SS3.SSS0.Px3.p1.7.m7.1.1.2.cmml" xref="S3.SS3.SSS0.Px3.p1.7.m7.1.1.2">𝑒</ci><ci id="S3.SS3.SSS0.Px3.p1.7.m7.1.1.3.cmml" xref="S3.SS3.SSS0.Px3.p1.7.m7.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px3.p1.7.m7.1c">e_{i}</annotation></semantics></math> is not selected. From these accumulated router scores, we can obtain an empirical probability distribution vector, denoted as <math id="S3.SS3.SSS0.Px3.p1.8.m8.1" class="ltx_Math" alttext="z=z^{\prime}/||z^{\prime}||_{1}" display="inline"><semantics id="S3.SS3.SSS0.Px3.p1.8.m8.1a"><mrow id="S3.SS3.SSS0.Px3.p1.8.m8.1.1" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.cmml"><mi id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.3" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.3.cmml">z</mi><mo id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.2" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.2.cmml">=</mo><mrow id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.cmml"><msup id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.3" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.3.cmml"><mi id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.3.2" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.3.2.cmml">z</mi><mo id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.3.3" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.3.3.cmml">′</mo></msup><mo id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.2" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.2.cmml">/</mo><msub id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.cmml"><mrow id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.1" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.1.2" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.2.1.cmml">‖</mo><msup id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.1.1" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.1.1.cmml"><mi id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.1.1.2" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.1.1.2.cmml">z</mi><mo id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.1.1.3" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.1.1.3.cmml">′</mo></msup><mo stretchy="false" id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.1.3" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.2.1.cmml">‖</mo></mrow><mn id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.3" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.3.cmml">1</mn></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px3.p1.8.m8.1b"><apply id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.cmml" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1"><eq id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.2.cmml" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.2"></eq><ci id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.3.cmml" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.3">𝑧</ci><apply id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.cmml" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1"><divide id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.2.cmml" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.2"></divide><apply id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.3.cmml" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.3.1.cmml" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.3">superscript</csymbol><ci id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.3.2.cmml" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.3.2">𝑧</ci><ci id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.3.3.cmml" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.3.3">′</ci></apply><apply id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.2.cmml" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1">subscript</csymbol><apply id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.2.cmml" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.1"><csymbol cd="latexml" id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.2.1.cmml" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.1.2">norm</csymbol><apply id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.1.1">superscript</csymbol><ci id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.1.1.2.cmml" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.1.1.2">𝑧</ci><ci id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.1.1.3.cmml" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.1.1.1.3">′</ci></apply></apply><cn type="integer" id="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.3.cmml" xref="S3.SS3.SSS0.Px3.p1.8.m8.1.1.1.1.3">1</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px3.p1.8.m8.1c">z=z^{\prime}/||z^{\prime}||_{1}</annotation></semantics></math>, representing the distribution of all experts over the C4 validation set. Then we computed the following metrics proposed by <cite class="ltx_cite ltx_citemacro_citet">Lample et al. (<a href="#bib.bib27" title="" class="ltx_ref">2019</a>)</cite> to assess the usage and distribution of experts:</p>
<ul id="S3.I1" class="ltx_itemize">
<li id="S3.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i1.p1" class="ltx_para">
<p id="S3.I1.i1.p1.1" class="ltx_p"><span id="S3.I1.i1.p1.1.1" class="ltx_text ltx_font_italic">Expert Usage</span>: the fraction of experts retrieved during inference: <math id="S3.I1.i1.p1.1.m1.1" class="ltx_Math" alttext="\#\{z_{i}\neq 0\}" display="inline"><semantics id="S3.I1.i1.p1.1.m1.1a"><mrow id="S3.I1.i1.p1.1.m1.1.1" xref="S3.I1.i1.p1.1.m1.1.1.cmml"><mi mathvariant="normal" id="S3.I1.i1.p1.1.m1.1.1.3" xref="S3.I1.i1.p1.1.m1.1.1.3.cmml">#</mi><mo lspace="0em" rspace="0em" id="S3.I1.i1.p1.1.m1.1.1.2" xref="S3.I1.i1.p1.1.m1.1.1.2.cmml">​</mo><mrow id="S3.I1.i1.p1.1.m1.1.1.1.1" xref="S3.I1.i1.p1.1.m1.1.1.1.2.cmml"><mo stretchy="false" id="S3.I1.i1.p1.1.m1.1.1.1.1.2" xref="S3.I1.i1.p1.1.m1.1.1.1.2.cmml">{</mo><mrow id="S3.I1.i1.p1.1.m1.1.1.1.1.1" xref="S3.I1.i1.p1.1.m1.1.1.1.1.1.cmml"><msub id="S3.I1.i1.p1.1.m1.1.1.1.1.1.2" xref="S3.I1.i1.p1.1.m1.1.1.1.1.1.2.cmml"><mi id="S3.I1.i1.p1.1.m1.1.1.1.1.1.2.2" xref="S3.I1.i1.p1.1.m1.1.1.1.1.1.2.2.cmml">z</mi><mi id="S3.I1.i1.p1.1.m1.1.1.1.1.1.2.3" xref="S3.I1.i1.p1.1.m1.1.1.1.1.1.2.3.cmml">i</mi></msub><mo id="S3.I1.i1.p1.1.m1.1.1.1.1.1.1" xref="S3.I1.i1.p1.1.m1.1.1.1.1.1.1.cmml">≠</mo><mn id="S3.I1.i1.p1.1.m1.1.1.1.1.1.3" xref="S3.I1.i1.p1.1.m1.1.1.1.1.1.3.cmml">0</mn></mrow><mo stretchy="false" id="S3.I1.i1.p1.1.m1.1.1.1.1.3" xref="S3.I1.i1.p1.1.m1.1.1.1.2.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.I1.i1.p1.1.m1.1b"><apply id="S3.I1.i1.p1.1.m1.1.1.cmml" xref="S3.I1.i1.p1.1.m1.1.1"><times id="S3.I1.i1.p1.1.m1.1.1.2.cmml" xref="S3.I1.i1.p1.1.m1.1.1.2"></times><ci id="S3.I1.i1.p1.1.m1.1.1.3.cmml" xref="S3.I1.i1.p1.1.m1.1.1.3">#</ci><set id="S3.I1.i1.p1.1.m1.1.1.1.2.cmml" xref="S3.I1.i1.p1.1.m1.1.1.1.1"><apply id="S3.I1.i1.p1.1.m1.1.1.1.1.1.cmml" xref="S3.I1.i1.p1.1.m1.1.1.1.1.1"><neq id="S3.I1.i1.p1.1.m1.1.1.1.1.1.1.cmml" xref="S3.I1.i1.p1.1.m1.1.1.1.1.1.1"></neq><apply id="S3.I1.i1.p1.1.m1.1.1.1.1.1.2.cmml" xref="S3.I1.i1.p1.1.m1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.I1.i1.p1.1.m1.1.1.1.1.1.2.1.cmml" xref="S3.I1.i1.p1.1.m1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.I1.i1.p1.1.m1.1.1.1.1.1.2.2.cmml" xref="S3.I1.i1.p1.1.m1.1.1.1.1.1.2.2">𝑧</ci><ci id="S3.I1.i1.p1.1.m1.1.1.1.1.1.2.3.cmml" xref="S3.I1.i1.p1.1.m1.1.1.1.1.1.2.3">𝑖</ci></apply><cn type="integer" id="S3.I1.i1.p1.1.m1.1.1.1.1.1.3.cmml" xref="S3.I1.i1.p1.1.m1.1.1.1.1.1.3">0</cn></apply></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i1.p1.1.m1.1c">\#\{z_{i}\neq 0\}</annotation></semantics></math></p>
</div>
</li>
<li id="S3.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i2.p1" class="ltx_para ltx_noindent">
<p id="S3.I1.i2.p1.2" class="ltx_p"><span id="S3.I1.i2.p1.2.1" class="ltx_text ltx_font_italic">Unevenness</span>: KL divergence between <math id="S3.I1.i2.p1.1.m1.1" class="ltx_Math" alttext="z" display="inline"><semantics id="S3.I1.i2.p1.1.m1.1a"><mi id="S3.I1.i2.p1.1.m1.1.1" xref="S3.I1.i2.p1.1.m1.1.1.cmml">z</mi><annotation-xml encoding="MathML-Content" id="S3.I1.i2.p1.1.m1.1b"><ci id="S3.I1.i2.p1.1.m1.1.1.cmml" xref="S3.I1.i2.p1.1.m1.1.1">𝑧</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i2.p1.1.m1.1c">z</annotation></semantics></math> and the uniform distribution: <math id="S3.I1.i2.p1.2.m2.4" class="ltx_Math" alttext="\log(N)+\sum_{i}z_{i}\log(z_{i})" display="inline"><semantics id="S3.I1.i2.p1.2.m2.4a"><mrow id="S3.I1.i2.p1.2.m2.4.4" xref="S3.I1.i2.p1.2.m2.4.4.cmml"><mrow id="S3.I1.i2.p1.2.m2.4.4.3.2" xref="S3.I1.i2.p1.2.m2.4.4.3.1.cmml"><mi id="S3.I1.i2.p1.2.m2.1.1" xref="S3.I1.i2.p1.2.m2.1.1.cmml">log</mi><mo id="S3.I1.i2.p1.2.m2.4.4.3.2a" xref="S3.I1.i2.p1.2.m2.4.4.3.1.cmml">⁡</mo><mrow id="S3.I1.i2.p1.2.m2.4.4.3.2.1" xref="S3.I1.i2.p1.2.m2.4.4.3.1.cmml"><mo stretchy="false" id="S3.I1.i2.p1.2.m2.4.4.3.2.1.1" xref="S3.I1.i2.p1.2.m2.4.4.3.1.cmml">(</mo><mi id="S3.I1.i2.p1.2.m2.2.2" xref="S3.I1.i2.p1.2.m2.2.2.cmml">N</mi><mo stretchy="false" id="S3.I1.i2.p1.2.m2.4.4.3.2.1.2" xref="S3.I1.i2.p1.2.m2.4.4.3.1.cmml">)</mo></mrow></mrow><mo rspace="0.055em" id="S3.I1.i2.p1.2.m2.4.4.2" xref="S3.I1.i2.p1.2.m2.4.4.2.cmml">+</mo><mrow id="S3.I1.i2.p1.2.m2.4.4.1" xref="S3.I1.i2.p1.2.m2.4.4.1.cmml"><msub id="S3.I1.i2.p1.2.m2.4.4.1.2" xref="S3.I1.i2.p1.2.m2.4.4.1.2.cmml"><mo id="S3.I1.i2.p1.2.m2.4.4.1.2.2" xref="S3.I1.i2.p1.2.m2.4.4.1.2.2.cmml">∑</mo><mi id="S3.I1.i2.p1.2.m2.4.4.1.2.3" xref="S3.I1.i2.p1.2.m2.4.4.1.2.3.cmml">i</mi></msub><mrow id="S3.I1.i2.p1.2.m2.4.4.1.1" xref="S3.I1.i2.p1.2.m2.4.4.1.1.cmml"><msub id="S3.I1.i2.p1.2.m2.4.4.1.1.3" xref="S3.I1.i2.p1.2.m2.4.4.1.1.3.cmml"><mi id="S3.I1.i2.p1.2.m2.4.4.1.1.3.2" xref="S3.I1.i2.p1.2.m2.4.4.1.1.3.2.cmml">z</mi><mi id="S3.I1.i2.p1.2.m2.4.4.1.1.3.3" xref="S3.I1.i2.p1.2.m2.4.4.1.1.3.3.cmml">i</mi></msub><mo lspace="0.167em" rspace="0em" id="S3.I1.i2.p1.2.m2.4.4.1.1.2" xref="S3.I1.i2.p1.2.m2.4.4.1.1.2.cmml">​</mo><mrow id="S3.I1.i2.p1.2.m2.4.4.1.1.1.1" xref="S3.I1.i2.p1.2.m2.4.4.1.1.1.2.cmml"><mi id="S3.I1.i2.p1.2.m2.3.3" xref="S3.I1.i2.p1.2.m2.3.3.cmml">log</mi><mo id="S3.I1.i2.p1.2.m2.4.4.1.1.1.1a" xref="S3.I1.i2.p1.2.m2.4.4.1.1.1.2.cmml">⁡</mo><mrow id="S3.I1.i2.p1.2.m2.4.4.1.1.1.1.1" xref="S3.I1.i2.p1.2.m2.4.4.1.1.1.2.cmml"><mo stretchy="false" id="S3.I1.i2.p1.2.m2.4.4.1.1.1.1.1.2" xref="S3.I1.i2.p1.2.m2.4.4.1.1.1.2.cmml">(</mo><msub id="S3.I1.i2.p1.2.m2.4.4.1.1.1.1.1.1" xref="S3.I1.i2.p1.2.m2.4.4.1.1.1.1.1.1.cmml"><mi id="S3.I1.i2.p1.2.m2.4.4.1.1.1.1.1.1.2" xref="S3.I1.i2.p1.2.m2.4.4.1.1.1.1.1.1.2.cmml">z</mi><mi id="S3.I1.i2.p1.2.m2.4.4.1.1.1.1.1.1.3" xref="S3.I1.i2.p1.2.m2.4.4.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S3.I1.i2.p1.2.m2.4.4.1.1.1.1.1.3" xref="S3.I1.i2.p1.2.m2.4.4.1.1.1.2.cmml">)</mo></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.I1.i2.p1.2.m2.4b"><apply id="S3.I1.i2.p1.2.m2.4.4.cmml" xref="S3.I1.i2.p1.2.m2.4.4"><plus id="S3.I1.i2.p1.2.m2.4.4.2.cmml" xref="S3.I1.i2.p1.2.m2.4.4.2"></plus><apply id="S3.I1.i2.p1.2.m2.4.4.3.1.cmml" xref="S3.I1.i2.p1.2.m2.4.4.3.2"><log id="S3.I1.i2.p1.2.m2.1.1.cmml" xref="S3.I1.i2.p1.2.m2.1.1"></log><ci id="S3.I1.i2.p1.2.m2.2.2.cmml" xref="S3.I1.i2.p1.2.m2.2.2">𝑁</ci></apply><apply id="S3.I1.i2.p1.2.m2.4.4.1.cmml" xref="S3.I1.i2.p1.2.m2.4.4.1"><apply id="S3.I1.i2.p1.2.m2.4.4.1.2.cmml" xref="S3.I1.i2.p1.2.m2.4.4.1.2"><csymbol cd="ambiguous" id="S3.I1.i2.p1.2.m2.4.4.1.2.1.cmml" xref="S3.I1.i2.p1.2.m2.4.4.1.2">subscript</csymbol><sum id="S3.I1.i2.p1.2.m2.4.4.1.2.2.cmml" xref="S3.I1.i2.p1.2.m2.4.4.1.2.2"></sum><ci id="S3.I1.i2.p1.2.m2.4.4.1.2.3.cmml" xref="S3.I1.i2.p1.2.m2.4.4.1.2.3">𝑖</ci></apply><apply id="S3.I1.i2.p1.2.m2.4.4.1.1.cmml" xref="S3.I1.i2.p1.2.m2.4.4.1.1"><times id="S3.I1.i2.p1.2.m2.4.4.1.1.2.cmml" xref="S3.I1.i2.p1.2.m2.4.4.1.1.2"></times><apply id="S3.I1.i2.p1.2.m2.4.4.1.1.3.cmml" xref="S3.I1.i2.p1.2.m2.4.4.1.1.3"><csymbol cd="ambiguous" id="S3.I1.i2.p1.2.m2.4.4.1.1.3.1.cmml" xref="S3.I1.i2.p1.2.m2.4.4.1.1.3">subscript</csymbol><ci id="S3.I1.i2.p1.2.m2.4.4.1.1.3.2.cmml" xref="S3.I1.i2.p1.2.m2.4.4.1.1.3.2">𝑧</ci><ci id="S3.I1.i2.p1.2.m2.4.4.1.1.3.3.cmml" xref="S3.I1.i2.p1.2.m2.4.4.1.1.3.3">𝑖</ci></apply><apply id="S3.I1.i2.p1.2.m2.4.4.1.1.1.2.cmml" xref="S3.I1.i2.p1.2.m2.4.4.1.1.1.1"><log id="S3.I1.i2.p1.2.m2.3.3.cmml" xref="S3.I1.i2.p1.2.m2.3.3"></log><apply id="S3.I1.i2.p1.2.m2.4.4.1.1.1.1.1.1.cmml" xref="S3.I1.i2.p1.2.m2.4.4.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.I1.i2.p1.2.m2.4.4.1.1.1.1.1.1.1.cmml" xref="S3.I1.i2.p1.2.m2.4.4.1.1.1.1.1.1">subscript</csymbol><ci id="S3.I1.i2.p1.2.m2.4.4.1.1.1.1.1.1.2.cmml" xref="S3.I1.i2.p1.2.m2.4.4.1.1.1.1.1.1.2">𝑧</ci><ci id="S3.I1.i2.p1.2.m2.4.4.1.1.1.1.1.1.3.cmml" xref="S3.I1.i2.p1.2.m2.4.4.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I1.i2.p1.2.m2.4c">\log(N)+\sum_{i}z_{i}\log(z_{i})</annotation></semantics></math></p>
</div>
</li>
</ul>
<p id="S3.SS3.SSS0.Px3.p1.9" class="ltx_p">where <math id="S3.SS3.SSS0.Px3.p1.9.m1.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S3.SS3.SSS0.Px3.p1.9.m1.1a"><mi id="S3.SS3.SSS0.Px3.p1.9.m1.1.1" xref="S3.SS3.SSS0.Px3.p1.9.m1.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px3.p1.9.m1.1b"><ci id="S3.SS3.SSS0.Px3.p1.9.m1.1.1.cmml" xref="S3.SS3.SSS0.Px3.p1.9.m1.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px3.p1.9.m1.1c">N</annotation></semantics></math> is the number of total experts.</p>
</div>
<div id="S3.SS3.SSS0.Px3.p2" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS0.Px3.p2.1" class="ltx_p">By default, we also added a batch normalization (BN) layer on top of the query network, as proposed by <cite class="ltx_cite ltx_citemacro_citet">Lample et al. (<a href="#bib.bib27" title="" class="ltx_ref">2019</a>)</cite> to increase the expert usage during training. Here we study the effect of adding this BN layer on the above-mentioned metrics.</p>
</div>
<div id="S3.SS3.SSS0.Px3.p3" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS0.Px3.p3.1" class="ltx_p">Table <a href="#S3.T2" title="Table 2 ‣ Varying the Number of Active Experts ‣ 3.3 Ablations ‣ 3 Experiments ‣ Mixture of A Million Experts" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> presents the expert usage and unevenness for varying numbers of experts, with and without BN. We can see that even for 1M experts, the expert usage is close to <math id="S3.SS3.SSS0.Px3.p3.1.m1.1" class="ltx_Math" alttext="100\%" display="inline"><semantics id="S3.SS3.SSS0.Px3.p3.1.m1.1a"><mrow id="S3.SS3.SSS0.Px3.p3.1.m1.1.1" xref="S3.SS3.SSS0.Px3.p3.1.m1.1.1.cmml"><mn id="S3.SS3.SSS0.Px3.p3.1.m1.1.1.2" xref="S3.SS3.SSS0.Px3.p3.1.m1.1.1.2.cmml">100</mn><mo id="S3.SS3.SSS0.Px3.p3.1.m1.1.1.1" xref="S3.SS3.SSS0.Px3.p3.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS0.Px3.p3.1.m1.1b"><apply id="S3.SS3.SSS0.Px3.p3.1.m1.1.1.cmml" xref="S3.SS3.SSS0.Px3.p3.1.m1.1.1"><csymbol cd="latexml" id="S3.SS3.SSS0.Px3.p3.1.m1.1.1.1.cmml" xref="S3.SS3.SSS0.Px3.p3.1.m1.1.1.1">percent</csymbol><cn type="integer" id="S3.SS3.SSS0.Px3.p3.1.m1.1.1.2.cmml" xref="S3.SS3.SSS0.Px3.p3.1.m1.1.1.2">100</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS0.Px3.p3.1.m1.1c">100\%</annotation></semantics></math>, and using BN can lead to more balanced utilization of the experts and lower perplexities. These findings demonstrate the effectiveness of the PEER model in utilizing a large number of experts.</p>
</div>
<figure id="S3.F4" class="ltx_figure"><img src="/html/2407.04153/assets/figs/ablation_bn.png" id="S3.F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="359" height="264" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span><span id="S3.F4.2.1" class="ltx_text ltx_font_bold">Query BatchNorm Ablation.</span> IsoFLOP curves of a PEER model with 1M experts on the C4 dataset, with and without query BatchNorm.</figcaption>
</figure>
<div id="S3.SS3.SSS0.Px3.p4" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS0.Px3.p4.1" class="ltx_p">We additionally compared isoFLOP curves with and without BN. Fig. <a href="#S3.F4" title="Figure 4 ‣ Expert Usage and Query Batch Normalization ‣ 3.3 Ablations ‣ 3 Experiments ‣ Mixture of A Million Experts" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> shows that the PEER model with BN generally achieves lower perplexities. While the difference is not significant, it is most pronounced around the isoFLOP-optimal region.</p>
</div>
</section>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Related Works</h2>

<section id="S4.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Mixture of Expert</h4>

<div id="S4.SS0.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS0.SSS0.Px1.p1.4" class="ltx_p">Since <cite class="ltx_cite ltx_citemacro_citet">Shazeer et al. (<a href="#bib.bib37" title="" class="ltx_ref">2017</a>)</cite> demonstrated the effectiveness of sparsely-gated Mixtures of Experts (MoEs) in efficiently increasing model capacity on GPU clusters, MoEs have emerged as a popular technique for scaling large models efficiently. Subsequent research <cite class="ltx_cite ltx_citemacro_citep">(Fedus et al., <a href="#bib.bib16" title="" class="ltx_ref">2022</a>; Lepikhin et al., <a href="#bib.bib29" title="" class="ltx_ref">2020</a>; Du et al., <a href="#bib.bib15" title="" class="ltx_ref">2022</a>)</cite> has proposed variations to address challenges such as load balancing, communication overhead, and training instability. These methods usually replace feedforward (FFW) layers in certain Transformer blocks with sparsely-gated MoE layers, which consist of multiple FFW layers as experts. Typically each expert matches the size of the regular dense FFW layer. Gating scores are calculated for each expert and token, and only the top k experts are activated for each token. These methods are known as token-choice methods. More recently, <cite class="ltx_cite ltx_citemacro_citet">Zhou et al. (<a href="#bib.bib41" title="" class="ltx_ref">2022</a>)</cite> introduced the Expert Choice routing method, where experts choose the top k tokens instead of tokens selecting experts. However, both token-choice and expert-choice methods require the top-k operator on a gating score matrix of size <math id="S4.SS0.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="N\times M" display="inline"><semantics id="S4.SS0.SSS0.Px1.p1.1.m1.1a"><mrow id="S4.SS0.SSS0.Px1.p1.1.m1.1.1" xref="S4.SS0.SSS0.Px1.p1.1.m1.1.1.cmml"><mi id="S4.SS0.SSS0.Px1.p1.1.m1.1.1.2" xref="S4.SS0.SSS0.Px1.p1.1.m1.1.1.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S4.SS0.SSS0.Px1.p1.1.m1.1.1.1" xref="S4.SS0.SSS0.Px1.p1.1.m1.1.1.1.cmml">×</mo><mi id="S4.SS0.SSS0.Px1.p1.1.m1.1.1.3" xref="S4.SS0.SSS0.Px1.p1.1.m1.1.1.3.cmml">M</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.SS0.SSS0.Px1.p1.1.m1.1b"><apply id="S4.SS0.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S4.SS0.SSS0.Px1.p1.1.m1.1.1"><times id="S4.SS0.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="S4.SS0.SSS0.Px1.p1.1.m1.1.1.1"></times><ci id="S4.SS0.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="S4.SS0.SSS0.Px1.p1.1.m1.1.1.2">𝑁</ci><ci id="S4.SS0.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="S4.SS0.SSS0.Px1.p1.1.m1.1.1.3">𝑀</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS0.SSS0.Px1.p1.1.m1.1c">N\times M</annotation></semantics></math> (<math id="S4.SS0.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S4.SS0.SSS0.Px1.p1.2.m2.1a"><mi id="S4.SS0.SSS0.Px1.p1.2.m2.1.1" xref="S4.SS0.SSS0.Px1.p1.2.m2.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S4.SS0.SSS0.Px1.p1.2.m2.1b"><ci id="S4.SS0.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S4.SS0.SSS0.Px1.p1.2.m2.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS0.SSS0.Px1.p1.2.m2.1c">N</annotation></semantics></math>: number of experts, <math id="S4.SS0.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S4.SS0.SSS0.Px1.p1.3.m3.1a"><mi id="S4.SS0.SSS0.Px1.p1.3.m3.1.1" xref="S4.SS0.SSS0.Px1.p1.3.m3.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S4.SS0.SSS0.Px1.p1.3.m3.1b"><ci id="S4.SS0.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S4.SS0.SSS0.Px1.p1.3.m3.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS0.SSS0.Px1.p1.3.m3.1c">M</annotation></semantics></math>: number of tokens), resulting in a routing cost of at least <math id="S4.SS0.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="O(N)" display="inline"><semantics id="S4.SS0.SSS0.Px1.p1.4.m4.1a"><mrow id="S4.SS0.SSS0.Px1.p1.4.m4.1.2" xref="S4.SS0.SSS0.Px1.p1.4.m4.1.2.cmml"><mi id="S4.SS0.SSS0.Px1.p1.4.m4.1.2.2" xref="S4.SS0.SSS0.Px1.p1.4.m4.1.2.2.cmml">O</mi><mo lspace="0em" rspace="0em" id="S4.SS0.SSS0.Px1.p1.4.m4.1.2.1" xref="S4.SS0.SSS0.Px1.p1.4.m4.1.2.1.cmml">​</mo><mrow id="S4.SS0.SSS0.Px1.p1.4.m4.1.2.3.2" xref="S4.SS0.SSS0.Px1.p1.4.m4.1.2.cmml"><mo stretchy="false" id="S4.SS0.SSS0.Px1.p1.4.m4.1.2.3.2.1" xref="S4.SS0.SSS0.Px1.p1.4.m4.1.2.cmml">(</mo><mi id="S4.SS0.SSS0.Px1.p1.4.m4.1.1" xref="S4.SS0.SSS0.Px1.p1.4.m4.1.1.cmml">N</mi><mo stretchy="false" id="S4.SS0.SSS0.Px1.p1.4.m4.1.2.3.2.2" xref="S4.SS0.SSS0.Px1.p1.4.m4.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS0.SSS0.Px1.p1.4.m4.1b"><apply id="S4.SS0.SSS0.Px1.p1.4.m4.1.2.cmml" xref="S4.SS0.SSS0.Px1.p1.4.m4.1.2"><times id="S4.SS0.SSS0.Px1.p1.4.m4.1.2.1.cmml" xref="S4.SS0.SSS0.Px1.p1.4.m4.1.2.1"></times><ci id="S4.SS0.SSS0.Px1.p1.4.m4.1.2.2.cmml" xref="S4.SS0.SSS0.Px1.p1.4.m4.1.2.2">𝑂</ci><ci id="S4.SS0.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S4.SS0.SSS0.Px1.p1.4.m4.1.1">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS0.SSS0.Px1.p1.4.m4.1c">O(N)</annotation></semantics></math>. This limits their practical application to a small number of experts (typically less than 128).</p>
</div>
<div id="S4.SS0.SSS0.Px1.p2" class="ltx_para ltx_noindent">
<p id="S4.SS0.SSS0.Px1.p2.2" class="ltx_p">Instead of using the top-k operator, some works also proposed using deterministic hash tables as routers <cite class="ltx_cite ltx_citemacro_citep">(Roller et al., <a href="#bib.bib35" title="" class="ltx_ref">2021</a>; dos Santos et al., <a href="#bib.bib14" title="" class="ltx_ref">2023</a>)</cite>. With <math id="S4.SS0.SSS0.Px1.p2.1.m1.1" class="ltx_Math" alttext="O(1)" display="inline"><semantics id="S4.SS0.SSS0.Px1.p2.1.m1.1a"><mrow id="S4.SS0.SSS0.Px1.p2.1.m1.1.2" xref="S4.SS0.SSS0.Px1.p2.1.m1.1.2.cmml"><mi id="S4.SS0.SSS0.Px1.p2.1.m1.1.2.2" xref="S4.SS0.SSS0.Px1.p2.1.m1.1.2.2.cmml">O</mi><mo lspace="0em" rspace="0em" id="S4.SS0.SSS0.Px1.p2.1.m1.1.2.1" xref="S4.SS0.SSS0.Px1.p2.1.m1.1.2.1.cmml">​</mo><mrow id="S4.SS0.SSS0.Px1.p2.1.m1.1.2.3.2" xref="S4.SS0.SSS0.Px1.p2.1.m1.1.2.cmml"><mo stretchy="false" id="S4.SS0.SSS0.Px1.p2.1.m1.1.2.3.2.1" xref="S4.SS0.SSS0.Px1.p2.1.m1.1.2.cmml">(</mo><mn id="S4.SS0.SSS0.Px1.p2.1.m1.1.1" xref="S4.SS0.SSS0.Px1.p2.1.m1.1.1.cmml">1</mn><mo stretchy="false" id="S4.SS0.SSS0.Px1.p2.1.m1.1.2.3.2.2" xref="S4.SS0.SSS0.Px1.p2.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS0.SSS0.Px1.p2.1.m1.1b"><apply id="S4.SS0.SSS0.Px1.p2.1.m1.1.2.cmml" xref="S4.SS0.SSS0.Px1.p2.1.m1.1.2"><times id="S4.SS0.SSS0.Px1.p2.1.m1.1.2.1.cmml" xref="S4.SS0.SSS0.Px1.p2.1.m1.1.2.1"></times><ci id="S4.SS0.SSS0.Px1.p2.1.m1.1.2.2.cmml" xref="S4.SS0.SSS0.Px1.p2.1.m1.1.2.2">𝑂</ci><cn type="integer" id="S4.SS0.SSS0.Px1.p2.1.m1.1.1.cmml" xref="S4.SS0.SSS0.Px1.p2.1.m1.1.1">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS0.SSS0.Px1.p2.1.m1.1c">O(1)</annotation></semantics></math> average lookup complexity, these methods offer potential scalability to a large number of experts. However, these routers are fixed and not learned. <cite class="ltx_cite ltx_citemacro_citet">Clark et al. (<a href="#bib.bib8" title="" class="ltx_ref">2022</a>)</cite> showed that deterministic routing does not scale as well as trainable routers. Furthermore, <cite class="ltx_cite ltx_citemacro_citet">Dikkala et al. (<a href="#bib.bib13" title="" class="ltx_ref">2023</a>)</cite> proved theoretically that learned routers offer non-trivial advantages over their fixed counterparts, such as removing spurious directions and identifying latent clusters in data. In contrast to previous works, the proposed PEER layer employs a learned router with sublinear (<math id="S4.SS0.SSS0.Px1.p2.2.m2.1" class="ltx_Math" alttext="O(\sqrt{N})" display="inline"><semantics id="S4.SS0.SSS0.Px1.p2.2.m2.1a"><mrow id="S4.SS0.SSS0.Px1.p2.2.m2.1.2" xref="S4.SS0.SSS0.Px1.p2.2.m2.1.2.cmml"><mi id="S4.SS0.SSS0.Px1.p2.2.m2.1.2.2" xref="S4.SS0.SSS0.Px1.p2.2.m2.1.2.2.cmml">O</mi><mo lspace="0em" rspace="0em" id="S4.SS0.SSS0.Px1.p2.2.m2.1.2.1" xref="S4.SS0.SSS0.Px1.p2.2.m2.1.2.1.cmml">​</mo><mrow id="S4.SS0.SSS0.Px1.p2.2.m2.1.2.3.2" xref="S4.SS0.SSS0.Px1.p2.2.m2.1.1.cmml"><mo stretchy="false" id="S4.SS0.SSS0.Px1.p2.2.m2.1.2.3.2.1" xref="S4.SS0.SSS0.Px1.p2.2.m2.1.1.cmml">(</mo><msqrt id="S4.SS0.SSS0.Px1.p2.2.m2.1.1" xref="S4.SS0.SSS0.Px1.p2.2.m2.1.1.cmml"><mi id="S4.SS0.SSS0.Px1.p2.2.m2.1.1.2" xref="S4.SS0.SSS0.Px1.p2.2.m2.1.1.2.cmml">N</mi></msqrt><mo stretchy="false" id="S4.SS0.SSS0.Px1.p2.2.m2.1.2.3.2.2" xref="S4.SS0.SSS0.Px1.p2.2.m2.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS0.SSS0.Px1.p2.2.m2.1b"><apply id="S4.SS0.SSS0.Px1.p2.2.m2.1.2.cmml" xref="S4.SS0.SSS0.Px1.p2.2.m2.1.2"><times id="S4.SS0.SSS0.Px1.p2.2.m2.1.2.1.cmml" xref="S4.SS0.SSS0.Px1.p2.2.m2.1.2.1"></times><ci id="S4.SS0.SSS0.Px1.p2.2.m2.1.2.2.cmml" xref="S4.SS0.SSS0.Px1.p2.2.m2.1.2.2">𝑂</ci><apply id="S4.SS0.SSS0.Px1.p2.2.m2.1.1.cmml" xref="S4.SS0.SSS0.Px1.p2.2.m2.1.2.3.2"><root id="S4.SS0.SSS0.Px1.p2.2.m2.1.1a.cmml" xref="S4.SS0.SSS0.Px1.p2.2.m2.1.2.3.2"></root><ci id="S4.SS0.SSS0.Px1.p2.2.m2.1.1.2.cmml" xref="S4.SS0.SSS0.Px1.p2.2.m2.1.1.2">𝑁</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS0.SSS0.Px1.p2.2.m2.1c">O(\sqrt{N})</annotation></semantics></math>) complexity.</p>
</div>
<div id="S4.SS0.SSS0.Px1.p3" class="ltx_para ltx_noindent">
<p id="S4.SS0.SSS0.Px1.p3.1" class="ltx_p">Since PEER uses lightweight experts, our work is also related to recent studies on parameter-efficient MoEs <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib38" title="" class="ltx_ref">2022</a>; Zadouri et al., <a href="#bib.bib40" title="" class="ltx_ref">2024</a>)</cite>. These methods utilize parameter efficient fine-tuning (PEFT) adapters as experts instead of full-sized FFWs. Their focus is on minimizing the number of parameters updated during fine-tuning, allowing storage of only one copy of the large backbone model. In PEER, parameter efficiency refers to the small number of active parameters in the MoE layer, which directly affects FLOPs and activation memory consumption during pre-training and inference. However, PEER could potentially be adapted to retrieve a large number of PEFT adapters.</p>
</div>
</section>
<section id="S4.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Retrieval-Augmented Models</h4>

<div id="S4.SS0.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS0.SSS0.Px2.p1.1" class="ltx_p">Our proposed method, with its retrieval mechanism for a large number of experts, aligns with the emerging field of retrieval-augmented models. These models facilitate large model memorization by retrieving knowledge from external databases, leading to improved accuracy and efficiency on knowledge-intensive tasks. Some notable works in this domain include ones by <cite class="ltx_cite ltx_citemacro_citet">Khandelwal et al. (<a href="#bib.bib24" title="" class="ltx_ref">2019</a>); Borgeaud et al. (<a href="#bib.bib5" title="" class="ltx_ref">2022a</a>); Guu et al. (<a href="#bib.bib20" title="" class="ltx_ref">2020</a>)</cite>. While these methods retrieve data in various formats, for instance, tokens <cite class="ltx_cite ltx_citemacro_citep">(Khandelwal et al., <a href="#bib.bib24" title="" class="ltx_ref">2019</a>)</cite>, chunks <cite class="ltx_cite ltx_citemacro_citep">(Borgeaud et al., <a href="#bib.bib6" title="" class="ltx_ref">2022b</a>)</cite> or knowledge graphs <cite class="ltx_cite ltx_citemacro_citep">(Kang et al., <a href="#bib.bib22" title="" class="ltx_ref">2023</a>)</cite> (see <cite class="ltx_cite ltx_citemacro_citep">(Gao et al., <a href="#bib.bib18" title="" class="ltx_ref">2023</a>)</cite> for a comprehensive survey on this topic), they differ from the proposed method in that they retrieve data rather than learned functions (experts). This distinction sets our parameter-efficient expert retrieval approach apart from existing retrieval-augmented models.</p>
</div>
</section>
<section id="S4.SS0.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Efficient Feedforward Layers</h4>

<div id="S4.SS0.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS0.SSS0.Px3.p1.3" class="ltx_p">Enhancing the efficiency of feedforward networks has been a long-standing area of research. Similar to PEER, most approaches are based on the idea of conditional computation <cite class="ltx_cite ltx_citemacro_citep">(Bengio, <a href="#bib.bib4" title="" class="ltx_ref">2013</a>)</cite>, where a gating mechanism is trained to determine which subset of neurons to compute. For instance, <cite class="ltx_cite ltx_citemacro_citet">Davis &amp; Arel (<a href="#bib.bib12" title="" class="ltx_ref">2013</a>)</cite> utilized low-rank weight matrix approximation to estimate the sign of pre-nonlinearity activations. Neurons with negative activations are omitted as they will produce zeros after the nonlinearity. <cite class="ltx_cite ltx_citemacro_citet">Bengio et al. (<a href="#bib.bib3" title="" class="ltx_ref">2015</a>)</cite> explored reinforcement learning to develop an activation-dependant policy for dropping blocks of neurons. More recently, <cite class="ltx_cite ltx_citemacro_citet">Belcak &amp; Wattenhofer (<a href="#bib.bib2" title="" class="ltx_ref">2023</a>)</cite> introduced the Fast FeedForward (FFF) layer that employs a differentiable balanced binary tree to select a neuron block for computation. During inference, only one leaf (corresponding to one block) is selected, hence it has <math id="S4.SS0.SSS0.Px3.p1.1.m1.3" class="ltx_Math" alttext="O(\log(N))" display="inline"><semantics id="S4.SS0.SSS0.Px3.p1.1.m1.3a"><mrow id="S4.SS0.SSS0.Px3.p1.1.m1.3.3" xref="S4.SS0.SSS0.Px3.p1.1.m1.3.3.cmml"><mi id="S4.SS0.SSS0.Px3.p1.1.m1.3.3.3" xref="S4.SS0.SSS0.Px3.p1.1.m1.3.3.3.cmml">O</mi><mo lspace="0em" rspace="0em" id="S4.SS0.SSS0.Px3.p1.1.m1.3.3.2" xref="S4.SS0.SSS0.Px3.p1.1.m1.3.3.2.cmml">​</mo><mrow id="S4.SS0.SSS0.Px3.p1.1.m1.3.3.1.1" xref="S4.SS0.SSS0.Px3.p1.1.m1.3.3.cmml"><mo stretchy="false" id="S4.SS0.SSS0.Px3.p1.1.m1.3.3.1.1.2" xref="S4.SS0.SSS0.Px3.p1.1.m1.3.3.cmml">(</mo><mrow id="S4.SS0.SSS0.Px3.p1.1.m1.3.3.1.1.1.2" xref="S4.SS0.SSS0.Px3.p1.1.m1.3.3.1.1.1.1.cmml"><mi id="S4.SS0.SSS0.Px3.p1.1.m1.1.1" xref="S4.SS0.SSS0.Px3.p1.1.m1.1.1.cmml">log</mi><mo id="S4.SS0.SSS0.Px3.p1.1.m1.3.3.1.1.1.2a" xref="S4.SS0.SSS0.Px3.p1.1.m1.3.3.1.1.1.1.cmml">⁡</mo><mrow id="S4.SS0.SSS0.Px3.p1.1.m1.3.3.1.1.1.2.1" xref="S4.SS0.SSS0.Px3.p1.1.m1.3.3.1.1.1.1.cmml"><mo stretchy="false" id="S4.SS0.SSS0.Px3.p1.1.m1.3.3.1.1.1.2.1.1" xref="S4.SS0.SSS0.Px3.p1.1.m1.3.3.1.1.1.1.cmml">(</mo><mi id="S4.SS0.SSS0.Px3.p1.1.m1.2.2" xref="S4.SS0.SSS0.Px3.p1.1.m1.2.2.cmml">N</mi><mo stretchy="false" id="S4.SS0.SSS0.Px3.p1.1.m1.3.3.1.1.1.2.1.2" xref="S4.SS0.SSS0.Px3.p1.1.m1.3.3.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S4.SS0.SSS0.Px3.p1.1.m1.3.3.1.1.3" xref="S4.SS0.SSS0.Px3.p1.1.m1.3.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS0.SSS0.Px3.p1.1.m1.3b"><apply id="S4.SS0.SSS0.Px3.p1.1.m1.3.3.cmml" xref="S4.SS0.SSS0.Px3.p1.1.m1.3.3"><times id="S4.SS0.SSS0.Px3.p1.1.m1.3.3.2.cmml" xref="S4.SS0.SSS0.Px3.p1.1.m1.3.3.2"></times><ci id="S4.SS0.SSS0.Px3.p1.1.m1.3.3.3.cmml" xref="S4.SS0.SSS0.Px3.p1.1.m1.3.3.3">𝑂</ci><apply id="S4.SS0.SSS0.Px3.p1.1.m1.3.3.1.1.1.1.cmml" xref="S4.SS0.SSS0.Px3.p1.1.m1.3.3.1.1.1.2"><log id="S4.SS0.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S4.SS0.SSS0.Px3.p1.1.m1.1.1"></log><ci id="S4.SS0.SSS0.Px3.p1.1.m1.2.2.cmml" xref="S4.SS0.SSS0.Px3.p1.1.m1.2.2">𝑁</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS0.SSS0.Px3.p1.1.m1.3c">O(\log(N))</annotation></semantics></math> complexity, where <math id="S4.SS0.SSS0.Px3.p1.2.m2.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S4.SS0.SSS0.Px3.p1.2.m2.1a"><mi id="S4.SS0.SSS0.Px3.p1.2.m2.1.1" xref="S4.SS0.SSS0.Px3.p1.2.m2.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S4.SS0.SSS0.Px3.p1.2.m2.1b"><ci id="S4.SS0.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S4.SS0.SSS0.Px3.p1.2.m2.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS0.SSS0.Px3.p1.2.m2.1c">N</annotation></semantics></math> is the total number of blocks in the tree. However, during training, all leaves and intermediate nodes are activated for gradient calculation, imposing a training complexity of <math id="S4.SS0.SSS0.Px3.p1.3.m3.1" class="ltx_Math" alttext="O(N)" display="inline"><semantics id="S4.SS0.SSS0.Px3.p1.3.m3.1a"><mrow id="S4.SS0.SSS0.Px3.p1.3.m3.1.2" xref="S4.SS0.SSS0.Px3.p1.3.m3.1.2.cmml"><mi id="S4.SS0.SSS0.Px3.p1.3.m3.1.2.2" xref="S4.SS0.SSS0.Px3.p1.3.m3.1.2.2.cmml">O</mi><mo lspace="0em" rspace="0em" id="S4.SS0.SSS0.Px3.p1.3.m3.1.2.1" xref="S4.SS0.SSS0.Px3.p1.3.m3.1.2.1.cmml">​</mo><mrow id="S4.SS0.SSS0.Px3.p1.3.m3.1.2.3.2" xref="S4.SS0.SSS0.Px3.p1.3.m3.1.2.cmml"><mo stretchy="false" id="S4.SS0.SSS0.Px3.p1.3.m3.1.2.3.2.1" xref="S4.SS0.SSS0.Px3.p1.3.m3.1.2.cmml">(</mo><mi id="S4.SS0.SSS0.Px3.p1.3.m3.1.1" xref="S4.SS0.SSS0.Px3.p1.3.m3.1.1.cmml">N</mi><mo stretchy="false" id="S4.SS0.SSS0.Px3.p1.3.m3.1.2.3.2.2" xref="S4.SS0.SSS0.Px3.p1.3.m3.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS0.SSS0.Px3.p1.3.m3.1b"><apply id="S4.SS0.SSS0.Px3.p1.3.m3.1.2.cmml" xref="S4.SS0.SSS0.Px3.p1.3.m3.1.2"><times id="S4.SS0.SSS0.Px3.p1.3.m3.1.2.1.cmml" xref="S4.SS0.SSS0.Px3.p1.3.m3.1.2.1"></times><ci id="S4.SS0.SSS0.Px3.p1.3.m3.1.2.2.cmml" xref="S4.SS0.SSS0.Px3.p1.3.m3.1.2.2">𝑂</ci><ci id="S4.SS0.SSS0.Px3.p1.3.m3.1.1.cmml" xref="S4.SS0.SSS0.Px3.p1.3.m3.1.1">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS0.SSS0.Px3.p1.3.m3.1c">O(N)</annotation></semantics></math> and limiting the total number of blocks. The most relevant work to ours is the Product Key Memory (PKM) <cite class="ltx_cite ltx_citemacro_citep">(Lample et al., <a href="#bib.bib27" title="" class="ltx_ref">2019</a>)</cite>, whose retrieval technique is utilized as the router in the PEER layer. However, PKM retrieves memory vectors instead of functions, thus their values cannot vary according to the inputs. As we show in Section <a href="#S3" title="3 Experiments ‣ Mixture of A Million Experts" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, by changing the memory vectors to input-dependent expert networks, PEER can achieve significantly higher efficiency than PKM. Finally, <cite class="ltx_cite ltx_citemacro_citet">Csordás et al. (<a href="#bib.bib9" title="" class="ltx_ref">2023</a>)</cite> presented a unified view encompassing FFW, MoE and PKM and proposed to change the router normalization function in MoE and PKM from softmax to sigmoid or ReLU.</p>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Conclusion</h2>

<div id="S5.p1" class="ltx_para ltx_noindent">
<p id="S5.p1.1" class="ltx_p">This work introduces a fine-grained MoE architecture that decomposes an extremely wide dense feedforward layer into a large number of small experts. This design is supported by the recent discovery of the fine-grained MoE scaling law. To overcome the computational overhead of routing to a large number of experts, we apply the product keys to efficiently select a small subset of hidden neurons within a wide MLP layer. Empirical analysis using language modeling tasks demonstrate that given the same compute budget, PEER significantly outperforms dense transformers, coarse-grained MoEs and product key memory layers.</p>
</div>
</section>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Acknowledgments</h2>

<div id="Sx1.p1" class="ltx_para ltx_noindent">
<p id="Sx1.p1.1" class="ltx_p">The author would like to thank Adam Santoro, Arthur Guez, Arthur Szlam, Andrei Rusu, Marc’aurelio Ranzato, Simon Schug, Utku Evci, Doina Precup and Razvan Pascanu for their insightful discussions and invaluable advice. The author is also grateful to Zhitao Gong, Daniel Toyama, Qixuan Feng and Jiajun Shen for their technical assistance. Special thanks are due to Adam Santoro for sharing the isoFLOP analysis scripts and to Andy Brock for building and maintaining the internal codebase used to train the models.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Aljundi et al. (2017)</span>
<span class="ltx_bibblock">
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars.

</span>
<span class="ltx_bibblock">Expert gate: Lifelong learning with a network of experts.

</span>
<span class="ltx_bibblock">In <em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE conference on computer vision and
pattern recognition</em>, pp.  3366–3375, 2017.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Belcak &amp; Wattenhofer (2023)</span>
<span class="ltx_bibblock">
Peter Belcak and Roger Wattenhofer.

</span>
<span class="ltx_bibblock">Fast feedforward networks.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2308.14711</em>, 2023.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bengio et al. (2015)</span>
<span class="ltx_bibblock">
Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup.

</span>
<span class="ltx_bibblock">Conditional computation in neural networks for faster models.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1511.06297</em>, 2015.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bengio (2013)</span>
<span class="ltx_bibblock">
Yoshua Bengio.

</span>
<span class="ltx_bibblock">Deep learning of representations: Looking forward.

</span>
<span class="ltx_bibblock">In <em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">International conference on statistical language and speech
processing</em>, pp.  1–37. Springer, 2013.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Borgeaud et al. (2022a)</span>
<span class="ltx_bibblock">
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza
Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste
Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob
Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones,
Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals,
Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre.

</span>
<span class="ltx_bibblock">Improving language models by retrieving from trillions of tokens.

</span>
<span class="ltx_bibblock">In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari,
Gang Niu, and Sivan Sabato (eds.), <em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 39th
International Conference on Machine Learning</em>, volume 162 of
<em id="bib.bib5.2.2" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning Research</em>, pp.  2206–2240. PMLR,
17–23 Jul 2022a.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.mlr.press/v162/borgeaud22a.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.mlr.press/v162/borgeaud22a.html</a>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Borgeaud et al. (2022b)</span>
<span class="ltx_bibblock">
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza
Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste
Lespiau, Bogdan Damoc, Aidan Clark, et al.

</span>
<span class="ltx_bibblock">Improving language models by retrieving from trillions of tokens.

</span>
<span class="ltx_bibblock">In <em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">International conference on machine learning</em>, pp. 2206–2240. PMLR, 2022b.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2023)</span>
<span class="ltx_bibblock">
Wuyang Chen, Yanqi Zhou, Nan Du, Yanping Huang, James Laudon, Zhifeng Chen, and
Claire Cui.

</span>
<span class="ltx_bibblock">Lifelong language pretraining with distribution-specialized experts.

</span>
<span class="ltx_bibblock">In <em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pp. 5383–5395. PMLR, 2023.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Clark et al. (2022)</span>
<span class="ltx_bibblock">
Aidan Clark, Diego De Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini,
Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian
Borgeaud, et al.

</span>
<span class="ltx_bibblock">Unified scaling laws for routed language models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pp. 4057–4086. PMLR, 2022.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Csordás et al. (2023)</span>
<span class="ltx_bibblock">
Róbert Csordás, Kazuki Irie, and Jürgen Schmidhuber.

</span>
<span class="ltx_bibblock">Approximating two-layer feedforward networks for efficient
transformers.

</span>
<span class="ltx_bibblock">In <em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">The 2023 Conference on Empirical Methods in Natural Language
Processing</em>, 2023.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Curation (2020)</span>
<span class="ltx_bibblock">
Curation.

</span>
<span class="ltx_bibblock">Curation corpus base, 2020.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dai et al. (2022)</span>
<span class="ltx_bibblock">
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei.

</span>
<span class="ltx_bibblock">Knowledge neurons in pretrained transformers.

</span>
<span class="ltx_bibblock">In <em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 60th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)</em>, pp.  8493–8502,
2022.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Davis &amp; Arel (2013)</span>
<span class="ltx_bibblock">
Andrew Davis and Itamar Arel.

</span>
<span class="ltx_bibblock">Low-rank approximations for conditional feedforward computation in
deep neural networks.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1312.4461</em>, 2013.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dikkala et al. (2023)</span>
<span class="ltx_bibblock">
Nishanth Dikkala, Nikhil Ghosh, Raghu Meka, Rina Panigrahy, Nikhil Vyas, and
Xin Wang.

</span>
<span class="ltx_bibblock">On the benefits of learning to route in mixture-of-experts models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">The 2023 Conference on Empirical Methods in Natural Language
Processing</em>, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=QV79qiKAjD" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=QV79qiKAjD</a>.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">dos Santos et al. (2023)</span>
<span class="ltx_bibblock">
Cicero Nogueira dos Santos, James Lee-Thorp, Isaac Noble, Chung-Ching Chang,
and David Uthus.

</span>
<span class="ltx_bibblock">Memory augmented language models through mixture of word experts,
2023.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Du et al. (2022)</span>
<span class="ltx_bibblock">
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu,
Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus,
Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang, Kellie Webster, Marie
Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun
Zhang, Quoc Le, Yonghui Wu, Zhifeng Chen, and Claire Cui.

</span>
<span class="ltx_bibblock">GLaM: Efficient scaling of language models with
mixture-of-experts.

</span>
<span class="ltx_bibblock">In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari,
Gang Niu, and Sivan Sabato (eds.), <em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 39th
International Conference on Machine Learning</em>, volume 162 of
<em id="bib.bib15.2.2" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning Research</em>, pp.  5547–5569. PMLR,
17–23 Jul 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.mlr.press/v162/du22c.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.mlr.press/v162/du22c.html</a>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fedus et al. (2022)</span>
<span class="ltx_bibblock">
William Fedus, Barret Zoph, and Noam Shazeer.

</span>
<span class="ltx_bibblock">Switch transformers: Scaling to trillion parameter models with simple
and efficient sparsity.

</span>
<span class="ltx_bibblock"><em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">The Journal of Machine Learning Research</em>, 23(1):5232–5270, 2022.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. (2020)</span>
<span class="ltx_bibblock">
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles
Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser,
and Connor Leahy.

</span>
<span class="ltx_bibblock">The Pile: An 800gb dataset of diverse text for language modeling.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2101.00027</em>, 2020.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gao et al. (2023)</span>
<span class="ltx_bibblock">
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai,
Jiawei Sun, and Haofen Wang.

</span>
<span class="ltx_bibblock">Retrieval-augmented generation for large language models: A survey.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2312.10997</em>, 2023.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Geva et al. (2021)</span>
<span class="ltx_bibblock">
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy.

</span>
<span class="ltx_bibblock">Transformer feed-forward layers are key-value memories.

</span>
<span class="ltx_bibblock">In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott
Wen-tau Yih (eds.), <em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2021 Conference on Empirical
Methods in Natural Language Processing</em>, pp.  5484–5495, Online and Punta
Cana, Dominican Republic, November 2021. Association for Computational
Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2021.emnlp-main.446</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2021.emnlp-main.446" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2021.emnlp-main.446</a>.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Guu et al. (2020)</span>
<span class="ltx_bibblock">
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang.

</span>
<span class="ltx_bibblock">Retrieval augmented language model pre-training.

</span>
<span class="ltx_bibblock">In <em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">International conference on machine learning</em>, pp. 3929–3938. PMLR, 2020.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hoffmann et al. (2022)</span>
<span class="ltx_bibblock">
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor
Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes
Welbl, Aidan Clark, et al.

</span>
<span class="ltx_bibblock">Training compute-optimal large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2203.15556</em>, 2022.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kang et al. (2023)</span>
<span class="ltx_bibblock">
Minki Kang, Jin Myung Kwak, Jinheon Baek, and Sung Ju Hwang.

</span>
<span class="ltx_bibblock">Knowledge graph-augmented language models for knowledge-grounded
dialogue generation.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.18846</em>, 2023.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kaplan et al. (2020)</span>
<span class="ltx_bibblock">
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon
Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.

</span>
<span class="ltx_bibblock">Scaling laws for neural language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2001.08361</em>, 2020.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Khandelwal et al. (2019)</span>
<span class="ltx_bibblock">
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis.

</span>
<span class="ltx_bibblock">Generalization through memorization: Nearest neighbor language
models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2019.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Krajewski et al. (2024)</span>
<span class="ltx_bibblock">
Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro,
Michał Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz
Odrzygóźdź, Piotr Sankowski, et al.

</span>
<span class="ltx_bibblock">Scaling laws for fine-grained mixture of experts.

</span>
<span class="ltx_bibblock"><em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.07871</em>, 2024.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kraska et al. (2018)</span>
<span class="ltx_bibblock">
Tim Kraska, Alex Beutel, Ed H Chi, Jeffrey Dean, and Neoklis Polyzotis.

</span>
<span class="ltx_bibblock">The case for learned index structures.

</span>
<span class="ltx_bibblock">In <em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2018 international conference on
management of data</em>, pp.  489–504, 2018.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lample et al. (2019)</span>
<span class="ltx_bibblock">
Guillaume Lample, Alexandre Sablayrolles, Marc’Aurelio Ranzato, Ludovic
Denoyer, and Hervé Jégou.

</span>
<span class="ltx_bibblock">Large memory layers with product keys.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 32, 2019.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lample et al. (2021)</span>
<span class="ltx_bibblock">
Guillaume Lample, Alexandre Sablayrolles, Marc’Aurelio Ranzato, Ludovic
Denoyer, and Hervé Jégou.

</span>
<span class="ltx_bibblock">Minimalist implementation of a product-key memory layer.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/facebookresearch/XLM/blob/main/PKM-layer.ipynb" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/facebookresearch/XLM/blob/main/PKM-layer.ipynb</a>,
2021.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lepikhin et al. (2020)</span>
<span class="ltx_bibblock">
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping
Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen.

</span>
<span class="ltx_bibblock">Gshard: Scaling giant models with conditional computation and
automatic sharding.

</span>
<span class="ltx_bibblock"><em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2006.16668</em>, 2020.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2024)</span>
<span class="ltx_bibblock">
Hongbo Li, Sen Lin, Lingjie Duan, Yingbin Liang, and Ness B. Shroff.

</span>
<span class="ltx_bibblock">Theory on mixture-of-experts in continual learning, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2406.16437" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2406.16437</a>.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Merity et al. (2016)</span>
<span class="ltx_bibblock">
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.

</span>
<span class="ltx_bibblock">Pointer sentinel mixture models, 2016.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mitchell et al. (2018)</span>
<span class="ltx_bibblock">
Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, Bishan Yang,
Justin Betteridge, Andrew Carlson, Bhavana Dalvi, Matt Gardner, Bryan Kisiel,
et al.

</span>
<span class="ltx_bibblock">Never-ending learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">Communications of the ACM</em>, 61(5):103–115,
2018.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Paperno et al. (2016)</span>
<span class="ltx_bibblock">
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham,
Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel
Fernández.

</span>
<span class="ltx_bibblock">The LAMBADA dataset: Word prediction requiring a broad discourse
context.

</span>
<span class="ltx_bibblock">In <em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers)</em>, pp.  1525–1534,
Berlin, Germany, August 2016. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/P16-1144</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://www.aclweb.org/anthology/P16-1144" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.aclweb.org/anthology/P16-1144</a>.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Raffel et al. (2020)</span>
<span class="ltx_bibblock">
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael
Matena, Yanqi Zhou, Wei Li, and Peter J Liu.

</span>
<span class="ltx_bibblock">Exploring the limits of transfer learning with a unified text-to-text
transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">Journal of machine learning research</em>, 21(140):1–67, 2020.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Roller et al. (2021)</span>
<span class="ltx_bibblock">
Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al.

</span>
<span class="ltx_bibblock">Hash layers for large sparse models.

</span>
<span class="ltx_bibblock"><em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
34:17555–17566, 2021.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shazeer (2020)</span>
<span class="ltx_bibblock">
Noam Shazeer.

</span>
<span class="ltx_bibblock">Glu variants improve transformer, 2020.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2002.05202" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2002.05202</a>.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shazeer et al. (2017)</span>
<span class="ltx_bibblock">
Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le,
Geoffrey Hinton, and Jeff Dean.

</span>
<span class="ltx_bibblock">Outrageously large neural networks: The sparsely-gated
mixture-of-experts layer.

</span>
<span class="ltx_bibblock">In <em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2017.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=B1ckMDqlg" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=B1ckMDqlg</a>.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2022)</span>
<span class="ltx_bibblock">
Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao,
Ahmed Hassan Awadallah, and Jianfeng Gao.

</span>
<span class="ltx_bibblock">AdaMix: Mixture-of-adaptations for parameter-efficient model
tuning.

</span>
<span class="ltx_bibblock">In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.),
<em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2022 Conference on Empirical Methods in Natural
Language Processing</em>, pp.  5744–5760, Abu Dhabi, United Arab Emirates,
December 2022. Association for Computational Linguistics.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.18653/v1/2022.emnlp-main.388</span>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2022.emnlp-main.388" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2022.emnlp-main.388</a>.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yu et al. (2024)</span>
<span class="ltx_bibblock">
Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He.

</span>
<span class="ltx_bibblock">Boosting continual learning of vision-language models via
mixture-of-experts adapters.

</span>
<span class="ltx_bibblock">In <em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR)</em>, pp.  23219–23230, June 2024.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zadouri et al. (2024)</span>
<span class="ltx_bibblock">
Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermis, Acyr Locatelli,
and Sara Hooker.

</span>
<span class="ltx_bibblock">Pushing mixture of experts to the limit: Extremely parameter
efficient moe for instruction tuning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">The Twelfth International Conference on Learning
Representations</em>, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=EvDeiLv7qc" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=EvDeiLv7qc</a>.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. (2022)</span>
<span class="ltx_bibblock">
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M
Dai, Quoc V Le, James Laudon, et al.

</span>
<span class="ltx_bibblock">Mixture-of-experts with expert choice routing.

</span>
<span class="ltx_bibblock"><em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
35:7103–7114, 2022.

</span>
</li>
</ul>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2407.04152" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2407.04153" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2407.04153">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2407.04153" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2407.04154" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Mon Aug  5 13:03:08 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2407.04153
The feedforward (FFW)  layers in standard transformer architectures incur a linear increase in computational costs and activation memory as the hidden layer width grows. Sparse mixture-of-experts (MoE)  architectures have emerged as a viable approach to address this issue by decoupling model size from computational cost. The recent discovery of the fine-grained MoE scaling law shows that higher granularity leads to better performance. However, existing MoE models are limited to a small number of experts due to computational and optimization challenges. This paper introduces PEER (parameter efficient expert retrieval) , a novel layer design that utilizes the product key technique for sparse retrieval from a vast pool of tiny experts (over a million) . Experiments on language modeling tasks demonstrate that PEER layers outperform dense FFWs and coarse-grained MoEs in terms of performance-compute trade-off. By enabling efficient utilization of a massive number of experts, PEER unlocks the potential for further scaling of transformer models while maintaining computational efficiency.
abstract
Mixture of A Million Experts
Abstract
2407.04153
1IntroductionThe past few years have seen the power of scaling: increasing the number of parameters, amount of training data, or the computational budget has proven to be a reliable way to improve model performance. Notably, feedforward (FFW)  layers, responsible for storing factual knowledge, account for two-thirds of the total parameters in a transformer. However, one drawback of these dense FFWs is that their computational footprint (FLOPs and device memory consumption)  is linearly proportional to their parameter count.To break the coupling between computational cost and parameter count, many recent workshave adopted the Mixture-of-Experts (MoE)  architecture, which uses a set of sparsely activated expert modules (often FFWs)  in place of a single dense FFW.studied the scaling law of MoE language models and showed that increasing the number of experts is an effective way to improve performance without increasing the inference cost. However, their experiments showed that the efficiency gains provided by MoEs plateau after a certain model size is reached. More recently,discovered that this plateau was caused by using a fixed number of training tokens. When the number of training tokens is compute-optimal, MoEs consistently outperform dense models in terms of FLOP efficiency. Moreover, they introduced granularity (the number of active experts)  as a new scaling axis and empirically showed that using higher granularity improves performance. Extrapolating this fine-grained MoE scaling law suggests that continued improvement of model capacity will ultimately lead to a large model with high granularity, corresponding to an architecture of an immense number of tiny experts.Beyond efficient scaling, another reason to have a vast number of experts is lifelong learning, where MoE has emerged as a promising approach. For instance,showed that, by simply adding new experts and regularizing them properly, MoE models can adapt to continuous data streams. Freezing old experts and updating only new ones prevents catastrophic forgetting and maintains plasticity by design. In lifelong learning settings, the data stream can be indefinitely long or never-ending, necessitating an expanding pool of experts.Although both efficient scaling and lifelong learning require MoE designs capable of handling a vast number of experts, to the best of our knowledge, the only architecture supporting more than ten thousands of experts is the Mixture of Word Experts (MoWE) . However, MoWE is language-specific and uses a fixed routing scheme. Theoretical and empirical evidencehighlights the advantages of learned routers over non-trainable ones. Thus, an MoE design with a learned router scalable to over a million experts remains an open area for exploration.This work introduces the Parameter Efficient Expert Retrieval (PEER)  architecture, leveraging product key retrievalfor efficient routing to an extremely large number of experts, decoupling computational cost from parameter count. This design demonstrates a superior compute-performance trade-off in our experiments, positioning it as a competitive alternative to dense FFW layers for scaling foundation models. The main contributions of this work are:•Exploration of Extreme MoE Setting:Deviating from the focus on a small number of large experts in previous MoE research, this work investigates the under-explored case of numerous tiny experts.•Learned Index Structure for Routing:Demonstrating for the first time that a learned index structurecan efficiently route to over a million experts.•New Layer Design:Combining product key routing with single-neuron experts, we introduce the PEER layer that expands layer capacity without significant computational overheads. Empirical results demonstrate its superior efficiency compared to dense FFW, coarse-grained MoEs and Product Key Memory (PKM)  layers.•Comprehensive Ablation Studies:We investigate the impact of different design choices of PEER such as number of experts, active parameters, number of heads and query batch normalization on language modeling tasks.Figure 2:Illustration of the PEER layer.A PEER layer can be inserted in the middle of a transformer backbone or can be used to replace FFW layers. Given the state vectorfrom the previous layer, a query networkmaps it to a query vector, which is then compared with the product keys to compute the router scores and to retrieve the topexperts. After the retrieved experts make their predictions, their outputs are linearly combined using the softmax-normalized router scores as weights.
introduction
Mixture of A Million Experts
1Introduction
2407.04153
2MethodIn this section, we introduce the Parameter Efficient Expert Retrieval (PEER)  layer, which is a Mixture of Experts architecture using product keysin the router and single-neuron MLPs as experts. Fig.2illustrates the computational process within a PEER layer.PEER OverviewFormally, a PEER layer is a functionthat consists of three parts: a pool ofexperts, where each expertshares the same signature as, a corresponding set ofproduct keys, and a query networkthat maps the input vectorto a query vector.
Letdenote the top-k operator. Given an input, we first retrieve a subset ofexperts whose corresponding product keys have the highest inner products with the query.# Retrieve topexperts(1) Then we apply nonlinear activations (such as softmax or sigmoid)  to the query-key inner products of these topexperts to obtain the router scores.# Compute router scores(2) Finally, we compute the output by linearly combining the expert outputs weighted by the router scores.# Aggregate expert outputs(3) Product Key RetrievalSince we intend to use a very large number of experts () , naively computing the topindices in Eq.1can be very expensive. Hence we apply the product key retrieval technique here. Instead of usingindependent-dimensional vectors as our keys, we create them by concatenating vectors from two independent sets of-dimensional sub-keys:(4) Note that herehave cardinalityandhave dimensionality. So in practice, we chooseto be a perfect square andto be an even number.This Cartesian product structure ofallows us to find the topexperts efficiently. Instead of comparingto allkeys inand selecting the top k matches, we can split the query vectorinto two sub-queriesandand apply the top k operations to the inner products between the sub-queries and sub-keys respectively:(5) This results in a set ofcandidate keys, and it is mathematically guaranteed that themost similar keys tofromare in this candidate set. Moreover, the inner product between the candidate key andis simply the sum of inner products between the sub-keys and sub-queries:. Hence we can apply the top-k operator again to theseinner products to get the top k matching keys from the original set of product keys. As explained in. This reduces the complexity of top k expert retrieval in Eq.1fromas done naively by exhaustive search to.Parameter Efficient Experts and Multi-Head RetrievalUnlike other MoE architectures, which often set the hidden layer of each expert to the same size as other FFW layers, in PEER, every expertis a singleton MLP, in other words, it has only one hidden layer with a single neuron:(6) whereare not matrices but vectors with the same dimension as, andis a nonlinear activation function such as ReLU or GELU. We omit bias terms here for brevity.Instead of varying the size of individual experts, we adjust the expressiveness of a PEER layer by using multi-head retrieval, similar to the multi-head attention mechanism in transformers and the multi-head memory in PKMs. In particular, we useindependent query networks instead of one, each computes its own query and retrieves a separate set ofexperts. However, different heads share the same pool of experts with the same set of product keys. The outputs of theseheads are simply summed up:(7) One can verify that when only one expert is retrieved ()  per head, using a PEER layer withheads is the same as using one expert withhidden neurons:(8) where. In other words, PEER dynamically assembles an MLP withneurons by aggregatingsingleton MLPs retrieved from a shared repository. Compared to existing MoE approaches that use MLPs with multiple hidden neurons as experts, this design allows shared hidden neurons among experts, enhancing knowledge transfer and parameter efficiency.Algorithm1shows a simplified implementation of the PEER forward pass, storing parameter-efficient expert weights in embedding layers and combining them with einsum operations. This implementation can be easily extended to experts of the GLU variantsby adding additional linear gating weights. In practice, an efficient implementation may require specialized hardware kernels to accelerate embedding lookup and fusion with the einsum operations.Why A Large Number of Small Experts?Given an MoE layer, we can characterize it by three hyperparameters: the total number of parameters, the number of active parameters per tokenand the size of a single expert.showed that the scaling law of MoE models has the following form:(9) whereis the final test loss,are constants,is the total number of training tokens and the granularityis the number of active experts:(10) In order to improve model performance, we need to scale up. On the other hand, it is essential to limitbecause the computational and memory costs are primarily determined by the active parameters during training and inference. Notably, the memory footprint corresponding tohas to be multiplied by the number of tokens in a batch, while the memory cost ofis independent of the batch size and sequence length because only one copy of the model needs to be stored.As a result, we want to increasebut not. Since the expert sizeand the number of experts, this implies that we should decrease the size of each expert,, and increase the number of experts. Hence we need a large number of small experts.In general, for experts that are MLPs with a single hidden layer.and, where,andare the hidden dimension of the transformer, the number of hidden neurons used in one expert and the total number of hidden neurons activated per token, respectively.In the case of PEER, we use the smallest expert size possible by setting, and the number of activated neurons is the number of retrieval heads multiplied by the number of experts retrieved per head:. Consequently, the granularity of PEER is always.⬇1defpeer_forward(self,x) :2#Embeddinglayersstoringthedown/upprojectionweightsofallexperts3self.w_down_embed=nn.Embed(num_embeddings=self.n_experts,features=self.d_model) 4self.w_up_embed=nn.Embed(num_embeddings=self.n_experts,features=self.d_model) 56#Retrievetheweightsofthetopmatchingexpertsusingproductkeys7#indicesandscoreshavetheshape’bthk’,wherehisthenumberofheads8indices,scores=self.get_indices(self.query_proj(x) ,self.sub_keys,top_k=self.k) 9w_down=self.w_down_embed(indices) 10w_up=self.w_up_embed(indices) 1112#Computeweightedaverageofexpertoutputs13x=jnp.einsum(’btd,bthkd->bthk’,x,w_down) 14x=self.activation(x) 15x=x*nn.softmax(scores) 16x=jnp.einsum(’bthk,bthkd->btd’,x,w_up) 17returnxAlgorithm 1:Pseudo code implementation of a PEER layer forward pass. An example implementation of the get_indices and query_proj functions in Pytorch can be found in
body
Mixture of A Million Experts
2Method
2407.04153
3Experiments3.1Pretraining isoFLOP AnalysisWe compare PEER with various baselines using isoFLOP analysis. We chose a fixed FLOP budget (and)  and jointly varied the model size and the number of training tokens from the C4 datasetto obtain isoFLOP curves. Each point on an isoFLOP curve has the same computational cost, and we plot them in terms of their model size and final validation perplexity on C4.For the dense baselines, we varied their size by changing the number of layers, attention heads and model dimensions. For MoE, PKM and PEER methods, we took each of the dense models considered and replaced the FFW layer in the middle block (e.g. in a 12 block transformer, we replace the FFN in block 6)  by a layer of MoE, PKM and PEER, respectively.In MoE, we used the expert-choicerouting algorithm, which effectively addresses the expert load imbalance issue and generally outperforms token-choice MoEs (see Section4for a review and comparison of these approaches) . Each expert has the same size as the original MLPs in the corresponding dense model, and we useexperts to cover the same range of model sizes as our PEER models. This type of MoE represents standard coarse-grained MoE approaches, which consist of a small number of large experts.In PKM, we usedmemories withheads and topmemories were selected per head. We also applied query batch normalization, as recommended in the original PKM paper, to enhance memory usage.In PEER, we usedexperts withheads and topexperts per head. By default, we also enabled query BatchNorm to increase expert usage. Ablation studies in subsection3.3investigate the effect of these hyperparameters. Unlike the expert-choice MoE baseline, PEER represents a fine-grained approach where a large number of small experts are employed.Across all model sizes and methods, we maintained a consistent batch size (128)  and sequence length (2048) . We calculated the number of training steps by dividing the total compute budget by the FLOPs per training step. Fig.1presents the isoFLOP profiles. Compared to the dense FFW baseline, the sparse alternatives shift the isoFLOP curves downward and to right because they introduce a larger number of total parametersbut utilize a smaller or equal number of active parameters. Given the same compute budget, a PEER model achieves the lowest compute-optimal perplexity.3.2Evaluation on Language Modeling DatasetsAfter determining the compute-optimal model for each method based on the isoFLOP curves, we evaluated the performance of these pretrained models on several popular language modeling datasets, including Curation Corpus, Lambada, the Pile, Wikitextand the pretraining dataset C4. Table1presents a summary of the evaluation results. We grouped the models based on their FLOP budgets used during training.Table 1:Perplexities of the compute-optimal models of each method on language modeling datasets.MethodCurationLambadaPileWikitextC4CorpusDense (6e18) 23.2621.9524.5529.1423.84MoE (6e18) 20.9819.0923.2626.1021.41PKM (6e18) 21.8019.3920.4927.0921.92PEER (6e18) 20.6817.6519.0125.4820.63Dense (2e19) 17.7012.2818.1921.2118.31MoE (2e19) 16.8812.9717.4120.2817.12PKM (2e19) 17.0311.1816.3420.2617.36PEER (2e19) 16.3410.3314.9919.0916.453.3AblationsVarying the Number of Total ExpertsThe models in the isoFLOP plot depicted in Fig.1all have over a million ()  experts. Here we conduct an ablation study on the effect of the number of experts, which determines the total parameter countin Eq.9. We selected the model at the isoFLOP-optimal position and vary the number of experts ()  in the PEER layer while keeping the number of active experts constant () . The results are shown in Fig.3(a) . As can be seen, the isoFLOP curve interpolates between the PEER model withexperts and the corresponding dense backbone without replacing the FFW layer in the middle block by a PEER layer. This demonstrates that simply increasing the number experts can improve model performance.(a) Varying Total Expert Num(b) Varying Active Expert NumFigure 3:We conduct two ablation studies using the same PEER model configuration. In (a) , we vary the total number of expertswhile keeping the same number of active experts. In (b) , we vary the number of active expertsby jointly changingandwhile keeping the total number of experts at.Varying the Number of Active ExpertsWe also conducted an ablation study on the effect of the number of active experts, which equals the granularityin Eq.9. We systematically varied the number of active experts ()  while keeping the number of total experts constant () . Furthermore, for a given, we jointly variedandto identify the optimal composition. The resulting isoFLOP curves, plotted over the number of heads () , are shown in Fig.3(b) .The results indicate that, within the range of values considered, highergenerally leads to improved performance. Notably, the optimalincreases asincreases. However, the performance gradually saturates, and increasing the number of active experts also increases device memory consumption and may necessitate additional accelerator devices. Thus in practice, the appropriatevalues should be selected based on the trade-off between performance, device number and computational resource requirements.Table 2:KL and expert usage for different memory sizes, with and without query BN.Similar to the findings in PKM, using query BN results in a more balanced usage of the experts.Expert num16k65k262k1MBatchNormNoYesNoYesNoYesNoYesPerplexityExpert Usage (%) Unevenness () Expert Usage and Query Batch NormalizationGiven the presence of over a million experts in the PEER layer, it is natural to inquire how many of these experts are actually selected during inference and whether their usage is evenly distributed. To analyze this, we kept an accumulated router score, denoted asfor each expertacross all tokenswithin the C4 validation set. Hereis the router score used to aggregate the expert output when tokenis given as input, withif expertis not selected. From these accumulated router scores, we can obtain an empirical probability distribution vector, denoted as, representing the distribution of all experts over the C4 validation set. Then we computed the following metrics proposed byto assess the usage and distribution of experts:•Expert Usage: the fraction of experts retrieved during inference:•Unevenness: KL divergence betweenand the uniform distribution:whereis the number of total experts.By default, we also added a batch normalization (BN)  layer on top of the query network, as proposed byto increase the expert usage during training. Here we study the effect of adding this BN layer on the above-mentioned metrics.Table2presents the expert usage and unevenness for varying numbers of experts, with and without BN. We can see that even for 1M experts, the expert usage is close to, and using BN can lead to more balanced utilization of the experts and lower perplexities. These findings demonstrate the effectiveness of the PEER model in utilizing a large number of experts.Figure 4:Query BatchNorm Ablation.IsoFLOP curves of a PEER model with 1M experts on the C4 dataset, with and without query BatchNorm.We additionally compared isoFLOP curves with and without BN. Fig.4shows that the PEER model with BN generally achieves lower perplexities. While the difference is not significant, it is most pronounced around the isoFLOP-optimal region.
body
Mixture of A Million Experts
3Experiments
2407.04153
4Related WorksMixture of ExpertSincedemonstrated the effectiveness of sparsely-gated Mixtures of Experts (MoEs)  in efficiently increasing model capacity on GPU clusters, MoEs have emerged as a popular technique for scaling large models efficiently. Subsequent researchhas proposed variations to address challenges such as load balancing, communication overhead, and training instability. These methods usually replace feedforward (FFW)  layers in certain Transformer blocks with sparsely-gated MoE layers, which consist of multiple FFW layers as experts. Typically each expert matches the size of the regular dense FFW layer. Gating scores are calculated for each expert and token, and only the top k experts are activated for each token. These methods are known as token-choice methods. More recently,introduced the Expert Choice routing method, where experts choose the top k tokens instead of tokens selecting experts. However, both token-choice and expert-choice methods require the top-k operator on a gating score matrix of size(: number of experts,: number of tokens) , resulting in a routing cost of at least. This limits their practical application to a small number of experts (typically less than 128) .Instead of using the top-k operator, some works also proposed using deterministic hash tables as routers. Withaverage lookup complexity, these methods offer potential scalability to a large number of experts. However, these routers are fixed and not learned.showed that deterministic routing does not scale as well as trainable routers. Furthermore,proved theoretically that learned routers offer non-trivial advantages over their fixed counterparts, such as removing spurious directions and identifying latent clusters in data. In contrast to previous works, the proposed PEER layer employs a learned router with sublinear ()  complexity.Since PEER uses lightweight experts, our work is also related to recent studies on parameter-efficient MoEs. These methods utilize parameter efficient fine-tuning (PEFT)  adapters as experts instead of full-sized FFWs. Their focus is on minimizing the number of parameters updated during fine-tuning, allowing storage of only one copy of the large backbone model. In PEER, parameter efficiency refers to the small number of active parameters in the MoE layer, which directly affects FLOPs and activation memory consumption during pre-training and inference. However, PEER could potentially be adapted to retrieve a large number of PEFT adapters.Retrieval-Augmented ModelsOur proposed method, with its retrieval mechanism for a large number of experts, aligns with the emerging field of retrieval-augmented models. These models facilitate large model memorization by retrieving knowledge from external databases, leading to improved accuracy and efficiency on knowledge-intensive tasks. Some notable works in this domain include ones by. While these methods retrieve data in various formats, for instance, tokens, chunksor knowledge graphs(seefor a comprehensive survey on this topic) , they differ from the proposed method in that they retrieve data rather than learned functions (experts) . This distinction sets our parameter-efficient expert retrieval approach apart from existing retrieval-augmented models.Efficient Feedforward LayersEnhancing the efficiency of feedforward networks has been a long-standing area of research. Similar to PEER, most approaches are based on the idea of conditional computation, where a gating mechanism is trained to determine which subset of neurons to compute. For instance,utilized low-rank weight matrix approximation to estimate the sign of pre-nonlinearity activations. Neurons with negative activations are omitted as they will produce zeros after the nonlinearity.explored reinforcement learning to develop an activation-dependant policy for dropping blocks of neurons. More recently,introduced the Fast FeedForward (FFF)  layer that employs a differentiable balanced binary tree to select a neuron block for computation. During inference, only one leaf (corresponding to one block)  is selected, hence it hascomplexity, whereis the total number of blocks in the tree. However, during training, all leaves and intermediate nodes are activated for gradient calculation, imposing a training complexity ofand limiting the total number of blocks. The most relevant work to ours is the Product Key Memory (PKM) , whose retrieval technique is utilized as the router in the PEER layer. However, PKM retrieves memory vectors instead of functions, thus their values cannot vary according to the inputs. As we show in Section3, by changing the memory vectors to input-dependent expert networks, PEER can achieve significantly higher efficiency than PKM. Finally,presented a unified view encompassing FFW, MoE and PKM and proposed to change the router normalization function in MoE and PKM from softmax to sigmoid or ReLU.
body
Mixture of A Million Experts
4Related Works
2407.04153
5ConclusionThis work introduces a fine-grained MoE architecture that decomposes an extremely wide dense feedforward layer into a large number of small experts. This design is supported by the recent discovery of the fine-grained MoE scaling law. To overcome the computational overhead of routing to a large number of experts, we apply the product keys to efficiently select a small subset of hidden neurons within a wide MLP layer. Empirical analysis using language modeling tasks demonstrate that given the same compute budget, PEER significantly outperforms dense transformers, coarse-grained MoEs and product key memory layers.
body
Mixture of A Million Experts
5Conclusion
2407.04153
AcknowledgmentsThe author would like to thank Adam Santoro, Arthur Guez, Arthur Szlam, Andrei Rusu, Marc’aurelio Ranzato, Simon Schug, Utku Evci, Doina Precup and Razvan Pascanu for their insightful discussions and invaluable advice. The author is also grateful to Zhitao Gong, Daniel Toyama, Qixuan Feng and Jiajun Shen for their technical assistance. Special thanks are due to Adam Santoro for sharing the isoFLOP analysis scripts and to Andy Brock for building and maintaining the internal codebase used to train the models.
conclusion
Mixture of A Million Experts
Acknowledgments