<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2010.13369] Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping</title><meta property="og:description" content="Recently, Transformer-based language models have demonstrated remarkable performance across many NLP domains.
However, the unsupervised pre-training step of these models suffers from unbearable overall computational ex…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2010.13369">

<!--Generated on Mon Mar 18 13:36:05 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Minjia Zhang  Yuxiong He 
<br class="ltx_break">Microsoft Corporation
<br class="ltx_break"><span id="id2.1.id1" class="ltx_text ltx_font_typewriter">{minjiaz,yuxhe}@microsoft.com</span> 
<br class="ltx_break">
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id1.1" class="ltx_p">Recently, Transformer-based language models have demonstrated remarkable performance across many NLP domains.
However, the unsupervised pre-training step of these models suffers from unbearable overall computational expenses.
Current methods for accelerating the pre-training either rely on massive parallelism with advanced hardware or are not applicable to language modeling.
In this work, we propose a method based on <em id="id1.1.1" class="ltx_emph ltx_font_italic">progressive layer dropping</em> that speeds the training of Transformer-based language models, not at the cost of excessive hardware resources but from model architecture change and training technique boosted efficiency.
Extensive experiments on BERT show that the proposed method achieves
a 24% time reduction on average per sample and allows the pre-training to be 2.5<math id="id1.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="id1.1.m1.1a"><mo id="id1.1.m1.1.1" xref="id1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="id1.1.m1.1b"><times id="id1.1.m1.1.1.cmml" xref="id1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="id1.1.m1.1c">\times</annotation></semantics></math> faster than the baseline to get a similar accuracy on downstream tasks. While being faster, our pre-trained models are equipped with strong knowledge transferability, achieving comparable and sometimes higher GLUE score than the baseline when pre-trained with the same number of samples.</p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Natural language processing (NLP) tasks, such as natural language inference <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib1" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">1</span></a>, <a href="#bib.bib2" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">2</span></a>]</cite> and question answering <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">3</span></a>, <a href="#bib.bib4" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">4</span></a>, <a href="#bib.bib5" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">5</span></a>]</cite>, have achieved great success with the development of neural networks. It has been demonstrated recently that Transformer-based networks have obtained superior performance in many NLP tasks (e.g., the GLUE benchmark <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">6</span></a>]</cite> and the challenging multi-hop reasoning task <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib7" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">7</span></a>]</cite>) than recurrent neural networks or convolutional neural networks. BERT trains a deep bidirectional Transformer and obtains outstanding results with transfer learning <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">3</span></a>]</cite>. RoBERTa <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib2" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">2</span></a>]</cite>, which is a robustly optimized version of BERT trained with more steps and larger corpora, achieves state-of-the-art results on 9 GLUE tasks. Megatron-LM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib8" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">8</span></a>]</cite> further advances the state-of-the-art in NLP by significantly increasing the size of BERT model. Finally, there are multiple research proposing different enhanced versions of Transformer-based networks, such as GPT-2/3 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib9" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">9</span></a>, <a href="#bib.bib10" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">10</span></a>]</cite>, XLNet <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib1" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">1</span></a>]</cite>, SpanBERT <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib11" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">11</span></a>]</cite>, BioBERT <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib12" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">12</span></a>]</cite>, UniLM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib13" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">13</span></a>]</cite>, Turing-NLG <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib14" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">14</span></a>]</cite>, and T5 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib15" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">15</span></a>]</cite>.
Due to the exciting prospect, pre-training Transformer networks with a large corpus of text followed by fine-tuning on specific tasks has become a new paradigm for natural language processing.
</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">Despite great success, a big challenge of Transformer networks comes from the training efficiency – even with self-attention and parallelizable recurrence <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">16</span></a>]</cite>, and extremely high performance hardware <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib17" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">17</span></a>]</cite>, the pre-training step still takes a significant amount of time.
To address this challenge, mixed-precision training is explored <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib8" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">8</span></a>, <a href="#bib.bib18" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">18</span></a>]</cite>, where the forward pass and backward pass are computed in half-precision and parameter update is in single precision. However, it requires Tensor Cores <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib19" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">19</span></a>]</cite>, which do not exist in all hardware. Some work resort to distributed training <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib20" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">20</span></a>, <a href="#bib.bib21" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">21</span></a>, <a href="#bib.bib8" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">8</span></a>]</cite>. However, distributed training uses large mini-batch sizes to increase the parallelism, where the training often converges to sharp local minima with poor generalizability even with significant hyperparameter tuning <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib22" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">22</span></a>]</cite>. Subsequently, Yang et al. propose a layer-wise adaptive large batch optimizer called LAMB <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib23" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">23</span></a>]</cite>, allowing to train BERT with 32K batch size on 1024 TPU chips. However, this type of approach often requires dedicated clusters with hundreds or even thousands of GPUs and sophisticated system techniques at managing and tuning distributed training, not to mention that the amount of computational resources is intractable for most research labs or individual practitioners.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">In this paper, we speedup pre-training Transformer networks by
exploring architectural change and training techniques, not at the cost of excessive hardware resources.
Given that the training cost grows linearly with the number of Transformer layers, one straightforward idea to reduce the computation cost is to reduce the depth of the Transformer networks. However, this is restrictive as it often results in lower accuracy in downstream tasks compared to full model pre-training, presumably because of having smaller model capacities <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib24" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">24</span></a>, <a href="#bib.bib25" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">25</span></a>]</cite>. Techniques such as Stochastic Depth have been demonstrated to be useful in accelerating supervised training in the image recognition domain <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib26" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">26</span></a>]</cite>. However, we observe that stochastically removing Transformer layers destabilizes the performance and
easily results in severe consequences such as model divergence or convergence to bad/suspicious local optima. Why are Transformer networks difficult to train with stochastic depth? Moreover, can we speed up the (unsupervised) pre-training of Transformer networks without hurting downstream performance?
</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">To address the above challenges,
we propose to accelerate pre-training of Transformer networks by
making the following contributions. (i) We conduct a comprehensive analysis to answer the
question: what makes Transformer networks difficult to train with stochastic depth. We find that both the choice of Transformer architecture as well as training dynamics would have a big impact on layer dropping. (ii) We propose a new architecture unit, called the <em id="S1.p4.1.1" class="ltx_emph ltx_font_italic">Switchable-Transformer</em> (ST) block, that
not only allows switching on/off a Transformer layer for only a set portion of the training schedule, excluding them from both forward and backward pass but also stabilizes Transformer network training. (iii) We
further propose a <em id="S1.p4.1.2" class="ltx_emph ltx_font_italic">progressive schedule</em> to add extra-stableness for pre-training Transformer networks with layer dropping –
our schedule smoothly increases the layer dropping rate for each mini-batch as training evolves by adapting in time the parameter of the Bernoulli distribution used for sampling. Within each gradient update, we distribute a global layer dropping rate across all the ST blocks to favor different layers. (iv) We use BERT as an example, and we conduct extensive experiments
to show that
the proposed method not only allows to train BERT 24% faster
than the baseline under the same number of samples but also allows the pre-training to be 2.5<math id="S1.p4.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S1.p4.1.m1.1a"><mo id="S1.p4.1.m1.1.1" xref="S1.p4.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.p4.1.m1.1b"><times id="S1.p4.1.m1.1.1.cmml" xref="S1.p4.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.p4.1.m1.1c">\times</annotation></semantics></math> faster to get similar accuracy on downstream tasks.
Furthermore, we evaluate the generalizability of models pre-trained with the same number of samples as the baseline, and we observe
that while faster to train,
our approach achieves a 1.1% higher GLUE score than the baseline, indicating a strong knowledge transferability.
</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Background and Related Work</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.8" class="ltx_p">Pre-training with Transformer-based architectures like BERT <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">3</span></a>]</cite> has been demonstrated as an effective strategy for language representation learning <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib2" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">2</span></a>, <a href="#bib.bib1" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">1</span></a>, <a href="#bib.bib27" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">27</span></a>, <a href="#bib.bib8" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">8</span></a>]</cite>. The approach provides a better model initialization for downstream tasks by training on large-scale unlabeled corpora, which often leads to a better generalization performance on the target task through fine-tuning on small data.
Consider BERT, which consists a stack of <math id="S2.p1.1.m1.1" class="ltx_Math" alttext="L" display="inline"><semantics id="S2.p1.1.m1.1a"><mi id="S2.p1.1.m1.1.1" xref="S2.p1.1.m1.1.1.cmml">L</mi><annotation-xml encoding="MathML-Content" id="S2.p1.1.m1.1b"><ci id="S2.p1.1.m1.1.1.cmml" xref="S2.p1.1.m1.1.1">𝐿</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.1.m1.1c">L</annotation></semantics></math> Transformer layers <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">16</span></a>]</cite>.
Each Transformer layer encodes the the input of the i-th Transformer layer <math id="S2.p1.2.m2.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="S2.p1.2.m2.1a"><msub id="S2.p1.2.m2.1.1" xref="S2.p1.2.m2.1.1.cmml"><mi id="S2.p1.2.m2.1.1.2" xref="S2.p1.2.m2.1.1.2.cmml">x</mi><mi id="S2.p1.2.m2.1.1.3" xref="S2.p1.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.p1.2.m2.1b"><apply id="S2.p1.2.m2.1.1.cmml" xref="S2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S2.p1.2.m2.1.1.1.cmml" xref="S2.p1.2.m2.1.1">subscript</csymbol><ci id="S2.p1.2.m2.1.1.2.cmml" xref="S2.p1.2.m2.1.1.2">𝑥</ci><ci id="S2.p1.2.m2.1.1.3.cmml" xref="S2.p1.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.2.m2.1c">x_{i}</annotation></semantics></math> with <math id="S2.p1.3.m3.1" class="ltx_Math" alttext="h_{i}=f_{LN}(x_{i}+f_{S-ATTN}(x_{i}))" display="inline"><semantics id="S2.p1.3.m3.1a"><mrow id="S2.p1.3.m3.1.1" xref="S2.p1.3.m3.1.1.cmml"><msub id="S2.p1.3.m3.1.1.3" xref="S2.p1.3.m3.1.1.3.cmml"><mi id="S2.p1.3.m3.1.1.3.2" xref="S2.p1.3.m3.1.1.3.2.cmml">h</mi><mi id="S2.p1.3.m3.1.1.3.3" xref="S2.p1.3.m3.1.1.3.3.cmml">i</mi></msub><mo id="S2.p1.3.m3.1.1.2" xref="S2.p1.3.m3.1.1.2.cmml">=</mo><mrow id="S2.p1.3.m3.1.1.1" xref="S2.p1.3.m3.1.1.1.cmml"><msub id="S2.p1.3.m3.1.1.1.3" xref="S2.p1.3.m3.1.1.1.3.cmml"><mi id="S2.p1.3.m3.1.1.1.3.2" xref="S2.p1.3.m3.1.1.1.3.2.cmml">f</mi><mrow id="S2.p1.3.m3.1.1.1.3.3" xref="S2.p1.3.m3.1.1.1.3.3.cmml"><mi id="S2.p1.3.m3.1.1.1.3.3.2" xref="S2.p1.3.m3.1.1.1.3.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S2.p1.3.m3.1.1.1.3.3.1" xref="S2.p1.3.m3.1.1.1.3.3.1.cmml">​</mo><mi id="S2.p1.3.m3.1.1.1.3.3.3" xref="S2.p1.3.m3.1.1.1.3.3.3.cmml">N</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S2.p1.3.m3.1.1.1.2" xref="S2.p1.3.m3.1.1.1.2.cmml">​</mo><mrow id="S2.p1.3.m3.1.1.1.1.1" xref="S2.p1.3.m3.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.p1.3.m3.1.1.1.1.1.2" xref="S2.p1.3.m3.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.p1.3.m3.1.1.1.1.1.1" xref="S2.p1.3.m3.1.1.1.1.1.1.cmml"><msub id="S2.p1.3.m3.1.1.1.1.1.1.3" xref="S2.p1.3.m3.1.1.1.1.1.1.3.cmml"><mi id="S2.p1.3.m3.1.1.1.1.1.1.3.2" xref="S2.p1.3.m3.1.1.1.1.1.1.3.2.cmml">x</mi><mi id="S2.p1.3.m3.1.1.1.1.1.1.3.3" xref="S2.p1.3.m3.1.1.1.1.1.1.3.3.cmml">i</mi></msub><mo id="S2.p1.3.m3.1.1.1.1.1.1.2" xref="S2.p1.3.m3.1.1.1.1.1.1.2.cmml">+</mo><mrow id="S2.p1.3.m3.1.1.1.1.1.1.1" xref="S2.p1.3.m3.1.1.1.1.1.1.1.cmml"><msub id="S2.p1.3.m3.1.1.1.1.1.1.1.3" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.cmml"><mi id="S2.p1.3.m3.1.1.1.1.1.1.1.3.2" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.2.cmml">f</mi><mrow id="S2.p1.3.m3.1.1.1.1.1.1.1.3.3" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.cmml"><mi id="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.2" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.2.cmml">S</mi><mo id="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.1" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.1.cmml">−</mo><mrow id="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.cmml"><mi id="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.2" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.2.cmml">A</mi><mo lspace="0em" rspace="0em" id="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.1" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.1.cmml">​</mo><mi id="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.3" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.3.cmml">T</mi><mo lspace="0em" rspace="0em" id="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.1a" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.1.cmml">​</mo><mi id="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.4" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.4.cmml">T</mi><mo lspace="0em" rspace="0em" id="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.1b" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.1.cmml">​</mo><mi id="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.5" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.5.cmml">N</mi></mrow></mrow></msub><mo lspace="0em" rspace="0em" id="S2.p1.3.m3.1.1.1.1.1.1.1.2" xref="S2.p1.3.m3.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S2.p1.3.m3.1.1.1.1.1.1.1.1.1" xref="S2.p1.3.m3.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.p1.3.m3.1.1.1.1.1.1.1.1.1.2" xref="S2.p1.3.m3.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="S2.p1.3.m3.1.1.1.1.1.1.1.1.1.1" xref="S2.p1.3.m3.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S2.p1.3.m3.1.1.1.1.1.1.1.1.1.1.2" xref="S2.p1.3.m3.1.1.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="S2.p1.3.m3.1.1.1.1.1.1.1.1.1.1.3" xref="S2.p1.3.m3.1.1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S2.p1.3.m3.1.1.1.1.1.1.1.1.1.3" xref="S2.p1.3.m3.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S2.p1.3.m3.1.1.1.1.1.3" xref="S2.p1.3.m3.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.p1.3.m3.1b"><apply id="S2.p1.3.m3.1.1.cmml" xref="S2.p1.3.m3.1.1"><eq id="S2.p1.3.m3.1.1.2.cmml" xref="S2.p1.3.m3.1.1.2"></eq><apply id="S2.p1.3.m3.1.1.3.cmml" xref="S2.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="S2.p1.3.m3.1.1.3.1.cmml" xref="S2.p1.3.m3.1.1.3">subscript</csymbol><ci id="S2.p1.3.m3.1.1.3.2.cmml" xref="S2.p1.3.m3.1.1.3.2">ℎ</ci><ci id="S2.p1.3.m3.1.1.3.3.cmml" xref="S2.p1.3.m3.1.1.3.3">𝑖</ci></apply><apply id="S2.p1.3.m3.1.1.1.cmml" xref="S2.p1.3.m3.1.1.1"><times id="S2.p1.3.m3.1.1.1.2.cmml" xref="S2.p1.3.m3.1.1.1.2"></times><apply id="S2.p1.3.m3.1.1.1.3.cmml" xref="S2.p1.3.m3.1.1.1.3"><csymbol cd="ambiguous" id="S2.p1.3.m3.1.1.1.3.1.cmml" xref="S2.p1.3.m3.1.1.1.3">subscript</csymbol><ci id="S2.p1.3.m3.1.1.1.3.2.cmml" xref="S2.p1.3.m3.1.1.1.3.2">𝑓</ci><apply id="S2.p1.3.m3.1.1.1.3.3.cmml" xref="S2.p1.3.m3.1.1.1.3.3"><times id="S2.p1.3.m3.1.1.1.3.3.1.cmml" xref="S2.p1.3.m3.1.1.1.3.3.1"></times><ci id="S2.p1.3.m3.1.1.1.3.3.2.cmml" xref="S2.p1.3.m3.1.1.1.3.3.2">𝐿</ci><ci id="S2.p1.3.m3.1.1.1.3.3.3.cmml" xref="S2.p1.3.m3.1.1.1.3.3.3">𝑁</ci></apply></apply><apply id="S2.p1.3.m3.1.1.1.1.1.1.cmml" xref="S2.p1.3.m3.1.1.1.1.1"><plus id="S2.p1.3.m3.1.1.1.1.1.1.2.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.2"></plus><apply id="S2.p1.3.m3.1.1.1.1.1.1.3.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.p1.3.m3.1.1.1.1.1.1.3.1.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.p1.3.m3.1.1.1.1.1.1.3.2.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.3.2">𝑥</ci><ci id="S2.p1.3.m3.1.1.1.1.1.1.3.3.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.3.3">𝑖</ci></apply><apply id="S2.p1.3.m3.1.1.1.1.1.1.1.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.1"><times id="S2.p1.3.m3.1.1.1.1.1.1.1.2.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.1.2"></times><apply id="S2.p1.3.m3.1.1.1.1.1.1.1.3.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.p1.3.m3.1.1.1.1.1.1.1.3.1.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.p1.3.m3.1.1.1.1.1.1.1.3.2.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.2">𝑓</ci><apply id="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.3"><minus id="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.1.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.1"></minus><ci id="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.2.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.2">𝑆</ci><apply id="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3"><times id="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.1.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.1"></times><ci id="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.2.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.2">𝐴</ci><ci id="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.3.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.3">𝑇</ci><ci id="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.4.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.4">𝑇</ci><ci id="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.5.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.1.3.3.3.5">𝑁</ci></apply></apply></apply><apply id="S2.p1.3.m3.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.p1.3.m3.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.p1.3.m3.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="S2.p1.3.m3.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.p1.3.m3.1.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.3.m3.1c">h_{i}=f_{LN}(x_{i}+f_{S-ATTN}(x_{i}))</annotation></semantics></math>, which is a multi-head self-attention sub-layer <math id="S2.p1.4.m4.1" class="ltx_Math" alttext="f_{ATTN}" display="inline"><semantics id="S2.p1.4.m4.1a"><msub id="S2.p1.4.m4.1.1" xref="S2.p1.4.m4.1.1.cmml"><mi id="S2.p1.4.m4.1.1.2" xref="S2.p1.4.m4.1.1.2.cmml">f</mi><mrow id="S2.p1.4.m4.1.1.3" xref="S2.p1.4.m4.1.1.3.cmml"><mi id="S2.p1.4.m4.1.1.3.2" xref="S2.p1.4.m4.1.1.3.2.cmml">A</mi><mo lspace="0em" rspace="0em" id="S2.p1.4.m4.1.1.3.1" xref="S2.p1.4.m4.1.1.3.1.cmml">​</mo><mi id="S2.p1.4.m4.1.1.3.3" xref="S2.p1.4.m4.1.1.3.3.cmml">T</mi><mo lspace="0em" rspace="0em" id="S2.p1.4.m4.1.1.3.1a" xref="S2.p1.4.m4.1.1.3.1.cmml">​</mo><mi id="S2.p1.4.m4.1.1.3.4" xref="S2.p1.4.m4.1.1.3.4.cmml">T</mi><mo lspace="0em" rspace="0em" id="S2.p1.4.m4.1.1.3.1b" xref="S2.p1.4.m4.1.1.3.1.cmml">​</mo><mi id="S2.p1.4.m4.1.1.3.5" xref="S2.p1.4.m4.1.1.3.5.cmml">N</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.p1.4.m4.1b"><apply id="S2.p1.4.m4.1.1.cmml" xref="S2.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S2.p1.4.m4.1.1.1.cmml" xref="S2.p1.4.m4.1.1">subscript</csymbol><ci id="S2.p1.4.m4.1.1.2.cmml" xref="S2.p1.4.m4.1.1.2">𝑓</ci><apply id="S2.p1.4.m4.1.1.3.cmml" xref="S2.p1.4.m4.1.1.3"><times id="S2.p1.4.m4.1.1.3.1.cmml" xref="S2.p1.4.m4.1.1.3.1"></times><ci id="S2.p1.4.m4.1.1.3.2.cmml" xref="S2.p1.4.m4.1.1.3.2">𝐴</ci><ci id="S2.p1.4.m4.1.1.3.3.cmml" xref="S2.p1.4.m4.1.1.3.3">𝑇</ci><ci id="S2.p1.4.m4.1.1.3.4.cmml" xref="S2.p1.4.m4.1.1.3.4">𝑇</ci><ci id="S2.p1.4.m4.1.1.3.5.cmml" xref="S2.p1.4.m4.1.1.3.5">𝑁</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.4.m4.1c">f_{ATTN}</annotation></semantics></math>, and then by <math id="S2.p1.5.m5.1" class="ltx_Math" alttext="x_{i+1}=f_{LN}(h_{i}+f_{FFN}(h_{i}))" display="inline"><semantics id="S2.p1.5.m5.1a"><mrow id="S2.p1.5.m5.1.1" xref="S2.p1.5.m5.1.1.cmml"><msub id="S2.p1.5.m5.1.1.3" xref="S2.p1.5.m5.1.1.3.cmml"><mi id="S2.p1.5.m5.1.1.3.2" xref="S2.p1.5.m5.1.1.3.2.cmml">x</mi><mrow id="S2.p1.5.m5.1.1.3.3" xref="S2.p1.5.m5.1.1.3.3.cmml"><mi id="S2.p1.5.m5.1.1.3.3.2" xref="S2.p1.5.m5.1.1.3.3.2.cmml">i</mi><mo id="S2.p1.5.m5.1.1.3.3.1" xref="S2.p1.5.m5.1.1.3.3.1.cmml">+</mo><mn id="S2.p1.5.m5.1.1.3.3.3" xref="S2.p1.5.m5.1.1.3.3.3.cmml">1</mn></mrow></msub><mo id="S2.p1.5.m5.1.1.2" xref="S2.p1.5.m5.1.1.2.cmml">=</mo><mrow id="S2.p1.5.m5.1.1.1" xref="S2.p1.5.m5.1.1.1.cmml"><msub id="S2.p1.5.m5.1.1.1.3" xref="S2.p1.5.m5.1.1.1.3.cmml"><mi id="S2.p1.5.m5.1.1.1.3.2" xref="S2.p1.5.m5.1.1.1.3.2.cmml">f</mi><mrow id="S2.p1.5.m5.1.1.1.3.3" xref="S2.p1.5.m5.1.1.1.3.3.cmml"><mi id="S2.p1.5.m5.1.1.1.3.3.2" xref="S2.p1.5.m5.1.1.1.3.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S2.p1.5.m5.1.1.1.3.3.1" xref="S2.p1.5.m5.1.1.1.3.3.1.cmml">​</mo><mi id="S2.p1.5.m5.1.1.1.3.3.3" xref="S2.p1.5.m5.1.1.1.3.3.3.cmml">N</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S2.p1.5.m5.1.1.1.2" xref="S2.p1.5.m5.1.1.1.2.cmml">​</mo><mrow id="S2.p1.5.m5.1.1.1.1.1" xref="S2.p1.5.m5.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.p1.5.m5.1.1.1.1.1.2" xref="S2.p1.5.m5.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.p1.5.m5.1.1.1.1.1.1" xref="S2.p1.5.m5.1.1.1.1.1.1.cmml"><msub id="S2.p1.5.m5.1.1.1.1.1.1.3" xref="S2.p1.5.m5.1.1.1.1.1.1.3.cmml"><mi id="S2.p1.5.m5.1.1.1.1.1.1.3.2" xref="S2.p1.5.m5.1.1.1.1.1.1.3.2.cmml">h</mi><mi id="S2.p1.5.m5.1.1.1.1.1.1.3.3" xref="S2.p1.5.m5.1.1.1.1.1.1.3.3.cmml">i</mi></msub><mo id="S2.p1.5.m5.1.1.1.1.1.1.2" xref="S2.p1.5.m5.1.1.1.1.1.1.2.cmml">+</mo><mrow id="S2.p1.5.m5.1.1.1.1.1.1.1" xref="S2.p1.5.m5.1.1.1.1.1.1.1.cmml"><msub id="S2.p1.5.m5.1.1.1.1.1.1.1.3" xref="S2.p1.5.m5.1.1.1.1.1.1.1.3.cmml"><mi id="S2.p1.5.m5.1.1.1.1.1.1.1.3.2" xref="S2.p1.5.m5.1.1.1.1.1.1.1.3.2.cmml">f</mi><mrow id="S2.p1.5.m5.1.1.1.1.1.1.1.3.3" xref="S2.p1.5.m5.1.1.1.1.1.1.1.3.3.cmml"><mi id="S2.p1.5.m5.1.1.1.1.1.1.1.3.3.2" xref="S2.p1.5.m5.1.1.1.1.1.1.1.3.3.2.cmml">F</mi><mo lspace="0em" rspace="0em" id="S2.p1.5.m5.1.1.1.1.1.1.1.3.3.1" xref="S2.p1.5.m5.1.1.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="S2.p1.5.m5.1.1.1.1.1.1.1.3.3.3" xref="S2.p1.5.m5.1.1.1.1.1.1.1.3.3.3.cmml">F</mi><mo lspace="0em" rspace="0em" id="S2.p1.5.m5.1.1.1.1.1.1.1.3.3.1a" xref="S2.p1.5.m5.1.1.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="S2.p1.5.m5.1.1.1.1.1.1.1.3.3.4" xref="S2.p1.5.m5.1.1.1.1.1.1.1.3.3.4.cmml">N</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S2.p1.5.m5.1.1.1.1.1.1.1.2" xref="S2.p1.5.m5.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S2.p1.5.m5.1.1.1.1.1.1.1.1.1" xref="S2.p1.5.m5.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.p1.5.m5.1.1.1.1.1.1.1.1.1.2" xref="S2.p1.5.m5.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="S2.p1.5.m5.1.1.1.1.1.1.1.1.1.1" xref="S2.p1.5.m5.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S2.p1.5.m5.1.1.1.1.1.1.1.1.1.1.2" xref="S2.p1.5.m5.1.1.1.1.1.1.1.1.1.1.2.cmml">h</mi><mi id="S2.p1.5.m5.1.1.1.1.1.1.1.1.1.1.3" xref="S2.p1.5.m5.1.1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S2.p1.5.m5.1.1.1.1.1.1.1.1.1.3" xref="S2.p1.5.m5.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S2.p1.5.m5.1.1.1.1.1.3" xref="S2.p1.5.m5.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.p1.5.m5.1b"><apply id="S2.p1.5.m5.1.1.cmml" xref="S2.p1.5.m5.1.1"><eq id="S2.p1.5.m5.1.1.2.cmml" xref="S2.p1.5.m5.1.1.2"></eq><apply id="S2.p1.5.m5.1.1.3.cmml" xref="S2.p1.5.m5.1.1.3"><csymbol cd="ambiguous" id="S2.p1.5.m5.1.1.3.1.cmml" xref="S2.p1.5.m5.1.1.3">subscript</csymbol><ci id="S2.p1.5.m5.1.1.3.2.cmml" xref="S2.p1.5.m5.1.1.3.2">𝑥</ci><apply id="S2.p1.5.m5.1.1.3.3.cmml" xref="S2.p1.5.m5.1.1.3.3"><plus id="S2.p1.5.m5.1.1.3.3.1.cmml" xref="S2.p1.5.m5.1.1.3.3.1"></plus><ci id="S2.p1.5.m5.1.1.3.3.2.cmml" xref="S2.p1.5.m5.1.1.3.3.2">𝑖</ci><cn type="integer" id="S2.p1.5.m5.1.1.3.3.3.cmml" xref="S2.p1.5.m5.1.1.3.3.3">1</cn></apply></apply><apply id="S2.p1.5.m5.1.1.1.cmml" xref="S2.p1.5.m5.1.1.1"><times id="S2.p1.5.m5.1.1.1.2.cmml" xref="S2.p1.5.m5.1.1.1.2"></times><apply id="S2.p1.5.m5.1.1.1.3.cmml" xref="S2.p1.5.m5.1.1.1.3"><csymbol cd="ambiguous" id="S2.p1.5.m5.1.1.1.3.1.cmml" xref="S2.p1.5.m5.1.1.1.3">subscript</csymbol><ci id="S2.p1.5.m5.1.1.1.3.2.cmml" xref="S2.p1.5.m5.1.1.1.3.2">𝑓</ci><apply id="S2.p1.5.m5.1.1.1.3.3.cmml" xref="S2.p1.5.m5.1.1.1.3.3"><times id="S2.p1.5.m5.1.1.1.3.3.1.cmml" xref="S2.p1.5.m5.1.1.1.3.3.1"></times><ci id="S2.p1.5.m5.1.1.1.3.3.2.cmml" xref="S2.p1.5.m5.1.1.1.3.3.2">𝐿</ci><ci id="S2.p1.5.m5.1.1.1.3.3.3.cmml" xref="S2.p1.5.m5.1.1.1.3.3.3">𝑁</ci></apply></apply><apply id="S2.p1.5.m5.1.1.1.1.1.1.cmml" xref="S2.p1.5.m5.1.1.1.1.1"><plus id="S2.p1.5.m5.1.1.1.1.1.1.2.cmml" xref="S2.p1.5.m5.1.1.1.1.1.1.2"></plus><apply id="S2.p1.5.m5.1.1.1.1.1.1.3.cmml" xref="S2.p1.5.m5.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.p1.5.m5.1.1.1.1.1.1.3.1.cmml" xref="S2.p1.5.m5.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.p1.5.m5.1.1.1.1.1.1.3.2.cmml" xref="S2.p1.5.m5.1.1.1.1.1.1.3.2">ℎ</ci><ci id="S2.p1.5.m5.1.1.1.1.1.1.3.3.cmml" xref="S2.p1.5.m5.1.1.1.1.1.1.3.3">𝑖</ci></apply><apply id="S2.p1.5.m5.1.1.1.1.1.1.1.cmml" xref="S2.p1.5.m5.1.1.1.1.1.1.1"><times id="S2.p1.5.m5.1.1.1.1.1.1.1.2.cmml" xref="S2.p1.5.m5.1.1.1.1.1.1.1.2"></times><apply id="S2.p1.5.m5.1.1.1.1.1.1.1.3.cmml" xref="S2.p1.5.m5.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.p1.5.m5.1.1.1.1.1.1.1.3.1.cmml" xref="S2.p1.5.m5.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.p1.5.m5.1.1.1.1.1.1.1.3.2.cmml" xref="S2.p1.5.m5.1.1.1.1.1.1.1.3.2">𝑓</ci><apply id="S2.p1.5.m5.1.1.1.1.1.1.1.3.3.cmml" xref="S2.p1.5.m5.1.1.1.1.1.1.1.3.3"><times id="S2.p1.5.m5.1.1.1.1.1.1.1.3.3.1.cmml" xref="S2.p1.5.m5.1.1.1.1.1.1.1.3.3.1"></times><ci id="S2.p1.5.m5.1.1.1.1.1.1.1.3.3.2.cmml" xref="S2.p1.5.m5.1.1.1.1.1.1.1.3.3.2">𝐹</ci><ci id="S2.p1.5.m5.1.1.1.1.1.1.1.3.3.3.cmml" xref="S2.p1.5.m5.1.1.1.1.1.1.1.3.3.3">𝐹</ci><ci id="S2.p1.5.m5.1.1.1.1.1.1.1.3.3.4.cmml" xref="S2.p1.5.m5.1.1.1.1.1.1.1.3.3.4">𝑁</ci></apply></apply><apply id="S2.p1.5.m5.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.p1.5.m5.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.p1.5.m5.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.p1.5.m5.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.p1.5.m5.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.p1.5.m5.1.1.1.1.1.1.1.1.1.1.2">ℎ</ci><ci id="S2.p1.5.m5.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.p1.5.m5.1.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.5.m5.1c">x_{i+1}=f_{LN}(h_{i}+f_{FFN}(h_{i}))</annotation></semantics></math>, which is a feed-forward network <math id="S2.p1.6.m6.1" class="ltx_Math" alttext="f_{FFN}" display="inline"><semantics id="S2.p1.6.m6.1a"><msub id="S2.p1.6.m6.1.1" xref="S2.p1.6.m6.1.1.cmml"><mi id="S2.p1.6.m6.1.1.2" xref="S2.p1.6.m6.1.1.2.cmml">f</mi><mrow id="S2.p1.6.m6.1.1.3" xref="S2.p1.6.m6.1.1.3.cmml"><mi id="S2.p1.6.m6.1.1.3.2" xref="S2.p1.6.m6.1.1.3.2.cmml">F</mi><mo lspace="0em" rspace="0em" id="S2.p1.6.m6.1.1.3.1" xref="S2.p1.6.m6.1.1.3.1.cmml">​</mo><mi id="S2.p1.6.m6.1.1.3.3" xref="S2.p1.6.m6.1.1.3.3.cmml">F</mi><mo lspace="0em" rspace="0em" id="S2.p1.6.m6.1.1.3.1a" xref="S2.p1.6.m6.1.1.3.1.cmml">​</mo><mi id="S2.p1.6.m6.1.1.3.4" xref="S2.p1.6.m6.1.1.3.4.cmml">N</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.p1.6.m6.1b"><apply id="S2.p1.6.m6.1.1.cmml" xref="S2.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S2.p1.6.m6.1.1.1.cmml" xref="S2.p1.6.m6.1.1">subscript</csymbol><ci id="S2.p1.6.m6.1.1.2.cmml" xref="S2.p1.6.m6.1.1.2">𝑓</ci><apply id="S2.p1.6.m6.1.1.3.cmml" xref="S2.p1.6.m6.1.1.3"><times id="S2.p1.6.m6.1.1.3.1.cmml" xref="S2.p1.6.m6.1.1.3.1"></times><ci id="S2.p1.6.m6.1.1.3.2.cmml" xref="S2.p1.6.m6.1.1.3.2">𝐹</ci><ci id="S2.p1.6.m6.1.1.3.3.cmml" xref="S2.p1.6.m6.1.1.3.3">𝐹</ci><ci id="S2.p1.6.m6.1.1.3.4.cmml" xref="S2.p1.6.m6.1.1.3.4">𝑁</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.6.m6.1c">f_{FFN}</annotation></semantics></math>, where <math id="S2.p1.7.m7.1" class="ltx_Math" alttext="x_{i+1}" display="inline"><semantics id="S2.p1.7.m7.1a"><msub id="S2.p1.7.m7.1.1" xref="S2.p1.7.m7.1.1.cmml"><mi id="S2.p1.7.m7.1.1.2" xref="S2.p1.7.m7.1.1.2.cmml">x</mi><mrow id="S2.p1.7.m7.1.1.3" xref="S2.p1.7.m7.1.1.3.cmml"><mi id="S2.p1.7.m7.1.1.3.2" xref="S2.p1.7.m7.1.1.3.2.cmml">i</mi><mo id="S2.p1.7.m7.1.1.3.1" xref="S2.p1.7.m7.1.1.3.1.cmml">+</mo><mn id="S2.p1.7.m7.1.1.3.3" xref="S2.p1.7.m7.1.1.3.3.cmml">1</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.p1.7.m7.1b"><apply id="S2.p1.7.m7.1.1.cmml" xref="S2.p1.7.m7.1.1"><csymbol cd="ambiguous" id="S2.p1.7.m7.1.1.1.cmml" xref="S2.p1.7.m7.1.1">subscript</csymbol><ci id="S2.p1.7.m7.1.1.2.cmml" xref="S2.p1.7.m7.1.1.2">𝑥</ci><apply id="S2.p1.7.m7.1.1.3.cmml" xref="S2.p1.7.m7.1.1.3"><plus id="S2.p1.7.m7.1.1.3.1.cmml" xref="S2.p1.7.m7.1.1.3.1"></plus><ci id="S2.p1.7.m7.1.1.3.2.cmml" xref="S2.p1.7.m7.1.1.3.2">𝑖</ci><cn type="integer" id="S2.p1.7.m7.1.1.3.3.cmml" xref="S2.p1.7.m7.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.7.m7.1c">x_{i+1}</annotation></semantics></math> is the output of the i-th Transformer layer. Both sub-layers have an AddNorm operation that consists a residual connection <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib28" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">28</span></a>]</cite> and a layer normalization (<math id="S2.p1.8.m8.1" class="ltx_Math" alttext="f_{LN}" display="inline"><semantics id="S2.p1.8.m8.1a"><msub id="S2.p1.8.m8.1.1" xref="S2.p1.8.m8.1.1.cmml"><mi id="S2.p1.8.m8.1.1.2" xref="S2.p1.8.m8.1.1.2.cmml">f</mi><mrow id="S2.p1.8.m8.1.1.3" xref="S2.p1.8.m8.1.1.3.cmml"><mi id="S2.p1.8.m8.1.1.3.2" xref="S2.p1.8.m8.1.1.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S2.p1.8.m8.1.1.3.1" xref="S2.p1.8.m8.1.1.3.1.cmml">​</mo><mi id="S2.p1.8.m8.1.1.3.3" xref="S2.p1.8.m8.1.1.3.3.cmml">N</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.p1.8.m8.1b"><apply id="S2.p1.8.m8.1.1.cmml" xref="S2.p1.8.m8.1.1"><csymbol cd="ambiguous" id="S2.p1.8.m8.1.1.1.cmml" xref="S2.p1.8.m8.1.1">subscript</csymbol><ci id="S2.p1.8.m8.1.1.2.cmml" xref="S2.p1.8.m8.1.1.2">𝑓</ci><apply id="S2.p1.8.m8.1.1.3.cmml" xref="S2.p1.8.m8.1.1.3"><times id="S2.p1.8.m8.1.1.3.1.cmml" xref="S2.p1.8.m8.1.1.3.1"></times><ci id="S2.p1.8.m8.1.1.3.2.cmml" xref="S2.p1.8.m8.1.1.3.2">𝐿</ci><ci id="S2.p1.8.m8.1.1.3.3.cmml" xref="S2.p1.8.m8.1.1.3.3">𝑁</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.8.m8.1c">f_{LN}</annotation></semantics></math>) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib29" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">29</span></a>]</cite>. The BERT model recursively applies the transformer block to the input to get the output.</p>
</div>
<div id="S2.p2" class="ltx_para">
<p id="S2.p2.1" class="ltx_p">While the Transformer-based architecture has achieved breakthrough results in modeling sequences for unsupervised language modeling <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">3</span></a>, <a href="#bib.bib9" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">9</span></a>]</cite>, previous work has also highlighted the training difficulties and excessively long training time <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib2" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">2</span></a>]</cite>.
To speed up the pre-training, ELECTRA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib30" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">30</span></a>]</cite> explores the adversarial training scheme by replacing masked tokens with alternatives sampled from a generator framework and training a discriminator to predict the replaced token. This increases the relative per-step cost, but leads to fewer steps, leading to the overall reduced costs.
Another line of work focus on reducing the per-step cost.
Since the total number of floating-point operations (FLOPS) of the forward and backward passes in the BERT pre-training process is linearly proportional to the depth of the Transformer blocks, reducing the number of Transformer layers brings opportunities to significantly speed up BERT pre-training. To show this, we plot the FLOPS per training iteration in Fig. <a href="#S4.F6" title="Figure 6 ‣ Identity mapping reordering. ‣ 4.1 Switchable-Transformer Blocks ‣ 4 Our Approach: Progressive Layer Dropping ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>, assuming we can remove a fraction of layers at each step. Each line in the figure shows the FLOPS using different layer removal schedules. Regardless of which schedule to choose, the majority of FLOPS are reduced in the later steps, with the rate of keep probability saturating to a fixed value <math id="S2.p2.1.m1.1" class="ltx_Math" alttext="\bar{\theta}" display="inline"><semantics id="S2.p2.1.m1.1a"><mover accent="true" id="S2.p2.1.m1.1.1" xref="S2.p2.1.m1.1.1.cmml"><mi id="S2.p2.1.m1.1.1.2" xref="S2.p2.1.m1.1.1.2.cmml">θ</mi><mo id="S2.p2.1.m1.1.1.1" xref="S2.p2.1.m1.1.1.1.cmml">¯</mo></mover><annotation-xml encoding="MathML-Content" id="S2.p2.1.m1.1b"><apply id="S2.p2.1.m1.1.1.cmml" xref="S2.p2.1.m1.1.1"><ci id="S2.p2.1.m1.1.1.1.cmml" xref="S2.p2.1.m1.1.1.1">¯</ci><ci id="S2.p2.1.m1.1.1.2.cmml" xref="S2.p2.1.m1.1.1.2">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p2.1.m1.1c">\bar{\theta}</annotation></semantics></math> (e.g., 0.5). We will describe our schedule in Section <a href="#S4.SS2" title="4.2 A Progressive Layer Dropping Schedule ‣ 4 Our Approach: Progressive Layer Dropping ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.2</span></a>.</p>
</div>
<figure id="S2.F3" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S2.F3.1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:138.8pt;"><img src="/html/2010.13369/assets/figs/stability-gradient-norm.png" id="S2.F3.1.g1" class="ltx_graphics ltx_img_landscape" width="178" height="140" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>The norm of the gradient with respect to the weights, with PostLN and PreLN.</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S2.F3.2" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:138.8pt;"><img src="/html/2010.13369/assets/figs/grad-norm-preserving.png" id="S2.F3.2.g1" class="ltx_graphics ltx_img_landscape" width="178" height="140" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>The norm preserving ratio with respect to the inputs, with PostLN and PreLN.</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S2.F3.3" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:138.8pt;"><img src="/html/2010.13369/assets/figs/leisioning-analysis.png" id="S2.F3.3.g1" class="ltx_graphics ltx_img_landscape" width="180" height="142" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Lesioning analysis with PostLN and PreLN.</figcaption>
</figure>
</div>
</div>
</figure>
<div id="S2.p3" class="ltx_para">
<p id="S2.p3.2" class="ltx_p">Despite the FLOPS reduction, directly training models like BERT with a smaller depth incurs a significant loss in accuracy even with knowledge distillation <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib24" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">24</span></a>, <a href="#bib.bib25" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">25</span></a>]</cite>. Prior work <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib31" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">31</span></a>]</cite> proposes to accelerate pre-training by first training a 3-layer BERT model and then growing the network depth to 6-layer and subsequently 12-layer.
However, the number of steps required at each depth before the network growth is not known a prior, making applying this approach challenging in practice. On the other hand, stochastic depth has been successfully demonstrated to train deep models with reduced expected depth <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib26" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">26</span></a>, <a href="#bib.bib32" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">32</span></a>]</cite>. However, we observe that directly pre-training BERT with randomly dropping <math id="S2.p3.1.m1.1" class="ltx_Math" alttext="f_{ATTN}" display="inline"><semantics id="S2.p3.1.m1.1a"><msub id="S2.p3.1.m1.1.1" xref="S2.p3.1.m1.1.1.cmml"><mi id="S2.p3.1.m1.1.1.2" xref="S2.p3.1.m1.1.1.2.cmml">f</mi><mrow id="S2.p3.1.m1.1.1.3" xref="S2.p3.1.m1.1.1.3.cmml"><mi id="S2.p3.1.m1.1.1.3.2" xref="S2.p3.1.m1.1.1.3.2.cmml">A</mi><mo lspace="0em" rspace="0em" id="S2.p3.1.m1.1.1.3.1" xref="S2.p3.1.m1.1.1.3.1.cmml">​</mo><mi id="S2.p3.1.m1.1.1.3.3" xref="S2.p3.1.m1.1.1.3.3.cmml">T</mi><mo lspace="0em" rspace="0em" id="S2.p3.1.m1.1.1.3.1a" xref="S2.p3.1.m1.1.1.3.1.cmml">​</mo><mi id="S2.p3.1.m1.1.1.3.4" xref="S2.p3.1.m1.1.1.3.4.cmml">T</mi><mo lspace="0em" rspace="0em" id="S2.p3.1.m1.1.1.3.1b" xref="S2.p3.1.m1.1.1.3.1.cmml">​</mo><mi id="S2.p3.1.m1.1.1.3.5" xref="S2.p3.1.m1.1.1.3.5.cmml">N</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.p3.1.m1.1b"><apply id="S2.p3.1.m1.1.1.cmml" xref="S2.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S2.p3.1.m1.1.1.1.cmml" xref="S2.p3.1.m1.1.1">subscript</csymbol><ci id="S2.p3.1.m1.1.1.2.cmml" xref="S2.p3.1.m1.1.1.2">𝑓</ci><apply id="S2.p3.1.m1.1.1.3.cmml" xref="S2.p3.1.m1.1.1.3"><times id="S2.p3.1.m1.1.1.3.1.cmml" xref="S2.p3.1.m1.1.1.3.1"></times><ci id="S2.p3.1.m1.1.1.3.2.cmml" xref="S2.p3.1.m1.1.1.3.2">𝐴</ci><ci id="S2.p3.1.m1.1.1.3.3.cmml" xref="S2.p3.1.m1.1.1.3.3">𝑇</ci><ci id="S2.p3.1.m1.1.1.3.4.cmml" xref="S2.p3.1.m1.1.1.3.4">𝑇</ci><ci id="S2.p3.1.m1.1.1.3.5.cmml" xref="S2.p3.1.m1.1.1.3.5">𝑁</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.1.m1.1c">f_{ATTN}</annotation></semantics></math> and <math id="S2.p3.2.m2.1" class="ltx_Math" alttext="f_{FFN}" display="inline"><semantics id="S2.p3.2.m2.1a"><msub id="S2.p3.2.m2.1.1" xref="S2.p3.2.m2.1.1.cmml"><mi id="S2.p3.2.m2.1.1.2" xref="S2.p3.2.m2.1.1.2.cmml">f</mi><mrow id="S2.p3.2.m2.1.1.3" xref="S2.p3.2.m2.1.1.3.cmml"><mi id="S2.p3.2.m2.1.1.3.2" xref="S2.p3.2.m2.1.1.3.2.cmml">F</mi><mo lspace="0em" rspace="0em" id="S2.p3.2.m2.1.1.3.1" xref="S2.p3.2.m2.1.1.3.1.cmml">​</mo><mi id="S2.p3.2.m2.1.1.3.3" xref="S2.p3.2.m2.1.1.3.3.cmml">F</mi><mo lspace="0em" rspace="0em" id="S2.p3.2.m2.1.1.3.1a" xref="S2.p3.2.m2.1.1.3.1.cmml">​</mo><mi id="S2.p3.2.m2.1.1.3.4" xref="S2.p3.2.m2.1.1.3.4.cmml">N</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.p3.2.m2.1b"><apply id="S2.p3.2.m2.1.1.cmml" xref="S2.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S2.p3.2.m2.1.1.1.cmml" xref="S2.p3.2.m2.1.1">subscript</csymbol><ci id="S2.p3.2.m2.1.1.2.cmml" xref="S2.p3.2.m2.1.1.2">𝑓</ci><apply id="S2.p3.2.m2.1.1.3.cmml" xref="S2.p3.2.m2.1.1.3"><times id="S2.p3.2.m2.1.1.3.1.cmml" xref="S2.p3.2.m2.1.1.3.1"></times><ci id="S2.p3.2.m2.1.1.3.2.cmml" xref="S2.p3.2.m2.1.1.3.2">𝐹</ci><ci id="S2.p3.2.m2.1.1.3.3.cmml" xref="S2.p3.2.m2.1.1.3.3">𝐹</ci><ci id="S2.p3.2.m2.1.1.3.4.cmml" xref="S2.p3.2.m2.1.1.3.4">𝑁</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p3.2.m2.1c">f_{FFN}</annotation></semantics></math> converges to bad/suspicious local optima under the same hyperparameter setting. When increasing the learning rate, the training often diverges even by tuning the warmup ratio.
What causes the instability of BERT pre-training with layer drop?
</p>
</div>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Preliminary Analysis</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">This section presents several studies that guided the design of the approach introduced in Section <a href="#S4" title="4 Our Approach: Progressive Layer Dropping ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>. We used BERT trained on Bookcorpus and Wikipedia dataset from Devlin et. al. with standard settings as the baseline <span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>Appendix <a href="#A1" title="Appendix A Pre-training Hyperparameters ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a> provides detailed training hyperparameters.</span></span></span>. First, we carry out a comparison between BERT with PostLN and PreLN. Our goal is to measure how effective these two methods at stabilizing BERT training.
Our second analysis considers measuring the dynamics of BERT pre-training, including both spatial and temporal dimensions. Finally, we analyze the effect of the removal of the Transformer layers. This leads us to identify appealing choices for our target operating points.</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Training Stability: PostLN or PreLN?</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.5" class="ltx_p">We consider two variants of BERT, namely the PostLN and PreLN. The default BERT employs PostLN, with layer normalization applied after the addition in Transformer blocks. The PreLN changes the placement of the location of <math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="f_{LN}" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><msub id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml"><mi id="S3.SS1.p1.1.m1.1.1.2" xref="S3.SS1.p1.1.m1.1.1.2.cmml">f</mi><mrow id="S3.SS1.p1.1.m1.1.1.3" xref="S3.SS1.p1.1.m1.1.1.3.cmml"><mi id="S3.SS1.p1.1.m1.1.1.3.2" xref="S3.SS1.p1.1.m1.1.1.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.1.m1.1.1.3.1" xref="S3.SS1.p1.1.m1.1.1.3.1.cmml">​</mo><mi id="S3.SS1.p1.1.m1.1.1.3.3" xref="S3.SS1.p1.1.m1.1.1.3.3.cmml">N</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><apply id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.1.m1.1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">subscript</csymbol><ci id="S3.SS1.p1.1.m1.1.1.2.cmml" xref="S3.SS1.p1.1.m1.1.1.2">𝑓</ci><apply id="S3.SS1.p1.1.m1.1.1.3.cmml" xref="S3.SS1.p1.1.m1.1.1.3"><times id="S3.SS1.p1.1.m1.1.1.3.1.cmml" xref="S3.SS1.p1.1.m1.1.1.3.1"></times><ci id="S3.SS1.p1.1.m1.1.1.3.2.cmml" xref="S3.SS1.p1.1.m1.1.1.3.2">𝐿</ci><ci id="S3.SS1.p1.1.m1.1.1.3.3.cmml" xref="S3.SS1.p1.1.m1.1.1.3.3">𝑁</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">f_{LN}</annotation></semantics></math> by placing it only on the input stream of the sublayers so that
<math id="S3.SS1.p1.2.m2.1" class="ltx_Math" alttext="{h}_{i}=x_{i}+f_{S-ATTN}(f_{LN}(x_{i}))" display="inline"><semantics id="S3.SS1.p1.2.m2.1a"><mrow id="S3.SS1.p1.2.m2.1.1" xref="S3.SS1.p1.2.m2.1.1.cmml"><msub id="S3.SS1.p1.2.m2.1.1.3" xref="S3.SS1.p1.2.m2.1.1.3.cmml"><mi id="S3.SS1.p1.2.m2.1.1.3.2" xref="S3.SS1.p1.2.m2.1.1.3.2.cmml">h</mi><mi id="S3.SS1.p1.2.m2.1.1.3.3" xref="S3.SS1.p1.2.m2.1.1.3.3.cmml">i</mi></msub><mo id="S3.SS1.p1.2.m2.1.1.2" xref="S3.SS1.p1.2.m2.1.1.2.cmml">=</mo><mrow id="S3.SS1.p1.2.m2.1.1.1" xref="S3.SS1.p1.2.m2.1.1.1.cmml"><msub id="S3.SS1.p1.2.m2.1.1.1.3" xref="S3.SS1.p1.2.m2.1.1.1.3.cmml"><mi id="S3.SS1.p1.2.m2.1.1.1.3.2" xref="S3.SS1.p1.2.m2.1.1.1.3.2.cmml">x</mi><mi id="S3.SS1.p1.2.m2.1.1.1.3.3" xref="S3.SS1.p1.2.m2.1.1.1.3.3.cmml">i</mi></msub><mo id="S3.SS1.p1.2.m2.1.1.1.2" xref="S3.SS1.p1.2.m2.1.1.1.2.cmml">+</mo><mrow id="S3.SS1.p1.2.m2.1.1.1.1" xref="S3.SS1.p1.2.m2.1.1.1.1.cmml"><msub id="S3.SS1.p1.2.m2.1.1.1.1.3" xref="S3.SS1.p1.2.m2.1.1.1.1.3.cmml"><mi id="S3.SS1.p1.2.m2.1.1.1.1.3.2" xref="S3.SS1.p1.2.m2.1.1.1.1.3.2.cmml">f</mi><mrow id="S3.SS1.p1.2.m2.1.1.1.1.3.3" xref="S3.SS1.p1.2.m2.1.1.1.1.3.3.cmml"><mi id="S3.SS1.p1.2.m2.1.1.1.1.3.3.2" xref="S3.SS1.p1.2.m2.1.1.1.1.3.3.2.cmml">S</mi><mo id="S3.SS1.p1.2.m2.1.1.1.1.3.3.1" xref="S3.SS1.p1.2.m2.1.1.1.1.3.3.1.cmml">−</mo><mrow id="S3.SS1.p1.2.m2.1.1.1.1.3.3.3" xref="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.cmml"><mi id="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.2" xref="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.2.cmml">A</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.1" xref="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.1.cmml">​</mo><mi id="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.3" xref="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.3.cmml">T</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.1a" xref="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.1.cmml">​</mo><mi id="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.4" xref="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.4.cmml">T</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.1b" xref="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.1.cmml">​</mo><mi id="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.5" xref="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.5.cmml">N</mi></mrow></mrow></msub><mo lspace="0em" rspace="0em" id="S3.SS1.p1.2.m2.1.1.1.1.2" xref="S3.SS1.p1.2.m2.1.1.1.1.2.cmml">​</mo><mrow id="S3.SS1.p1.2.m2.1.1.1.1.1.1" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p1.2.m2.1.1.1.1.1.1.2" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.cmml"><msub id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.cmml"><mi id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.2" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.2.cmml">f</mi><mrow id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.3" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.3.cmml"><mi id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.3.2" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.3.1" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.3.3" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.3.3.cmml">N</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.2" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.1" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.1.2" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.1.1" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.1.1.2" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.1.1.3" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.1.3" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S3.SS1.p1.2.m2.1.1.1.1.1.1.3" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.2.m2.1b"><apply id="S3.SS1.p1.2.m2.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1"><eq id="S3.SS1.p1.2.m2.1.1.2.cmml" xref="S3.SS1.p1.2.m2.1.1.2"></eq><apply id="S3.SS1.p1.2.m2.1.1.3.cmml" xref="S3.SS1.p1.2.m2.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p1.2.m2.1.1.3.1.cmml" xref="S3.SS1.p1.2.m2.1.1.3">subscript</csymbol><ci id="S3.SS1.p1.2.m2.1.1.3.2.cmml" xref="S3.SS1.p1.2.m2.1.1.3.2">ℎ</ci><ci id="S3.SS1.p1.2.m2.1.1.3.3.cmml" xref="S3.SS1.p1.2.m2.1.1.3.3">𝑖</ci></apply><apply id="S3.SS1.p1.2.m2.1.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1.1"><plus id="S3.SS1.p1.2.m2.1.1.1.2.cmml" xref="S3.SS1.p1.2.m2.1.1.1.2"></plus><apply id="S3.SS1.p1.2.m2.1.1.1.3.cmml" xref="S3.SS1.p1.2.m2.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p1.2.m2.1.1.1.3.1.cmml" xref="S3.SS1.p1.2.m2.1.1.1.3">subscript</csymbol><ci id="S3.SS1.p1.2.m2.1.1.1.3.2.cmml" xref="S3.SS1.p1.2.m2.1.1.1.3.2">𝑥</ci><ci id="S3.SS1.p1.2.m2.1.1.1.3.3.cmml" xref="S3.SS1.p1.2.m2.1.1.1.3.3">𝑖</ci></apply><apply id="S3.SS1.p1.2.m2.1.1.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1"><times id="S3.SS1.p1.2.m2.1.1.1.1.2.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.2"></times><apply id="S3.SS1.p1.2.m2.1.1.1.1.3.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p1.2.m2.1.1.1.1.3.1.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.3">subscript</csymbol><ci id="S3.SS1.p1.2.m2.1.1.1.1.3.2.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.3.2">𝑓</ci><apply id="S3.SS1.p1.2.m2.1.1.1.1.3.3.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.3.3"><minus id="S3.SS1.p1.2.m2.1.1.1.1.3.3.1.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.3.3.1"></minus><ci id="S3.SS1.p1.2.m2.1.1.1.1.3.3.2.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.3.3.2">𝑆</ci><apply id="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.3.3.3"><times id="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.1.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.1"></times><ci id="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.2.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.2">𝐴</ci><ci id="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.3.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.3">𝑇</ci><ci id="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.4.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.4">𝑇</ci><ci id="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.5.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.3.3.3.5">𝑁</ci></apply></apply></apply><apply id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1"><times id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.2"></times><apply id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.1.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.2.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.2">𝑓</ci><apply id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.3.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.3"><times id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.3.1.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.3.1"></times><ci id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.3.2.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.3.2">𝐿</ci><ci id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.3.3.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.3.3">𝑁</ci></apply></apply><apply id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.2.m2.1c">{h}_{i}=x_{i}+f_{S-ATTN}(f_{LN}(x_{i}))</annotation></semantics></math> and then <math id="S3.SS1.p1.3.m3.1" class="ltx_Math" alttext="x_{i+1}=h_{i}+f_{FFN}(f_{LN}(h_{i}))" display="inline"><semantics id="S3.SS1.p1.3.m3.1a"><mrow id="S3.SS1.p1.3.m3.1.1" xref="S3.SS1.p1.3.m3.1.1.cmml"><msub id="S3.SS1.p1.3.m3.1.1.3" xref="S3.SS1.p1.3.m3.1.1.3.cmml"><mi id="S3.SS1.p1.3.m3.1.1.3.2" xref="S3.SS1.p1.3.m3.1.1.3.2.cmml">x</mi><mrow id="S3.SS1.p1.3.m3.1.1.3.3" xref="S3.SS1.p1.3.m3.1.1.3.3.cmml"><mi id="S3.SS1.p1.3.m3.1.1.3.3.2" xref="S3.SS1.p1.3.m3.1.1.3.3.2.cmml">i</mi><mo id="S3.SS1.p1.3.m3.1.1.3.3.1" xref="S3.SS1.p1.3.m3.1.1.3.3.1.cmml">+</mo><mn id="S3.SS1.p1.3.m3.1.1.3.3.3" xref="S3.SS1.p1.3.m3.1.1.3.3.3.cmml">1</mn></mrow></msub><mo id="S3.SS1.p1.3.m3.1.1.2" xref="S3.SS1.p1.3.m3.1.1.2.cmml">=</mo><mrow id="S3.SS1.p1.3.m3.1.1.1" xref="S3.SS1.p1.3.m3.1.1.1.cmml"><msub id="S3.SS1.p1.3.m3.1.1.1.3" xref="S3.SS1.p1.3.m3.1.1.1.3.cmml"><mi id="S3.SS1.p1.3.m3.1.1.1.3.2" xref="S3.SS1.p1.3.m3.1.1.1.3.2.cmml">h</mi><mi id="S3.SS1.p1.3.m3.1.1.1.3.3" xref="S3.SS1.p1.3.m3.1.1.1.3.3.cmml">i</mi></msub><mo id="S3.SS1.p1.3.m3.1.1.1.2" xref="S3.SS1.p1.3.m3.1.1.1.2.cmml">+</mo><mrow id="S3.SS1.p1.3.m3.1.1.1.1" xref="S3.SS1.p1.3.m3.1.1.1.1.cmml"><msub id="S3.SS1.p1.3.m3.1.1.1.1.3" xref="S3.SS1.p1.3.m3.1.1.1.1.3.cmml"><mi id="S3.SS1.p1.3.m3.1.1.1.1.3.2" xref="S3.SS1.p1.3.m3.1.1.1.1.3.2.cmml">f</mi><mrow id="S3.SS1.p1.3.m3.1.1.1.1.3.3" xref="S3.SS1.p1.3.m3.1.1.1.1.3.3.cmml"><mi id="S3.SS1.p1.3.m3.1.1.1.1.3.3.2" xref="S3.SS1.p1.3.m3.1.1.1.1.3.3.2.cmml">F</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.3.m3.1.1.1.1.3.3.1" xref="S3.SS1.p1.3.m3.1.1.1.1.3.3.1.cmml">​</mo><mi id="S3.SS1.p1.3.m3.1.1.1.1.3.3.3" xref="S3.SS1.p1.3.m3.1.1.1.1.3.3.3.cmml">F</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.3.m3.1.1.1.1.3.3.1a" xref="S3.SS1.p1.3.m3.1.1.1.1.3.3.1.cmml">​</mo><mi id="S3.SS1.p1.3.m3.1.1.1.1.3.3.4" xref="S3.SS1.p1.3.m3.1.1.1.1.3.3.4.cmml">N</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S3.SS1.p1.3.m3.1.1.1.1.2" xref="S3.SS1.p1.3.m3.1.1.1.1.2.cmml">​</mo><mrow id="S3.SS1.p1.3.m3.1.1.1.1.1.1" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p1.3.m3.1.1.1.1.1.1.2" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.cmml"><msub id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.cmml"><mi id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.2" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.2.cmml">f</mi><mrow id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.3" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.3.cmml"><mi id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.3.2" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.3.1" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.3.3" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.3.3.cmml">N</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.2" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.1" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.1.2" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.1.1" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.1.1.2" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.1.1.2.cmml">h</mi><mi id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.1.1.3" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.1.3" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S3.SS1.p1.3.m3.1.1.1.1.1.1.3" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.3.m3.1b"><apply id="S3.SS1.p1.3.m3.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1"><eq id="S3.SS1.p1.3.m3.1.1.2.cmml" xref="S3.SS1.p1.3.m3.1.1.2"></eq><apply id="S3.SS1.p1.3.m3.1.1.3.cmml" xref="S3.SS1.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p1.3.m3.1.1.3.1.cmml" xref="S3.SS1.p1.3.m3.1.1.3">subscript</csymbol><ci id="S3.SS1.p1.3.m3.1.1.3.2.cmml" xref="S3.SS1.p1.3.m3.1.1.3.2">𝑥</ci><apply id="S3.SS1.p1.3.m3.1.1.3.3.cmml" xref="S3.SS1.p1.3.m3.1.1.3.3"><plus id="S3.SS1.p1.3.m3.1.1.3.3.1.cmml" xref="S3.SS1.p1.3.m3.1.1.3.3.1"></plus><ci id="S3.SS1.p1.3.m3.1.1.3.3.2.cmml" xref="S3.SS1.p1.3.m3.1.1.3.3.2">𝑖</ci><cn type="integer" id="S3.SS1.p1.3.m3.1.1.3.3.3.cmml" xref="S3.SS1.p1.3.m3.1.1.3.3.3">1</cn></apply></apply><apply id="S3.SS1.p1.3.m3.1.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1"><plus id="S3.SS1.p1.3.m3.1.1.1.2.cmml" xref="S3.SS1.p1.3.m3.1.1.1.2"></plus><apply id="S3.SS1.p1.3.m3.1.1.1.3.cmml" xref="S3.SS1.p1.3.m3.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p1.3.m3.1.1.1.3.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1.3">subscript</csymbol><ci id="S3.SS1.p1.3.m3.1.1.1.3.2.cmml" xref="S3.SS1.p1.3.m3.1.1.1.3.2">ℎ</ci><ci id="S3.SS1.p1.3.m3.1.1.1.3.3.cmml" xref="S3.SS1.p1.3.m3.1.1.1.3.3">𝑖</ci></apply><apply id="S3.SS1.p1.3.m3.1.1.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1"><times id="S3.SS1.p1.3.m3.1.1.1.1.2.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.2"></times><apply id="S3.SS1.p1.3.m3.1.1.1.1.3.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p1.3.m3.1.1.1.1.3.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.3">subscript</csymbol><ci id="S3.SS1.p1.3.m3.1.1.1.1.3.2.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.3.2">𝑓</ci><apply id="S3.SS1.p1.3.m3.1.1.1.1.3.3.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.3.3"><times id="S3.SS1.p1.3.m3.1.1.1.1.3.3.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.3.3.1"></times><ci id="S3.SS1.p1.3.m3.1.1.1.1.3.3.2.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.3.3.2">𝐹</ci><ci id="S3.SS1.p1.3.m3.1.1.1.1.3.3.3.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.3.3.3">𝐹</ci><ci id="S3.SS1.p1.3.m3.1.1.1.1.3.3.4.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.3.3.4">𝑁</ci></apply></apply><apply id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1"><times id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.2"></times><apply id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.2.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.2">𝑓</ci><apply id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.3.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.3"><times id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.3.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.3.1"></times><ci id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.3.2.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.3.2">𝐿</ci><ci id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.3.3.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.3.3.3">𝑁</ci></apply></apply><apply id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.1.1.2">ℎ</ci><ci id="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.3.m3.1c">x_{i+1}=h_{i}+f_{FFN}(f_{LN}(h_{i}))</annotation></semantics></math>, which is a modification described by several recent works to establish identity mapping for neural machine translation <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib33" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">33</span></a>, <a href="#bib.bib34" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">34</span></a>, <a href="#bib.bib35" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">35</span></a>, <a href="#bib.bib36" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">36</span></a>, <a href="#bib.bib37" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">37</span></a>]</cite>. Fig. <a href="#S2.F3" title="Figure 3 ‣ 2 Background and Related Work ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> reports the norm of gradients with respect to weights in backward propagation for both methods, varying the depth <math id="S3.SS1.p1.4.m4.1" class="ltx_Math" alttext="L" display="inline"><semantics id="S3.SS1.p1.4.m4.1a"><mi id="S3.SS1.p1.4.m4.1.1" xref="S3.SS1.p1.4.m4.1.1.cmml">L</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.4.m4.1b"><ci id="S3.SS1.p1.4.m4.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1">𝐿</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.4.m4.1c">L</annotation></semantics></math> (e.g., 12, 24, 48). The plot shows that while PostLN suffers from unbalanced gradients (e.g., vanishing gradients as the layer ID decreases), PreLN eliminates the unbalanced gradient problem (solid green lines) and the gradient norm stays almost same for any layer. Furthermore, Fig. <a href="#S2.F3" title="Figure 3 ‣ 2 Background and Related Work ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> shows that for PreLN the gradients with respect to input <math id="S3.SS1.p1.5.m5.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="S3.SS1.p1.5.m5.1a"><msub id="S3.SS1.p1.5.m5.1.1" xref="S3.SS1.p1.5.m5.1.1.cmml"><mi id="S3.SS1.p1.5.m5.1.1.2" xref="S3.SS1.p1.5.m5.1.1.2.cmml">x</mi><mi id="S3.SS1.p1.5.m5.1.1.3" xref="S3.SS1.p1.5.m5.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.5.m5.1b"><apply id="S3.SS1.p1.5.m5.1.1.cmml" xref="S3.SS1.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.5.m5.1.1.1.cmml" xref="S3.SS1.p1.5.m5.1.1">subscript</csymbol><ci id="S3.SS1.p1.5.m5.1.1.2.cmml" xref="S3.SS1.p1.5.m5.1.1.2">𝑥</ci><ci id="S3.SS1.p1.5.m5.1.1.3.cmml" xref="S3.SS1.p1.5.m5.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.5.m5.1c">x_{i}</annotation></semantics></math> have very
similar magnitudes (norm preserving ratio close to 1) at different layers, which is consistent with prior findings that a neural model should preserve the gradient norm between layers so as to have well-conditioning and faster convergence <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib38" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">38</span></a>, <a href="#bib.bib39" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">39</span></a>]</cite>.
Indeed, we find that PostLN is more sensitive to the choice of hyperparameters, and training often diverges with more aggressive learning rates (more results in Section <a href="#S5" title="5 Evaluation ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>), whereas PreLN avoids vanishing gradients and leads to more stable optimization. We also provide preliminary theoretical results in Appendix <a href="#A2" title="Appendix B Establishing Identity Mapping with PreLN ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a> on why PreLN is beneficial.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Corroboration of Training Dynamics</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">Hereafter we investigate the representation <math id="S3.SS2.p1.1.m1.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="S3.SS2.p1.1.m1.1a"><msub id="S3.SS2.p1.1.m1.1.1" xref="S3.SS2.p1.1.m1.1.1.cmml"><mi id="S3.SS2.p1.1.m1.1.1.2" xref="S3.SS2.p1.1.m1.1.1.2.cmml">x</mi><mi id="S3.SS2.p1.1.m1.1.1.3" xref="S3.SS2.p1.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.1.m1.1b"><apply id="S3.SS2.p1.1.m1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.1.m1.1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1">subscript</csymbol><ci id="S3.SS2.p1.1.m1.1.1.2.cmml" xref="S3.SS2.p1.1.m1.1.1.2">𝑥</ci><ci id="S3.SS2.p1.1.m1.1.1.3.cmml" xref="S3.SS2.p1.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.1.m1.1c">x_{i}</annotation></semantics></math> learned at different phases of BERT pre-training and at different layers. Fig. <a href="#S3.F4" title="Figure 4 ‣ 3.2 Corroboration of Training Dynamics ‣ 3 Preliminary Analysis ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> shows the L2 norm distances and cosine similarity, which measures the angle between two vectors and ignores their norms, between the input and output embeddings, with PostLN and PreLN, respectively. We draw several observations.</p>
</div>
<div id="S3.SS2.p2" class="ltx_para">
<p id="S3.SS2.p2.1" class="ltx_p">First, the dissimilarity (Fig. <a href="#S3.F4.sf1" title="In Figure 4 ‣ 3.2 Corroboration of Training Dynamics ‣ 3 Preliminary Analysis ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4a</span></a> and Fig. <a href="#S3.F4.sf2" title="In Figure 4 ‣ 3.2 Corroboration of Training Dynamics ‣ 3 Preliminary Analysis ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4b</span></a>) stays high for both PostLN and PreLN at those higher layers in the beginning, and the L2 and cosine similarity seems to be less correlated (e.g., step = 300).
This is presumably because, at the beginning of the training, the model weights are randomly initialized, and the network is still actively adjusting weights to derive richer features from input data. Since the model is still positively self-organizing on the network parameters toward their optimal configuration,
dropping layers at this stage is not an interesting strategy, because it can create inputs with large noise and disturb the positive co-adaption process.
</p>
</div>
<div id="S3.SS2.p3" class="ltx_para">
<p id="S3.SS2.p3.1" class="ltx_p">Second, as the training proceeds (Fig. <a href="#S3.F4.sf3" title="In Figure 4 ‣ 3.2 Corroboration of Training Dynamics ‣ 3 Preliminary Analysis ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4c</span></a> and Fig. <a href="#S3.F4.sf4" title="In Figure 4 ‣ 3.2 Corroboration of Training Dynamics ‣ 3 Preliminary Analysis ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4d</span></a>), although the dissimilarity remains relatively high and bumpy for PostLN, the similarity from PreLN starts to increase over successive layers, indicating that while PostLN
is still trying to produce new representations that are very different across layers, the dissimilarity from PreLN is getting close to zero for upper layers, indicating that the upper layers are getting similar estimations. This can be viewed as doing an unrolled iterative refinement <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib40" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">40</span></a>]</cite>, where a group of successive layers iteratively refine their estimates of the same representations instead of computing an entirely new representation.
Although the viewpoint was originally proposed to explain ResNet, we demonstrate that it is also true for language modeling and Transformer-based networks. Appendix <a href="#A3" title="Appendix C PreLN From the View of Unrolled Iterative Refinement ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a> provides additional analysis on how PreLN provides extra preservation of feature identity through unrolled iterative refinement.</p>
</div>
<figure id="S3.F4" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S3.F4.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/similarity/similarity-l2-norm-step300.png" id="S3.F4.sf1.g1" class="ltx_graphics ltx_img_landscape" width="139" height="110" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(a) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S3.F4.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/similarity/similarity-cosine-step300.png" id="S3.F4.sf2.g1" class="ltx_graphics ltx_img_landscape" width="139" height="110" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(b) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S3.F4.sf3" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/similarity/similarity-l2-norm-step2000.png" id="S3.F4.sf3.g1" class="ltx_graphics ltx_img_landscape" width="139" height="110" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(c) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S3.F4.sf4" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/similarity/similarity-cosine-step2000.png" id="S3.F4.sf4.g1" class="ltx_graphics ltx_img_landscape" width="139" height="110" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(d) </span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>The L2 distance and cosine similarity of the input and output embeddings for BERT with PostLN and PreLN, at different layers and different steps. We plot the inverse of cosine similarity (arccosine) in degrees, so that for both L2 and arccosine, the lower the more similar.</figcaption>
</figure>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Effect of Lesioning</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">We randomly drop layers with a keep ratio <math id="S3.SS3.p1.1.m1.1" class="ltx_Math" alttext="\theta=0.5" display="inline"><semantics id="S3.SS3.p1.1.m1.1a"><mrow id="S3.SS3.p1.1.m1.1.1" xref="S3.SS3.p1.1.m1.1.1.cmml"><mi id="S3.SS3.p1.1.m1.1.1.2" xref="S3.SS3.p1.1.m1.1.1.2.cmml">θ</mi><mo id="S3.SS3.p1.1.m1.1.1.1" xref="S3.SS3.p1.1.m1.1.1.1.cmml">=</mo><mn id="S3.SS3.p1.1.m1.1.1.3" xref="S3.SS3.p1.1.m1.1.1.3.cmml">0.5</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.1.m1.1b"><apply id="S3.SS3.p1.1.m1.1.1.cmml" xref="S3.SS3.p1.1.m1.1.1"><eq id="S3.SS3.p1.1.m1.1.1.1.cmml" xref="S3.SS3.p1.1.m1.1.1.1"></eq><ci id="S3.SS3.p1.1.m1.1.1.2.cmml" xref="S3.SS3.p1.1.m1.1.1.2">𝜃</ci><cn type="float" id="S3.SS3.p1.1.m1.1.1.3.cmml" xref="S3.SS3.p1.1.m1.1.1.3">0.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.1.m1.1c">\theta=0.5</annotation></semantics></math> to test if dropping layers would break the training because dropping any layer changes the input distribution of all subsequent layers. The results are shown in Fig. <a href="#S2.F3" title="Figure 3 ‣ 2 Background and Related Work ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. As shown, removing layers in PostLN significantly reduces performance. Moreover, when increasing the learning rate, it results in diverged training.
In contrast, this is not the case for PreLN. Given that later layers in PreLN tend to refine an estimate of the representation, the model with PreLN has less dependence on the downsampling individual layers. As a result, removing Transformer layers with PreLN has a modest impact on performance (slightly worse validation loss at the same number of training samples). However, the change is much smaller than with PostLN. It further indicates that if we remove layers, especially those higher ones, it should have only a mild effect on the final result because doing so does not change the overall estimation the next layer receives, only its quality. The following layers can still perform mostly the same operation, even with some relatively little noisy input. Furthermore, as Fig. <a href="#S3.F4" title="Figure 4 ‣ 3.2 Corroboration of Training Dynamics ‣ 3 Preliminary Analysis ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> indicates, since the lower layers remain to have a relatively high dissimilarity (deriving new features), they should be less frequently dropped. Overall, these results show that, to some extent, the structure of a Transformer network with PreLN can be changed at runtime without significantly affecting performance.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Our Approach: Progressive Layer Dropping</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">This section describes our approach, namely progressive layer dropping (PLD), to accelerate the pre-training of Transformer-based models. We first present the Switchable-Transformer blocks, a new unit that allows us to train models like BERT with layer drop and improved stability. Then we introduce the progressive layer drop procedure.</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Switchable-Transformer Blocks</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">In this work, we propose a novel transformer unit, which we call "Switchable-Transformer " (ST) block. Compared with the original Transformer block (Fig. <a href="#S4.F5.sf1" title="In Figure 6 ‣ Identity mapping reordering. ‣ 4.1 Switchable-Transformer Blocks ‣ 4 Our Approach: Progressive Layer Dropping ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5a</span></a>), it contains two changes.</p>
</div>
<section id="S4.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Identity mapping reordering.</h4>

<div id="S4.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS1.SSS0.Px1.p1.1" class="ltx_p">The first change is to establish identity mapping within a transformer block by placing the layer normalization only on the input stream of the sublayers (i.e., use PreLN to replace PostLN) (Fig. <a href="#S4.F5.sf2" title="In Figure 6 ‣ Identity mapping reordering. ‣ 4.1 Switchable-Transformer Blocks ‣ 4 Our Approach: Progressive Layer Dropping ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5b</span></a>) for the stability reason described in Section <a href="#S3.SS1" title="3.1 Training Stability: PostLN or PreLN? ‣ 3 Preliminary Analysis ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a>.</p>
</div>
<figure id="S4.F6" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F6.fig1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:260.2pt;">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F5.sf1" class="ltx_figure ltx_figure_panel"><img src="/html/2010.13369/assets/figs/architecture/gated-transformer-v2a.png" id="S4.F5.sf1.g1" class="ltx_graphics ltx_img_portrait" width="154" height="313" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span>Original</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F5.sf2" class="ltx_figure ltx_figure_panel"><img src="/html/2010.13369/assets/figs/architecture/gated-transformer-v2b.png" id="S4.F5.sf2.g1" class="ltx_graphics ltx_img_portrait" width="155" height="313" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span>Identity mapping reordering</figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S4.F5.sf3" class="ltx_figure ltx_figure_panel"><img src="/html/2010.13369/assets/figs/architecture/gated-transformer-v2c.png" id="S4.F5.sf3.g1" class="ltx_graphics ltx_img_portrait" width="197" height="308" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(c) </span>Switchable Transformer</figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>Transformer variants, showing a single layer block.</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F6.1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:147.4pt;"><img src="/html/2010.13369/assets/figs/expected-GFLOPS-curve-graph.png" id="S4.F6.1.g1" class="ltx_graphics ltx_img_landscape" width="186" height="146" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>FLOPS per training iteration normalized to the baseline.</figcaption>
</figure>
</div>
</div>
</figure>
</section>
<section id="S4.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Switchable gates.</h4>

<div id="S4.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS1.SSS0.Px2.p1.1" class="ltx_p">Next, we extend the architecture to include a gate for each sub-layer (Fig. <a href="#S4.F5.sf3" title="In Figure 6 ‣ Identity mapping reordering. ‣ 4.1 Switchable-Transformer Blocks ‣ 4 Our Approach: Progressive Layer Dropping ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5c</span></a>), which controls whether a sub-layer is disabled or not during training. In particular, for each mini-batch, the two gates for the two sublayers decide whether to remove their corresponding transformation functions and only keep the identify mapping connection, which is equivalent to applying a conditional gate function <math id="S4.SS1.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="G" display="inline"><semantics id="S4.SS1.SSS0.Px2.p1.1.m1.1a"><mi id="S4.SS1.SSS0.Px2.p1.1.m1.1.1" xref="S4.SS1.SSS0.Px2.p1.1.m1.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p1.1.m1.1b"><ci id="S4.SS1.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p1.1.m1.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p1.1.m1.1c">G</annotation></semantics></math> to each sub-layer as follows:</p>
</div>
<div id="S4.SS1.SSS0.Px2.p2" class="ltx_para">
<table id="S4.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E1.m1.46" class="ltx_Math" alttext="\begin{split}h_{i}&amp;=x_{i}+G_{i}\times f_{S-ATTN}(f_{LN}(x_{i}))\times\frac{1}{p_{i}}\\
x_{i+1}&amp;=h_{i}+G_{i}\times f_{FFN}(f_{LN}(h_{i}))\times\frac{1}{p_{i}}\\
\end{split}" display="block"><semantics id="S4.E1.m1.46a"><mtable columnspacing="0pt" displaystyle="true" rowspacing="0pt" id="S4.E1.m1.46.46.4" xref="S4.E1.m1.44.44.2.cmml"><mtr id="S4.E1.m1.46.46.4a" xref="S4.E1.m1.44.44.2.cmml"><mtd class="ltx_align_right" columnalign="right" id="S4.E1.m1.46.46.4b" xref="S4.E1.m1.44.44.2.cmml"><msub id="S4.E1.m1.2.2.2.2.2" xref="S4.E1.m1.44.44.2.cmml"><mi id="S4.E1.m1.1.1.1.1.1.1" xref="S4.E1.m1.1.1.1.1.1.1.cmml">h</mi><mi id="S4.E1.m1.2.2.2.2.2.2.1" xref="S4.E1.m1.2.2.2.2.2.2.1.cmml">i</mi></msub></mtd><mtd class="ltx_align_left" columnalign="left" id="S4.E1.m1.46.46.4c" xref="S4.E1.m1.44.44.2.cmml"><mrow id="S4.E1.m1.45.45.3.43.22.20" xref="S4.E1.m1.44.44.2.cmml"><mi id="S4.E1.m1.45.45.3.43.22.20.21" xref="S4.E1.m1.44.44.2a.cmml"></mi><mo id="S4.E1.m1.3.3.3.3.1.1" xref="S4.E1.m1.3.3.3.3.1.1.cmml">=</mo><mrow id="S4.E1.m1.45.45.3.43.22.20.20" xref="S4.E1.m1.44.44.2.cmml"><msub id="S4.E1.m1.45.45.3.43.22.20.20.2" xref="S4.E1.m1.44.44.2.cmml"><mi id="S4.E1.m1.4.4.4.4.2.2" xref="S4.E1.m1.4.4.4.4.2.2.cmml">x</mi><mi id="S4.E1.m1.5.5.5.5.3.3.1" xref="S4.E1.m1.5.5.5.5.3.3.1.cmml">i</mi></msub><mo id="S4.E1.m1.6.6.6.6.4.4" xref="S4.E1.m1.6.6.6.6.4.4.cmml">+</mo><mrow id="S4.E1.m1.45.45.3.43.22.20.20.1" xref="S4.E1.m1.44.44.2.cmml"><mrow id="S4.E1.m1.45.45.3.43.22.20.20.1.1" xref="S4.E1.m1.44.44.2.cmml"><mrow id="S4.E1.m1.45.45.3.43.22.20.20.1.1.3" xref="S4.E1.m1.44.44.2.cmml"><msub id="S4.E1.m1.45.45.3.43.22.20.20.1.1.3.1" xref="S4.E1.m1.44.44.2.cmml"><mi id="S4.E1.m1.7.7.7.7.5.5" xref="S4.E1.m1.7.7.7.7.5.5.cmml">G</mi><mi id="S4.E1.m1.8.8.8.8.6.6.1" xref="S4.E1.m1.8.8.8.8.6.6.1.cmml">i</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S4.E1.m1.9.9.9.9.7.7" xref="S4.E1.m1.9.9.9.9.7.7.cmml">×</mo><msub id="S4.E1.m1.45.45.3.43.22.20.20.1.1.3.2" xref="S4.E1.m1.44.44.2.cmml"><mi id="S4.E1.m1.10.10.10.10.8.8" xref="S4.E1.m1.10.10.10.10.8.8.cmml">f</mi><mrow id="S4.E1.m1.11.11.11.11.9.9.1" xref="S4.E1.m1.11.11.11.11.9.9.1.cmml"><mi id="S4.E1.m1.11.11.11.11.9.9.1.2" xref="S4.E1.m1.11.11.11.11.9.9.1.2.cmml">S</mi><mo id="S4.E1.m1.11.11.11.11.9.9.1.1" xref="S4.E1.m1.11.11.11.11.9.9.1.1.cmml">−</mo><mrow id="S4.E1.m1.11.11.11.11.9.9.1.3" xref="S4.E1.m1.11.11.11.11.9.9.1.3.cmml"><mi id="S4.E1.m1.11.11.11.11.9.9.1.3.2" xref="S4.E1.m1.11.11.11.11.9.9.1.3.2.cmml">A</mi><mo lspace="0em" rspace="0em" id="S4.E1.m1.11.11.11.11.9.9.1.3.1" xref="S4.E1.m1.11.11.11.11.9.9.1.3.1.cmml">​</mo><mi id="S4.E1.m1.11.11.11.11.9.9.1.3.3" xref="S4.E1.m1.11.11.11.11.9.9.1.3.3.cmml">T</mi><mo lspace="0em" rspace="0em" id="S4.E1.m1.11.11.11.11.9.9.1.3.1a" xref="S4.E1.m1.11.11.11.11.9.9.1.3.1.cmml">​</mo><mi id="S4.E1.m1.11.11.11.11.9.9.1.3.4" xref="S4.E1.m1.11.11.11.11.9.9.1.3.4.cmml">T</mi><mo lspace="0em" rspace="0em" id="S4.E1.m1.11.11.11.11.9.9.1.3.1b" xref="S4.E1.m1.11.11.11.11.9.9.1.3.1.cmml">​</mo><mi id="S4.E1.m1.11.11.11.11.9.9.1.3.5" xref="S4.E1.m1.11.11.11.11.9.9.1.3.5.cmml">N</mi></mrow></mrow></msub></mrow><mo lspace="0em" rspace="0em" id="S4.E1.m1.45.45.3.43.22.20.20.1.1.2" xref="S4.E1.m1.44.44.2a.cmml">​</mo><mrow id="S4.E1.m1.45.45.3.43.22.20.20.1.1.1.1" xref="S4.E1.m1.44.44.2.cmml"><mo stretchy="false" id="S4.E1.m1.12.12.12.12.10.10" xref="S4.E1.m1.44.44.2a.cmml">(</mo><mrow id="S4.E1.m1.45.45.3.43.22.20.20.1.1.1.1.1" xref="S4.E1.m1.44.44.2.cmml"><msub id="S4.E1.m1.45.45.3.43.22.20.20.1.1.1.1.1.3" xref="S4.E1.m1.44.44.2.cmml"><mi id="S4.E1.m1.13.13.13.13.11.11" xref="S4.E1.m1.13.13.13.13.11.11.cmml">f</mi><mrow id="S4.E1.m1.14.14.14.14.12.12.1" xref="S4.E1.m1.14.14.14.14.12.12.1.cmml"><mi id="S4.E1.m1.14.14.14.14.12.12.1.2" xref="S4.E1.m1.14.14.14.14.12.12.1.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S4.E1.m1.14.14.14.14.12.12.1.1" xref="S4.E1.m1.14.14.14.14.12.12.1.1.cmml">​</mo><mi id="S4.E1.m1.14.14.14.14.12.12.1.3" xref="S4.E1.m1.14.14.14.14.12.12.1.3.cmml">N</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S4.E1.m1.45.45.3.43.22.20.20.1.1.1.1.1.2" xref="S4.E1.m1.44.44.2a.cmml">​</mo><mrow id="S4.E1.m1.45.45.3.43.22.20.20.1.1.1.1.1.1.1" xref="S4.E1.m1.44.44.2.cmml"><mo stretchy="false" id="S4.E1.m1.15.15.15.15.13.13" xref="S4.E1.m1.44.44.2a.cmml">(</mo><msub id="S4.E1.m1.45.45.3.43.22.20.20.1.1.1.1.1.1.1.1" xref="S4.E1.m1.44.44.2.cmml"><mi id="S4.E1.m1.16.16.16.16.14.14" xref="S4.E1.m1.16.16.16.16.14.14.cmml">x</mi><mi id="S4.E1.m1.17.17.17.17.15.15.1" xref="S4.E1.m1.17.17.17.17.15.15.1.cmml">i</mi></msub><mo stretchy="false" id="S4.E1.m1.18.18.18.18.16.16" xref="S4.E1.m1.44.44.2a.cmml">)</mo></mrow></mrow><mo rspace="0.055em" stretchy="false" id="S4.E1.m1.19.19.19.19.17.17" xref="S4.E1.m1.44.44.2a.cmml">)</mo></mrow></mrow><mo rspace="0.222em" id="S4.E1.m1.20.20.20.20.18.18" xref="S4.E1.m1.20.20.20.20.18.18.cmml">×</mo><mfrac id="S4.E1.m1.21.21.21.21.19.19" xref="S4.E1.m1.21.21.21.21.19.19.cmml"><mn id="S4.E1.m1.21.21.21.21.19.19.2" xref="S4.E1.m1.21.21.21.21.19.19.2.cmml">1</mn><msub id="S4.E1.m1.21.21.21.21.19.19.3" xref="S4.E1.m1.21.21.21.21.19.19.3.cmml"><mi id="S4.E1.m1.21.21.21.21.19.19.3.2" xref="S4.E1.m1.21.21.21.21.19.19.3.2.cmml">p</mi><mi id="S4.E1.m1.21.21.21.21.19.19.3.3" xref="S4.E1.m1.21.21.21.21.19.19.3.3.cmml">i</mi></msub></mfrac></mrow></mrow></mrow></mtd></mtr><mtr id="S4.E1.m1.46.46.4d" xref="S4.E1.m1.44.44.2.cmml"><mtd class="ltx_align_right" columnalign="right" id="S4.E1.m1.46.46.4e" xref="S4.E1.m1.44.44.2.cmml"><msub id="S4.E1.m1.23.23.23.2.2" xref="S4.E1.m1.44.44.2.cmml"><mi id="S4.E1.m1.22.22.22.1.1.1" xref="S4.E1.m1.22.22.22.1.1.1.cmml">x</mi><mrow id="S4.E1.m1.23.23.23.2.2.2.1" xref="S4.E1.m1.23.23.23.2.2.2.1.cmml"><mi id="S4.E1.m1.23.23.23.2.2.2.1.2" xref="S4.E1.m1.23.23.23.2.2.2.1.2.cmml">i</mi><mo id="S4.E1.m1.23.23.23.2.2.2.1.1" xref="S4.E1.m1.23.23.23.2.2.2.1.1.cmml">+</mo><mn id="S4.E1.m1.23.23.23.2.2.2.1.3" xref="S4.E1.m1.23.23.23.2.2.2.1.3.cmml">1</mn></mrow></msub></mtd><mtd class="ltx_align_left" columnalign="left" id="S4.E1.m1.46.46.4f" xref="S4.E1.m1.44.44.2.cmml"><mrow id="S4.E1.m1.46.46.4.44.22.20" xref="S4.E1.m1.44.44.2.cmml"><mi id="S4.E1.m1.46.46.4.44.22.20.21" xref="S4.E1.m1.44.44.2a.cmml"></mi><mo id="S4.E1.m1.24.24.24.3.1.1" xref="S4.E1.m1.24.24.24.3.1.1.cmml">=</mo><mrow id="S4.E1.m1.46.46.4.44.22.20.20" xref="S4.E1.m1.44.44.2.cmml"><msub id="S4.E1.m1.46.46.4.44.22.20.20.2" xref="S4.E1.m1.44.44.2.cmml"><mi id="S4.E1.m1.25.25.25.4.2.2" xref="S4.E1.m1.25.25.25.4.2.2.cmml">h</mi><mi id="S4.E1.m1.26.26.26.5.3.3.1" xref="S4.E1.m1.26.26.26.5.3.3.1.cmml">i</mi></msub><mo id="S4.E1.m1.27.27.27.6.4.4" xref="S4.E1.m1.27.27.27.6.4.4.cmml">+</mo><mrow id="S4.E1.m1.46.46.4.44.22.20.20.1" xref="S4.E1.m1.44.44.2.cmml"><mrow id="S4.E1.m1.46.46.4.44.22.20.20.1.1" xref="S4.E1.m1.44.44.2.cmml"><mrow id="S4.E1.m1.46.46.4.44.22.20.20.1.1.3" xref="S4.E1.m1.44.44.2.cmml"><msub id="S4.E1.m1.46.46.4.44.22.20.20.1.1.3.1" xref="S4.E1.m1.44.44.2.cmml"><mi id="S4.E1.m1.28.28.28.7.5.5" xref="S4.E1.m1.28.28.28.7.5.5.cmml">G</mi><mi id="S4.E1.m1.29.29.29.8.6.6.1" xref="S4.E1.m1.29.29.29.8.6.6.1.cmml">i</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S4.E1.m1.30.30.30.9.7.7" xref="S4.E1.m1.30.30.30.9.7.7.cmml">×</mo><msub id="S4.E1.m1.46.46.4.44.22.20.20.1.1.3.2" xref="S4.E1.m1.44.44.2.cmml"><mi id="S4.E1.m1.31.31.31.10.8.8" xref="S4.E1.m1.31.31.31.10.8.8.cmml">f</mi><mrow id="S4.E1.m1.32.32.32.11.9.9.1" xref="S4.E1.m1.32.32.32.11.9.9.1.cmml"><mi id="S4.E1.m1.32.32.32.11.9.9.1.2" xref="S4.E1.m1.32.32.32.11.9.9.1.2.cmml">F</mi><mo lspace="0em" rspace="0em" id="S4.E1.m1.32.32.32.11.9.9.1.1" xref="S4.E1.m1.32.32.32.11.9.9.1.1.cmml">​</mo><mi id="S4.E1.m1.32.32.32.11.9.9.1.3" xref="S4.E1.m1.32.32.32.11.9.9.1.3.cmml">F</mi><mo lspace="0em" rspace="0em" id="S4.E1.m1.32.32.32.11.9.9.1.1a" xref="S4.E1.m1.32.32.32.11.9.9.1.1.cmml">​</mo><mi id="S4.E1.m1.32.32.32.11.9.9.1.4" xref="S4.E1.m1.32.32.32.11.9.9.1.4.cmml">N</mi></mrow></msub></mrow><mo lspace="0em" rspace="0em" id="S4.E1.m1.46.46.4.44.22.20.20.1.1.2" xref="S4.E1.m1.44.44.2a.cmml">​</mo><mrow id="S4.E1.m1.46.46.4.44.22.20.20.1.1.1.1" xref="S4.E1.m1.44.44.2.cmml"><mo stretchy="false" id="S4.E1.m1.33.33.33.12.10.10" xref="S4.E1.m1.44.44.2a.cmml">(</mo><mrow id="S4.E1.m1.46.46.4.44.22.20.20.1.1.1.1.1" xref="S4.E1.m1.44.44.2.cmml"><msub id="S4.E1.m1.46.46.4.44.22.20.20.1.1.1.1.1.3" xref="S4.E1.m1.44.44.2.cmml"><mi id="S4.E1.m1.34.34.34.13.11.11" xref="S4.E1.m1.34.34.34.13.11.11.cmml">f</mi><mrow id="S4.E1.m1.35.35.35.14.12.12.1" xref="S4.E1.m1.35.35.35.14.12.12.1.cmml"><mi id="S4.E1.m1.35.35.35.14.12.12.1.2" xref="S4.E1.m1.35.35.35.14.12.12.1.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S4.E1.m1.35.35.35.14.12.12.1.1" xref="S4.E1.m1.35.35.35.14.12.12.1.1.cmml">​</mo><mi id="S4.E1.m1.35.35.35.14.12.12.1.3" xref="S4.E1.m1.35.35.35.14.12.12.1.3.cmml">N</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S4.E1.m1.46.46.4.44.22.20.20.1.1.1.1.1.2" xref="S4.E1.m1.44.44.2a.cmml">​</mo><mrow id="S4.E1.m1.46.46.4.44.22.20.20.1.1.1.1.1.1.1" xref="S4.E1.m1.44.44.2.cmml"><mo stretchy="false" id="S4.E1.m1.36.36.36.15.13.13" xref="S4.E1.m1.44.44.2a.cmml">(</mo><msub id="S4.E1.m1.46.46.4.44.22.20.20.1.1.1.1.1.1.1.1" xref="S4.E1.m1.44.44.2.cmml"><mi id="S4.E1.m1.37.37.37.16.14.14" xref="S4.E1.m1.37.37.37.16.14.14.cmml">h</mi><mi id="S4.E1.m1.38.38.38.17.15.15.1" xref="S4.E1.m1.38.38.38.17.15.15.1.cmml">i</mi></msub><mo stretchy="false" id="S4.E1.m1.39.39.39.18.16.16" xref="S4.E1.m1.44.44.2a.cmml">)</mo></mrow></mrow><mo rspace="0.055em" stretchy="false" id="S4.E1.m1.40.40.40.19.17.17" xref="S4.E1.m1.44.44.2a.cmml">)</mo></mrow></mrow><mo rspace="0.222em" id="S4.E1.m1.41.41.41.20.18.18" xref="S4.E1.m1.41.41.41.20.18.18.cmml">×</mo><mfrac id="S4.E1.m1.42.42.42.21.19.19" xref="S4.E1.m1.42.42.42.21.19.19.cmml"><mn id="S4.E1.m1.42.42.42.21.19.19.2" xref="S4.E1.m1.42.42.42.21.19.19.2.cmml">1</mn><msub id="S4.E1.m1.42.42.42.21.19.19.3" xref="S4.E1.m1.42.42.42.21.19.19.3.cmml"><mi id="S4.E1.m1.42.42.42.21.19.19.3.2" xref="S4.E1.m1.42.42.42.21.19.19.3.2.cmml">p</mi><mi id="S4.E1.m1.42.42.42.21.19.19.3.3" xref="S4.E1.m1.42.42.42.21.19.19.3.3.cmml">i</mi></msub></mfrac></mrow></mrow></mrow></mtd></mtr></mtable><annotation-xml encoding="MathML-Content" id="S4.E1.m1.46b"><apply id="S4.E1.m1.44.44.2.cmml" xref="S4.E1.m1.46.46.4"><and id="S4.E1.m1.44.44.2a.cmml" xref="S4.E1.m1.45.45.3.43.22.20.21"></and><apply id="S4.E1.m1.44.44.2b.cmml" xref="S4.E1.m1.46.46.4"><eq id="S4.E1.m1.3.3.3.3.1.1.cmml" xref="S4.E1.m1.3.3.3.3.1.1"></eq><apply id="S4.E1.m1.44.44.2.4.cmml" xref="S4.E1.m1.46.46.4"><csymbol cd="ambiguous" id="S4.E1.m1.44.44.2.4.1.cmml" xref="S4.E1.m1.45.45.3.43.22.20.21">subscript</csymbol><ci id="S4.E1.m1.1.1.1.1.1.1.cmml" xref="S4.E1.m1.1.1.1.1.1.1">ℎ</ci><ci id="S4.E1.m1.2.2.2.2.2.2.1.cmml" xref="S4.E1.m1.2.2.2.2.2.2.1">𝑖</ci></apply><apply id="S4.E1.m1.43.43.1.1.cmml" xref="S4.E1.m1.46.46.4"><plus id="S4.E1.m1.6.6.6.6.4.4.cmml" xref="S4.E1.m1.6.6.6.6.4.4"></plus><apply id="S4.E1.m1.43.43.1.1.3.cmml" xref="S4.E1.m1.46.46.4"><csymbol cd="ambiguous" id="S4.E1.m1.43.43.1.1.3.1.cmml" xref="S4.E1.m1.45.45.3.43.22.20.21">subscript</csymbol><ci id="S4.E1.m1.4.4.4.4.2.2.cmml" xref="S4.E1.m1.4.4.4.4.2.2">𝑥</ci><ci id="S4.E1.m1.5.5.5.5.3.3.1.cmml" xref="S4.E1.m1.5.5.5.5.3.3.1">𝑖</ci></apply><apply id="S4.E1.m1.43.43.1.1.1.cmml" xref="S4.E1.m1.46.46.4"><times id="S4.E1.m1.43.43.1.1.1.2.cmml" xref="S4.E1.m1.45.45.3.43.22.20.21"></times><apply id="S4.E1.m1.43.43.1.1.1.1.cmml" xref="S4.E1.m1.46.46.4"><times id="S4.E1.m1.20.20.20.20.18.18.cmml" xref="S4.E1.m1.20.20.20.20.18.18"></times><apply id="S4.E1.m1.43.43.1.1.1.1.1.cmml" xref="S4.E1.m1.46.46.4"><times id="S4.E1.m1.43.43.1.1.1.1.1.2.cmml" xref="S4.E1.m1.45.45.3.43.22.20.21"></times><apply id="S4.E1.m1.43.43.1.1.1.1.1.3.cmml" xref="S4.E1.m1.46.46.4"><times id="S4.E1.m1.9.9.9.9.7.7.cmml" xref="S4.E1.m1.9.9.9.9.7.7"></times><apply id="S4.E1.m1.43.43.1.1.1.1.1.3.2.cmml" xref="S4.E1.m1.46.46.4"><csymbol cd="ambiguous" id="S4.E1.m1.43.43.1.1.1.1.1.3.2.1.cmml" xref="S4.E1.m1.45.45.3.43.22.20.21">subscript</csymbol><ci id="S4.E1.m1.7.7.7.7.5.5.cmml" xref="S4.E1.m1.7.7.7.7.5.5">𝐺</ci><ci id="S4.E1.m1.8.8.8.8.6.6.1.cmml" xref="S4.E1.m1.8.8.8.8.6.6.1">𝑖</ci></apply><apply id="S4.E1.m1.43.43.1.1.1.1.1.3.3.cmml" xref="S4.E1.m1.46.46.4"><csymbol cd="ambiguous" id="S4.E1.m1.43.43.1.1.1.1.1.3.3.1.cmml" xref="S4.E1.m1.45.45.3.43.22.20.21">subscript</csymbol><ci id="S4.E1.m1.10.10.10.10.8.8.cmml" xref="S4.E1.m1.10.10.10.10.8.8">𝑓</ci><apply id="S4.E1.m1.11.11.11.11.9.9.1.cmml" xref="S4.E1.m1.11.11.11.11.9.9.1"><minus id="S4.E1.m1.11.11.11.11.9.9.1.1.cmml" xref="S4.E1.m1.11.11.11.11.9.9.1.1"></minus><ci id="S4.E1.m1.11.11.11.11.9.9.1.2.cmml" xref="S4.E1.m1.11.11.11.11.9.9.1.2">𝑆</ci><apply id="S4.E1.m1.11.11.11.11.9.9.1.3.cmml" xref="S4.E1.m1.11.11.11.11.9.9.1.3"><times id="S4.E1.m1.11.11.11.11.9.9.1.3.1.cmml" xref="S4.E1.m1.11.11.11.11.9.9.1.3.1"></times><ci id="S4.E1.m1.11.11.11.11.9.9.1.3.2.cmml" xref="S4.E1.m1.11.11.11.11.9.9.1.3.2">𝐴</ci><ci id="S4.E1.m1.11.11.11.11.9.9.1.3.3.cmml" xref="S4.E1.m1.11.11.11.11.9.9.1.3.3">𝑇</ci><ci id="S4.E1.m1.11.11.11.11.9.9.1.3.4.cmml" xref="S4.E1.m1.11.11.11.11.9.9.1.3.4">𝑇</ci><ci id="S4.E1.m1.11.11.11.11.9.9.1.3.5.cmml" xref="S4.E1.m1.11.11.11.11.9.9.1.3.5">𝑁</ci></apply></apply></apply></apply><apply id="S4.E1.m1.43.43.1.1.1.1.1.1.1.1.cmml" xref="S4.E1.m1.46.46.4"><times id="S4.E1.m1.43.43.1.1.1.1.1.1.1.1.2.cmml" xref="S4.E1.m1.46.46.4"></times><apply id="S4.E1.m1.43.43.1.1.1.1.1.1.1.1.3.cmml" xref="S4.E1.m1.46.46.4"><csymbol cd="ambiguous" id="S4.E1.m1.43.43.1.1.1.1.1.1.1.1.3.1.cmml" xref="S4.E1.m1.46.46.4">subscript</csymbol><ci id="S4.E1.m1.13.13.13.13.11.11.cmml" xref="S4.E1.m1.13.13.13.13.11.11">𝑓</ci><apply id="S4.E1.m1.14.14.14.14.12.12.1.cmml" xref="S4.E1.m1.14.14.14.14.12.12.1"><times id="S4.E1.m1.14.14.14.14.12.12.1.1.cmml" xref="S4.E1.m1.14.14.14.14.12.12.1.1"></times><ci id="S4.E1.m1.14.14.14.14.12.12.1.2.cmml" xref="S4.E1.m1.14.14.14.14.12.12.1.2">𝐿</ci><ci id="S4.E1.m1.14.14.14.14.12.12.1.3.cmml" xref="S4.E1.m1.14.14.14.14.12.12.1.3">𝑁</ci></apply></apply><apply id="S4.E1.m1.43.43.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E1.m1.46.46.4"><csymbol cd="ambiguous" id="S4.E1.m1.43.43.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E1.m1.46.46.4">subscript</csymbol><ci id="S4.E1.m1.16.16.16.16.14.14.cmml" xref="S4.E1.m1.16.16.16.16.14.14">𝑥</ci><ci id="S4.E1.m1.17.17.17.17.15.15.1.cmml" xref="S4.E1.m1.17.17.17.17.15.15.1">𝑖</ci></apply></apply></apply><apply id="S4.E1.m1.21.21.21.21.19.19.cmml" xref="S4.E1.m1.21.21.21.21.19.19"><divide id="S4.E1.m1.21.21.21.21.19.19.1.cmml" xref="S4.E1.m1.21.21.21.21.19.19"></divide><cn type="integer" id="S4.E1.m1.21.21.21.21.19.19.2.cmml" xref="S4.E1.m1.21.21.21.21.19.19.2">1</cn><apply id="S4.E1.m1.21.21.21.21.19.19.3.cmml" xref="S4.E1.m1.21.21.21.21.19.19.3"><csymbol cd="ambiguous" id="S4.E1.m1.21.21.21.21.19.19.3.1.cmml" xref="S4.E1.m1.21.21.21.21.19.19.3">subscript</csymbol><ci id="S4.E1.m1.21.21.21.21.19.19.3.2.cmml" xref="S4.E1.m1.21.21.21.21.19.19.3.2">𝑝</ci><ci id="S4.E1.m1.21.21.21.21.19.19.3.3.cmml" xref="S4.E1.m1.21.21.21.21.19.19.3.3">𝑖</ci></apply></apply></apply><apply id="S4.E1.m1.43.43.1.1.1.3.cmml" xref="S4.E1.m1.46.46.4"><csymbol cd="ambiguous" id="S4.E1.m1.43.43.1.1.1.3.1.cmml" xref="S4.E1.m1.45.45.3.43.22.20.21">subscript</csymbol><ci id="S4.E1.m1.22.22.22.1.1.1.cmml" xref="S4.E1.m1.22.22.22.1.1.1">𝑥</ci><apply id="S4.E1.m1.23.23.23.2.2.2.1.cmml" xref="S4.E1.m1.23.23.23.2.2.2.1"><plus id="S4.E1.m1.23.23.23.2.2.2.1.1.cmml" xref="S4.E1.m1.23.23.23.2.2.2.1.1"></plus><ci id="S4.E1.m1.23.23.23.2.2.2.1.2.cmml" xref="S4.E1.m1.23.23.23.2.2.2.1.2">𝑖</ci><cn type="integer" id="S4.E1.m1.23.23.23.2.2.2.1.3.cmml" xref="S4.E1.m1.23.23.23.2.2.2.1.3">1</cn></apply></apply></apply></apply></apply><apply id="S4.E1.m1.44.44.2c.cmml" xref="S4.E1.m1.46.46.4"><eq id="S4.E1.m1.24.24.24.3.1.1.cmml" xref="S4.E1.m1.24.24.24.3.1.1"></eq><share href="#S4.E1.m1.43.43.1.1.cmml" id="S4.E1.m1.44.44.2d.cmml" xref="S4.E1.m1.45.45.3.43.22.20.21"></share><apply id="S4.E1.m1.44.44.2.2.cmml" xref="S4.E1.m1.46.46.4"><plus id="S4.E1.m1.27.27.27.6.4.4.cmml" xref="S4.E1.m1.27.27.27.6.4.4"></plus><apply id="S4.E1.m1.44.44.2.2.3.cmml" xref="S4.E1.m1.46.46.4"><csymbol cd="ambiguous" id="S4.E1.m1.44.44.2.2.3.1.cmml" xref="S4.E1.m1.45.45.3.43.22.20.21">subscript</csymbol><ci id="S4.E1.m1.25.25.25.4.2.2.cmml" xref="S4.E1.m1.25.25.25.4.2.2">ℎ</ci><ci id="S4.E1.m1.26.26.26.5.3.3.1.cmml" xref="S4.E1.m1.26.26.26.5.3.3.1">𝑖</ci></apply><apply id="S4.E1.m1.44.44.2.2.1.cmml" xref="S4.E1.m1.46.46.4"><times id="S4.E1.m1.41.41.41.20.18.18.cmml" xref="S4.E1.m1.41.41.41.20.18.18"></times><apply id="S4.E1.m1.44.44.2.2.1.1.cmml" xref="S4.E1.m1.46.46.4"><times id="S4.E1.m1.44.44.2.2.1.1.2.cmml" xref="S4.E1.m1.45.45.3.43.22.20.21"></times><apply id="S4.E1.m1.44.44.2.2.1.1.3.cmml" xref="S4.E1.m1.46.46.4"><times id="S4.E1.m1.30.30.30.9.7.7.cmml" xref="S4.E1.m1.30.30.30.9.7.7"></times><apply id="S4.E1.m1.44.44.2.2.1.1.3.2.cmml" xref="S4.E1.m1.46.46.4"><csymbol cd="ambiguous" id="S4.E1.m1.44.44.2.2.1.1.3.2.1.cmml" xref="S4.E1.m1.45.45.3.43.22.20.21">subscript</csymbol><ci id="S4.E1.m1.28.28.28.7.5.5.cmml" xref="S4.E1.m1.28.28.28.7.5.5">𝐺</ci><ci id="S4.E1.m1.29.29.29.8.6.6.1.cmml" xref="S4.E1.m1.29.29.29.8.6.6.1">𝑖</ci></apply><apply id="S4.E1.m1.44.44.2.2.1.1.3.3.cmml" xref="S4.E1.m1.46.46.4"><csymbol cd="ambiguous" id="S4.E1.m1.44.44.2.2.1.1.3.3.1.cmml" xref="S4.E1.m1.45.45.3.43.22.20.21">subscript</csymbol><ci id="S4.E1.m1.31.31.31.10.8.8.cmml" xref="S4.E1.m1.31.31.31.10.8.8">𝑓</ci><apply id="S4.E1.m1.32.32.32.11.9.9.1.cmml" xref="S4.E1.m1.32.32.32.11.9.9.1"><times id="S4.E1.m1.32.32.32.11.9.9.1.1.cmml" xref="S4.E1.m1.32.32.32.11.9.9.1.1"></times><ci id="S4.E1.m1.32.32.32.11.9.9.1.2.cmml" xref="S4.E1.m1.32.32.32.11.9.9.1.2">𝐹</ci><ci id="S4.E1.m1.32.32.32.11.9.9.1.3.cmml" xref="S4.E1.m1.32.32.32.11.9.9.1.3">𝐹</ci><ci id="S4.E1.m1.32.32.32.11.9.9.1.4.cmml" xref="S4.E1.m1.32.32.32.11.9.9.1.4">𝑁</ci></apply></apply></apply><apply id="S4.E1.m1.44.44.2.2.1.1.1.1.1.cmml" xref="S4.E1.m1.46.46.4"><times id="S4.E1.m1.44.44.2.2.1.1.1.1.1.2.cmml" xref="S4.E1.m1.46.46.4"></times><apply id="S4.E1.m1.44.44.2.2.1.1.1.1.1.3.cmml" xref="S4.E1.m1.46.46.4"><csymbol cd="ambiguous" id="S4.E1.m1.44.44.2.2.1.1.1.1.1.3.1.cmml" xref="S4.E1.m1.46.46.4">subscript</csymbol><ci id="S4.E1.m1.34.34.34.13.11.11.cmml" xref="S4.E1.m1.34.34.34.13.11.11">𝑓</ci><apply id="S4.E1.m1.35.35.35.14.12.12.1.cmml" xref="S4.E1.m1.35.35.35.14.12.12.1"><times id="S4.E1.m1.35.35.35.14.12.12.1.1.cmml" xref="S4.E1.m1.35.35.35.14.12.12.1.1"></times><ci id="S4.E1.m1.35.35.35.14.12.12.1.2.cmml" xref="S4.E1.m1.35.35.35.14.12.12.1.2">𝐿</ci><ci id="S4.E1.m1.35.35.35.14.12.12.1.3.cmml" xref="S4.E1.m1.35.35.35.14.12.12.1.3">𝑁</ci></apply></apply><apply id="S4.E1.m1.44.44.2.2.1.1.1.1.1.1.1.1.cmml" xref="S4.E1.m1.46.46.4"><csymbol cd="ambiguous" id="S4.E1.m1.44.44.2.2.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E1.m1.46.46.4">subscript</csymbol><ci id="S4.E1.m1.37.37.37.16.14.14.cmml" xref="S4.E1.m1.37.37.37.16.14.14">ℎ</ci><ci id="S4.E1.m1.38.38.38.17.15.15.1.cmml" xref="S4.E1.m1.38.38.38.17.15.15.1">𝑖</ci></apply></apply></apply><apply id="S4.E1.m1.42.42.42.21.19.19.cmml" xref="S4.E1.m1.42.42.42.21.19.19"><divide id="S4.E1.m1.42.42.42.21.19.19.1.cmml" xref="S4.E1.m1.42.42.42.21.19.19"></divide><cn type="integer" id="S4.E1.m1.42.42.42.21.19.19.2.cmml" xref="S4.E1.m1.42.42.42.21.19.19.2">1</cn><apply id="S4.E1.m1.42.42.42.21.19.19.3.cmml" xref="S4.E1.m1.42.42.42.21.19.19.3"><csymbol cd="ambiguous" id="S4.E1.m1.42.42.42.21.19.19.3.1.cmml" xref="S4.E1.m1.42.42.42.21.19.19.3">subscript</csymbol><ci id="S4.E1.m1.42.42.42.21.19.19.3.2.cmml" xref="S4.E1.m1.42.42.42.21.19.19.3.2">𝑝</ci><ci id="S4.E1.m1.42.42.42.21.19.19.3.3.cmml" xref="S4.E1.m1.42.42.42.21.19.19.3.3">𝑖</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E1.m1.46c">\begin{split}h_{i}&amp;=x_{i}+G_{i}\times f_{S-ATTN}(f_{LN}(x_{i}))\times\frac{1}{p_{i}}\\
x_{i+1}&amp;=h_{i}+G_{i}\times f_{FFN}(f_{LN}(h_{i}))\times\frac{1}{p_{i}}\\
\end{split}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
</div>
<div id="S4.SS1.SSS0.Px2.p3" class="ltx_para">
<p id="S4.SS1.SSS0.Px2.p3.5" class="ltx_p">In our design, the function <math id="S4.SS1.SSS0.Px2.p3.1.m1.1" class="ltx_Math" alttext="G_{i}" display="inline"><semantics id="S4.SS1.SSS0.Px2.p3.1.m1.1a"><msub id="S4.SS1.SSS0.Px2.p3.1.m1.1.1" xref="S4.SS1.SSS0.Px2.p3.1.m1.1.1.cmml"><mi id="S4.SS1.SSS0.Px2.p3.1.m1.1.1.2" xref="S4.SS1.SSS0.Px2.p3.1.m1.1.1.2.cmml">G</mi><mi id="S4.SS1.SSS0.Px2.p3.1.m1.1.1.3" xref="S4.SS1.SSS0.Px2.p3.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p3.1.m1.1b"><apply id="S4.SS1.SSS0.Px2.p3.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS0.Px2.p3.1.m1.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.1.m1.1.1">subscript</csymbol><ci id="S4.SS1.SSS0.Px2.p3.1.m1.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p3.1.m1.1.1.2">𝐺</ci><ci id="S4.SS1.SSS0.Px2.p3.1.m1.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p3.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p3.1.m1.1c">G_{i}</annotation></semantics></math> only takes 0 or 1 as values, which is chosen randomly from a Bernoulli distribution (with two possible outcomes), <math id="S4.SS1.SSS0.Px2.p3.2.m2.2" class="ltx_Math" alttext="G_{i}\sim B(1,p_{i})" display="inline"><semantics id="S4.SS1.SSS0.Px2.p3.2.m2.2a"><mrow id="S4.SS1.SSS0.Px2.p3.2.m2.2.2" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.cmml"><msub id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.3" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.3.cmml"><mi id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.3.2" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.3.2.cmml">G</mi><mi id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.3.3" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.3.3.cmml">i</mi></msub><mo id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.2" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.2.cmml">∼</mo><mrow id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.cmml"><mi id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.3" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.3.cmml">B</mi><mo lspace="0em" rspace="0em" id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.2" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.2.cmml">​</mo><mrow id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.1" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.2.cmml"><mo stretchy="false" id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.1.2" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.2.cmml">(</mo><mn id="S4.SS1.SSS0.Px2.p3.2.m2.1.1" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.cmml">1</mn><mo id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.1.3" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.2.cmml">,</mo><msub id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.1.1" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.1.1.cmml"><mi id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.1.1.2" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.1.1.2.cmml">p</mi><mi id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.1.1.3" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.1.4" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p3.2.m2.2b"><apply id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2"><csymbol cd="latexml" id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.2.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.2">similar-to</csymbol><apply id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.3.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.3"><csymbol cd="ambiguous" id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.3.1.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.3">subscript</csymbol><ci id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.3.2.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.3.2">𝐺</ci><ci id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.3.3.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.3.3">𝑖</ci></apply><apply id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1"><times id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.2.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.2"></times><ci id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.3.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.3">𝐵</ci><interval closure="open" id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.1"><cn type="integer" id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1">1</cn><apply id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.1.1">subscript</csymbol><ci id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.1.1.2">𝑝</ci><ci id="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.2.2.1.1.1.1.3">𝑖</ci></apply></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p3.2.m2.2c">G_{i}\sim B(1,p_{i})</annotation></semantics></math>,
where <math id="S4.SS1.SSS0.Px2.p3.3.m3.1" class="ltx_Math" alttext="p_{i}" display="inline"><semantics id="S4.SS1.SSS0.Px2.p3.3.m3.1a"><msub id="S4.SS1.SSS0.Px2.p3.3.m3.1.1" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.cmml"><mi id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.2" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.2.cmml">p</mi><mi id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p3.3.m3.1b"><apply id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1">subscript</csymbol><ci id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.2">𝑝</ci><ci id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p3.3.m3.1c">p_{i}</annotation></semantics></math> is the probability of choosing 1. Because the blocks are selected with probability <math id="S4.SS1.SSS0.Px2.p3.4.m4.1" class="ltx_Math" alttext="p_{i}" display="inline"><semantics id="S4.SS1.SSS0.Px2.p3.4.m4.1a"><msub id="S4.SS1.SSS0.Px2.p3.4.m4.1.1" xref="S4.SS1.SSS0.Px2.p3.4.m4.1.1.cmml"><mi id="S4.SS1.SSS0.Px2.p3.4.m4.1.1.2" xref="S4.SS1.SSS0.Px2.p3.4.m4.1.1.2.cmml">p</mi><mi id="S4.SS1.SSS0.Px2.p3.4.m4.1.1.3" xref="S4.SS1.SSS0.Px2.p3.4.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p3.4.m4.1b"><apply id="S4.SS1.SSS0.Px2.p3.4.m4.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.4.m4.1.1"><csymbol cd="ambiguous" id="S4.SS1.SSS0.Px2.p3.4.m4.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.4.m4.1.1">subscript</csymbol><ci id="S4.SS1.SSS0.Px2.p3.4.m4.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p3.4.m4.1.1.2">𝑝</ci><ci id="S4.SS1.SSS0.Px2.p3.4.m4.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p3.4.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p3.4.m4.1c">p_{i}</annotation></semantics></math> during training and are always presented during inference, we re-calibrate the layers’ output by a scaling factor of <math id="S4.SS1.SSS0.Px2.p3.5.m5.1" class="ltx_Math" alttext="\frac{1}{p_{i}}" display="inline"><semantics id="S4.SS1.SSS0.Px2.p3.5.m5.1a"><mfrac id="S4.SS1.SSS0.Px2.p3.5.m5.1.1" xref="S4.SS1.SSS0.Px2.p3.5.m5.1.1.cmml"><mn id="S4.SS1.SSS0.Px2.p3.5.m5.1.1.2" xref="S4.SS1.SSS0.Px2.p3.5.m5.1.1.2.cmml">1</mn><msub id="S4.SS1.SSS0.Px2.p3.5.m5.1.1.3" xref="S4.SS1.SSS0.Px2.p3.5.m5.1.1.3.cmml"><mi id="S4.SS1.SSS0.Px2.p3.5.m5.1.1.3.2" xref="S4.SS1.SSS0.Px2.p3.5.m5.1.1.3.2.cmml">p</mi><mi id="S4.SS1.SSS0.Px2.p3.5.m5.1.1.3.3" xref="S4.SS1.SSS0.Px2.p3.5.m5.1.1.3.3.cmml">i</mi></msub></mfrac><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p3.5.m5.1b"><apply id="S4.SS1.SSS0.Px2.p3.5.m5.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.5.m5.1.1"><divide id="S4.SS1.SSS0.Px2.p3.5.m5.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.5.m5.1.1"></divide><cn type="integer" id="S4.SS1.SSS0.Px2.p3.5.m5.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p3.5.m5.1.1.2">1</cn><apply id="S4.SS1.SSS0.Px2.p3.5.m5.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p3.5.m5.1.1.3"><csymbol cd="ambiguous" id="S4.SS1.SSS0.Px2.p3.5.m5.1.1.3.1.cmml" xref="S4.SS1.SSS0.Px2.p3.5.m5.1.1.3">subscript</csymbol><ci id="S4.SS1.SSS0.Px2.p3.5.m5.1.1.3.2.cmml" xref="S4.SS1.SSS0.Px2.p3.5.m5.1.1.3.2">𝑝</ci><ci id="S4.SS1.SSS0.Px2.p3.5.m5.1.1.3.3.cmml" xref="S4.SS1.SSS0.Px2.p3.5.m5.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p3.5.m5.1c">\frac{1}{p_{i}}</annotation></semantics></math> whenever they are selected.</p>
</div>
</section>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>A Progressive Layer Dropping Schedule</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.6" class="ltx_p">Based on the insights from Section <a href="#S3.SS2" title="3.2 Corroboration of Training Dynamics ‣ 3 Preliminary Analysis ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>, and inspired by prior work on curriculum learning <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib41" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">41</span></a>, <a href="#bib.bib42" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">42</span></a>]</cite>

we propose a progressive schedule <math id="S4.SS2.p1.1.m1.1" class="ltx_Math" alttext="\theta(t)" display="inline"><semantics id="S4.SS2.p1.1.m1.1a"><mrow id="S4.SS2.p1.1.m1.1.2" xref="S4.SS2.p1.1.m1.1.2.cmml"><mi id="S4.SS2.p1.1.m1.1.2.2" xref="S4.SS2.p1.1.m1.1.2.2.cmml">θ</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p1.1.m1.1.2.1" xref="S4.SS2.p1.1.m1.1.2.1.cmml">​</mo><mrow id="S4.SS2.p1.1.m1.1.2.3.2" xref="S4.SS2.p1.1.m1.1.2.cmml"><mo stretchy="false" id="S4.SS2.p1.1.m1.1.2.3.2.1" xref="S4.SS2.p1.1.m1.1.2.cmml">(</mo><mi id="S4.SS2.p1.1.m1.1.1" xref="S4.SS2.p1.1.m1.1.1.cmml">t</mi><mo stretchy="false" id="S4.SS2.p1.1.m1.1.2.3.2.2" xref="S4.SS2.p1.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.1.m1.1b"><apply id="S4.SS2.p1.1.m1.1.2.cmml" xref="S4.SS2.p1.1.m1.1.2"><times id="S4.SS2.p1.1.m1.1.2.1.cmml" xref="S4.SS2.p1.1.m1.1.2.1"></times><ci id="S4.SS2.p1.1.m1.1.2.2.cmml" xref="S4.SS2.p1.1.m1.1.2.2">𝜃</ci><ci id="S4.SS2.p1.1.m1.1.1.cmml" xref="S4.SS2.p1.1.m1.1.1">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.1.m1.1c">\theta(t)</annotation></semantics></math> – a temporal schedule for the expected number of ST blocks that are retained.
We limit ourselves to monotonically decreasing functions so that the likelihood of layer dropping can only increase along the temporal dimension.
We constrain <math id="S4.SS2.p1.2.m2.1" class="ltx_Math" alttext="{\theta}(t)" display="inline"><semantics id="S4.SS2.p1.2.m2.1a"><mrow id="S4.SS2.p1.2.m2.1.2" xref="S4.SS2.p1.2.m2.1.2.cmml"><mi id="S4.SS2.p1.2.m2.1.2.2" xref="S4.SS2.p1.2.m2.1.2.2.cmml">θ</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p1.2.m2.1.2.1" xref="S4.SS2.p1.2.m2.1.2.1.cmml">​</mo><mrow id="S4.SS2.p1.2.m2.1.2.3.2" xref="S4.SS2.p1.2.m2.1.2.cmml"><mo stretchy="false" id="S4.SS2.p1.2.m2.1.2.3.2.1" xref="S4.SS2.p1.2.m2.1.2.cmml">(</mo><mi id="S4.SS2.p1.2.m2.1.1" xref="S4.SS2.p1.2.m2.1.1.cmml">t</mi><mo stretchy="false" id="S4.SS2.p1.2.m2.1.2.3.2.2" xref="S4.SS2.p1.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.2.m2.1b"><apply id="S4.SS2.p1.2.m2.1.2.cmml" xref="S4.SS2.p1.2.m2.1.2"><times id="S4.SS2.p1.2.m2.1.2.1.cmml" xref="S4.SS2.p1.2.m2.1.2.1"></times><ci id="S4.SS2.p1.2.m2.1.2.2.cmml" xref="S4.SS2.p1.2.m2.1.2.2">𝜃</ci><ci id="S4.SS2.p1.2.m2.1.1.cmml" xref="S4.SS2.p1.2.m2.1.1">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.2.m2.1c">{\theta}(t)</annotation></semantics></math> to be <math id="S4.SS2.p1.3.m3.1" class="ltx_Math" alttext="{\theta}(t)\geq\bar{\theta}" display="inline"><semantics id="S4.SS2.p1.3.m3.1a"><mrow id="S4.SS2.p1.3.m3.1.2" xref="S4.SS2.p1.3.m3.1.2.cmml"><mrow id="S4.SS2.p1.3.m3.1.2.2" xref="S4.SS2.p1.3.m3.1.2.2.cmml"><mi id="S4.SS2.p1.3.m3.1.2.2.2" xref="S4.SS2.p1.3.m3.1.2.2.2.cmml">θ</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p1.3.m3.1.2.2.1" xref="S4.SS2.p1.3.m3.1.2.2.1.cmml">​</mo><mrow id="S4.SS2.p1.3.m3.1.2.2.3.2" xref="S4.SS2.p1.3.m3.1.2.2.cmml"><mo stretchy="false" id="S4.SS2.p1.3.m3.1.2.2.3.2.1" xref="S4.SS2.p1.3.m3.1.2.2.cmml">(</mo><mi id="S4.SS2.p1.3.m3.1.1" xref="S4.SS2.p1.3.m3.1.1.cmml">t</mi><mo stretchy="false" id="S4.SS2.p1.3.m3.1.2.2.3.2.2" xref="S4.SS2.p1.3.m3.1.2.2.cmml">)</mo></mrow></mrow><mo id="S4.SS2.p1.3.m3.1.2.1" xref="S4.SS2.p1.3.m3.1.2.1.cmml">≥</mo><mover accent="true" id="S4.SS2.p1.3.m3.1.2.3" xref="S4.SS2.p1.3.m3.1.2.3.cmml"><mi id="S4.SS2.p1.3.m3.1.2.3.2" xref="S4.SS2.p1.3.m3.1.2.3.2.cmml">θ</mi><mo id="S4.SS2.p1.3.m3.1.2.3.1" xref="S4.SS2.p1.3.m3.1.2.3.1.cmml">¯</mo></mover></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.3.m3.1b"><apply id="S4.SS2.p1.3.m3.1.2.cmml" xref="S4.SS2.p1.3.m3.1.2"><geq id="S4.SS2.p1.3.m3.1.2.1.cmml" xref="S4.SS2.p1.3.m3.1.2.1"></geq><apply id="S4.SS2.p1.3.m3.1.2.2.cmml" xref="S4.SS2.p1.3.m3.1.2.2"><times id="S4.SS2.p1.3.m3.1.2.2.1.cmml" xref="S4.SS2.p1.3.m3.1.2.2.1"></times><ci id="S4.SS2.p1.3.m3.1.2.2.2.cmml" xref="S4.SS2.p1.3.m3.1.2.2.2">𝜃</ci><ci id="S4.SS2.p1.3.m3.1.1.cmml" xref="S4.SS2.p1.3.m3.1.1">𝑡</ci></apply><apply id="S4.SS2.p1.3.m3.1.2.3.cmml" xref="S4.SS2.p1.3.m3.1.2.3"><ci id="S4.SS2.p1.3.m3.1.2.3.1.cmml" xref="S4.SS2.p1.3.m3.1.2.3.1">¯</ci><ci id="S4.SS2.p1.3.m3.1.2.3.2.cmml" xref="S4.SS2.p1.3.m3.1.2.3.2">𝜃</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.3.m3.1c">{\theta}(t)\geq\bar{\theta}</annotation></semantics></math> for any t, where <math id="S4.SS2.p1.4.m4.1" class="ltx_Math" alttext="\bar{\theta}" display="inline"><semantics id="S4.SS2.p1.4.m4.1a"><mover accent="true" id="S4.SS2.p1.4.m4.1.1" xref="S4.SS2.p1.4.m4.1.1.cmml"><mi id="S4.SS2.p1.4.m4.1.1.2" xref="S4.SS2.p1.4.m4.1.1.2.cmml">θ</mi><mo id="S4.SS2.p1.4.m4.1.1.1" xref="S4.SS2.p1.4.m4.1.1.1.cmml">¯</mo></mover><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.4.m4.1b"><apply id="S4.SS2.p1.4.m4.1.1.cmml" xref="S4.SS2.p1.4.m4.1.1"><ci id="S4.SS2.p1.4.m4.1.1.1.cmml" xref="S4.SS2.p1.4.m4.1.1.1">¯</ci><ci id="S4.SS2.p1.4.m4.1.1.2.cmml" xref="S4.SS2.p1.4.m4.1.1.2">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.4.m4.1c">\bar{\theta}</annotation></semantics></math> is a limit value, to be taken as <math id="S4.SS2.p1.5.m5.1" class="ltx_Math" alttext="0.5\leq\bar{\theta}\leq 0.9" display="inline"><semantics id="S4.SS2.p1.5.m5.1a"><mrow id="S4.SS2.p1.5.m5.1.1" xref="S4.SS2.p1.5.m5.1.1.cmml"><mn id="S4.SS2.p1.5.m5.1.1.2" xref="S4.SS2.p1.5.m5.1.1.2.cmml">0.5</mn><mo id="S4.SS2.p1.5.m5.1.1.3" xref="S4.SS2.p1.5.m5.1.1.3.cmml">≤</mo><mover accent="true" id="S4.SS2.p1.5.m5.1.1.4" xref="S4.SS2.p1.5.m5.1.1.4.cmml"><mi id="S4.SS2.p1.5.m5.1.1.4.2" xref="S4.SS2.p1.5.m5.1.1.4.2.cmml">θ</mi><mo id="S4.SS2.p1.5.m5.1.1.4.1" xref="S4.SS2.p1.5.m5.1.1.4.1.cmml">¯</mo></mover><mo id="S4.SS2.p1.5.m5.1.1.5" xref="S4.SS2.p1.5.m5.1.1.5.cmml">≤</mo><mn id="S4.SS2.p1.5.m5.1.1.6" xref="S4.SS2.p1.5.m5.1.1.6.cmml">0.9</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.5.m5.1b"><apply id="S4.SS2.p1.5.m5.1.1.cmml" xref="S4.SS2.p1.5.m5.1.1"><and id="S4.SS2.p1.5.m5.1.1a.cmml" xref="S4.SS2.p1.5.m5.1.1"></and><apply id="S4.SS2.p1.5.m5.1.1b.cmml" xref="S4.SS2.p1.5.m5.1.1"><leq id="S4.SS2.p1.5.m5.1.1.3.cmml" xref="S4.SS2.p1.5.m5.1.1.3"></leq><cn type="float" id="S4.SS2.p1.5.m5.1.1.2.cmml" xref="S4.SS2.p1.5.m5.1.1.2">0.5</cn><apply id="S4.SS2.p1.5.m5.1.1.4.cmml" xref="S4.SS2.p1.5.m5.1.1.4"><ci id="S4.SS2.p1.5.m5.1.1.4.1.cmml" xref="S4.SS2.p1.5.m5.1.1.4.1">¯</ci><ci id="S4.SS2.p1.5.m5.1.1.4.2.cmml" xref="S4.SS2.p1.5.m5.1.1.4.2">𝜃</ci></apply></apply><apply id="S4.SS2.p1.5.m5.1.1c.cmml" xref="S4.SS2.p1.5.m5.1.1"><leq id="S4.SS2.p1.5.m5.1.1.5.cmml" xref="S4.SS2.p1.5.m5.1.1.5"></leq><share href="#S4.SS2.p1.5.m5.1.1.4.cmml" id="S4.SS2.p1.5.m5.1.1d.cmml" xref="S4.SS2.p1.5.m5.1.1"></share><cn type="float" id="S4.SS2.p1.5.m5.1.1.6.cmml" xref="S4.SS2.p1.5.m5.1.1.6">0.9</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.5.m5.1c">0.5\leq\bar{\theta}\leq 0.9</annotation></semantics></math> (Section <a href="#S5" title="5 Evaluation ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>).
Based on this, we define the progressive schedule <math id="S4.SS2.p1.6.m6.1" class="ltx_Math" alttext="\theta(t)" display="inline"><semantics id="S4.SS2.p1.6.m6.1a"><mrow id="S4.SS2.p1.6.m6.1.2" xref="S4.SS2.p1.6.m6.1.2.cmml"><mi id="S4.SS2.p1.6.m6.1.2.2" xref="S4.SS2.p1.6.m6.1.2.2.cmml">θ</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p1.6.m6.1.2.1" xref="S4.SS2.p1.6.m6.1.2.1.cmml">​</mo><mrow id="S4.SS2.p1.6.m6.1.2.3.2" xref="S4.SS2.p1.6.m6.1.2.cmml"><mo stretchy="false" id="S4.SS2.p1.6.m6.1.2.3.2.1" xref="S4.SS2.p1.6.m6.1.2.cmml">(</mo><mi id="S4.SS2.p1.6.m6.1.1" xref="S4.SS2.p1.6.m6.1.1.cmml">t</mi><mo stretchy="false" id="S4.SS2.p1.6.m6.1.2.3.2.2" xref="S4.SS2.p1.6.m6.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.6.m6.1b"><apply id="S4.SS2.p1.6.m6.1.2.cmml" xref="S4.SS2.p1.6.m6.1.2"><times id="S4.SS2.p1.6.m6.1.2.1.cmml" xref="S4.SS2.p1.6.m6.1.2.1"></times><ci id="S4.SS2.p1.6.m6.1.2.2.cmml" xref="S4.SS2.p1.6.m6.1.2.2">𝜃</ci><ci id="S4.SS2.p1.6.m6.1.1.cmml" xref="S4.SS2.p1.6.m6.1.1">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.6.m6.1c">\theta(t)</annotation></semantics></math> as:</p>
</div>
<div id="S4.Thmdefn1" class="ltx_theorem ltx_theorem_defn">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="S4.Thmdefn1.1.1.1" class="ltx_text ltx_font_bold">Definition 4.1</span></span><span id="S4.Thmdefn1.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="S4.Thmdefn1.p1" class="ltx_para">
<p id="S4.Thmdefn1.p1.4" class="ltx_p"><span id="S4.Thmdefn1.p1.4.4" class="ltx_text ltx_font_italic">A progressive schedule is a function <math id="S4.Thmdefn1.p1.1.1.m1.1" class="ltx_Math" alttext="t\rightarrow\theta(t)" display="inline"><semantics id="S4.Thmdefn1.p1.1.1.m1.1a"><mrow id="S4.Thmdefn1.p1.1.1.m1.1.2" xref="S4.Thmdefn1.p1.1.1.m1.1.2.cmml"><mi id="S4.Thmdefn1.p1.1.1.m1.1.2.2" xref="S4.Thmdefn1.p1.1.1.m1.1.2.2.cmml">t</mi><mo stretchy="false" id="S4.Thmdefn1.p1.1.1.m1.1.2.1" xref="S4.Thmdefn1.p1.1.1.m1.1.2.1.cmml">→</mo><mrow id="S4.Thmdefn1.p1.1.1.m1.1.2.3" xref="S4.Thmdefn1.p1.1.1.m1.1.2.3.cmml"><mi id="S4.Thmdefn1.p1.1.1.m1.1.2.3.2" xref="S4.Thmdefn1.p1.1.1.m1.1.2.3.2.cmml">θ</mi><mo lspace="0em" rspace="0em" id="S4.Thmdefn1.p1.1.1.m1.1.2.3.1" xref="S4.Thmdefn1.p1.1.1.m1.1.2.3.1.cmml">​</mo><mrow id="S4.Thmdefn1.p1.1.1.m1.1.2.3.3.2" xref="S4.Thmdefn1.p1.1.1.m1.1.2.3.cmml"><mo stretchy="false" id="S4.Thmdefn1.p1.1.1.m1.1.2.3.3.2.1" xref="S4.Thmdefn1.p1.1.1.m1.1.2.3.cmml">(</mo><mi id="S4.Thmdefn1.p1.1.1.m1.1.1" xref="S4.Thmdefn1.p1.1.1.m1.1.1.cmml">t</mi><mo stretchy="false" id="S4.Thmdefn1.p1.1.1.m1.1.2.3.3.2.2" xref="S4.Thmdefn1.p1.1.1.m1.1.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmdefn1.p1.1.1.m1.1b"><apply id="S4.Thmdefn1.p1.1.1.m1.1.2.cmml" xref="S4.Thmdefn1.p1.1.1.m1.1.2"><ci id="S4.Thmdefn1.p1.1.1.m1.1.2.1.cmml" xref="S4.Thmdefn1.p1.1.1.m1.1.2.1">→</ci><ci id="S4.Thmdefn1.p1.1.1.m1.1.2.2.cmml" xref="S4.Thmdefn1.p1.1.1.m1.1.2.2">𝑡</ci><apply id="S4.Thmdefn1.p1.1.1.m1.1.2.3.cmml" xref="S4.Thmdefn1.p1.1.1.m1.1.2.3"><times id="S4.Thmdefn1.p1.1.1.m1.1.2.3.1.cmml" xref="S4.Thmdefn1.p1.1.1.m1.1.2.3.1"></times><ci id="S4.Thmdefn1.p1.1.1.m1.1.2.3.2.cmml" xref="S4.Thmdefn1.p1.1.1.m1.1.2.3.2">𝜃</ci><ci id="S4.Thmdefn1.p1.1.1.m1.1.1.cmml" xref="S4.Thmdefn1.p1.1.1.m1.1.1">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmdefn1.p1.1.1.m1.1c">t\rightarrow\theta(t)</annotation></semantics></math> such that <math id="S4.Thmdefn1.p1.2.2.m2.1" class="ltx_Math" alttext="\theta(0)" display="inline"><semantics id="S4.Thmdefn1.p1.2.2.m2.1a"><mrow id="S4.Thmdefn1.p1.2.2.m2.1.2" xref="S4.Thmdefn1.p1.2.2.m2.1.2.cmml"><mi id="S4.Thmdefn1.p1.2.2.m2.1.2.2" xref="S4.Thmdefn1.p1.2.2.m2.1.2.2.cmml">θ</mi><mo lspace="0em" rspace="0em" id="S4.Thmdefn1.p1.2.2.m2.1.2.1" xref="S4.Thmdefn1.p1.2.2.m2.1.2.1.cmml">​</mo><mrow id="S4.Thmdefn1.p1.2.2.m2.1.2.3.2" xref="S4.Thmdefn1.p1.2.2.m2.1.2.cmml"><mo stretchy="false" id="S4.Thmdefn1.p1.2.2.m2.1.2.3.2.1" xref="S4.Thmdefn1.p1.2.2.m2.1.2.cmml">(</mo><mn id="S4.Thmdefn1.p1.2.2.m2.1.1" xref="S4.Thmdefn1.p1.2.2.m2.1.1.cmml">0</mn><mo stretchy="false" id="S4.Thmdefn1.p1.2.2.m2.1.2.3.2.2" xref="S4.Thmdefn1.p1.2.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmdefn1.p1.2.2.m2.1b"><apply id="S4.Thmdefn1.p1.2.2.m2.1.2.cmml" xref="S4.Thmdefn1.p1.2.2.m2.1.2"><times id="S4.Thmdefn1.p1.2.2.m2.1.2.1.cmml" xref="S4.Thmdefn1.p1.2.2.m2.1.2.1"></times><ci id="S4.Thmdefn1.p1.2.2.m2.1.2.2.cmml" xref="S4.Thmdefn1.p1.2.2.m2.1.2.2">𝜃</ci><cn type="integer" id="S4.Thmdefn1.p1.2.2.m2.1.1.cmml" xref="S4.Thmdefn1.p1.2.2.m2.1.1">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmdefn1.p1.2.2.m2.1c">\theta(0)</annotation></semantics></math> = 1 and <math id="S4.Thmdefn1.p1.3.3.m3.1" class="ltx_Math" alttext="\lim_{t\rightarrow\infty}{\theta(t)}\rightarrow\bar{\theta}" display="inline"><semantics id="S4.Thmdefn1.p1.3.3.m3.1a"><mrow id="S4.Thmdefn1.p1.3.3.m3.1.2" xref="S4.Thmdefn1.p1.3.3.m3.1.2.cmml"><mrow id="S4.Thmdefn1.p1.3.3.m3.1.2.2" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.cmml"><msub id="S4.Thmdefn1.p1.3.3.m3.1.2.2.1" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.cmml"><mo id="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.2" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.2.cmml">lim</mo><mrow id="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.3" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.3.cmml"><mi id="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.3.2" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.3.2.cmml">t</mi><mo stretchy="false" id="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.3.1" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.3.1.cmml">→</mo><mi mathvariant="normal" id="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.3.3" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.3.3.cmml">∞</mi></mrow></msub><mrow id="S4.Thmdefn1.p1.3.3.m3.1.2.2.2" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.2.cmml"><mi id="S4.Thmdefn1.p1.3.3.m3.1.2.2.2.2" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.2.2.cmml">θ</mi><mo lspace="0em" rspace="0em" id="S4.Thmdefn1.p1.3.3.m3.1.2.2.2.1" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.2.1.cmml">​</mo><mrow id="S4.Thmdefn1.p1.3.3.m3.1.2.2.2.3.2" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.2.cmml"><mo stretchy="false" id="S4.Thmdefn1.p1.3.3.m3.1.2.2.2.3.2.1" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.2.cmml">(</mo><mi id="S4.Thmdefn1.p1.3.3.m3.1.1" xref="S4.Thmdefn1.p1.3.3.m3.1.1.cmml">t</mi><mo stretchy="false" id="S4.Thmdefn1.p1.3.3.m3.1.2.2.2.3.2.2" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.2.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S4.Thmdefn1.p1.3.3.m3.1.2.1" xref="S4.Thmdefn1.p1.3.3.m3.1.2.1.cmml">→</mo><mover accent="true" id="S4.Thmdefn1.p1.3.3.m3.1.2.3" xref="S4.Thmdefn1.p1.3.3.m3.1.2.3.cmml"><mi id="S4.Thmdefn1.p1.3.3.m3.1.2.3.2" xref="S4.Thmdefn1.p1.3.3.m3.1.2.3.2.cmml">θ</mi><mo id="S4.Thmdefn1.p1.3.3.m3.1.2.3.1" xref="S4.Thmdefn1.p1.3.3.m3.1.2.3.1.cmml">¯</mo></mover></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmdefn1.p1.3.3.m3.1b"><apply id="S4.Thmdefn1.p1.3.3.m3.1.2.cmml" xref="S4.Thmdefn1.p1.3.3.m3.1.2"><ci id="S4.Thmdefn1.p1.3.3.m3.1.2.1.cmml" xref="S4.Thmdefn1.p1.3.3.m3.1.2.1">→</ci><apply id="S4.Thmdefn1.p1.3.3.m3.1.2.2.cmml" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2"><apply id="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.cmml" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.1"><csymbol cd="ambiguous" id="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.1.cmml" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.1">subscript</csymbol><limit id="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.2.cmml" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.2"></limit><apply id="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.3.cmml" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.3"><ci id="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.3.1.cmml" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.3.1">→</ci><ci id="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.3.2.cmml" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.3.2">𝑡</ci><infinity id="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.3.3.cmml" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.1.3.3"></infinity></apply></apply><apply id="S4.Thmdefn1.p1.3.3.m3.1.2.2.2.cmml" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.2"><times id="S4.Thmdefn1.p1.3.3.m3.1.2.2.2.1.cmml" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.2.1"></times><ci id="S4.Thmdefn1.p1.3.3.m3.1.2.2.2.2.cmml" xref="S4.Thmdefn1.p1.3.3.m3.1.2.2.2.2">𝜃</ci><ci id="S4.Thmdefn1.p1.3.3.m3.1.1.cmml" xref="S4.Thmdefn1.p1.3.3.m3.1.1">𝑡</ci></apply></apply><apply id="S4.Thmdefn1.p1.3.3.m3.1.2.3.cmml" xref="S4.Thmdefn1.p1.3.3.m3.1.2.3"><ci id="S4.Thmdefn1.p1.3.3.m3.1.2.3.1.cmml" xref="S4.Thmdefn1.p1.3.3.m3.1.2.3.1">¯</ci><ci id="S4.Thmdefn1.p1.3.3.m3.1.2.3.2.cmml" xref="S4.Thmdefn1.p1.3.3.m3.1.2.3.2">𝜃</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmdefn1.p1.3.3.m3.1c">\lim_{t\rightarrow\infty}{\theta(t)}\rightarrow\bar{\theta}</annotation></semantics></math>, where <math id="S4.Thmdefn1.p1.4.4.m4.2" class="ltx_Math" alttext="\bar{\theta}\in(0,1]" display="inline"><semantics id="S4.Thmdefn1.p1.4.4.m4.2a"><mrow id="S4.Thmdefn1.p1.4.4.m4.2.3" xref="S4.Thmdefn1.p1.4.4.m4.2.3.cmml"><mover accent="true" id="S4.Thmdefn1.p1.4.4.m4.2.3.2" xref="S4.Thmdefn1.p1.4.4.m4.2.3.2.cmml"><mi id="S4.Thmdefn1.p1.4.4.m4.2.3.2.2" xref="S4.Thmdefn1.p1.4.4.m4.2.3.2.2.cmml">θ</mi><mo id="S4.Thmdefn1.p1.4.4.m4.2.3.2.1" xref="S4.Thmdefn1.p1.4.4.m4.2.3.2.1.cmml">¯</mo></mover><mo id="S4.Thmdefn1.p1.4.4.m4.2.3.1" xref="S4.Thmdefn1.p1.4.4.m4.2.3.1.cmml">∈</mo><mrow id="S4.Thmdefn1.p1.4.4.m4.2.3.3.2" xref="S4.Thmdefn1.p1.4.4.m4.2.3.3.1.cmml"><mo stretchy="false" id="S4.Thmdefn1.p1.4.4.m4.2.3.3.2.1" xref="S4.Thmdefn1.p1.4.4.m4.2.3.3.1.cmml">(</mo><mn id="S4.Thmdefn1.p1.4.4.m4.1.1" xref="S4.Thmdefn1.p1.4.4.m4.1.1.cmml">0</mn><mo id="S4.Thmdefn1.p1.4.4.m4.2.3.3.2.2" xref="S4.Thmdefn1.p1.4.4.m4.2.3.3.1.cmml">,</mo><mn id="S4.Thmdefn1.p1.4.4.m4.2.2" xref="S4.Thmdefn1.p1.4.4.m4.2.2.cmml">1</mn><mo stretchy="false" id="S4.Thmdefn1.p1.4.4.m4.2.3.3.2.3" xref="S4.Thmdefn1.p1.4.4.m4.2.3.3.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.Thmdefn1.p1.4.4.m4.2b"><apply id="S4.Thmdefn1.p1.4.4.m4.2.3.cmml" xref="S4.Thmdefn1.p1.4.4.m4.2.3"><in id="S4.Thmdefn1.p1.4.4.m4.2.3.1.cmml" xref="S4.Thmdefn1.p1.4.4.m4.2.3.1"></in><apply id="S4.Thmdefn1.p1.4.4.m4.2.3.2.cmml" xref="S4.Thmdefn1.p1.4.4.m4.2.3.2"><ci id="S4.Thmdefn1.p1.4.4.m4.2.3.2.1.cmml" xref="S4.Thmdefn1.p1.4.4.m4.2.3.2.1">¯</ci><ci id="S4.Thmdefn1.p1.4.4.m4.2.3.2.2.cmml" xref="S4.Thmdefn1.p1.4.4.m4.2.3.2.2">𝜃</ci></apply><interval closure="open-closed" id="S4.Thmdefn1.p1.4.4.m4.2.3.3.1.cmml" xref="S4.Thmdefn1.p1.4.4.m4.2.3.3.2"><cn type="integer" id="S4.Thmdefn1.p1.4.4.m4.1.1.cmml" xref="S4.Thmdefn1.p1.4.4.m4.1.1">0</cn><cn type="integer" id="S4.Thmdefn1.p1.4.4.m4.2.2.cmml" xref="S4.Thmdefn1.p1.4.4.m4.2.2">1</cn></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Thmdefn1.p1.4.4.m4.2c">\bar{\theta}\in(0,1]</annotation></semantics></math>.</span></p>
</div>
</div>
<section id="S4.SS2.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Progress along the time dimension.</h4>

<div id="S4.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS2.SSS0.Px1.p1.3" class="ltx_p">Starting from the initial condition <math id="S4.SS2.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="\theta(0)=1" display="inline"><semantics id="S4.SS2.SSS0.Px1.p1.1.m1.1a"><mrow id="S4.SS2.SSS0.Px1.p1.1.m1.1.2" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.2.cmml"><mrow id="S4.SS2.SSS0.Px1.p1.1.m1.1.2.2" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.2.2.cmml"><mi id="S4.SS2.SSS0.Px1.p1.1.m1.1.2.2.2" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.2.2.2.cmml">θ</mi><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px1.p1.1.m1.1.2.2.1" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.2.2.1.cmml">​</mo><mrow id="S4.SS2.SSS0.Px1.p1.1.m1.1.2.2.3.2" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.2.2.cmml"><mo stretchy="false" id="S4.SS2.SSS0.Px1.p1.1.m1.1.2.2.3.2.1" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.2.2.cmml">(</mo><mn id="S4.SS2.SSS0.Px1.p1.1.m1.1.1" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.1.cmml">0</mn><mo stretchy="false" id="S4.SS2.SSS0.Px1.p1.1.m1.1.2.2.3.2.2" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.2.2.cmml">)</mo></mrow></mrow><mo id="S4.SS2.SSS0.Px1.p1.1.m1.1.2.1" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.2.1.cmml">=</mo><mn id="S4.SS2.SSS0.Px1.p1.1.m1.1.2.3" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.2.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p1.1.m1.1b"><apply id="S4.SS2.SSS0.Px1.p1.1.m1.1.2.cmml" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.2"><eq id="S4.SS2.SSS0.Px1.p1.1.m1.1.2.1.cmml" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.2.1"></eq><apply id="S4.SS2.SSS0.Px1.p1.1.m1.1.2.2.cmml" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.2.2"><times id="S4.SS2.SSS0.Px1.p1.1.m1.1.2.2.1.cmml" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.2.2.1"></times><ci id="S4.SS2.SSS0.Px1.p1.1.m1.1.2.2.2.cmml" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.2.2.2">𝜃</ci><cn type="integer" id="S4.SS2.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.1">0</cn></apply><cn type="integer" id="S4.SS2.SSS0.Px1.p1.1.m1.1.2.3.cmml" xref="S4.SS2.SSS0.Px1.p1.1.m1.1.2.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p1.1.m1.1c">\theta(0)=1</annotation></semantics></math> where no layer drop is performed, layer drop is gradually introduced. Eventually (i.e., when <math id="S4.SS2.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S4.SS2.SSS0.Px1.p1.2.m2.1a"><mi id="S4.SS2.SSS0.Px1.p1.2.m2.1.1" xref="S4.SS2.SSS0.Px1.p1.2.m2.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p1.2.m2.1b"><ci id="S4.SS2.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S4.SS2.SSS0.Px1.p1.2.m2.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p1.2.m2.1c">t</annotation></semantics></math> is sufficiently large), <math id="S4.SS2.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="\theta(t)\rightarrow\bar{\theta}" display="inline"><semantics id="S4.SS2.SSS0.Px1.p1.3.m3.1a"><mrow id="S4.SS2.SSS0.Px1.p1.3.m3.1.2" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.2.cmml"><mrow id="S4.SS2.SSS0.Px1.p1.3.m3.1.2.2" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.2.2.cmml"><mi id="S4.SS2.SSS0.Px1.p1.3.m3.1.2.2.2" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.2.2.2.cmml">θ</mi><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px1.p1.3.m3.1.2.2.1" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.2.2.1.cmml">​</mo><mrow id="S4.SS2.SSS0.Px1.p1.3.m3.1.2.2.3.2" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.2.2.cmml"><mo stretchy="false" id="S4.SS2.SSS0.Px1.p1.3.m3.1.2.2.3.2.1" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.2.2.cmml">(</mo><mi id="S4.SS2.SSS0.Px1.p1.3.m3.1.1" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.1.cmml">t</mi><mo stretchy="false" id="S4.SS2.SSS0.Px1.p1.3.m3.1.2.2.3.2.2" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.2.2.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S4.SS2.SSS0.Px1.p1.3.m3.1.2.1" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.2.1.cmml">→</mo><mover accent="true" id="S4.SS2.SSS0.Px1.p1.3.m3.1.2.3" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.2.3.cmml"><mi id="S4.SS2.SSS0.Px1.p1.3.m3.1.2.3.2" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.2.3.2.cmml">θ</mi><mo id="S4.SS2.SSS0.Px1.p1.3.m3.1.2.3.1" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.2.3.1.cmml">¯</mo></mover></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p1.3.m3.1b"><apply id="S4.SS2.SSS0.Px1.p1.3.m3.1.2.cmml" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.2"><ci id="S4.SS2.SSS0.Px1.p1.3.m3.1.2.1.cmml" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.2.1">→</ci><apply id="S4.SS2.SSS0.Px1.p1.3.m3.1.2.2.cmml" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.2.2"><times id="S4.SS2.SSS0.Px1.p1.3.m3.1.2.2.1.cmml" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.2.2.1"></times><ci id="S4.SS2.SSS0.Px1.p1.3.m3.1.2.2.2.cmml" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.2.2.2">𝜃</ci><ci id="S4.SS2.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.1">𝑡</ci></apply><apply id="S4.SS2.SSS0.Px1.p1.3.m3.1.2.3.cmml" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.2.3"><ci id="S4.SS2.SSS0.Px1.p1.3.m3.1.2.3.1.cmml" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.2.3.1">¯</ci><ci id="S4.SS2.SSS0.Px1.p1.3.m3.1.2.3.2.cmml" xref="S4.SS2.SSS0.Px1.p1.3.m3.1.2.3.2">𝜃</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p1.3.m3.1c">\theta(t)\rightarrow\bar{\theta}</annotation></semantics></math>.
According to Def. <a href="#S4.Thmdefn1" title="Definition 4.1. ‣ 4.2 A Progressive Layer Dropping Schedule ‣ 4 Our Approach: Progressive Layer Dropping ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.1</span></a>, in our work, we use the following schedule function:</p>
<table id="S4.E2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E2.m1.3" class="ltx_Math" alttext="\bar{\theta}(t)=(1-\bar{\theta})exp(-\gamma\cdot t)+\bar{\theta},\gamma&gt;0" display="block"><semantics id="S4.E2.m1.3a"><mrow id="S4.E2.m1.3.3.2" xref="S4.E2.m1.3.3.3.cmml"><mrow id="S4.E2.m1.2.2.1.1" xref="S4.E2.m1.2.2.1.1.cmml"><mrow id="S4.E2.m1.2.2.1.1.4" xref="S4.E2.m1.2.2.1.1.4.cmml"><mover accent="true" id="S4.E2.m1.2.2.1.1.4.2" xref="S4.E2.m1.2.2.1.1.4.2.cmml"><mi id="S4.E2.m1.2.2.1.1.4.2.2" xref="S4.E2.m1.2.2.1.1.4.2.2.cmml">θ</mi><mo id="S4.E2.m1.2.2.1.1.4.2.1" xref="S4.E2.m1.2.2.1.1.4.2.1.cmml">¯</mo></mover><mo lspace="0em" rspace="0em" id="S4.E2.m1.2.2.1.1.4.1" xref="S4.E2.m1.2.2.1.1.4.1.cmml">​</mo><mrow id="S4.E2.m1.2.2.1.1.4.3.2" xref="S4.E2.m1.2.2.1.1.4.cmml"><mo stretchy="false" id="S4.E2.m1.2.2.1.1.4.3.2.1" xref="S4.E2.m1.2.2.1.1.4.cmml">(</mo><mi id="S4.E2.m1.1.1" xref="S4.E2.m1.1.1.cmml">t</mi><mo stretchy="false" id="S4.E2.m1.2.2.1.1.4.3.2.2" xref="S4.E2.m1.2.2.1.1.4.cmml">)</mo></mrow></mrow><mo id="S4.E2.m1.2.2.1.1.3" xref="S4.E2.m1.2.2.1.1.3.cmml">=</mo><mrow id="S4.E2.m1.2.2.1.1.2" xref="S4.E2.m1.2.2.1.1.2.cmml"><mrow id="S4.E2.m1.2.2.1.1.2.2" xref="S4.E2.m1.2.2.1.1.2.2.cmml"><mrow id="S4.E2.m1.2.2.1.1.1.1.1.1" xref="S4.E2.m1.2.2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.E2.m1.2.2.1.1.1.1.1.1.2" xref="S4.E2.m1.2.2.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.E2.m1.2.2.1.1.1.1.1.1.1" xref="S4.E2.m1.2.2.1.1.1.1.1.1.1.cmml"><mn id="S4.E2.m1.2.2.1.1.1.1.1.1.1.2" xref="S4.E2.m1.2.2.1.1.1.1.1.1.1.2.cmml">1</mn><mo id="S4.E2.m1.2.2.1.1.1.1.1.1.1.1" xref="S4.E2.m1.2.2.1.1.1.1.1.1.1.1.cmml">−</mo><mover accent="true" id="S4.E2.m1.2.2.1.1.1.1.1.1.1.3" xref="S4.E2.m1.2.2.1.1.1.1.1.1.1.3.cmml"><mi id="S4.E2.m1.2.2.1.1.1.1.1.1.1.3.2" xref="S4.E2.m1.2.2.1.1.1.1.1.1.1.3.2.cmml">θ</mi><mo id="S4.E2.m1.2.2.1.1.1.1.1.1.1.3.1" xref="S4.E2.m1.2.2.1.1.1.1.1.1.1.3.1.cmml">¯</mo></mover></mrow><mo stretchy="false" id="S4.E2.m1.2.2.1.1.1.1.1.1.3" xref="S4.E2.m1.2.2.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S4.E2.m1.2.2.1.1.2.2.3" xref="S4.E2.m1.2.2.1.1.2.2.3.cmml">​</mo><mi id="S4.E2.m1.2.2.1.1.2.2.4" xref="S4.E2.m1.2.2.1.1.2.2.4.cmml">e</mi><mo lspace="0em" rspace="0em" id="S4.E2.m1.2.2.1.1.2.2.3a" xref="S4.E2.m1.2.2.1.1.2.2.3.cmml">​</mo><mi id="S4.E2.m1.2.2.1.1.2.2.5" xref="S4.E2.m1.2.2.1.1.2.2.5.cmml">x</mi><mo lspace="0em" rspace="0em" id="S4.E2.m1.2.2.1.1.2.2.3b" xref="S4.E2.m1.2.2.1.1.2.2.3.cmml">​</mo><mi id="S4.E2.m1.2.2.1.1.2.2.6" xref="S4.E2.m1.2.2.1.1.2.2.6.cmml">p</mi><mo lspace="0em" rspace="0em" id="S4.E2.m1.2.2.1.1.2.2.3c" xref="S4.E2.m1.2.2.1.1.2.2.3.cmml">​</mo><mrow id="S4.E2.m1.2.2.1.1.2.2.2.1" xref="S4.E2.m1.2.2.1.1.2.2.2.1.1.cmml"><mo stretchy="false" id="S4.E2.m1.2.2.1.1.2.2.2.1.2" xref="S4.E2.m1.2.2.1.1.2.2.2.1.1.cmml">(</mo><mrow id="S4.E2.m1.2.2.1.1.2.2.2.1.1" xref="S4.E2.m1.2.2.1.1.2.2.2.1.1.cmml"><mo id="S4.E2.m1.2.2.1.1.2.2.2.1.1a" xref="S4.E2.m1.2.2.1.1.2.2.2.1.1.cmml">−</mo><mrow id="S4.E2.m1.2.2.1.1.2.2.2.1.1.2" xref="S4.E2.m1.2.2.1.1.2.2.2.1.1.2.cmml"><mi id="S4.E2.m1.2.2.1.1.2.2.2.1.1.2.2" xref="S4.E2.m1.2.2.1.1.2.2.2.1.1.2.2.cmml">γ</mi><mo lspace="0.222em" rspace="0.222em" id="S4.E2.m1.2.2.1.1.2.2.2.1.1.2.1" xref="S4.E2.m1.2.2.1.1.2.2.2.1.1.2.1.cmml">⋅</mo><mi id="S4.E2.m1.2.2.1.1.2.2.2.1.1.2.3" xref="S4.E2.m1.2.2.1.1.2.2.2.1.1.2.3.cmml">t</mi></mrow></mrow><mo stretchy="false" id="S4.E2.m1.2.2.1.1.2.2.2.1.3" xref="S4.E2.m1.2.2.1.1.2.2.2.1.1.cmml">)</mo></mrow></mrow><mo id="S4.E2.m1.2.2.1.1.2.3" xref="S4.E2.m1.2.2.1.1.2.3.cmml">+</mo><mover accent="true" id="S4.E2.m1.2.2.1.1.2.4" xref="S4.E2.m1.2.2.1.1.2.4.cmml"><mi id="S4.E2.m1.2.2.1.1.2.4.2" xref="S4.E2.m1.2.2.1.1.2.4.2.cmml">θ</mi><mo id="S4.E2.m1.2.2.1.1.2.4.1" xref="S4.E2.m1.2.2.1.1.2.4.1.cmml">¯</mo></mover></mrow></mrow><mo id="S4.E2.m1.3.3.2.3" xref="S4.E2.m1.3.3.3a.cmml">,</mo><mrow id="S4.E2.m1.3.3.2.2" xref="S4.E2.m1.3.3.2.2.cmml"><mi id="S4.E2.m1.3.3.2.2.2" xref="S4.E2.m1.3.3.2.2.2.cmml">γ</mi><mo id="S4.E2.m1.3.3.2.2.1" xref="S4.E2.m1.3.3.2.2.1.cmml">&gt;</mo><mn id="S4.E2.m1.3.3.2.2.3" xref="S4.E2.m1.3.3.2.2.3.cmml">0</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.E2.m1.3b"><apply id="S4.E2.m1.3.3.3.cmml" xref="S4.E2.m1.3.3.2"><csymbol cd="ambiguous" id="S4.E2.m1.3.3.3a.cmml" xref="S4.E2.m1.3.3.2.3">formulae-sequence</csymbol><apply id="S4.E2.m1.2.2.1.1.cmml" xref="S4.E2.m1.2.2.1.1"><eq id="S4.E2.m1.2.2.1.1.3.cmml" xref="S4.E2.m1.2.2.1.1.3"></eq><apply id="S4.E2.m1.2.2.1.1.4.cmml" xref="S4.E2.m1.2.2.1.1.4"><times id="S4.E2.m1.2.2.1.1.4.1.cmml" xref="S4.E2.m1.2.2.1.1.4.1"></times><apply id="S4.E2.m1.2.2.1.1.4.2.cmml" xref="S4.E2.m1.2.2.1.1.4.2"><ci id="S4.E2.m1.2.2.1.1.4.2.1.cmml" xref="S4.E2.m1.2.2.1.1.4.2.1">¯</ci><ci id="S4.E2.m1.2.2.1.1.4.2.2.cmml" xref="S4.E2.m1.2.2.1.1.4.2.2">𝜃</ci></apply><ci id="S4.E2.m1.1.1.cmml" xref="S4.E2.m1.1.1">𝑡</ci></apply><apply id="S4.E2.m1.2.2.1.1.2.cmml" xref="S4.E2.m1.2.2.1.1.2"><plus id="S4.E2.m1.2.2.1.1.2.3.cmml" xref="S4.E2.m1.2.2.1.1.2.3"></plus><apply id="S4.E2.m1.2.2.1.1.2.2.cmml" xref="S4.E2.m1.2.2.1.1.2.2"><times id="S4.E2.m1.2.2.1.1.2.2.3.cmml" xref="S4.E2.m1.2.2.1.1.2.2.3"></times><apply id="S4.E2.m1.2.2.1.1.1.1.1.1.1.cmml" xref="S4.E2.m1.2.2.1.1.1.1.1.1"><minus id="S4.E2.m1.2.2.1.1.1.1.1.1.1.1.cmml" xref="S4.E2.m1.2.2.1.1.1.1.1.1.1.1"></minus><cn type="integer" id="S4.E2.m1.2.2.1.1.1.1.1.1.1.2.cmml" xref="S4.E2.m1.2.2.1.1.1.1.1.1.1.2">1</cn><apply id="S4.E2.m1.2.2.1.1.1.1.1.1.1.3.cmml" xref="S4.E2.m1.2.2.1.1.1.1.1.1.1.3"><ci id="S4.E2.m1.2.2.1.1.1.1.1.1.1.3.1.cmml" xref="S4.E2.m1.2.2.1.1.1.1.1.1.1.3.1">¯</ci><ci id="S4.E2.m1.2.2.1.1.1.1.1.1.1.3.2.cmml" xref="S4.E2.m1.2.2.1.1.1.1.1.1.1.3.2">𝜃</ci></apply></apply><ci id="S4.E2.m1.2.2.1.1.2.2.4.cmml" xref="S4.E2.m1.2.2.1.1.2.2.4">𝑒</ci><ci id="S4.E2.m1.2.2.1.1.2.2.5.cmml" xref="S4.E2.m1.2.2.1.1.2.2.5">𝑥</ci><ci id="S4.E2.m1.2.2.1.1.2.2.6.cmml" xref="S4.E2.m1.2.2.1.1.2.2.6">𝑝</ci><apply id="S4.E2.m1.2.2.1.1.2.2.2.1.1.cmml" xref="S4.E2.m1.2.2.1.1.2.2.2.1"><minus id="S4.E2.m1.2.2.1.1.2.2.2.1.1.1.cmml" xref="S4.E2.m1.2.2.1.1.2.2.2.1"></minus><apply id="S4.E2.m1.2.2.1.1.2.2.2.1.1.2.cmml" xref="S4.E2.m1.2.2.1.1.2.2.2.1.1.2"><ci id="S4.E2.m1.2.2.1.1.2.2.2.1.1.2.1.cmml" xref="S4.E2.m1.2.2.1.1.2.2.2.1.1.2.1">⋅</ci><ci id="S4.E2.m1.2.2.1.1.2.2.2.1.1.2.2.cmml" xref="S4.E2.m1.2.2.1.1.2.2.2.1.1.2.2">𝛾</ci><ci id="S4.E2.m1.2.2.1.1.2.2.2.1.1.2.3.cmml" xref="S4.E2.m1.2.2.1.1.2.2.2.1.1.2.3">𝑡</ci></apply></apply></apply><apply id="S4.E2.m1.2.2.1.1.2.4.cmml" xref="S4.E2.m1.2.2.1.1.2.4"><ci id="S4.E2.m1.2.2.1.1.2.4.1.cmml" xref="S4.E2.m1.2.2.1.1.2.4.1">¯</ci><ci id="S4.E2.m1.2.2.1.1.2.4.2.cmml" xref="S4.E2.m1.2.2.1.1.2.4.2">𝜃</ci></apply></apply></apply><apply id="S4.E2.m1.3.3.2.2.cmml" xref="S4.E2.m1.3.3.2.2"><gt id="S4.E2.m1.3.3.2.2.1.cmml" xref="S4.E2.m1.3.3.2.2.1"></gt><ci id="S4.E2.m1.3.3.2.2.2.cmml" xref="S4.E2.m1.3.3.2.2.2">𝛾</ci><cn type="integer" id="S4.E2.m1.3.3.2.2.3.cmml" xref="S4.E2.m1.3.3.2.2.3">0</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E2.m1.3c">\bar{\theta}(t)=(1-\bar{\theta})exp(-\gamma\cdot t)+\bar{\theta},\gamma&gt;0</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
</div>
<div id="S4.SS2.SSS0.Px1.p2" class="ltx_para">
<p id="S4.SS2.SSS0.Px1.p2.13" class="ltx_p">By considering Fig. <a href="#S4.F6" title="Figure 6 ‣ Identity mapping reordering. ‣ 4.1 Switchable-Transformer Blocks ‣ 4 Our Approach: Progressive Layer Dropping ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>, we provide intuitive and straightforward motivation for our choice. The blue curve in Fig. <a href="#S4.F6" title="Figure 6 ‣ Identity mapping reordering. ‣ 4.1 Switchable-Transformer Blocks ‣ 4 Our Approach: Progressive Layer Dropping ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> are polynomials of increasing degree <math id="S4.SS2.SSS0.Px1.p2.1.m1.1" class="ltx_math_unparsed" alttext="\delta=\{1,..,8\}" display="inline"><semantics id="S4.SS2.SSS0.Px1.p2.1.m1.1a"><mrow id="S4.SS2.SSS0.Px1.p2.1.m1.1b"><mi id="S4.SS2.SSS0.Px1.p2.1.m1.1.1">δ</mi><mo id="S4.SS2.SSS0.Px1.p2.1.m1.1.2">=</mo><mrow id="S4.SS2.SSS0.Px1.p2.1.m1.1.3"><mo stretchy="false" id="S4.SS2.SSS0.Px1.p2.1.m1.1.3.1">{</mo><mn id="S4.SS2.SSS0.Px1.p2.1.m1.1.3.2">1</mn><mo id="S4.SS2.SSS0.Px1.p2.1.m1.1.3.3">,</mo><mo lspace="0em" rspace="0.0835em" id="S4.SS2.SSS0.Px1.p2.1.m1.1.3.4">.</mo><mo lspace="0.0835em" rspace="0.167em" id="S4.SS2.SSS0.Px1.p2.1.m1.1.3.5">.</mo><mo id="S4.SS2.SSS0.Px1.p2.1.m1.1.3.6">,</mo><mn id="S4.SS2.SSS0.Px1.p2.1.m1.1.3.7">8</mn><mo stretchy="false" id="S4.SS2.SSS0.Px1.p2.1.m1.1.3.8">}</mo></mrow></mrow><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p2.1.m1.1c">\delta=\{1,..,8\}</annotation></semantics></math> (left to right). Despite fulfilling the initial constraint <math id="S4.SS2.SSS0.Px1.p2.2.m2.1" class="ltx_Math" alttext="\theta(0)=1" display="inline"><semantics id="S4.SS2.SSS0.Px1.p2.2.m2.1a"><mrow id="S4.SS2.SSS0.Px1.p2.2.m2.1.2" xref="S4.SS2.SSS0.Px1.p2.2.m2.1.2.cmml"><mrow id="S4.SS2.SSS0.Px1.p2.2.m2.1.2.2" xref="S4.SS2.SSS0.Px1.p2.2.m2.1.2.2.cmml"><mi id="S4.SS2.SSS0.Px1.p2.2.m2.1.2.2.2" xref="S4.SS2.SSS0.Px1.p2.2.m2.1.2.2.2.cmml">θ</mi><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px1.p2.2.m2.1.2.2.1" xref="S4.SS2.SSS0.Px1.p2.2.m2.1.2.2.1.cmml">​</mo><mrow id="S4.SS2.SSS0.Px1.p2.2.m2.1.2.2.3.2" xref="S4.SS2.SSS0.Px1.p2.2.m2.1.2.2.cmml"><mo stretchy="false" id="S4.SS2.SSS0.Px1.p2.2.m2.1.2.2.3.2.1" xref="S4.SS2.SSS0.Px1.p2.2.m2.1.2.2.cmml">(</mo><mn id="S4.SS2.SSS0.Px1.p2.2.m2.1.1" xref="S4.SS2.SSS0.Px1.p2.2.m2.1.1.cmml">0</mn><mo stretchy="false" id="S4.SS2.SSS0.Px1.p2.2.m2.1.2.2.3.2.2" xref="S4.SS2.SSS0.Px1.p2.2.m2.1.2.2.cmml">)</mo></mrow></mrow><mo id="S4.SS2.SSS0.Px1.p2.2.m2.1.2.1" xref="S4.SS2.SSS0.Px1.p2.2.m2.1.2.1.cmml">=</mo><mn id="S4.SS2.SSS0.Px1.p2.2.m2.1.2.3" xref="S4.SS2.SSS0.Px1.p2.2.m2.1.2.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p2.2.m2.1b"><apply id="S4.SS2.SSS0.Px1.p2.2.m2.1.2.cmml" xref="S4.SS2.SSS0.Px1.p2.2.m2.1.2"><eq id="S4.SS2.SSS0.Px1.p2.2.m2.1.2.1.cmml" xref="S4.SS2.SSS0.Px1.p2.2.m2.1.2.1"></eq><apply id="S4.SS2.SSS0.Px1.p2.2.m2.1.2.2.cmml" xref="S4.SS2.SSS0.Px1.p2.2.m2.1.2.2"><times id="S4.SS2.SSS0.Px1.p2.2.m2.1.2.2.1.cmml" xref="S4.SS2.SSS0.Px1.p2.2.m2.1.2.2.1"></times><ci id="S4.SS2.SSS0.Px1.p2.2.m2.1.2.2.2.cmml" xref="S4.SS2.SSS0.Px1.p2.2.m2.1.2.2.2">𝜃</ci><cn type="integer" id="S4.SS2.SSS0.Px1.p2.2.m2.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.2.m2.1.1">0</cn></apply><cn type="integer" id="S4.SS2.SSS0.Px1.p2.2.m2.1.2.3.cmml" xref="S4.SS2.SSS0.Px1.p2.2.m2.1.2.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p2.2.m2.1c">\theta(0)=1</annotation></semantics></math>, they have to be manually thresholded to impose <math id="S4.SS2.SSS0.Px1.p2.3.m3.1" class="ltx_Math" alttext="\theta(t)\rightarrow\bar{\theta}" display="inline"><semantics id="S4.SS2.SSS0.Px1.p2.3.m3.1a"><mrow id="S4.SS2.SSS0.Px1.p2.3.m3.1.2" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.2.cmml"><mrow id="S4.SS2.SSS0.Px1.p2.3.m3.1.2.2" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.2.2.cmml"><mi id="S4.SS2.SSS0.Px1.p2.3.m3.1.2.2.2" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.2.2.2.cmml">θ</mi><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px1.p2.3.m3.1.2.2.1" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.2.2.1.cmml">​</mo><mrow id="S4.SS2.SSS0.Px1.p2.3.m3.1.2.2.3.2" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.2.2.cmml"><mo stretchy="false" id="S4.SS2.SSS0.Px1.p2.3.m3.1.2.2.3.2.1" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.2.2.cmml">(</mo><mi id="S4.SS2.SSS0.Px1.p2.3.m3.1.1" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.1.cmml">t</mi><mo stretchy="false" id="S4.SS2.SSS0.Px1.p2.3.m3.1.2.2.3.2.2" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.2.2.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S4.SS2.SSS0.Px1.p2.3.m3.1.2.1" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.2.1.cmml">→</mo><mover accent="true" id="S4.SS2.SSS0.Px1.p2.3.m3.1.2.3" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.2.3.cmml"><mi id="S4.SS2.SSS0.Px1.p2.3.m3.1.2.3.2" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.2.3.2.cmml">θ</mi><mo id="S4.SS2.SSS0.Px1.p2.3.m3.1.2.3.1" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.2.3.1.cmml">¯</mo></mover></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p2.3.m3.1b"><apply id="S4.SS2.SSS0.Px1.p2.3.m3.1.2.cmml" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.2"><ci id="S4.SS2.SSS0.Px1.p2.3.m3.1.2.1.cmml" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.2.1">→</ci><apply id="S4.SS2.SSS0.Px1.p2.3.m3.1.2.2.cmml" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.2.2"><times id="S4.SS2.SSS0.Px1.p2.3.m3.1.2.2.1.cmml" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.2.2.1"></times><ci id="S4.SS2.SSS0.Px1.p2.3.m3.1.2.2.2.cmml" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.2.2.2">𝜃</ci><ci id="S4.SS2.SSS0.Px1.p2.3.m3.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.1">𝑡</ci></apply><apply id="S4.SS2.SSS0.Px1.p2.3.m3.1.2.3.cmml" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.2.3"><ci id="S4.SS2.SSS0.Px1.p2.3.m3.1.2.3.1.cmml" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.2.3.1">¯</ci><ci id="S4.SS2.SSS0.Px1.p2.3.m3.1.2.3.2.cmml" xref="S4.SS2.SSS0.Px1.p2.3.m3.1.2.3.2">𝜃</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p2.3.m3.1c">\theta(t)\rightarrow\bar{\theta}</annotation></semantics></math> when <math id="S4.SS2.SSS0.Px1.p2.4.m4.1" class="ltx_Math" alttext="t\rightarrow\infty" display="inline"><semantics id="S4.SS2.SSS0.Px1.p2.4.m4.1a"><mrow id="S4.SS2.SSS0.Px1.p2.4.m4.1.1" xref="S4.SS2.SSS0.Px1.p2.4.m4.1.1.cmml"><mi id="S4.SS2.SSS0.Px1.p2.4.m4.1.1.2" xref="S4.SS2.SSS0.Px1.p2.4.m4.1.1.2.cmml">t</mi><mo stretchy="false" id="S4.SS2.SSS0.Px1.p2.4.m4.1.1.1" xref="S4.SS2.SSS0.Px1.p2.4.m4.1.1.1.cmml">→</mo><mi mathvariant="normal" id="S4.SS2.SSS0.Px1.p2.4.m4.1.1.3" xref="S4.SS2.SSS0.Px1.p2.4.m4.1.1.3.cmml">∞</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p2.4.m4.1b"><apply id="S4.SS2.SSS0.Px1.p2.4.m4.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.4.m4.1.1"><ci id="S4.SS2.SSS0.Px1.p2.4.m4.1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.4.m4.1.1.1">→</ci><ci id="S4.SS2.SSS0.Px1.p2.4.m4.1.1.2.cmml" xref="S4.SS2.SSS0.Px1.p2.4.m4.1.1.2">𝑡</ci><infinity id="S4.SS2.SSS0.Px1.p2.4.m4.1.1.3.cmml" xref="S4.SS2.SSS0.Px1.p2.4.m4.1.1.3"></infinity></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p2.4.m4.1c">t\rightarrow\infty</annotation></semantics></math>, which introduces two more parameters (<math id="S4.SS2.SSS0.Px1.p2.5.m5.1" class="ltx_Math" alttext="\delta" display="inline"><semantics id="S4.SS2.SSS0.Px1.p2.5.m5.1a"><mi id="S4.SS2.SSS0.Px1.p2.5.m5.1.1" xref="S4.SS2.SSS0.Px1.p2.5.m5.1.1.cmml">δ</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p2.5.m5.1b"><ci id="S4.SS2.SSS0.Px1.p2.5.m5.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.5.m5.1.1">𝛿</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p2.5.m5.1c">\delta</annotation></semantics></math> and the threshold).

In contrast, in our schedule, we fix <math id="S4.SS2.SSS0.Px1.p2.6.m6.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="S4.SS2.SSS0.Px1.p2.6.m6.1a"><mi id="S4.SS2.SSS0.Px1.p2.6.m6.1.1" xref="S4.SS2.SSS0.Px1.p2.6.m6.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p2.6.m6.1b"><ci id="S4.SS2.SSS0.Px1.p2.6.m6.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.6.m6.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p2.6.m6.1c">\gamma</annotation></semantics></math> using the following simple heuristics <math id="S4.SS2.SSS0.Px1.p2.7.m7.1" class="ltx_Math" alttext="\gamma=\frac{100}{T}" display="inline"><semantics id="S4.SS2.SSS0.Px1.p2.7.m7.1a"><mrow id="S4.SS2.SSS0.Px1.p2.7.m7.1.1" xref="S4.SS2.SSS0.Px1.p2.7.m7.1.1.cmml"><mi id="S4.SS2.SSS0.Px1.p2.7.m7.1.1.2" xref="S4.SS2.SSS0.Px1.p2.7.m7.1.1.2.cmml">γ</mi><mo id="S4.SS2.SSS0.Px1.p2.7.m7.1.1.1" xref="S4.SS2.SSS0.Px1.p2.7.m7.1.1.1.cmml">=</mo><mfrac id="S4.SS2.SSS0.Px1.p2.7.m7.1.1.3" xref="S4.SS2.SSS0.Px1.p2.7.m7.1.1.3.cmml"><mn id="S4.SS2.SSS0.Px1.p2.7.m7.1.1.3.2" xref="S4.SS2.SSS0.Px1.p2.7.m7.1.1.3.2.cmml">100</mn><mi id="S4.SS2.SSS0.Px1.p2.7.m7.1.1.3.3" xref="S4.SS2.SSS0.Px1.p2.7.m7.1.1.3.3.cmml">T</mi></mfrac></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p2.7.m7.1b"><apply id="S4.SS2.SSS0.Px1.p2.7.m7.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.7.m7.1.1"><eq id="S4.SS2.SSS0.Px1.p2.7.m7.1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.7.m7.1.1.1"></eq><ci id="S4.SS2.SSS0.Px1.p2.7.m7.1.1.2.cmml" xref="S4.SS2.SSS0.Px1.p2.7.m7.1.1.2">𝛾</ci><apply id="S4.SS2.SSS0.Px1.p2.7.m7.1.1.3.cmml" xref="S4.SS2.SSS0.Px1.p2.7.m7.1.1.3"><divide id="S4.SS2.SSS0.Px1.p2.7.m7.1.1.3.1.cmml" xref="S4.SS2.SSS0.Px1.p2.7.m7.1.1.3"></divide><cn type="integer" id="S4.SS2.SSS0.Px1.p2.7.m7.1.1.3.2.cmml" xref="S4.SS2.SSS0.Px1.p2.7.m7.1.1.3.2">100</cn><ci id="S4.SS2.SSS0.Px1.p2.7.m7.1.1.3.3.cmml" xref="S4.SS2.SSS0.Px1.p2.7.m7.1.1.3.3">𝑇</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p2.7.m7.1c">\gamma=\frac{100}{T}</annotation></semantics></math>, since it implies <math id="S4.SS2.SSS0.Px1.p2.8.m8.2" class="ltx_Math" alttext="|\theta(T)-\bar{\theta}|&lt;10^{-5}" display="inline"><semantics id="S4.SS2.SSS0.Px1.p2.8.m8.2a"><mrow id="S4.SS2.SSS0.Px1.p2.8.m8.2.2" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.cmml"><mrow id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.2.cmml"><mo stretchy="false" id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.2" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.2.1.cmml">|</mo><mrow id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.cmml"><mrow id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.2" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.2.cmml"><mi id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.2.2" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.2.2.cmml">θ</mi><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.2.1" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.2.1.cmml">​</mo><mrow id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.2.3.2" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.2.cmml"><mo stretchy="false" id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.2.3.2.1" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.2.cmml">(</mo><mi id="S4.SS2.SSS0.Px1.p2.8.m8.1.1" xref="S4.SS2.SSS0.Px1.p2.8.m8.1.1.cmml">T</mi><mo stretchy="false" id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.2.3.2.2" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.1" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.1.cmml">−</mo><mover accent="true" id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.3" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.3.cmml"><mi id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.3.2" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.3.2.cmml">θ</mi><mo id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.3.1" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.3.1.cmml">¯</mo></mover></mrow><mo stretchy="false" id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.3" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.2.1.cmml">|</mo></mrow><mo id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.2" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.2.cmml">&lt;</mo><msup id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3.cmml"><mn id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3.2" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3.2.cmml">10</mn><mrow id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3.3" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3.3.cmml"><mo id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3.3a" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3.3.cmml">−</mo><mn id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3.3.2" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p2.8.m8.2b"><apply id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.cmml" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2"><lt id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.2.cmml" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.2"></lt><apply id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.2.cmml" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1"><abs id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.2.1.cmml" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.2"></abs><apply id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1"><minus id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.1"></minus><apply id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.2.cmml" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.2"><times id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.2.1.cmml" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.2.1"></times><ci id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.2.2.cmml" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.2.2">𝜃</ci><ci id="S4.SS2.SSS0.Px1.p2.8.m8.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.8.m8.1.1">𝑇</ci></apply><apply id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.3.cmml" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.3"><ci id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.3.1.cmml" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.3.1">¯</ci><ci id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.3.2.cmml" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.1.1.1.3.2">𝜃</ci></apply></apply></apply><apply id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3.cmml" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3.1.cmml" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3">superscript</csymbol><cn type="integer" id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3.2.cmml" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3.2">10</cn><apply id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3.3.cmml" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3.3"><minus id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3.3.1.cmml" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3.3"></minus><cn type="integer" id="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3.3.2.cmml" xref="S4.SS2.SSS0.Px1.p2.8.m8.2.2.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p2.8.m8.2c">|\theta(T)-\bar{\theta}|&lt;10^{-5}</annotation></semantics></math> for <math id="S4.SS2.SSS0.Px1.p2.9.m9.1" class="ltx_Math" alttext="\theta(t)\approx\bar{\theta}" display="inline"><semantics id="S4.SS2.SSS0.Px1.p2.9.m9.1a"><mrow id="S4.SS2.SSS0.Px1.p2.9.m9.1.2" xref="S4.SS2.SSS0.Px1.p2.9.m9.1.2.cmml"><mrow id="S4.SS2.SSS0.Px1.p2.9.m9.1.2.2" xref="S4.SS2.SSS0.Px1.p2.9.m9.1.2.2.cmml"><mi id="S4.SS2.SSS0.Px1.p2.9.m9.1.2.2.2" xref="S4.SS2.SSS0.Px1.p2.9.m9.1.2.2.2.cmml">θ</mi><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px1.p2.9.m9.1.2.2.1" xref="S4.SS2.SSS0.Px1.p2.9.m9.1.2.2.1.cmml">​</mo><mrow id="S4.SS2.SSS0.Px1.p2.9.m9.1.2.2.3.2" xref="S4.SS2.SSS0.Px1.p2.9.m9.1.2.2.cmml"><mo stretchy="false" id="S4.SS2.SSS0.Px1.p2.9.m9.1.2.2.3.2.1" xref="S4.SS2.SSS0.Px1.p2.9.m9.1.2.2.cmml">(</mo><mi id="S4.SS2.SSS0.Px1.p2.9.m9.1.1" xref="S4.SS2.SSS0.Px1.p2.9.m9.1.1.cmml">t</mi><mo stretchy="false" id="S4.SS2.SSS0.Px1.p2.9.m9.1.2.2.3.2.2" xref="S4.SS2.SSS0.Px1.p2.9.m9.1.2.2.cmml">)</mo></mrow></mrow><mo id="S4.SS2.SSS0.Px1.p2.9.m9.1.2.1" xref="S4.SS2.SSS0.Px1.p2.9.m9.1.2.1.cmml">≈</mo><mover accent="true" id="S4.SS2.SSS0.Px1.p2.9.m9.1.2.3" xref="S4.SS2.SSS0.Px1.p2.9.m9.1.2.3.cmml"><mi id="S4.SS2.SSS0.Px1.p2.9.m9.1.2.3.2" xref="S4.SS2.SSS0.Px1.p2.9.m9.1.2.3.2.cmml">θ</mi><mo id="S4.SS2.SSS0.Px1.p2.9.m9.1.2.3.1" xref="S4.SS2.SSS0.Px1.p2.9.m9.1.2.3.1.cmml">¯</mo></mover></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p2.9.m9.1b"><apply id="S4.SS2.SSS0.Px1.p2.9.m9.1.2.cmml" xref="S4.SS2.SSS0.Px1.p2.9.m9.1.2"><approx id="S4.SS2.SSS0.Px1.p2.9.m9.1.2.1.cmml" xref="S4.SS2.SSS0.Px1.p2.9.m9.1.2.1"></approx><apply id="S4.SS2.SSS0.Px1.p2.9.m9.1.2.2.cmml" xref="S4.SS2.SSS0.Px1.p2.9.m9.1.2.2"><times id="S4.SS2.SSS0.Px1.p2.9.m9.1.2.2.1.cmml" xref="S4.SS2.SSS0.Px1.p2.9.m9.1.2.2.1"></times><ci id="S4.SS2.SSS0.Px1.p2.9.m9.1.2.2.2.cmml" xref="S4.SS2.SSS0.Px1.p2.9.m9.1.2.2.2">𝜃</ci><ci id="S4.SS2.SSS0.Px1.p2.9.m9.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.9.m9.1.1">𝑡</ci></apply><apply id="S4.SS2.SSS0.Px1.p2.9.m9.1.2.3.cmml" xref="S4.SS2.SSS0.Px1.p2.9.m9.1.2.3"><ci id="S4.SS2.SSS0.Px1.p2.9.m9.1.2.3.1.cmml" xref="S4.SS2.SSS0.Px1.p2.9.m9.1.2.3.1">¯</ci><ci id="S4.SS2.SSS0.Px1.p2.9.m9.1.2.3.2.cmml" xref="S4.SS2.SSS0.Px1.p2.9.m9.1.2.3.2">𝜃</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p2.9.m9.1c">\theta(t)\approx\bar{\theta}</annotation></semantics></math> when <math id="S4.SS2.SSS0.Px1.p2.10.m10.1" class="ltx_Math" alttext="t\approx T" display="inline"><semantics id="S4.SS2.SSS0.Px1.p2.10.m10.1a"><mrow id="S4.SS2.SSS0.Px1.p2.10.m10.1.1" xref="S4.SS2.SSS0.Px1.p2.10.m10.1.1.cmml"><mi id="S4.SS2.SSS0.Px1.p2.10.m10.1.1.2" xref="S4.SS2.SSS0.Px1.p2.10.m10.1.1.2.cmml">t</mi><mo id="S4.SS2.SSS0.Px1.p2.10.m10.1.1.1" xref="S4.SS2.SSS0.Px1.p2.10.m10.1.1.1.cmml">≈</mo><mi id="S4.SS2.SSS0.Px1.p2.10.m10.1.1.3" xref="S4.SS2.SSS0.Px1.p2.10.m10.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p2.10.m10.1b"><apply id="S4.SS2.SSS0.Px1.p2.10.m10.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.10.m10.1.1"><approx id="S4.SS2.SSS0.Px1.p2.10.m10.1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.10.m10.1.1.1"></approx><ci id="S4.SS2.SSS0.Px1.p2.10.m10.1.1.2.cmml" xref="S4.SS2.SSS0.Px1.p2.10.m10.1.1.2">𝑡</ci><ci id="S4.SS2.SSS0.Px1.p2.10.m10.1.1.3.cmml" xref="S4.SS2.SSS0.Px1.p2.10.m10.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p2.10.m10.1c">t\approx T</annotation></semantics></math>, and it is reasonable to assume that T is at the order of magnitude of <math id="S4.SS2.SSS0.Px1.p2.11.m11.1" class="ltx_Math" alttext="10^{4}" display="inline"><semantics id="S4.SS2.SSS0.Px1.p2.11.m11.1a"><msup id="S4.SS2.SSS0.Px1.p2.11.m11.1.1" xref="S4.SS2.SSS0.Px1.p2.11.m11.1.1.cmml"><mn id="S4.SS2.SSS0.Px1.p2.11.m11.1.1.2" xref="S4.SS2.SSS0.Px1.p2.11.m11.1.1.2.cmml">10</mn><mn id="S4.SS2.SSS0.Px1.p2.11.m11.1.1.3" xref="S4.SS2.SSS0.Px1.p2.11.m11.1.1.3.cmml">4</mn></msup><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p2.11.m11.1b"><apply id="S4.SS2.SSS0.Px1.p2.11.m11.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.11.m11.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px1.p2.11.m11.1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.11.m11.1.1">superscript</csymbol><cn type="integer" id="S4.SS2.SSS0.Px1.p2.11.m11.1.1.2.cmml" xref="S4.SS2.SSS0.Px1.p2.11.m11.1.1.2">10</cn><cn type="integer" id="S4.SS2.SSS0.Px1.p2.11.m11.1.1.3.cmml" xref="S4.SS2.SSS0.Px1.p2.11.m11.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p2.11.m11.1c">10^{4}</annotation></semantics></math> or <math id="S4.SS2.SSS0.Px1.p2.12.m12.1" class="ltx_Math" alttext="10^{5}" display="inline"><semantics id="S4.SS2.SSS0.Px1.p2.12.m12.1a"><msup id="S4.SS2.SSS0.Px1.p2.12.m12.1.1" xref="S4.SS2.SSS0.Px1.p2.12.m12.1.1.cmml"><mn id="S4.SS2.SSS0.Px1.p2.12.m12.1.1.2" xref="S4.SS2.SSS0.Px1.p2.12.m12.1.1.2.cmml">10</mn><mn id="S4.SS2.SSS0.Px1.p2.12.m12.1.1.3" xref="S4.SS2.SSS0.Px1.p2.12.m12.1.1.3.cmml">5</mn></msup><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p2.12.m12.1b"><apply id="S4.SS2.SSS0.Px1.p2.12.m12.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.12.m12.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px1.p2.12.m12.1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.12.m12.1.1">superscript</csymbol><cn type="integer" id="S4.SS2.SSS0.Px1.p2.12.m12.1.1.2.cmml" xref="S4.SS2.SSS0.Px1.p2.12.m12.1.1.2">10</cn><cn type="integer" id="S4.SS2.SSS0.Px1.p2.12.m12.1.1.3.cmml" xref="S4.SS2.SSS0.Px1.p2.12.m12.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p2.12.m12.1c">10^{5}</annotation></semantics></math> when training Transformer networks. In other words, this means that for a big portion of the training, we are dropping (<math id="S4.SS2.SSS0.Px1.p2.13.m13.1" class="ltx_Math" alttext="1-\bar{\theta}" display="inline"><semantics id="S4.SS2.SSS0.Px1.p2.13.m13.1a"><mrow id="S4.SS2.SSS0.Px1.p2.13.m13.1.1" xref="S4.SS2.SSS0.Px1.p2.13.m13.1.1.cmml"><mn id="S4.SS2.SSS0.Px1.p2.13.m13.1.1.2" xref="S4.SS2.SSS0.Px1.p2.13.m13.1.1.2.cmml">1</mn><mo id="S4.SS2.SSS0.Px1.p2.13.m13.1.1.1" xref="S4.SS2.SSS0.Px1.p2.13.m13.1.1.1.cmml">−</mo><mover accent="true" id="S4.SS2.SSS0.Px1.p2.13.m13.1.1.3" xref="S4.SS2.SSS0.Px1.p2.13.m13.1.1.3.cmml"><mi id="S4.SS2.SSS0.Px1.p2.13.m13.1.1.3.2" xref="S4.SS2.SSS0.Px1.p2.13.m13.1.1.3.2.cmml">θ</mi><mo id="S4.SS2.SSS0.Px1.p2.13.m13.1.1.3.1" xref="S4.SS2.SSS0.Px1.p2.13.m13.1.1.3.1.cmml">¯</mo></mover></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px1.p2.13.m13.1b"><apply id="S4.SS2.SSS0.Px1.p2.13.m13.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.13.m13.1.1"><minus id="S4.SS2.SSS0.Px1.p2.13.m13.1.1.1.cmml" xref="S4.SS2.SSS0.Px1.p2.13.m13.1.1.1"></minus><cn type="integer" id="S4.SS2.SSS0.Px1.p2.13.m13.1.1.2.cmml" xref="S4.SS2.SSS0.Px1.p2.13.m13.1.1.2">1</cn><apply id="S4.SS2.SSS0.Px1.p2.13.m13.1.1.3.cmml" xref="S4.SS2.SSS0.Px1.p2.13.m13.1.1.3"><ci id="S4.SS2.SSS0.Px1.p2.13.m13.1.1.3.1.cmml" xref="S4.SS2.SSS0.Px1.p2.13.m13.1.1.3.1">¯</ci><ci id="S4.SS2.SSS0.Px1.p2.13.m13.1.1.3.2.cmml" xref="S4.SS2.SSS0.Px1.p2.13.m13.1.1.3.2">𝜃</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px1.p2.13.m13.1c">1-\bar{\theta}</annotation></semantics></math>) ST blocks, accelerating the training efficiency.</p>
</div>
</section>
<section id="S4.SS2.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Distribution along the depth dimension.</h4>

<div id="S4.SS2.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS2.SSS0.Px2.p1.6" class="ltx_p">The above progressive schedule assumes all gates in ST blocks take the same <math id="S4.SS2.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="p" display="inline"><semantics id="S4.SS2.SSS0.Px2.p1.1.m1.1a"><mi id="S4.SS2.SSS0.Px2.p1.1.m1.1.1" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1.cmml">p</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p1.1.m1.1b"><ci id="S4.SS2.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.1.m1.1.1">𝑝</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p1.1.m1.1c">p</annotation></semantics></math> value at each step t. However, as shown in Fig. <a href="#S3.F4" title="Figure 4 ‣ 3.2 Corroboration of Training Dynamics ‣ 3 Preliminary Analysis ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>,
the lower layers of the networks
should be more reliably present. Therefore, we distribute the global <math id="S4.SS2.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="\bar{\theta}" display="inline"><semantics id="S4.SS2.SSS0.Px2.p1.2.m2.1a"><mover accent="true" id="S4.SS2.SSS0.Px2.p1.2.m2.1.1" xref="S4.SS2.SSS0.Px2.p1.2.m2.1.1.cmml"><mi id="S4.SS2.SSS0.Px2.p1.2.m2.1.1.2" xref="S4.SS2.SSS0.Px2.p1.2.m2.1.1.2.cmml">θ</mi><mo id="S4.SS2.SSS0.Px2.p1.2.m2.1.1.1" xref="S4.SS2.SSS0.Px2.p1.2.m2.1.1.1.cmml">¯</mo></mover><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p1.2.m2.1b"><apply id="S4.SS2.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.2.m2.1.1"><ci id="S4.SS2.SSS0.Px2.p1.2.m2.1.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.2.m2.1.1.1">¯</ci><ci id="S4.SS2.SSS0.Px2.p1.2.m2.1.1.2.cmml" xref="S4.SS2.SSS0.Px2.p1.2.m2.1.1.2">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p1.2.m2.1c">\bar{\theta}</annotation></semantics></math> across the entire stack so that lower layers have lower drop probability linearly scaled by their depth according to equation <a href="#S4.E3" title="In Distribution along the depth dimension. ‣ 4.2 A Progressive Layer Dropping Schedule ‣ 4 Our Approach: Progressive Layer Dropping ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>.
Furthermore, we let the sub-layers inside each block share the same schedule, so
when <math id="S4.SS2.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="G_{i}" display="inline"><semantics id="S4.SS2.SSS0.Px2.p1.3.m3.1a"><msub id="S4.SS2.SSS0.Px2.p1.3.m3.1.1" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.cmml"><mi id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.cmml">G</mi><mi id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p1.3.m3.1b"><apply id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.2">𝐺</ci><ci id="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3.cmml" xref="S4.SS2.SSS0.Px2.p1.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p1.3.m3.1c">G_{i}</annotation></semantics></math> = 1, both the inner function <math id="S4.SS2.SSS0.Px2.p1.4.m4.1" class="ltx_Math" alttext="f_{ATTN}" display="inline"><semantics id="S4.SS2.SSS0.Px2.p1.4.m4.1a"><msub id="S4.SS2.SSS0.Px2.p1.4.m4.1.1" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.cmml"><mi id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.2" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.2.cmml">f</mi><mrow id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.cmml"><mi id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.2" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.2.cmml">A</mi><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.1" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.1.cmml">​</mo><mi id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.3" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.3.cmml">T</mi><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.1a" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.1.cmml">​</mo><mi id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.4" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.4.cmml">T</mi><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.1b" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.1.cmml">​</mo><mi id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.5" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.5.cmml">N</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p1.4.m4.1b"><apply id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.2.cmml" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.2">𝑓</ci><apply id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.cmml" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3"><times id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.1.cmml" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.1"></times><ci id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.2.cmml" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.2">𝐴</ci><ci id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.3.cmml" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.3">𝑇</ci><ci id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.4.cmml" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.4">𝑇</ci><ci id="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.5.cmml" xref="S4.SS2.SSS0.Px2.p1.4.m4.1.1.3.5">𝑁</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p1.4.m4.1c">f_{ATTN}</annotation></semantics></math> and <math id="S4.SS2.SSS0.Px2.p1.5.m5.1" class="ltx_Math" alttext="f_{FFN}" display="inline"><semantics id="S4.SS2.SSS0.Px2.p1.5.m5.1a"><msub id="S4.SS2.SSS0.Px2.p1.5.m5.1.1" xref="S4.SS2.SSS0.Px2.p1.5.m5.1.1.cmml"><mi id="S4.SS2.SSS0.Px2.p1.5.m5.1.1.2" xref="S4.SS2.SSS0.Px2.p1.5.m5.1.1.2.cmml">f</mi><mrow id="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3" xref="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3.cmml"><mi id="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3.2" xref="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3.2.cmml">F</mi><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3.1" xref="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3.1.cmml">​</mo><mi id="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3.3" xref="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3.3.cmml">F</mi><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3.1a" xref="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3.1.cmml">​</mo><mi id="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3.4" xref="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3.4.cmml">N</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p1.5.m5.1b"><apply id="S4.SS2.SSS0.Px2.p1.5.m5.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px2.p1.5.m5.1.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.5.m5.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px2.p1.5.m5.1.1.2.cmml" xref="S4.SS2.SSS0.Px2.p1.5.m5.1.1.2">𝑓</ci><apply id="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3.cmml" xref="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3"><times id="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3.1.cmml" xref="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3.1"></times><ci id="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3.2.cmml" xref="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3.2">𝐹</ci><ci id="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3.3.cmml" xref="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3.3">𝐹</ci><ci id="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3.4.cmml" xref="S4.SS2.SSS0.Px2.p1.5.m5.1.1.3.4">𝑁</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p1.5.m5.1c">f_{FFN}</annotation></semantics></math> are activated, while they are skipped when <math id="S4.SS2.SSS0.Px2.p1.6.m6.1" class="ltx_Math" alttext="G_{i}" display="inline"><semantics id="S4.SS2.SSS0.Px2.p1.6.m6.1a"><msub id="S4.SS2.SSS0.Px2.p1.6.m6.1.1" xref="S4.SS2.SSS0.Px2.p1.6.m6.1.1.cmml"><mi id="S4.SS2.SSS0.Px2.p1.6.m6.1.1.2" xref="S4.SS2.SSS0.Px2.p1.6.m6.1.1.2.cmml">G</mi><mi id="S4.SS2.SSS0.Px2.p1.6.m6.1.1.3" xref="S4.SS2.SSS0.Px2.p1.6.m6.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px2.p1.6.m6.1b"><apply id="S4.SS2.SSS0.Px2.p1.6.m6.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px2.p1.6.m6.1.1.1.cmml" xref="S4.SS2.SSS0.Px2.p1.6.m6.1.1">subscript</csymbol><ci id="S4.SS2.SSS0.Px2.p1.6.m6.1.1.2.cmml" xref="S4.SS2.SSS0.Px2.p1.6.m6.1.1.2">𝐺</ci><ci id="S4.SS2.SSS0.Px2.p1.6.m6.1.1.3.cmml" xref="S4.SS2.SSS0.Px2.p1.6.m6.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px2.p1.6.m6.1c">G_{i}</annotation></semantics></math> = 0.
Therefore, each gate has the following form during training:</p>
<table id="S4.E3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E3.m1.3" class="ltx_Math" alttext="p_{l}(t)=\frac{i}{L}(1-\bar{\theta}(t))" display="block"><semantics id="S4.E3.m1.3a"><mrow id="S4.E3.m1.3.3" xref="S4.E3.m1.3.3.cmml"><mrow id="S4.E3.m1.3.3.3" xref="S4.E3.m1.3.3.3.cmml"><msub id="S4.E3.m1.3.3.3.2" xref="S4.E3.m1.3.3.3.2.cmml"><mi id="S4.E3.m1.3.3.3.2.2" xref="S4.E3.m1.3.3.3.2.2.cmml">p</mi><mi id="S4.E3.m1.3.3.3.2.3" xref="S4.E3.m1.3.3.3.2.3.cmml">l</mi></msub><mo lspace="0em" rspace="0em" id="S4.E3.m1.3.3.3.1" xref="S4.E3.m1.3.3.3.1.cmml">​</mo><mrow id="S4.E3.m1.3.3.3.3.2" xref="S4.E3.m1.3.3.3.cmml"><mo stretchy="false" id="S4.E3.m1.3.3.3.3.2.1" xref="S4.E3.m1.3.3.3.cmml">(</mo><mi id="S4.E3.m1.1.1" xref="S4.E3.m1.1.1.cmml">t</mi><mo stretchy="false" id="S4.E3.m1.3.3.3.3.2.2" xref="S4.E3.m1.3.3.3.cmml">)</mo></mrow></mrow><mo id="S4.E3.m1.3.3.2" xref="S4.E3.m1.3.3.2.cmml">=</mo><mrow id="S4.E3.m1.3.3.1" xref="S4.E3.m1.3.3.1.cmml"><mfrac id="S4.E3.m1.3.3.1.3" xref="S4.E3.m1.3.3.1.3.cmml"><mi id="S4.E3.m1.3.3.1.3.2" xref="S4.E3.m1.3.3.1.3.2.cmml">i</mi><mi id="S4.E3.m1.3.3.1.3.3" xref="S4.E3.m1.3.3.1.3.3.cmml">L</mi></mfrac><mo lspace="0em" rspace="0em" id="S4.E3.m1.3.3.1.2" xref="S4.E3.m1.3.3.1.2.cmml">​</mo><mrow id="S4.E3.m1.3.3.1.1.1" xref="S4.E3.m1.3.3.1.1.1.1.cmml"><mo stretchy="false" id="S4.E3.m1.3.3.1.1.1.2" xref="S4.E3.m1.3.3.1.1.1.1.cmml">(</mo><mrow id="S4.E3.m1.3.3.1.1.1.1" xref="S4.E3.m1.3.3.1.1.1.1.cmml"><mn id="S4.E3.m1.3.3.1.1.1.1.2" xref="S4.E3.m1.3.3.1.1.1.1.2.cmml">1</mn><mo id="S4.E3.m1.3.3.1.1.1.1.1" xref="S4.E3.m1.3.3.1.1.1.1.1.cmml">−</mo><mrow id="S4.E3.m1.3.3.1.1.1.1.3" xref="S4.E3.m1.3.3.1.1.1.1.3.cmml"><mover accent="true" id="S4.E3.m1.3.3.1.1.1.1.3.2" xref="S4.E3.m1.3.3.1.1.1.1.3.2.cmml"><mi id="S4.E3.m1.3.3.1.1.1.1.3.2.2" xref="S4.E3.m1.3.3.1.1.1.1.3.2.2.cmml">θ</mi><mo id="S4.E3.m1.3.3.1.1.1.1.3.2.1" xref="S4.E3.m1.3.3.1.1.1.1.3.2.1.cmml">¯</mo></mover><mo lspace="0em" rspace="0em" id="S4.E3.m1.3.3.1.1.1.1.3.1" xref="S4.E3.m1.3.3.1.1.1.1.3.1.cmml">​</mo><mrow id="S4.E3.m1.3.3.1.1.1.1.3.3.2" xref="S4.E3.m1.3.3.1.1.1.1.3.cmml"><mo stretchy="false" id="S4.E3.m1.3.3.1.1.1.1.3.3.2.1" xref="S4.E3.m1.3.3.1.1.1.1.3.cmml">(</mo><mi id="S4.E3.m1.2.2" xref="S4.E3.m1.2.2.cmml">t</mi><mo stretchy="false" id="S4.E3.m1.3.3.1.1.1.1.3.3.2.2" xref="S4.E3.m1.3.3.1.1.1.1.3.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S4.E3.m1.3.3.1.1.1.3" xref="S4.E3.m1.3.3.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.E3.m1.3b"><apply id="S4.E3.m1.3.3.cmml" xref="S4.E3.m1.3.3"><eq id="S4.E3.m1.3.3.2.cmml" xref="S4.E3.m1.3.3.2"></eq><apply id="S4.E3.m1.3.3.3.cmml" xref="S4.E3.m1.3.3.3"><times id="S4.E3.m1.3.3.3.1.cmml" xref="S4.E3.m1.3.3.3.1"></times><apply id="S4.E3.m1.3.3.3.2.cmml" xref="S4.E3.m1.3.3.3.2"><csymbol cd="ambiguous" id="S4.E3.m1.3.3.3.2.1.cmml" xref="S4.E3.m1.3.3.3.2">subscript</csymbol><ci id="S4.E3.m1.3.3.3.2.2.cmml" xref="S4.E3.m1.3.3.3.2.2">𝑝</ci><ci id="S4.E3.m1.3.3.3.2.3.cmml" xref="S4.E3.m1.3.3.3.2.3">𝑙</ci></apply><ci id="S4.E3.m1.1.1.cmml" xref="S4.E3.m1.1.1">𝑡</ci></apply><apply id="S4.E3.m1.3.3.1.cmml" xref="S4.E3.m1.3.3.1"><times id="S4.E3.m1.3.3.1.2.cmml" xref="S4.E3.m1.3.3.1.2"></times><apply id="S4.E3.m1.3.3.1.3.cmml" xref="S4.E3.m1.3.3.1.3"><divide id="S4.E3.m1.3.3.1.3.1.cmml" xref="S4.E3.m1.3.3.1.3"></divide><ci id="S4.E3.m1.3.3.1.3.2.cmml" xref="S4.E3.m1.3.3.1.3.2">𝑖</ci><ci id="S4.E3.m1.3.3.1.3.3.cmml" xref="S4.E3.m1.3.3.1.3.3">𝐿</ci></apply><apply id="S4.E3.m1.3.3.1.1.1.1.cmml" xref="S4.E3.m1.3.3.1.1.1"><minus id="S4.E3.m1.3.3.1.1.1.1.1.cmml" xref="S4.E3.m1.3.3.1.1.1.1.1"></minus><cn type="integer" id="S4.E3.m1.3.3.1.1.1.1.2.cmml" xref="S4.E3.m1.3.3.1.1.1.1.2">1</cn><apply id="S4.E3.m1.3.3.1.1.1.1.3.cmml" xref="S4.E3.m1.3.3.1.1.1.1.3"><times id="S4.E3.m1.3.3.1.1.1.1.3.1.cmml" xref="S4.E3.m1.3.3.1.1.1.1.3.1"></times><apply id="S4.E3.m1.3.3.1.1.1.1.3.2.cmml" xref="S4.E3.m1.3.3.1.1.1.1.3.2"><ci id="S4.E3.m1.3.3.1.1.1.1.3.2.1.cmml" xref="S4.E3.m1.3.3.1.1.1.1.3.2.1">¯</ci><ci id="S4.E3.m1.3.3.1.1.1.1.3.2.2.cmml" xref="S4.E3.m1.3.3.1.1.1.1.3.2.2">𝜃</ci></apply><ci id="S4.E3.m1.2.2.cmml" xref="S4.E3.m1.2.2">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E3.m1.3c">p_{l}(t)=\frac{i}{L}(1-\bar{\theta}(t))</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
</div>
<div id="S4.SS2.SSS0.Px2.p2" class="ltx_para">
<p id="S4.SS2.SSS0.Px2.p2.1" class="ltx_p">Combining Eqn. <a href="#S4.E3" title="In Distribution along the depth dimension. ‣ 4.2 A Progressive Layer Dropping Schedule ‣ 4 Our Approach: Progressive Layer Dropping ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> and Eqn. <a href="#S4.E2" title="In Progress along the time dimension. ‣ 4.2 A Progressive Layer Dropping Schedule ‣ 4 Our Approach: Progressive Layer Dropping ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, we have the progressive schedule for an ST block below.</p>
<table id="S4.E4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E4.m1.4" class="ltx_Math" alttext="\theta_{i}(t)=\frac{i}{L}(1-(1-\bar{\theta}(t))exp(-\gamma\cdot t)-\bar{\theta}(t))" display="block"><semantics id="S4.E4.m1.4a"><mrow id="S4.E4.m1.4.4" xref="S4.E4.m1.4.4.cmml"><mrow id="S4.E4.m1.4.4.3" xref="S4.E4.m1.4.4.3.cmml"><msub id="S4.E4.m1.4.4.3.2" xref="S4.E4.m1.4.4.3.2.cmml"><mi id="S4.E4.m1.4.4.3.2.2" xref="S4.E4.m1.4.4.3.2.2.cmml">θ</mi><mi id="S4.E4.m1.4.4.3.2.3" xref="S4.E4.m1.4.4.3.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S4.E4.m1.4.4.3.1" xref="S4.E4.m1.4.4.3.1.cmml">​</mo><mrow id="S4.E4.m1.4.4.3.3.2" xref="S4.E4.m1.4.4.3.cmml"><mo stretchy="false" id="S4.E4.m1.4.4.3.3.2.1" xref="S4.E4.m1.4.4.3.cmml">(</mo><mi id="S4.E4.m1.1.1" xref="S4.E4.m1.1.1.cmml">t</mi><mo stretchy="false" id="S4.E4.m1.4.4.3.3.2.2" xref="S4.E4.m1.4.4.3.cmml">)</mo></mrow></mrow><mo id="S4.E4.m1.4.4.2" xref="S4.E4.m1.4.4.2.cmml">=</mo><mrow id="S4.E4.m1.4.4.1" xref="S4.E4.m1.4.4.1.cmml"><mfrac id="S4.E4.m1.4.4.1.3" xref="S4.E4.m1.4.4.1.3.cmml"><mi id="S4.E4.m1.4.4.1.3.2" xref="S4.E4.m1.4.4.1.3.2.cmml">i</mi><mi id="S4.E4.m1.4.4.1.3.3" xref="S4.E4.m1.4.4.1.3.3.cmml">L</mi></mfrac><mo lspace="0em" rspace="0em" id="S4.E4.m1.4.4.1.2" xref="S4.E4.m1.4.4.1.2.cmml">​</mo><mrow id="S4.E4.m1.4.4.1.1.1" xref="S4.E4.m1.4.4.1.1.1.1.cmml"><mo stretchy="false" id="S4.E4.m1.4.4.1.1.1.2" xref="S4.E4.m1.4.4.1.1.1.1.cmml">(</mo><mrow id="S4.E4.m1.4.4.1.1.1.1" xref="S4.E4.m1.4.4.1.1.1.1.cmml"><mn id="S4.E4.m1.4.4.1.1.1.1.4" xref="S4.E4.m1.4.4.1.1.1.1.4.cmml">1</mn><mo id="S4.E4.m1.4.4.1.1.1.1.3" xref="S4.E4.m1.4.4.1.1.1.1.3.cmml">−</mo><mrow id="S4.E4.m1.4.4.1.1.1.1.2" xref="S4.E4.m1.4.4.1.1.1.1.2.cmml"><mrow id="S4.E4.m1.4.4.1.1.1.1.1.1.1" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.E4.m1.4.4.1.1.1.1.1.1.1.2" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.E4.m1.4.4.1.1.1.1.1.1.1.1" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.cmml"><mn id="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.2" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.2.cmml">1</mn><mo id="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.1" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.1.cmml">−</mo><mrow id="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.cmml"><mover accent="true" id="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.2" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.2.cmml"><mi id="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.2.2" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.2.2.cmml">θ</mi><mo id="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.2.1" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.2.1.cmml">¯</mo></mover><mo lspace="0em" rspace="0em" id="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.1" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mrow id="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.3.2" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.cmml"><mo stretchy="false" id="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.3.2.1" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.cmml">(</mo><mi id="S4.E4.m1.2.2" xref="S4.E4.m1.2.2.cmml">t</mi><mo stretchy="false" id="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.3.2.2" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S4.E4.m1.4.4.1.1.1.1.1.1.1.3" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S4.E4.m1.4.4.1.1.1.1.2.3" xref="S4.E4.m1.4.4.1.1.1.1.2.3.cmml">​</mo><mi id="S4.E4.m1.4.4.1.1.1.1.2.4" xref="S4.E4.m1.4.4.1.1.1.1.2.4.cmml">e</mi><mo lspace="0em" rspace="0em" id="S4.E4.m1.4.4.1.1.1.1.2.3a" xref="S4.E4.m1.4.4.1.1.1.1.2.3.cmml">​</mo><mi id="S4.E4.m1.4.4.1.1.1.1.2.5" xref="S4.E4.m1.4.4.1.1.1.1.2.5.cmml">x</mi><mo lspace="0em" rspace="0em" id="S4.E4.m1.4.4.1.1.1.1.2.3b" xref="S4.E4.m1.4.4.1.1.1.1.2.3.cmml">​</mo><mi id="S4.E4.m1.4.4.1.1.1.1.2.6" xref="S4.E4.m1.4.4.1.1.1.1.2.6.cmml">p</mi><mo lspace="0em" rspace="0em" id="S4.E4.m1.4.4.1.1.1.1.2.3c" xref="S4.E4.m1.4.4.1.1.1.1.2.3.cmml">​</mo><mrow id="S4.E4.m1.4.4.1.1.1.1.2.2.1" xref="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.cmml"><mo stretchy="false" id="S4.E4.m1.4.4.1.1.1.1.2.2.1.2" xref="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.cmml">(</mo><mrow id="S4.E4.m1.4.4.1.1.1.1.2.2.1.1" xref="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.cmml"><mo id="S4.E4.m1.4.4.1.1.1.1.2.2.1.1a" xref="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.cmml">−</mo><mrow id="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.2" xref="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.2.cmml"><mi id="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.2.2" xref="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.2.2.cmml">γ</mi><mo lspace="0.222em" rspace="0.222em" id="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.2.1" xref="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.2.1.cmml">⋅</mo><mi id="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.2.3" xref="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.2.3.cmml">t</mi></mrow></mrow><mo stretchy="false" id="S4.E4.m1.4.4.1.1.1.1.2.2.1.3" xref="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.cmml">)</mo></mrow></mrow><mo id="S4.E4.m1.4.4.1.1.1.1.3a" xref="S4.E4.m1.4.4.1.1.1.1.3.cmml">−</mo><mrow id="S4.E4.m1.4.4.1.1.1.1.5" xref="S4.E4.m1.4.4.1.1.1.1.5.cmml"><mover accent="true" id="S4.E4.m1.4.4.1.1.1.1.5.2" xref="S4.E4.m1.4.4.1.1.1.1.5.2.cmml"><mi id="S4.E4.m1.4.4.1.1.1.1.5.2.2" xref="S4.E4.m1.4.4.1.1.1.1.5.2.2.cmml">θ</mi><mo id="S4.E4.m1.4.4.1.1.1.1.5.2.1" xref="S4.E4.m1.4.4.1.1.1.1.5.2.1.cmml">¯</mo></mover><mo lspace="0em" rspace="0em" id="S4.E4.m1.4.4.1.1.1.1.5.1" xref="S4.E4.m1.4.4.1.1.1.1.5.1.cmml">​</mo><mrow id="S4.E4.m1.4.4.1.1.1.1.5.3.2" xref="S4.E4.m1.4.4.1.1.1.1.5.cmml"><mo stretchy="false" id="S4.E4.m1.4.4.1.1.1.1.5.3.2.1" xref="S4.E4.m1.4.4.1.1.1.1.5.cmml">(</mo><mi id="S4.E4.m1.3.3" xref="S4.E4.m1.3.3.cmml">t</mi><mo stretchy="false" id="S4.E4.m1.4.4.1.1.1.1.5.3.2.2" xref="S4.E4.m1.4.4.1.1.1.1.5.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S4.E4.m1.4.4.1.1.1.3" xref="S4.E4.m1.4.4.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.E4.m1.4b"><apply id="S4.E4.m1.4.4.cmml" xref="S4.E4.m1.4.4"><eq id="S4.E4.m1.4.4.2.cmml" xref="S4.E4.m1.4.4.2"></eq><apply id="S4.E4.m1.4.4.3.cmml" xref="S4.E4.m1.4.4.3"><times id="S4.E4.m1.4.4.3.1.cmml" xref="S4.E4.m1.4.4.3.1"></times><apply id="S4.E4.m1.4.4.3.2.cmml" xref="S4.E4.m1.4.4.3.2"><csymbol cd="ambiguous" id="S4.E4.m1.4.4.3.2.1.cmml" xref="S4.E4.m1.4.4.3.2">subscript</csymbol><ci id="S4.E4.m1.4.4.3.2.2.cmml" xref="S4.E4.m1.4.4.3.2.2">𝜃</ci><ci id="S4.E4.m1.4.4.3.2.3.cmml" xref="S4.E4.m1.4.4.3.2.3">𝑖</ci></apply><ci id="S4.E4.m1.1.1.cmml" xref="S4.E4.m1.1.1">𝑡</ci></apply><apply id="S4.E4.m1.4.4.1.cmml" xref="S4.E4.m1.4.4.1"><times id="S4.E4.m1.4.4.1.2.cmml" xref="S4.E4.m1.4.4.1.2"></times><apply id="S4.E4.m1.4.4.1.3.cmml" xref="S4.E4.m1.4.4.1.3"><divide id="S4.E4.m1.4.4.1.3.1.cmml" xref="S4.E4.m1.4.4.1.3"></divide><ci id="S4.E4.m1.4.4.1.3.2.cmml" xref="S4.E4.m1.4.4.1.3.2">𝑖</ci><ci id="S4.E4.m1.4.4.1.3.3.cmml" xref="S4.E4.m1.4.4.1.3.3">𝐿</ci></apply><apply id="S4.E4.m1.4.4.1.1.1.1.cmml" xref="S4.E4.m1.4.4.1.1.1"><minus id="S4.E4.m1.4.4.1.1.1.1.3.cmml" xref="S4.E4.m1.4.4.1.1.1.1.3"></minus><cn type="integer" id="S4.E4.m1.4.4.1.1.1.1.4.cmml" xref="S4.E4.m1.4.4.1.1.1.1.4">1</cn><apply id="S4.E4.m1.4.4.1.1.1.1.2.cmml" xref="S4.E4.m1.4.4.1.1.1.1.2"><times id="S4.E4.m1.4.4.1.1.1.1.2.3.cmml" xref="S4.E4.m1.4.4.1.1.1.1.2.3"></times><apply id="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.cmml" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1"><minus id="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.1"></minus><cn type="integer" id="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.2.cmml" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.2">1</cn><apply id="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.cmml" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3"><times id="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.1.cmml" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.1"></times><apply id="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.2.cmml" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.2"><ci id="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.2.1.cmml" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.2.1">¯</ci><ci id="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.2.2.cmml" xref="S4.E4.m1.4.4.1.1.1.1.1.1.1.1.3.2.2">𝜃</ci></apply><ci id="S4.E4.m1.2.2.cmml" xref="S4.E4.m1.2.2">𝑡</ci></apply></apply><ci id="S4.E4.m1.4.4.1.1.1.1.2.4.cmml" xref="S4.E4.m1.4.4.1.1.1.1.2.4">𝑒</ci><ci id="S4.E4.m1.4.4.1.1.1.1.2.5.cmml" xref="S4.E4.m1.4.4.1.1.1.1.2.5">𝑥</ci><ci id="S4.E4.m1.4.4.1.1.1.1.2.6.cmml" xref="S4.E4.m1.4.4.1.1.1.1.2.6">𝑝</ci><apply id="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.cmml" xref="S4.E4.m1.4.4.1.1.1.1.2.2.1"><minus id="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.1.cmml" xref="S4.E4.m1.4.4.1.1.1.1.2.2.1"></minus><apply id="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.2.cmml" xref="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.2"><ci id="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.2.1.cmml" xref="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.2.1">⋅</ci><ci id="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.2.2.cmml" xref="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.2.2">𝛾</ci><ci id="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.2.3.cmml" xref="S4.E4.m1.4.4.1.1.1.1.2.2.1.1.2.3">𝑡</ci></apply></apply></apply><apply id="S4.E4.m1.4.4.1.1.1.1.5.cmml" xref="S4.E4.m1.4.4.1.1.1.1.5"><times id="S4.E4.m1.4.4.1.1.1.1.5.1.cmml" xref="S4.E4.m1.4.4.1.1.1.1.5.1"></times><apply id="S4.E4.m1.4.4.1.1.1.1.5.2.cmml" xref="S4.E4.m1.4.4.1.1.1.1.5.2"><ci id="S4.E4.m1.4.4.1.1.1.1.5.2.1.cmml" xref="S4.E4.m1.4.4.1.1.1.1.5.2.1">¯</ci><ci id="S4.E4.m1.4.4.1.1.1.1.5.2.2.cmml" xref="S4.E4.m1.4.4.1.1.1.1.5.2.2">𝜃</ci></apply><ci id="S4.E4.m1.3.3.cmml" xref="S4.E4.m1.3.3">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E4.m1.4c">\theta_{i}(t)=\frac{i}{L}(1-(1-\bar{\theta}(t))exp(-\gamma\cdot t)-\bar{\theta}(t))</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
</div>
<figure id="S4.SS2.SSS0.Px2.fig1" class="ltx_figure">
<figure id="alg1" class="ltx_float ltx_minipage ltx_align_middle ltx_framed ltx_framed_top" style="width:216.8pt;">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_float"><span id="alg1.3.1.1" class="ltx_text ltx_font_bold">Algorithm 1</span> </span>  <span id="alg1.4.2" class="ltx_text ltx_font_bold">Progressive_Layer_Dropping</span></figcaption>
<div id="alg1.5" class="ltx_listing ltx_align_center ltx_listing">
<div id="alg1.l1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l1.1.1.1" class="ltx_text" style="font-size:80%;">1:</span></span><span id="alg1.l1.2" class="ltx_text ltx_font_bold">Input:</span> <math id="alg1.l1.m1.1" class="ltx_Math" alttext="keep\_prob" display="inline"><semantics id="alg1.l1.m1.1a"><mrow id="alg1.l1.m1.1.1" xref="alg1.l1.m1.1.1.cmml"><mi id="alg1.l1.m1.1.1.2" xref="alg1.l1.m1.1.1.2.cmml">k</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m1.1.1.1" xref="alg1.l1.m1.1.1.1.cmml">​</mo><mi id="alg1.l1.m1.1.1.3" xref="alg1.l1.m1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m1.1.1.1a" xref="alg1.l1.m1.1.1.1.cmml">​</mo><mi id="alg1.l1.m1.1.1.4" xref="alg1.l1.m1.1.1.4.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m1.1.1.1b" xref="alg1.l1.m1.1.1.1.cmml">​</mo><mi id="alg1.l1.m1.1.1.5" xref="alg1.l1.m1.1.1.5.cmml">p</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m1.1.1.1c" xref="alg1.l1.m1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l1.m1.1.1.6" xref="alg1.l1.m1.1.1.6.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m1.1.1.1d" xref="alg1.l1.m1.1.1.1.cmml">​</mo><mi id="alg1.l1.m1.1.1.7" xref="alg1.l1.m1.1.1.7.cmml">p</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m1.1.1.1e" xref="alg1.l1.m1.1.1.1.cmml">​</mo><mi id="alg1.l1.m1.1.1.8" xref="alg1.l1.m1.1.1.8.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m1.1.1.1f" xref="alg1.l1.m1.1.1.1.cmml">​</mo><mi id="alg1.l1.m1.1.1.9" xref="alg1.l1.m1.1.1.9.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg1.l1.m1.1.1.1g" xref="alg1.l1.m1.1.1.1.cmml">​</mo><mi id="alg1.l1.m1.1.1.10" xref="alg1.l1.m1.1.1.10.cmml">b</mi></mrow><annotation-xml encoding="MathML-Content" id="alg1.l1.m1.1b"><apply id="alg1.l1.m1.1.1.cmml" xref="alg1.l1.m1.1.1"><times id="alg1.l1.m1.1.1.1.cmml" xref="alg1.l1.m1.1.1.1"></times><ci id="alg1.l1.m1.1.1.2.cmml" xref="alg1.l1.m1.1.1.2">𝑘</ci><ci id="alg1.l1.m1.1.1.3.cmml" xref="alg1.l1.m1.1.1.3">𝑒</ci><ci id="alg1.l1.m1.1.1.4.cmml" xref="alg1.l1.m1.1.1.4">𝑒</ci><ci id="alg1.l1.m1.1.1.5.cmml" xref="alg1.l1.m1.1.1.5">𝑝</ci><ci id="alg1.l1.m1.1.1.6.cmml" xref="alg1.l1.m1.1.1.6">_</ci><ci id="alg1.l1.m1.1.1.7.cmml" xref="alg1.l1.m1.1.1.7">𝑝</ci><ci id="alg1.l1.m1.1.1.8.cmml" xref="alg1.l1.m1.1.1.8">𝑟</ci><ci id="alg1.l1.m1.1.1.9.cmml" xref="alg1.l1.m1.1.1.9">𝑜</ci><ci id="alg1.l1.m1.1.1.10.cmml" xref="alg1.l1.m1.1.1.10">𝑏</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m1.1c">keep\_prob</annotation></semantics></math> <math id="alg1.l1.m2.1" class="ltx_Math" alttext="\bar{\theta}" display="inline"><semantics id="alg1.l1.m2.1a"><mover accent="true" id="alg1.l1.m2.1.1" xref="alg1.l1.m2.1.1.cmml"><mi id="alg1.l1.m2.1.1.2" xref="alg1.l1.m2.1.1.2.cmml">θ</mi><mo id="alg1.l1.m2.1.1.1" xref="alg1.l1.m2.1.1.1.cmml">¯</mo></mover><annotation-xml encoding="MathML-Content" id="alg1.l1.m2.1b"><apply id="alg1.l1.m2.1.1.cmml" xref="alg1.l1.m2.1.1"><ci id="alg1.l1.m2.1.1.1.cmml" xref="alg1.l1.m2.1.1.1">¯</ci><ci id="alg1.l1.m2.1.1.2.cmml" xref="alg1.l1.m2.1.1.2">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m2.1c">\bar{\theta}</annotation></semantics></math>


</div>
<div id="alg1.l2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l2.1.1.1" class="ltx_text" style="font-size:80%;">2:</span></span>InitBERT(<math id="alg1.l2.m1.1" class="ltx_Math" alttext="switchable\_transformer\_block" display="inline"><semantics id="alg1.l2.m1.1a"><mrow id="alg1.l2.m1.1.1" xref="alg1.l2.m1.1.1.cmml"><mi id="alg1.l2.m1.1.1.2" xref="alg1.l2.m1.1.1.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.3" xref="alg1.l2.m1.1.1.3.cmml">w</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1a" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.4" xref="alg1.l2.m1.1.1.4.cmml">i</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1b" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.5" xref="alg1.l2.m1.1.1.5.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1c" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.6" xref="alg1.l2.m1.1.1.6.cmml">c</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1d" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.7" xref="alg1.l2.m1.1.1.7.cmml">h</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1e" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.8" xref="alg1.l2.m1.1.1.8.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1f" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.9" xref="alg1.l2.m1.1.1.9.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1g" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.10" xref="alg1.l2.m1.1.1.10.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1h" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.11" xref="alg1.l2.m1.1.1.11.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1i" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l2.m1.1.1.12" xref="alg1.l2.m1.1.1.12.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1j" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.13" xref="alg1.l2.m1.1.1.13.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1k" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.14" xref="alg1.l2.m1.1.1.14.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1l" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.15" xref="alg1.l2.m1.1.1.15.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1m" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.16" xref="alg1.l2.m1.1.1.16.cmml">n</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1n" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.17" xref="alg1.l2.m1.1.1.17.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1o" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.18" xref="alg1.l2.m1.1.1.18.cmml">f</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1p" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.19" xref="alg1.l2.m1.1.1.19.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1q" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.20" xref="alg1.l2.m1.1.1.20.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1r" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.21" xref="alg1.l2.m1.1.1.21.cmml">m</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1s" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.22" xref="alg1.l2.m1.1.1.22.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1t" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.23" xref="alg1.l2.m1.1.1.23.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1u" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l2.m1.1.1.24" xref="alg1.l2.m1.1.1.24.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1v" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.25" xref="alg1.l2.m1.1.1.25.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1w" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.26" xref="alg1.l2.m1.1.1.26.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1x" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.27" xref="alg1.l2.m1.1.1.27.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1y" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.28" xref="alg1.l2.m1.1.1.28.cmml">c</mi><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.1.1z" xref="alg1.l2.m1.1.1.1.cmml">​</mo><mi id="alg1.l2.m1.1.1.29" xref="alg1.l2.m1.1.1.29.cmml">k</mi></mrow><annotation-xml encoding="MathML-Content" id="alg1.l2.m1.1b"><apply id="alg1.l2.m1.1.1.cmml" xref="alg1.l2.m1.1.1"><times id="alg1.l2.m1.1.1.1.cmml" xref="alg1.l2.m1.1.1.1"></times><ci id="alg1.l2.m1.1.1.2.cmml" xref="alg1.l2.m1.1.1.2">𝑠</ci><ci id="alg1.l2.m1.1.1.3.cmml" xref="alg1.l2.m1.1.1.3">𝑤</ci><ci id="alg1.l2.m1.1.1.4.cmml" xref="alg1.l2.m1.1.1.4">𝑖</ci><ci id="alg1.l2.m1.1.1.5.cmml" xref="alg1.l2.m1.1.1.5">𝑡</ci><ci id="alg1.l2.m1.1.1.6.cmml" xref="alg1.l2.m1.1.1.6">𝑐</ci><ci id="alg1.l2.m1.1.1.7.cmml" xref="alg1.l2.m1.1.1.7">ℎ</ci><ci id="alg1.l2.m1.1.1.8.cmml" xref="alg1.l2.m1.1.1.8">𝑎</ci><ci id="alg1.l2.m1.1.1.9.cmml" xref="alg1.l2.m1.1.1.9">𝑏</ci><ci id="alg1.l2.m1.1.1.10.cmml" xref="alg1.l2.m1.1.1.10">𝑙</ci><ci id="alg1.l2.m1.1.1.11.cmml" xref="alg1.l2.m1.1.1.11">𝑒</ci><ci id="alg1.l2.m1.1.1.12.cmml" xref="alg1.l2.m1.1.1.12">_</ci><ci id="alg1.l2.m1.1.1.13.cmml" xref="alg1.l2.m1.1.1.13">𝑡</ci><ci id="alg1.l2.m1.1.1.14.cmml" xref="alg1.l2.m1.1.1.14">𝑟</ci><ci id="alg1.l2.m1.1.1.15.cmml" xref="alg1.l2.m1.1.1.15">𝑎</ci><ci id="alg1.l2.m1.1.1.16.cmml" xref="alg1.l2.m1.1.1.16">𝑛</ci><ci id="alg1.l2.m1.1.1.17.cmml" xref="alg1.l2.m1.1.1.17">𝑠</ci><ci id="alg1.l2.m1.1.1.18.cmml" xref="alg1.l2.m1.1.1.18">𝑓</ci><ci id="alg1.l2.m1.1.1.19.cmml" xref="alg1.l2.m1.1.1.19">𝑜</ci><ci id="alg1.l2.m1.1.1.20.cmml" xref="alg1.l2.m1.1.1.20">𝑟</ci><ci id="alg1.l2.m1.1.1.21.cmml" xref="alg1.l2.m1.1.1.21">𝑚</ci><ci id="alg1.l2.m1.1.1.22.cmml" xref="alg1.l2.m1.1.1.22">𝑒</ci><ci id="alg1.l2.m1.1.1.23.cmml" xref="alg1.l2.m1.1.1.23">𝑟</ci><ci id="alg1.l2.m1.1.1.24.cmml" xref="alg1.l2.m1.1.1.24">_</ci><ci id="alg1.l2.m1.1.1.25.cmml" xref="alg1.l2.m1.1.1.25">𝑏</ci><ci id="alg1.l2.m1.1.1.26.cmml" xref="alg1.l2.m1.1.1.26">𝑙</ci><ci id="alg1.l2.m1.1.1.27.cmml" xref="alg1.l2.m1.1.1.27">𝑜</ci><ci id="alg1.l2.m1.1.1.28.cmml" xref="alg1.l2.m1.1.1.28">𝑐</ci><ci id="alg1.l2.m1.1.1.29.cmml" xref="alg1.l2.m1.1.1.29">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l2.m1.1c">switchable\_transformer\_block</annotation></semantics></math>)

</div>
<div id="alg1.l3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l3.1.1.1" class="ltx_text" style="font-size:80%;">3:</span></span><math id="alg1.l3.m1.1" class="ltx_Math" alttext="\gamma\ \leftarrow\frac{100}{T}" display="inline"><semantics id="alg1.l3.m1.1a"><mrow id="alg1.l3.m1.1.1" xref="alg1.l3.m1.1.1.cmml"><mi id="alg1.l3.m1.1.1.2" xref="alg1.l3.m1.1.1.2.cmml">γ</mi><mo lspace="0.778em" stretchy="false" id="alg1.l3.m1.1.1.1" xref="alg1.l3.m1.1.1.1.cmml">←</mo><mfrac id="alg1.l3.m1.1.1.3" xref="alg1.l3.m1.1.1.3.cmml"><mn id="alg1.l3.m1.1.1.3.2" xref="alg1.l3.m1.1.1.3.2.cmml">100</mn><mi id="alg1.l3.m1.1.1.3.3" xref="alg1.l3.m1.1.1.3.3.cmml">T</mi></mfrac></mrow><annotation-xml encoding="MathML-Content" id="alg1.l3.m1.1b"><apply id="alg1.l3.m1.1.1.cmml" xref="alg1.l3.m1.1.1"><ci id="alg1.l3.m1.1.1.1.cmml" xref="alg1.l3.m1.1.1.1">←</ci><ci id="alg1.l3.m1.1.1.2.cmml" xref="alg1.l3.m1.1.1.2">𝛾</ci><apply id="alg1.l3.m1.1.1.3.cmml" xref="alg1.l3.m1.1.1.3"><divide id="alg1.l3.m1.1.1.3.1.cmml" xref="alg1.l3.m1.1.1.3"></divide><cn type="integer" id="alg1.l3.m1.1.1.3.2.cmml" xref="alg1.l3.m1.1.1.3.2">100</cn><ci id="alg1.l3.m1.1.1.3.3.cmml" xref="alg1.l3.m1.1.1.3.3">𝑇</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l3.m1.1c">\gamma\ \leftarrow\frac{100}{T}</annotation></semantics></math>

</div>
<div id="alg1.l4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l4.1.1.1" class="ltx_text" style="font-size:80%;">4:</span></span><span id="alg1.l4.2" class="ltx_text ltx_font_bold">for</span> t <math id="alg1.l4.m1.1" class="ltx_Math" alttext="\leftarrow" display="inline"><semantics id="alg1.l4.m1.1a"><mo stretchy="false" id="alg1.l4.m1.1.1" xref="alg1.l4.m1.1.1.cmml">←</mo><annotation-xml encoding="MathML-Content" id="alg1.l4.m1.1b"><ci id="alg1.l4.m1.1.1.cmml" xref="alg1.l4.m1.1.1">←</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l4.m1.1c">\leftarrow</annotation></semantics></math> 1 to T <span id="alg1.l4.3" class="ltx_text ltx_font_bold">do</span>

</div>
<div id="alg1.l5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l5.1.1.1" class="ltx_text" style="font-size:80%;">5:</span></span>     <math id="alg1.l5.m1.2" class="ltx_Math" alttext="\theta_{t}\leftarrow(1-\bar{\theta})exp(-\gamma\cdot t)+\bar{\theta}" display="inline"><semantics id="alg1.l5.m1.2a"><mrow id="alg1.l5.m1.2.2" xref="alg1.l5.m1.2.2.cmml"><msub id="alg1.l5.m1.2.2.4" xref="alg1.l5.m1.2.2.4.cmml"><mi id="alg1.l5.m1.2.2.4.2" xref="alg1.l5.m1.2.2.4.2.cmml">θ</mi><mi id="alg1.l5.m1.2.2.4.3" xref="alg1.l5.m1.2.2.4.3.cmml">t</mi></msub><mo stretchy="false" id="alg1.l5.m1.2.2.3" xref="alg1.l5.m1.2.2.3.cmml">←</mo><mrow id="alg1.l5.m1.2.2.2" xref="alg1.l5.m1.2.2.2.cmml"><mrow id="alg1.l5.m1.2.2.2.2" xref="alg1.l5.m1.2.2.2.2.cmml"><mrow id="alg1.l5.m1.1.1.1.1.1.1" xref="alg1.l5.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l5.m1.1.1.1.1.1.1.2" xref="alg1.l5.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="alg1.l5.m1.1.1.1.1.1.1.1" xref="alg1.l5.m1.1.1.1.1.1.1.1.cmml"><mn id="alg1.l5.m1.1.1.1.1.1.1.1.2" xref="alg1.l5.m1.1.1.1.1.1.1.1.2.cmml">1</mn><mo id="alg1.l5.m1.1.1.1.1.1.1.1.1" xref="alg1.l5.m1.1.1.1.1.1.1.1.1.cmml">−</mo><mover accent="true" id="alg1.l5.m1.1.1.1.1.1.1.1.3" xref="alg1.l5.m1.1.1.1.1.1.1.1.3.cmml"><mi id="alg1.l5.m1.1.1.1.1.1.1.1.3.2" xref="alg1.l5.m1.1.1.1.1.1.1.1.3.2.cmml">θ</mi><mo id="alg1.l5.m1.1.1.1.1.1.1.1.3.1" xref="alg1.l5.m1.1.1.1.1.1.1.1.3.1.cmml">¯</mo></mover></mrow><mo stretchy="false" id="alg1.l5.m1.1.1.1.1.1.1.3" xref="alg1.l5.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="alg1.l5.m1.2.2.2.2.3" xref="alg1.l5.m1.2.2.2.2.3.cmml">​</mo><mi id="alg1.l5.m1.2.2.2.2.4" xref="alg1.l5.m1.2.2.2.2.4.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l5.m1.2.2.2.2.3a" xref="alg1.l5.m1.2.2.2.2.3.cmml">​</mo><mi id="alg1.l5.m1.2.2.2.2.5" xref="alg1.l5.m1.2.2.2.2.5.cmml">x</mi><mo lspace="0em" rspace="0em" id="alg1.l5.m1.2.2.2.2.3b" xref="alg1.l5.m1.2.2.2.2.3.cmml">​</mo><mi id="alg1.l5.m1.2.2.2.2.6" xref="alg1.l5.m1.2.2.2.2.6.cmml">p</mi><mo lspace="0em" rspace="0em" id="alg1.l5.m1.2.2.2.2.3c" xref="alg1.l5.m1.2.2.2.2.3.cmml">​</mo><mrow id="alg1.l5.m1.2.2.2.2.2.1" xref="alg1.l5.m1.2.2.2.2.2.1.1.cmml"><mo stretchy="false" id="alg1.l5.m1.2.2.2.2.2.1.2" xref="alg1.l5.m1.2.2.2.2.2.1.1.cmml">(</mo><mrow id="alg1.l5.m1.2.2.2.2.2.1.1" xref="alg1.l5.m1.2.2.2.2.2.1.1.cmml"><mo id="alg1.l5.m1.2.2.2.2.2.1.1a" xref="alg1.l5.m1.2.2.2.2.2.1.1.cmml">−</mo><mrow id="alg1.l5.m1.2.2.2.2.2.1.1.2" xref="alg1.l5.m1.2.2.2.2.2.1.1.2.cmml"><mi id="alg1.l5.m1.2.2.2.2.2.1.1.2.2" xref="alg1.l5.m1.2.2.2.2.2.1.1.2.2.cmml">γ</mi><mo lspace="0.222em" rspace="0.222em" id="alg1.l5.m1.2.2.2.2.2.1.1.2.1" xref="alg1.l5.m1.2.2.2.2.2.1.1.2.1.cmml">⋅</mo><mi id="alg1.l5.m1.2.2.2.2.2.1.1.2.3" xref="alg1.l5.m1.2.2.2.2.2.1.1.2.3.cmml">t</mi></mrow></mrow><mo stretchy="false" id="alg1.l5.m1.2.2.2.2.2.1.3" xref="alg1.l5.m1.2.2.2.2.2.1.1.cmml">)</mo></mrow></mrow><mo id="alg1.l5.m1.2.2.2.3" xref="alg1.l5.m1.2.2.2.3.cmml">+</mo><mover accent="true" id="alg1.l5.m1.2.2.2.4" xref="alg1.l5.m1.2.2.2.4.cmml"><mi id="alg1.l5.m1.2.2.2.4.2" xref="alg1.l5.m1.2.2.2.4.2.cmml">θ</mi><mo id="alg1.l5.m1.2.2.2.4.1" xref="alg1.l5.m1.2.2.2.4.1.cmml">¯</mo></mover></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l5.m1.2b"><apply id="alg1.l5.m1.2.2.cmml" xref="alg1.l5.m1.2.2"><ci id="alg1.l5.m1.2.2.3.cmml" xref="alg1.l5.m1.2.2.3">←</ci><apply id="alg1.l5.m1.2.2.4.cmml" xref="alg1.l5.m1.2.2.4"><csymbol cd="ambiguous" id="alg1.l5.m1.2.2.4.1.cmml" xref="alg1.l5.m1.2.2.4">subscript</csymbol><ci id="alg1.l5.m1.2.2.4.2.cmml" xref="alg1.l5.m1.2.2.4.2">𝜃</ci><ci id="alg1.l5.m1.2.2.4.3.cmml" xref="alg1.l5.m1.2.2.4.3">𝑡</ci></apply><apply id="alg1.l5.m1.2.2.2.cmml" xref="alg1.l5.m1.2.2.2"><plus id="alg1.l5.m1.2.2.2.3.cmml" xref="alg1.l5.m1.2.2.2.3"></plus><apply id="alg1.l5.m1.2.2.2.2.cmml" xref="alg1.l5.m1.2.2.2.2"><times id="alg1.l5.m1.2.2.2.2.3.cmml" xref="alg1.l5.m1.2.2.2.2.3"></times><apply id="alg1.l5.m1.1.1.1.1.1.1.1.cmml" xref="alg1.l5.m1.1.1.1.1.1.1"><minus id="alg1.l5.m1.1.1.1.1.1.1.1.1.cmml" xref="alg1.l5.m1.1.1.1.1.1.1.1.1"></minus><cn type="integer" id="alg1.l5.m1.1.1.1.1.1.1.1.2.cmml" xref="alg1.l5.m1.1.1.1.1.1.1.1.2">1</cn><apply id="alg1.l5.m1.1.1.1.1.1.1.1.3.cmml" xref="alg1.l5.m1.1.1.1.1.1.1.1.3"><ci id="alg1.l5.m1.1.1.1.1.1.1.1.3.1.cmml" xref="alg1.l5.m1.1.1.1.1.1.1.1.3.1">¯</ci><ci id="alg1.l5.m1.1.1.1.1.1.1.1.3.2.cmml" xref="alg1.l5.m1.1.1.1.1.1.1.1.3.2">𝜃</ci></apply></apply><ci id="alg1.l5.m1.2.2.2.2.4.cmml" xref="alg1.l5.m1.2.2.2.2.4">𝑒</ci><ci id="alg1.l5.m1.2.2.2.2.5.cmml" xref="alg1.l5.m1.2.2.2.2.5">𝑥</ci><ci id="alg1.l5.m1.2.2.2.2.6.cmml" xref="alg1.l5.m1.2.2.2.2.6">𝑝</ci><apply id="alg1.l5.m1.2.2.2.2.2.1.1.cmml" xref="alg1.l5.m1.2.2.2.2.2.1"><minus id="alg1.l5.m1.2.2.2.2.2.1.1.1.cmml" xref="alg1.l5.m1.2.2.2.2.2.1"></minus><apply id="alg1.l5.m1.2.2.2.2.2.1.1.2.cmml" xref="alg1.l5.m1.2.2.2.2.2.1.1.2"><ci id="alg1.l5.m1.2.2.2.2.2.1.1.2.1.cmml" xref="alg1.l5.m1.2.2.2.2.2.1.1.2.1">⋅</ci><ci id="alg1.l5.m1.2.2.2.2.2.1.1.2.2.cmml" xref="alg1.l5.m1.2.2.2.2.2.1.1.2.2">𝛾</ci><ci id="alg1.l5.m1.2.2.2.2.2.1.1.2.3.cmml" xref="alg1.l5.m1.2.2.2.2.2.1.1.2.3">𝑡</ci></apply></apply></apply><apply id="alg1.l5.m1.2.2.2.4.cmml" xref="alg1.l5.m1.2.2.2.4"><ci id="alg1.l5.m1.2.2.2.4.1.cmml" xref="alg1.l5.m1.2.2.2.4.1">¯</ci><ci id="alg1.l5.m1.2.2.2.4.2.cmml" xref="alg1.l5.m1.2.2.2.4.2">𝜃</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m1.2c">\theta_{t}\leftarrow(1-\bar{\theta})exp(-\gamma\cdot t)+\bar{\theta}</annotation></semantics></math>

</div>
<div id="alg1.l6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l6.1.1.1" class="ltx_text" style="font-size:80%;">6:</span></span>     step <math id="alg1.l6.m1.1" class="ltx_Math" alttext="\leftarrow\frac{1-\theta_{t}}{L}" display="inline"><semantics id="alg1.l6.m1.1a"><mrow id="alg1.l6.m1.1.1" xref="alg1.l6.m1.1.1.cmml"><mi id="alg1.l6.m1.1.1.2" xref="alg1.l6.m1.1.1.2.cmml"></mi><mo stretchy="false" id="alg1.l6.m1.1.1.1" xref="alg1.l6.m1.1.1.1.cmml">←</mo><mfrac id="alg1.l6.m1.1.1.3" xref="alg1.l6.m1.1.1.3.cmml"><mrow id="alg1.l6.m1.1.1.3.2" xref="alg1.l6.m1.1.1.3.2.cmml"><mn id="alg1.l6.m1.1.1.3.2.2" xref="alg1.l6.m1.1.1.3.2.2.cmml">1</mn><mo id="alg1.l6.m1.1.1.3.2.1" xref="alg1.l6.m1.1.1.3.2.1.cmml">−</mo><msub id="alg1.l6.m1.1.1.3.2.3" xref="alg1.l6.m1.1.1.3.2.3.cmml"><mi id="alg1.l6.m1.1.1.3.2.3.2" xref="alg1.l6.m1.1.1.3.2.3.2.cmml">θ</mi><mi id="alg1.l6.m1.1.1.3.2.3.3" xref="alg1.l6.m1.1.1.3.2.3.3.cmml">t</mi></msub></mrow><mi id="alg1.l6.m1.1.1.3.3" xref="alg1.l6.m1.1.1.3.3.cmml">L</mi></mfrac></mrow><annotation-xml encoding="MathML-Content" id="alg1.l6.m1.1b"><apply id="alg1.l6.m1.1.1.cmml" xref="alg1.l6.m1.1.1"><ci id="alg1.l6.m1.1.1.1.cmml" xref="alg1.l6.m1.1.1.1">←</ci><csymbol cd="latexml" id="alg1.l6.m1.1.1.2.cmml" xref="alg1.l6.m1.1.1.2">absent</csymbol><apply id="alg1.l6.m1.1.1.3.cmml" xref="alg1.l6.m1.1.1.3"><divide id="alg1.l6.m1.1.1.3.1.cmml" xref="alg1.l6.m1.1.1.3"></divide><apply id="alg1.l6.m1.1.1.3.2.cmml" xref="alg1.l6.m1.1.1.3.2"><minus id="alg1.l6.m1.1.1.3.2.1.cmml" xref="alg1.l6.m1.1.1.3.2.1"></minus><cn type="integer" id="alg1.l6.m1.1.1.3.2.2.cmml" xref="alg1.l6.m1.1.1.3.2.2">1</cn><apply id="alg1.l6.m1.1.1.3.2.3.cmml" xref="alg1.l6.m1.1.1.3.2.3"><csymbol cd="ambiguous" id="alg1.l6.m1.1.1.3.2.3.1.cmml" xref="alg1.l6.m1.1.1.3.2.3">subscript</csymbol><ci id="alg1.l6.m1.1.1.3.2.3.2.cmml" xref="alg1.l6.m1.1.1.3.2.3.2">𝜃</ci><ci id="alg1.l6.m1.1.1.3.2.3.3.cmml" xref="alg1.l6.m1.1.1.3.2.3.3">𝑡</ci></apply></apply><ci id="alg1.l6.m1.1.1.3.3.cmml" xref="alg1.l6.m1.1.1.3.3">𝐿</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l6.m1.1c">\leftarrow\frac{1-\theta_{t}}{L}</annotation></semantics></math>

</div>
<div id="alg1.l7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l7.1.1.1" class="ltx_text" style="font-size:80%;">7:</span></span>     <math id="alg1.l7.m1.1" class="ltx_Math" alttext="p\leftarrow 1" display="inline"><semantics id="alg1.l7.m1.1a"><mrow id="alg1.l7.m1.1.1" xref="alg1.l7.m1.1.1.cmml"><mi id="alg1.l7.m1.1.1.2" xref="alg1.l7.m1.1.1.2.cmml">p</mi><mo stretchy="false" id="alg1.l7.m1.1.1.1" xref="alg1.l7.m1.1.1.1.cmml">←</mo><mn id="alg1.l7.m1.1.1.3" xref="alg1.l7.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg1.l7.m1.1b"><apply id="alg1.l7.m1.1.1.cmml" xref="alg1.l7.m1.1.1"><ci id="alg1.l7.m1.1.1.1.cmml" xref="alg1.l7.m1.1.1.1">←</ci><ci id="alg1.l7.m1.1.1.2.cmml" xref="alg1.l7.m1.1.1.2">𝑝</ci><cn type="integer" id="alg1.l7.m1.1.1.3.cmml" xref="alg1.l7.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l7.m1.1c">p\leftarrow 1</annotation></semantics></math>

</div>
<div id="alg1.l8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l8.1.1.1" class="ltx_text" style="font-size:80%;">8:</span></span>     <span id="alg1.l8.2" class="ltx_text ltx_font_bold">for</span> l <math id="alg1.l8.m1.1" class="ltx_Math" alttext="\leftarrow" display="inline"><semantics id="alg1.l8.m1.1a"><mo stretchy="false" id="alg1.l8.m1.1.1" xref="alg1.l8.m1.1.1.cmml">←</mo><annotation-xml encoding="MathML-Content" id="alg1.l8.m1.1b"><ci id="alg1.l8.m1.1.1.cmml" xref="alg1.l8.m1.1.1">←</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l8.m1.1c">\leftarrow</annotation></semantics></math> 1 to L <span id="alg1.l8.3" class="ltx_text ltx_font_bold">do</span>

</div>
<div id="alg1.l9" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l9.1.1.1" class="ltx_text" style="font-size:80%;">9:</span></span>         action <math id="alg1.l9.m1.1" class="ltx_Math" alttext="\sim" display="inline"><semantics id="alg1.l9.m1.1a"><mo id="alg1.l9.m1.1.1" xref="alg1.l9.m1.1.1.cmml">∼</mo><annotation-xml encoding="MathML-Content" id="alg1.l9.m1.1b"><csymbol cd="latexml" id="alg1.l9.m1.1.1.cmml" xref="alg1.l9.m1.1.1">similar-to</csymbol></annotation-xml><annotation encoding="application/x-tex" id="alg1.l9.m1.1c">\sim</annotation></semantics></math> Bernoulli(p)

</div>
<div id="alg1.l10" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l10.1.1.1" class="ltx_text" style="font-size:80%;">10:</span></span>         <span id="alg1.l10.2" class="ltx_text ltx_font_bold">if</span> action == 0 <span id="alg1.l10.3" class="ltx_text ltx_font_bold">then</span>

</div>
<div id="alg1.l11" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l11.1.1.1" class="ltx_text" style="font-size:80%;">11:</span></span>              <math id="alg1.l11.m1.1" class="ltx_Math" alttext="x_{i+1}\leftarrow x_{i}" display="inline"><semantics id="alg1.l11.m1.1a"><mrow id="alg1.l11.m1.1.1" xref="alg1.l11.m1.1.1.cmml"><msub id="alg1.l11.m1.1.1.2" xref="alg1.l11.m1.1.1.2.cmml"><mi id="alg1.l11.m1.1.1.2.2" xref="alg1.l11.m1.1.1.2.2.cmml">x</mi><mrow id="alg1.l11.m1.1.1.2.3" xref="alg1.l11.m1.1.1.2.3.cmml"><mi id="alg1.l11.m1.1.1.2.3.2" xref="alg1.l11.m1.1.1.2.3.2.cmml">i</mi><mo id="alg1.l11.m1.1.1.2.3.1" xref="alg1.l11.m1.1.1.2.3.1.cmml">+</mo><mn id="alg1.l11.m1.1.1.2.3.3" xref="alg1.l11.m1.1.1.2.3.3.cmml">1</mn></mrow></msub><mo stretchy="false" id="alg1.l11.m1.1.1.1" xref="alg1.l11.m1.1.1.1.cmml">←</mo><msub id="alg1.l11.m1.1.1.3" xref="alg1.l11.m1.1.1.3.cmml"><mi id="alg1.l11.m1.1.1.3.2" xref="alg1.l11.m1.1.1.3.2.cmml">x</mi><mi id="alg1.l11.m1.1.1.3.3" xref="alg1.l11.m1.1.1.3.3.cmml">i</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="alg1.l11.m1.1b"><apply id="alg1.l11.m1.1.1.cmml" xref="alg1.l11.m1.1.1"><ci id="alg1.l11.m1.1.1.1.cmml" xref="alg1.l11.m1.1.1.1">←</ci><apply id="alg1.l11.m1.1.1.2.cmml" xref="alg1.l11.m1.1.1.2"><csymbol cd="ambiguous" id="alg1.l11.m1.1.1.2.1.cmml" xref="alg1.l11.m1.1.1.2">subscript</csymbol><ci id="alg1.l11.m1.1.1.2.2.cmml" xref="alg1.l11.m1.1.1.2.2">𝑥</ci><apply id="alg1.l11.m1.1.1.2.3.cmml" xref="alg1.l11.m1.1.1.2.3"><plus id="alg1.l11.m1.1.1.2.3.1.cmml" xref="alg1.l11.m1.1.1.2.3.1"></plus><ci id="alg1.l11.m1.1.1.2.3.2.cmml" xref="alg1.l11.m1.1.1.2.3.2">𝑖</ci><cn type="integer" id="alg1.l11.m1.1.1.2.3.3.cmml" xref="alg1.l11.m1.1.1.2.3.3">1</cn></apply></apply><apply id="alg1.l11.m1.1.1.3.cmml" xref="alg1.l11.m1.1.1.3"><csymbol cd="ambiguous" id="alg1.l11.m1.1.1.3.1.cmml" xref="alg1.l11.m1.1.1.3">subscript</csymbol><ci id="alg1.l11.m1.1.1.3.2.cmml" xref="alg1.l11.m1.1.1.3.2">𝑥</ci><ci id="alg1.l11.m1.1.1.3.3.cmml" xref="alg1.l11.m1.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l11.m1.1c">x_{i+1}\leftarrow x_{i}</annotation></semantics></math>

</div>
<div id="alg1.l12" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l12.1.1.1" class="ltx_text" style="font-size:80%;">12:</span></span>         <span id="alg1.l12.2" class="ltx_text ltx_font_bold">else</span>
</div>
<div id="alg1.l13" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l13.1.1.1" class="ltx_text" style="font-size:80%;">13:</span></span>              <math id="alg1.l13.m1.1" class="ltx_Math" alttext="x_{i}^{{}^{\prime}}\leftarrow x_{i}+f_{ATTN}(f_{LN}(x_{i}))\times\frac{1}{p}" display="inline"><semantics id="alg1.l13.m1.1a"><mrow id="alg1.l13.m1.1.1" xref="alg1.l13.m1.1.1.cmml"><msubsup id="alg1.l13.m1.1.1.3" xref="alg1.l13.m1.1.1.3.cmml"><mi id="alg1.l13.m1.1.1.3.2.2" xref="alg1.l13.m1.1.1.3.2.2.cmml">x</mi><mi id="alg1.l13.m1.1.1.3.2.3" xref="alg1.l13.m1.1.1.3.2.3.cmml">i</mi><msup id="alg1.l13.m1.1.1.3.3" xref="alg1.l13.m1.1.1.3.3.cmml"><mi id="alg1.l13.m1.1.1.3.3a" xref="alg1.l13.m1.1.1.3.3.cmml"></mi><mo id="alg1.l13.m1.1.1.3.3.1" xref="alg1.l13.m1.1.1.3.3.1.cmml">′</mo></msup></msubsup><mo stretchy="false" id="alg1.l13.m1.1.1.2" xref="alg1.l13.m1.1.1.2.cmml">←</mo><mrow id="alg1.l13.m1.1.1.1" xref="alg1.l13.m1.1.1.1.cmml"><msub id="alg1.l13.m1.1.1.1.3" xref="alg1.l13.m1.1.1.1.3.cmml"><mi id="alg1.l13.m1.1.1.1.3.2" xref="alg1.l13.m1.1.1.1.3.2.cmml">x</mi><mi id="alg1.l13.m1.1.1.1.3.3" xref="alg1.l13.m1.1.1.1.3.3.cmml">i</mi></msub><mo id="alg1.l13.m1.1.1.1.2" xref="alg1.l13.m1.1.1.1.2.cmml">+</mo><mrow id="alg1.l13.m1.1.1.1.1" xref="alg1.l13.m1.1.1.1.1.cmml"><mrow id="alg1.l13.m1.1.1.1.1.1" xref="alg1.l13.m1.1.1.1.1.1.cmml"><msub id="alg1.l13.m1.1.1.1.1.1.3" xref="alg1.l13.m1.1.1.1.1.1.3.cmml"><mi id="alg1.l13.m1.1.1.1.1.1.3.2" xref="alg1.l13.m1.1.1.1.1.1.3.2.cmml">f</mi><mrow id="alg1.l13.m1.1.1.1.1.1.3.3" xref="alg1.l13.m1.1.1.1.1.1.3.3.cmml"><mi id="alg1.l13.m1.1.1.1.1.1.3.3.2" xref="alg1.l13.m1.1.1.1.1.1.3.3.2.cmml">A</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.1.1.1.1.1.3.3.1" xref="alg1.l13.m1.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="alg1.l13.m1.1.1.1.1.1.3.3.3" xref="alg1.l13.m1.1.1.1.1.1.3.3.3.cmml">T</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.1.1.1.1.1.3.3.1a" xref="alg1.l13.m1.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="alg1.l13.m1.1.1.1.1.1.3.3.4" xref="alg1.l13.m1.1.1.1.1.1.3.3.4.cmml">T</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.1.1.1.1.1.3.3.1b" xref="alg1.l13.m1.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="alg1.l13.m1.1.1.1.1.1.3.3.5" xref="alg1.l13.m1.1.1.1.1.1.3.3.5.cmml">N</mi></mrow></msub><mo lspace="0em" rspace="0em" id="alg1.l13.m1.1.1.1.1.1.2" xref="alg1.l13.m1.1.1.1.1.1.2.cmml">​</mo><mrow id="alg1.l13.m1.1.1.1.1.1.1.1" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l13.m1.1.1.1.1.1.1.1.2" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="alg1.l13.m1.1.1.1.1.1.1.1.1" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.cmml"><msub id="alg1.l13.m1.1.1.1.1.1.1.1.1.3" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.3.cmml"><mi id="alg1.l13.m1.1.1.1.1.1.1.1.1.3.2" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.3.2.cmml">f</mi><mrow id="alg1.l13.m1.1.1.1.1.1.1.1.1.3.3" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.3.3.cmml"><mi id="alg1.l13.m1.1.1.1.1.1.1.1.1.3.3.2" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.3.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.1.1.1.1.1.1.1.1.3.3.1" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="alg1.l13.m1.1.1.1.1.1.1.1.1.3.3.3" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.3.3.3.cmml">N</mi></mrow></msub><mo lspace="0em" rspace="0em" id="alg1.l13.m1.1.1.1.1.1.1.1.1.2" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="alg1.l13.m1.1.1.1.1.1.1.1.1.1.1" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l13.m1.1.1.1.1.1.1.1.1.1.1.2" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="alg1.l13.m1.1.1.1.1.1.1.1.1.1.1.1" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="alg1.l13.m1.1.1.1.1.1.1.1.1.1.1.1.2" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="alg1.l13.m1.1.1.1.1.1.1.1.1.1.1.1.3" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="alg1.l13.m1.1.1.1.1.1.1.1.1.1.1.3" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo rspace="0.055em" stretchy="false" id="alg1.l13.m1.1.1.1.1.1.1.1.3" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo rspace="0.222em" id="alg1.l13.m1.1.1.1.1.2" xref="alg1.l13.m1.1.1.1.1.2.cmml">×</mo><mfrac id="alg1.l13.m1.1.1.1.1.3" xref="alg1.l13.m1.1.1.1.1.3.cmml"><mn id="alg1.l13.m1.1.1.1.1.3.2" xref="alg1.l13.m1.1.1.1.1.3.2.cmml">1</mn><mi id="alg1.l13.m1.1.1.1.1.3.3" xref="alg1.l13.m1.1.1.1.1.3.3.cmml">p</mi></mfrac></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l13.m1.1b"><apply id="alg1.l13.m1.1.1.cmml" xref="alg1.l13.m1.1.1"><ci id="alg1.l13.m1.1.1.2.cmml" xref="alg1.l13.m1.1.1.2">←</ci><apply id="alg1.l13.m1.1.1.3.cmml" xref="alg1.l13.m1.1.1.3"><csymbol cd="ambiguous" id="alg1.l13.m1.1.1.3.1.cmml" xref="alg1.l13.m1.1.1.3">superscript</csymbol><apply id="alg1.l13.m1.1.1.3.2.cmml" xref="alg1.l13.m1.1.1.3"><csymbol cd="ambiguous" id="alg1.l13.m1.1.1.3.2.1.cmml" xref="alg1.l13.m1.1.1.3">subscript</csymbol><ci id="alg1.l13.m1.1.1.3.2.2.cmml" xref="alg1.l13.m1.1.1.3.2.2">𝑥</ci><ci id="alg1.l13.m1.1.1.3.2.3.cmml" xref="alg1.l13.m1.1.1.3.2.3">𝑖</ci></apply><apply id="alg1.l13.m1.1.1.3.3.cmml" xref="alg1.l13.m1.1.1.3.3"><ci id="alg1.l13.m1.1.1.3.3.1.cmml" xref="alg1.l13.m1.1.1.3.3.1">′</ci></apply></apply><apply id="alg1.l13.m1.1.1.1.cmml" xref="alg1.l13.m1.1.1.1"><plus id="alg1.l13.m1.1.1.1.2.cmml" xref="alg1.l13.m1.1.1.1.2"></plus><apply id="alg1.l13.m1.1.1.1.3.cmml" xref="alg1.l13.m1.1.1.1.3"><csymbol cd="ambiguous" id="alg1.l13.m1.1.1.1.3.1.cmml" xref="alg1.l13.m1.1.1.1.3">subscript</csymbol><ci id="alg1.l13.m1.1.1.1.3.2.cmml" xref="alg1.l13.m1.1.1.1.3.2">𝑥</ci><ci id="alg1.l13.m1.1.1.1.3.3.cmml" xref="alg1.l13.m1.1.1.1.3.3">𝑖</ci></apply><apply id="alg1.l13.m1.1.1.1.1.cmml" xref="alg1.l13.m1.1.1.1.1"><times id="alg1.l13.m1.1.1.1.1.2.cmml" xref="alg1.l13.m1.1.1.1.1.2"></times><apply id="alg1.l13.m1.1.1.1.1.1.cmml" xref="alg1.l13.m1.1.1.1.1.1"><times id="alg1.l13.m1.1.1.1.1.1.2.cmml" xref="alg1.l13.m1.1.1.1.1.1.2"></times><apply id="alg1.l13.m1.1.1.1.1.1.3.cmml" xref="alg1.l13.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="alg1.l13.m1.1.1.1.1.1.3.1.cmml" xref="alg1.l13.m1.1.1.1.1.1.3">subscript</csymbol><ci id="alg1.l13.m1.1.1.1.1.1.3.2.cmml" xref="alg1.l13.m1.1.1.1.1.1.3.2">𝑓</ci><apply id="alg1.l13.m1.1.1.1.1.1.3.3.cmml" xref="alg1.l13.m1.1.1.1.1.1.3.3"><times id="alg1.l13.m1.1.1.1.1.1.3.3.1.cmml" xref="alg1.l13.m1.1.1.1.1.1.3.3.1"></times><ci id="alg1.l13.m1.1.1.1.1.1.3.3.2.cmml" xref="alg1.l13.m1.1.1.1.1.1.3.3.2">𝐴</ci><ci id="alg1.l13.m1.1.1.1.1.1.3.3.3.cmml" xref="alg1.l13.m1.1.1.1.1.1.3.3.3">𝑇</ci><ci id="alg1.l13.m1.1.1.1.1.1.3.3.4.cmml" xref="alg1.l13.m1.1.1.1.1.1.3.3.4">𝑇</ci><ci id="alg1.l13.m1.1.1.1.1.1.3.3.5.cmml" xref="alg1.l13.m1.1.1.1.1.1.3.3.5">𝑁</ci></apply></apply><apply id="alg1.l13.m1.1.1.1.1.1.1.1.1.cmml" xref="alg1.l13.m1.1.1.1.1.1.1.1"><times id="alg1.l13.m1.1.1.1.1.1.1.1.1.2.cmml" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.2"></times><apply id="alg1.l13.m1.1.1.1.1.1.1.1.1.3.cmml" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="alg1.l13.m1.1.1.1.1.1.1.1.1.3.1.cmml" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="alg1.l13.m1.1.1.1.1.1.1.1.1.3.2.cmml" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.3.2">𝑓</ci><apply id="alg1.l13.m1.1.1.1.1.1.1.1.1.3.3.cmml" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.3.3"><times id="alg1.l13.m1.1.1.1.1.1.1.1.1.3.3.1.cmml" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.3.3.1"></times><ci id="alg1.l13.m1.1.1.1.1.1.1.1.1.3.3.2.cmml" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.3.3.2">𝐿</ci><ci id="alg1.l13.m1.1.1.1.1.1.1.1.1.3.3.3.cmml" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.3.3.3">𝑁</ci></apply></apply><apply id="alg1.l13.m1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l13.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="alg1.l13.m1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="alg1.l13.m1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="alg1.l13.m1.1.1.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply><apply id="alg1.l13.m1.1.1.1.1.3.cmml" xref="alg1.l13.m1.1.1.1.1.3"><divide id="alg1.l13.m1.1.1.1.1.3.1.cmml" xref="alg1.l13.m1.1.1.1.1.3"></divide><cn type="integer" id="alg1.l13.m1.1.1.1.1.3.2.cmml" xref="alg1.l13.m1.1.1.1.1.3.2">1</cn><ci id="alg1.l13.m1.1.1.1.1.3.3.cmml" xref="alg1.l13.m1.1.1.1.1.3.3">𝑝</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l13.m1.1c">x_{i}^{{}^{\prime}}\leftarrow x_{i}+f_{ATTN}(f_{LN}(x_{i}))\times\frac{1}{p}</annotation></semantics></math>

</div>
<div id="alg1.l14" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l14.1.1.1" class="ltx_text" style="font-size:80%;">14:</span></span>              <math id="alg1.l14.m1.1" class="ltx_Math" alttext="x_{i+1}\leftarrow x_{i}^{{}^{\prime}}+f_{FFN}(f_{LN}(x_{i}^{{}^{\prime}}))\times\frac{1}{p}" display="inline"><semantics id="alg1.l14.m1.1a"><mrow id="alg1.l14.m1.1.1" xref="alg1.l14.m1.1.1.cmml"><msub id="alg1.l14.m1.1.1.3" xref="alg1.l14.m1.1.1.3.cmml"><mi id="alg1.l14.m1.1.1.3.2" xref="alg1.l14.m1.1.1.3.2.cmml">x</mi><mrow id="alg1.l14.m1.1.1.3.3" xref="alg1.l14.m1.1.1.3.3.cmml"><mi id="alg1.l14.m1.1.1.3.3.2" xref="alg1.l14.m1.1.1.3.3.2.cmml">i</mi><mo id="alg1.l14.m1.1.1.3.3.1" xref="alg1.l14.m1.1.1.3.3.1.cmml">+</mo><mn id="alg1.l14.m1.1.1.3.3.3" xref="alg1.l14.m1.1.1.3.3.3.cmml">1</mn></mrow></msub><mo stretchy="false" id="alg1.l14.m1.1.1.2" xref="alg1.l14.m1.1.1.2.cmml">←</mo><mrow id="alg1.l14.m1.1.1.1" xref="alg1.l14.m1.1.1.1.cmml"><msubsup id="alg1.l14.m1.1.1.1.3" xref="alg1.l14.m1.1.1.1.3.cmml"><mi id="alg1.l14.m1.1.1.1.3.2.2" xref="alg1.l14.m1.1.1.1.3.2.2.cmml">x</mi><mi id="alg1.l14.m1.1.1.1.3.2.3" xref="alg1.l14.m1.1.1.1.3.2.3.cmml">i</mi><msup id="alg1.l14.m1.1.1.1.3.3" xref="alg1.l14.m1.1.1.1.3.3.cmml"><mi id="alg1.l14.m1.1.1.1.3.3a" xref="alg1.l14.m1.1.1.1.3.3.cmml"></mi><mo id="alg1.l14.m1.1.1.1.3.3.1" xref="alg1.l14.m1.1.1.1.3.3.1.cmml">′</mo></msup></msubsup><mo id="alg1.l14.m1.1.1.1.2" xref="alg1.l14.m1.1.1.1.2.cmml">+</mo><mrow id="alg1.l14.m1.1.1.1.1" xref="alg1.l14.m1.1.1.1.1.cmml"><mrow id="alg1.l14.m1.1.1.1.1.1" xref="alg1.l14.m1.1.1.1.1.1.cmml"><msub id="alg1.l14.m1.1.1.1.1.1.3" xref="alg1.l14.m1.1.1.1.1.1.3.cmml"><mi id="alg1.l14.m1.1.1.1.1.1.3.2" xref="alg1.l14.m1.1.1.1.1.1.3.2.cmml">f</mi><mrow id="alg1.l14.m1.1.1.1.1.1.3.3" xref="alg1.l14.m1.1.1.1.1.1.3.3.cmml"><mi id="alg1.l14.m1.1.1.1.1.1.3.3.2" xref="alg1.l14.m1.1.1.1.1.1.3.3.2.cmml">F</mi><mo lspace="0em" rspace="0em" id="alg1.l14.m1.1.1.1.1.1.3.3.1" xref="alg1.l14.m1.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="alg1.l14.m1.1.1.1.1.1.3.3.3" xref="alg1.l14.m1.1.1.1.1.1.3.3.3.cmml">F</mi><mo lspace="0em" rspace="0em" id="alg1.l14.m1.1.1.1.1.1.3.3.1a" xref="alg1.l14.m1.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="alg1.l14.m1.1.1.1.1.1.3.3.4" xref="alg1.l14.m1.1.1.1.1.1.3.3.4.cmml">N</mi></mrow></msub><mo lspace="0em" rspace="0em" id="alg1.l14.m1.1.1.1.1.1.2" xref="alg1.l14.m1.1.1.1.1.1.2.cmml">​</mo><mrow id="alg1.l14.m1.1.1.1.1.1.1.1" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l14.m1.1.1.1.1.1.1.1.2" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="alg1.l14.m1.1.1.1.1.1.1.1.1" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.cmml"><msub id="alg1.l14.m1.1.1.1.1.1.1.1.1.3" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.3.cmml"><mi id="alg1.l14.m1.1.1.1.1.1.1.1.1.3.2" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.3.2.cmml">f</mi><mrow id="alg1.l14.m1.1.1.1.1.1.1.1.1.3.3" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.3.3.cmml"><mi id="alg1.l14.m1.1.1.1.1.1.1.1.1.3.3.2" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.3.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="alg1.l14.m1.1.1.1.1.1.1.1.1.3.3.1" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="alg1.l14.m1.1.1.1.1.1.1.1.1.3.3.3" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.3.3.3.cmml">N</mi></mrow></msub><mo lspace="0em" rspace="0em" id="alg1.l14.m1.1.1.1.1.1.1.1.1.2" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.2" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.2.2" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.2.3" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml">i</mi><msup id="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.3" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.3a" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.3.cmml"></mi><mo id="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.3.1" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml">′</mo></msup></msubsup><mo stretchy="false" id="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.3" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo rspace="0.055em" stretchy="false" id="alg1.l14.m1.1.1.1.1.1.1.1.3" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo rspace="0.222em" id="alg1.l14.m1.1.1.1.1.2" xref="alg1.l14.m1.1.1.1.1.2.cmml">×</mo><mfrac id="alg1.l14.m1.1.1.1.1.3" xref="alg1.l14.m1.1.1.1.1.3.cmml"><mn id="alg1.l14.m1.1.1.1.1.3.2" xref="alg1.l14.m1.1.1.1.1.3.2.cmml">1</mn><mi id="alg1.l14.m1.1.1.1.1.3.3" xref="alg1.l14.m1.1.1.1.1.3.3.cmml">p</mi></mfrac></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l14.m1.1b"><apply id="alg1.l14.m1.1.1.cmml" xref="alg1.l14.m1.1.1"><ci id="alg1.l14.m1.1.1.2.cmml" xref="alg1.l14.m1.1.1.2">←</ci><apply id="alg1.l14.m1.1.1.3.cmml" xref="alg1.l14.m1.1.1.3"><csymbol cd="ambiguous" id="alg1.l14.m1.1.1.3.1.cmml" xref="alg1.l14.m1.1.1.3">subscript</csymbol><ci id="alg1.l14.m1.1.1.3.2.cmml" xref="alg1.l14.m1.1.1.3.2">𝑥</ci><apply id="alg1.l14.m1.1.1.3.3.cmml" xref="alg1.l14.m1.1.1.3.3"><plus id="alg1.l14.m1.1.1.3.3.1.cmml" xref="alg1.l14.m1.1.1.3.3.1"></plus><ci id="alg1.l14.m1.1.1.3.3.2.cmml" xref="alg1.l14.m1.1.1.3.3.2">𝑖</ci><cn type="integer" id="alg1.l14.m1.1.1.3.3.3.cmml" xref="alg1.l14.m1.1.1.3.3.3">1</cn></apply></apply><apply id="alg1.l14.m1.1.1.1.cmml" xref="alg1.l14.m1.1.1.1"><plus id="alg1.l14.m1.1.1.1.2.cmml" xref="alg1.l14.m1.1.1.1.2"></plus><apply id="alg1.l14.m1.1.1.1.3.cmml" xref="alg1.l14.m1.1.1.1.3"><csymbol cd="ambiguous" id="alg1.l14.m1.1.1.1.3.1.cmml" xref="alg1.l14.m1.1.1.1.3">superscript</csymbol><apply id="alg1.l14.m1.1.1.1.3.2.cmml" xref="alg1.l14.m1.1.1.1.3"><csymbol cd="ambiguous" id="alg1.l14.m1.1.1.1.3.2.1.cmml" xref="alg1.l14.m1.1.1.1.3">subscript</csymbol><ci id="alg1.l14.m1.1.1.1.3.2.2.cmml" xref="alg1.l14.m1.1.1.1.3.2.2">𝑥</ci><ci id="alg1.l14.m1.1.1.1.3.2.3.cmml" xref="alg1.l14.m1.1.1.1.3.2.3">𝑖</ci></apply><apply id="alg1.l14.m1.1.1.1.3.3.cmml" xref="alg1.l14.m1.1.1.1.3.3"><ci id="alg1.l14.m1.1.1.1.3.3.1.cmml" xref="alg1.l14.m1.1.1.1.3.3.1">′</ci></apply></apply><apply id="alg1.l14.m1.1.1.1.1.cmml" xref="alg1.l14.m1.1.1.1.1"><times id="alg1.l14.m1.1.1.1.1.2.cmml" xref="alg1.l14.m1.1.1.1.1.2"></times><apply id="alg1.l14.m1.1.1.1.1.1.cmml" xref="alg1.l14.m1.1.1.1.1.1"><times id="alg1.l14.m1.1.1.1.1.1.2.cmml" xref="alg1.l14.m1.1.1.1.1.1.2"></times><apply id="alg1.l14.m1.1.1.1.1.1.3.cmml" xref="alg1.l14.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="alg1.l14.m1.1.1.1.1.1.3.1.cmml" xref="alg1.l14.m1.1.1.1.1.1.3">subscript</csymbol><ci id="alg1.l14.m1.1.1.1.1.1.3.2.cmml" xref="alg1.l14.m1.1.1.1.1.1.3.2">𝑓</ci><apply id="alg1.l14.m1.1.1.1.1.1.3.3.cmml" xref="alg1.l14.m1.1.1.1.1.1.3.3"><times id="alg1.l14.m1.1.1.1.1.1.3.3.1.cmml" xref="alg1.l14.m1.1.1.1.1.1.3.3.1"></times><ci id="alg1.l14.m1.1.1.1.1.1.3.3.2.cmml" xref="alg1.l14.m1.1.1.1.1.1.3.3.2">𝐹</ci><ci id="alg1.l14.m1.1.1.1.1.1.3.3.3.cmml" xref="alg1.l14.m1.1.1.1.1.1.3.3.3">𝐹</ci><ci id="alg1.l14.m1.1.1.1.1.1.3.3.4.cmml" xref="alg1.l14.m1.1.1.1.1.1.3.3.4">𝑁</ci></apply></apply><apply id="alg1.l14.m1.1.1.1.1.1.1.1.1.cmml" xref="alg1.l14.m1.1.1.1.1.1.1.1"><times id="alg1.l14.m1.1.1.1.1.1.1.1.1.2.cmml" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.2"></times><apply id="alg1.l14.m1.1.1.1.1.1.1.1.1.3.cmml" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="alg1.l14.m1.1.1.1.1.1.1.1.1.3.1.cmml" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="alg1.l14.m1.1.1.1.1.1.1.1.1.3.2.cmml" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.3.2">𝑓</ci><apply id="alg1.l14.m1.1.1.1.1.1.1.1.1.3.3.cmml" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.3.3"><times id="alg1.l14.m1.1.1.1.1.1.1.1.1.3.3.1.cmml" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.3.3.1"></times><ci id="alg1.l14.m1.1.1.1.1.1.1.1.1.3.3.2.cmml" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.3.3.2">𝐿</ci><ci id="alg1.l14.m1.1.1.1.1.1.1.1.1.3.3.3.cmml" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.3.3.3">𝑁</ci></apply></apply><apply id="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1">superscript</csymbol><apply id="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.3"><ci id="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="alg1.l14.m1.1.1.1.1.1.1.1.1.1.1.1.3.1">′</ci></apply></apply></apply></apply><apply id="alg1.l14.m1.1.1.1.1.3.cmml" xref="alg1.l14.m1.1.1.1.1.3"><divide id="alg1.l14.m1.1.1.1.1.3.1.cmml" xref="alg1.l14.m1.1.1.1.1.3"></divide><cn type="integer" id="alg1.l14.m1.1.1.1.1.3.2.cmml" xref="alg1.l14.m1.1.1.1.1.3.2">1</cn><ci id="alg1.l14.m1.1.1.1.1.3.3.cmml" xref="alg1.l14.m1.1.1.1.1.3.3">𝑝</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l14.m1.1c">x_{i+1}\leftarrow x_{i}^{{}^{\prime}}+f_{FFN}(f_{LN}(x_{i}^{{}^{\prime}}))\times\frac{1}{p}</annotation></semantics></math>
         
</div>
<div id="alg1.l15" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l15.1.1.1" class="ltx_text" style="font-size:80%;">15:</span></span>         <math id="alg1.l15.m1.1" class="ltx_Math" alttext="x_{i}\leftarrow x_{i+1}" display="inline"><semantics id="alg1.l15.m1.1a"><mrow id="alg1.l15.m1.1.1" xref="alg1.l15.m1.1.1.cmml"><msub id="alg1.l15.m1.1.1.2" xref="alg1.l15.m1.1.1.2.cmml"><mi id="alg1.l15.m1.1.1.2.2" xref="alg1.l15.m1.1.1.2.2.cmml">x</mi><mi id="alg1.l15.m1.1.1.2.3" xref="alg1.l15.m1.1.1.2.3.cmml">i</mi></msub><mo stretchy="false" id="alg1.l15.m1.1.1.1" xref="alg1.l15.m1.1.1.1.cmml">←</mo><msub id="alg1.l15.m1.1.1.3" xref="alg1.l15.m1.1.1.3.cmml"><mi id="alg1.l15.m1.1.1.3.2" xref="alg1.l15.m1.1.1.3.2.cmml">x</mi><mrow id="alg1.l15.m1.1.1.3.3" xref="alg1.l15.m1.1.1.3.3.cmml"><mi id="alg1.l15.m1.1.1.3.3.2" xref="alg1.l15.m1.1.1.3.3.2.cmml">i</mi><mo id="alg1.l15.m1.1.1.3.3.1" xref="alg1.l15.m1.1.1.3.3.1.cmml">+</mo><mn id="alg1.l15.m1.1.1.3.3.3" xref="alg1.l15.m1.1.1.3.3.3.cmml">1</mn></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="alg1.l15.m1.1b"><apply id="alg1.l15.m1.1.1.cmml" xref="alg1.l15.m1.1.1"><ci id="alg1.l15.m1.1.1.1.cmml" xref="alg1.l15.m1.1.1.1">←</ci><apply id="alg1.l15.m1.1.1.2.cmml" xref="alg1.l15.m1.1.1.2"><csymbol cd="ambiguous" id="alg1.l15.m1.1.1.2.1.cmml" xref="alg1.l15.m1.1.1.2">subscript</csymbol><ci id="alg1.l15.m1.1.1.2.2.cmml" xref="alg1.l15.m1.1.1.2.2">𝑥</ci><ci id="alg1.l15.m1.1.1.2.3.cmml" xref="alg1.l15.m1.1.1.2.3">𝑖</ci></apply><apply id="alg1.l15.m1.1.1.3.cmml" xref="alg1.l15.m1.1.1.3"><csymbol cd="ambiguous" id="alg1.l15.m1.1.1.3.1.cmml" xref="alg1.l15.m1.1.1.3">subscript</csymbol><ci id="alg1.l15.m1.1.1.3.2.cmml" xref="alg1.l15.m1.1.1.3.2">𝑥</ci><apply id="alg1.l15.m1.1.1.3.3.cmml" xref="alg1.l15.m1.1.1.3.3"><plus id="alg1.l15.m1.1.1.3.3.1.cmml" xref="alg1.l15.m1.1.1.3.3.1"></plus><ci id="alg1.l15.m1.1.1.3.3.2.cmml" xref="alg1.l15.m1.1.1.3.3.2">𝑖</ci><cn type="integer" id="alg1.l15.m1.1.1.3.3.3.cmml" xref="alg1.l15.m1.1.1.3.3.3">1</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l15.m1.1c">x_{i}\leftarrow x_{i+1}</annotation></semantics></math>

</div>
<div id="alg1.l16" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l16.1.1.1" class="ltx_text" style="font-size:80%;">16:</span></span>         p <math id="alg1.l16.m1.1" class="ltx_Math" alttext="\leftarrow" display="inline"><semantics id="alg1.l16.m1.1a"><mo stretchy="false" id="alg1.l16.m1.1.1" xref="alg1.l16.m1.1.1.cmml">←</mo><annotation-xml encoding="MathML-Content" id="alg1.l16.m1.1b"><ci id="alg1.l16.m1.1.1.cmml" xref="alg1.l16.m1.1.1">←</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l16.m1.1c">\leftarrow</annotation></semantics></math> p - step
     
</div>
<div id="alg1.l17" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l17.1.1.1" class="ltx_text" style="font-size:80%;">17:</span></span>     Y <math id="alg1.l17.m1.1" class="ltx_Math" alttext="\leftarrow" display="inline"><semantics id="alg1.l17.m1.1a"><mo stretchy="false" id="alg1.l17.m1.1.1" xref="alg1.l17.m1.1.1.cmml">←</mo><annotation-xml encoding="MathML-Content" id="alg1.l17.m1.1b"><ci id="alg1.l17.m1.1.1.cmml" xref="alg1.l17.m1.1.1">←</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l17.m1.1c">\leftarrow</annotation></semantics></math> <math id="alg1.l17.m2.1" class="ltx_Math" alttext="output\_layer(x_{L})" display="inline"><semantics id="alg1.l17.m2.1a"><mrow id="alg1.l17.m2.1.1" xref="alg1.l17.m2.1.1.cmml"><mi id="alg1.l17.m2.1.1.3" xref="alg1.l17.m2.1.1.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m2.1.1.2" xref="alg1.l17.m2.1.1.2.cmml">​</mo><mi id="alg1.l17.m2.1.1.4" xref="alg1.l17.m2.1.1.4.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m2.1.1.2a" xref="alg1.l17.m2.1.1.2.cmml">​</mo><mi id="alg1.l17.m2.1.1.5" xref="alg1.l17.m2.1.1.5.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m2.1.1.2b" xref="alg1.l17.m2.1.1.2.cmml">​</mo><mi id="alg1.l17.m2.1.1.6" xref="alg1.l17.m2.1.1.6.cmml">p</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m2.1.1.2c" xref="alg1.l17.m2.1.1.2.cmml">​</mo><mi id="alg1.l17.m2.1.1.7" xref="alg1.l17.m2.1.1.7.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m2.1.1.2d" xref="alg1.l17.m2.1.1.2.cmml">​</mo><mi id="alg1.l17.m2.1.1.8" xref="alg1.l17.m2.1.1.8.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m2.1.1.2e" xref="alg1.l17.m2.1.1.2.cmml">​</mo><mi mathvariant="normal" id="alg1.l17.m2.1.1.9" xref="alg1.l17.m2.1.1.9.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m2.1.1.2f" xref="alg1.l17.m2.1.1.2.cmml">​</mo><mi id="alg1.l17.m2.1.1.10" xref="alg1.l17.m2.1.1.10.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m2.1.1.2g" xref="alg1.l17.m2.1.1.2.cmml">​</mo><mi id="alg1.l17.m2.1.1.11" xref="alg1.l17.m2.1.1.11.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m2.1.1.2h" xref="alg1.l17.m2.1.1.2.cmml">​</mo><mi id="alg1.l17.m2.1.1.12" xref="alg1.l17.m2.1.1.12.cmml">y</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m2.1.1.2i" xref="alg1.l17.m2.1.1.2.cmml">​</mo><mi id="alg1.l17.m2.1.1.13" xref="alg1.l17.m2.1.1.13.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m2.1.1.2j" xref="alg1.l17.m2.1.1.2.cmml">​</mo><mi id="alg1.l17.m2.1.1.14" xref="alg1.l17.m2.1.1.14.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg1.l17.m2.1.1.2k" xref="alg1.l17.m2.1.1.2.cmml">​</mo><mrow id="alg1.l17.m2.1.1.1.1" xref="alg1.l17.m2.1.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l17.m2.1.1.1.1.2" xref="alg1.l17.m2.1.1.1.1.1.cmml">(</mo><msub id="alg1.l17.m2.1.1.1.1.1" xref="alg1.l17.m2.1.1.1.1.1.cmml"><mi id="alg1.l17.m2.1.1.1.1.1.2" xref="alg1.l17.m2.1.1.1.1.1.2.cmml">x</mi><mi id="alg1.l17.m2.1.1.1.1.1.3" xref="alg1.l17.m2.1.1.1.1.1.3.cmml">L</mi></msub><mo stretchy="false" id="alg1.l17.m2.1.1.1.1.3" xref="alg1.l17.m2.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l17.m2.1b"><apply id="alg1.l17.m2.1.1.cmml" xref="alg1.l17.m2.1.1"><times id="alg1.l17.m2.1.1.2.cmml" xref="alg1.l17.m2.1.1.2"></times><ci id="alg1.l17.m2.1.1.3.cmml" xref="alg1.l17.m2.1.1.3">𝑜</ci><ci id="alg1.l17.m2.1.1.4.cmml" xref="alg1.l17.m2.1.1.4">𝑢</ci><ci id="alg1.l17.m2.1.1.5.cmml" xref="alg1.l17.m2.1.1.5">𝑡</ci><ci id="alg1.l17.m2.1.1.6.cmml" xref="alg1.l17.m2.1.1.6">𝑝</ci><ci id="alg1.l17.m2.1.1.7.cmml" xref="alg1.l17.m2.1.1.7">𝑢</ci><ci id="alg1.l17.m2.1.1.8.cmml" xref="alg1.l17.m2.1.1.8">𝑡</ci><ci id="alg1.l17.m2.1.1.9.cmml" xref="alg1.l17.m2.1.1.9">_</ci><ci id="alg1.l17.m2.1.1.10.cmml" xref="alg1.l17.m2.1.1.10">𝑙</ci><ci id="alg1.l17.m2.1.1.11.cmml" xref="alg1.l17.m2.1.1.11">𝑎</ci><ci id="alg1.l17.m2.1.1.12.cmml" xref="alg1.l17.m2.1.1.12">𝑦</ci><ci id="alg1.l17.m2.1.1.13.cmml" xref="alg1.l17.m2.1.1.13">𝑒</ci><ci id="alg1.l17.m2.1.1.14.cmml" xref="alg1.l17.m2.1.1.14">𝑟</ci><apply id="alg1.l17.m2.1.1.1.1.1.cmml" xref="alg1.l17.m2.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l17.m2.1.1.1.1.1.1.cmml" xref="alg1.l17.m2.1.1.1.1">subscript</csymbol><ci id="alg1.l17.m2.1.1.1.1.1.2.cmml" xref="alg1.l17.m2.1.1.1.1.1.2">𝑥</ci><ci id="alg1.l17.m2.1.1.1.1.1.3.cmml" xref="alg1.l17.m2.1.1.1.1.1.3">𝐿</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l17.m2.1c">output\_layer(x_{L})</annotation></semantics></math>

</div>
<div id="alg1.l18" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l18.1.1.1" class="ltx_text" style="font-size:80%;">18:</span></span>     loss <math id="alg1.l18.m1.2" class="ltx_Math" alttext="\leftarrow loss\_fn(\bar{Y},Y)" display="inline"><semantics id="alg1.l18.m1.2a"><mrow id="alg1.l18.m1.2.3" xref="alg1.l18.m1.2.3.cmml"><mi id="alg1.l18.m1.2.3.2" xref="alg1.l18.m1.2.3.2.cmml"></mi><mo stretchy="false" id="alg1.l18.m1.2.3.1" xref="alg1.l18.m1.2.3.1.cmml">←</mo><mrow id="alg1.l18.m1.2.3.3" xref="alg1.l18.m1.2.3.3.cmml"><mi id="alg1.l18.m1.2.3.3.2" xref="alg1.l18.m1.2.3.3.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l18.m1.2.3.3.1" xref="alg1.l18.m1.2.3.3.1.cmml">​</mo><mi id="alg1.l18.m1.2.3.3.3" xref="alg1.l18.m1.2.3.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg1.l18.m1.2.3.3.1a" xref="alg1.l18.m1.2.3.3.1.cmml">​</mo><mi id="alg1.l18.m1.2.3.3.4" xref="alg1.l18.m1.2.3.3.4.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l18.m1.2.3.3.1b" xref="alg1.l18.m1.2.3.3.1.cmml">​</mo><mi id="alg1.l18.m1.2.3.3.5" xref="alg1.l18.m1.2.3.3.5.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l18.m1.2.3.3.1c" xref="alg1.l18.m1.2.3.3.1.cmml">​</mo><mi mathvariant="normal" id="alg1.l18.m1.2.3.3.6" xref="alg1.l18.m1.2.3.3.6.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg1.l18.m1.2.3.3.1d" xref="alg1.l18.m1.2.3.3.1.cmml">​</mo><mi id="alg1.l18.m1.2.3.3.7" xref="alg1.l18.m1.2.3.3.7.cmml">f</mi><mo lspace="0em" rspace="0em" id="alg1.l18.m1.2.3.3.1e" xref="alg1.l18.m1.2.3.3.1.cmml">​</mo><mi id="alg1.l18.m1.2.3.3.8" xref="alg1.l18.m1.2.3.3.8.cmml">n</mi><mo lspace="0em" rspace="0em" id="alg1.l18.m1.2.3.3.1f" xref="alg1.l18.m1.2.3.3.1.cmml">​</mo><mrow id="alg1.l18.m1.2.3.3.9.2" xref="alg1.l18.m1.2.3.3.9.1.cmml"><mo stretchy="false" id="alg1.l18.m1.2.3.3.9.2.1" xref="alg1.l18.m1.2.3.3.9.1.cmml">(</mo><mover accent="true" id="alg1.l18.m1.1.1" xref="alg1.l18.m1.1.1.cmml"><mi id="alg1.l18.m1.1.1.2" xref="alg1.l18.m1.1.1.2.cmml">Y</mi><mo id="alg1.l18.m1.1.1.1" xref="alg1.l18.m1.1.1.1.cmml">¯</mo></mover><mo id="alg1.l18.m1.2.3.3.9.2.2" xref="alg1.l18.m1.2.3.3.9.1.cmml">,</mo><mi id="alg1.l18.m1.2.2" xref="alg1.l18.m1.2.2.cmml">Y</mi><mo stretchy="false" id="alg1.l18.m1.2.3.3.9.2.3" xref="alg1.l18.m1.2.3.3.9.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l18.m1.2b"><apply id="alg1.l18.m1.2.3.cmml" xref="alg1.l18.m1.2.3"><ci id="alg1.l18.m1.2.3.1.cmml" xref="alg1.l18.m1.2.3.1">←</ci><csymbol cd="latexml" id="alg1.l18.m1.2.3.2.cmml" xref="alg1.l18.m1.2.3.2">absent</csymbol><apply id="alg1.l18.m1.2.3.3.cmml" xref="alg1.l18.m1.2.3.3"><times id="alg1.l18.m1.2.3.3.1.cmml" xref="alg1.l18.m1.2.3.3.1"></times><ci id="alg1.l18.m1.2.3.3.2.cmml" xref="alg1.l18.m1.2.3.3.2">𝑙</ci><ci id="alg1.l18.m1.2.3.3.3.cmml" xref="alg1.l18.m1.2.3.3.3">𝑜</ci><ci id="alg1.l18.m1.2.3.3.4.cmml" xref="alg1.l18.m1.2.3.3.4">𝑠</ci><ci id="alg1.l18.m1.2.3.3.5.cmml" xref="alg1.l18.m1.2.3.3.5">𝑠</ci><ci id="alg1.l18.m1.2.3.3.6.cmml" xref="alg1.l18.m1.2.3.3.6">_</ci><ci id="alg1.l18.m1.2.3.3.7.cmml" xref="alg1.l18.m1.2.3.3.7">𝑓</ci><ci id="alg1.l18.m1.2.3.3.8.cmml" xref="alg1.l18.m1.2.3.3.8">𝑛</ci><interval closure="open" id="alg1.l18.m1.2.3.3.9.1.cmml" xref="alg1.l18.m1.2.3.3.9.2"><apply id="alg1.l18.m1.1.1.cmml" xref="alg1.l18.m1.1.1"><ci id="alg1.l18.m1.1.1.1.cmml" xref="alg1.l18.m1.1.1.1">¯</ci><ci id="alg1.l18.m1.1.1.2.cmml" xref="alg1.l18.m1.1.1.2">𝑌</ci></apply><ci id="alg1.l18.m1.2.2.cmml" xref="alg1.l18.m1.2.2">𝑌</ci></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l18.m1.2c">\leftarrow loss\_fn(\bar{Y},Y)</annotation></semantics></math>

</div>
<div id="alg1.l19" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l19.1.1.1" class="ltx_text" style="font-size:80%;">19:</span></span>     backward(loss)

</div>
</div>
</figure>
</figure>
</section>
<section id="S4.SS2.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Putting it together.</h4>

<div id="S4.SS2.SSS0.Px3.p1" class="ltx_para">
<p id="S4.SS2.SSS0.Px3.p1.7" class="ltx_p">Note that because of the exist of the identity mapping, when an ST block is bypassed for a specific iteration,
there is no need to perform forward-backward computation or gradient updates, and there will be updates with significantly shorter networks and more direct paths to individual layers.
Based on this, we design a stochastic training algorithm based on ST blocks and the progressive layer-drop schedule to train models like BERT faster, which we call <em id="S4.SS2.SSS0.Px3.p1.7.1" class="ltx_emph ltx_font_italic">progressive layer dropping</em> (Alg. <a href="#alg1" title="Algorithm 1 ‣ Distribution along the depth dimension. ‣ 4.2 A Progressive Layer Dropping Schedule ‣ 4 Our Approach: Progressive Layer Dropping ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>).

The expected network depth, denoted as <math id="S4.SS2.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="\bar{L}" display="inline"><semantics id="S4.SS2.SSS0.Px3.p1.1.m1.1a"><mover accent="true" id="S4.SS2.SSS0.Px3.p1.1.m1.1.1" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.cmml"><mi id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2.cmml">L</mi><mo id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.1" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.1.cmml">¯</mo></mover><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px3.p1.1.m1.1b"><apply id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1"><ci id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.1">¯</ci><ci id="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2.cmml" xref="S4.SS2.SSS0.Px3.p1.1.m1.1.1.2">𝐿</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px3.p1.1.m1.1c">\bar{L}</annotation></semantics></math>, becomes a random variable. Its expectation is given by: <math id="S4.SS2.SSS0.Px3.p1.2.m2.3" class="ltx_Math" alttext="E(\bar{L})=\sum_{t=0}^{T}\sum_{i=1}^{L}\theta(i,t)" display="inline"><semantics id="S4.SS2.SSS0.Px3.p1.2.m2.3a"><mrow id="S4.SS2.SSS0.Px3.p1.2.m2.3.4" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.cmml"><mrow id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.2" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.2.cmml"><mi id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.2.2" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.2.2.cmml">E</mi><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.2.1" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.2.1.cmml">​</mo><mrow id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.2.3.2" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.cmml"><mo stretchy="false" id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.2.3.2.1" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.cmml">(</mo><mover accent="true" id="S4.SS2.SSS0.Px3.p1.2.m2.1.1" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.cmml"><mi id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2.cmml">L</mi><mo id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.1" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.1.cmml">¯</mo></mover><mo stretchy="false" id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.2.3.2.2" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.cmml">)</mo></mrow></mrow><mo rspace="0.111em" id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.1" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.1.cmml">=</mo><mrow id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.cmml"><msubsup id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.cmml"><mo rspace="0em" id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.2" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.2.cmml">∑</mo><mrow id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.3" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.3.cmml"><mi id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.3.2" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.3.2.cmml">t</mi><mo id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.3.1" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.3.1.cmml">=</mo><mn id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.3.3" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.3.3.cmml">0</mn></mrow><mi id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.3" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.3.cmml">T</mi></msubsup><mrow id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.cmml"><msubsup id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.cmml"><mo id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.2" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.2.cmml">∑</mo><mrow id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.3" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.3.cmml"><mi id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.3.2" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.3.2.cmml">i</mi><mo id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.3.1" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.3.1.cmml">=</mo><mn id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.3.3" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.3.3.cmml">1</mn></mrow><mi id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.3" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.3.cmml">L</mi></msubsup><mrow id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2.cmml"><mi id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2.2" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2.2.cmml">θ</mi><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2.1" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2.1.cmml">​</mo><mrow id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2.3.2" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2.3.1.cmml"><mo stretchy="false" id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2.3.2.1" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2.3.1.cmml">(</mo><mi id="S4.SS2.SSS0.Px3.p1.2.m2.2.2" xref="S4.SS2.SSS0.Px3.p1.2.m2.2.2.cmml">i</mi><mo id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2.3.2.2" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2.3.1.cmml">,</mo><mi id="S4.SS2.SSS0.Px3.p1.2.m2.3.3" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.3.cmml">t</mi><mo stretchy="false" id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2.3.2.3" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2.3.1.cmml">)</mo></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px3.p1.2.m2.3b"><apply id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4"><eq id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.1.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.1"></eq><apply id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.2.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.2"><times id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.2.1.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.2.1"></times><ci id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.2.2.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.2.2">𝐸</ci><apply id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.2.3.2"><ci id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.1">¯</ci><ci id="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.1.1.2">𝐿</ci></apply></apply><apply id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3"><apply id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1">superscript</csymbol><apply id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.1.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1">subscript</csymbol><sum id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.2.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.2"></sum><apply id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.3.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.3"><eq id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.3.1.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.3.1"></eq><ci id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.3.2.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.3.2">𝑡</ci><cn type="integer" id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.3.3.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.2.3.3">0</cn></apply></apply><ci id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.3.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.1.3">𝑇</ci></apply><apply id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2"><apply id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1">superscript</csymbol><apply id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1"><csymbol cd="ambiguous" id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.1.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1">subscript</csymbol><sum id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.2.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.2"></sum><apply id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.3.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.3"><eq id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.3.1.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.3.1"></eq><ci id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.3.2.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.3.2">𝑖</ci><cn type="integer" id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.3.3.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.2.3.3">1</cn></apply></apply><ci id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.3.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.1.3">𝐿</ci></apply><apply id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2"><times id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2.1.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2.1"></times><ci id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2.2.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2.2">𝜃</ci><interval closure="open" id="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2.3.1.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.4.3.2.2.3.2"><ci id="S4.SS2.SSS0.Px3.p1.2.m2.2.2.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.2.2">𝑖</ci><ci id="S4.SS2.SSS0.Px3.p1.2.m2.3.3.cmml" xref="S4.SS2.SSS0.Px3.p1.2.m2.3.3">𝑡</ci></interval></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px3.p1.2.m2.3c">E(\bar{L})=\sum_{t=0}^{T}\sum_{i=1}^{L}\theta(i,t)</annotation></semantics></math>.
With <math id="S4.SS2.SSS0.Px3.p1.3.m3.1" class="ltx_Math" alttext="\bar{\theta}=0.5" display="inline"><semantics id="S4.SS2.SSS0.Px3.p1.3.m3.1a"><mrow id="S4.SS2.SSS0.Px3.p1.3.m3.1.1" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1.cmml"><mover accent="true" id="S4.SS2.SSS0.Px3.p1.3.m3.1.1.2" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1.2.cmml"><mi id="S4.SS2.SSS0.Px3.p1.3.m3.1.1.2.2" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1.2.2.cmml">θ</mi><mo id="S4.SS2.SSS0.Px3.p1.3.m3.1.1.2.1" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1.2.1.cmml">¯</mo></mover><mo id="S4.SS2.SSS0.Px3.p1.3.m3.1.1.1" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1.1.cmml">=</mo><mn id="S4.SS2.SSS0.Px3.p1.3.m3.1.1.3" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1.3.cmml">0.5</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px3.p1.3.m3.1b"><apply id="S4.SS2.SSS0.Px3.p1.3.m3.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1"><eq id="S4.SS2.SSS0.Px3.p1.3.m3.1.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1.1"></eq><apply id="S4.SS2.SSS0.Px3.p1.3.m3.1.1.2.cmml" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1.2"><ci id="S4.SS2.SSS0.Px3.p1.3.m3.1.1.2.1.cmml" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1.2.1">¯</ci><ci id="S4.SS2.SSS0.Px3.p1.3.m3.1.1.2.2.cmml" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1.2.2">𝜃</ci></apply><cn type="float" id="S4.SS2.SSS0.Px3.p1.3.m3.1.1.3.cmml" xref="S4.SS2.SSS0.Px3.p1.3.m3.1.1.3">0.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px3.p1.3.m3.1c">\bar{\theta}=0.5</annotation></semantics></math>, the expected number of ST blocks during training reduces to <math id="S4.SS2.SSS0.Px3.p1.4.m4.2" class="ltx_Math" alttext="E(\bar{L})=(3L-1)/4" display="inline"><semantics id="S4.SS2.SSS0.Px3.p1.4.m4.2a"><mrow id="S4.SS2.SSS0.Px3.p1.4.m4.2.2" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.cmml"><mrow id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.3" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.3.cmml"><mi id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.3.2" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.3.2.cmml">E</mi><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.3.1" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.3.1.cmml">​</mo><mrow id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.3.3.2" xref="S4.SS2.SSS0.Px3.p1.4.m4.1.1.cmml"><mo stretchy="false" id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.3.3.2.1" xref="S4.SS2.SSS0.Px3.p1.4.m4.1.1.cmml">(</mo><mover accent="true" id="S4.SS2.SSS0.Px3.p1.4.m4.1.1" xref="S4.SS2.SSS0.Px3.p1.4.m4.1.1.cmml"><mi id="S4.SS2.SSS0.Px3.p1.4.m4.1.1.2" xref="S4.SS2.SSS0.Px3.p1.4.m4.1.1.2.cmml">L</mi><mo id="S4.SS2.SSS0.Px3.p1.4.m4.1.1.1" xref="S4.SS2.SSS0.Px3.p1.4.m4.1.1.1.cmml">¯</mo></mover><mo stretchy="false" id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.3.3.2.2" xref="S4.SS2.SSS0.Px3.p1.4.m4.1.1.cmml">)</mo></mrow></mrow><mo id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.2" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.2.cmml">=</mo><mrow id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.cmml"><mrow id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.cmml"><mo stretchy="false" id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.2" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.cmml">(</mo><mrow id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.cmml"><mrow id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.2" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.2.cmml"><mn id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.2.2" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.2.2.cmml">3</mn><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.2.1" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.2.1.cmml">​</mo><mi id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.2.3" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.2.3.cmml">L</mi></mrow><mo id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.1" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.1.cmml">−</mo><mn id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.3" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.3" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.cmml">)</mo></mrow><mo id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.2" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.2.cmml">/</mo><mn id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.3" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.3.cmml">4</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px3.p1.4.m4.2b"><apply id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.cmml" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2"><eq id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.2.cmml" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.2"></eq><apply id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.3.cmml" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.3"><times id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.3.1.cmml" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.3.1"></times><ci id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.3.2.cmml" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.3.2">𝐸</ci><apply id="S4.SS2.SSS0.Px3.p1.4.m4.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.3.3.2"><ci id="S4.SS2.SSS0.Px3.p1.4.m4.1.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.4.m4.1.1.1">¯</ci><ci id="S4.SS2.SSS0.Px3.p1.4.m4.1.1.2.cmml" xref="S4.SS2.SSS0.Px3.p1.4.m4.1.1.2">𝐿</ci></apply></apply><apply id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.cmml" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1"><divide id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.2.cmml" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.2"></divide><apply id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1"><minus id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.1"></minus><apply id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.2.cmml" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.2"><times id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.2.1.cmml" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.2.1"></times><cn type="integer" id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.2.2.cmml" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.2.2">3</cn><ci id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.2.3.cmml" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.2.3">𝐿</ci></apply><cn type="integer" id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.3.cmml" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.1.1.1.3">1</cn></apply><cn type="integer" id="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.3.cmml" xref="S4.SS2.SSS0.Px3.p1.4.m4.2.2.1.3">4</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px3.p1.4.m4.2c">E(\bar{L})=(3L-1)/4</annotation></semantics></math> or <math id="S4.SS2.SSS0.Px3.p1.5.m5.1" class="ltx_Math" alttext="E(\bar{L})\approx 3L/4" display="inline"><semantics id="S4.SS2.SSS0.Px3.p1.5.m5.1a"><mrow id="S4.SS2.SSS0.Px3.p1.5.m5.1.2" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.cmml"><mrow id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.2" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.2.cmml"><mi id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.2.2" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.2.2.cmml">E</mi><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.2.1" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.2.1.cmml">​</mo><mrow id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.2.3.2" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.1.cmml"><mo stretchy="false" id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.2.3.2.1" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.1.cmml">(</mo><mover accent="true" id="S4.SS2.SSS0.Px3.p1.5.m5.1.1" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.1.cmml"><mi id="S4.SS2.SSS0.Px3.p1.5.m5.1.1.2" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.1.2.cmml">L</mi><mo id="S4.SS2.SSS0.Px3.p1.5.m5.1.1.1" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.1.1.cmml">¯</mo></mover><mo stretchy="false" id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.2.3.2.2" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.1.cmml">)</mo></mrow></mrow><mo id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.1" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.1.cmml">≈</mo><mrow id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.cmml"><mrow id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.2" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.2.cmml"><mn id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.2.2" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.2.2.cmml">3</mn><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.2.1" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.2.1.cmml">​</mo><mi id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.2.3" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.2.3.cmml">L</mi></mrow><mo id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.1" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.1.cmml">/</mo><mn id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.3" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.3.cmml">4</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px3.p1.5.m5.1b"><apply id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.cmml" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2"><approx id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.1.cmml" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.1"></approx><apply id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.2.cmml" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.2"><times id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.2.1.cmml" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.2.1"></times><ci id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.2.2.cmml" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.2.2">𝐸</ci><apply id="S4.SS2.SSS0.Px3.p1.5.m5.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.2.3.2"><ci id="S4.SS2.SSS0.Px3.p1.5.m5.1.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.1.1">¯</ci><ci id="S4.SS2.SSS0.Px3.p1.5.m5.1.1.2.cmml" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.1.2">𝐿</ci></apply></apply><apply id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.cmml" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3"><divide id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.1.cmml" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.1"></divide><apply id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.2.cmml" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.2"><times id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.2.1.cmml" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.2.1"></times><cn type="integer" id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.2.2.cmml" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.2.2">3</cn><ci id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.2.3.cmml" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.2.3">𝐿</ci></apply><cn type="integer" id="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.3.cmml" xref="S4.SS2.SSS0.Px3.p1.5.m5.1.2.3.3">4</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px3.p1.5.m5.1c">E(\bar{L})\approx 3L/4</annotation></semantics></math> when T is large. For the 12-layer BERT model with L=12 used in our experiments, we have <math id="S4.SS2.SSS0.Px3.p1.6.m6.1" class="ltx_Math" alttext="E(\bar{L})\approx 9" display="inline"><semantics id="S4.SS2.SSS0.Px3.p1.6.m6.1a"><mrow id="S4.SS2.SSS0.Px3.p1.6.m6.1.2" xref="S4.SS2.SSS0.Px3.p1.6.m6.1.2.cmml"><mrow id="S4.SS2.SSS0.Px3.p1.6.m6.1.2.2" xref="S4.SS2.SSS0.Px3.p1.6.m6.1.2.2.cmml"><mi id="S4.SS2.SSS0.Px3.p1.6.m6.1.2.2.2" xref="S4.SS2.SSS0.Px3.p1.6.m6.1.2.2.2.cmml">E</mi><mo lspace="0em" rspace="0em" id="S4.SS2.SSS0.Px3.p1.6.m6.1.2.2.1" xref="S4.SS2.SSS0.Px3.p1.6.m6.1.2.2.1.cmml">​</mo><mrow id="S4.SS2.SSS0.Px3.p1.6.m6.1.2.2.3.2" xref="S4.SS2.SSS0.Px3.p1.6.m6.1.1.cmml"><mo stretchy="false" id="S4.SS2.SSS0.Px3.p1.6.m6.1.2.2.3.2.1" xref="S4.SS2.SSS0.Px3.p1.6.m6.1.1.cmml">(</mo><mover accent="true" id="S4.SS2.SSS0.Px3.p1.6.m6.1.1" xref="S4.SS2.SSS0.Px3.p1.6.m6.1.1.cmml"><mi id="S4.SS2.SSS0.Px3.p1.6.m6.1.1.2" xref="S4.SS2.SSS0.Px3.p1.6.m6.1.1.2.cmml">L</mi><mo id="S4.SS2.SSS0.Px3.p1.6.m6.1.1.1" xref="S4.SS2.SSS0.Px3.p1.6.m6.1.1.1.cmml">¯</mo></mover><mo stretchy="false" id="S4.SS2.SSS0.Px3.p1.6.m6.1.2.2.3.2.2" xref="S4.SS2.SSS0.Px3.p1.6.m6.1.1.cmml">)</mo></mrow></mrow><mo id="S4.SS2.SSS0.Px3.p1.6.m6.1.2.1" xref="S4.SS2.SSS0.Px3.p1.6.m6.1.2.1.cmml">≈</mo><mn id="S4.SS2.SSS0.Px3.p1.6.m6.1.2.3" xref="S4.SS2.SSS0.Px3.p1.6.m6.1.2.3.cmml">9</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px3.p1.6.m6.1b"><apply id="S4.SS2.SSS0.Px3.p1.6.m6.1.2.cmml" xref="S4.SS2.SSS0.Px3.p1.6.m6.1.2"><approx id="S4.SS2.SSS0.Px3.p1.6.m6.1.2.1.cmml" xref="S4.SS2.SSS0.Px3.p1.6.m6.1.2.1"></approx><apply id="S4.SS2.SSS0.Px3.p1.6.m6.1.2.2.cmml" xref="S4.SS2.SSS0.Px3.p1.6.m6.1.2.2"><times id="S4.SS2.SSS0.Px3.p1.6.m6.1.2.2.1.cmml" xref="S4.SS2.SSS0.Px3.p1.6.m6.1.2.2.1"></times><ci id="S4.SS2.SSS0.Px3.p1.6.m6.1.2.2.2.cmml" xref="S4.SS2.SSS0.Px3.p1.6.m6.1.2.2.2">𝐸</ci><apply id="S4.SS2.SSS0.Px3.p1.6.m6.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.6.m6.1.2.2.3.2"><ci id="S4.SS2.SSS0.Px3.p1.6.m6.1.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.6.m6.1.1.1">¯</ci><ci id="S4.SS2.SSS0.Px3.p1.6.m6.1.1.2.cmml" xref="S4.SS2.SSS0.Px3.p1.6.m6.1.1.2">𝐿</ci></apply></apply><cn type="integer" id="S4.SS2.SSS0.Px3.p1.6.m6.1.2.3.cmml" xref="S4.SS2.SSS0.Px3.p1.6.m6.1.2.3">9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px3.p1.6.m6.1c">E(\bar{L})\approx 9</annotation></semantics></math>. In other words, with progressive layer dropping, we train BERT with an average number of 9 layers. This significantly improves the pre-training speed of the BERT model. Following the calculations
above, approximately 25% of FLOPS could be saved under the drop schedule with <math id="S4.SS2.SSS0.Px3.p1.7.m7.1" class="ltx_Math" alttext="\bar{\theta}" display="inline"><semantics id="S4.SS2.SSS0.Px3.p1.7.m7.1a"><mover accent="true" id="S4.SS2.SSS0.Px3.p1.7.m7.1.1" xref="S4.SS2.SSS0.Px3.p1.7.m7.1.1.cmml"><mi id="S4.SS2.SSS0.Px3.p1.7.m7.1.1.2" xref="S4.SS2.SSS0.Px3.p1.7.m7.1.1.2.cmml">θ</mi><mo id="S4.SS2.SSS0.Px3.p1.7.m7.1.1.1" xref="S4.SS2.SSS0.Px3.p1.7.m7.1.1.1.cmml">¯</mo></mover><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS0.Px3.p1.7.m7.1b"><apply id="S4.SS2.SSS0.Px3.p1.7.m7.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.7.m7.1.1"><ci id="S4.SS2.SSS0.Px3.p1.7.m7.1.1.1.cmml" xref="S4.SS2.SSS0.Px3.p1.7.m7.1.1.1">¯</ci><ci id="S4.SS2.SSS0.Px3.p1.7.m7.1.1.2.cmml" xref="S4.SS2.SSS0.Px3.p1.7.m7.1.1.2">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS0.Px3.p1.7.m7.1c">\bar{\theta}</annotation></semantics></math> = 0.5. We recover the model with full-depth blocks at fine-tuning and testing time.</p>
</div>
</section>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Evaluation</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">We show that our method improves the pre-training efficiency of Transformer networks, and the trained models achieve competitive or even better performance compared to the baseline on transfer learning downstream tasks. We also show ablation studies to analyze the proposed training techniques.</p>
</div>
<section id="S5.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Datasets.</h4>

<div id="S5.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S5.SS0.SSS0.Px1.p1.1" class="ltx_p">We follow Devlin et al. <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">3</span></a>]</cite> to use English Wikipedia corpus and BookCorpus for pre-training. By concatenating the two datasets, we obtain our corpus with roughly 2.8B word tokens in total, which is comparable with the data corpus used in Devlin et al. <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">3</span></a>]</cite>.



We segment
documents into sentences with 128 tokens; We normalize,
lower-case, and tokenize texts using Wordpiece tokenizer <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">3</span></a>]</cite>. The
final vocabulary contains 30,528 word pieces.
We split documents into one training set and one validation set (300:1).
For fine-tuning, we use
GLUE (General Language Understanding Evaluation), a collection of 9 sentence or sentence-pair natural language understanding tasks including question answering, sentiment analysis, and textual entailment. It is designed to favor sample-efficient learning and knowledge-transfer across a range of different linguistic tasks in different domains.
</p>
</div>
</section>
<section id="S5.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Training details.</h4>

<div id="S5.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S5.SS0.SSS0.Px2.p1.4" class="ltx_p">We use our own implementation of the BERT model <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">3</span></a>]</cite> based on the Huggingface[1] PyTorch implementation. All experiments are performed on 4<math id="S5.SS0.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.1.m1.1a"><mo id="S5.SS0.SSS0.Px2.p1.1.m1.1.1" xref="S5.SS0.SSS0.Px2.p1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.1.m1.1b"><times id="S5.SS0.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.1.m1.1c">\times</annotation></semantics></math>DGX-2 boxes with 64<math id="S5.SS0.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S5.SS0.SSS0.Px2.p1.2.m2.1a"><mo id="S5.SS0.SSS0.Px2.p1.2.m2.1.1" xref="S5.SS0.SSS0.Px2.p1.2.m2.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px2.p1.2.m2.1b"><times id="S5.SS0.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S5.SS0.SSS0.Px2.p1.2.m2.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px2.p1.2.m2.1c">\times</annotation></semantics></math>V100 GPUs. Data parallelism is handled via PyTorch DDP (Distributed Data Parallel) library <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib43" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">43</span></a>]</cite>. We recognize and eliminate additional computation overhead: we overlap data loading with computation through the asynchronous prefetching queue; we optimize the BERT output processing through sparse computation on masked tokens. Using our pre-processed data, we train a 12-layer BERT-base model from scratch as the baseline. We use a warm-up ratio of 0.02 with lr<sub id="S5.SS0.SSS0.Px2.p1.4.1" class="ltx_sub"><span id="S5.SS0.SSS0.Px2.p1.4.1.1" class="ltx_text ltx_font_italic">max</span></sub>=1e<sup id="S5.SS0.SSS0.Px2.p1.4.2" class="ltx_sup"><span id="S5.SS0.SSS0.Px2.p1.4.2.1" class="ltx_text ltx_font_italic">-4</span></sup>. Following <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">3</span></a>]</cite>, we use Adam as the optimizer. We train with batch size 4K for 200K steps, which is approximately 186 epochs. The detailed parameter settings are listed in the Appendix <a href="#A1" title="Appendix A Pre-training Hyperparameters ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>. We fine-tune GLUE tasks for 5 epochs and report the median development
set results for each task over five random initializations.</p>
</div>
</section>
<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>Experimental Results</h3>

<section id="S5.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Pre-training convergence comparisons.</h4>

<div id="S5.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S5.SS1.SSS0.Px1.p1.4" class="ltx_p">Fig. <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:fig:different-lr</span> visualizes the convergence of validation loss regarding the computational time. We make the following observations. First, with lr<sub id="S5.SS1.SSS0.Px1.p1.4.1" class="ltx_sub"><span id="S5.SS1.SSS0.Px1.p1.4.1.1" class="ltx_text ltx_font_italic">max</span></sub>=1e<sup id="S5.SS1.SSS0.Px1.p1.4.2" class="ltx_sup"><span id="S5.SS1.SSS0.Px1.p1.4.2.1" class="ltx_text ltx_font_italic">-4</span></sup>, the convergence rate of our algorithm and the baseline is very close. This verifies empirically that our progressive layer dropping method does not hurt model convergence. Second, when using a larger learning rate lr<sub id="S5.SS1.SSS0.Px1.p1.4.3" class="ltx_sub"><span id="S5.SS1.SSS0.Px1.p1.4.3.1" class="ltx_text ltx_font_italic">max</span></sub>=1e<sup id="S5.SS1.SSS0.Px1.p1.4.4" class="ltx_sup"><span id="S5.SS1.SSS0.Px1.p1.4.4.1" class="ltx_text ltx_font_italic">-3</span></sup>, the baseline diverges. In contrast, our method shows a healthy convergence curve and is much faster. This confirms that our architectural changes stabilize training and allows BERT training with more aggressive learning rates.</p>
</div>
<figure id="S5.F7" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S5.F7.fig1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:286.2pt;">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S5.F7.sf1" class="ltx_figure ltx_figure_panel"><img src="/html/2010.13369/assets/figs/different-lr.png" id="S5.F7.sf1.g1" class="ltx_graphics ltx_img_landscape" width="180" height="142" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S5.F7.sf2" class="ltx_figure ltx_figure_panel"><img src="/html/2010.13369/assets/figs/baseline-vs-pst-main.png" id="S5.F7.sf2.g1" class="ltx_graphics ltx_img_landscape" width="180" height="142" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 7: </span>The convergence curve of the baseline and our proposed method regarding the wall-clock time. </figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S5.F7.3" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:143.1pt;">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S5.T1" class="ltx_table ltx_figure_panel">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_table">Table 1: </span>Training time comparison. Sample RD standards for sample reduction. SPD represents speedup.</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<table id="S5.F7.3.3" class="ltx_tabular ltx_centering ltx_figure_panel ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.F7.3.3.4.1" class="ltx_tr">
<th id="S5.F7.3.3.4.1.1" class="ltx_td ltx_nopad_r ltx_th ltx_th_column ltx_border_l ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"></th>
<th id="S5.F7.3.3.4.1.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.F7.3.3.4.1.2.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.F7.3.3.4.1.2.1.1" class="ltx_tr">
<td id="S5.F7.3.3.4.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.3.3.4.1.2.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Training</span></td>
</tr>
<tr id="S5.F7.3.3.4.1.2.1.2" class="ltx_tr">
<td id="S5.F7.3.3.4.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.3.3.4.1.2.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Time</span></td>
</tr>
</table>
</th>
<th id="S5.F7.3.3.4.1.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.F7.3.3.4.1.3.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.F7.3.3.4.1.3.1.1" class="ltx_tr">
<td id="S5.F7.3.3.4.1.3.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.3.3.4.1.3.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Sample</span></td>
</tr>
<tr id="S5.F7.3.3.4.1.3.1.2" class="ltx_tr">
<td id="S5.F7.3.3.4.1.3.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.3.3.4.1.3.1.2.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">RD</span></td>
</tr>
</table>
</th>
<th id="S5.F7.3.3.4.1.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.3.3.4.1.4.1" class="ltx_text ltx_font_bold" style="font-size:90%;">SPD</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.F7.3.3.5.1" class="ltx_tr">
<td id="S5.F7.3.3.5.1.1" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_l ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.F7.3.3.5.1.1.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.F7.3.3.5.1.1.1.1" class="ltx_tr">
<td id="S5.F7.3.3.5.1.1.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.3.3.5.1.1.1.1.1.1" class="ltx_text" style="font-size:90%;">Baseline</span></td>
</tr>
<tr id="S5.F7.3.3.5.1.1.1.2" class="ltx_tr">
<td id="S5.F7.3.3.5.1.1.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.3.3.5.1.1.1.2.1.1" class="ltx_text" style="font-size:90%;">ckp186</span></td>
</tr>
</table>
</td>
<td id="S5.F7.3.3.5.1.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.3.3.5.1.2.1" class="ltx_text" style="font-size:90%;">38.45h</span></td>
<td id="S5.F7.3.3.5.1.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.3.3.5.1.3.1" class="ltx_text" style="font-size:90%;">0</span></td>
<td id="S5.F7.3.3.5.1.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.3.3.5.1.4.1" class="ltx_text" style="font-size:90%;">1</span></td>
</tr>
<tr id="S5.F7.1.1.1" class="ltx_tr">
<td id="S5.F7.1.1.1.2" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_l ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.F7.1.1.1.2.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.F7.1.1.1.2.1.1" class="ltx_tr">
<td id="S5.F7.1.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.1.1.1.2.1.1.1.1" class="ltx_text" style="font-size:90%;">PLD</span></td>
</tr>
<tr id="S5.F7.1.1.1.2.1.2" class="ltx_tr">
<td id="S5.F7.1.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.1.1.1.2.1.2.1.1" class="ltx_text" style="font-size:90%;">ckp186</span></td>
</tr>
</table>
</td>
<td id="S5.F7.1.1.1.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.1.1.1.3.1" class="ltx_text" style="font-size:90%;">29.22h</span></td>
<td id="S5.F7.1.1.1.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.1.1.1.4.1" class="ltx_text" style="font-size:90%;">0</span></td>
<td id="S5.F7.1.1.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S5.F7.1.1.1.1.1" class="ltx_text" style="font-size:90%;">1.3</span><math id="S5.F7.1.1.1.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S5.F7.1.1.1.1.m1.1a"><mo mathsize="90%" id="S5.F7.1.1.1.1.m1.1.1" xref="S5.F7.1.1.1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S5.F7.1.1.1.1.m1.1b"><times id="S5.F7.1.1.1.1.m1.1.1.cmml" xref="S5.F7.1.1.1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S5.F7.1.1.1.1.m1.1c">\times</annotation></semantics></math>
</td>
</tr>
<tr id="S5.F7.2.2.2" class="ltx_tr">
<td id="S5.F7.2.2.2.2" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_l ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.F7.2.2.2.2.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.F7.2.2.2.2.1.1" class="ltx_tr">
<td id="S5.F7.2.2.2.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.2.2.2.2.1.1.1.1" class="ltx_text" style="font-size:90%;">PLD</span></td>
</tr>
<tr id="S5.F7.2.2.2.2.1.2" class="ltx_tr">
<td id="S5.F7.2.2.2.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.2.2.2.2.1.2.1.1" class="ltx_text" style="font-size:90%;">ckp100</span></td>
</tr>
</table>
</td>
<td id="S5.F7.2.2.2.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.2.2.2.3.1" class="ltx_text" style="font-size:90%;">15.56h</span></td>
<td id="S5.F7.2.2.2.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.2.2.2.4.1" class="ltx_text" style="font-size:90%;">46%</span></td>
<td id="S5.F7.2.2.2.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S5.F7.2.2.2.1.1" class="ltx_text" style="font-size:90%;">2.5</span><math id="S5.F7.2.2.2.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S5.F7.2.2.2.1.m1.1a"><mo mathsize="90%" id="S5.F7.2.2.2.1.m1.1.1" xref="S5.F7.2.2.2.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S5.F7.2.2.2.1.m1.1b"><times id="S5.F7.2.2.2.1.m1.1.1.cmml" xref="S5.F7.2.2.2.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S5.F7.2.2.2.1.m1.1c">\times</annotation></semantics></math>
</td>
</tr>
<tr id="S5.F7.3.3.3" class="ltx_tr">
<td id="S5.F7.3.3.3.2" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_b ltx_border_l ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.F7.3.3.3.2.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.F7.3.3.3.2.1.1" class="ltx_tr">
<td id="S5.F7.3.3.3.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.3.3.3.2.1.1.1.1" class="ltx_text" style="font-size:90%;">PLD</span></td>
</tr>
<tr id="S5.F7.3.3.3.2.1.2" class="ltx_tr">
<td id="S5.F7.3.3.3.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.3.3.3.2.1.2.1.1" class="ltx_text" style="font-size:90%;">ckp87</span></td>
</tr>
</table>
</td>
<td id="S5.F7.3.3.3.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_b ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.3.3.3.3.1" class="ltx_text" style="font-size:90%;">13.53h</span></td>
<td id="S5.F7.3.3.3.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_b ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.F7.3.3.3.4.1" class="ltx_text" style="font-size:90%;">53%</span></td>
<td id="S5.F7.3.3.3.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_b ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S5.F7.3.3.3.1.1" class="ltx_text" style="font-size:90%;">2.8</span><math id="S5.F7.3.3.3.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S5.F7.3.3.3.1.m1.1a"><mo mathsize="90%" id="S5.F7.3.3.3.1.m1.1.1" xref="S5.F7.3.3.3.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S5.F7.3.3.3.1.m1.1b"><times id="S5.F7.3.3.3.1.m1.1.1.cmml" xref="S5.F7.3.3.3.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S5.F7.3.3.3.1.m1.1c">\times</annotation></semantics></math>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</figure>
</div>
</div>
</figure>
</section>
<section id="S5.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Speedup.</h4>

<div id="S5.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="S5.SS1.SSS0.Px2.p1.1" class="ltx_p">Fig. <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:fig:baseline-vs-pst-main</span> shows both the training curve (dotted) and the validation curve (solid) of the baseline and PLD with a zoomed-in view. The baseline curve becomes almost flat at epoch 186, getting a validation loss of 1.75. In contrast, PLD reaches the same validation loss at epoch 87, with 53% fewer training samples. Furthermore, PLD achieves a 24% time reduction when training the same number of samples. This is because our approach trains the model with a smaller number of expected depth for the same number of steps. It is slightly lower than the 25% GFLOPS reduction in the analysis because the output layer still takes a small amount of computation even after optimizations. The combination of these two factors, yields 2.8<math id="S5.SS1.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S5.SS1.SSS0.Px2.p1.1.m1.1a"><mo id="S5.SS1.SSS0.Px2.p1.1.m1.1.1" xref="S5.SS1.SSS0.Px2.p1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px2.p1.1.m1.1b"><times id="S5.SS1.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S5.SS1.SSS0.Px2.p1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px2.p1.1.m1.1c">\times</annotation></semantics></math> speedup in end-to-end wall-clock training time over the baseline, as shown in Table <a href="#S5.F7" title="Figure 7 ‣ Pre-training convergence comparisons. ‣ 5.1 Experimental Results ‣ 5 Evaluation ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>.</p>
</div>
</section>
<section id="S5.SS1.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Downstream task accuracy.</h4>

<div id="S5.SS1.SSS0.Px3.p1" class="ltx_para">
<p id="S5.SS1.SSS0.Px3.p1.1" class="ltx_p">Despite improved training speed, one may still wonder whether such a method is as effective as the baseline model on downstream tasks.
Table <a href="#S5.T2" title="Table 2 ‣ Downstream task accuracy. ‣ 5.1 Experimental Results ‣ 5 Evaluation ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> shows our results on the GLUE dataset compared to the baseline. Our baseline is comparable with the original BERT-Base (on the test set), and our PLD method achieves a higher GLUE score than our baseline (83.2 vs. 82.1) when fine-tuning the checkpoint (186). We also dump model checkpoints from different epochs during pre-training and fine-tune these models.
The checkpoint 87 corresponds to the validation loss at 1.75 achieved by PLD. The GLUE score is slightly worse than the baseline (81.6 vs. 82.1). However, by fine-tuning at checkpoint 100, PLD achieves a higher score than the baseline (82.3 vs. 82.1) at checkpoint 186. In terms of the pre-training wall clock time, PLD requires 15.56h vs. the baseline with 39.15h to get similar accuracy on downstream tasks, which corresponds to a 2.5<math id="S5.SS1.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S5.SS1.SSS0.Px3.p1.1.m1.1a"><mo id="S5.SS1.SSS0.Px3.p1.1.m1.1.1" xref="S5.SS1.SSS0.Px3.p1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px3.p1.1.m1.1b"><times id="S5.SS1.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S5.SS1.SSS0.Px3.p1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px3.p1.1.m1.1c">\times</annotation></semantics></math> speedup.</p>
</div>
<figure id="S5.T2" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:90%;"><span class="ltx_tag ltx_tag_table">Table 2: </span>The results on the GLUE benchmark. The number below each task denotes the number of training examples.
The metrics for these tasks can be found in the GLUE paper <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">6</span></a>]</cite>. We compute the geometric mean of the metrics as the GLUE score.</figcaption>
<table id="S5.T2.5" class="ltx_tabular ltx_centering ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S5.T2.5.6.1" class="ltx_tr">
<td id="S5.T2.5.6.1.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_l ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;" rowspan="2"><span id="S5.T2.5.6.1.1.1" class="ltx_text" style="font-size:90%;">Model</span></td>
<td id="S5.T2.5.6.1.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.T2.5.6.1.2.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T2.5.6.1.2.1.1" class="ltx_tr">
<td id="S5.T2.5.6.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.6.1.2.1.1.1.1" class="ltx_text" style="font-size:90%;">RTE</span></td>
</tr>
<tr id="S5.T2.5.6.1.2.1.2" class="ltx_tr">
<td id="S5.T2.5.6.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.6.1.2.1.2.1.1" class="ltx_text" style="font-size:90%;">(Acc.)</span></td>
</tr>
</table>
</td>
<td id="S5.T2.5.6.1.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.T2.5.6.1.3.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T2.5.6.1.3.1.1" class="ltx_tr">
<td id="S5.T2.5.6.1.3.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.6.1.3.1.1.1.1" class="ltx_text" style="font-size:90%;">MRPC</span></td>
</tr>
<tr id="S5.T2.5.6.1.3.1.2" class="ltx_tr">
<td id="S5.T2.5.6.1.3.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.6.1.3.1.2.1.1" class="ltx_text" style="font-size:90%;">(F1/Acc.)</span></td>
</tr>
</table>
</td>
<td id="S5.T2.5.6.1.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.T2.5.6.1.4.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T2.5.6.1.4.1.1" class="ltx_tr">
<td id="S5.T2.5.6.1.4.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.6.1.4.1.1.1.1" class="ltx_text" style="font-size:90%;">STS-B</span></td>
</tr>
<tr id="S5.T2.5.6.1.4.1.2" class="ltx_tr">
<td id="S5.T2.5.6.1.4.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.6.1.4.1.2.1.1" class="ltx_text" style="font-size:90%;">(PCC/SCC)</span></td>
</tr>
</table>
</td>
<td id="S5.T2.5.6.1.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.T2.5.6.1.5.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T2.5.6.1.5.1.1" class="ltx_tr">
<td id="S5.T2.5.6.1.5.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.6.1.5.1.1.1.1" class="ltx_text" style="font-size:90%;">CoLA</span></td>
</tr>
<tr id="S5.T2.5.6.1.5.1.2" class="ltx_tr">
<td id="S5.T2.5.6.1.5.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.6.1.5.1.2.1.1" class="ltx_text" style="font-size:90%;">(MCC)</span></td>
</tr>
</table>
</td>
<td id="S5.T2.5.6.1.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.T2.5.6.1.6.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T2.5.6.1.6.1.1" class="ltx_tr">
<td id="S5.T2.5.6.1.6.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.6.1.6.1.1.1.1" class="ltx_text" style="font-size:90%;">SST-2</span></td>
</tr>
<tr id="S5.T2.5.6.1.6.1.2" class="ltx_tr">
<td id="S5.T2.5.6.1.6.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.6.1.6.1.2.1.1" class="ltx_text" style="font-size:90%;">(Acc.)</span></td>
</tr>
</table>
</td>
<td id="S5.T2.5.6.1.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.T2.5.6.1.7.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T2.5.6.1.7.1.1" class="ltx_tr">
<td id="S5.T2.5.6.1.7.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.6.1.7.1.1.1.1" class="ltx_text" style="font-size:90%;">QNLI</span></td>
</tr>
<tr id="S5.T2.5.6.1.7.1.2" class="ltx_tr">
<td id="S5.T2.5.6.1.7.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.6.1.7.1.2.1.1" class="ltx_text" style="font-size:90%;">(Acc.)</span></td>
</tr>
</table>
</td>
<td id="S5.T2.5.6.1.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.T2.5.6.1.8.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T2.5.6.1.8.1.1" class="ltx_tr">
<td id="S5.T2.5.6.1.8.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.6.1.8.1.1.1.1" class="ltx_text" style="font-size:90%;">QQP</span></td>
</tr>
<tr id="S5.T2.5.6.1.8.1.2" class="ltx_tr">
<td id="S5.T2.5.6.1.8.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.6.1.8.1.2.1.1" class="ltx_text" style="font-size:90%;">(F1/Acc.)</span></td>
</tr>
</table>
</td>
<td id="S5.T2.5.6.1.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.T2.5.6.1.9.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T2.5.6.1.9.1.1" class="ltx_tr">
<td id="S5.T2.5.6.1.9.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.6.1.9.1.1.1.1" class="ltx_text" style="font-size:90%;">MNLI-mm</span></td>
</tr>
<tr id="S5.T2.5.6.1.9.1.2" class="ltx_tr">
<td id="S5.T2.5.6.1.9.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.6.1.9.1.2.1.1" class="ltx_text" style="font-size:90%;">-/m (Acc.)</span></td>
</tr>
</table>
</td>
<td id="S5.T2.5.6.1.10" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;" rowspan="2"><span id="S5.T2.5.6.1.10.1" class="ltx_text" style="font-size:90%;">GLUE</span></td>
</tr>
<tr id="S5.T2.5.7.2" class="ltx_tr">
<td id="S5.T2.5.7.2.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.7.2.1.1" class="ltx_text" style="font-size:90%;">2.5K</span></td>
<td id="S5.T2.5.7.2.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.7.2.2.1" class="ltx_text" style="font-size:90%;">3.7K</span></td>
<td id="S5.T2.5.7.2.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.7.2.3.1" class="ltx_text" style="font-size:90%;">5.7K</span></td>
<td id="S5.T2.5.7.2.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.7.2.4.1" class="ltx_text" style="font-size:90%;">8.5K</span></td>
<td id="S5.T2.5.7.2.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.7.2.5.1" class="ltx_text" style="font-size:90%;">67K</span></td>
<td id="S5.T2.5.7.2.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.7.2.6.1" class="ltx_text" style="font-size:90%;">108K</span></td>
<td id="S5.T2.5.7.2.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.7.2.7.1" class="ltx_text" style="font-size:90%;">368K</span></td>
<td id="S5.T2.5.7.2.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.7.2.8.1" class="ltx_text" style="font-size:90%;">393K</span></td>
</tr>
<tr id="S5.T2.1.1" class="ltx_tr">
<td id="S5.T2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_l ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S5.T2.1.1.1.1" class="ltx_text" style="font-size:90%;">BERT</span><sub id="S5.T2.1.1.1.2" class="ltx_sub"><span id="S5.T2.1.1.1.2.1" class="ltx_text ltx_font_italic" style="font-size:90%;">base</span></sub><span id="S5.T2.1.1.1.3" class="ltx_text" style="font-size:90%;"> (original)</span>
</td>
<td id="S5.T2.1.1.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.1.1.2.1" class="ltx_text" style="font-size:90%;">66.4</span></td>
<td id="S5.T2.1.1.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.1.1.3.1" class="ltx_text" style="font-size:90%;">88.9/84.8</span></td>
<td id="S5.T2.1.1.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.1.1.4.1" class="ltx_text" style="font-size:90%;">87.1/89.2</span></td>
<td id="S5.T2.1.1.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.1.1.5.1" class="ltx_text" style="font-size:90%;">52.1</span></td>
<td id="S5.T2.1.1.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.1.1.6.1" class="ltx_text ltx_font_bold" style="font-size:90%;">93.5</span></td>
<td id="S5.T2.1.1.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.1.1.7.1" class="ltx_text ltx_font_bold" style="font-size:90%;">90.5</span></td>
<td id="S5.T2.1.1.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.1.1.8.1" class="ltx_text" style="font-size:90%;">71.2/89.2</span></td>
<td id="S5.T2.1.1.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S5.T2.1.1.9.1" class="ltx_text ltx_font_bold" style="font-size:90%;">84.6</span><span id="S5.T2.1.1.9.2" class="ltx_text" style="font-size:90%;">/83.4</span>
</td>
<td id="S5.T2.1.1.10" class="ltx_td ltx_nopad_l ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.1.1.10.1" class="ltx_text" style="font-size:90%;">80.7</span></td>
</tr>
<tr id="S5.T2.2.2" class="ltx_tr">
<td id="S5.T2.2.2.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_l ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S5.T2.2.2.1.1" class="ltx_text" style="font-size:90%;">BERT</span><sub id="S5.T2.2.2.1.2" class="ltx_sub"><span id="S5.T2.2.2.1.2.1" class="ltx_text ltx_font_italic" style="font-size:90%;">base</span></sub><span id="S5.T2.2.2.1.3" class="ltx_text" style="font-size:90%;"> (Baseline, ckp186)</span>
</td>
<td id="S5.T2.2.2.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.2.2.2.1" class="ltx_text" style="font-size:90%;">67.8</span></td>
<td id="S5.T2.2.2.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.2.2.3.1" class="ltx_text" style="font-size:90%;">88.0/86.0</span></td>
<td id="S5.T2.2.2.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S5.T2.2.2.4.1" class="ltx_text" style="font-size:90%;">89.5/</span><span id="S5.T2.2.2.4.2" class="ltx_text ltx_font_bold" style="font-size:90%;">89.2</span>
</td>
<td id="S5.T2.2.2.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.2.2.5.1" class="ltx_text" style="font-size:90%;">52.5</span></td>
<td id="S5.T2.2.2.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.2.2.6.1" class="ltx_text" style="font-size:90%;">91.2</span></td>
<td id="S5.T2.2.2.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.2.2.7.1" class="ltx_text" style="font-size:90%;">87.1</span></td>
<td id="S5.T2.2.2.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.2.2.8.1" class="ltx_text" style="font-size:90%;">89.0/90.6</span></td>
<td id="S5.T2.2.2.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.2.2.9.1" class="ltx_text" style="font-size:90%;">82.5/83.4</span></td>
<td id="S5.T2.2.2.10" class="ltx_td ltx_nopad_l ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.2.2.10.1" class="ltx_text" style="font-size:90%;">82.1</span></td>
</tr>
<tr id="S5.T2.3.3" class="ltx_tr">
<td id="S5.T2.3.3.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_l ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S5.T2.3.3.1.1" class="ltx_text" style="font-size:90%;">BERT</span><sub id="S5.T2.3.3.1.2" class="ltx_sub"><span id="S5.T2.3.3.1.2.1" class="ltx_text ltx_font_italic" style="font-size:90%;">base</span></sub><span id="S5.T2.3.3.1.3" class="ltx_text" style="font-size:90%;"> (PLD, ckp87)</span>
</td>
<td id="S5.T2.3.3.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.3.3.2.1" class="ltx_text" style="font-size:90%;">66</span></td>
<td id="S5.T2.3.3.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.3.3.3.1" class="ltx_text" style="font-size:90%;">88.2/85.6</span></td>
<td id="S5.T2.3.3.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.3.3.4.1" class="ltx_text" style="font-size:90%;">88.9/88.4</span></td>
<td id="S5.T2.3.3.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.3.3.5.1" class="ltx_text" style="font-size:90%;">54.5</span></td>
<td id="S5.T2.3.3.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.3.3.6.1" class="ltx_text" style="font-size:90%;">91</span></td>
<td id="S5.T2.3.3.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.3.3.7.1" class="ltx_text" style="font-size:90%;">86.3</span></td>
<td id="S5.T2.3.3.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.3.3.8.1" class="ltx_text" style="font-size:90%;">87.4/89.1</span></td>
<td id="S5.T2.3.3.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.3.3.9.1" class="ltx_text" style="font-size:90%;">81.6/82.4</span></td>
<td id="S5.T2.3.3.10" class="ltx_td ltx_nopad_l ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.3.3.10.1" class="ltx_text" style="font-size:90%;">81.6</span></td>
</tr>
<tr id="S5.T2.4.4" class="ltx_tr">
<td id="S5.T2.4.4.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_l ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S5.T2.4.4.1.1" class="ltx_text" style="font-size:90%;">BERT</span><sub id="S5.T2.4.4.1.2" class="ltx_sub"><span id="S5.T2.4.4.1.2.1" class="ltx_text ltx_font_italic" style="font-size:90%;">base</span></sub><span id="S5.T2.4.4.1.3" class="ltx_text" style="font-size:90%;"> (PLD, ckp100)</span>
</td>
<td id="S5.T2.4.4.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.4.4.2.1" class="ltx_text" style="font-size:90%;">68.2</span></td>
<td id="S5.T2.4.4.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.4.4.3.1" class="ltx_text" style="font-size:90%;">88.2/85.8</span></td>
<td id="S5.T2.4.4.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.4.4.4.1" class="ltx_text" style="font-size:90%;">89.3/88.9</span></td>
<td id="S5.T2.4.4.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.4.4.5.1" class="ltx_text" style="font-size:90%;">56.1</span></td>
<td id="S5.T2.4.4.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.4.4.6.1" class="ltx_text" style="font-size:90%;">91.5</span></td>
<td id="S5.T2.4.4.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.4.4.7.1" class="ltx_text" style="font-size:90%;">86.9</span></td>
<td id="S5.T2.4.4.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.4.4.8.1" class="ltx_text" style="font-size:90%;">87.7/89.3</span></td>
<td id="S5.T2.4.4.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.4.4.9.1" class="ltx_text" style="font-size:90%;">82.4/82.6</span></td>
<td id="S5.T2.4.4.10" class="ltx_td ltx_nopad_l ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.4.4.10.1" class="ltx_text" style="font-size:90%;">82.3</span></td>
</tr>
<tr id="S5.T2.5.5" class="ltx_tr">
<td id="S5.T2.5.5.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_b ltx_border_l ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S5.T2.5.5.1.1" class="ltx_text" style="font-size:90%;">BERT</span><sub id="S5.T2.5.5.1.2" class="ltx_sub"><span id="S5.T2.5.5.1.2.1" class="ltx_text ltx_font_italic" style="font-size:90%;">base</span></sub><span id="S5.T2.5.5.1.3" class="ltx_text" style="font-size:90%;"> (PLD, ckp186)</span>
</td>
<td id="S5.T2.5.5.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_b ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.5.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">69</span></td>
<td id="S5.T2.5.5.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_b ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.5.3.1" class="ltx_text ltx_font_bold" style="font-size:90%;">88.9/86.5</span></td>
<td id="S5.T2.5.5.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_b ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S5.T2.5.5.4.1" class="ltx_text ltx_font_bold" style="font-size:90%;">89.6</span><span id="S5.T2.5.5.4.2" class="ltx_text" style="font-size:90%;">/89.1</span>
</td>
<td id="S5.T2.5.5.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_b ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.5.5.1" class="ltx_text ltx_font_bold" style="font-size:90%;">59.4</span></td>
<td id="S5.T2.5.5.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_b ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.5.6.1" class="ltx_text" style="font-size:90%;">91.8</span></td>
<td id="S5.T2.5.5.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_b ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.5.7.1" class="ltx_text" style="font-size:90%;">88</span></td>
<td id="S5.T2.5.5.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_b ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.5.8.1" class="ltx_text ltx_font_bold" style="font-size:90%;">89.4/90.9</span></td>
<td id="S5.T2.5.5.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_b ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S5.T2.5.5.9.1" class="ltx_text" style="font-size:90%;">83.1/</span><span id="S5.T2.5.5.9.2" class="ltx_text ltx_font_bold" style="font-size:90%;">83.5</span>
</td>
<td id="S5.T2.5.5.10" class="ltx_td ltx_nopad_l ltx_align_left ltx_border_b ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T2.5.5.10.1" class="ltx_text ltx_font_bold" style="font-size:90%;">83.2</span></td>
</tr>
</tbody>
</table>
</figure>
<div id="S5.SS1.SSS0.Px3.p2" class="ltx_para">
<p id="S5.SS1.SSS0.Px3.p2.1" class="ltx_p">Fig. <a href="#S5.F8" title="Figure 8 ‣ Downstream task accuracy. ‣ 5.1 Experimental Results ‣ 5 Evaluation ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> illustrates the fine-tuning results between the baseline and PLD on GLUE tasks over different checkpoints
Overall, we observe that PLD not only trains BERT faster in pre-training but also preserves the performance on downstream tasks. In each figure, we observe that both curves have a similar shape at the beginning because no layer drop is added. For later checkpoints, PLD smoothly adds layer drop. Interestingly, we note that the baseline model has fluctuations in testing accuracy. In contrast, the downstream task accuracy from PLD is consistently increasing as the
number of training epochs increases. This indicates that PLD takes a more robust optimization path toward the optimum. We also observe that our model achieves higher performance on MNLI, QNLI, QQP, RTE, SST-2, and CoLA on later checkpoints, indicating that the model trained with our approach also generalizes better on downstream tasks than our baseline does.</p>
</div>
<div id="S5.SS1.SSS0.Px3.p3" class="ltx_para">
<p id="S5.SS1.SSS0.Px3.p3.1" class="ltx_p">From a knowledge transferability perspective, the goal of training a language model is to learn a good representation of natural language that ideally ignores the data-dependent noise and generalizes well to downstream tasks. However, training a model with a constant depth is at least somewhat noisy and can bias the model to prefer certain representations, whereas PLD enables more sub-network configurations to be created during training Transformer networks.
Each of the L ST blocks is either active or inactive, resulting in <math id="S5.SS1.SSS0.Px3.p3.1.m1.1" class="ltx_Math" alttext="2^{L}" display="inline"><semantics id="S5.SS1.SSS0.Px3.p3.1.m1.1a"><msup id="S5.SS1.SSS0.Px3.p3.1.m1.1.1" xref="S5.SS1.SSS0.Px3.p3.1.m1.1.1.cmml"><mn id="S5.SS1.SSS0.Px3.p3.1.m1.1.1.2" xref="S5.SS1.SSS0.Px3.p3.1.m1.1.1.2.cmml">2</mn><mi id="S5.SS1.SSS0.Px3.p3.1.m1.1.1.3" xref="S5.SS1.SSS0.Px3.p3.1.m1.1.1.3.cmml">L</mi></msup><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px3.p3.1.m1.1b"><apply id="S5.SS1.SSS0.Px3.p3.1.m1.1.1.cmml" xref="S5.SS1.SSS0.Px3.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S5.SS1.SSS0.Px3.p3.1.m1.1.1.1.cmml" xref="S5.SS1.SSS0.Px3.p3.1.m1.1.1">superscript</csymbol><cn type="integer" id="S5.SS1.SSS0.Px3.p3.1.m1.1.1.2.cmml" xref="S5.SS1.SSS0.Px3.p3.1.m1.1.1.2">2</cn><ci id="S5.SS1.SSS0.Px3.p3.1.m1.1.1.3.cmml" xref="S5.SS1.SSS0.Px3.p3.1.m1.1.1.3">𝐿</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px3.p3.1.m1.1c">2^{L}</annotation></semantics></math> possible network combinations. By selecting a different submodular in each mini-batch, PLD encourages the submodular to produce good results independently. This allows the unsupervised pre-training model to obtain a more general representation by averaging the noise patterns, which helps the model to better generalize to new tasks.
On the other hand, during inference, the full network is presented, causing the effect of ensembling different sub-networks.</p>
</div>
<figure id="S5.F8" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S5.F8.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/fine-tune/fine-tune-MNLI-m-ci.png" id="S5.F8.sf1.g1" class="ltx_graphics ltx_img_landscape" width="154" height="121" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(a) </span>MNLI-m</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S5.F8.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/fine-tune/fine-tune-MNLI-mm-ci.png" id="S5.F8.sf2.g1" class="ltx_graphics ltx_img_landscape" width="154" height="121" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(b) </span>MNLI-mm</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S5.F8.sf3" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/fine-tune/fine-tune-QNLI-ci.png" id="S5.F8.sf3.g1" class="ltx_graphics ltx_img_landscape" width="154" height="121" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(c) </span>QNLI</figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S5.F8.sf4" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/fine-tune/fine-tune-QQP-acc-ci.png" id="S5.F8.sf4.g1" class="ltx_graphics ltx_img_landscape" width="154" height="121" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(d) </span>QQP</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S5.F8.sf5" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/fine-tune/fine-tune-RTE-acc-ci.png" id="S5.F8.sf5.g1" class="ltx_graphics ltx_img_landscape" width="154" height="121" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(e) </span>RTE</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S5.F8.sf6" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/fine-tune/fine-tune-SST-2-ci.png" id="S5.F8.sf6.g1" class="ltx_graphics ltx_img_landscape" width="154" height="121" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(f) </span>SST-2</figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S5.F8.sf7" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/fine-tune/fine-tune-WNLI-ci.png" id="S5.F8.sf7.g1" class="ltx_graphics ltx_img_landscape" width="154" height="121" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(g) </span>WNLI</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S5.F8.sf8" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/fine-tune/fine-tune-CoLA-ci.png" id="S5.F8.sf8.g1" class="ltx_graphics ltx_img_landscape" width="154" height="121" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(h) </span>CoLA</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S5.F8.sf9" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/fine-tune/fine-tune-MRPC-acc-ci.png" id="S5.F8.sf9.g1" class="ltx_graphics ltx_img_landscape" width="154" height="121" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(i) </span>MRPC (acc.)</figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S5.F8.sf10" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/fine-tune/fine-tune-MRPC-f1-ci.png" id="S5.F8.sf10.g1" class="ltx_graphics ltx_img_landscape" width="154" height="121" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(j) </span>MRPC (F1.)</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S5.F8.sf11" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/fine-tune/fine-tune-SST-B-Pearson-ci.png" id="S5.F8.sf11.g1" class="ltx_graphics ltx_img_landscape" width="154" height="121" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(k) </span>SST-B (PCC)</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S5.F8.sf12" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/fine-tune/fine-tune-SST-B-Spearman-ci.png" id="S5.F8.sf12.g1" class="ltx_graphics ltx_img_landscape" width="154" height="121" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(l) </span>SST-B (SCC)</figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">Figure 8: </span>The fine-tuning results at different checkpoints.</figcaption>
</figure>
</section>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>Ablation Studies</h3>

<section id="S5.SS2.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Downstream task fine-tuning sensitivity.</h4>

<div id="S5.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="S5.SS2.SSS0.Px1.p1.1" class="ltx_p">To further verify that our approach not only stabilizes training but also improves downstream tasks, we show a grid search on learning rates
{1e-5, 3e-5, 5e-5, 7e-5, 9e-5, 1e-4}. As illustrated in Fig. <a href="#S5.F9" title="Figure 9 ‣ Downstream task fine-tuning sensitivity. ‣ 5.2 Ablation Studies ‣ 5 Evaluation ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>, the baseline is vulnerable to the choice of learning rates. Specifically, the fine-tuning results are often much worse with a large learning rate, while PLD is more robust and often achieves better results with large learning rates.</p>
</div>
<figure id="S5.F9" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S5.F9.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/heatmap/heatmap-MNLI-m.png" id="S5.F9.sf1.g1" class="ltx_graphics ltx_img_landscape" width="158" height="121" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(a) </span>MNLI-m</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S5.F9.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/heatmap/heatmap-MNLI-mm.png" id="S5.F9.sf2.g1" class="ltx_graphics ltx_img_landscape" width="158" height="121" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(b) </span>MNLI-mm</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S5.F9.sf3" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/heatmap/heatmap-QNLI.png" id="S5.F9.sf3.g1" class="ltx_graphics ltx_img_landscape" width="158" height="121" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(c) </span>QNLI</figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S5.F9.sf4" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/heatmap/heatmap-QQP.png" id="S5.F9.sf4.g1" class="ltx_graphics ltx_img_landscape" width="158" height="121" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(d) </span>QQP</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S5.F9.sf5" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/heatmap/heatmap-RTE.png" id="S5.F9.sf5.g1" class="ltx_graphics ltx_img_landscape" width="158" height="121" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(e) </span>RTE</figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_3">
<figure id="S5.F9.sf6" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2010.13369/assets/figs/heatmap/heatmap-SST-2.png" id="S5.F9.sf6.g1" class="ltx_graphics ltx_img_landscape" width="158" height="121" alt="Refer to caption">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">(f) </span>SST-2</figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering" style="font-size:90%;"><span class="ltx_tag ltx_tag_figure">Figure 9: </span>The fine-tuning results at different checkpoints.</figcaption>
</figure>
</section>
<section id="S5.SS2.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">The Effect of <math id="S5.SS2.SSS0.Px2.1.m1.1" class="ltx_Math" alttext="\bar{\theta}" display="inline"><semantics id="S5.SS2.SSS0.Px2.1.m1.1b"><mover accent="true" id="S5.SS2.SSS0.Px2.1.m1.1.1" xref="S5.SS2.SSS0.Px2.1.m1.1.1.cmml"><mi id="S5.SS2.SSS0.Px2.1.m1.1.1.2" xref="S5.SS2.SSS0.Px2.1.m1.1.1.2.cmml">θ</mi><mo id="S5.SS2.SSS0.Px2.1.m1.1.1.1" xref="S5.SS2.SSS0.Px2.1.m1.1.1.1.cmml">¯</mo></mover><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS0.Px2.1.m1.1c"><apply id="S5.SS2.SSS0.Px2.1.m1.1.1.cmml" xref="S5.SS2.SSS0.Px2.1.m1.1.1"><ci id="S5.SS2.SSS0.Px2.1.m1.1.1.1.cmml" xref="S5.SS2.SSS0.Px2.1.m1.1.1.1">¯</ci><ci id="S5.SS2.SSS0.Px2.1.m1.1.1.2.cmml" xref="S5.SS2.SSS0.Px2.1.m1.1.1.2">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS0.Px2.1.m1.1d">\bar{\theta}</annotation></semantics></math>.</h4>

<div id="S5.SS2.SSS0.Px2.p1" class="ltx_para">
<p id="S5.SS2.SSS0.Px2.p1.3" class="ltx_p">We test different values of the keep ratio <math id="S5.SS2.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="\bar{\theta}" display="inline"><semantics id="S5.SS2.SSS0.Px2.p1.1.m1.1a"><mover accent="true" id="S5.SS2.SSS0.Px2.p1.1.m1.1.1" xref="S5.SS2.SSS0.Px2.p1.1.m1.1.1.cmml"><mi id="S5.SS2.SSS0.Px2.p1.1.m1.1.1.2" xref="S5.SS2.SSS0.Px2.p1.1.m1.1.1.2.cmml">θ</mi><mo id="S5.SS2.SSS0.Px2.p1.1.m1.1.1.1" xref="S5.SS2.SSS0.Px2.p1.1.m1.1.1.1.cmml">¯</mo></mover><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS0.Px2.p1.1.m1.1b"><apply id="S5.SS2.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S5.SS2.SSS0.Px2.p1.1.m1.1.1"><ci id="S5.SS2.SSS0.Px2.p1.1.m1.1.1.1.cmml" xref="S5.SS2.SSS0.Px2.p1.1.m1.1.1.1">¯</ci><ci id="S5.SS2.SSS0.Px2.p1.1.m1.1.1.2.cmml" xref="S5.SS2.SSS0.Px2.p1.1.m1.1.1.2">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS0.Px2.p1.1.m1.1c">\bar{\theta}</annotation></semantics></math> and identify <math id="S5.SS2.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="0.5\leq\bar{\theta}\leq 0.9" display="inline"><semantics id="S5.SS2.SSS0.Px2.p1.2.m2.1a"><mrow id="S5.SS2.SSS0.Px2.p1.2.m2.1.1" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1.cmml"><mn id="S5.SS2.SSS0.Px2.p1.2.m2.1.1.2" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1.2.cmml">0.5</mn><mo id="S5.SS2.SSS0.Px2.p1.2.m2.1.1.3" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1.3.cmml">≤</mo><mover accent="true" id="S5.SS2.SSS0.Px2.p1.2.m2.1.1.4" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1.4.cmml"><mi id="S5.SS2.SSS0.Px2.p1.2.m2.1.1.4.2" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1.4.2.cmml">θ</mi><mo id="S5.SS2.SSS0.Px2.p1.2.m2.1.1.4.1" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1.4.1.cmml">¯</mo></mover><mo id="S5.SS2.SSS0.Px2.p1.2.m2.1.1.5" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1.5.cmml">≤</mo><mn id="S5.SS2.SSS0.Px2.p1.2.m2.1.1.6" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1.6.cmml">0.9</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS0.Px2.p1.2.m2.1b"><apply id="S5.SS2.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1"><and id="S5.SS2.SSS0.Px2.p1.2.m2.1.1a.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1"></and><apply id="S5.SS2.SSS0.Px2.p1.2.m2.1.1b.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1"><leq id="S5.SS2.SSS0.Px2.p1.2.m2.1.1.3.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1.3"></leq><cn type="float" id="S5.SS2.SSS0.Px2.p1.2.m2.1.1.2.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1.2">0.5</cn><apply id="S5.SS2.SSS0.Px2.p1.2.m2.1.1.4.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1.4"><ci id="S5.SS2.SSS0.Px2.p1.2.m2.1.1.4.1.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1.4.1">¯</ci><ci id="S5.SS2.SSS0.Px2.p1.2.m2.1.1.4.2.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1.4.2">𝜃</ci></apply></apply><apply id="S5.SS2.SSS0.Px2.p1.2.m2.1.1c.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1"><leq id="S5.SS2.SSS0.Px2.p1.2.m2.1.1.5.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1.5"></leq><share href="#S5.SS2.SSS0.Px2.p1.2.m2.1.1.4.cmml" id="S5.SS2.SSS0.Px2.p1.2.m2.1.1d.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1"></share><cn type="float" id="S5.SS2.SSS0.Px2.p1.2.m2.1.1.6.cmml" xref="S5.SS2.SSS0.Px2.p1.2.m2.1.1.6">0.9</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS0.Px2.p1.2.m2.1c">0.5\leq\bar{\theta}\leq 0.9</annotation></semantics></math> as a good range, as shown in Fig. <a href="#S5.F10" title="Figure 10 ‣ The Effect of 𝜃̄. ‣ 5.2 Ablation Studies ‣ 5 Evaluation ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a> in the Appendix. We observe that the algorithm may diverge if <math id="S5.SS2.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="\bar{\theta}" display="inline"><semantics id="S5.SS2.SSS0.Px2.p1.3.m3.1a"><mover accent="true" id="S5.SS2.SSS0.Px2.p1.3.m3.1.1" xref="S5.SS2.SSS0.Px2.p1.3.m3.1.1.cmml"><mi id="S5.SS2.SSS0.Px2.p1.3.m3.1.1.2" xref="S5.SS2.SSS0.Px2.p1.3.m3.1.1.2.cmml">θ</mi><mo id="S5.SS2.SSS0.Px2.p1.3.m3.1.1.1" xref="S5.SS2.SSS0.Px2.p1.3.m3.1.1.1.cmml">¯</mo></mover><annotation-xml encoding="MathML-Content" id="S5.SS2.SSS0.Px2.p1.3.m3.1b"><apply id="S5.SS2.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S5.SS2.SSS0.Px2.p1.3.m3.1.1"><ci id="S5.SS2.SSS0.Px2.p1.3.m3.1.1.1.cmml" xref="S5.SS2.SSS0.Px2.p1.3.m3.1.1.1">¯</ci><ci id="S5.SS2.SSS0.Px2.p1.3.m3.1.1.2.cmml" xref="S5.SS2.SSS0.Px2.p1.3.m3.1.1.2">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.SSS0.Px2.p1.3.m3.1c">\bar{\theta}</annotation></semantics></math> is too small (e.g., 0.3).</p>
</div>
<figure id="S5.F10" class="ltx_figure"><img src="/html/2010.13369/assets/figs/varying-keep-ratio.png" id="S5.F10.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="236" height="186" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 10: </span>Convergence curves varying the keep ratio <math id="S5.F10.2.m1.1" class="ltx_Math" alttext="\bar{\theta}" display="inline"><semantics id="S5.F10.2.m1.1b"><mover accent="true" id="S5.F10.2.m1.1.1" xref="S5.F10.2.m1.1.1.cmml"><mi id="S5.F10.2.m1.1.1.2" xref="S5.F10.2.m1.1.1.2.cmml">θ</mi><mo id="S5.F10.2.m1.1.1.1" xref="S5.F10.2.m1.1.1.1.cmml">¯</mo></mover><annotation-xml encoding="MathML-Content" id="S5.F10.2.m1.1c"><apply id="S5.F10.2.m1.1.1.cmml" xref="S5.F10.2.m1.1.1"><ci id="S5.F10.2.m1.1.1.1.cmml" xref="S5.F10.2.m1.1.1.1">¯</ci><ci id="S5.F10.2.m1.1.1.2.cmml" xref="S5.F10.2.m1.1.1.2">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.F10.2.m1.1d">\bar{\theta}</annotation></semantics></math>.</figcaption>
</figure>
</section>
<section id="S5.SS2.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">PLD vs. PreLN.</h4>

<div id="S5.SS2.SSS0.Px3.p1" class="ltx_para">
<p id="S5.SS2.SSS0.Px3.p1.1" class="ltx_p">To investigate the question on how PLD compares with PreLN, we run both PreLN with the hyperparameters used for training PostLN (lr=1e-4) and the hyperparameters used for PLD (lr=1e-3) to address the effect from the choice of hyperparameters. We train all configurations for the same number of epochs and fine-tune following the standard procedure. In both cases, PreLN is 24% slower than PLD, because PreLN still needs to perform the full forward and backward propagation in each iteration.</p>
</div>
<div id="S5.SS2.SSS0.Px3.p2" class="ltx_para">
<p id="S5.SS2.SSS0.Px3.p2.1" class="ltx_p">Table <a href="#S5.T3" title="Table 3 ‣ PLD vs. PreLN. ‣ 5.2 Ablation Studies ‣ 5 Evaluation ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> shows the fine-tuning results on GLUE tasks. When trained with the same hyperparameters as PostLN, PreLN appears to have a much worse GLUE score (80.2) compared with PostLN (82.1) on downstream tasks. This is because PreLN restricts layer outputs from depending too much on their own residual branches and inhibits the network from reaching its full potential, as recently studied in <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">44</span></a>]</cite>. When trained with the large learning rate as PLD, PreLN’s result have improved to 82.6 but is 0.6 points worse than PLD (83.2), despite using 24% more compute resource. PLD achieves better accuracy than PreLN because it encourages each residual branch to produce good results independently.</p>
</div>
<figure id="S5.T3" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:90%;"><span class="ltx_tag ltx_tag_table">Table 3: </span>Ablation studies of the fine-tuning results on the GLUE benchmark.</figcaption>
<table id="S5.T3.5" class="ltx_tabular ltx_centering ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S5.T3.5.6.1" class="ltx_tr">
<td id="S5.T3.5.6.1.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_l ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.6.1.1.1" class="ltx_text" style="font-size:90%;">Model</span></td>
<td id="S5.T3.5.6.1.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.T3.5.6.1.2.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T3.5.6.1.2.1.1" class="ltx_tr">
<td id="S5.T3.5.6.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.6.1.2.1.1.1.1" class="ltx_text" style="font-size:90%;">RTE</span></td>
</tr>
<tr id="S5.T3.5.6.1.2.1.2" class="ltx_tr">
<td id="S5.T3.5.6.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.6.1.2.1.2.1.1" class="ltx_text" style="font-size:90%;">(Acc.)</span></td>
</tr>
</table>
</td>
<td id="S5.T3.5.6.1.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.T3.5.6.1.3.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T3.5.6.1.3.1.1" class="ltx_tr">
<td id="S5.T3.5.6.1.3.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.6.1.3.1.1.1.1" class="ltx_text" style="font-size:90%;">MRPC</span></td>
</tr>
<tr id="S5.T3.5.6.1.3.1.2" class="ltx_tr">
<td id="S5.T3.5.6.1.3.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.6.1.3.1.2.1.1" class="ltx_text" style="font-size:90%;">(F1/Acc.)</span></td>
</tr>
</table>
</td>
<td id="S5.T3.5.6.1.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.T3.5.6.1.4.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T3.5.6.1.4.1.1" class="ltx_tr">
<td id="S5.T3.5.6.1.4.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.6.1.4.1.1.1.1" class="ltx_text" style="font-size:90%;">STS-B</span></td>
</tr>
<tr id="S5.T3.5.6.1.4.1.2" class="ltx_tr">
<td id="S5.T3.5.6.1.4.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.6.1.4.1.2.1.1" class="ltx_text" style="font-size:90%;">(PCC/SCC)</span></td>
</tr>
</table>
</td>
<td id="S5.T3.5.6.1.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.T3.5.6.1.5.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T3.5.6.1.5.1.1" class="ltx_tr">
<td id="S5.T3.5.6.1.5.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.6.1.5.1.1.1.1" class="ltx_text" style="font-size:90%;">CoLA</span></td>
</tr>
<tr id="S5.T3.5.6.1.5.1.2" class="ltx_tr">
<td id="S5.T3.5.6.1.5.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.6.1.5.1.2.1.1" class="ltx_text" style="font-size:90%;">(MCC)</span></td>
</tr>
</table>
</td>
<td id="S5.T3.5.6.1.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.T3.5.6.1.6.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T3.5.6.1.6.1.1" class="ltx_tr">
<td id="S5.T3.5.6.1.6.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.6.1.6.1.1.1.1" class="ltx_text" style="font-size:90%;">SST-2</span></td>
</tr>
<tr id="S5.T3.5.6.1.6.1.2" class="ltx_tr">
<td id="S5.T3.5.6.1.6.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.6.1.6.1.2.1.1" class="ltx_text" style="font-size:90%;">(Acc.)</span></td>
</tr>
</table>
</td>
<td id="S5.T3.5.6.1.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.T3.5.6.1.7.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T3.5.6.1.7.1.1" class="ltx_tr">
<td id="S5.T3.5.6.1.7.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.6.1.7.1.1.1.1" class="ltx_text" style="font-size:90%;">QNLI</span></td>
</tr>
<tr id="S5.T3.5.6.1.7.1.2" class="ltx_tr">
<td id="S5.T3.5.6.1.7.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.6.1.7.1.2.1.1" class="ltx_text" style="font-size:90%;">(Acc.)</span></td>
</tr>
</table>
</td>
<td id="S5.T3.5.6.1.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.T3.5.6.1.8.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T3.5.6.1.8.1.1" class="ltx_tr">
<td id="S5.T3.5.6.1.8.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.6.1.8.1.1.1.1" class="ltx_text" style="font-size:90%;">QQP</span></td>
</tr>
<tr id="S5.T3.5.6.1.8.1.2" class="ltx_tr">
<td id="S5.T3.5.6.1.8.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.6.1.8.1.2.1.1" class="ltx_text" style="font-size:90%;">(F1/Acc.)</span></td>
</tr>
</table>
</td>
<td id="S5.T3.5.6.1.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<table id="S5.T3.5.6.1.9.1" class="ltx_tabular ltx_align_middle">
<tr id="S5.T3.5.6.1.9.1.1" class="ltx_tr">
<td id="S5.T3.5.6.1.9.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.6.1.9.1.1.1.1" class="ltx_text" style="font-size:90%;">MNLI-m/mm</span></td>
</tr>
<tr id="S5.T3.5.6.1.9.1.2" class="ltx_tr">
<td id="S5.T3.5.6.1.9.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.6.1.9.1.2.1.1" class="ltx_text" style="font-size:90%;">(Acc.)</span></td>
</tr>
</table>
</td>
<td id="S5.T3.5.6.1.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.6.1.10.1" class="ltx_text" style="font-size:90%;">GLUE</span></td>
</tr>
<tr id="S5.T3.5.7.2" class="ltx_tr">
<td id="S5.T3.5.7.2.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_l ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.7.2.1.1" class="ltx_text" style="font-size:90%;">BERT (Original)</span></td>
<td id="S5.T3.5.7.2.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.7.2.2.1" class="ltx_text" style="font-size:90%;">66.4</span></td>
<td id="S5.T3.5.7.2.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S5.T3.5.7.2.3.1" class="ltx_text ltx_font_bold" style="font-size:90%;">88.9</span><span id="S5.T3.5.7.2.3.2" class="ltx_text" style="font-size:90%;">/84.8</span>
</td>
<td id="S5.T3.5.7.2.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.7.2.4.1" class="ltx_text" style="font-size:90%;">87.1/89.2</span></td>
<td id="S5.T3.5.7.2.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.7.2.5.1" class="ltx_text" style="font-size:90%;">52.1</span></td>
<td id="S5.T3.5.7.2.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.7.2.6.1" class="ltx_text ltx_font_bold" style="font-size:90%;">93.5</span></td>
<td id="S5.T3.5.7.2.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.7.2.7.1" class="ltx_text ltx_font_bold" style="font-size:90%;">90.5</span></td>
<td id="S5.T3.5.7.2.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.7.2.8.1" class="ltx_text" style="font-size:90%;">71.2/89.2</span></td>
<td id="S5.T3.5.7.2.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.7.2.9.1" class="ltx_text ltx_font_bold" style="font-size:90%;">84.6/83.4</span></td>
<td id="S5.T3.5.7.2.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.7.2.10.1" class="ltx_text" style="font-size:90%;">80.7</span></td>
</tr>
<tr id="S5.T3.5.8.3" class="ltx_tr">
<td id="S5.T3.5.8.3.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_l ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.8.3.1.1" class="ltx_text" style="font-size:90%;">BERT + PostLN</span></td>
<td id="S5.T3.5.8.3.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.8.3.2.1" class="ltx_text" style="font-size:90%;">67.8</span></td>
<td id="S5.T3.5.8.3.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.8.3.3.1" class="ltx_text" style="font-size:90%;">88.0/86.0</span></td>
<td id="S5.T3.5.8.3.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.8.3.4.1" class="ltx_text" style="font-size:90%;">89.5/89.2</span></td>
<td id="S5.T3.5.8.3.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.8.3.5.1" class="ltx_text" style="font-size:90%;">52.5</span></td>
<td id="S5.T3.5.8.3.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.8.3.6.1" class="ltx_text" style="font-size:90%;">91.2</span></td>
<td id="S5.T3.5.8.3.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.8.3.7.1" class="ltx_text" style="font-size:90%;">87.1</span></td>
<td id="S5.T3.5.8.3.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.8.3.8.1" class="ltx_text" style="font-size:90%;">89.0/90.6</span></td>
<td id="S5.T3.5.8.3.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.8.3.9.1" class="ltx_text" style="font-size:90%;">82.5/83.4</span></td>
<td id="S5.T3.5.8.3.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.8.3.10.1" class="ltx_text" style="font-size:90%;">82.1</span></td>
</tr>
<tr id="S5.T3.5.9.4" class="ltx_tr">
<td id="S5.T3.5.9.4.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_l ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.9.4.1.1" class="ltx_text" style="font-size:90%;">BERT + PreLN + Same lr</span></td>
<td id="S5.T3.5.9.4.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.9.4.2.1" class="ltx_text" style="font-size:90%;">66.0</span></td>
<td id="S5.T3.5.9.4.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.9.4.3.1" class="ltx_text" style="font-size:90%;">85.9/83.3</span></td>
<td id="S5.T3.5.9.4.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.9.4.4.1" class="ltx_text" style="font-size:90%;">88.2/87.9</span></td>
<td id="S5.T3.5.9.4.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.9.4.5.1" class="ltx_text" style="font-size:90%;">46.4</span></td>
<td id="S5.T3.5.9.4.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.9.4.6.1" class="ltx_text" style="font-size:90%;">90.5</span></td>
<td id="S5.T3.5.9.4.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.9.4.7.1" class="ltx_text" style="font-size:90%;">85.5</span></td>
<td id="S5.T3.5.9.4.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.9.4.8.1" class="ltx_text" style="font-size:90%;">89.0/90.6</span></td>
<td id="S5.T3.5.9.4.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.9.4.9.1" class="ltx_text" style="font-size:90%;">81.6/81.6</span></td>
<td id="S5.T3.5.9.4.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.9.4.10.1" class="ltx_text" style="font-size:90%;">80.2</span></td>
</tr>
<tr id="S5.T3.1.1" class="ltx_tr">
<td id="S5.T3.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_l ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S5.T3.1.1.1.1" class="ltx_text" style="font-size:90%;">BERT + PreLN + lr</span><math id="S5.T3.1.1.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S5.T3.1.1.1.m1.1a"><mo mathsize="90%" stretchy="false" id="S5.T3.1.1.1.m1.1.1" xref="S5.T3.1.1.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S5.T3.1.1.1.m1.1b"><ci id="S5.T3.1.1.1.m1.1.1.cmml" xref="S5.T3.1.1.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.T3.1.1.1.m1.1c">\uparrow</annotation></semantics></math>
</td>
<td id="S5.T3.1.1.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.1.1.2.1" class="ltx_text" style="font-size:90%;">67.8</span></td>
<td id="S5.T3.1.1.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.1.1.3.1" class="ltx_text" style="font-size:90%;">86.7/84.5</span></td>
<td id="S5.T3.1.1.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.1.1.4.1" class="ltx_text ltx_font_bold" style="font-size:90%;">89.6/89.1</span></td>
<td id="S5.T3.1.1.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.1.1.5.1" class="ltx_text" style="font-size:90%;">54.6</span></td>
<td id="S5.T3.1.1.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.1.1.6.1" class="ltx_text" style="font-size:90%;">91.9</span></td>
<td id="S5.T3.1.1.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.1.1.7.1" class="ltx_text" style="font-size:90%;">88.1</span></td>
<td id="S5.T3.1.1.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S5.T3.1.1.8.1" class="ltx_text" style="font-size:90%;">89.3/</span><span id="S5.T3.1.1.8.2" class="ltx_text ltx_font_bold" style="font-size:90%;">90.9</span>
</td>
<td id="S5.T3.1.1.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.1.1.9.1" class="ltx_text ltx_font_bold" style="font-size:90%;">83.6/83.7</span></td>
<td id="S5.T3.1.1.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.1.1.10.1" class="ltx_text" style="font-size:90%;">82.6</span></td>
</tr>
<tr id="S5.T3.2.2" class="ltx_tr">
<td id="S5.T3.2.2.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_l ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S5.T3.2.2.1.1" class="ltx_text" style="font-size:90%;">Shallow BERT + PreLN + lr</span><math id="S5.T3.2.2.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S5.T3.2.2.1.m1.1a"><mo mathsize="90%" stretchy="false" id="S5.T3.2.2.1.m1.1.1" xref="S5.T3.2.2.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S5.T3.2.2.1.m1.1b"><ci id="S5.T3.2.2.1.m1.1.1.cmml" xref="S5.T3.2.2.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.T3.2.2.1.m1.1c">\uparrow</annotation></semantics></math>
</td>
<td id="S5.T3.2.2.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.2.2.2.1" class="ltx_text" style="font-size:90%;">66.0</span></td>
<td id="S5.T3.2.2.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.2.2.3.1" class="ltx_text" style="font-size:90%;">85.9/83.5</span></td>
<td id="S5.T3.2.2.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.2.2.4.1" class="ltx_text" style="font-size:90%;">89.5/88.9</span></td>
<td id="S5.T3.2.2.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.2.2.5.1" class="ltx_text" style="font-size:90%;">54.7</span></td>
<td id="S5.T3.2.2.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.2.2.6.1" class="ltx_text" style="font-size:90%;">91.8</span></td>
<td id="S5.T3.2.2.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.2.2.7.1" class="ltx_text" style="font-size:90%;">86.1</span></td>
<td id="S5.T3.2.2.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.2.2.8.1" class="ltx_text" style="font-size:90%;">89.0/90.6</span></td>
<td id="S5.T3.2.2.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.2.2.9.1" class="ltx_text" style="font-size:90%;">82.7/82.9</span></td>
<td id="S5.T3.2.2.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.2.2.10.1" class="ltx_text" style="font-size:90%;">81.8</span></td>
</tr>
<tr id="S5.T3.3.3" class="ltx_tr">
<td id="S5.T3.3.3.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_l ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S5.T3.3.3.1.1" class="ltx_text" style="font-size:90%;">BERT + PreLN + lr</span><math id="S5.T3.3.3.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S5.T3.3.3.1.m1.1a"><mo mathsize="90%" stretchy="false" id="S5.T3.3.3.1.m1.1.1" xref="S5.T3.3.3.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S5.T3.3.3.1.m1.1b"><ci id="S5.T3.3.3.1.m1.1.1.cmml" xref="S5.T3.3.3.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.T3.3.3.1.m1.1c">\uparrow</annotation></semantics></math><span id="S5.T3.3.3.1.2" class="ltx_text" style="font-size:90%;"> + Rand.</span>
</td>
<td id="S5.T3.3.3.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.3.3.2.1" class="ltx_text" style="font-size:90%;">68.2</span></td>
<td id="S5.T3.3.3.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.3.3.3.1" class="ltx_text" style="font-size:90%;">88.2/86.2</span></td>
<td id="S5.T3.3.3.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.3.3.4.1" class="ltx_text" style="font-size:90%;">89.3/88.8</span></td>
<td id="S5.T3.3.3.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.3.3.5.1" class="ltx_text" style="font-size:90%;">56.8</span></td>
<td id="S5.T3.3.3.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.3.3.6.1" class="ltx_text" style="font-size:90%;">91.5</span></td>
<td id="S5.T3.3.3.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.3.3.7.1" class="ltx_text" style="font-size:90%;">87.2</span></td>
<td id="S5.T3.3.3.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.3.3.8.1" class="ltx_text" style="font-size:90%;">88.6/90.3</span></td>
<td id="S5.T3.3.3.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.3.3.9.1" class="ltx_text" style="font-size:90%;">82.9/83.3</span></td>
<td id="S5.T3.3.3.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.3.3.10.1" class="ltx_text" style="font-size:90%;">82.7</span></td>
</tr>
<tr id="S5.T3.4.4" class="ltx_tr">
<td id="S5.T3.4.4.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_l ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S5.T3.4.4.1.1" class="ltx_text" style="font-size:90%;">BERT + PreLN + lr</span><math id="S5.T3.4.4.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S5.T3.4.4.1.m1.1a"><mo mathsize="90%" stretchy="false" id="S5.T3.4.4.1.m1.1.1" xref="S5.T3.4.4.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S5.T3.4.4.1.m1.1b"><ci id="S5.T3.4.4.1.m1.1.1.cmml" xref="S5.T3.4.4.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.T3.4.4.1.m1.1c">\uparrow</annotation></semantics></math><span id="S5.T3.4.4.1.2" class="ltx_text" style="font-size:90%;"> + TD</span>
</td>
<td id="S5.T3.4.4.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.4.4.2.1" class="ltx_text" style="font-size:90%;">68.2</span></td>
<td id="S5.T3.4.4.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S5.T3.4.4.3.1" class="ltx_text" style="font-size:90%;">88.6/</span><span id="S5.T3.4.4.3.2" class="ltx_text ltx_font_bold" style="font-size:90%;">86.7</span>
</td>
<td id="S5.T3.4.4.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.4.4.4.1" class="ltx_text" style="font-size:90%;">89.4/88.9</span></td>
<td id="S5.T3.4.4.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.4.4.5.1" class="ltx_text" style="font-size:90%;">55.9</span></td>
<td id="S5.T3.4.4.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.4.4.6.1" class="ltx_text" style="font-size:90%;">91.3</span></td>
<td id="S5.T3.4.4.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.4.4.7.1" class="ltx_text" style="font-size:90%;">86.8</span></td>
<td id="S5.T3.4.4.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.4.4.8.1" class="ltx_text" style="font-size:90%;">89.1/90.7</span></td>
<td id="S5.T3.4.4.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.4.4.9.1" class="ltx_text" style="font-size:90%;">82.7/83.1</span></td>
<td id="S5.T3.4.4.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.4.4.10.1" class="ltx_text" style="font-size:90%;">82.7</span></td>
</tr>
<tr id="S5.T3.5.5" class="ltx_tr">
<td id="S5.T3.5.5.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_b ltx_border_l ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S5.T3.5.5.1.1" class="ltx_text" style="font-size:90%;">BERT + PreLN + lr</span><math id="S5.T3.5.5.1.m1.1" class="ltx_Math" alttext="\uparrow" display="inline"><semantics id="S5.T3.5.5.1.m1.1a"><mo mathsize="90%" stretchy="false" id="S5.T3.5.5.1.m1.1.1" xref="S5.T3.5.5.1.m1.1.1.cmml">↑</mo><annotation-xml encoding="MathML-Content" id="S5.T3.5.5.1.m1.1b"><ci id="S5.T3.5.5.1.m1.1.1.cmml" xref="S5.T3.5.5.1.m1.1.1">↑</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.T3.5.5.1.m1.1c">\uparrow</annotation></semantics></math><span id="S5.T3.5.5.1.2" class="ltx_text" style="font-size:90%;"> + PLD</span>
</td>
<td id="S5.T3.5.5.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_b ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.5.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">69.0</span></td>
<td id="S5.T3.5.5.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_b ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S5.T3.5.5.3.1" class="ltx_text ltx_font_bold" style="font-size:90%;">88.9</span><span id="S5.T3.5.5.3.2" class="ltx_text" style="font-size:90%;">/86.5</span>
</td>
<td id="S5.T3.5.5.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_b ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.5.4.1" class="ltx_text ltx_font_bold" style="font-size:90%;">89.6/89.1</span></td>
<td id="S5.T3.5.5.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_b ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.5.5.1" class="ltx_text ltx_font_bold" style="font-size:90%;">59.4</span></td>
<td id="S5.T3.5.5.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_b ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.5.6.1" class="ltx_text" style="font-size:90%;">91.8</span></td>
<td id="S5.T3.5.5.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_b ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.5.7.1" class="ltx_text" style="font-size:90%;">88.0</span></td>
<td id="S5.T3.5.5.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_b ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.5.8.1" class="ltx_text ltx_font_bold" style="font-size:90%;">89.4/90.9</span></td>
<td id="S5.T3.5.5.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_b ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.5.9.1" class="ltx_text" style="font-size:90%;">83.1/83.5</span></td>
<td id="S5.T3.5.5.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_border_b ltx_border_r ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S5.T3.5.5.10.1" class="ltx_text ltx_font_bold" style="font-size:90%;">83.2</span></td>
</tr>
</tbody>
</table>
</figure>
</section>
<section id="S5.SS2.SSS0.Px4" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">PLD vs. Shallow network.</h4>

<div id="S5.SS2.SSS0.Px4.p1" class="ltx_para">
<p id="S5.SS2.SSS0.Px4.p1.1" class="ltx_p"><em id="S5.SS2.SSS0.Px4.p1.1.1" class="ltx_emph ltx_font_italic">Shallow BERT + PreLN + Large lr</em> in Table <a href="#S5.T3" title="Table 3 ‣ PLD vs. PreLN. ‣ 5.2 Ablation Studies ‣ 5 Evaluation ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> shows the downstream task accuracy of the 9-layer BERT. Although having the same same number of training computational GFLOPS as ours, the shallow BERT
underperforms PreLN by 0.8 points and is 1.4 points worse than PLD likely because the model capacity has been reduced by the loss of parameters.</p>
</div>
</section>
<section id="S5.SS2.SSS0.Px5" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">PLD vs. Random drop.</h4>

<div id="S5.SS2.SSS0.Px5.p1" class="ltx_para">
<p id="S5.SS2.SSS0.Px5.p1.1" class="ltx_p"><em id="S5.SS2.SSS0.Px5.p1.1.1" class="ltx_emph ltx_font_italic">BERT + PreLN + Large lr + Random</em> drops layers randomly with a fixed ratio (i.e., it has the same compute cost but without any schedule), similar to Stochastic Depth <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib26" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">26</span></a>]</cite>. The GLUE score is 0.9 points better than shallow BERT under the same compute cost and 0.1 point better than PreLN while being 24% faster, indicating the strong regularization effect from stochastic depth. It is 0.5 points worse than PLD, presumably because a fixed ratio does not take into account the training dynamics of Transformer networks.</p>
</div>
</section>
<section id="S5.SS2.SSS0.Px6" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Schedule impact analysis.</h4>

<div id="S5.SS2.SSS0.Px6.p1" class="ltx_para">
<p id="S5.SS2.SSS0.Px6.p1.1" class="ltx_p"><em id="S5.SS2.SSS0.Px6.p1.1.1" class="ltx_emph ltx_font_italic">BERT + PreLN + Large lr + TD only (32-bit*)</em> disables the schedule along the depth dimension (DD) and enables only the schedule along the temporal dimension (TD) in training. Its GLUE score matches "Random", suggesting that the temporal schedule has similar performance as the fixed constant schedule along the time dimension and accuracy gains of PLD is mostly from the depth dimension. However, without the temporal schedule enabled, the model diverges with <em id="S5.SS2.SSS0.Px6.p1.1.2" class="ltx_emph ltx_font_italic">NaN</em> in the middle of half-precision (16-bit) training and has to switch to full-precision (32-bit) training, slowing down training speed. Furthermore, this concept of starting-easy and gradually increasing the difficulty of the learning problem has its roots in curriculum learning and often makes optimization easier. We adopt the temporal schedule since it is robust and helpful for training stability, retaining similar accuracy while reducing training cost considerably.</p>
</div>
</section>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Conclusion</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">Unsupervised language model pre-training is a crucial step for getting state-of-the-art performance on NLP tasks. The current time for training such a model is excruciatingly long, and it is very much desirable to reduce the turnaround time for training such models. In this paper, we study the efficient training algorithms for pre-training BERT model for NLP tasks. We have conducted extensive analysis and found that model architecture is important when training Transformer-based models with stochastic depth. Using this insight, we propose the Switchable-Transformer block and a progressive layer-wise drop schedule. Our experiment results show that our training strategy achieves competitive performance to training a deep model from scratch at a faster rate.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography" style="font-size:90%;">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib1.2.2.1" class="ltx_text" style="font-size:90%;">[1]</span></span>
<span class="ltx_bibblock"><span id="bib.bib1.4.1" class="ltx_text" style="font-size:90%;">
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov,
and Quoc V. Le.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib1.5.1" class="ltx_text" style="font-size:90%;">Xlnet: Generalized autoregressive pretraining for language
understanding.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib1.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib1.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Advances in Neural Information Processing Systems 32: Annual
Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14
December 2019, Vancouver, BC, Canada</span><span id="bib.bib1.8.3" class="ltx_text" style="font-size:90%;">, pages 5754–5764, 2019.
</span>
</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib2.2.2.1" class="ltx_text" style="font-size:90%;">[2]</span></span>
<span class="ltx_bibblock"><span id="bib.bib2.4.1" class="ltx_text" style="font-size:90%;">
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib2.5.1" class="ltx_text" style="font-size:90%;">Roberta: A robustly optimized BERT pretraining approach.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib2.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">CoRR</span><span id="bib.bib2.7.2" class="ltx_text" style="font-size:90%;">, abs/1907.11692, 2019.
</span>
</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib3.2.2.1" class="ltx_text" style="font-size:90%;">[3]</span></span>
<span class="ltx_bibblock"><span id="bib.bib3.4.1" class="ltx_text" style="font-size:90%;">
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib3.5.1" class="ltx_text" style="font-size:90%;">BERT: pre-training of deep bidirectional transformers for language
understanding.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib3.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib3.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, NAACL-HLT 2019)</span><span id="bib.bib3.8.3" class="ltx_text" style="font-size:90%;">, pages 4171–4186, 2019.
</span>
</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib4.2.2.1" class="ltx_text" style="font-size:90%;">[4]</span></span>
<span class="ltx_bibblock"><span id="bib.bib4.4.1" class="ltx_text" style="font-size:90%;">
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib4.5.1" class="ltx_text" style="font-size:90%;">Commonsenseqa: A question answering challenge targeting commonsense
knowledge.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib4.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib4.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume
1 (Long and Short Papers)</span><span id="bib.bib4.8.3" class="ltx_text" style="font-size:90%;">, pages 4149–4158, 2019.
</span>
</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib5.2.2.1" class="ltx_text" style="font-size:90%;">[5]</span></span>
<span class="ltx_bibblock"><span id="bib.bib5.4.1" class="ltx_text" style="font-size:90%;">
Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li,
and Jimmy Lin.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib5.5.1" class="ltx_text" style="font-size:90%;">End-to-end open-domain question answering with bertserini.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib5.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib5.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019,
Demonstrations</span><span id="bib.bib5.8.3" class="ltx_text" style="font-size:90%;">, pages 72–77, 2019.
</span>
</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib6.2.2.1" class="ltx_text" style="font-size:90%;">[6]</span></span>
<span class="ltx_bibblock"><span id="bib.bib6.4.1" class="ltx_text" style="font-size:90%;">
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and
Samuel R. Bowman.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib6.5.1" class="ltx_text" style="font-size:90%;">GLUE: A multi-task benchmark and analysis platform for natural
language understanding.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib6.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib6.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">7th International Conference on Learning Representations</span><span id="bib.bib6.8.3" class="ltx_text" style="font-size:90%;">,
2019.
</span>
</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib7.2.2.1" class="ltx_text" style="font-size:90%;">[7]</span></span>
<span class="ltx_bibblock"><span id="bib.bib7.4.1" class="ltx_text" style="font-size:90%;">
Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib7.5.1" class="ltx_text" style="font-size:90%;">Cognitive graph for multi-hop reading comprehension at scale.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib7.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib7.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Proceedings of the 57th Conference of the Association for
Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2,
2019, Volume 1: Long Papers</span><span id="bib.bib7.8.3" class="ltx_text" style="font-size:90%;">, pages 2694–2703, 2019.
</span>
</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib8.2.2.1" class="ltx_text" style="font-size:90%;">[8]</span></span>
<span class="ltx_bibblock"><span id="bib.bib8.4.1" class="ltx_text" style="font-size:90%;">
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper,
and Bryan Catanzaro.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib8.5.1" class="ltx_text" style="font-size:90%;">Megatron-lm: Training multi-billion parameter language models using
model parallelism.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib8.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">CoRR</span><span id="bib.bib8.7.2" class="ltx_text" style="font-size:90%;">, abs/1909.08053, 2019.
</span>
</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib9.2.2.1" class="ltx_text" style="font-size:90%;">[9]</span></span>
<span class="ltx_bibblock"><span id="bib.bib9.4.1" class="ltx_text" style="font-size:90%;">
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya
Sutskever.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib9.5.1" class="ltx_text" style="font-size:90%;">Language models are unsupervised multitask learners.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib9.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">Online post</span><span id="bib.bib9.7.2" class="ltx_text" style="font-size:90%;">, 2019.
</span>
</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib10.2.2.1" class="ltx_text" style="font-size:90%;">[10]</span></span>
<span class="ltx_bibblock"><span id="bib.bib10.4.1" class="ltx_text" style="font-size:90%;">
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan,
Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom
Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens
Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott
Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec
Radford, Ilya Sutskever, and Dario Amodei.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib10.5.1" class="ltx_text" style="font-size:90%;">Language models are few-shot learners.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib10.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">CoRR</span><span id="bib.bib10.7.2" class="ltx_text" style="font-size:90%;">, abs/2005.14165, 2020.
</span>
</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib11.2.2.1" class="ltx_text" style="font-size:90%;">[11]</span></span>
<span class="ltx_bibblock"><span id="bib.bib11.4.1" class="ltx_text" style="font-size:90%;">
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and
Omer Levy.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib11.5.1" class="ltx_text" style="font-size:90%;">Spanbert: Improving pre-training by representing and predicting
spans.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib11.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">CoRR</span><span id="bib.bib11.7.2" class="ltx_text" style="font-size:90%;">, abs/1907.10529, 2019.
</span>
</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib12.2.2.1" class="ltx_text" style="font-size:90%;">[12]</span></span>
<span class="ltx_bibblock"><span id="bib.bib12.4.1" class="ltx_text" style="font-size:90%;">
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So,
and Jaewoo Kang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib12.5.1" class="ltx_text" style="font-size:90%;">Biobert: a pre-trained biomedical language representation model for
biomedical text mining.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib12.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">Bioinform.</span><span id="bib.bib12.7.2" class="ltx_text" style="font-size:90%;">, 36(4):1234–1240, 2020.
</span>
</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib13.2.2.1" class="ltx_text" style="font-size:90%;">[13]</span></span>
<span class="ltx_bibblock"><span id="bib.bib13.4.1" class="ltx_text" style="font-size:90%;">
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao,
Ming Zhou, and Hsiao-Wuen Hon.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib13.5.1" class="ltx_text" style="font-size:90%;">Unified language model pre-training for natural language
understanding and generation.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib13.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib13.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Advances in Neural Information Processing Systems</span><span id="bib.bib13.8.3" class="ltx_text" style="font-size:90%;">, pages
13042–13054, 2019.
</span>
</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib14.2.2.1" class="ltx_text" style="font-size:90%;">[14]</span></span>
<span class="ltx_bibblock"><span id="bib.bib14.4.1" class="ltx_text" style="font-size:90%;">
Turing-NLG: A 17-billion-parameter language model by Microsoft.
</span>
</span>
<span class="ltx_bibblock"><a target="_blank" href="https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/" title="" class="ltx_ref ltx_url ltx_font_typewriter" style="font-size:90%;">https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/</a><span id="bib.bib14.5.1" class="ltx_text" style="font-size:90%;">.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib14.6.1" class="ltx_text" style="font-size:90%;">Accessed: 19-May-2020.
</span>
</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib15.2.2.1" class="ltx_text" style="font-size:90%;">[15]</span></span>
<span class="ltx_bibblock"><span id="bib.bib15.4.1" class="ltx_text" style="font-size:90%;">
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael
Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib15.5.1" class="ltx_text" style="font-size:90%;">Exploring the limits of transfer learning with a unified text-to-text
transformer.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib15.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">CoRR</span><span id="bib.bib15.7.2" class="ltx_text" style="font-size:90%;">, abs/1910.10683, 2019.
</span>
</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib16.2.2.1" class="ltx_text" style="font-size:90%;">[16]</span></span>
<span class="ltx_bibblock"><span id="bib.bib16.4.1" class="ltx_text" style="font-size:90%;">
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib16.5.1" class="ltx_text" style="font-size:90%;">Attention is all you need.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib16.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib16.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Advances in Neural Information Processing Systems 30: Annual
Conference on Neural Information Processing Systems 2017</span><span id="bib.bib16.8.3" class="ltx_text" style="font-size:90%;">, pages 5998–6008,
2017.
</span>
</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib17.2.2.1" class="ltx_text" style="font-size:90%;">[17]</span></span>
<span class="ltx_bibblock"><span id="bib.bib17.4.1" class="ltx_text" style="font-size:90%;">
Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal,
Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick
Boyle, Pierre-luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike
Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra
Gottipati, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John
Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski,
Alexander Kaplan, Harshit Khaitan, Daniel Killebrew, Andy Koch, Naveen Kumar,
Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu,
Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony,
Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas
Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt
Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew
Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory
Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter,
Walter Wang, Eric Wilcox, and Doe Hyun Yoon.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib17.5.1" class="ltx_text" style="font-size:90%;">In-Datacenter Performance Analysis of a Tensor Processing Unit.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib17.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib17.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Proceedings of the 44th Annual International Symposium on
Computer Architecture</span><span id="bib.bib17.8.3" class="ltx_text" style="font-size:90%;">, ISCA ’17, pages 1–12, 2017.
</span>
</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib18.2.2.1" class="ltx_text" style="font-size:90%;">[18]</span></span>
<span class="ltx_bibblock"><span id="bib.bib18.4.1" class="ltx_text" style="font-size:90%;">
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich
Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii
Kuchaiev, Ganesh Venkatesh, and Hao Wu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib18.5.1" class="ltx_text" style="font-size:90%;">Mixed precision training.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib18.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib18.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">6th International Conference on Learning Representations,
ICLR 2018</span><span id="bib.bib18.8.3" class="ltx_text" style="font-size:90%;">, 2018.
</span>
</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib19.2.2.1" class="ltx_text" style="font-size:90%;">[19]</span></span>
<span class="ltx_bibblock"><span id="bib.bib19.4.1" class="ltx_text" style="font-size:90%;">
Stefano Markidis, Steven Wei Der Chien, Erwin Laure, Ivy Bo Peng, and
Jeffrey S. Vetter.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib19.5.1" class="ltx_text" style="font-size:90%;">NVIDIA tensor core programmability, performance &amp; precision.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib19.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib19.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">2018 IEEE International Parallel and Distributed Processing
Symposium Workshops, IPDPS Workshops 2018, Vancouver, BC, Canada, May
21-25, 2018</span><span id="bib.bib19.8.3" class="ltx_text" style="font-size:90%;">, pages 522–531, 2018.
</span>
</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib20.2.2.1" class="ltx_text" style="font-size:90%;">[20]</span></span>
<span class="ltx_bibblock"><span id="bib.bib20.4.1" class="ltx_text" style="font-size:90%;">
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu
Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib20.5.1" class="ltx_text" style="font-size:90%;">Gpipe: Efficient training of giant neural networks using pipeline
parallelism.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib20.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib20.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Advances in Neural Information Processing Systems 32: Annual
Conference on Neural Information Processing Systems 2019</span><span id="bib.bib20.8.3" class="ltx_text" style="font-size:90%;">, pages 103–112,
2019.
</span>
</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib21.2.2.1" class="ltx_text" style="font-size:90%;">[21]</span></span>
<span class="ltx_bibblock"><span id="bib.bib21.4.1" class="ltx_text" style="font-size:90%;">
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn
Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young,
Ryan Sepassi, and Blake A. Hechtman.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib21.5.1" class="ltx_text" style="font-size:90%;">Mesh-tensorflow: Deep learning for supercomputers.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib21.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib21.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Advances in Neural Information Processing Systems 31: Annual
Conference on Neural Information Processing Systems 2018</span><span id="bib.bib21.8.3" class="ltx_text" style="font-size:90%;">, pages
10435–10444, 2018.
</span>
</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib22.2.2.1" class="ltx_text" style="font-size:90%;">[22]</span></span>
<span class="ltx_bibblock"><span id="bib.bib22.4.1" class="ltx_text" style="font-size:90%;">
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy,
and Ping Tak Peter Tang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib22.5.1" class="ltx_text" style="font-size:90%;">On large-batch training for deep learning: Generalization gap and
sharp minima.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib22.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib22.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">5th International Conference on Learning Representations,
ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track
Proceedings</span><span id="bib.bib22.8.3" class="ltx_text" style="font-size:90%;">, 2017.
</span>
</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib23.2.2.1" class="ltx_text" style="font-size:90%;">[23]</span></span>
<span class="ltx_bibblock"><span id="bib.bib23.4.1" class="ltx_text" style="font-size:90%;">
Yang You, Jing Li, Jonathan Hseu, Xiaodan Song, James Demmel, and Cho-Jui
Hsieh.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib23.5.1" class="ltx_text" style="font-size:90%;">Reducing BERT pre-training time from 3 days to 76 minutes.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib23.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">CoRR</span><span id="bib.bib23.7.2" class="ltx_text" style="font-size:90%;">, abs/1904.00962, 2019.
</span>
</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib24.2.2.1" class="ltx_text" style="font-size:90%;">[24]</span></span>
<span class="ltx_bibblock"><span id="bib.bib24.4.1" class="ltx_text" style="font-size:90%;">
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib24.5.1" class="ltx_text" style="font-size:90%;">Distilbert, a distilled version of BERT: smaller, faster, cheaper
and lighter.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib24.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">CoRR</span><span id="bib.bib24.7.2" class="ltx_text" style="font-size:90%;">, abs/1910.01108, 2019.
</span>
</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib25.2.2.1" class="ltx_text" style="font-size:90%;">[25]</span></span>
<span class="ltx_bibblock"><span id="bib.bib25.4.1" class="ltx_text" style="font-size:90%;">
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang
Wang, and Qun Liu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib25.5.1" class="ltx_text" style="font-size:90%;">Tinybert: Distilling BERT for natural language understanding.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib25.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">CoRR</span><span id="bib.bib25.7.2" class="ltx_text" style="font-size:90%;">, abs/1909.10351, 2019.
</span>
</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib26.2.2.1" class="ltx_text" style="font-size:90%;">[26]</span></span>
<span class="ltx_bibblock"><span id="bib.bib26.4.1" class="ltx_text" style="font-size:90%;">
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib26.5.1" class="ltx_text" style="font-size:90%;">Deep networks with stochastic depth.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib26.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib26.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Computer Vision - ECCV 2016 - 14th European Conference,
Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV</span><span id="bib.bib26.8.3" class="ltx_text" style="font-size:90%;">,
pages 646–661, 2016.
</span>
</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib27.2.2.1" class="ltx_text" style="font-size:90%;">[27]</span></span>
<span class="ltx_bibblock"><span id="bib.bib27.4.1" class="ltx_text" style="font-size:90%;">
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and
Radu Soricut.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib27.5.1" class="ltx_text" style="font-size:90%;">ALBERT: A lite BERT for self-supervised learning of language
representations.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib27.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib27.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">8th International Conference on Learning Representations,
ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020</span><span id="bib.bib27.8.3" class="ltx_text" style="font-size:90%;">, 2020.
</span>
</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib28.2.2.1" class="ltx_text" style="font-size:90%;">[28]</span></span>
<span class="ltx_bibblock"><span id="bib.bib28.4.1" class="ltx_text" style="font-size:90%;">
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib28.5.1" class="ltx_text" style="font-size:90%;">Deep residual learning for image recognition.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib28.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib28.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">2016 IEEE Conference on Computer Vision and Pattern
Recognition</span><span id="bib.bib28.8.3" class="ltx_text" style="font-size:90%;">, pages 770–778, 2016.
</span>
</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib29.2.2.1" class="ltx_text" style="font-size:90%;">[29]</span></span>
<span class="ltx_bibblock"><span id="bib.bib29.4.1" class="ltx_text" style="font-size:90%;">
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib29.5.1" class="ltx_text" style="font-size:90%;">Layer normalization.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib29.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">CoRR</span><span id="bib.bib29.7.2" class="ltx_text" style="font-size:90%;">, abs/1607.06450, 2016.
</span>
</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib30.2.2.1" class="ltx_text" style="font-size:90%;">[30]</span></span>
<span class="ltx_bibblock"><span id="bib.bib30.4.1" class="ltx_text" style="font-size:90%;">
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib30.5.1" class="ltx_text" style="font-size:90%;">ELECTRA: pre-training text encoders as discriminators rather than
generators.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib30.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib30.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">8th International Conference on Learning Representations,
ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020</span><span id="bib.bib30.8.3" class="ltx_text" style="font-size:90%;">. OpenReview.net, 2020.
</span>
</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib31.2.2.1" class="ltx_text" style="font-size:90%;">[31]</span></span>
<span class="ltx_bibblock"><span id="bib.bib31.4.1" class="ltx_text" style="font-size:90%;">
Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tie-Yan Liu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib31.5.1" class="ltx_text" style="font-size:90%;">Efficient training of BERT by progressively stacking.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib31.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib31.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Proceedings of the 36th International Conference on Machine
Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA</span><span id="bib.bib31.8.3" class="ltx_text" style="font-size:90%;">, pages
2337–2346, 2019.
</span>
</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib32.2.2.1" class="ltx_text" style="font-size:90%;">[32]</span></span>
<span class="ltx_bibblock"><span id="bib.bib32.4.1" class="ltx_text" style="font-size:90%;">
Angela Fan, Edouard Grave, and Armand Joulin.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib32.5.1" class="ltx_text" style="font-size:90%;">Reducing transformer depth on demand with structured dropout.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib32.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib32.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">8th International Conference on Learning Representations,
ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020</span><span id="bib.bib32.8.3" class="ltx_text" style="font-size:90%;">. OpenReview.net, 2020.
</span>
</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib33.2.2.1" class="ltx_text" style="font-size:90%;">[33]</span></span>
<span class="ltx_bibblock"><span id="bib.bib33.4.1" class="ltx_text" style="font-size:90%;">
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and
Lidia S. Chao.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib33.5.1" class="ltx_text" style="font-size:90%;">Learning deep transformer models for machine translation.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib33.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib33.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Proceedings of the 57th Conference of the Association for
Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2,
2019, Volume 1: Long Papers</span><span id="bib.bib33.8.3" class="ltx_text" style="font-size:90%;">, pages 1810–1822, 2019.
</span>
</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib34.2.2.1" class="ltx_text" style="font-size:90%;">[34]</span></span>
<span class="ltx_bibblock"><span id="bib.bib34.4.1" class="ltx_text" style="font-size:90%;">
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing,
Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib34.5.1" class="ltx_text" style="font-size:90%;">On layer normalization in the transformer architecture.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib34.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2002.04745</span><span id="bib.bib34.7.2" class="ltx_text" style="font-size:90%;">, 2020.
</span>
</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib35.2.2.1" class="ltx_text" style="font-size:90%;">[35]</span></span>
<span class="ltx_bibblock"><span id="bib.bib35.4.1" class="ltx_text" style="font-size:90%;">
Alexei Baevski and Michael Auli.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib35.5.1" class="ltx_text" style="font-size:90%;">Adaptive input representations for neural language modeling.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib35.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib35.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">7th International Conference on Learning Representations</span><span id="bib.bib35.8.3" class="ltx_text" style="font-size:90%;">,
2019.
</span>
</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib36.2.2.1" class="ltx_text" style="font-size:90%;">[36]</span></span>
<span class="ltx_bibblock"><span id="bib.bib36.4.1" class="ltx_text" style="font-size:90%;">
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib36.5.1" class="ltx_text" style="font-size:90%;">Generating long sequences with sparse transformers.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib36.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">CoRR</span><span id="bib.bib36.7.2" class="ltx_text" style="font-size:90%;">, abs/1904.10509, 2019.
</span>
</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib37.2.2.1" class="ltx_text" style="font-size:90%;">[37]</span></span>
<span class="ltx_bibblock"><span id="bib.bib37.4.1" class="ltx_text" style="font-size:90%;">
Toan Q. Nguyen and Julian Salazar.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib37.5.1" class="ltx_text" style="font-size:90%;">Transformers without tears: Improving the normalization of
self-attention.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib37.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">CoRR</span><span id="bib.bib37.7.2" class="ltx_text" style="font-size:90%;">, abs/1910.05895, 2019.
</span>
</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib38.2.2.1" class="ltx_text" style="font-size:90%;">[38]</span></span>
<span class="ltx_bibblock"><span id="bib.bib38.4.1" class="ltx_text" style="font-size:90%;">
Xavier Glorot and Yoshua Bengio.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib38.5.1" class="ltx_text" style="font-size:90%;">Understanding the difficulty of training deep feedforward neural
networks.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib38.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib38.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Proceedings of the Thirteenth International Conference on
Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort,
Sardinia, Italy, May 13-15, 2010</span><span id="bib.bib38.8.3" class="ltx_text" style="font-size:90%;">, pages 249–256, 2010.
</span>
</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib39.2.2.1" class="ltx_text" style="font-size:90%;">[39]</span></span>
<span class="ltx_bibblock"><span id="bib.bib39.4.1" class="ltx_text" style="font-size:90%;">
Alireza Zaeemzadeh, Nazanin Rahnavard, and Mubarak Shah.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib39.5.1" class="ltx_text" style="font-size:90%;">Norm-preservation: Why residual networks can become extremely deep?
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib39.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">CoRR</span><span id="bib.bib39.7.2" class="ltx_text" style="font-size:90%;">, abs/1805.07477, 2018.
</span>
</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib40.2.2.1" class="ltx_text" style="font-size:90%;">[40]</span></span>
<span class="ltx_bibblock"><span id="bib.bib40.4.1" class="ltx_text" style="font-size:90%;">
Klaus Greff, Rupesh Kumar Srivastava, and Jürgen Schmidhuber.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib40.5.1" class="ltx_text" style="font-size:90%;">Highway and residual networks learn unrolled iterative estimation.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib40.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib40.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">5th International Conference on Learning Representations,
ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track
Proceedings</span><span id="bib.bib40.8.3" class="ltx_text" style="font-size:90%;">, 2017.
</span>
</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib41.2.2.1" class="ltx_text" style="font-size:90%;">[41]</span></span>
<span class="ltx_bibblock"><span id="bib.bib41.4.1" class="ltx_text" style="font-size:90%;">
Pietro Morerio, Jacopo Cavazza, Riccardo Volpi, René Vidal, and Vittorio
Murino.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib41.5.1" class="ltx_text" style="font-size:90%;">Curriculum dropout.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib41.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib41.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">IEEE International Conference on Computer Vision, ICCV
2017, Venice, Italy, October 22-29, 2017</span><span id="bib.bib41.8.3" class="ltx_text" style="font-size:90%;">, pages 3564–3572, 2017.
</span>
</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib42.2.2.1" class="ltx_text" style="font-size:90%;">[42]</span></span>
<span class="ltx_bibblock"><span id="bib.bib42.4.1" class="ltx_text" style="font-size:90%;">
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib42.5.1" class="ltx_text" style="font-size:90%;">Curriculum learning.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib42.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib42.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Proceedings of the 26th Annual International Conference on
Machine Learning, ICML 2009, Montreal, Quebec, Canada, June 14-18, 2009</span><span id="bib.bib42.8.3" class="ltx_text" style="font-size:90%;">,
pages 41–48, 2009.
</span>
</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib43.2.2.1" class="ltx_text" style="font-size:90%;">[43]</span></span>
<span class="ltx_bibblock"><span id="bib.bib43.4.1" class="ltx_text" style="font-size:90%;">
PyTorch Distributed Data Parallel.
</span>
</span>
<span class="ltx_bibblock"><a target="_blank" href="https://pytorch.org/docs/stable/notes/ddp.html" title="" class="ltx_ref ltx_url ltx_font_typewriter" style="font-size:90%;">https://pytorch.org/docs/stable/notes/ddp.html</a><span id="bib.bib43.5.1" class="ltx_text" style="font-size:90%;">.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib43.6.1" class="ltx_text" style="font-size:90%;">Accessed: 28-April-2020.
</span>
</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib44.2.2.1" class="ltx_text" style="font-size:90%;">[44]</span></span>
<span class="ltx_bibblock"><span id="bib.bib44.4.1" class="ltx_text" style="font-size:90%;">
Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib44.5.1" class="ltx_text" style="font-size:90%;">Understanding the difficulty of training transformers.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib44.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">CoRR</span><span id="bib.bib44.7.2" class="ltx_text" style="font-size:90%;">, abs/2004.08249, 2020.
</span>
</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib45.2.2.1" class="ltx_text" style="font-size:90%;">[45]</span></span>
<span class="ltx_bibblock"><span id="bib.bib45.4.1" class="ltx_text" style="font-size:90%;">
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib45.5.1" class="ltx_text" style="font-size:90%;">Identity mappings in deep residual networks.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib45.6.1" class="ltx_text" style="font-size:90%;">In </span><span id="bib.bib45.7.2" class="ltx_text ltx_font_italic" style="font-size:90%;">Computer Vision - ECCV 2016 - 14th European Conference</span><span id="bib.bib45.8.3" class="ltx_text" style="font-size:90%;">,
pages 630–645, 2016.
</span>
</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib46.2.2.1" class="ltx_text" style="font-size:90%;">[46]</span></span>
<span class="ltx_bibblock"><span id="bib.bib46.4.1" class="ltx_text" style="font-size:90%;">
Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, and
Jeffrey Mark Siskind.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib46.5.1" class="ltx_text" style="font-size:90%;">Automatic differentiation in machine learning: a survey.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib46.6.1" class="ltx_text ltx_font_italic" style="font-size:90%;">J. Mach. Learn. Res.</span><span id="bib.bib46.7.2" class="ltx_text" style="font-size:90%;">, 18:153:1–153:43, 2017.
</span>
</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Pre-training Hyperparameters</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.1" class="ltx_p">Table <a href="#A1.T4" title="Table 4 ‣ Appendix A Pre-training Hyperparameters ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> describes the hyperparameters for pre-training the baseline and PLD.</p>
</div>
<figure id="A1.T4" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>Hyperparameters for pre-training the baseline and PLD.</figcaption>
<table id="A1.T4.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A1.T4.1.1.1" class="ltx_tr">
<th id="A1.T4.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t"><span id="A1.T4.1.1.1.1.1" class="ltx_text ltx_font_bold">Hyperparameter</span></th>
<td id="A1.T4.1.1.1.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t"><span id="A1.T4.1.1.1.2.1" class="ltx_text ltx_font_bold">Baseline</span></td>
<td id="A1.T4.1.1.1.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t"><span id="A1.T4.1.1.1.3.1" class="ltx_text ltx_font_bold">PLD</span></td>
</tr>
<tr id="A1.T4.1.2.2" class="ltx_tr">
<th id="A1.T4.1.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t">Number of Layers</th>
<td id="A1.T4.1.2.2.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">12</td>
<td id="A1.T4.1.2.2.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">12</td>
</tr>
<tr id="A1.T4.1.3.3" class="ltx_tr">
<th id="A1.T4.1.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t">Hidden zies</th>
<td id="A1.T4.1.3.3.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">768</td>
<td id="A1.T4.1.3.3.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">768</td>
</tr>
<tr id="A1.T4.1.4.4" class="ltx_tr">
<th id="A1.T4.1.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t">Attention heads</th>
<td id="A1.T4.1.4.4.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">12</td>
<td id="A1.T4.1.4.4.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">12</td>
</tr>
<tr id="A1.T4.1.5.5" class="ltx_tr">
<th id="A1.T4.1.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t">Dropout</th>
<td id="A1.T4.1.5.5.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">0.1</td>
<td id="A1.T4.1.5.5.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">0.1</td>
</tr>
<tr id="A1.T4.1.6.6" class="ltx_tr">
<th id="A1.T4.1.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t">Attention dropout</th>
<td id="A1.T4.1.6.6.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">0.1</td>
<td id="A1.T4.1.6.6.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">0.1</td>
</tr>
<tr id="A1.T4.1.7.7" class="ltx_tr">
<th id="A1.T4.1.7.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t">Total batch size</th>
<td id="A1.T4.1.7.7.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">4K</td>
<td id="A1.T4.1.7.7.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">4K</td>
</tr>
<tr id="A1.T4.1.8.8" class="ltx_tr">
<th id="A1.T4.1.8.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t">Train micro batch size per gpu</th>
<td id="A1.T4.1.8.8.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">16</td>
<td id="A1.T4.1.8.8.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">16</td>
</tr>
<tr id="A1.T4.1.9.9" class="ltx_tr">
<th id="A1.T4.1.9.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t">Optimizer</th>
<td id="A1.T4.1.9.9.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Adam</td>
<td id="A1.T4.1.9.9.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Adam</td>
</tr>
<tr id="A1.T4.1.10.10" class="ltx_tr">
<th id="A1.T4.1.10.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t">Peak learning rate</th>
<td id="A1.T4.1.10.10.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">1e-04</td>
<td id="A1.T4.1.10.10.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">1e-03</td>
</tr>
<tr id="A1.T4.1.11.11" class="ltx_tr">
<th id="A1.T4.1.11.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t">Learning rate scheduler</th>
<td id="A1.T4.1.11.11.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">warmup_linear_decay_exp</td>
<td id="A1.T4.1.11.11.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">warmup_linear_decay_exp</td>
</tr>
<tr id="A1.T4.1.12.12" class="ltx_tr">
<th id="A1.T4.1.12.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t">Warmup ratio</th>
<td id="A1.T4.1.12.12.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">0.02</td>
<td id="A1.T4.1.12.12.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">0.02</td>
</tr>
<tr id="A1.T4.1.13.13" class="ltx_tr">
<th id="A1.T4.1.13.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t">Decay rate</th>
<td id="A1.T4.1.13.13.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">0.99</td>
<td id="A1.T4.1.13.13.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">0.99</td>
</tr>
<tr id="A1.T4.1.14.14" class="ltx_tr">
<th id="A1.T4.1.14.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t">Decay step</th>
<td id="A1.T4.1.14.14.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">1000</td>
<td id="A1.T4.1.14.14.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">1000</td>
</tr>
<tr id="A1.T4.1.15.15" class="ltx_tr">
<th id="A1.T4.1.15.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t">Max Training steps</th>
<td id="A1.T4.1.15.15.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">200000</td>
<td id="A1.T4.1.15.15.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">200000</td>
</tr>
<tr id="A1.T4.1.16.16" class="ltx_tr">
<th id="A1.T4.1.16.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t">Weight decay</th>
<td id="A1.T4.1.16.16.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">0.01</td>
<td id="A1.T4.1.16.16.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">0.01</td>
</tr>
<tr id="A1.T4.1.17.17" class="ltx_tr">
<th id="A1.T4.1.17.17.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_b ltx_border_l ltx_border_r ltx_border_t">Gradient clipping</th>
<td id="A1.T4.1.17.17.2" class="ltx_td ltx_align_left ltx_border_b ltx_border_r ltx_border_t">1</td>
<td id="A1.T4.1.17.17.3" class="ltx_td ltx_align_left ltx_border_b ltx_border_r ltx_border_t">1</td>
</tr>
</tbody>
</table>
</figure>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Establishing Identity Mapping with PreLN</h2>

<div id="A2.p1" class="ltx_para">
<p id="A2.p1.6" class="ltx_p">Prior studies <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib28" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">28</span></a>, <a href="#bib.bib45" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">45</span></a>]</cite> suggest that establishing <em id="A2.p1.6.1" class="ltx_emph ltx_font_italic">identity mapping</em> to keep a <em id="A2.p1.6.2" class="ltx_emph ltx_font_italic">clean</em> information path (no operations except addition) is helpful for easing optimization of networks with residual connections. With the change of PreLN, we can express the output of the i-th Transformer layer as the input <math id="A2.p1.1.m1.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="A2.p1.1.m1.1a"><msub id="A2.p1.1.m1.1.1" xref="A2.p1.1.m1.1.1.cmml"><mi id="A2.p1.1.m1.1.1.2" xref="A2.p1.1.m1.1.1.2.cmml">x</mi><mi id="A2.p1.1.m1.1.1.3" xref="A2.p1.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A2.p1.1.m1.1b"><apply id="A2.p1.1.m1.1.1.cmml" xref="A2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A2.p1.1.m1.1.1.1.cmml" xref="A2.p1.1.m1.1.1">subscript</csymbol><ci id="A2.p1.1.m1.1.1.2.cmml" xref="A2.p1.1.m1.1.1.2">𝑥</ci><ci id="A2.p1.1.m1.1.1.3.cmml" xref="A2.p1.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.1.m1.1c">x_{i}</annotation></semantics></math> of that layer plus a residual transformation function <math id="A2.p1.2.m2.2" class="ltx_Math" alttext="f_{RT}=f_{S-ATTN}(f_{LN}(x_{i}))+f_{FFN}(f_{LN}(x_{i}^{{}^{\prime}}))" display="inline"><semantics id="A2.p1.2.m2.2a"><mrow id="A2.p1.2.m2.2.2" xref="A2.p1.2.m2.2.2.cmml"><msub id="A2.p1.2.m2.2.2.4" xref="A2.p1.2.m2.2.2.4.cmml"><mi id="A2.p1.2.m2.2.2.4.2" xref="A2.p1.2.m2.2.2.4.2.cmml">f</mi><mrow id="A2.p1.2.m2.2.2.4.3" xref="A2.p1.2.m2.2.2.4.3.cmml"><mi id="A2.p1.2.m2.2.2.4.3.2" xref="A2.p1.2.m2.2.2.4.3.2.cmml">R</mi><mo lspace="0em" rspace="0em" id="A2.p1.2.m2.2.2.4.3.1" xref="A2.p1.2.m2.2.2.4.3.1.cmml">​</mo><mi id="A2.p1.2.m2.2.2.4.3.3" xref="A2.p1.2.m2.2.2.4.3.3.cmml">T</mi></mrow></msub><mo id="A2.p1.2.m2.2.2.3" xref="A2.p1.2.m2.2.2.3.cmml">=</mo><mrow id="A2.p1.2.m2.2.2.2" xref="A2.p1.2.m2.2.2.2.cmml"><mrow id="A2.p1.2.m2.1.1.1.1" xref="A2.p1.2.m2.1.1.1.1.cmml"><msub id="A2.p1.2.m2.1.1.1.1.3" xref="A2.p1.2.m2.1.1.1.1.3.cmml"><mi id="A2.p1.2.m2.1.1.1.1.3.2" xref="A2.p1.2.m2.1.1.1.1.3.2.cmml">f</mi><mrow id="A2.p1.2.m2.1.1.1.1.3.3" xref="A2.p1.2.m2.1.1.1.1.3.3.cmml"><mi id="A2.p1.2.m2.1.1.1.1.3.3.2" xref="A2.p1.2.m2.1.1.1.1.3.3.2.cmml">S</mi><mo id="A2.p1.2.m2.1.1.1.1.3.3.1" xref="A2.p1.2.m2.1.1.1.1.3.3.1.cmml">−</mo><mrow id="A2.p1.2.m2.1.1.1.1.3.3.3" xref="A2.p1.2.m2.1.1.1.1.3.3.3.cmml"><mi id="A2.p1.2.m2.1.1.1.1.3.3.3.2" xref="A2.p1.2.m2.1.1.1.1.3.3.3.2.cmml">A</mi><mo lspace="0em" rspace="0em" id="A2.p1.2.m2.1.1.1.1.3.3.3.1" xref="A2.p1.2.m2.1.1.1.1.3.3.3.1.cmml">​</mo><mi id="A2.p1.2.m2.1.1.1.1.3.3.3.3" xref="A2.p1.2.m2.1.1.1.1.3.3.3.3.cmml">T</mi><mo lspace="0em" rspace="0em" id="A2.p1.2.m2.1.1.1.1.3.3.3.1a" xref="A2.p1.2.m2.1.1.1.1.3.3.3.1.cmml">​</mo><mi id="A2.p1.2.m2.1.1.1.1.3.3.3.4" xref="A2.p1.2.m2.1.1.1.1.3.3.3.4.cmml">T</mi><mo lspace="0em" rspace="0em" id="A2.p1.2.m2.1.1.1.1.3.3.3.1b" xref="A2.p1.2.m2.1.1.1.1.3.3.3.1.cmml">​</mo><mi id="A2.p1.2.m2.1.1.1.1.3.3.3.5" xref="A2.p1.2.m2.1.1.1.1.3.3.3.5.cmml">N</mi></mrow></mrow></msub><mo lspace="0em" rspace="0em" id="A2.p1.2.m2.1.1.1.1.2" xref="A2.p1.2.m2.1.1.1.1.2.cmml">​</mo><mrow id="A2.p1.2.m2.1.1.1.1.1.1" xref="A2.p1.2.m2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A2.p1.2.m2.1.1.1.1.1.1.2" xref="A2.p1.2.m2.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A2.p1.2.m2.1.1.1.1.1.1.1" xref="A2.p1.2.m2.1.1.1.1.1.1.1.cmml"><msub id="A2.p1.2.m2.1.1.1.1.1.1.1.3" xref="A2.p1.2.m2.1.1.1.1.1.1.1.3.cmml"><mi id="A2.p1.2.m2.1.1.1.1.1.1.1.3.2" xref="A2.p1.2.m2.1.1.1.1.1.1.1.3.2.cmml">f</mi><mrow id="A2.p1.2.m2.1.1.1.1.1.1.1.3.3" xref="A2.p1.2.m2.1.1.1.1.1.1.1.3.3.cmml"><mi id="A2.p1.2.m2.1.1.1.1.1.1.1.3.3.2" xref="A2.p1.2.m2.1.1.1.1.1.1.1.3.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="A2.p1.2.m2.1.1.1.1.1.1.1.3.3.1" xref="A2.p1.2.m2.1.1.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="A2.p1.2.m2.1.1.1.1.1.1.1.3.3.3" xref="A2.p1.2.m2.1.1.1.1.1.1.1.3.3.3.cmml">N</mi></mrow></msub><mo lspace="0em" rspace="0em" id="A2.p1.2.m2.1.1.1.1.1.1.1.2" xref="A2.p1.2.m2.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="A2.p1.2.m2.1.1.1.1.1.1.1.1.1" xref="A2.p1.2.m2.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A2.p1.2.m2.1.1.1.1.1.1.1.1.1.2" xref="A2.p1.2.m2.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="A2.p1.2.m2.1.1.1.1.1.1.1.1.1.1" xref="A2.p1.2.m2.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="A2.p1.2.m2.1.1.1.1.1.1.1.1.1.1.2" xref="A2.p1.2.m2.1.1.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="A2.p1.2.m2.1.1.1.1.1.1.1.1.1.1.3" xref="A2.p1.2.m2.1.1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A2.p1.2.m2.1.1.1.1.1.1.1.1.1.3" xref="A2.p1.2.m2.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="A2.p1.2.m2.1.1.1.1.1.1.3" xref="A2.p1.2.m2.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="A2.p1.2.m2.2.2.2.3" xref="A2.p1.2.m2.2.2.2.3.cmml">+</mo><mrow id="A2.p1.2.m2.2.2.2.2" xref="A2.p1.2.m2.2.2.2.2.cmml"><msub id="A2.p1.2.m2.2.2.2.2.3" xref="A2.p1.2.m2.2.2.2.2.3.cmml"><mi id="A2.p1.2.m2.2.2.2.2.3.2" xref="A2.p1.2.m2.2.2.2.2.3.2.cmml">f</mi><mrow id="A2.p1.2.m2.2.2.2.2.3.3" xref="A2.p1.2.m2.2.2.2.2.3.3.cmml"><mi id="A2.p1.2.m2.2.2.2.2.3.3.2" xref="A2.p1.2.m2.2.2.2.2.3.3.2.cmml">F</mi><mo lspace="0em" rspace="0em" id="A2.p1.2.m2.2.2.2.2.3.3.1" xref="A2.p1.2.m2.2.2.2.2.3.3.1.cmml">​</mo><mi id="A2.p1.2.m2.2.2.2.2.3.3.3" xref="A2.p1.2.m2.2.2.2.2.3.3.3.cmml">F</mi><mo lspace="0em" rspace="0em" id="A2.p1.2.m2.2.2.2.2.3.3.1a" xref="A2.p1.2.m2.2.2.2.2.3.3.1.cmml">​</mo><mi id="A2.p1.2.m2.2.2.2.2.3.3.4" xref="A2.p1.2.m2.2.2.2.2.3.3.4.cmml">N</mi></mrow></msub><mo lspace="0em" rspace="0em" id="A2.p1.2.m2.2.2.2.2.2" xref="A2.p1.2.m2.2.2.2.2.2.cmml">​</mo><mrow id="A2.p1.2.m2.2.2.2.2.1.1" xref="A2.p1.2.m2.2.2.2.2.1.1.1.cmml"><mo stretchy="false" id="A2.p1.2.m2.2.2.2.2.1.1.2" xref="A2.p1.2.m2.2.2.2.2.1.1.1.cmml">(</mo><mrow id="A2.p1.2.m2.2.2.2.2.1.1.1" xref="A2.p1.2.m2.2.2.2.2.1.1.1.cmml"><msub id="A2.p1.2.m2.2.2.2.2.1.1.1.3" xref="A2.p1.2.m2.2.2.2.2.1.1.1.3.cmml"><mi id="A2.p1.2.m2.2.2.2.2.1.1.1.3.2" xref="A2.p1.2.m2.2.2.2.2.1.1.1.3.2.cmml">f</mi><mrow id="A2.p1.2.m2.2.2.2.2.1.1.1.3.3" xref="A2.p1.2.m2.2.2.2.2.1.1.1.3.3.cmml"><mi id="A2.p1.2.m2.2.2.2.2.1.1.1.3.3.2" xref="A2.p1.2.m2.2.2.2.2.1.1.1.3.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="A2.p1.2.m2.2.2.2.2.1.1.1.3.3.1" xref="A2.p1.2.m2.2.2.2.2.1.1.1.3.3.1.cmml">​</mo><mi id="A2.p1.2.m2.2.2.2.2.1.1.1.3.3.3" xref="A2.p1.2.m2.2.2.2.2.1.1.1.3.3.3.cmml">N</mi></mrow></msub><mo lspace="0em" rspace="0em" id="A2.p1.2.m2.2.2.2.2.1.1.1.2" xref="A2.p1.2.m2.2.2.2.2.1.1.1.2.cmml">​</mo><mrow id="A2.p1.2.m2.2.2.2.2.1.1.1.1.1" xref="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.2" xref="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.cmml">(</mo><msubsup id="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1" xref="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.cmml"><mi id="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.2.2" xref="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.2.3" xref="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.2.3.cmml">i</mi><msup id="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.3" xref="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.3.cmml"><mi id="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.3a" xref="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.3.cmml"></mi><mo id="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.3.1" xref="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.3.1.cmml">′</mo></msup></msubsup><mo stretchy="false" id="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.3" xref="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="A2.p1.2.m2.2.2.2.2.1.1.3" xref="A2.p1.2.m2.2.2.2.2.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.p1.2.m2.2b"><apply id="A2.p1.2.m2.2.2.cmml" xref="A2.p1.2.m2.2.2"><eq id="A2.p1.2.m2.2.2.3.cmml" xref="A2.p1.2.m2.2.2.3"></eq><apply id="A2.p1.2.m2.2.2.4.cmml" xref="A2.p1.2.m2.2.2.4"><csymbol cd="ambiguous" id="A2.p1.2.m2.2.2.4.1.cmml" xref="A2.p1.2.m2.2.2.4">subscript</csymbol><ci id="A2.p1.2.m2.2.2.4.2.cmml" xref="A2.p1.2.m2.2.2.4.2">𝑓</ci><apply id="A2.p1.2.m2.2.2.4.3.cmml" xref="A2.p1.2.m2.2.2.4.3"><times id="A2.p1.2.m2.2.2.4.3.1.cmml" xref="A2.p1.2.m2.2.2.4.3.1"></times><ci id="A2.p1.2.m2.2.2.4.3.2.cmml" xref="A2.p1.2.m2.2.2.4.3.2">𝑅</ci><ci id="A2.p1.2.m2.2.2.4.3.3.cmml" xref="A2.p1.2.m2.2.2.4.3.3">𝑇</ci></apply></apply><apply id="A2.p1.2.m2.2.2.2.cmml" xref="A2.p1.2.m2.2.2.2"><plus id="A2.p1.2.m2.2.2.2.3.cmml" xref="A2.p1.2.m2.2.2.2.3"></plus><apply id="A2.p1.2.m2.1.1.1.1.cmml" xref="A2.p1.2.m2.1.1.1.1"><times id="A2.p1.2.m2.1.1.1.1.2.cmml" xref="A2.p1.2.m2.1.1.1.1.2"></times><apply id="A2.p1.2.m2.1.1.1.1.3.cmml" xref="A2.p1.2.m2.1.1.1.1.3"><csymbol cd="ambiguous" id="A2.p1.2.m2.1.1.1.1.3.1.cmml" xref="A2.p1.2.m2.1.1.1.1.3">subscript</csymbol><ci id="A2.p1.2.m2.1.1.1.1.3.2.cmml" xref="A2.p1.2.m2.1.1.1.1.3.2">𝑓</ci><apply id="A2.p1.2.m2.1.1.1.1.3.3.cmml" xref="A2.p1.2.m2.1.1.1.1.3.3"><minus id="A2.p1.2.m2.1.1.1.1.3.3.1.cmml" xref="A2.p1.2.m2.1.1.1.1.3.3.1"></minus><ci id="A2.p1.2.m2.1.1.1.1.3.3.2.cmml" xref="A2.p1.2.m2.1.1.1.1.3.3.2">𝑆</ci><apply id="A2.p1.2.m2.1.1.1.1.3.3.3.cmml" xref="A2.p1.2.m2.1.1.1.1.3.3.3"><times id="A2.p1.2.m2.1.1.1.1.3.3.3.1.cmml" xref="A2.p1.2.m2.1.1.1.1.3.3.3.1"></times><ci id="A2.p1.2.m2.1.1.1.1.3.3.3.2.cmml" xref="A2.p1.2.m2.1.1.1.1.3.3.3.2">𝐴</ci><ci id="A2.p1.2.m2.1.1.1.1.3.3.3.3.cmml" xref="A2.p1.2.m2.1.1.1.1.3.3.3.3">𝑇</ci><ci id="A2.p1.2.m2.1.1.1.1.3.3.3.4.cmml" xref="A2.p1.2.m2.1.1.1.1.3.3.3.4">𝑇</ci><ci id="A2.p1.2.m2.1.1.1.1.3.3.3.5.cmml" xref="A2.p1.2.m2.1.1.1.1.3.3.3.5">𝑁</ci></apply></apply></apply><apply id="A2.p1.2.m2.1.1.1.1.1.1.1.cmml" xref="A2.p1.2.m2.1.1.1.1.1.1"><times id="A2.p1.2.m2.1.1.1.1.1.1.1.2.cmml" xref="A2.p1.2.m2.1.1.1.1.1.1.1.2"></times><apply id="A2.p1.2.m2.1.1.1.1.1.1.1.3.cmml" xref="A2.p1.2.m2.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A2.p1.2.m2.1.1.1.1.1.1.1.3.1.cmml" xref="A2.p1.2.m2.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="A2.p1.2.m2.1.1.1.1.1.1.1.3.2.cmml" xref="A2.p1.2.m2.1.1.1.1.1.1.1.3.2">𝑓</ci><apply id="A2.p1.2.m2.1.1.1.1.1.1.1.3.3.cmml" xref="A2.p1.2.m2.1.1.1.1.1.1.1.3.3"><times id="A2.p1.2.m2.1.1.1.1.1.1.1.3.3.1.cmml" xref="A2.p1.2.m2.1.1.1.1.1.1.1.3.3.1"></times><ci id="A2.p1.2.m2.1.1.1.1.1.1.1.3.3.2.cmml" xref="A2.p1.2.m2.1.1.1.1.1.1.1.3.3.2">𝐿</ci><ci id="A2.p1.2.m2.1.1.1.1.1.1.1.3.3.3.cmml" xref="A2.p1.2.m2.1.1.1.1.1.1.1.3.3.3">𝑁</ci></apply></apply><apply id="A2.p1.2.m2.1.1.1.1.1.1.1.1.1.1.cmml" xref="A2.p1.2.m2.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A2.p1.2.m2.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A2.p1.2.m2.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="A2.p1.2.m2.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A2.p1.2.m2.1.1.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="A2.p1.2.m2.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A2.p1.2.m2.1.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply><apply id="A2.p1.2.m2.2.2.2.2.cmml" xref="A2.p1.2.m2.2.2.2.2"><times id="A2.p1.2.m2.2.2.2.2.2.cmml" xref="A2.p1.2.m2.2.2.2.2.2"></times><apply id="A2.p1.2.m2.2.2.2.2.3.cmml" xref="A2.p1.2.m2.2.2.2.2.3"><csymbol cd="ambiguous" id="A2.p1.2.m2.2.2.2.2.3.1.cmml" xref="A2.p1.2.m2.2.2.2.2.3">subscript</csymbol><ci id="A2.p1.2.m2.2.2.2.2.3.2.cmml" xref="A2.p1.2.m2.2.2.2.2.3.2">𝑓</ci><apply id="A2.p1.2.m2.2.2.2.2.3.3.cmml" xref="A2.p1.2.m2.2.2.2.2.3.3"><times id="A2.p1.2.m2.2.2.2.2.3.3.1.cmml" xref="A2.p1.2.m2.2.2.2.2.3.3.1"></times><ci id="A2.p1.2.m2.2.2.2.2.3.3.2.cmml" xref="A2.p1.2.m2.2.2.2.2.3.3.2">𝐹</ci><ci id="A2.p1.2.m2.2.2.2.2.3.3.3.cmml" xref="A2.p1.2.m2.2.2.2.2.3.3.3">𝐹</ci><ci id="A2.p1.2.m2.2.2.2.2.3.3.4.cmml" xref="A2.p1.2.m2.2.2.2.2.3.3.4">𝑁</ci></apply></apply><apply id="A2.p1.2.m2.2.2.2.2.1.1.1.cmml" xref="A2.p1.2.m2.2.2.2.2.1.1"><times id="A2.p1.2.m2.2.2.2.2.1.1.1.2.cmml" xref="A2.p1.2.m2.2.2.2.2.1.1.1.2"></times><apply id="A2.p1.2.m2.2.2.2.2.1.1.1.3.cmml" xref="A2.p1.2.m2.2.2.2.2.1.1.1.3"><csymbol cd="ambiguous" id="A2.p1.2.m2.2.2.2.2.1.1.1.3.1.cmml" xref="A2.p1.2.m2.2.2.2.2.1.1.1.3">subscript</csymbol><ci id="A2.p1.2.m2.2.2.2.2.1.1.1.3.2.cmml" xref="A2.p1.2.m2.2.2.2.2.1.1.1.3.2">𝑓</ci><apply id="A2.p1.2.m2.2.2.2.2.1.1.1.3.3.cmml" xref="A2.p1.2.m2.2.2.2.2.1.1.1.3.3"><times id="A2.p1.2.m2.2.2.2.2.1.1.1.3.3.1.cmml" xref="A2.p1.2.m2.2.2.2.2.1.1.1.3.3.1"></times><ci id="A2.p1.2.m2.2.2.2.2.1.1.1.3.3.2.cmml" xref="A2.p1.2.m2.2.2.2.2.1.1.1.3.3.2">𝐿</ci><ci id="A2.p1.2.m2.2.2.2.2.1.1.1.3.3.3.cmml" xref="A2.p1.2.m2.2.2.2.2.1.1.1.3.3.3">𝑁</ci></apply></apply><apply id="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.cmml" xref="A2.p1.2.m2.2.2.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.1.cmml" xref="A2.p1.2.m2.2.2.2.2.1.1.1.1.1">superscript</csymbol><apply id="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.2.cmml" xref="A2.p1.2.m2.2.2.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.2.1.cmml" xref="A2.p1.2.m2.2.2.2.2.1.1.1.1.1">subscript</csymbol><ci id="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.2.2.cmml" xref="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.2.2">𝑥</ci><ci id="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.2.3.cmml" xref="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.3.cmml" xref="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.3"><ci id="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.3.1.cmml" xref="A2.p1.2.m2.2.2.2.2.1.1.1.1.1.1.3.1">′</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.2.m2.2c">f_{RT}=f_{S-ATTN}(f_{LN}(x_{i}))+f_{FFN}(f_{LN}(x_{i}^{{}^{\prime}}))</annotation></semantics></math>, and the output layer <math id="A2.p1.3.m3.1" class="ltx_Math" alttext="x_{L}=x_{l}+\sum_{i=l}^{L-1}f_{RT}(x_{i})" display="inline"><semantics id="A2.p1.3.m3.1a"><mrow id="A2.p1.3.m3.1.1" xref="A2.p1.3.m3.1.1.cmml"><msub id="A2.p1.3.m3.1.1.3" xref="A2.p1.3.m3.1.1.3.cmml"><mi id="A2.p1.3.m3.1.1.3.2" xref="A2.p1.3.m3.1.1.3.2.cmml">x</mi><mi id="A2.p1.3.m3.1.1.3.3" xref="A2.p1.3.m3.1.1.3.3.cmml">L</mi></msub><mo id="A2.p1.3.m3.1.1.2" xref="A2.p1.3.m3.1.1.2.cmml">=</mo><mrow id="A2.p1.3.m3.1.1.1" xref="A2.p1.3.m3.1.1.1.cmml"><msub id="A2.p1.3.m3.1.1.1.3" xref="A2.p1.3.m3.1.1.1.3.cmml"><mi id="A2.p1.3.m3.1.1.1.3.2" xref="A2.p1.3.m3.1.1.1.3.2.cmml">x</mi><mi id="A2.p1.3.m3.1.1.1.3.3" xref="A2.p1.3.m3.1.1.1.3.3.cmml">l</mi></msub><mo rspace="0.055em" id="A2.p1.3.m3.1.1.1.2" xref="A2.p1.3.m3.1.1.1.2.cmml">+</mo><mrow id="A2.p1.3.m3.1.1.1.1" xref="A2.p1.3.m3.1.1.1.1.cmml"><msubsup id="A2.p1.3.m3.1.1.1.1.2" xref="A2.p1.3.m3.1.1.1.1.2.cmml"><mo id="A2.p1.3.m3.1.1.1.1.2.2.2" xref="A2.p1.3.m3.1.1.1.1.2.2.2.cmml">∑</mo><mrow id="A2.p1.3.m3.1.1.1.1.2.2.3" xref="A2.p1.3.m3.1.1.1.1.2.2.3.cmml"><mi id="A2.p1.3.m3.1.1.1.1.2.2.3.2" xref="A2.p1.3.m3.1.1.1.1.2.2.3.2.cmml">i</mi><mo id="A2.p1.3.m3.1.1.1.1.2.2.3.1" xref="A2.p1.3.m3.1.1.1.1.2.2.3.1.cmml">=</mo><mi id="A2.p1.3.m3.1.1.1.1.2.2.3.3" xref="A2.p1.3.m3.1.1.1.1.2.2.3.3.cmml">l</mi></mrow><mrow id="A2.p1.3.m3.1.1.1.1.2.3" xref="A2.p1.3.m3.1.1.1.1.2.3.cmml"><mi id="A2.p1.3.m3.1.1.1.1.2.3.2" xref="A2.p1.3.m3.1.1.1.1.2.3.2.cmml">L</mi><mo id="A2.p1.3.m3.1.1.1.1.2.3.1" xref="A2.p1.3.m3.1.1.1.1.2.3.1.cmml">−</mo><mn id="A2.p1.3.m3.1.1.1.1.2.3.3" xref="A2.p1.3.m3.1.1.1.1.2.3.3.cmml">1</mn></mrow></msubsup><mrow id="A2.p1.3.m3.1.1.1.1.1" xref="A2.p1.3.m3.1.1.1.1.1.cmml"><msub id="A2.p1.3.m3.1.1.1.1.1.3" xref="A2.p1.3.m3.1.1.1.1.1.3.cmml"><mi id="A2.p1.3.m3.1.1.1.1.1.3.2" xref="A2.p1.3.m3.1.1.1.1.1.3.2.cmml">f</mi><mrow id="A2.p1.3.m3.1.1.1.1.1.3.3" xref="A2.p1.3.m3.1.1.1.1.1.3.3.cmml"><mi id="A2.p1.3.m3.1.1.1.1.1.3.3.2" xref="A2.p1.3.m3.1.1.1.1.1.3.3.2.cmml">R</mi><mo lspace="0em" rspace="0em" id="A2.p1.3.m3.1.1.1.1.1.3.3.1" xref="A2.p1.3.m3.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="A2.p1.3.m3.1.1.1.1.1.3.3.3" xref="A2.p1.3.m3.1.1.1.1.1.3.3.3.cmml">T</mi></mrow></msub><mo lspace="0em" rspace="0em" id="A2.p1.3.m3.1.1.1.1.1.2" xref="A2.p1.3.m3.1.1.1.1.1.2.cmml">​</mo><mrow id="A2.p1.3.m3.1.1.1.1.1.1.1" xref="A2.p1.3.m3.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A2.p1.3.m3.1.1.1.1.1.1.1.2" xref="A2.p1.3.m3.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="A2.p1.3.m3.1.1.1.1.1.1.1.1" xref="A2.p1.3.m3.1.1.1.1.1.1.1.1.cmml"><mi id="A2.p1.3.m3.1.1.1.1.1.1.1.1.2" xref="A2.p1.3.m3.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="A2.p1.3.m3.1.1.1.1.1.1.1.1.3" xref="A2.p1.3.m3.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A2.p1.3.m3.1.1.1.1.1.1.1.3" xref="A2.p1.3.m3.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.p1.3.m3.1b"><apply id="A2.p1.3.m3.1.1.cmml" xref="A2.p1.3.m3.1.1"><eq id="A2.p1.3.m3.1.1.2.cmml" xref="A2.p1.3.m3.1.1.2"></eq><apply id="A2.p1.3.m3.1.1.3.cmml" xref="A2.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="A2.p1.3.m3.1.1.3.1.cmml" xref="A2.p1.3.m3.1.1.3">subscript</csymbol><ci id="A2.p1.3.m3.1.1.3.2.cmml" xref="A2.p1.3.m3.1.1.3.2">𝑥</ci><ci id="A2.p1.3.m3.1.1.3.3.cmml" xref="A2.p1.3.m3.1.1.3.3">𝐿</ci></apply><apply id="A2.p1.3.m3.1.1.1.cmml" xref="A2.p1.3.m3.1.1.1"><plus id="A2.p1.3.m3.1.1.1.2.cmml" xref="A2.p1.3.m3.1.1.1.2"></plus><apply id="A2.p1.3.m3.1.1.1.3.cmml" xref="A2.p1.3.m3.1.1.1.3"><csymbol cd="ambiguous" id="A2.p1.3.m3.1.1.1.3.1.cmml" xref="A2.p1.3.m3.1.1.1.3">subscript</csymbol><ci id="A2.p1.3.m3.1.1.1.3.2.cmml" xref="A2.p1.3.m3.1.1.1.3.2">𝑥</ci><ci id="A2.p1.3.m3.1.1.1.3.3.cmml" xref="A2.p1.3.m3.1.1.1.3.3">𝑙</ci></apply><apply id="A2.p1.3.m3.1.1.1.1.cmml" xref="A2.p1.3.m3.1.1.1.1"><apply id="A2.p1.3.m3.1.1.1.1.2.cmml" xref="A2.p1.3.m3.1.1.1.1.2"><csymbol cd="ambiguous" id="A2.p1.3.m3.1.1.1.1.2.1.cmml" xref="A2.p1.3.m3.1.1.1.1.2">superscript</csymbol><apply id="A2.p1.3.m3.1.1.1.1.2.2.cmml" xref="A2.p1.3.m3.1.1.1.1.2"><csymbol cd="ambiguous" id="A2.p1.3.m3.1.1.1.1.2.2.1.cmml" xref="A2.p1.3.m3.1.1.1.1.2">subscript</csymbol><sum id="A2.p1.3.m3.1.1.1.1.2.2.2.cmml" xref="A2.p1.3.m3.1.1.1.1.2.2.2"></sum><apply id="A2.p1.3.m3.1.1.1.1.2.2.3.cmml" xref="A2.p1.3.m3.1.1.1.1.2.2.3"><eq id="A2.p1.3.m3.1.1.1.1.2.2.3.1.cmml" xref="A2.p1.3.m3.1.1.1.1.2.2.3.1"></eq><ci id="A2.p1.3.m3.1.1.1.1.2.2.3.2.cmml" xref="A2.p1.3.m3.1.1.1.1.2.2.3.2">𝑖</ci><ci id="A2.p1.3.m3.1.1.1.1.2.2.3.3.cmml" xref="A2.p1.3.m3.1.1.1.1.2.2.3.3">𝑙</ci></apply></apply><apply id="A2.p1.3.m3.1.1.1.1.2.3.cmml" xref="A2.p1.3.m3.1.1.1.1.2.3"><minus id="A2.p1.3.m3.1.1.1.1.2.3.1.cmml" xref="A2.p1.3.m3.1.1.1.1.2.3.1"></minus><ci id="A2.p1.3.m3.1.1.1.1.2.3.2.cmml" xref="A2.p1.3.m3.1.1.1.1.2.3.2">𝐿</ci><cn type="integer" id="A2.p1.3.m3.1.1.1.1.2.3.3.cmml" xref="A2.p1.3.m3.1.1.1.1.2.3.3">1</cn></apply></apply><apply id="A2.p1.3.m3.1.1.1.1.1.cmml" xref="A2.p1.3.m3.1.1.1.1.1"><times id="A2.p1.3.m3.1.1.1.1.1.2.cmml" xref="A2.p1.3.m3.1.1.1.1.1.2"></times><apply id="A2.p1.3.m3.1.1.1.1.1.3.cmml" xref="A2.p1.3.m3.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A2.p1.3.m3.1.1.1.1.1.3.1.cmml" xref="A2.p1.3.m3.1.1.1.1.1.3">subscript</csymbol><ci id="A2.p1.3.m3.1.1.1.1.1.3.2.cmml" xref="A2.p1.3.m3.1.1.1.1.1.3.2">𝑓</ci><apply id="A2.p1.3.m3.1.1.1.1.1.3.3.cmml" xref="A2.p1.3.m3.1.1.1.1.1.3.3"><times id="A2.p1.3.m3.1.1.1.1.1.3.3.1.cmml" xref="A2.p1.3.m3.1.1.1.1.1.3.3.1"></times><ci id="A2.p1.3.m3.1.1.1.1.1.3.3.2.cmml" xref="A2.p1.3.m3.1.1.1.1.1.3.3.2">𝑅</ci><ci id="A2.p1.3.m3.1.1.1.1.1.3.3.3.cmml" xref="A2.p1.3.m3.1.1.1.1.1.3.3.3">𝑇</ci></apply></apply><apply id="A2.p1.3.m3.1.1.1.1.1.1.1.1.cmml" xref="A2.p1.3.m3.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A2.p1.3.m3.1.1.1.1.1.1.1.1.1.cmml" xref="A2.p1.3.m3.1.1.1.1.1.1.1">subscript</csymbol><ci id="A2.p1.3.m3.1.1.1.1.1.1.1.1.2.cmml" xref="A2.p1.3.m3.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="A2.p1.3.m3.1.1.1.1.1.1.1.1.3.cmml" xref="A2.p1.3.m3.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.3.m3.1c">x_{L}=x_{l}+\sum_{i=l}^{L-1}f_{RT}(x_{i})</annotation></semantics></math> as the recursive summation of preceding <math id="A2.p1.4.m4.1" class="ltx_Math" alttext="f_{RT}" display="inline"><semantics id="A2.p1.4.m4.1a"><msub id="A2.p1.4.m4.1.1" xref="A2.p1.4.m4.1.1.cmml"><mi id="A2.p1.4.m4.1.1.2" xref="A2.p1.4.m4.1.1.2.cmml">f</mi><mrow id="A2.p1.4.m4.1.1.3" xref="A2.p1.4.m4.1.1.3.cmml"><mi id="A2.p1.4.m4.1.1.3.2" xref="A2.p1.4.m4.1.1.3.2.cmml">R</mi><mo lspace="0em" rspace="0em" id="A2.p1.4.m4.1.1.3.1" xref="A2.p1.4.m4.1.1.3.1.cmml">​</mo><mi id="A2.p1.4.m4.1.1.3.3" xref="A2.p1.4.m4.1.1.3.3.cmml">T</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A2.p1.4.m4.1b"><apply id="A2.p1.4.m4.1.1.cmml" xref="A2.p1.4.m4.1.1"><csymbol cd="ambiguous" id="A2.p1.4.m4.1.1.1.cmml" xref="A2.p1.4.m4.1.1">subscript</csymbol><ci id="A2.p1.4.m4.1.1.2.cmml" xref="A2.p1.4.m4.1.1.2">𝑓</ci><apply id="A2.p1.4.m4.1.1.3.cmml" xref="A2.p1.4.m4.1.1.3"><times id="A2.p1.4.m4.1.1.3.1.cmml" xref="A2.p1.4.m4.1.1.3.1"></times><ci id="A2.p1.4.m4.1.1.3.2.cmml" xref="A2.p1.4.m4.1.1.3.2">𝑅</ci><ci id="A2.p1.4.m4.1.1.3.3.cmml" xref="A2.p1.4.m4.1.1.3.3">𝑇</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.4.m4.1c">f_{RT}</annotation></semantics></math> functions in shallower layers (plus <math id="A2.p1.5.m5.1" class="ltx_Math" alttext="x_{l}" display="inline"><semantics id="A2.p1.5.m5.1a"><msub id="A2.p1.5.m5.1.1" xref="A2.p1.5.m5.1.1.cmml"><mi id="A2.p1.5.m5.1.1.2" xref="A2.p1.5.m5.1.1.2.cmml">x</mi><mi id="A2.p1.5.m5.1.1.3" xref="A2.p1.5.m5.1.1.3.cmml">l</mi></msub><annotation-xml encoding="MathML-Content" id="A2.p1.5.m5.1b"><apply id="A2.p1.5.m5.1.1.cmml" xref="A2.p1.5.m5.1.1"><csymbol cd="ambiguous" id="A2.p1.5.m5.1.1.1.cmml" xref="A2.p1.5.m5.1.1">subscript</csymbol><ci id="A2.p1.5.m5.1.1.2.cmml" xref="A2.p1.5.m5.1.1.2">𝑥</ci><ci id="A2.p1.5.m5.1.1.3.cmml" xref="A2.p1.5.m5.1.1.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.5.m5.1c">x_{l}</annotation></semantics></math>). If we denote the loss function as <math id="A2.p1.6.m6.1" class="ltx_Math" alttext="\mathcal{E}" display="inline"><semantics id="A2.p1.6.m6.1a"><mi class="ltx_font_mathcaligraphic" id="A2.p1.6.m6.1.1" xref="A2.p1.6.m6.1.1.cmml">ℰ</mi><annotation-xml encoding="MathML-Content" id="A2.p1.6.m6.1b"><ci id="A2.p1.6.m6.1.1.cmml" xref="A2.p1.6.m6.1.1">ℰ</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.6.m6.1c">\mathcal{E}</annotation></semantics></math>, from the chain rule of backpropagation <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib46" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">46</span></a>]</cite> we have:</p>
</div>
<div id="A2.p2" class="ltx_para">
<table id="A2.E5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A2.E5.m1.1" class="ltx_Math" alttext="\frac{\partial\mathcal{E}}{\partial x_{l}}=\frac{\partial\mathcal{E}}{\partial x_{L}}\frac{\partial x_{L}}{\partial x_{l}}=\frac{\partial\mathcal{E}}{\partial x_{L}}(1+\frac{\partial}{\partial x_{l}}\sum_{i=l}^{L-1}f_{RT}(x_{i}))" display="block"><semantics id="A2.E5.m1.1a"><mrow id="A2.E5.m1.1.1" xref="A2.E5.m1.1.1.cmml"><mfrac id="A2.E5.m1.1.1.3" xref="A2.E5.m1.1.1.3.cmml"><mrow id="A2.E5.m1.1.1.3.2" xref="A2.E5.m1.1.1.3.2.cmml"><mo rspace="0em" id="A2.E5.m1.1.1.3.2.1" xref="A2.E5.m1.1.1.3.2.1.cmml">∂</mo><mi class="ltx_font_mathcaligraphic" id="A2.E5.m1.1.1.3.2.2" xref="A2.E5.m1.1.1.3.2.2.cmml">ℰ</mi></mrow><mrow id="A2.E5.m1.1.1.3.3" xref="A2.E5.m1.1.1.3.3.cmml"><mo rspace="0em" id="A2.E5.m1.1.1.3.3.1" xref="A2.E5.m1.1.1.3.3.1.cmml">∂</mo><msub id="A2.E5.m1.1.1.3.3.2" xref="A2.E5.m1.1.1.3.3.2.cmml"><mi id="A2.E5.m1.1.1.3.3.2.2" xref="A2.E5.m1.1.1.3.3.2.2.cmml">x</mi><mi id="A2.E5.m1.1.1.3.3.2.3" xref="A2.E5.m1.1.1.3.3.2.3.cmml">l</mi></msub></mrow></mfrac><mo id="A2.E5.m1.1.1.4" xref="A2.E5.m1.1.1.4.cmml">=</mo><mrow id="A2.E5.m1.1.1.5" xref="A2.E5.m1.1.1.5.cmml"><mfrac id="A2.E5.m1.1.1.5.2" xref="A2.E5.m1.1.1.5.2.cmml"><mrow id="A2.E5.m1.1.1.5.2.2" xref="A2.E5.m1.1.1.5.2.2.cmml"><mo rspace="0em" id="A2.E5.m1.1.1.5.2.2.1" xref="A2.E5.m1.1.1.5.2.2.1.cmml">∂</mo><mi class="ltx_font_mathcaligraphic" id="A2.E5.m1.1.1.5.2.2.2" xref="A2.E5.m1.1.1.5.2.2.2.cmml">ℰ</mi></mrow><mrow id="A2.E5.m1.1.1.5.2.3" xref="A2.E5.m1.1.1.5.2.3.cmml"><mo rspace="0em" id="A2.E5.m1.1.1.5.2.3.1" xref="A2.E5.m1.1.1.5.2.3.1.cmml">∂</mo><msub id="A2.E5.m1.1.1.5.2.3.2" xref="A2.E5.m1.1.1.5.2.3.2.cmml"><mi id="A2.E5.m1.1.1.5.2.3.2.2" xref="A2.E5.m1.1.1.5.2.3.2.2.cmml">x</mi><mi id="A2.E5.m1.1.1.5.2.3.2.3" xref="A2.E5.m1.1.1.5.2.3.2.3.cmml">L</mi></msub></mrow></mfrac><mo lspace="0em" rspace="0em" id="A2.E5.m1.1.1.5.1" xref="A2.E5.m1.1.1.5.1.cmml">​</mo><mfrac id="A2.E5.m1.1.1.5.3" xref="A2.E5.m1.1.1.5.3.cmml"><mrow id="A2.E5.m1.1.1.5.3.2" xref="A2.E5.m1.1.1.5.3.2.cmml"><mo rspace="0em" id="A2.E5.m1.1.1.5.3.2.1" xref="A2.E5.m1.1.1.5.3.2.1.cmml">∂</mo><msub id="A2.E5.m1.1.1.5.3.2.2" xref="A2.E5.m1.1.1.5.3.2.2.cmml"><mi id="A2.E5.m1.1.1.5.3.2.2.2" xref="A2.E5.m1.1.1.5.3.2.2.2.cmml">x</mi><mi id="A2.E5.m1.1.1.5.3.2.2.3" xref="A2.E5.m1.1.1.5.3.2.2.3.cmml">L</mi></msub></mrow><mrow id="A2.E5.m1.1.1.5.3.3" xref="A2.E5.m1.1.1.5.3.3.cmml"><mo rspace="0em" id="A2.E5.m1.1.1.5.3.3.1" xref="A2.E5.m1.1.1.5.3.3.1.cmml">∂</mo><msub id="A2.E5.m1.1.1.5.3.3.2" xref="A2.E5.m1.1.1.5.3.3.2.cmml"><mi id="A2.E5.m1.1.1.5.3.3.2.2" xref="A2.E5.m1.1.1.5.3.3.2.2.cmml">x</mi><mi id="A2.E5.m1.1.1.5.3.3.2.3" xref="A2.E5.m1.1.1.5.3.3.2.3.cmml">l</mi></msub></mrow></mfrac></mrow><mo id="A2.E5.m1.1.1.6" xref="A2.E5.m1.1.1.6.cmml">=</mo><mrow id="A2.E5.m1.1.1.1" xref="A2.E5.m1.1.1.1.cmml"><mfrac id="A2.E5.m1.1.1.1.3" xref="A2.E5.m1.1.1.1.3.cmml"><mrow id="A2.E5.m1.1.1.1.3.2" xref="A2.E5.m1.1.1.1.3.2.cmml"><mo rspace="0em" id="A2.E5.m1.1.1.1.3.2.1" xref="A2.E5.m1.1.1.1.3.2.1.cmml">∂</mo><mi class="ltx_font_mathcaligraphic" id="A2.E5.m1.1.1.1.3.2.2" xref="A2.E5.m1.1.1.1.3.2.2.cmml">ℰ</mi></mrow><mrow id="A2.E5.m1.1.1.1.3.3" xref="A2.E5.m1.1.1.1.3.3.cmml"><mo rspace="0em" id="A2.E5.m1.1.1.1.3.3.1" xref="A2.E5.m1.1.1.1.3.3.1.cmml">∂</mo><msub id="A2.E5.m1.1.1.1.3.3.2" xref="A2.E5.m1.1.1.1.3.3.2.cmml"><mi id="A2.E5.m1.1.1.1.3.3.2.2" xref="A2.E5.m1.1.1.1.3.3.2.2.cmml">x</mi><mi id="A2.E5.m1.1.1.1.3.3.2.3" xref="A2.E5.m1.1.1.1.3.3.2.3.cmml">L</mi></msub></mrow></mfrac><mo lspace="0em" rspace="0em" id="A2.E5.m1.1.1.1.2" xref="A2.E5.m1.1.1.1.2.cmml">​</mo><mrow id="A2.E5.m1.1.1.1.1.1" xref="A2.E5.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A2.E5.m1.1.1.1.1.1.2" xref="A2.E5.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="A2.E5.m1.1.1.1.1.1.1" xref="A2.E5.m1.1.1.1.1.1.1.cmml"><mn id="A2.E5.m1.1.1.1.1.1.1.3" xref="A2.E5.m1.1.1.1.1.1.1.3.cmml">1</mn><mo id="A2.E5.m1.1.1.1.1.1.1.2" xref="A2.E5.m1.1.1.1.1.1.1.2.cmml">+</mo><mrow id="A2.E5.m1.1.1.1.1.1.1.1" xref="A2.E5.m1.1.1.1.1.1.1.1.cmml"><mfrac id="A2.E5.m1.1.1.1.1.1.1.1.3" xref="A2.E5.m1.1.1.1.1.1.1.1.3.cmml"><mo id="A2.E5.m1.1.1.1.1.1.1.1.3.2" xref="A2.E5.m1.1.1.1.1.1.1.1.3.2.cmml">∂</mo><mrow id="A2.E5.m1.1.1.1.1.1.1.1.3.3" xref="A2.E5.m1.1.1.1.1.1.1.1.3.3.cmml"><mo rspace="0em" id="A2.E5.m1.1.1.1.1.1.1.1.3.3.1" xref="A2.E5.m1.1.1.1.1.1.1.1.3.3.1.cmml">∂</mo><msub id="A2.E5.m1.1.1.1.1.1.1.1.3.3.2" xref="A2.E5.m1.1.1.1.1.1.1.1.3.3.2.cmml"><mi id="A2.E5.m1.1.1.1.1.1.1.1.3.3.2.2" xref="A2.E5.m1.1.1.1.1.1.1.1.3.3.2.2.cmml">x</mi><mi id="A2.E5.m1.1.1.1.1.1.1.1.3.3.2.3" xref="A2.E5.m1.1.1.1.1.1.1.1.3.3.2.3.cmml">l</mi></msub></mrow></mfrac><mo lspace="0em" rspace="0em" id="A2.E5.m1.1.1.1.1.1.1.1.2" xref="A2.E5.m1.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="A2.E5.m1.1.1.1.1.1.1.1.1" xref="A2.E5.m1.1.1.1.1.1.1.1.1.cmml"><munderover id="A2.E5.m1.1.1.1.1.1.1.1.1.2" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2.cmml"><mo movablelimits="false" id="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.2" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.2.cmml">∑</mo><mrow id="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.3" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.3.cmml"><mi id="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.3.2" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.3.2.cmml">i</mi><mo id="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.3.1" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.3.1.cmml">=</mo><mi id="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.3.3" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.3.3.cmml">l</mi></mrow><mrow id="A2.E5.m1.1.1.1.1.1.1.1.1.2.3" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2.3.cmml"><mi id="A2.E5.m1.1.1.1.1.1.1.1.1.2.3.2" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2.3.2.cmml">L</mi><mo id="A2.E5.m1.1.1.1.1.1.1.1.1.2.3.1" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2.3.1.cmml">−</mo><mn id="A2.E5.m1.1.1.1.1.1.1.1.1.2.3.3" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2.3.3.cmml">1</mn></mrow></munderover><mrow id="A2.E5.m1.1.1.1.1.1.1.1.1.1" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.cmml"><msub id="A2.E5.m1.1.1.1.1.1.1.1.1.1.3" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.2" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.2.cmml">f</mi><mrow id="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.3" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.3.cmml"><mi id="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.3.2" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.3.2.cmml">R</mi><mo lspace="0em" rspace="0em" id="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.3.1" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.3.3" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.3.3.cmml">T</mi></mrow></msub><mo lspace="0em" rspace="0em" id="A2.E5.m1.1.1.1.1.1.1.1.1.1.2" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="A2.E5.m1.1.1.1.1.1.1.1.1.1.1.1" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A2.E5.m1.1.1.1.1.1.1.1.1.1.1.1.2" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="A2.E5.m1.1.1.1.1.1.1.1.1.1.1.1.1" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="A2.E5.m1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="A2.E5.m1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A2.E5.m1.1.1.1.1.1.1.1.1.1.1.1.3" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow></mrow><mo stretchy="false" id="A2.E5.m1.1.1.1.1.1.3" xref="A2.E5.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.E5.m1.1b"><apply id="A2.E5.m1.1.1.cmml" xref="A2.E5.m1.1.1"><and id="A2.E5.m1.1.1a.cmml" xref="A2.E5.m1.1.1"></and><apply id="A2.E5.m1.1.1b.cmml" xref="A2.E5.m1.1.1"><eq id="A2.E5.m1.1.1.4.cmml" xref="A2.E5.m1.1.1.4"></eq><apply id="A2.E5.m1.1.1.3.cmml" xref="A2.E5.m1.1.1.3"><divide id="A2.E5.m1.1.1.3.1.cmml" xref="A2.E5.m1.1.1.3"></divide><apply id="A2.E5.m1.1.1.3.2.cmml" xref="A2.E5.m1.1.1.3.2"><partialdiff id="A2.E5.m1.1.1.3.2.1.cmml" xref="A2.E5.m1.1.1.3.2.1"></partialdiff><ci id="A2.E5.m1.1.1.3.2.2.cmml" xref="A2.E5.m1.1.1.3.2.2">ℰ</ci></apply><apply id="A2.E5.m1.1.1.3.3.cmml" xref="A2.E5.m1.1.1.3.3"><partialdiff id="A2.E5.m1.1.1.3.3.1.cmml" xref="A2.E5.m1.1.1.3.3.1"></partialdiff><apply id="A2.E5.m1.1.1.3.3.2.cmml" xref="A2.E5.m1.1.1.3.3.2"><csymbol cd="ambiguous" id="A2.E5.m1.1.1.3.3.2.1.cmml" xref="A2.E5.m1.1.1.3.3.2">subscript</csymbol><ci id="A2.E5.m1.1.1.3.3.2.2.cmml" xref="A2.E5.m1.1.1.3.3.2.2">𝑥</ci><ci id="A2.E5.m1.1.1.3.3.2.3.cmml" xref="A2.E5.m1.1.1.3.3.2.3">𝑙</ci></apply></apply></apply><apply id="A2.E5.m1.1.1.5.cmml" xref="A2.E5.m1.1.1.5"><times id="A2.E5.m1.1.1.5.1.cmml" xref="A2.E5.m1.1.1.5.1"></times><apply id="A2.E5.m1.1.1.5.2.cmml" xref="A2.E5.m1.1.1.5.2"><divide id="A2.E5.m1.1.1.5.2.1.cmml" xref="A2.E5.m1.1.1.5.2"></divide><apply id="A2.E5.m1.1.1.5.2.2.cmml" xref="A2.E5.m1.1.1.5.2.2"><partialdiff id="A2.E5.m1.1.1.5.2.2.1.cmml" xref="A2.E5.m1.1.1.5.2.2.1"></partialdiff><ci id="A2.E5.m1.1.1.5.2.2.2.cmml" xref="A2.E5.m1.1.1.5.2.2.2">ℰ</ci></apply><apply id="A2.E5.m1.1.1.5.2.3.cmml" xref="A2.E5.m1.1.1.5.2.3"><partialdiff id="A2.E5.m1.1.1.5.2.3.1.cmml" xref="A2.E5.m1.1.1.5.2.3.1"></partialdiff><apply id="A2.E5.m1.1.1.5.2.3.2.cmml" xref="A2.E5.m1.1.1.5.2.3.2"><csymbol cd="ambiguous" id="A2.E5.m1.1.1.5.2.3.2.1.cmml" xref="A2.E5.m1.1.1.5.2.3.2">subscript</csymbol><ci id="A2.E5.m1.1.1.5.2.3.2.2.cmml" xref="A2.E5.m1.1.1.5.2.3.2.2">𝑥</ci><ci id="A2.E5.m1.1.1.5.2.3.2.3.cmml" xref="A2.E5.m1.1.1.5.2.3.2.3">𝐿</ci></apply></apply></apply><apply id="A2.E5.m1.1.1.5.3.cmml" xref="A2.E5.m1.1.1.5.3"><divide id="A2.E5.m1.1.1.5.3.1.cmml" xref="A2.E5.m1.1.1.5.3"></divide><apply id="A2.E5.m1.1.1.5.3.2.cmml" xref="A2.E5.m1.1.1.5.3.2"><partialdiff id="A2.E5.m1.1.1.5.3.2.1.cmml" xref="A2.E5.m1.1.1.5.3.2.1"></partialdiff><apply id="A2.E5.m1.1.1.5.3.2.2.cmml" xref="A2.E5.m1.1.1.5.3.2.2"><csymbol cd="ambiguous" id="A2.E5.m1.1.1.5.3.2.2.1.cmml" xref="A2.E5.m1.1.1.5.3.2.2">subscript</csymbol><ci id="A2.E5.m1.1.1.5.3.2.2.2.cmml" xref="A2.E5.m1.1.1.5.3.2.2.2">𝑥</ci><ci id="A2.E5.m1.1.1.5.3.2.2.3.cmml" xref="A2.E5.m1.1.1.5.3.2.2.3">𝐿</ci></apply></apply><apply id="A2.E5.m1.1.1.5.3.3.cmml" xref="A2.E5.m1.1.1.5.3.3"><partialdiff id="A2.E5.m1.1.1.5.3.3.1.cmml" xref="A2.E5.m1.1.1.5.3.3.1"></partialdiff><apply id="A2.E5.m1.1.1.5.3.3.2.cmml" xref="A2.E5.m1.1.1.5.3.3.2"><csymbol cd="ambiguous" id="A2.E5.m1.1.1.5.3.3.2.1.cmml" xref="A2.E5.m1.1.1.5.3.3.2">subscript</csymbol><ci id="A2.E5.m1.1.1.5.3.3.2.2.cmml" xref="A2.E5.m1.1.1.5.3.3.2.2">𝑥</ci><ci id="A2.E5.m1.1.1.5.3.3.2.3.cmml" xref="A2.E5.m1.1.1.5.3.3.2.3">𝑙</ci></apply></apply></apply></apply></apply><apply id="A2.E5.m1.1.1c.cmml" xref="A2.E5.m1.1.1"><eq id="A2.E5.m1.1.1.6.cmml" xref="A2.E5.m1.1.1.6"></eq><share href="#A2.E5.m1.1.1.5.cmml" id="A2.E5.m1.1.1d.cmml" xref="A2.E5.m1.1.1"></share><apply id="A2.E5.m1.1.1.1.cmml" xref="A2.E5.m1.1.1.1"><times id="A2.E5.m1.1.1.1.2.cmml" xref="A2.E5.m1.1.1.1.2"></times><apply id="A2.E5.m1.1.1.1.3.cmml" xref="A2.E5.m1.1.1.1.3"><divide id="A2.E5.m1.1.1.1.3.1.cmml" xref="A2.E5.m1.1.1.1.3"></divide><apply id="A2.E5.m1.1.1.1.3.2.cmml" xref="A2.E5.m1.1.1.1.3.2"><partialdiff id="A2.E5.m1.1.1.1.3.2.1.cmml" xref="A2.E5.m1.1.1.1.3.2.1"></partialdiff><ci id="A2.E5.m1.1.1.1.3.2.2.cmml" xref="A2.E5.m1.1.1.1.3.2.2">ℰ</ci></apply><apply id="A2.E5.m1.1.1.1.3.3.cmml" xref="A2.E5.m1.1.1.1.3.3"><partialdiff id="A2.E5.m1.1.1.1.3.3.1.cmml" xref="A2.E5.m1.1.1.1.3.3.1"></partialdiff><apply id="A2.E5.m1.1.1.1.3.3.2.cmml" xref="A2.E5.m1.1.1.1.3.3.2"><csymbol cd="ambiguous" id="A2.E5.m1.1.1.1.3.3.2.1.cmml" xref="A2.E5.m1.1.1.1.3.3.2">subscript</csymbol><ci id="A2.E5.m1.1.1.1.3.3.2.2.cmml" xref="A2.E5.m1.1.1.1.3.3.2.2">𝑥</ci><ci id="A2.E5.m1.1.1.1.3.3.2.3.cmml" xref="A2.E5.m1.1.1.1.3.3.2.3">𝐿</ci></apply></apply></apply><apply id="A2.E5.m1.1.1.1.1.1.1.cmml" xref="A2.E5.m1.1.1.1.1.1"><plus id="A2.E5.m1.1.1.1.1.1.1.2.cmml" xref="A2.E5.m1.1.1.1.1.1.1.2"></plus><cn type="integer" id="A2.E5.m1.1.1.1.1.1.1.3.cmml" xref="A2.E5.m1.1.1.1.1.1.1.3">1</cn><apply id="A2.E5.m1.1.1.1.1.1.1.1.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1"><times id="A2.E5.m1.1.1.1.1.1.1.1.2.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.2"></times><apply id="A2.E5.m1.1.1.1.1.1.1.1.3.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.3"><divide id="A2.E5.m1.1.1.1.1.1.1.1.3.1.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.3"></divide><partialdiff id="A2.E5.m1.1.1.1.1.1.1.1.3.2.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.3.2"></partialdiff><apply id="A2.E5.m1.1.1.1.1.1.1.1.3.3.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.3.3"><partialdiff id="A2.E5.m1.1.1.1.1.1.1.1.3.3.1.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.3.3.1"></partialdiff><apply id="A2.E5.m1.1.1.1.1.1.1.1.3.3.2.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.3.3.2"><csymbol cd="ambiguous" id="A2.E5.m1.1.1.1.1.1.1.1.3.3.2.1.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.3.3.2">subscript</csymbol><ci id="A2.E5.m1.1.1.1.1.1.1.1.3.3.2.2.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.3.3.2.2">𝑥</ci><ci id="A2.E5.m1.1.1.1.1.1.1.1.3.3.2.3.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.3.3.2.3">𝑙</ci></apply></apply></apply><apply id="A2.E5.m1.1.1.1.1.1.1.1.1.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1"><apply id="A2.E5.m1.1.1.1.1.1.1.1.1.2.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A2.E5.m1.1.1.1.1.1.1.1.1.2.1.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2">superscript</csymbol><apply id="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.1.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2">subscript</csymbol><sum id="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.2.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.2"></sum><apply id="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.3.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.3"><eq id="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.3.1.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.3.1"></eq><ci id="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.3.2.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.3.2">𝑖</ci><ci id="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.3.3.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2.2.3.3">𝑙</ci></apply></apply><apply id="A2.E5.m1.1.1.1.1.1.1.1.1.2.3.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2.3"><minus id="A2.E5.m1.1.1.1.1.1.1.1.1.2.3.1.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2.3.1"></minus><ci id="A2.E5.m1.1.1.1.1.1.1.1.1.2.3.2.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2.3.2">𝐿</ci><cn type="integer" id="A2.E5.m1.1.1.1.1.1.1.1.1.2.3.3.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.2.3.3">1</cn></apply></apply><apply id="A2.E5.m1.1.1.1.1.1.1.1.1.1.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1"><times id="A2.E5.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.2"></times><apply id="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.2">𝑓</ci><apply id="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.3"><times id="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.3.1.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.3.1"></times><ci id="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.3.2.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.3.2">𝑅</ci><ci id="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.3.3.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.3.3.3">𝑇</ci></apply></apply><apply id="A2.E5.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A2.E5.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="A2.E5.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="A2.E5.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A2.E5.m1.1.1.1.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.E5.m1.1c">\frac{\partial\mathcal{E}}{\partial x_{l}}=\frac{\partial\mathcal{E}}{\partial x_{L}}\frac{\partial x_{L}}{\partial x_{l}}=\frac{\partial\mathcal{E}}{\partial x_{L}}(1+\frac{\partial}{\partial x_{l}}\sum_{i=l}^{L-1}f_{RT}(x_{i}))</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(5)</span></td>
</tr></tbody>
</table>
</div>
<div id="A2.p3" class="ltx_para">
<p id="A2.p3.7" class="ltx_p">Eqn. <a href="#A2.E5" title="In Appendix B Establishing Identity Mapping with PreLN ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> indicates that the gradient <math id="A2.p3.1.m1.1" class="ltx_Math" alttext="\frac{\partial\mathcal{E}}{\partial X_{l}}" display="inline"><semantics id="A2.p3.1.m1.1a"><mfrac id="A2.p3.1.m1.1.1" xref="A2.p3.1.m1.1.1.cmml"><mrow id="A2.p3.1.m1.1.1.2" xref="A2.p3.1.m1.1.1.2.cmml"><mo rspace="0em" id="A2.p3.1.m1.1.1.2.1" xref="A2.p3.1.m1.1.1.2.1.cmml">∂</mo><mi class="ltx_font_mathcaligraphic" id="A2.p3.1.m1.1.1.2.2" xref="A2.p3.1.m1.1.1.2.2.cmml">ℰ</mi></mrow><mrow id="A2.p3.1.m1.1.1.3" xref="A2.p3.1.m1.1.1.3.cmml"><mo rspace="0em" id="A2.p3.1.m1.1.1.3.1" xref="A2.p3.1.m1.1.1.3.1.cmml">∂</mo><msub id="A2.p3.1.m1.1.1.3.2" xref="A2.p3.1.m1.1.1.3.2.cmml"><mi id="A2.p3.1.m1.1.1.3.2.2" xref="A2.p3.1.m1.1.1.3.2.2.cmml">X</mi><mi id="A2.p3.1.m1.1.1.3.2.3" xref="A2.p3.1.m1.1.1.3.2.3.cmml">l</mi></msub></mrow></mfrac><annotation-xml encoding="MathML-Content" id="A2.p3.1.m1.1b"><apply id="A2.p3.1.m1.1.1.cmml" xref="A2.p3.1.m1.1.1"><divide id="A2.p3.1.m1.1.1.1.cmml" xref="A2.p3.1.m1.1.1"></divide><apply id="A2.p3.1.m1.1.1.2.cmml" xref="A2.p3.1.m1.1.1.2"><partialdiff id="A2.p3.1.m1.1.1.2.1.cmml" xref="A2.p3.1.m1.1.1.2.1"></partialdiff><ci id="A2.p3.1.m1.1.1.2.2.cmml" xref="A2.p3.1.m1.1.1.2.2">ℰ</ci></apply><apply id="A2.p3.1.m1.1.1.3.cmml" xref="A2.p3.1.m1.1.1.3"><partialdiff id="A2.p3.1.m1.1.1.3.1.cmml" xref="A2.p3.1.m1.1.1.3.1"></partialdiff><apply id="A2.p3.1.m1.1.1.3.2.cmml" xref="A2.p3.1.m1.1.1.3.2"><csymbol cd="ambiguous" id="A2.p3.1.m1.1.1.3.2.1.cmml" xref="A2.p3.1.m1.1.1.3.2">subscript</csymbol><ci id="A2.p3.1.m1.1.1.3.2.2.cmml" xref="A2.p3.1.m1.1.1.3.2.2">𝑋</ci><ci id="A2.p3.1.m1.1.1.3.2.3.cmml" xref="A2.p3.1.m1.1.1.3.2.3">𝑙</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p3.1.m1.1c">\frac{\partial\mathcal{E}}{\partial X_{l}}</annotation></semantics></math> can be decomposed into two additive terms: a term of <math id="A2.p3.2.m2.1" class="ltx_Math" alttext="\frac{\partial\mathcal{E}}{\partial X_{L}}" display="inline"><semantics id="A2.p3.2.m2.1a"><mfrac id="A2.p3.2.m2.1.1" xref="A2.p3.2.m2.1.1.cmml"><mrow id="A2.p3.2.m2.1.1.2" xref="A2.p3.2.m2.1.1.2.cmml"><mo rspace="0em" id="A2.p3.2.m2.1.1.2.1" xref="A2.p3.2.m2.1.1.2.1.cmml">∂</mo><mi class="ltx_font_mathcaligraphic" id="A2.p3.2.m2.1.1.2.2" xref="A2.p3.2.m2.1.1.2.2.cmml">ℰ</mi></mrow><mrow id="A2.p3.2.m2.1.1.3" xref="A2.p3.2.m2.1.1.3.cmml"><mo rspace="0em" id="A2.p3.2.m2.1.1.3.1" xref="A2.p3.2.m2.1.1.3.1.cmml">∂</mo><msub id="A2.p3.2.m2.1.1.3.2" xref="A2.p3.2.m2.1.1.3.2.cmml"><mi id="A2.p3.2.m2.1.1.3.2.2" xref="A2.p3.2.m2.1.1.3.2.2.cmml">X</mi><mi id="A2.p3.2.m2.1.1.3.2.3" xref="A2.p3.2.m2.1.1.3.2.3.cmml">L</mi></msub></mrow></mfrac><annotation-xml encoding="MathML-Content" id="A2.p3.2.m2.1b"><apply id="A2.p3.2.m2.1.1.cmml" xref="A2.p3.2.m2.1.1"><divide id="A2.p3.2.m2.1.1.1.cmml" xref="A2.p3.2.m2.1.1"></divide><apply id="A2.p3.2.m2.1.1.2.cmml" xref="A2.p3.2.m2.1.1.2"><partialdiff id="A2.p3.2.m2.1.1.2.1.cmml" xref="A2.p3.2.m2.1.1.2.1"></partialdiff><ci id="A2.p3.2.m2.1.1.2.2.cmml" xref="A2.p3.2.m2.1.1.2.2">ℰ</ci></apply><apply id="A2.p3.2.m2.1.1.3.cmml" xref="A2.p3.2.m2.1.1.3"><partialdiff id="A2.p3.2.m2.1.1.3.1.cmml" xref="A2.p3.2.m2.1.1.3.1"></partialdiff><apply id="A2.p3.2.m2.1.1.3.2.cmml" xref="A2.p3.2.m2.1.1.3.2"><csymbol cd="ambiguous" id="A2.p3.2.m2.1.1.3.2.1.cmml" xref="A2.p3.2.m2.1.1.3.2">subscript</csymbol><ci id="A2.p3.2.m2.1.1.3.2.2.cmml" xref="A2.p3.2.m2.1.1.3.2.2">𝑋</ci><ci id="A2.p3.2.m2.1.1.3.2.3.cmml" xref="A2.p3.2.m2.1.1.3.2.3">𝐿</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p3.2.m2.1c">\frac{\partial\mathcal{E}}{\partial X_{L}}</annotation></semantics></math> that propagates information directly back to any shallower <math id="A2.p3.3.m3.1" class="ltx_Math" alttext="l" display="inline"><semantics id="A2.p3.3.m3.1a"><mi id="A2.p3.3.m3.1.1" xref="A2.p3.3.m3.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="A2.p3.3.m3.1b"><ci id="A2.p3.3.m3.1.1.cmml" xref="A2.p3.3.m3.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.p3.3.m3.1c">l</annotation></semantics></math>-th block without concerning how complex <math id="A2.p3.4.m4.1" class="ltx_math_unparsed" alttext="\frac{\partial}{\partial x_{l}}\sum_{i=l}^{L-1}f_{RT}(x_{i}))" display="inline"><semantics id="A2.p3.4.m4.1a"><mrow id="A2.p3.4.m4.1b"><mfrac id="A2.p3.4.m4.1.1"><mo id="A2.p3.4.m4.1.1.2">∂</mo><mrow id="A2.p3.4.m4.1.1.3"><mo rspace="0em" id="A2.p3.4.m4.1.1.3.1">∂</mo><msub id="A2.p3.4.m4.1.1.3.2"><mi id="A2.p3.4.m4.1.1.3.2.2">x</mi><mi id="A2.p3.4.m4.1.1.3.2.3">l</mi></msub></mrow></mfrac><msubsup id="A2.p3.4.m4.1.2"><mo id="A2.p3.4.m4.1.2.2.2">∑</mo><mrow id="A2.p3.4.m4.1.2.2.3"><mi id="A2.p3.4.m4.1.2.2.3.2">i</mi><mo id="A2.p3.4.m4.1.2.2.3.1">=</mo><mi id="A2.p3.4.m4.1.2.2.3.3">l</mi></mrow><mrow id="A2.p3.4.m4.1.2.3"><mi id="A2.p3.4.m4.1.2.3.2">L</mi><mo id="A2.p3.4.m4.1.2.3.1">−</mo><mn id="A2.p3.4.m4.1.2.3.3">1</mn></mrow></msubsup><msub id="A2.p3.4.m4.1.3"><mi id="A2.p3.4.m4.1.3.2">f</mi><mrow id="A2.p3.4.m4.1.3.3"><mi id="A2.p3.4.m4.1.3.3.2">R</mi><mo lspace="0em" rspace="0em" id="A2.p3.4.m4.1.3.3.1">​</mo><mi id="A2.p3.4.m4.1.3.3.3">T</mi></mrow></msub><mrow id="A2.p3.4.m4.1.4"><mo stretchy="false" id="A2.p3.4.m4.1.4.1">(</mo><msub id="A2.p3.4.m4.1.4.2"><mi id="A2.p3.4.m4.1.4.2.2">x</mi><mi id="A2.p3.4.m4.1.4.2.3">i</mi></msub><mo stretchy="false" id="A2.p3.4.m4.1.4.3">)</mo></mrow><mo stretchy="false" id="A2.p3.4.m4.1.5">)</mo></mrow><annotation encoding="application/x-tex" id="A2.p3.4.m4.1c">\frac{\partial}{\partial x_{l}}\sum_{i=l}^{L-1}f_{RT}(x_{i}))</annotation></semantics></math> would be, and another term of <math id="A2.p3.5.m5.1" class="ltx_Math" alttext="\frac{\partial\mathcal{E}}{\partial X_{L}}(\frac{\partial}{\partial X_{l}}\sum_{i=l}^{L-1}f_{RT}(X_{i}))" display="inline"><semantics id="A2.p3.5.m5.1a"><mrow id="A2.p3.5.m5.1.1" xref="A2.p3.5.m5.1.1.cmml"><mfrac id="A2.p3.5.m5.1.1.3" xref="A2.p3.5.m5.1.1.3.cmml"><mrow id="A2.p3.5.m5.1.1.3.2" xref="A2.p3.5.m5.1.1.3.2.cmml"><mo rspace="0em" id="A2.p3.5.m5.1.1.3.2.1" xref="A2.p3.5.m5.1.1.3.2.1.cmml">∂</mo><mi class="ltx_font_mathcaligraphic" id="A2.p3.5.m5.1.1.3.2.2" xref="A2.p3.5.m5.1.1.3.2.2.cmml">ℰ</mi></mrow><mrow id="A2.p3.5.m5.1.1.3.3" xref="A2.p3.5.m5.1.1.3.3.cmml"><mo rspace="0em" id="A2.p3.5.m5.1.1.3.3.1" xref="A2.p3.5.m5.1.1.3.3.1.cmml">∂</mo><msub id="A2.p3.5.m5.1.1.3.3.2" xref="A2.p3.5.m5.1.1.3.3.2.cmml"><mi id="A2.p3.5.m5.1.1.3.3.2.2" xref="A2.p3.5.m5.1.1.3.3.2.2.cmml">X</mi><mi id="A2.p3.5.m5.1.1.3.3.2.3" xref="A2.p3.5.m5.1.1.3.3.2.3.cmml">L</mi></msub></mrow></mfrac><mo lspace="0em" rspace="0em" id="A2.p3.5.m5.1.1.2" xref="A2.p3.5.m5.1.1.2.cmml">​</mo><mrow id="A2.p3.5.m5.1.1.1.1" xref="A2.p3.5.m5.1.1.1.1.1.cmml"><mo stretchy="false" id="A2.p3.5.m5.1.1.1.1.2" xref="A2.p3.5.m5.1.1.1.1.1.cmml">(</mo><mrow id="A2.p3.5.m5.1.1.1.1.1" xref="A2.p3.5.m5.1.1.1.1.1.cmml"><mfrac id="A2.p3.5.m5.1.1.1.1.1.3" xref="A2.p3.5.m5.1.1.1.1.1.3.cmml"><mo id="A2.p3.5.m5.1.1.1.1.1.3.2" xref="A2.p3.5.m5.1.1.1.1.1.3.2.cmml">∂</mo><mrow id="A2.p3.5.m5.1.1.1.1.1.3.3" xref="A2.p3.5.m5.1.1.1.1.1.3.3.cmml"><mo rspace="0em" id="A2.p3.5.m5.1.1.1.1.1.3.3.1" xref="A2.p3.5.m5.1.1.1.1.1.3.3.1.cmml">∂</mo><msub id="A2.p3.5.m5.1.1.1.1.1.3.3.2" xref="A2.p3.5.m5.1.1.1.1.1.3.3.2.cmml"><mi id="A2.p3.5.m5.1.1.1.1.1.3.3.2.2" xref="A2.p3.5.m5.1.1.1.1.1.3.3.2.2.cmml">X</mi><mi id="A2.p3.5.m5.1.1.1.1.1.3.3.2.3" xref="A2.p3.5.m5.1.1.1.1.1.3.3.2.3.cmml">l</mi></msub></mrow></mfrac><mo lspace="0em" rspace="0em" id="A2.p3.5.m5.1.1.1.1.1.2" xref="A2.p3.5.m5.1.1.1.1.1.2.cmml">​</mo><mrow id="A2.p3.5.m5.1.1.1.1.1.1" xref="A2.p3.5.m5.1.1.1.1.1.1.cmml"><msubsup id="A2.p3.5.m5.1.1.1.1.1.1.2" xref="A2.p3.5.m5.1.1.1.1.1.1.2.cmml"><mo id="A2.p3.5.m5.1.1.1.1.1.1.2.2.2" xref="A2.p3.5.m5.1.1.1.1.1.1.2.2.2.cmml">∑</mo><mrow id="A2.p3.5.m5.1.1.1.1.1.1.2.2.3" xref="A2.p3.5.m5.1.1.1.1.1.1.2.2.3.cmml"><mi id="A2.p3.5.m5.1.1.1.1.1.1.2.2.3.2" xref="A2.p3.5.m5.1.1.1.1.1.1.2.2.3.2.cmml">i</mi><mo id="A2.p3.5.m5.1.1.1.1.1.1.2.2.3.1" xref="A2.p3.5.m5.1.1.1.1.1.1.2.2.3.1.cmml">=</mo><mi id="A2.p3.5.m5.1.1.1.1.1.1.2.2.3.3" xref="A2.p3.5.m5.1.1.1.1.1.1.2.2.3.3.cmml">l</mi></mrow><mrow id="A2.p3.5.m5.1.1.1.1.1.1.2.3" xref="A2.p3.5.m5.1.1.1.1.1.1.2.3.cmml"><mi id="A2.p3.5.m5.1.1.1.1.1.1.2.3.2" xref="A2.p3.5.m5.1.1.1.1.1.1.2.3.2.cmml">L</mi><mo id="A2.p3.5.m5.1.1.1.1.1.1.2.3.1" xref="A2.p3.5.m5.1.1.1.1.1.1.2.3.1.cmml">−</mo><mn id="A2.p3.5.m5.1.1.1.1.1.1.2.3.3" xref="A2.p3.5.m5.1.1.1.1.1.1.2.3.3.cmml">1</mn></mrow></msubsup><mrow id="A2.p3.5.m5.1.1.1.1.1.1.1" xref="A2.p3.5.m5.1.1.1.1.1.1.1.cmml"><msub id="A2.p3.5.m5.1.1.1.1.1.1.1.3" xref="A2.p3.5.m5.1.1.1.1.1.1.1.3.cmml"><mi id="A2.p3.5.m5.1.1.1.1.1.1.1.3.2" xref="A2.p3.5.m5.1.1.1.1.1.1.1.3.2.cmml">f</mi><mrow id="A2.p3.5.m5.1.1.1.1.1.1.1.3.3" xref="A2.p3.5.m5.1.1.1.1.1.1.1.3.3.cmml"><mi id="A2.p3.5.m5.1.1.1.1.1.1.1.3.3.2" xref="A2.p3.5.m5.1.1.1.1.1.1.1.3.3.2.cmml">R</mi><mo lspace="0em" rspace="0em" id="A2.p3.5.m5.1.1.1.1.1.1.1.3.3.1" xref="A2.p3.5.m5.1.1.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="A2.p3.5.m5.1.1.1.1.1.1.1.3.3.3" xref="A2.p3.5.m5.1.1.1.1.1.1.1.3.3.3.cmml">T</mi></mrow></msub><mo lspace="0em" rspace="0em" id="A2.p3.5.m5.1.1.1.1.1.1.1.2" xref="A2.p3.5.m5.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="A2.p3.5.m5.1.1.1.1.1.1.1.1.1" xref="A2.p3.5.m5.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A2.p3.5.m5.1.1.1.1.1.1.1.1.1.2" xref="A2.p3.5.m5.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="A2.p3.5.m5.1.1.1.1.1.1.1.1.1.1" xref="A2.p3.5.m5.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="A2.p3.5.m5.1.1.1.1.1.1.1.1.1.1.2" xref="A2.p3.5.m5.1.1.1.1.1.1.1.1.1.1.2.cmml">X</mi><mi id="A2.p3.5.m5.1.1.1.1.1.1.1.1.1.1.3" xref="A2.p3.5.m5.1.1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A2.p3.5.m5.1.1.1.1.1.1.1.1.1.3" xref="A2.p3.5.m5.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><mo stretchy="false" id="A2.p3.5.m5.1.1.1.1.3" xref="A2.p3.5.m5.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.p3.5.m5.1b"><apply id="A2.p3.5.m5.1.1.cmml" xref="A2.p3.5.m5.1.1"><times id="A2.p3.5.m5.1.1.2.cmml" xref="A2.p3.5.m5.1.1.2"></times><apply id="A2.p3.5.m5.1.1.3.cmml" xref="A2.p3.5.m5.1.1.3"><divide id="A2.p3.5.m5.1.1.3.1.cmml" xref="A2.p3.5.m5.1.1.3"></divide><apply id="A2.p3.5.m5.1.1.3.2.cmml" xref="A2.p3.5.m5.1.1.3.2"><partialdiff id="A2.p3.5.m5.1.1.3.2.1.cmml" xref="A2.p3.5.m5.1.1.3.2.1"></partialdiff><ci id="A2.p3.5.m5.1.1.3.2.2.cmml" xref="A2.p3.5.m5.1.1.3.2.2">ℰ</ci></apply><apply id="A2.p3.5.m5.1.1.3.3.cmml" xref="A2.p3.5.m5.1.1.3.3"><partialdiff id="A2.p3.5.m5.1.1.3.3.1.cmml" xref="A2.p3.5.m5.1.1.3.3.1"></partialdiff><apply id="A2.p3.5.m5.1.1.3.3.2.cmml" xref="A2.p3.5.m5.1.1.3.3.2"><csymbol cd="ambiguous" id="A2.p3.5.m5.1.1.3.3.2.1.cmml" xref="A2.p3.5.m5.1.1.3.3.2">subscript</csymbol><ci id="A2.p3.5.m5.1.1.3.3.2.2.cmml" xref="A2.p3.5.m5.1.1.3.3.2.2">𝑋</ci><ci id="A2.p3.5.m5.1.1.3.3.2.3.cmml" xref="A2.p3.5.m5.1.1.3.3.2.3">𝐿</ci></apply></apply></apply><apply id="A2.p3.5.m5.1.1.1.1.1.cmml" xref="A2.p3.5.m5.1.1.1.1"><times id="A2.p3.5.m5.1.1.1.1.1.2.cmml" xref="A2.p3.5.m5.1.1.1.1.1.2"></times><apply id="A2.p3.5.m5.1.1.1.1.1.3.cmml" xref="A2.p3.5.m5.1.1.1.1.1.3"><divide id="A2.p3.5.m5.1.1.1.1.1.3.1.cmml" xref="A2.p3.5.m5.1.1.1.1.1.3"></divide><partialdiff id="A2.p3.5.m5.1.1.1.1.1.3.2.cmml" xref="A2.p3.5.m5.1.1.1.1.1.3.2"></partialdiff><apply id="A2.p3.5.m5.1.1.1.1.1.3.3.cmml" xref="A2.p3.5.m5.1.1.1.1.1.3.3"><partialdiff id="A2.p3.5.m5.1.1.1.1.1.3.3.1.cmml" xref="A2.p3.5.m5.1.1.1.1.1.3.3.1"></partialdiff><apply id="A2.p3.5.m5.1.1.1.1.1.3.3.2.cmml" xref="A2.p3.5.m5.1.1.1.1.1.3.3.2"><csymbol cd="ambiguous" id="A2.p3.5.m5.1.1.1.1.1.3.3.2.1.cmml" xref="A2.p3.5.m5.1.1.1.1.1.3.3.2">subscript</csymbol><ci id="A2.p3.5.m5.1.1.1.1.1.3.3.2.2.cmml" xref="A2.p3.5.m5.1.1.1.1.1.3.3.2.2">𝑋</ci><ci id="A2.p3.5.m5.1.1.1.1.1.3.3.2.3.cmml" xref="A2.p3.5.m5.1.1.1.1.1.3.3.2.3">𝑙</ci></apply></apply></apply><apply id="A2.p3.5.m5.1.1.1.1.1.1.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1"><apply id="A2.p3.5.m5.1.1.1.1.1.1.2.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A2.p3.5.m5.1.1.1.1.1.1.2.1.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.2">superscript</csymbol><apply id="A2.p3.5.m5.1.1.1.1.1.1.2.2.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A2.p3.5.m5.1.1.1.1.1.1.2.2.1.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.2">subscript</csymbol><sum id="A2.p3.5.m5.1.1.1.1.1.1.2.2.2.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.2.2.2"></sum><apply id="A2.p3.5.m5.1.1.1.1.1.1.2.2.3.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.2.2.3"><eq id="A2.p3.5.m5.1.1.1.1.1.1.2.2.3.1.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.2.2.3.1"></eq><ci id="A2.p3.5.m5.1.1.1.1.1.1.2.2.3.2.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.2.2.3.2">𝑖</ci><ci id="A2.p3.5.m5.1.1.1.1.1.1.2.2.3.3.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.2.2.3.3">𝑙</ci></apply></apply><apply id="A2.p3.5.m5.1.1.1.1.1.1.2.3.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.2.3"><minus id="A2.p3.5.m5.1.1.1.1.1.1.2.3.1.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.2.3.1"></minus><ci id="A2.p3.5.m5.1.1.1.1.1.1.2.3.2.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.2.3.2">𝐿</ci><cn type="integer" id="A2.p3.5.m5.1.1.1.1.1.1.2.3.3.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.2.3.3">1</cn></apply></apply><apply id="A2.p3.5.m5.1.1.1.1.1.1.1.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.1"><times id="A2.p3.5.m5.1.1.1.1.1.1.1.2.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.1.2"></times><apply id="A2.p3.5.m5.1.1.1.1.1.1.1.3.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A2.p3.5.m5.1.1.1.1.1.1.1.3.1.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="A2.p3.5.m5.1.1.1.1.1.1.1.3.2.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.1.3.2">𝑓</ci><apply id="A2.p3.5.m5.1.1.1.1.1.1.1.3.3.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.1.3.3"><times id="A2.p3.5.m5.1.1.1.1.1.1.1.3.3.1.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.1.3.3.1"></times><ci id="A2.p3.5.m5.1.1.1.1.1.1.1.3.3.2.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.1.3.3.2">𝑅</ci><ci id="A2.p3.5.m5.1.1.1.1.1.1.1.3.3.3.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.1.3.3.3">𝑇</ci></apply></apply><apply id="A2.p3.5.m5.1.1.1.1.1.1.1.1.1.1.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A2.p3.5.m5.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="A2.p3.5.m5.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.1.1.1.1.2">𝑋</ci><ci id="A2.p3.5.m5.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A2.p3.5.m5.1.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p3.5.m5.1c">\frac{\partial\mathcal{E}}{\partial X_{L}}(\frac{\partial}{\partial X_{l}}\sum_{i=l}^{L-1}f_{RT}(X_{i}))</annotation></semantics></math> that propagates through the Transformer blocks.
The equation also suggests that
it is unlikely for the gradient <math id="A2.p3.6.m6.1" class="ltx_Math" alttext="\frac{\partial}{\partial X_{l}}" display="inline"><semantics id="A2.p3.6.m6.1a"><mfrac id="A2.p3.6.m6.1.1" xref="A2.p3.6.m6.1.1.cmml"><mo id="A2.p3.6.m6.1.1.2" xref="A2.p3.6.m6.1.1.2.cmml">∂</mo><mrow id="A2.p3.6.m6.1.1.3" xref="A2.p3.6.m6.1.1.3.cmml"><mo rspace="0em" id="A2.p3.6.m6.1.1.3.1" xref="A2.p3.6.m6.1.1.3.1.cmml">∂</mo><msub id="A2.p3.6.m6.1.1.3.2" xref="A2.p3.6.m6.1.1.3.2.cmml"><mi id="A2.p3.6.m6.1.1.3.2.2" xref="A2.p3.6.m6.1.1.3.2.2.cmml">X</mi><mi id="A2.p3.6.m6.1.1.3.2.3" xref="A2.p3.6.m6.1.1.3.2.3.cmml">l</mi></msub></mrow></mfrac><annotation-xml encoding="MathML-Content" id="A2.p3.6.m6.1b"><apply id="A2.p3.6.m6.1.1.cmml" xref="A2.p3.6.m6.1.1"><divide id="A2.p3.6.m6.1.1.1.cmml" xref="A2.p3.6.m6.1.1"></divide><partialdiff id="A2.p3.6.m6.1.1.2.cmml" xref="A2.p3.6.m6.1.1.2"></partialdiff><apply id="A2.p3.6.m6.1.1.3.cmml" xref="A2.p3.6.m6.1.1.3"><partialdiff id="A2.p3.6.m6.1.1.3.1.cmml" xref="A2.p3.6.m6.1.1.3.1"></partialdiff><apply id="A2.p3.6.m6.1.1.3.2.cmml" xref="A2.p3.6.m6.1.1.3.2"><csymbol cd="ambiguous" id="A2.p3.6.m6.1.1.3.2.1.cmml" xref="A2.p3.6.m6.1.1.3.2">subscript</csymbol><ci id="A2.p3.6.m6.1.1.3.2.2.cmml" xref="A2.p3.6.m6.1.1.3.2.2">𝑋</ci><ci id="A2.p3.6.m6.1.1.3.2.3.cmml" xref="A2.p3.6.m6.1.1.3.2.3">𝑙</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p3.6.m6.1c">\frac{\partial}{\partial X_{l}}</annotation></semantics></math> to be canceled out for a mini-batch, and in general the term <math id="A2.p3.7.m7.1" class="ltx_Math" alttext="\frac{\partial}{\partial X_{l}}\sum_{i=l}^{L-1}f_{RT}(X_{i})" display="inline"><semantics id="A2.p3.7.m7.1a"><mrow id="A2.p3.7.m7.1.1" xref="A2.p3.7.m7.1.1.cmml"><mfrac id="A2.p3.7.m7.1.1.3" xref="A2.p3.7.m7.1.1.3.cmml"><mo id="A2.p3.7.m7.1.1.3.2" xref="A2.p3.7.m7.1.1.3.2.cmml">∂</mo><mrow id="A2.p3.7.m7.1.1.3.3" xref="A2.p3.7.m7.1.1.3.3.cmml"><mo rspace="0em" id="A2.p3.7.m7.1.1.3.3.1" xref="A2.p3.7.m7.1.1.3.3.1.cmml">∂</mo><msub id="A2.p3.7.m7.1.1.3.3.2" xref="A2.p3.7.m7.1.1.3.3.2.cmml"><mi id="A2.p3.7.m7.1.1.3.3.2.2" xref="A2.p3.7.m7.1.1.3.3.2.2.cmml">X</mi><mi id="A2.p3.7.m7.1.1.3.3.2.3" xref="A2.p3.7.m7.1.1.3.3.2.3.cmml">l</mi></msub></mrow></mfrac><mo lspace="0em" rspace="0em" id="A2.p3.7.m7.1.1.2" xref="A2.p3.7.m7.1.1.2.cmml">​</mo><mrow id="A2.p3.7.m7.1.1.1" xref="A2.p3.7.m7.1.1.1.cmml"><msubsup id="A2.p3.7.m7.1.1.1.2" xref="A2.p3.7.m7.1.1.1.2.cmml"><mo id="A2.p3.7.m7.1.1.1.2.2.2" xref="A2.p3.7.m7.1.1.1.2.2.2.cmml">∑</mo><mrow id="A2.p3.7.m7.1.1.1.2.2.3" xref="A2.p3.7.m7.1.1.1.2.2.3.cmml"><mi id="A2.p3.7.m7.1.1.1.2.2.3.2" xref="A2.p3.7.m7.1.1.1.2.2.3.2.cmml">i</mi><mo id="A2.p3.7.m7.1.1.1.2.2.3.1" xref="A2.p3.7.m7.1.1.1.2.2.3.1.cmml">=</mo><mi id="A2.p3.7.m7.1.1.1.2.2.3.3" xref="A2.p3.7.m7.1.1.1.2.2.3.3.cmml">l</mi></mrow><mrow id="A2.p3.7.m7.1.1.1.2.3" xref="A2.p3.7.m7.1.1.1.2.3.cmml"><mi id="A2.p3.7.m7.1.1.1.2.3.2" xref="A2.p3.7.m7.1.1.1.2.3.2.cmml">L</mi><mo id="A2.p3.7.m7.1.1.1.2.3.1" xref="A2.p3.7.m7.1.1.1.2.3.1.cmml">−</mo><mn id="A2.p3.7.m7.1.1.1.2.3.3" xref="A2.p3.7.m7.1.1.1.2.3.3.cmml">1</mn></mrow></msubsup><mrow id="A2.p3.7.m7.1.1.1.1" xref="A2.p3.7.m7.1.1.1.1.cmml"><msub id="A2.p3.7.m7.1.1.1.1.3" xref="A2.p3.7.m7.1.1.1.1.3.cmml"><mi id="A2.p3.7.m7.1.1.1.1.3.2" xref="A2.p3.7.m7.1.1.1.1.3.2.cmml">f</mi><mrow id="A2.p3.7.m7.1.1.1.1.3.3" xref="A2.p3.7.m7.1.1.1.1.3.3.cmml"><mi id="A2.p3.7.m7.1.1.1.1.3.3.2" xref="A2.p3.7.m7.1.1.1.1.3.3.2.cmml">R</mi><mo lspace="0em" rspace="0em" id="A2.p3.7.m7.1.1.1.1.3.3.1" xref="A2.p3.7.m7.1.1.1.1.3.3.1.cmml">​</mo><mi id="A2.p3.7.m7.1.1.1.1.3.3.3" xref="A2.p3.7.m7.1.1.1.1.3.3.3.cmml">T</mi></mrow></msub><mo lspace="0em" rspace="0em" id="A2.p3.7.m7.1.1.1.1.2" xref="A2.p3.7.m7.1.1.1.1.2.cmml">​</mo><mrow id="A2.p3.7.m7.1.1.1.1.1.1" xref="A2.p3.7.m7.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A2.p3.7.m7.1.1.1.1.1.1.2" xref="A2.p3.7.m7.1.1.1.1.1.1.1.cmml">(</mo><msub id="A2.p3.7.m7.1.1.1.1.1.1.1" xref="A2.p3.7.m7.1.1.1.1.1.1.1.cmml"><mi id="A2.p3.7.m7.1.1.1.1.1.1.1.2" xref="A2.p3.7.m7.1.1.1.1.1.1.1.2.cmml">X</mi><mi id="A2.p3.7.m7.1.1.1.1.1.1.1.3" xref="A2.p3.7.m7.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A2.p3.7.m7.1.1.1.1.1.1.3" xref="A2.p3.7.m7.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.p3.7.m7.1b"><apply id="A2.p3.7.m7.1.1.cmml" xref="A2.p3.7.m7.1.1"><times id="A2.p3.7.m7.1.1.2.cmml" xref="A2.p3.7.m7.1.1.2"></times><apply id="A2.p3.7.m7.1.1.3.cmml" xref="A2.p3.7.m7.1.1.3"><divide id="A2.p3.7.m7.1.1.3.1.cmml" xref="A2.p3.7.m7.1.1.3"></divide><partialdiff id="A2.p3.7.m7.1.1.3.2.cmml" xref="A2.p3.7.m7.1.1.3.2"></partialdiff><apply id="A2.p3.7.m7.1.1.3.3.cmml" xref="A2.p3.7.m7.1.1.3.3"><partialdiff id="A2.p3.7.m7.1.1.3.3.1.cmml" xref="A2.p3.7.m7.1.1.3.3.1"></partialdiff><apply id="A2.p3.7.m7.1.1.3.3.2.cmml" xref="A2.p3.7.m7.1.1.3.3.2"><csymbol cd="ambiguous" id="A2.p3.7.m7.1.1.3.3.2.1.cmml" xref="A2.p3.7.m7.1.1.3.3.2">subscript</csymbol><ci id="A2.p3.7.m7.1.1.3.3.2.2.cmml" xref="A2.p3.7.m7.1.1.3.3.2.2">𝑋</ci><ci id="A2.p3.7.m7.1.1.3.3.2.3.cmml" xref="A2.p3.7.m7.1.1.3.3.2.3">𝑙</ci></apply></apply></apply><apply id="A2.p3.7.m7.1.1.1.cmml" xref="A2.p3.7.m7.1.1.1"><apply id="A2.p3.7.m7.1.1.1.2.cmml" xref="A2.p3.7.m7.1.1.1.2"><csymbol cd="ambiguous" id="A2.p3.7.m7.1.1.1.2.1.cmml" xref="A2.p3.7.m7.1.1.1.2">superscript</csymbol><apply id="A2.p3.7.m7.1.1.1.2.2.cmml" xref="A2.p3.7.m7.1.1.1.2"><csymbol cd="ambiguous" id="A2.p3.7.m7.1.1.1.2.2.1.cmml" xref="A2.p3.7.m7.1.1.1.2">subscript</csymbol><sum id="A2.p3.7.m7.1.1.1.2.2.2.cmml" xref="A2.p3.7.m7.1.1.1.2.2.2"></sum><apply id="A2.p3.7.m7.1.1.1.2.2.3.cmml" xref="A2.p3.7.m7.1.1.1.2.2.3"><eq id="A2.p3.7.m7.1.1.1.2.2.3.1.cmml" xref="A2.p3.7.m7.1.1.1.2.2.3.1"></eq><ci id="A2.p3.7.m7.1.1.1.2.2.3.2.cmml" xref="A2.p3.7.m7.1.1.1.2.2.3.2">𝑖</ci><ci id="A2.p3.7.m7.1.1.1.2.2.3.3.cmml" xref="A2.p3.7.m7.1.1.1.2.2.3.3">𝑙</ci></apply></apply><apply id="A2.p3.7.m7.1.1.1.2.3.cmml" xref="A2.p3.7.m7.1.1.1.2.3"><minus id="A2.p3.7.m7.1.1.1.2.3.1.cmml" xref="A2.p3.7.m7.1.1.1.2.3.1"></minus><ci id="A2.p3.7.m7.1.1.1.2.3.2.cmml" xref="A2.p3.7.m7.1.1.1.2.3.2">𝐿</ci><cn type="integer" id="A2.p3.7.m7.1.1.1.2.3.3.cmml" xref="A2.p3.7.m7.1.1.1.2.3.3">1</cn></apply></apply><apply id="A2.p3.7.m7.1.1.1.1.cmml" xref="A2.p3.7.m7.1.1.1.1"><times id="A2.p3.7.m7.1.1.1.1.2.cmml" xref="A2.p3.7.m7.1.1.1.1.2"></times><apply id="A2.p3.7.m7.1.1.1.1.3.cmml" xref="A2.p3.7.m7.1.1.1.1.3"><csymbol cd="ambiguous" id="A2.p3.7.m7.1.1.1.1.3.1.cmml" xref="A2.p3.7.m7.1.1.1.1.3">subscript</csymbol><ci id="A2.p3.7.m7.1.1.1.1.3.2.cmml" xref="A2.p3.7.m7.1.1.1.1.3.2">𝑓</ci><apply id="A2.p3.7.m7.1.1.1.1.3.3.cmml" xref="A2.p3.7.m7.1.1.1.1.3.3"><times id="A2.p3.7.m7.1.1.1.1.3.3.1.cmml" xref="A2.p3.7.m7.1.1.1.1.3.3.1"></times><ci id="A2.p3.7.m7.1.1.1.1.3.3.2.cmml" xref="A2.p3.7.m7.1.1.1.1.3.3.2">𝑅</ci><ci id="A2.p3.7.m7.1.1.1.1.3.3.3.cmml" xref="A2.p3.7.m7.1.1.1.1.3.3.3">𝑇</ci></apply></apply><apply id="A2.p3.7.m7.1.1.1.1.1.1.1.cmml" xref="A2.p3.7.m7.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A2.p3.7.m7.1.1.1.1.1.1.1.1.cmml" xref="A2.p3.7.m7.1.1.1.1.1.1">subscript</csymbol><ci id="A2.p3.7.m7.1.1.1.1.1.1.1.2.cmml" xref="A2.p3.7.m7.1.1.1.1.1.1.1.2">𝑋</ci><ci id="A2.p3.7.m7.1.1.1.1.1.1.1.3.cmml" xref="A2.p3.7.m7.1.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p3.7.m7.1c">\frac{\partial}{\partial X_{l}}\sum_{i=l}^{L-1}f_{RT}(X_{i})</annotation></semantics></math> cannot be always -1 for all samples in a mini-batch.
This explains why the gradients of Transformer layers in Fig. <a href="#S2.F3" title="Figure 3 ‣ 2 Background and Related Work ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> become more balanced and do not vanish after identity mapping reordering. In contrast, the PostLN architecture has a series of layer normalization operations that constantly alter the signal that passes through the skip connection and impedes information propagation, causing both vanishing gradients and training instability.
Overall, PreLN results in several useful characteristics such as avoiding vanishing/exploding gradient, stable optimization, and performance gain.</p>
</div>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>PreLN From the View of Unrolled Iterative Refinement</h2>

<div id="A3.p1" class="ltx_para">
<p id="A3.p1.5" class="ltx_p">From a theoretical point of view <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib40" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">40</span></a>]</cite>, a noisy estimate for a representation by the first Transformer layer should, on average, be correct even though it might have high variance. The unrolled iterative refinement view says if we treat "identity mapping" (as in PreLN) as being an unbiased estimator for the target representation, then beyond the first layer, the subsequent Transformer layer outputs <math id="A3.p1.1.m1.1" class="ltx_Math" alttext="x_{i}^{n}" display="inline"><semantics id="A3.p1.1.m1.1a"><msubsup id="A3.p1.1.m1.1.1" xref="A3.p1.1.m1.1.1.cmml"><mi id="A3.p1.1.m1.1.1.2.2" xref="A3.p1.1.m1.1.1.2.2.cmml">x</mi><mi id="A3.p1.1.m1.1.1.2.3" xref="A3.p1.1.m1.1.1.2.3.cmml">i</mi><mi id="A3.p1.1.m1.1.1.3" xref="A3.p1.1.m1.1.1.3.cmml">n</mi></msubsup><annotation-xml encoding="MathML-Content" id="A3.p1.1.m1.1b"><apply id="A3.p1.1.m1.1.1.cmml" xref="A3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A3.p1.1.m1.1.1.1.cmml" xref="A3.p1.1.m1.1.1">superscript</csymbol><apply id="A3.p1.1.m1.1.1.2.cmml" xref="A3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A3.p1.1.m1.1.1.2.1.cmml" xref="A3.p1.1.m1.1.1">subscript</csymbol><ci id="A3.p1.1.m1.1.1.2.2.cmml" xref="A3.p1.1.m1.1.1.2.2">𝑥</ci><ci id="A3.p1.1.m1.1.1.2.3.cmml" xref="A3.p1.1.m1.1.1.2.3">𝑖</ci></apply><ci id="A3.p1.1.m1.1.1.3.cmml" xref="A3.p1.1.m1.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.1.m1.1c">x_{i}^{n}</annotation></semantics></math> (e.g., <math id="A3.p1.2.m2.1" class="ltx_Math" alttext="i\in{2...L}" display="inline"><semantics id="A3.p1.2.m2.1a"><mrow id="A3.p1.2.m2.1.1" xref="A3.p1.2.m2.1.1.cmml"><mi id="A3.p1.2.m2.1.1.2" xref="A3.p1.2.m2.1.1.2.cmml">i</mi><mo id="A3.p1.2.m2.1.1.1" xref="A3.p1.2.m2.1.1.1.cmml">∈</mo><mrow id="A3.p1.2.m2.1.1.3" xref="A3.p1.2.m2.1.1.3.cmml"><mn id="A3.p1.2.m2.1.1.3.2" xref="A3.p1.2.m2.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A3.p1.2.m2.1.1.3.1" xref="A3.p1.2.m2.1.1.3.1.cmml">​</mo><mi mathvariant="normal" id="A3.p1.2.m2.1.1.3.3" xref="A3.p1.2.m2.1.1.3.3.cmml">…</mi><mo lspace="0em" rspace="0em" id="A3.p1.2.m2.1.1.3.1a" xref="A3.p1.2.m2.1.1.3.1.cmml">​</mo><mi id="A3.p1.2.m2.1.1.3.4" xref="A3.p1.2.m2.1.1.3.4.cmml">L</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.p1.2.m2.1b"><apply id="A3.p1.2.m2.1.1.cmml" xref="A3.p1.2.m2.1.1"><in id="A3.p1.2.m2.1.1.1.cmml" xref="A3.p1.2.m2.1.1.1"></in><ci id="A3.p1.2.m2.1.1.2.cmml" xref="A3.p1.2.m2.1.1.2">𝑖</ci><apply id="A3.p1.2.m2.1.1.3.cmml" xref="A3.p1.2.m2.1.1.3"><times id="A3.p1.2.m2.1.1.3.1.cmml" xref="A3.p1.2.m2.1.1.3.1"></times><cn type="integer" id="A3.p1.2.m2.1.1.3.2.cmml" xref="A3.p1.2.m2.1.1.3.2">2</cn><ci id="A3.p1.2.m2.1.1.3.3.cmml" xref="A3.p1.2.m2.1.1.3.3">…</ci><ci id="A3.p1.2.m2.1.1.3.4.cmml" xref="A3.p1.2.m2.1.1.3.4">𝐿</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.2.m2.1c">i\in{2...L}</annotation></semantics></math>) are all estimators for the same latent representation <math id="A3.p1.3.m3.1" class="ltx_Math" alttext="H^{n}" display="inline"><semantics id="A3.p1.3.m3.1a"><msup id="A3.p1.3.m3.1.1" xref="A3.p1.3.m3.1.1.cmml"><mi id="A3.p1.3.m3.1.1.2" xref="A3.p1.3.m3.1.1.2.cmml">H</mi><mi id="A3.p1.3.m3.1.1.3" xref="A3.p1.3.m3.1.1.3.cmml">n</mi></msup><annotation-xml encoding="MathML-Content" id="A3.p1.3.m3.1b"><apply id="A3.p1.3.m3.1.1.cmml" xref="A3.p1.3.m3.1.1"><csymbol cd="ambiguous" id="A3.p1.3.m3.1.1.1.cmml" xref="A3.p1.3.m3.1.1">superscript</csymbol><ci id="A3.p1.3.m3.1.1.2.cmml" xref="A3.p1.3.m3.1.1.2">𝐻</ci><ci id="A3.p1.3.m3.1.1.3.cmml" xref="A3.p1.3.m3.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.3.m3.1c">H^{n}</annotation></semantics></math>, where <math id="A3.p1.4.m4.1" class="ltx_Math" alttext="H^{n}" display="inline"><semantics id="A3.p1.4.m4.1a"><msup id="A3.p1.4.m4.1.1" xref="A3.p1.4.m4.1.1.cmml"><mi id="A3.p1.4.m4.1.1.2" xref="A3.p1.4.m4.1.1.2.cmml">H</mi><mi id="A3.p1.4.m4.1.1.3" xref="A3.p1.4.m4.1.1.3.cmml">n</mi></msup><annotation-xml encoding="MathML-Content" id="A3.p1.4.m4.1b"><apply id="A3.p1.4.m4.1.1.cmml" xref="A3.p1.4.m4.1.1"><csymbol cd="ambiguous" id="A3.p1.4.m4.1.1.1.cmml" xref="A3.p1.4.m4.1.1">superscript</csymbol><ci id="A3.p1.4.m4.1.1.2.cmml" xref="A3.p1.4.m4.1.1.2">𝐻</ci><ci id="A3.p1.4.m4.1.1.3.cmml" xref="A3.p1.4.m4.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.4.m4.1c">H^{n}</annotation></semantics></math> refers to the (unknown) value towards which the <math id="A3.p1.5.m5.1" class="ltx_Math" alttext="n" display="inline"><semantics id="A3.p1.5.m5.1a"><mi id="A3.p1.5.m5.1.1" xref="A3.p1.5.m5.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="A3.p1.5.m5.1b"><ci id="A3.p1.5.m5.1.1.cmml" xref="A3.p1.5.m5.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.p1.5.m5.1c">n</annotation></semantics></math>-th representation is converging. The unbiased estimator condition can then be written as the expected difference between the estimator and the final representation:</p>
<table id="A3.E6" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A3.E6.m1.1" class="ltx_Math" alttext="\underset{x\in X}{\mathds{E}}[x_{i}^{n}-H^{n}]=0" display="block"><semantics id="A3.E6.m1.1a"><mrow id="A3.E6.m1.1.1" xref="A3.E6.m1.1.1.cmml"><mrow id="A3.E6.m1.1.1.1" xref="A3.E6.m1.1.1.1.cmml"><munder accentunder="true" id="A3.E6.m1.1.1.1.3" xref="A3.E6.m1.1.1.1.3.cmml"><mi id="A3.E6.m1.1.1.1.3.2" xref="A3.E6.m1.1.1.1.3.2.cmml">𝔼</mi><mrow id="A3.E6.m1.1.1.1.3.1" xref="A3.E6.m1.1.1.1.3.1.cmml"><mi id="A3.E6.m1.1.1.1.3.1.2" xref="A3.E6.m1.1.1.1.3.1.2.cmml">x</mi><mo id="A3.E6.m1.1.1.1.3.1.1" xref="A3.E6.m1.1.1.1.3.1.1.cmml">∈</mo><mi id="A3.E6.m1.1.1.1.3.1.3" xref="A3.E6.m1.1.1.1.3.1.3.cmml">X</mi></mrow></munder><mo lspace="0em" rspace="0em" id="A3.E6.m1.1.1.1.2" xref="A3.E6.m1.1.1.1.2.cmml">​</mo><mrow id="A3.E6.m1.1.1.1.1.1" xref="A3.E6.m1.1.1.1.1.2.cmml"><mo stretchy="false" id="A3.E6.m1.1.1.1.1.1.2" xref="A3.E6.m1.1.1.1.1.2.1.cmml">[</mo><mrow id="A3.E6.m1.1.1.1.1.1.1" xref="A3.E6.m1.1.1.1.1.1.1.cmml"><msubsup id="A3.E6.m1.1.1.1.1.1.1.2" xref="A3.E6.m1.1.1.1.1.1.1.2.cmml"><mi id="A3.E6.m1.1.1.1.1.1.1.2.2.2" xref="A3.E6.m1.1.1.1.1.1.1.2.2.2.cmml">x</mi><mi id="A3.E6.m1.1.1.1.1.1.1.2.2.3" xref="A3.E6.m1.1.1.1.1.1.1.2.2.3.cmml">i</mi><mi id="A3.E6.m1.1.1.1.1.1.1.2.3" xref="A3.E6.m1.1.1.1.1.1.1.2.3.cmml">n</mi></msubsup><mo id="A3.E6.m1.1.1.1.1.1.1.1" xref="A3.E6.m1.1.1.1.1.1.1.1.cmml">−</mo><msup id="A3.E6.m1.1.1.1.1.1.1.3" xref="A3.E6.m1.1.1.1.1.1.1.3.cmml"><mi id="A3.E6.m1.1.1.1.1.1.1.3.2" xref="A3.E6.m1.1.1.1.1.1.1.3.2.cmml">H</mi><mi id="A3.E6.m1.1.1.1.1.1.1.3.3" xref="A3.E6.m1.1.1.1.1.1.1.3.3.cmml">n</mi></msup></mrow><mo stretchy="false" id="A3.E6.m1.1.1.1.1.1.3" xref="A3.E6.m1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="A3.E6.m1.1.1.2" xref="A3.E6.m1.1.1.2.cmml">=</mo><mn id="A3.E6.m1.1.1.3" xref="A3.E6.m1.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A3.E6.m1.1b"><apply id="A3.E6.m1.1.1.cmml" xref="A3.E6.m1.1.1"><eq id="A3.E6.m1.1.1.2.cmml" xref="A3.E6.m1.1.1.2"></eq><apply id="A3.E6.m1.1.1.1.cmml" xref="A3.E6.m1.1.1.1"><times id="A3.E6.m1.1.1.1.2.cmml" xref="A3.E6.m1.1.1.1.2"></times><apply id="A3.E6.m1.1.1.1.3.cmml" xref="A3.E6.m1.1.1.1.3"><apply id="A3.E6.m1.1.1.1.3.1.cmml" xref="A3.E6.m1.1.1.1.3.1"><in id="A3.E6.m1.1.1.1.3.1.1.cmml" xref="A3.E6.m1.1.1.1.3.1.1"></in><ci id="A3.E6.m1.1.1.1.3.1.2.cmml" xref="A3.E6.m1.1.1.1.3.1.2">𝑥</ci><ci id="A3.E6.m1.1.1.1.3.1.3.cmml" xref="A3.E6.m1.1.1.1.3.1.3">𝑋</ci></apply><ci id="A3.E6.m1.1.1.1.3.2.cmml" xref="A3.E6.m1.1.1.1.3.2">𝔼</ci></apply><apply id="A3.E6.m1.1.1.1.1.2.cmml" xref="A3.E6.m1.1.1.1.1.1"><csymbol cd="latexml" id="A3.E6.m1.1.1.1.1.2.1.cmml" xref="A3.E6.m1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A3.E6.m1.1.1.1.1.1.1.cmml" xref="A3.E6.m1.1.1.1.1.1.1"><minus id="A3.E6.m1.1.1.1.1.1.1.1.cmml" xref="A3.E6.m1.1.1.1.1.1.1.1"></minus><apply id="A3.E6.m1.1.1.1.1.1.1.2.cmml" xref="A3.E6.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A3.E6.m1.1.1.1.1.1.1.2.1.cmml" xref="A3.E6.m1.1.1.1.1.1.1.2">superscript</csymbol><apply id="A3.E6.m1.1.1.1.1.1.1.2.2.cmml" xref="A3.E6.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A3.E6.m1.1.1.1.1.1.1.2.2.1.cmml" xref="A3.E6.m1.1.1.1.1.1.1.2">subscript</csymbol><ci id="A3.E6.m1.1.1.1.1.1.1.2.2.2.cmml" xref="A3.E6.m1.1.1.1.1.1.1.2.2.2">𝑥</ci><ci id="A3.E6.m1.1.1.1.1.1.1.2.2.3.cmml" xref="A3.E6.m1.1.1.1.1.1.1.2.2.3">𝑖</ci></apply><ci id="A3.E6.m1.1.1.1.1.1.1.2.3.cmml" xref="A3.E6.m1.1.1.1.1.1.1.2.3">𝑛</ci></apply><apply id="A3.E6.m1.1.1.1.1.1.1.3.cmml" xref="A3.E6.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A3.E6.m1.1.1.1.1.1.1.3.1.cmml" xref="A3.E6.m1.1.1.1.1.1.1.3">superscript</csymbol><ci id="A3.E6.m1.1.1.1.1.1.1.3.2.cmml" xref="A3.E6.m1.1.1.1.1.1.1.3.2">𝐻</ci><ci id="A3.E6.m1.1.1.1.1.1.1.3.3.cmml" xref="A3.E6.m1.1.1.1.1.1.1.3.3">𝑛</ci></apply></apply></apply></apply><cn type="integer" id="A3.E6.m1.1.1.3.cmml" xref="A3.E6.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.E6.m1.1c">\underset{x\in X}{\mathds{E}}[x_{i}^{n}-H^{n}]=0</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(6)</span></td>
</tr></tbody>
</table>
</div>
<div id="A3.p2" class="ltx_para">
<p id="A3.p2.1" class="ltx_p">With the PreLN equation, it follows that the expected difference between outputs of two consecutive layers is zero, because</p>
<table id="A3.E7" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A3.E7.m1.3" class="ltx_Math" alttext="{\mathds{E}}[x_{i}^{n}-H^{n}]-{\mathds{E}}[x_{i-1}^{n}-H^{n}]=0\Rightarrow{\mathds{E}}[x_{i}^{n}-x_{i-1}^{n}]=0" display="block"><semantics id="A3.E7.m1.3a"><mrow id="A3.E7.m1.3.3" xref="A3.E7.m1.3.3.cmml"><mrow id="A3.E7.m1.2.2.2" xref="A3.E7.m1.2.2.2.cmml"><mrow id="A3.E7.m1.1.1.1.1" xref="A3.E7.m1.1.1.1.1.cmml"><mi id="A3.E7.m1.1.1.1.1.3" xref="A3.E7.m1.1.1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A3.E7.m1.1.1.1.1.2" xref="A3.E7.m1.1.1.1.1.2.cmml">​</mo><mrow id="A3.E7.m1.1.1.1.1.1.1" xref="A3.E7.m1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A3.E7.m1.1.1.1.1.1.1.2" xref="A3.E7.m1.1.1.1.1.1.2.1.cmml">[</mo><mrow id="A3.E7.m1.1.1.1.1.1.1.1" xref="A3.E7.m1.1.1.1.1.1.1.1.cmml"><msubsup id="A3.E7.m1.1.1.1.1.1.1.1.2" xref="A3.E7.m1.1.1.1.1.1.1.1.2.cmml"><mi id="A3.E7.m1.1.1.1.1.1.1.1.2.2.2" xref="A3.E7.m1.1.1.1.1.1.1.1.2.2.2.cmml">x</mi><mi id="A3.E7.m1.1.1.1.1.1.1.1.2.2.3" xref="A3.E7.m1.1.1.1.1.1.1.1.2.2.3.cmml">i</mi><mi id="A3.E7.m1.1.1.1.1.1.1.1.2.3" xref="A3.E7.m1.1.1.1.1.1.1.1.2.3.cmml">n</mi></msubsup><mo id="A3.E7.m1.1.1.1.1.1.1.1.1" xref="A3.E7.m1.1.1.1.1.1.1.1.1.cmml">−</mo><msup id="A3.E7.m1.1.1.1.1.1.1.1.3" xref="A3.E7.m1.1.1.1.1.1.1.1.3.cmml"><mi id="A3.E7.m1.1.1.1.1.1.1.1.3.2" xref="A3.E7.m1.1.1.1.1.1.1.1.3.2.cmml">H</mi><mi id="A3.E7.m1.1.1.1.1.1.1.1.3.3" xref="A3.E7.m1.1.1.1.1.1.1.1.3.3.cmml">n</mi></msup></mrow><mo stretchy="false" id="A3.E7.m1.1.1.1.1.1.1.3" xref="A3.E7.m1.1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="A3.E7.m1.2.2.2.3" xref="A3.E7.m1.2.2.2.3.cmml">−</mo><mrow id="A3.E7.m1.2.2.2.2" xref="A3.E7.m1.2.2.2.2.cmml"><mi id="A3.E7.m1.2.2.2.2.3" xref="A3.E7.m1.2.2.2.2.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A3.E7.m1.2.2.2.2.2" xref="A3.E7.m1.2.2.2.2.2.cmml">​</mo><mrow id="A3.E7.m1.2.2.2.2.1.1" xref="A3.E7.m1.2.2.2.2.1.2.cmml"><mo stretchy="false" id="A3.E7.m1.2.2.2.2.1.1.2" xref="A3.E7.m1.2.2.2.2.1.2.1.cmml">[</mo><mrow id="A3.E7.m1.2.2.2.2.1.1.1" xref="A3.E7.m1.2.2.2.2.1.1.1.cmml"><msubsup id="A3.E7.m1.2.2.2.2.1.1.1.2" xref="A3.E7.m1.2.2.2.2.1.1.1.2.cmml"><mi id="A3.E7.m1.2.2.2.2.1.1.1.2.2.2" xref="A3.E7.m1.2.2.2.2.1.1.1.2.2.2.cmml">x</mi><mrow id="A3.E7.m1.2.2.2.2.1.1.1.2.2.3" xref="A3.E7.m1.2.2.2.2.1.1.1.2.2.3.cmml"><mi id="A3.E7.m1.2.2.2.2.1.1.1.2.2.3.2" xref="A3.E7.m1.2.2.2.2.1.1.1.2.2.3.2.cmml">i</mi><mo id="A3.E7.m1.2.2.2.2.1.1.1.2.2.3.1" xref="A3.E7.m1.2.2.2.2.1.1.1.2.2.3.1.cmml">−</mo><mn id="A3.E7.m1.2.2.2.2.1.1.1.2.2.3.3" xref="A3.E7.m1.2.2.2.2.1.1.1.2.2.3.3.cmml">1</mn></mrow><mi id="A3.E7.m1.2.2.2.2.1.1.1.2.3" xref="A3.E7.m1.2.2.2.2.1.1.1.2.3.cmml">n</mi></msubsup><mo id="A3.E7.m1.2.2.2.2.1.1.1.1" xref="A3.E7.m1.2.2.2.2.1.1.1.1.cmml">−</mo><msup id="A3.E7.m1.2.2.2.2.1.1.1.3" xref="A3.E7.m1.2.2.2.2.1.1.1.3.cmml"><mi id="A3.E7.m1.2.2.2.2.1.1.1.3.2" xref="A3.E7.m1.2.2.2.2.1.1.1.3.2.cmml">H</mi><mi id="A3.E7.m1.2.2.2.2.1.1.1.3.3" xref="A3.E7.m1.2.2.2.2.1.1.1.3.3.cmml">n</mi></msup></mrow><mo stretchy="false" id="A3.E7.m1.2.2.2.2.1.1.3" xref="A3.E7.m1.2.2.2.2.1.2.1.cmml">]</mo></mrow></mrow></mrow><mo id="A3.E7.m1.3.3.5" xref="A3.E7.m1.3.3.5.cmml">=</mo><mn id="A3.E7.m1.3.3.6" xref="A3.E7.m1.3.3.6.cmml">0</mn><mo stretchy="false" id="A3.E7.m1.3.3.7" xref="A3.E7.m1.3.3.7.cmml">⇒</mo><mrow id="A3.E7.m1.3.3.3" xref="A3.E7.m1.3.3.3.cmml"><mi id="A3.E7.m1.3.3.3.3" xref="A3.E7.m1.3.3.3.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A3.E7.m1.3.3.3.2" xref="A3.E7.m1.3.3.3.2.cmml">​</mo><mrow id="A3.E7.m1.3.3.3.1.1" xref="A3.E7.m1.3.3.3.1.2.cmml"><mo stretchy="false" id="A3.E7.m1.3.3.3.1.1.2" xref="A3.E7.m1.3.3.3.1.2.1.cmml">[</mo><mrow id="A3.E7.m1.3.3.3.1.1.1" xref="A3.E7.m1.3.3.3.1.1.1.cmml"><msubsup id="A3.E7.m1.3.3.3.1.1.1.2" xref="A3.E7.m1.3.3.3.1.1.1.2.cmml"><mi id="A3.E7.m1.3.3.3.1.1.1.2.2.2" xref="A3.E7.m1.3.3.3.1.1.1.2.2.2.cmml">x</mi><mi id="A3.E7.m1.3.3.3.1.1.1.2.2.3" xref="A3.E7.m1.3.3.3.1.1.1.2.2.3.cmml">i</mi><mi id="A3.E7.m1.3.3.3.1.1.1.2.3" xref="A3.E7.m1.3.3.3.1.1.1.2.3.cmml">n</mi></msubsup><mo id="A3.E7.m1.3.3.3.1.1.1.1" xref="A3.E7.m1.3.3.3.1.1.1.1.cmml">−</mo><msubsup id="A3.E7.m1.3.3.3.1.1.1.3" xref="A3.E7.m1.3.3.3.1.1.1.3.cmml"><mi id="A3.E7.m1.3.3.3.1.1.1.3.2.2" xref="A3.E7.m1.3.3.3.1.1.1.3.2.2.cmml">x</mi><mrow id="A3.E7.m1.3.3.3.1.1.1.3.2.3" xref="A3.E7.m1.3.3.3.1.1.1.3.2.3.cmml"><mi id="A3.E7.m1.3.3.3.1.1.1.3.2.3.2" xref="A3.E7.m1.3.3.3.1.1.1.3.2.3.2.cmml">i</mi><mo id="A3.E7.m1.3.3.3.1.1.1.3.2.3.1" xref="A3.E7.m1.3.3.3.1.1.1.3.2.3.1.cmml">−</mo><mn id="A3.E7.m1.3.3.3.1.1.1.3.2.3.3" xref="A3.E7.m1.3.3.3.1.1.1.3.2.3.3.cmml">1</mn></mrow><mi id="A3.E7.m1.3.3.3.1.1.1.3.3" xref="A3.E7.m1.3.3.3.1.1.1.3.3.cmml">n</mi></msubsup></mrow><mo stretchy="false" id="A3.E7.m1.3.3.3.1.1.3" xref="A3.E7.m1.3.3.3.1.2.1.cmml">]</mo></mrow></mrow><mo id="A3.E7.m1.3.3.8" xref="A3.E7.m1.3.3.8.cmml">=</mo><mn id="A3.E7.m1.3.3.9" xref="A3.E7.m1.3.3.9.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A3.E7.m1.3b"><apply id="A3.E7.m1.3.3.cmml" xref="A3.E7.m1.3.3"><and id="A3.E7.m1.3.3a.cmml" xref="A3.E7.m1.3.3"></and><apply id="A3.E7.m1.3.3b.cmml" xref="A3.E7.m1.3.3"><eq id="A3.E7.m1.3.3.5.cmml" xref="A3.E7.m1.3.3.5"></eq><apply id="A3.E7.m1.2.2.2.cmml" xref="A3.E7.m1.2.2.2"><minus id="A3.E7.m1.2.2.2.3.cmml" xref="A3.E7.m1.2.2.2.3"></minus><apply id="A3.E7.m1.1.1.1.1.cmml" xref="A3.E7.m1.1.1.1.1"><times id="A3.E7.m1.1.1.1.1.2.cmml" xref="A3.E7.m1.1.1.1.1.2"></times><ci id="A3.E7.m1.1.1.1.1.3.cmml" xref="A3.E7.m1.1.1.1.1.3">𝔼</ci><apply id="A3.E7.m1.1.1.1.1.1.2.cmml" xref="A3.E7.m1.1.1.1.1.1.1"><csymbol cd="latexml" id="A3.E7.m1.1.1.1.1.1.2.1.cmml" xref="A3.E7.m1.1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A3.E7.m1.1.1.1.1.1.1.1.cmml" xref="A3.E7.m1.1.1.1.1.1.1.1"><minus id="A3.E7.m1.1.1.1.1.1.1.1.1.cmml" xref="A3.E7.m1.1.1.1.1.1.1.1.1"></minus><apply id="A3.E7.m1.1.1.1.1.1.1.1.2.cmml" xref="A3.E7.m1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A3.E7.m1.1.1.1.1.1.1.1.2.1.cmml" xref="A3.E7.m1.1.1.1.1.1.1.1.2">superscript</csymbol><apply id="A3.E7.m1.1.1.1.1.1.1.1.2.2.cmml" xref="A3.E7.m1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A3.E7.m1.1.1.1.1.1.1.1.2.2.1.cmml" xref="A3.E7.m1.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="A3.E7.m1.1.1.1.1.1.1.1.2.2.2.cmml" xref="A3.E7.m1.1.1.1.1.1.1.1.2.2.2">𝑥</ci><ci id="A3.E7.m1.1.1.1.1.1.1.1.2.2.3.cmml" xref="A3.E7.m1.1.1.1.1.1.1.1.2.2.3">𝑖</ci></apply><ci id="A3.E7.m1.1.1.1.1.1.1.1.2.3.cmml" xref="A3.E7.m1.1.1.1.1.1.1.1.2.3">𝑛</ci></apply><apply id="A3.E7.m1.1.1.1.1.1.1.1.3.cmml" xref="A3.E7.m1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A3.E7.m1.1.1.1.1.1.1.1.3.1.cmml" xref="A3.E7.m1.1.1.1.1.1.1.1.3">superscript</csymbol><ci id="A3.E7.m1.1.1.1.1.1.1.1.3.2.cmml" xref="A3.E7.m1.1.1.1.1.1.1.1.3.2">𝐻</ci><ci id="A3.E7.m1.1.1.1.1.1.1.1.3.3.cmml" xref="A3.E7.m1.1.1.1.1.1.1.1.3.3">𝑛</ci></apply></apply></apply></apply><apply id="A3.E7.m1.2.2.2.2.cmml" xref="A3.E7.m1.2.2.2.2"><times id="A3.E7.m1.2.2.2.2.2.cmml" xref="A3.E7.m1.2.2.2.2.2"></times><ci id="A3.E7.m1.2.2.2.2.3.cmml" xref="A3.E7.m1.2.2.2.2.3">𝔼</ci><apply id="A3.E7.m1.2.2.2.2.1.2.cmml" xref="A3.E7.m1.2.2.2.2.1.1"><csymbol cd="latexml" id="A3.E7.m1.2.2.2.2.1.2.1.cmml" xref="A3.E7.m1.2.2.2.2.1.1.2">delimited-[]</csymbol><apply id="A3.E7.m1.2.2.2.2.1.1.1.cmml" xref="A3.E7.m1.2.2.2.2.1.1.1"><minus id="A3.E7.m1.2.2.2.2.1.1.1.1.cmml" xref="A3.E7.m1.2.2.2.2.1.1.1.1"></minus><apply id="A3.E7.m1.2.2.2.2.1.1.1.2.cmml" xref="A3.E7.m1.2.2.2.2.1.1.1.2"><csymbol cd="ambiguous" id="A3.E7.m1.2.2.2.2.1.1.1.2.1.cmml" xref="A3.E7.m1.2.2.2.2.1.1.1.2">superscript</csymbol><apply id="A3.E7.m1.2.2.2.2.1.1.1.2.2.cmml" xref="A3.E7.m1.2.2.2.2.1.1.1.2"><csymbol cd="ambiguous" id="A3.E7.m1.2.2.2.2.1.1.1.2.2.1.cmml" xref="A3.E7.m1.2.2.2.2.1.1.1.2">subscript</csymbol><ci id="A3.E7.m1.2.2.2.2.1.1.1.2.2.2.cmml" xref="A3.E7.m1.2.2.2.2.1.1.1.2.2.2">𝑥</ci><apply id="A3.E7.m1.2.2.2.2.1.1.1.2.2.3.cmml" xref="A3.E7.m1.2.2.2.2.1.1.1.2.2.3"><minus id="A3.E7.m1.2.2.2.2.1.1.1.2.2.3.1.cmml" xref="A3.E7.m1.2.2.2.2.1.1.1.2.2.3.1"></minus><ci id="A3.E7.m1.2.2.2.2.1.1.1.2.2.3.2.cmml" xref="A3.E7.m1.2.2.2.2.1.1.1.2.2.3.2">𝑖</ci><cn type="integer" id="A3.E7.m1.2.2.2.2.1.1.1.2.2.3.3.cmml" xref="A3.E7.m1.2.2.2.2.1.1.1.2.2.3.3">1</cn></apply></apply><ci id="A3.E7.m1.2.2.2.2.1.1.1.2.3.cmml" xref="A3.E7.m1.2.2.2.2.1.1.1.2.3">𝑛</ci></apply><apply id="A3.E7.m1.2.2.2.2.1.1.1.3.cmml" xref="A3.E7.m1.2.2.2.2.1.1.1.3"><csymbol cd="ambiguous" id="A3.E7.m1.2.2.2.2.1.1.1.3.1.cmml" xref="A3.E7.m1.2.2.2.2.1.1.1.3">superscript</csymbol><ci id="A3.E7.m1.2.2.2.2.1.1.1.3.2.cmml" xref="A3.E7.m1.2.2.2.2.1.1.1.3.2">𝐻</ci><ci id="A3.E7.m1.2.2.2.2.1.1.1.3.3.cmml" xref="A3.E7.m1.2.2.2.2.1.1.1.3.3">𝑛</ci></apply></apply></apply></apply></apply><cn type="integer" id="A3.E7.m1.3.3.6.cmml" xref="A3.E7.m1.3.3.6">0</cn></apply><apply id="A3.E7.m1.3.3c.cmml" xref="A3.E7.m1.3.3"><ci id="A3.E7.m1.3.3.7.cmml" xref="A3.E7.m1.3.3.7">⇒</ci><share href="#A3.E7.m1.3.3.6.cmml" id="A3.E7.m1.3.3d.cmml" xref="A3.E7.m1.3.3"></share><apply id="A3.E7.m1.3.3.3.cmml" xref="A3.E7.m1.3.3.3"><times id="A3.E7.m1.3.3.3.2.cmml" xref="A3.E7.m1.3.3.3.2"></times><ci id="A3.E7.m1.3.3.3.3.cmml" xref="A3.E7.m1.3.3.3.3">𝔼</ci><apply id="A3.E7.m1.3.3.3.1.2.cmml" xref="A3.E7.m1.3.3.3.1.1"><csymbol cd="latexml" id="A3.E7.m1.3.3.3.1.2.1.cmml" xref="A3.E7.m1.3.3.3.1.1.2">delimited-[]</csymbol><apply id="A3.E7.m1.3.3.3.1.1.1.cmml" xref="A3.E7.m1.3.3.3.1.1.1"><minus id="A3.E7.m1.3.3.3.1.1.1.1.cmml" xref="A3.E7.m1.3.3.3.1.1.1.1"></minus><apply id="A3.E7.m1.3.3.3.1.1.1.2.cmml" xref="A3.E7.m1.3.3.3.1.1.1.2"><csymbol cd="ambiguous" id="A3.E7.m1.3.3.3.1.1.1.2.1.cmml" xref="A3.E7.m1.3.3.3.1.1.1.2">superscript</csymbol><apply id="A3.E7.m1.3.3.3.1.1.1.2.2.cmml" xref="A3.E7.m1.3.3.3.1.1.1.2"><csymbol cd="ambiguous" id="A3.E7.m1.3.3.3.1.1.1.2.2.1.cmml" xref="A3.E7.m1.3.3.3.1.1.1.2">subscript</csymbol><ci id="A3.E7.m1.3.3.3.1.1.1.2.2.2.cmml" xref="A3.E7.m1.3.3.3.1.1.1.2.2.2">𝑥</ci><ci id="A3.E7.m1.3.3.3.1.1.1.2.2.3.cmml" xref="A3.E7.m1.3.3.3.1.1.1.2.2.3">𝑖</ci></apply><ci id="A3.E7.m1.3.3.3.1.1.1.2.3.cmml" xref="A3.E7.m1.3.3.3.1.1.1.2.3">𝑛</ci></apply><apply id="A3.E7.m1.3.3.3.1.1.1.3.cmml" xref="A3.E7.m1.3.3.3.1.1.1.3"><csymbol cd="ambiguous" id="A3.E7.m1.3.3.3.1.1.1.3.1.cmml" xref="A3.E7.m1.3.3.3.1.1.1.3">superscript</csymbol><apply id="A3.E7.m1.3.3.3.1.1.1.3.2.cmml" xref="A3.E7.m1.3.3.3.1.1.1.3"><csymbol cd="ambiguous" id="A3.E7.m1.3.3.3.1.1.1.3.2.1.cmml" xref="A3.E7.m1.3.3.3.1.1.1.3">subscript</csymbol><ci id="A3.E7.m1.3.3.3.1.1.1.3.2.2.cmml" xref="A3.E7.m1.3.3.3.1.1.1.3.2.2">𝑥</ci><apply id="A3.E7.m1.3.3.3.1.1.1.3.2.3.cmml" xref="A3.E7.m1.3.3.3.1.1.1.3.2.3"><minus id="A3.E7.m1.3.3.3.1.1.1.3.2.3.1.cmml" xref="A3.E7.m1.3.3.3.1.1.1.3.2.3.1"></minus><ci id="A3.E7.m1.3.3.3.1.1.1.3.2.3.2.cmml" xref="A3.E7.m1.3.3.3.1.1.1.3.2.3.2">𝑖</ci><cn type="integer" id="A3.E7.m1.3.3.3.1.1.1.3.2.3.3.cmml" xref="A3.E7.m1.3.3.3.1.1.1.3.2.3.3">1</cn></apply></apply><ci id="A3.E7.m1.3.3.3.1.1.1.3.3.cmml" xref="A3.E7.m1.3.3.3.1.1.1.3.3">𝑛</ci></apply></apply></apply></apply></apply><apply id="A3.E7.m1.3.3e.cmml" xref="A3.E7.m1.3.3"><eq id="A3.E7.m1.3.3.8.cmml" xref="A3.E7.m1.3.3.8"></eq><share href="#A3.E7.m1.3.3.3.cmml" id="A3.E7.m1.3.3f.cmml" xref="A3.E7.m1.3.3"></share><cn type="integer" id="A3.E7.m1.3.3.9.cmml" xref="A3.E7.m1.3.3.9">0</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.E7.m1.3c">{\mathds{E}}[x_{i}^{n}-H^{n}]-{\mathds{E}}[x_{i-1}^{n}-H^{n}]=0\Rightarrow{\mathds{E}}[x_{i}^{n}-x_{i-1}^{n}]=0</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(7)</span></td>
</tr></tbody>
</table>
</div>
<div id="A3.p3" class="ltx_para">
<p id="A3.p3.3" class="ltx_p">If we write representation <math id="A3.p3.1.m1.1" class="ltx_Math" alttext="x_{i}^{n}" display="inline"><semantics id="A3.p3.1.m1.1a"><msubsup id="A3.p3.1.m1.1.1" xref="A3.p3.1.m1.1.1.cmml"><mi id="A3.p3.1.m1.1.1.2.2" xref="A3.p3.1.m1.1.1.2.2.cmml">x</mi><mi id="A3.p3.1.m1.1.1.2.3" xref="A3.p3.1.m1.1.1.2.3.cmml">i</mi><mi id="A3.p3.1.m1.1.1.3" xref="A3.p3.1.m1.1.1.3.cmml">n</mi></msubsup><annotation-xml encoding="MathML-Content" id="A3.p3.1.m1.1b"><apply id="A3.p3.1.m1.1.1.cmml" xref="A3.p3.1.m1.1.1"><csymbol cd="ambiguous" id="A3.p3.1.m1.1.1.1.cmml" xref="A3.p3.1.m1.1.1">superscript</csymbol><apply id="A3.p3.1.m1.1.1.2.cmml" xref="A3.p3.1.m1.1.1"><csymbol cd="ambiguous" id="A3.p3.1.m1.1.1.2.1.cmml" xref="A3.p3.1.m1.1.1">subscript</csymbol><ci id="A3.p3.1.m1.1.1.2.2.cmml" xref="A3.p3.1.m1.1.1.2.2">𝑥</ci><ci id="A3.p3.1.m1.1.1.2.3.cmml" xref="A3.p3.1.m1.1.1.2.3">𝑖</ci></apply><ci id="A3.p3.1.m1.1.1.3.cmml" xref="A3.p3.1.m1.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p3.1.m1.1c">x_{i}^{n}</annotation></semantics></math> as a combination of <math id="A3.p3.2.m2.1" class="ltx_Math" alttext="x_{i-1^{n}}" display="inline"><semantics id="A3.p3.2.m2.1a"><msub id="A3.p3.2.m2.1.1" xref="A3.p3.2.m2.1.1.cmml"><mi id="A3.p3.2.m2.1.1.2" xref="A3.p3.2.m2.1.1.2.cmml">x</mi><mrow id="A3.p3.2.m2.1.1.3" xref="A3.p3.2.m2.1.1.3.cmml"><mi id="A3.p3.2.m2.1.1.3.2" xref="A3.p3.2.m2.1.1.3.2.cmml">i</mi><mo id="A3.p3.2.m2.1.1.3.1" xref="A3.p3.2.m2.1.1.3.1.cmml">−</mo><msup id="A3.p3.2.m2.1.1.3.3" xref="A3.p3.2.m2.1.1.3.3.cmml"><mn id="A3.p3.2.m2.1.1.3.3.2" xref="A3.p3.2.m2.1.1.3.3.2.cmml">1</mn><mi id="A3.p3.2.m2.1.1.3.3.3" xref="A3.p3.2.m2.1.1.3.3.3.cmml">n</mi></msup></mrow></msub><annotation-xml encoding="MathML-Content" id="A3.p3.2.m2.1b"><apply id="A3.p3.2.m2.1.1.cmml" xref="A3.p3.2.m2.1.1"><csymbol cd="ambiguous" id="A3.p3.2.m2.1.1.1.cmml" xref="A3.p3.2.m2.1.1">subscript</csymbol><ci id="A3.p3.2.m2.1.1.2.cmml" xref="A3.p3.2.m2.1.1.2">𝑥</ci><apply id="A3.p3.2.m2.1.1.3.cmml" xref="A3.p3.2.m2.1.1.3"><minus id="A3.p3.2.m2.1.1.3.1.cmml" xref="A3.p3.2.m2.1.1.3.1"></minus><ci id="A3.p3.2.m2.1.1.3.2.cmml" xref="A3.p3.2.m2.1.1.3.2">𝑖</ci><apply id="A3.p3.2.m2.1.1.3.3.cmml" xref="A3.p3.2.m2.1.1.3.3"><csymbol cd="ambiguous" id="A3.p3.2.m2.1.1.3.3.1.cmml" xref="A3.p3.2.m2.1.1.3.3">superscript</csymbol><cn type="integer" id="A3.p3.2.m2.1.1.3.3.2.cmml" xref="A3.p3.2.m2.1.1.3.3.2">1</cn><ci id="A3.p3.2.m2.1.1.3.3.3.cmml" xref="A3.p3.2.m2.1.1.3.3.3">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p3.2.m2.1c">x_{i-1^{n}}</annotation></semantics></math> and a residual <math id="A3.p3.3.m3.1" class="ltx_Math" alttext="{f_{RT}}^{n}" display="inline"><semantics id="A3.p3.3.m3.1a"><mmultiscripts id="A3.p3.3.m3.1.1" xref="A3.p3.3.m3.1.1.cmml"><mi id="A3.p3.3.m3.1.1.2.2" xref="A3.p3.3.m3.1.1.2.2.cmml">f</mi><mrow id="A3.p3.3.m3.1.1.2.3" xref="A3.p3.3.m3.1.1.2.3.cmml"><mi id="A3.p3.3.m3.1.1.2.3.2" xref="A3.p3.3.m3.1.1.2.3.2.cmml">R</mi><mo lspace="0em" rspace="0em" id="A3.p3.3.m3.1.1.2.3.1" xref="A3.p3.3.m3.1.1.2.3.1.cmml">​</mo><mi id="A3.p3.3.m3.1.1.2.3.3" xref="A3.p3.3.m3.1.1.2.3.3.cmml">T</mi></mrow><mrow id="A3.p3.3.m3.1.1a" xref="A3.p3.3.m3.1.1.cmml"></mrow><mrow id="A3.p3.3.m3.1.1b" xref="A3.p3.3.m3.1.1.cmml"></mrow><mi id="A3.p3.3.m3.1.1.3" xref="A3.p3.3.m3.1.1.3.cmml">n</mi></mmultiscripts><annotation-xml encoding="MathML-Content" id="A3.p3.3.m3.1b"><apply id="A3.p3.3.m3.1.1.cmml" xref="A3.p3.3.m3.1.1"><csymbol cd="ambiguous" id="A3.p3.3.m3.1.1.1.cmml" xref="A3.p3.3.m3.1.1">superscript</csymbol><apply id="A3.p3.3.m3.1.1.2.cmml" xref="A3.p3.3.m3.1.1"><csymbol cd="ambiguous" id="A3.p3.3.m3.1.1.2.1.cmml" xref="A3.p3.3.m3.1.1">subscript</csymbol><ci id="A3.p3.3.m3.1.1.2.2.cmml" xref="A3.p3.3.m3.1.1.2.2">𝑓</ci><apply id="A3.p3.3.m3.1.1.2.3.cmml" xref="A3.p3.3.m3.1.1.2.3"><times id="A3.p3.3.m3.1.1.2.3.1.cmml" xref="A3.p3.3.m3.1.1.2.3.1"></times><ci id="A3.p3.3.m3.1.1.2.3.2.cmml" xref="A3.p3.3.m3.1.1.2.3.2">𝑅</ci><ci id="A3.p3.3.m3.1.1.2.3.3.cmml" xref="A3.p3.3.m3.1.1.2.3.3">𝑇</ci></apply></apply><ci id="A3.p3.3.m3.1.1.3.cmml" xref="A3.p3.3.m3.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p3.3.m3.1c">{f_{RT}}^{n}</annotation></semantics></math>, it follows from the above equation that the residual has to be zero-mean:</p>
<table id="A3.E8" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A3.E8.m1.1" class="ltx_Math" alttext="x_{i}^{n}=x_{i-1}^{n}+{f_{RT}}^{n}\Rightarrow{\mathds{E}}[{f_{RT}}^{n}]=0" display="block"><semantics id="A3.E8.m1.1a"><mrow id="A3.E8.m1.1.1" xref="A3.E8.m1.1.1.cmml"><msubsup id="A3.E8.m1.1.1.3" xref="A3.E8.m1.1.1.3.cmml"><mi id="A3.E8.m1.1.1.3.2.2" xref="A3.E8.m1.1.1.3.2.2.cmml">x</mi><mi id="A3.E8.m1.1.1.3.2.3" xref="A3.E8.m1.1.1.3.2.3.cmml">i</mi><mi id="A3.E8.m1.1.1.3.3" xref="A3.E8.m1.1.1.3.3.cmml">n</mi></msubsup><mo id="A3.E8.m1.1.1.4" xref="A3.E8.m1.1.1.4.cmml">=</mo><mrow id="A3.E8.m1.1.1.5" xref="A3.E8.m1.1.1.5.cmml"><msubsup id="A3.E8.m1.1.1.5.2" xref="A3.E8.m1.1.1.5.2.cmml"><mi id="A3.E8.m1.1.1.5.2.2.2" xref="A3.E8.m1.1.1.5.2.2.2.cmml">x</mi><mrow id="A3.E8.m1.1.1.5.2.2.3" xref="A3.E8.m1.1.1.5.2.2.3.cmml"><mi id="A3.E8.m1.1.1.5.2.2.3.2" xref="A3.E8.m1.1.1.5.2.2.3.2.cmml">i</mi><mo id="A3.E8.m1.1.1.5.2.2.3.1" xref="A3.E8.m1.1.1.5.2.2.3.1.cmml">−</mo><mn id="A3.E8.m1.1.1.5.2.2.3.3" xref="A3.E8.m1.1.1.5.2.2.3.3.cmml">1</mn></mrow><mi id="A3.E8.m1.1.1.5.2.3" xref="A3.E8.m1.1.1.5.2.3.cmml">n</mi></msubsup><mo id="A3.E8.m1.1.1.5.1" xref="A3.E8.m1.1.1.5.1.cmml">+</mo><mmultiscripts id="A3.E8.m1.1.1.5.3" xref="A3.E8.m1.1.1.5.3.cmml"><mi id="A3.E8.m1.1.1.5.3.2.2" xref="A3.E8.m1.1.1.5.3.2.2.cmml">f</mi><mrow id="A3.E8.m1.1.1.5.3.2.3" xref="A3.E8.m1.1.1.5.3.2.3.cmml"><mi id="A3.E8.m1.1.1.5.3.2.3.2" xref="A3.E8.m1.1.1.5.3.2.3.2.cmml">R</mi><mo lspace="0em" rspace="0em" id="A3.E8.m1.1.1.5.3.2.3.1" xref="A3.E8.m1.1.1.5.3.2.3.1.cmml">​</mo><mi id="A3.E8.m1.1.1.5.3.2.3.3" xref="A3.E8.m1.1.1.5.3.2.3.3.cmml">T</mi></mrow><mrow id="A3.E8.m1.1.1.5.3a" xref="A3.E8.m1.1.1.5.3.cmml"></mrow><mrow id="A3.E8.m1.1.1.5.3b" xref="A3.E8.m1.1.1.5.3.cmml"></mrow><mi id="A3.E8.m1.1.1.5.3.3" xref="A3.E8.m1.1.1.5.3.3.cmml">n</mi></mmultiscripts></mrow><mo stretchy="false" id="A3.E8.m1.1.1.6" xref="A3.E8.m1.1.1.6.cmml">⇒</mo><mrow id="A3.E8.m1.1.1.1" xref="A3.E8.m1.1.1.1.cmml"><mi id="A3.E8.m1.1.1.1.3" xref="A3.E8.m1.1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A3.E8.m1.1.1.1.2" xref="A3.E8.m1.1.1.1.2.cmml">​</mo><mrow id="A3.E8.m1.1.1.1.1.1" xref="A3.E8.m1.1.1.1.1.2.cmml"><mo stretchy="false" id="A3.E8.m1.1.1.1.1.1.2" xref="A3.E8.m1.1.1.1.1.2.1.cmml">[</mo><mmultiscripts id="A3.E8.m1.1.1.1.1.1.1" xref="A3.E8.m1.1.1.1.1.1.1.cmml"><mi id="A3.E8.m1.1.1.1.1.1.1.2.2" xref="A3.E8.m1.1.1.1.1.1.1.2.2.cmml">f</mi><mrow id="A3.E8.m1.1.1.1.1.1.1.2.3" xref="A3.E8.m1.1.1.1.1.1.1.2.3.cmml"><mi id="A3.E8.m1.1.1.1.1.1.1.2.3.2" xref="A3.E8.m1.1.1.1.1.1.1.2.3.2.cmml">R</mi><mo lspace="0em" rspace="0em" id="A3.E8.m1.1.1.1.1.1.1.2.3.1" xref="A3.E8.m1.1.1.1.1.1.1.2.3.1.cmml">​</mo><mi id="A3.E8.m1.1.1.1.1.1.1.2.3.3" xref="A3.E8.m1.1.1.1.1.1.1.2.3.3.cmml">T</mi></mrow><mrow id="A3.E8.m1.1.1.1.1.1.1a" xref="A3.E8.m1.1.1.1.1.1.1.cmml"></mrow><mrow id="A3.E8.m1.1.1.1.1.1.1b" xref="A3.E8.m1.1.1.1.1.1.1.cmml"></mrow><mi id="A3.E8.m1.1.1.1.1.1.1.3" xref="A3.E8.m1.1.1.1.1.1.1.3.cmml">n</mi></mmultiscripts><mo stretchy="false" id="A3.E8.m1.1.1.1.1.1.3" xref="A3.E8.m1.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo id="A3.E8.m1.1.1.7" xref="A3.E8.m1.1.1.7.cmml">=</mo><mn id="A3.E8.m1.1.1.8" xref="A3.E8.m1.1.1.8.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A3.E8.m1.1b"><apply id="A3.E8.m1.1.1.cmml" xref="A3.E8.m1.1.1"><and id="A3.E8.m1.1.1a.cmml" xref="A3.E8.m1.1.1"></and><apply id="A3.E8.m1.1.1b.cmml" xref="A3.E8.m1.1.1"><eq id="A3.E8.m1.1.1.4.cmml" xref="A3.E8.m1.1.1.4"></eq><apply id="A3.E8.m1.1.1.3.cmml" xref="A3.E8.m1.1.1.3"><csymbol cd="ambiguous" id="A3.E8.m1.1.1.3.1.cmml" xref="A3.E8.m1.1.1.3">superscript</csymbol><apply id="A3.E8.m1.1.1.3.2.cmml" xref="A3.E8.m1.1.1.3"><csymbol cd="ambiguous" id="A3.E8.m1.1.1.3.2.1.cmml" xref="A3.E8.m1.1.1.3">subscript</csymbol><ci id="A3.E8.m1.1.1.3.2.2.cmml" xref="A3.E8.m1.1.1.3.2.2">𝑥</ci><ci id="A3.E8.m1.1.1.3.2.3.cmml" xref="A3.E8.m1.1.1.3.2.3">𝑖</ci></apply><ci id="A3.E8.m1.1.1.3.3.cmml" xref="A3.E8.m1.1.1.3.3">𝑛</ci></apply><apply id="A3.E8.m1.1.1.5.cmml" xref="A3.E8.m1.1.1.5"><plus id="A3.E8.m1.1.1.5.1.cmml" xref="A3.E8.m1.1.1.5.1"></plus><apply id="A3.E8.m1.1.1.5.2.cmml" xref="A3.E8.m1.1.1.5.2"><csymbol cd="ambiguous" id="A3.E8.m1.1.1.5.2.1.cmml" xref="A3.E8.m1.1.1.5.2">superscript</csymbol><apply id="A3.E8.m1.1.1.5.2.2.cmml" xref="A3.E8.m1.1.1.5.2"><csymbol cd="ambiguous" id="A3.E8.m1.1.1.5.2.2.1.cmml" xref="A3.E8.m1.1.1.5.2">subscript</csymbol><ci id="A3.E8.m1.1.1.5.2.2.2.cmml" xref="A3.E8.m1.1.1.5.2.2.2">𝑥</ci><apply id="A3.E8.m1.1.1.5.2.2.3.cmml" xref="A3.E8.m1.1.1.5.2.2.3"><minus id="A3.E8.m1.1.1.5.2.2.3.1.cmml" xref="A3.E8.m1.1.1.5.2.2.3.1"></minus><ci id="A3.E8.m1.1.1.5.2.2.3.2.cmml" xref="A3.E8.m1.1.1.5.2.2.3.2">𝑖</ci><cn type="integer" id="A3.E8.m1.1.1.5.2.2.3.3.cmml" xref="A3.E8.m1.1.1.5.2.2.3.3">1</cn></apply></apply><ci id="A3.E8.m1.1.1.5.2.3.cmml" xref="A3.E8.m1.1.1.5.2.3">𝑛</ci></apply><apply id="A3.E8.m1.1.1.5.3.cmml" xref="A3.E8.m1.1.1.5.3"><csymbol cd="ambiguous" id="A3.E8.m1.1.1.5.3.1.cmml" xref="A3.E8.m1.1.1.5.3">superscript</csymbol><apply id="A3.E8.m1.1.1.5.3.2.cmml" xref="A3.E8.m1.1.1.5.3"><csymbol cd="ambiguous" id="A3.E8.m1.1.1.5.3.2.1.cmml" xref="A3.E8.m1.1.1.5.3">subscript</csymbol><ci id="A3.E8.m1.1.1.5.3.2.2.cmml" xref="A3.E8.m1.1.1.5.3.2.2">𝑓</ci><apply id="A3.E8.m1.1.1.5.3.2.3.cmml" xref="A3.E8.m1.1.1.5.3.2.3"><times id="A3.E8.m1.1.1.5.3.2.3.1.cmml" xref="A3.E8.m1.1.1.5.3.2.3.1"></times><ci id="A3.E8.m1.1.1.5.3.2.3.2.cmml" xref="A3.E8.m1.1.1.5.3.2.3.2">𝑅</ci><ci id="A3.E8.m1.1.1.5.3.2.3.3.cmml" xref="A3.E8.m1.1.1.5.3.2.3.3">𝑇</ci></apply></apply><ci id="A3.E8.m1.1.1.5.3.3.cmml" xref="A3.E8.m1.1.1.5.3.3">𝑛</ci></apply></apply></apply><apply id="A3.E8.m1.1.1c.cmml" xref="A3.E8.m1.1.1"><ci id="A3.E8.m1.1.1.6.cmml" xref="A3.E8.m1.1.1.6">⇒</ci><share href="#A3.E8.m1.1.1.5.cmml" id="A3.E8.m1.1.1d.cmml" xref="A3.E8.m1.1.1"></share><apply id="A3.E8.m1.1.1.1.cmml" xref="A3.E8.m1.1.1.1"><times id="A3.E8.m1.1.1.1.2.cmml" xref="A3.E8.m1.1.1.1.2"></times><ci id="A3.E8.m1.1.1.1.3.cmml" xref="A3.E8.m1.1.1.1.3">𝔼</ci><apply id="A3.E8.m1.1.1.1.1.2.cmml" xref="A3.E8.m1.1.1.1.1.1"><csymbol cd="latexml" id="A3.E8.m1.1.1.1.1.2.1.cmml" xref="A3.E8.m1.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A3.E8.m1.1.1.1.1.1.1.cmml" xref="A3.E8.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A3.E8.m1.1.1.1.1.1.1.1.cmml" xref="A3.E8.m1.1.1.1.1.1.1">superscript</csymbol><apply id="A3.E8.m1.1.1.1.1.1.1.2.cmml" xref="A3.E8.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A3.E8.m1.1.1.1.1.1.1.2.1.cmml" xref="A3.E8.m1.1.1.1.1.1.1">subscript</csymbol><ci id="A3.E8.m1.1.1.1.1.1.1.2.2.cmml" xref="A3.E8.m1.1.1.1.1.1.1.2.2">𝑓</ci><apply id="A3.E8.m1.1.1.1.1.1.1.2.3.cmml" xref="A3.E8.m1.1.1.1.1.1.1.2.3"><times id="A3.E8.m1.1.1.1.1.1.1.2.3.1.cmml" xref="A3.E8.m1.1.1.1.1.1.1.2.3.1"></times><ci id="A3.E8.m1.1.1.1.1.1.1.2.3.2.cmml" xref="A3.E8.m1.1.1.1.1.1.1.2.3.2">𝑅</ci><ci id="A3.E8.m1.1.1.1.1.1.1.2.3.3.cmml" xref="A3.E8.m1.1.1.1.1.1.1.2.3.3">𝑇</ci></apply></apply><ci id="A3.E8.m1.1.1.1.1.1.1.3.cmml" xref="A3.E8.m1.1.1.1.1.1.1.3">𝑛</ci></apply></apply></apply></apply><apply id="A3.E8.m1.1.1e.cmml" xref="A3.E8.m1.1.1"><eq id="A3.E8.m1.1.1.7.cmml" xref="A3.E8.m1.1.1.7"></eq><share href="#A3.E8.m1.1.1.1.cmml" id="A3.E8.m1.1.1f.cmml" xref="A3.E8.m1.1.1"></share><cn type="integer" id="A3.E8.m1.1.1.8.cmml" xref="A3.E8.m1.1.1.8">0</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.E8.m1.1c">x_{i}^{n}=x_{i-1}^{n}+{f_{RT}}^{n}\Rightarrow{\mathds{E}}[{f_{RT}}^{n}]=0</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(8)</span></td>
</tr></tbody>
</table>
<p id="A3.p3.4" class="ltx_p">which we have empirically verified to be correct, as shown in Figure <a href="#S2.F3" title="Figure 3 ‣ 2 Background and Related Work ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. Therefore, PreLN ensures that the expectation of the new estimate will be correct, and the iterative summation of the residual functions in the remaining layers determines the variance of the new estimate <math id="A3.p3.4.m1.1" class="ltx_Math" alttext="{\mathds{E}}[{F_{RT}}_{i}]" display="inline"><semantics id="A3.p3.4.m1.1a"><mrow id="A3.p3.4.m1.1.1" xref="A3.p3.4.m1.1.1.cmml"><mi id="A3.p3.4.m1.1.1.3" xref="A3.p3.4.m1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A3.p3.4.m1.1.1.2" xref="A3.p3.4.m1.1.1.2.cmml">​</mo><mrow id="A3.p3.4.m1.1.1.1.1" xref="A3.p3.4.m1.1.1.1.2.cmml"><mo stretchy="false" id="A3.p3.4.m1.1.1.1.1.2" xref="A3.p3.4.m1.1.1.1.2.1.cmml">[</mo><mmultiscripts id="A3.p3.4.m1.1.1.1.1.1" xref="A3.p3.4.m1.1.1.1.1.1.cmml"><mi id="A3.p3.4.m1.1.1.1.1.1.2.2" xref="A3.p3.4.m1.1.1.1.1.1.2.2.cmml">F</mi><mrow id="A3.p3.4.m1.1.1.1.1.1.2.3" xref="A3.p3.4.m1.1.1.1.1.1.2.3.cmml"><mi id="A3.p3.4.m1.1.1.1.1.1.2.3.2" xref="A3.p3.4.m1.1.1.1.1.1.2.3.2.cmml">R</mi><mo lspace="0em" rspace="0em" id="A3.p3.4.m1.1.1.1.1.1.2.3.1" xref="A3.p3.4.m1.1.1.1.1.1.2.3.1.cmml">​</mo><mi id="A3.p3.4.m1.1.1.1.1.1.2.3.3" xref="A3.p3.4.m1.1.1.1.1.1.2.3.3.cmml">T</mi></mrow><mrow id="A3.p3.4.m1.1.1.1.1.1a" xref="A3.p3.4.m1.1.1.1.1.1.cmml"></mrow><mi id="A3.p3.4.m1.1.1.1.1.1.3" xref="A3.p3.4.m1.1.1.1.1.1.3.cmml">i</mi><mrow id="A3.p3.4.m1.1.1.1.1.1b" xref="A3.p3.4.m1.1.1.1.1.1.cmml"></mrow></mmultiscripts><mo stretchy="false" id="A3.p3.4.m1.1.1.1.1.3" xref="A3.p3.4.m1.1.1.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.p3.4.m1.1b"><apply id="A3.p3.4.m1.1.1.cmml" xref="A3.p3.4.m1.1.1"><times id="A3.p3.4.m1.1.1.2.cmml" xref="A3.p3.4.m1.1.1.2"></times><ci id="A3.p3.4.m1.1.1.3.cmml" xref="A3.p3.4.m1.1.1.3">𝔼</ci><apply id="A3.p3.4.m1.1.1.1.2.cmml" xref="A3.p3.4.m1.1.1.1.1"><csymbol cd="latexml" id="A3.p3.4.m1.1.1.1.2.1.cmml" xref="A3.p3.4.m1.1.1.1.1.2">delimited-[]</csymbol><apply id="A3.p3.4.m1.1.1.1.1.1.cmml" xref="A3.p3.4.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="A3.p3.4.m1.1.1.1.1.1.1.cmml" xref="A3.p3.4.m1.1.1.1.1.1">subscript</csymbol><apply id="A3.p3.4.m1.1.1.1.1.1.2.cmml" xref="A3.p3.4.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="A3.p3.4.m1.1.1.1.1.1.2.1.cmml" xref="A3.p3.4.m1.1.1.1.1.1">subscript</csymbol><ci id="A3.p3.4.m1.1.1.1.1.1.2.2.cmml" xref="A3.p3.4.m1.1.1.1.1.1.2.2">𝐹</ci><apply id="A3.p3.4.m1.1.1.1.1.1.2.3.cmml" xref="A3.p3.4.m1.1.1.1.1.1.2.3"><times id="A3.p3.4.m1.1.1.1.1.1.2.3.1.cmml" xref="A3.p3.4.m1.1.1.1.1.1.2.3.1"></times><ci id="A3.p3.4.m1.1.1.1.1.1.2.3.2.cmml" xref="A3.p3.4.m1.1.1.1.1.1.2.3.2">𝑅</ci><ci id="A3.p3.4.m1.1.1.1.1.1.2.3.3.cmml" xref="A3.p3.4.m1.1.1.1.1.1.2.3.3">𝑇</ci></apply></apply><ci id="A3.p3.4.m1.1.1.1.1.1.3.cmml" xref="A3.p3.4.m1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p3.4.m1.1c">{\mathds{E}}[{F_{RT}}_{i}]</annotation></semantics></math>.</p>
</div>
<section id="A3.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">The effect of learning rates on downstream tasks.</h4>

<div id="A3.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="A3.SS0.SSS0.Px1.p1.1" class="ltx_p">We focus on evaluating larger datasets and exclude very small datasets, as we find that the validation scores on those datasets have a large variance for different random seeds.</p>
</div>
<div id="A3.SS0.SSS0.Px1.p2" class="ltx_para">
<p id="A3.SS0.SSS0.Px1.p2.1" class="ltx_p">For fine-tuning models on downstream tasks,
we consider training with batch size 32 and performing a
linear warmup for the first 10% of steps followed by
a linear decay to 0. We fine-tune for 5 epochs and
perform the evaluation on the development set.
We report the median development
set results for each task over five random initializations, without model ensemble.
</p>
</div>
<div id="A3.SS0.SSS0.Px1.p3" class="ltx_para">
<p id="A3.SS0.SSS0.Px1.p3.1" class="ltx_p">Results are visualized in Fig. <a href="#S5.F9" title="Figure 9 ‣ Downstream task fine-tuning sensitivity. ‣ 5.2 Ablation Studies ‣ 5 Evaluation ‣ Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>, which shows that the baseline is less robust on the choice of learning rates. Specifically, the fine-tuning results are often much worse with a large learning rate. In comparison, PLD is more robust and often achieves better results with large learning rates.</p>
</div>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2010.13368" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2010.13369" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2010.13369">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2010.13369" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2010.13370" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Mon Mar 18 13:36:05 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2010.13369
Recently, Transformer-based language models have demonstrated remarkable performance across many NLP domains.
However, the unsupervised pre-training step of these models suffers from unbearable overall computational expenses.
Current methods for accelerating the pre-training either rely on massive parallelism with advanced hardware or are not applicable to language modeling.
In this work, we propose a method based onprogressive layer droppingthat speeds the training of Transformer-based language models, not at the cost of excessive hardware resources but from model architecture change and training technique boosted efficiency.
Extensive experiments on BERT show that the proposed method achieves
a 24% time reduction on average per sample and allows the pre-training to be 2.5faster than the baseline to get a similar accuracy on downstream tasks. While being faster, our pre-trained models are equipped with strong knowledge transferability, achieving comparable and sometimes higher GLUE score than the baseline when pre-trained with the same number of samples.
abstract
Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping
Abstract
2010.13369
1IntroductionNatural language processing (NLP)  tasks, such as natural language inferenceand question answering, have achieved great success with the development of neural networks. It has been demonstrated recently that Transformer-based networks have obtained superior performance in many NLP tasks (e.g., the GLUE benchmarkand the challenging multi-hop reasoning task)  than recurrent neural networks or convolutional neural networks. BERT trains a deep bidirectional Transformer and obtains outstanding results with transfer learning. RoBERTa, which is a robustly optimized version of BERT trained with more steps and larger corpora, achieves state-of-the-art results on 9 GLUE tasks. Megatron-LMfurther advances the state-of-the-art in NLP by significantly increasing the size of BERT model. Finally, there are multiple research proposing different enhanced versions of Transformer-based networks, such as GPT-2/3, XLNet, SpanBERT, BioBERT, UniLM, Turing-NLG, and T5.
Due to the exciting prospect, pre-training Transformer networks with a large corpus of text followed by fine-tuning on specific tasks has become a new paradigm for natural language processing.Despite great success, a big challenge of Transformer networks comes from the training efficiency – even with self-attention and parallelizable recurrence, and extremely high performance hardware, the pre-training step still takes a significant amount of time.
To address this challenge, mixed-precision training is explored, where the forward pass and backward pass are computed in half-precision and parameter update is in single precision. However, it requires Tensor Cores, which do not exist in all hardware. Some work resort to distributed training. However, distributed training uses large mini-batch sizes to increase the parallelism, where the training often converges to sharp local minima with poor generalizability even with significant hyperparameter tuning. Subsequently, Yang et al. propose a layer-wise adaptive large batch optimizer called LAMB, allowing to train BERT with 32K batch size on 1024 TPU chips. However, this type of approach often requires dedicated clusters with hundreds or even thousands of GPUs and sophisticated system techniques at managing and tuning distributed training, not to mention that the amount of computational resources is intractable for most research labs or individual practitioners.In this paper, we speedup pre-training Transformer networks by
exploring architectural change and training techniques, not at the cost of excessive hardware resources.
Given that the training cost grows linearly with the number of Transformer layers, one straightforward idea to reduce the computation cost is to reduce the depth of the Transformer networks. However, this is restrictive as it often results in lower accuracy in downstream tasks compared to full model pre-training, presumably because of having smaller model capacities. Techniques such as Stochastic Depth have been demonstrated to be useful in accelerating supervised training in the image recognition domain. However, we observe that stochastically removing Transformer layers destabilizes the performance and
easily results in severe consequences such as model divergence or convergence to bad/suspicious local optima. Why are Transformer networks difficult to train with stochastic depth? Moreover, can we speed up the (unsupervised)  pre-training of Transformer networks without hurting downstream performance?To address the above challenges,
we propose to accelerate pre-training of Transformer networks by
making the following contributions. (i)  We conduct a comprehensive analysis to answer the
question: what makes Transformer networks difficult to train with stochastic depth. We find that both the choice of Transformer architecture as well as training dynamics would have a big impact on layer dropping. (ii)  We propose a new architecture unit, called theSwitchable-Transformer(ST)  block, that
not only allows switching on/off a Transformer layer for only a set portion of the training schedule, excluding them from both forward and backward pass but also stabilizes Transformer network training. (iii)  We
further propose aprogressive scheduleto add extra-stableness for pre-training Transformer networks with layer dropping –
our schedule smoothly increases the layer dropping rate for each mini-batch as training evolves by adapting in time the parameter of the Bernoulli distribution used for sampling. Within each gradient update, we distribute a global layer dropping rate across all the ST blocks to favor different layers. (iv)  We use BERT as an example, and we conduct extensive experiments
to show that
the proposed method not only allows to train BERT 24% faster
than the baseline under the same number of samples but also allows the pre-training to be 2.5faster to get similar accuracy on downstream tasks.
Furthermore, we evaluate the generalizability of models pre-trained with the same number of samples as the baseline, and we observe
that while faster to train,
our approach achieves a 1.1% higher GLUE score than the baseline, indicating a strong knowledge transferability.
introduction
Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping
1Introduction
2010.13369
2Background and Related WorkPre-training with Transformer-based architectures like BERThas been demonstrated as an effective strategy for language representation learning. The approach provides a better model initialization for downstream tasks by training on large-scale unlabeled corpora, which often leads to a better generalization performance on the target task through fine-tuning on small data.
Consider BERT, which consists a stack ofTransformer layers.
Each Transformer layer encodes the the input of the i-th Transformer layerwith, which is a multi-head self-attention sub-layer, and then by, which is a feed-forward network, whereis the output of the i-th Transformer layer. Both sub-layers have an AddNorm operation that consists a residual connectionand a layer normalization () . The BERT model recursively applies the transformer block to the input to get the output.While the Transformer-based architecture has achieved breakthrough results in modeling sequences for unsupervised language modeling, previous work has also highlighted the training difficulties and excessively long training time.
To speed up the pre-training, ELECTRAexplores the adversarial training scheme by replacing masked tokens with alternatives sampled from a generator framework and training a discriminator to predict the replaced token. This increases the relative per-step cost, but leads to fewer steps, leading to the overall reduced costs.
Another line of work focus on reducing the per-step cost.
Since the total number of floating-point operations (FLOPS)  of the forward and backward passes in the BERT pre-training process is linearly proportional to the depth of the Transformer blocks, reducing the number of Transformer layers brings opportunities to significantly speed up BERT pre-training. To show this, we plot the FLOPS per training iteration in Fig.6, assuming we can remove a fraction of layers at each step. Each line in the figure shows the FLOPS using different layer removal schedules. Regardless of which schedule to choose, the majority of FLOPS are reduced in the later steps, with the rate of keep probability saturating to a fixed value(e.g., 0.5) . We will describe our schedule in Section4.2.Figure 1:The norm of the gradient with respect to the weights, with PostLN and PreLN.Figure 2:The norm preserving ratio with respect to the inputs, with PostLN and PreLN.Figure 3:Lesioning analysis with PostLN and PreLN.Despite the FLOPS reduction, directly training models like BERT with a smaller depth incurs a significant loss in accuracy even with knowledge distillation. Prior workproposes to accelerate pre-training by first training a 3-layer BERT model and then growing the network depth to 6-layer and subsequently 12-layer.
However, the number of steps required at each depth before the network growth is not known a prior, making applying this approach challenging in practice. On the other hand, stochastic depth has been successfully demonstrated to train deep models with reduced expected depth. However, we observe that directly pre-training BERT with randomly droppingandconverges to bad/suspicious local optima under the same hyperparameter setting. When increasing the learning rate, the training often diverges even by tuning the warmup ratio.
What causes the instability of BERT pre-training with layer drop?
body
Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping
2Background and Related Work
2010.13369
3Preliminary AnalysisThis section presents several studies that guided the design of the approach introduced in Section4. We used BERT trained on Bookcorpus and Wikipedia dataset from Devlin et. al. with standard settings as the baseline111AppendixAprovides detailed training hyperparameters.. First, we carry out a comparison between BERT with PostLN and PreLN. Our goal is to measure how effective these two methods at stabilizing BERT training.
Our second analysis considers measuring the dynamics of BERT pre-training, including both spatial and temporal dimensions. Finally, we analyze the effect of the removal of the Transformer layers. This leads us to identify appealing choices for our target operating points.3.1Training Stability: PostLN or PreLN?We consider two variants of BERT, namely the PostLN and PreLN. The default BERT employs PostLN, with layer normalization applied after the addition in Transformer blocks. The PreLN changes the placement of the location ofby placing it only on the input stream of the sublayers so thatand then, which is a modification described by several recent works to establish identity mapping for neural machine translation. Fig.3reports the norm of gradients with respect to weights in backward propagation for both methods, varying the depth(e.g., 12, 24, 48) . The plot shows that while PostLN suffers from unbalanced gradients (e.g., vanishing gradients as the layer ID decreases) , PreLN eliminates the unbalanced gradient problem (solid green lines)  and the gradient norm stays almost same for any layer. Furthermore, Fig.3shows that for PreLN the gradients with respect to inputhave very
similar magnitudes (norm preserving ratio close to 1)  at different layers, which is consistent with prior findings that a neural model should preserve the gradient norm between layers so as to have well-conditioning and faster convergence.
Indeed, we find that PostLN is more sensitive to the choice of hyperparameters, and training often diverges with more aggressive learning rates (more results in Section5) , whereas PreLN avoids vanishing gradients and leads to more stable optimization. We also provide preliminary theoretical results in AppendixBon why PreLN is beneficial.3.2Corroboration of Training DynamicsHereafter we investigate the representationlearned at different phases of BERT pre-training and at different layers. Fig.4shows the L2 norm distances and cosine similarity, which measures the angle between two vectors and ignores their norms, between the input and output embeddings, with PostLN and PreLN, respectively. We draw several observations.First, the dissimilarity (Fig.4aand Fig.4b)  stays high for both PostLN and PreLN at those higher layers in the beginning, and the L2 and cosine similarity seems to be less correlated (e.g., step = 300) .
This is presumably because, at the beginning of the training, the model weights are randomly initialized, and the network is still actively adjusting weights to derive richer features from input data. Since the model is still positively self-organizing on the network parameters toward their optimal configuration,
dropping layers at this stage is not an interesting strategy, because it can create inputs with large noise and disturb the positive co-adaption process.Second, as the training proceeds (Fig.4cand Fig.4d) , although the dissimilarity remains relatively high and bumpy for PostLN, the similarity from PreLN starts to increase over successive layers, indicating that while PostLN
is still trying to produce new representations that are very different across layers, the dissimilarity from PreLN is getting close to zero for upper layers, indicating that the upper layers are getting similar estimations. This can be viewed as doing an unrolled iterative refinement, where a group of successive layers iteratively refine their estimates of the same representations instead of computing an entirely new representation.
Although the viewpoint was originally proposed to explain ResNet, we demonstrate that it is also true for language modeling and Transformer-based networks. AppendixCprovides additional analysis on how PreLN provides extra preservation of feature identity through unrolled iterative refinement.(a) (b) (c) (d) Figure 4:The L2 distance and cosine similarity of the input and output embeddings for BERT with PostLN and PreLN, at different layers and different steps. We plot the inverse of cosine similarity (arccosine)  in degrees, so that for both L2 and arccosine, the lower the more similar.3.3Effect of LesioningWe randomly drop layers with a keep ratioto test if dropping layers would break the training because dropping any layer changes the input distribution of all subsequent layers. The results are shown in Fig.3. As shown, removing layers in PostLN significantly reduces performance. Moreover, when increasing the learning rate, it results in diverged training.
In contrast, this is not the case for PreLN. Given that later layers in PreLN tend to refine an estimate of the representation, the model with PreLN has less dependence on the downsampling individual layers. As a result, removing Transformer layers with PreLN has a modest impact on performance (slightly worse validation loss at the same number of training samples) . However, the change is much smaller than with PostLN. It further indicates that if we remove layers, especially those higher ones, it should have only a mild effect on the final result because doing so does not change the overall estimation the next layer receives, only its quality. The following layers can still perform mostly the same operation, even with some relatively little noisy input. Furthermore, as Fig.4indicates, since the lower layers remain to have a relatively high dissimilarity (deriving new features) , they should be less frequently dropped. Overall, these results show that, to some extent, the structure of a Transformer network with PreLN can be changed at runtime without significantly affecting performance.
body
Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping
3Preliminary Analysis
2010.13369
4Our Approach: Progressive Layer DroppingThis section describes our approach, namely progressive layer dropping (PLD) , to accelerate the pre-training of Transformer-based models. We first present the Switchable-Transformer blocks, a new unit that allows us to train models like BERT with layer drop and improved stability. Then we introduce the progressive layer drop procedure.4.1Switchable-Transformer BlocksIn this work, we propose a novel transformer unit, which we call "Switchable-Transformer " (ST)  block. Compared with the original Transformer block (Fig.5a) , it contains two changes.Identity mapping reordering.The first change is to establish identity mapping within a transformer block by placing the layer normalization only on the input stream of the sublayers (i.e., use PreLN to replace PostLN)  (Fig.5b)  for the stability reason described in Section3.1.(a) Original(b) Identity mapping reordering(c) Switchable TransformerFigure 5:Transformer variants, showing a single layer block.Figure 6:FLOPS per training iteration normalized to the baseline.Switchable gates.Next, we extend the architecture to include a gate for each sub-layer (Fig.5c) , which controls whether a sub-layer is disabled or not during training. In particular, for each mini-batch, the two gates for the two sublayers decide whether to remove their corresponding transformation functions and only keep the identify mapping connection, which is equivalent to applying a conditional gate functionto each sub-layer as follows:(1) In our design, the functiononly takes 0 or 1 as values, which is chosen randomly from a Bernoulli distribution (with two possible outcomes) ,,
whereis the probability of choosing 1. Because the blocks are selected with probabilityduring training and are always presented during inference, we re-calibrate the layers’ output by a scaling factor ofwhenever they are selected.4.2A Progressive Layer Dropping ScheduleBased on the insights from Section3.2, and inspired by prior work on curriculum learningwe propose a progressive schedule– a temporal schedule for the expected number of ST blocks that are retained.
We limit ourselves to monotonically decreasing functions so that the likelihood of layer dropping can only increase along the temporal dimension.
We constrainto befor any t, whereis a limit value, to be taken as(Section5) .
Based on this, we define the progressive scheduleas:Definition 4.1.A progressive schedule is a functionsuch that= 1 and, where.Progress along the time dimension.Starting from the initial conditionwhere no layer drop is performed, layer drop is gradually introduced. Eventually (i.e., whenis sufficiently large) ,.
According to Def.4.1, in our work, we use the following schedule function:(2) By considering Fig.6, we provide intuitive and straightforward motivation for our choice. The blue curve in Fig.6are polynomials of increasing degree(left to right) . Despite fulfilling the initial constraint, they have to be manually thresholded to imposewhen, which introduces two more parameters (and the threshold) .

In contrast, in our schedule, we fixusing the following simple heuristics, since it impliesforwhen, and it is reasonable to assume that T is at the order of magnitude oforwhen training Transformer networks. In other words, this means that for a big portion of the training, we are dropping ()  ST blocks, accelerating the training efficiency.Distribution along the depth dimension.The above progressive schedule assumes all gates in ST blocks take the samevalue at each step t. However, as shown in Fig.4,
the lower layers of the networks
should be more reliably present. Therefore, we distribute the globalacross the entire stack so that lower layers have lower drop probability linearly scaled by their depth according to equation3.
Furthermore, we let the sub-layers inside each block share the same schedule, so
when= 1, both the inner functionandare activated, while they are skipped when= 0.
Therefore, each gate has the following form during training:(3) Combining Eqn.3and Eqn.2, we have the progressive schedule for an ST block below.(4) Algorithm 1Progressive_Layer_Dropping1:Input:2:InitBERT() 3:4:fort1 to Tdo5:6:step7:8:forl1 to Ldo9:actionBernoulli(p) 10:ifaction == 0then11:12:else13:14:15:16:pp - step17:Y18:loss19:backward(loss) Putting it together.Note that because of the exist of the identity mapping, when an ST block is bypassed for a specific iteration,
there is no need to perform forward-backward computation or gradient updates, and there will be updates with significantly shorter networks and more direct paths to individual layers.
Based on this, we design a stochastic training algorithm based on ST blocks and the progressive layer-drop schedule to train models like BERT faster, which we callprogressive layer dropping(Alg.1) .

The expected network depth, denoted as, becomes a random variable. Its expectation is given by:.
With, the expected number of ST blocks during training reduces toorwhen T is large. For the 12-layer BERT model with L=12 used in our experiments, we have. In other words, with progressive layer dropping, we train BERT with an average number of 9 layers. This significantly improves the pre-training speed of the BERT model. Following the calculations
above, approximately 25% of FLOPS could be saved under the drop schedule with= 0.5. We recover the model with full-depth blocks at fine-tuning and testing time.
body
Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping
4Our Approach: Progressive Layer Dropping
2010.13369
5EvaluationWe show that our method improves the pre-training efficiency of Transformer networks, and the trained models achieve competitive or even better performance compared to the baseline on transfer learning downstream tasks. We also show ablation studies to analyze the proposed training techniques.Datasets.We follow Devlin et al.to use English Wikipedia corpus and BookCorpus for pre-training. By concatenating the two datasets, we obtain our corpus with roughly 2.8B word tokens in total, which is comparable with the data corpus used in Devlin et al..



We segment
documents into sentences with 128 tokens; We normalize,
lower-case, and tokenize texts using Wordpiece tokenizer. The
final vocabulary contains 30,528 word pieces.
We split documents into one training set and one validation set (300:1) .
For fine-tuning, we use
GLUE (General Language Understanding Evaluation) , a collection of 9 sentence or sentence-pair natural language understanding tasks including question answering, sentiment analysis, and textual entailment. It is designed to favor sample-efficient learning and knowledge-transfer across a range of different linguistic tasks in different domains.Training details.We use our own implementation of the BERT modelbased on the Huggingface[1] PyTorch implementation. All experiments are performed on 4DGX-2 boxes with 64V100 GPUs. Data parallelism is handled via PyTorch DDP (Distributed Data Parallel)  library. We recognize and eliminate additional computation overhead: we overlap data loading with computation through the asynchronous prefetching queue; we optimize the BERT output processing through sparse computation on masked tokens. Using our pre-processed data, we train a 12-layer BERT-base model from scratch as the baseline. We use a warm-up ratio of 0.02 with lrmax=1e-4. Following, we use Adam as the optimizer. We train with batch size 4K for 200K steps, which is approximately 186 epochs. The detailed parameter settings are listed in the AppendixA. We fine-tune GLUE tasks for 5 epochs and report the median development
set results for each task over five random initializations.5.1Experimental ResultsPre-training convergence comparisons.Fig.LABEL:fig:different-lrvisualizes the convergence of validation loss regarding the computational time. We make the following observations. First, with lrmax=1e-4, the convergence rate of our algorithm and the baseline is very close. This verifies empirically that our progressive layer dropping method does not hurt model convergence. Second, when using a larger learning rate lrmax=1e-3, the baseline diverges. In contrast, our method shows a healthy convergence curve and is much faster. This confirms that our architectural changes stabilize training and allows BERT training with more aggressive learning rates.(a) (b) Figure 7:The convergence curve of the baseline and our proposed method regarding the wall-clock time.Table 1:Training time comparison. Sample RD standards for sample reduction. SPD represents speedup.TrainingTimeSampleRDSPDBaselineckp18638.45h01PLDckp18629.22h01.3PLDckp10015.56h46%2.5PLDckp8713.53h53%2.8Speedup.Fig.LABEL:fig:baseline-vs-pst-mainshows both the training curve (dotted)  and the validation curve (solid)  of the baseline and PLD with a zoomed-in view. The baseline curve becomes almost flat at epoch 186, getting a validation loss of 1.75. In contrast, PLD reaches the same validation loss at epoch 87, with 53% fewer training samples. Furthermore, PLD achieves a 24% time reduction when training the same number of samples. This is because our approach trains the model with a smaller number of expected depth for the same number of steps. It is slightly lower than the 25% GFLOPS reduction in the analysis because the output layer still takes a small amount of computation even after optimizations. The combination of these two factors, yields 2.8speedup in end-to-end wall-clock training time over the baseline, as shown in Table7.Downstream task accuracy.Despite improved training speed, one may still wonder whether such a method is as effective as the baseline model on downstream tasks.
Table2shows our results on the GLUE dataset compared to the baseline. Our baseline is comparable with the original BERT-Base (on the test set) , and our PLD method achieves a higher GLUE score than our baseline (83.2 vs. 82.1)  when fine-tuning the checkpoint (186) . We also dump model checkpoints from different epochs during pre-training and fine-tune these models.
The checkpoint 87 corresponds to the validation loss at 1.75 achieved by PLD. The GLUE score is slightly worse than the baseline (81.6 vs. 82.1) . However, by fine-tuning at checkpoint 100, PLD achieves a higher score than the baseline (82.3 vs. 82.1)  at checkpoint 186. In terms of the pre-training wall clock time, PLD requires 15.56h vs. the baseline with 39.15h to get similar accuracy on downstream tasks, which corresponds to a 2.5speedup.Table 2:The results on the GLUE benchmark. The number below each task denotes the number of training examples.
The metrics for these tasks can be found in the GLUE paper. We compute the geometric mean of the metrics as the GLUE score.ModelRTE(Acc.) MRPC(F1/Acc.) STS-B(PCC/SCC) CoLA(MCC) SST-2(Acc.) QNLI(Acc.) QQP(F1/Acc.) MNLI-mm-/m (Acc.) GLUE2.5K3.7K5.7K8.5K67K108K368K393KBERTbase(original) 66.488.9/84.887.1/89.252.193.590.571.2/89.284.6/83.480.7BERTbase(Baseline, ckp186) 67.888.0/86.089.5/89.252.591.287.189.0/90.682.5/83.482.1BERTbase(PLD, ckp87) 6688.2/85.688.9/88.454.59186.387.4/89.181.6/82.481.6BERTbase(PLD, ckp100) 68.288.2/85.889.3/88.956.191.586.987.7/89.382.4/82.682.3BERTbase(PLD, ckp186) 6988.9/86.589.6/89.159.491.88889.4/90.983.1/83.583.2Fig.8illustrates the fine-tuning results between the baseline and PLD on GLUE tasks over different checkpoints
Overall, we observe that PLD not only trains BERT faster in pre-training but also preserves the performance on downstream tasks. In each figure, we observe that both curves have a similar shape at the beginning because no layer drop is added. For later checkpoints, PLD smoothly adds layer drop. Interestingly, we note that the baseline model has fluctuations in testing accuracy. In contrast, the downstream task accuracy from PLD is consistently increasing as the
number of training epochs increases. This indicates that PLD takes a more robust optimization path toward the optimum. We also observe that our model achieves higher performance on MNLI, QNLI, QQP, RTE, SST-2, and CoLA on later checkpoints, indicating that the model trained with our approach also generalizes better on downstream tasks than our baseline does.From a knowledge transferability perspective, the goal of training a language model is to learn a good representation of natural language that ideally ignores the data-dependent noise and generalizes well to downstream tasks. However, training a model with a constant depth is at least somewhat noisy and can bias the model to prefer certain representations, whereas PLD enables more sub-network configurations to be created during training Transformer networks.
Each of the L ST blocks is either active or inactive, resulting inpossible network combinations. By selecting a different submodular in each mini-batch, PLD encourages the submodular to produce good results independently. This allows the unsupervised pre-training model to obtain a more general representation by averaging the noise patterns, which helps the model to better generalize to new tasks.
On the other hand, during inference, the full network is presented, causing the effect of ensembling different sub-networks.(a) MNLI-m(b) MNLI-mm(c) QNLI(d) QQP(e) RTE(f) SST-2(g) WNLI(h) CoLA(i) MRPC (acc.) (j) MRPC (F1.) (k) SST-B (PCC) (l) SST-B (SCC) Figure 8:The fine-tuning results at different checkpoints.5.2Ablation StudiesDownstream task fine-tuning sensitivity.To further verify that our approach not only stabilizes training but also improves downstream tasks, we show a grid search on learning rates
{1e-5, 3e-5, 5e-5, 7e-5, 9e-5, 1e-4}. As illustrated in Fig.9, the baseline is vulnerable to the choice of learning rates. Specifically, the fine-tuning results are often much worse with a large learning rate, while PLD is more robust and often achieves better results with large learning rates.(a) MNLI-m(b) MNLI-mm(c) QNLI(d) QQP(e) RTE(f) SST-2Figure 9:The fine-tuning results at different checkpoints.The Effect of.We test different values of the keep ratioand identifyas a good range, as shown in Fig.10in the Appendix. We observe that the algorithm may diverge ifis too small (e.g., 0.3) .Figure 10:Convergence curves varying the keep ratio.PLD vs. PreLN.To investigate the question on how PLD compares with PreLN, we run both PreLN with the hyperparameters used for training PostLN (lr=1e-4)  and the hyperparameters used for PLD (lr=1e-3)  to address the effect from the choice of hyperparameters. We train all configurations for the same number of epochs and fine-tune following the standard procedure. In both cases, PreLN is 24% slower than PLD, because PreLN still needs to perform the full forward and backward propagation in each iteration.Table3shows the fine-tuning results on GLUE tasks. When trained with the same hyperparameters as PostLN, PreLN appears to have a much worse GLUE score (80.2)  compared with PostLN (82.1)  on downstream tasks. This is because PreLN restricts layer outputs from depending too much on their own residual branches and inhibits the network from reaching its full potential, as recently studied in. When trained with the large learning rate as PLD, PreLN’s result have improved to 82.6 but is 0.6 points worse than PLD (83.2) , despite using 24% more compute resource. PLD achieves better accuracy than PreLN because it encourages each residual branch to produce good results independently.Table 3:Ablation studies of the fine-tuning results on the GLUE benchmark.ModelRTE(Acc.) MRPC(F1/Acc.) STS-B(PCC/SCC) CoLA(MCC) SST-2(Acc.) QNLI(Acc.) QQP(F1/Acc.) MNLI-m/mm(Acc.) GLUEBERT (Original) 66.488.9/84.887.1/89.252.193.590.571.2/89.284.6/83.480.7BERT + PostLN67.888.0/86.089.5/89.252.591.287.189.0/90.682.5/83.482.1BERT + PreLN + Same lr66.085.9/83.388.2/87.946.490.585.589.0/90.681.6/81.680.2BERT + PreLN + lr67.886.7/84.589.6/89.154.691.988.189.3/90.983.6/83.782.6Shallow BERT + PreLN + lr66.085.9/83.589.5/88.954.791.886.189.0/90.682.7/82.981.8BERT + PreLN + lr+ Rand.68.288.2/86.289.3/88.856.891.587.288.6/90.382.9/83.382.7BERT + PreLN + lr+ TD68.288.6/86.789.4/88.955.991.386.889.1/90.782.7/83.182.7BERT + PreLN + lr+ PLD69.088.9/86.589.6/89.159.491.888.089.4/90.983.1/83.583.2PLD vs. Shallow network.Shallow BERT + PreLN + Large lrin Table3shows the downstream task accuracy of the 9-layer BERT. Although having the same same number of training computational GFLOPS as ours, the shallow BERT
underperforms PreLN by 0.8 points and is 1.4 points worse than PLD likely because the model capacity has been reduced by the loss of parameters.PLD vs. Random drop.BERT + PreLN + Large lr + Randomdrops layers randomly with a fixed ratio (i.e., it has the same compute cost but without any schedule) , similar to Stochastic Depth. The GLUE score is 0.9 points better than shallow BERT under the same compute cost and 0.1 point better than PreLN while being 24% faster, indicating the strong regularization effect from stochastic depth. It is 0.5 points worse than PLD, presumably because a fixed ratio does not take into account the training dynamics of Transformer networks.Schedule impact analysis.BERT + PreLN + Large lr + TD only (32-bit*) disables the schedule along the depth dimension (DD)  and enables only the schedule along the temporal dimension (TD)  in training. Its GLUE score matches "Random", suggesting that the temporal schedule has similar performance as the fixed constant schedule along the time dimension and accuracy gains of PLD is mostly from the depth dimension. However, without the temporal schedule enabled, the model diverges withNaNin the middle of half-precision (16-bit)  training and has to switch to full-precision (32-bit)  training, slowing down training speed. Furthermore, this concept of starting-easy and gradually increasing the difficulty of the learning problem has its roots in curriculum learning and often makes optimization easier. We adopt the temporal schedule since it is robust and helpful for training stability, retaining similar accuracy while reducing training cost considerably.
body
Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping
5Evaluation
2010.13369
6ConclusionUnsupervised language model pre-training is a crucial step for getting state-of-the-art performance on NLP tasks. The current time for training such a model is excruciatingly long, and it is very much desirable to reduce the turnaround time for training such models. In this paper, we study the efficient training algorithms for pre-training BERT model for NLP tasks. We have conducted extensive analysis and found that model architecture is important when training Transformer-based models with stochastic depth. Using this insight, we propose the Switchable-Transformer block and a progressive layer-wise drop schedule. Our experiment results show that our training strategy achieves competitive performance to training a deep model from scratch at a faster rate.
conclusion
Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping
6Conclusion