<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2403.09347] BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences</title><meta property="og:description" content="Effective attention modules have played a crucial role in the success of Transformer-based large language models
(LLMs), but the quadratic time and memory complexities of these attention modules also pose a challenge w…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2403.09347">

<!--Generated on Fri Apr  5 14:14:59 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_pruned_first">
<h1 class="ltx_title ltx_title_document">BurstAttention: An Efficient Distributed Attention Framework for 
<br class="ltx_break">Extremely Long Sequences</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Ao Sun
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Weilin Zhao
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Xu Han
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Cheng Yang
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Zhiyuan Liu
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Chuan Shi
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Maosong Sun
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Shengnan Wan
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Teng Su
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id2.2" class="ltx_p">Effective attention modules have played a crucial role in the success of Transformer-based large language models
(LLMs), but the quadratic time and memory complexities of these attention modules also pose a challenge when processing long sequences.
One potential solution for the long sequence problem is to utilize distributed clusters to parallelize the computation of attention modules across multiple devices
(e.g., GPUs).
However, adopting a distributed approach inevitably introduces extra memory overheads to store local attention results and incurs additional communication costs to aggregate local results into global ones.
In this paper, we propose a distributed attention framework named “BurstAttention” to optimize memory access and communication operations at both the global cluster and local device levels by partitioning attention along the sequence dimension across device.
Through our experiments, we compare BurstAttention with other competitive long-sequence distributed attention solutions.
The experimental results under different lengths demonstrate that BurstAttention offers significant advantages for processing long sequences compared with these competitive baselines, especially tensor parallelism (Megatron-V3) with FlashAttention, reducing 40% communication overheads and achieving 2<math id="id1.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="id1.1.m1.1a"><mo id="id1.1.m1.1.1" xref="id1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="id1.1.m1.1b"><times id="id1.1.m1.1.1.cmml" xref="id1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="id1.1.m1.1c">\times</annotation></semantics></math> speedup during training 128K sequence length on 8<math id="id2.2.m2.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="id2.2.m2.1a"><mo id="id2.2.m2.1.1" xref="id2.2.m2.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="id2.2.m2.1b"><times id="id2.2.m2.1.1.cmml" xref="id2.2.m2.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="id2.2.m2.1c">\times</annotation></semantics></math>A100.</p>
</div>
<div id="p2" class="ltx_para">
<br class="ltx_break">
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Transformers <cite class="ltx_cite ltx_citemacro_citep">(Vaswani et al., <a href="#bib.bib27" title="" class="ltx_ref">2017</a>)</cite> have emerged as the dominant architectures for large language models (LLMs) <cite class="ltx_cite ltx_citemacro_citep">(Brown et al., <a href="#bib.bib3" title="" class="ltx_ref">2020</a>; Chowdhery et al., <a href="#bib.bib6" title="" class="ltx_ref">2022</a>)</cite> due to their remarkable capacities to understand complex text and generate controllable responses. Empirically, the power of Transformers lies largely in their multi-head attention modules, which enable Transformers to capture rich semantic information from textual contexts effectively. For every plus, there is a minus. Despite the success of Transformers’ attention modules, these modules exhibit quadratic time and memory complexity concerning sequence length, posing challenges in terms of both computing time and memory overheads as sequence length increases.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">Various efforts have been devoted to making attention modules more efficient and enabling LLMs to process longer sequences.
One direction is taking full advantage of a single device’s compute and storage units (e.g., a GPU) to process long sequences, such as FlashAttention <cite class="ltx_cite ltx_citemacro_citep">(Dao et al., <a href="#bib.bib7" title="" class="ltx_ref">2022</a>)</cite>. FlashAttention can significantly accelerate the computation of attention modules by using more efficient static random access memory (SRAM) instead of high-bandwidth memory (HBM) in devices to store intermediate attention states.
Another direction is using distributed clusters containing multiple devices (e.g., multiple GPUs) to process long sequences, such as RingAttention <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib15" title="" class="ltx_ref">2021</a>)</cite>. RingAttention divides sequences into multiple subsequences and processes subsequences separately on different devices.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2403.09347/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="392" height="211" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S1.F1.6.3.1" class="ltx_text" style="font-size:90%;">Figure 1</span>: </span><span id="S1.F1.4.2" class="ltx_text" style="font-size:90%;">BurstAttention undertakes a two-step partitioning: dividing the sequence across multiple devices (inter-device), and then splitting the subsequences within each single device (intra-device). First, BurstAttention partitions the query, key, and value across devices and pass each sliced subsequence through all devices in a ring-like communication. This allows each device to process only a local attention at a time, and avoids the burden on memory caused by processing extremely long sequence at once.
By transmitting <math id="S1.F1.3.1.m1.2" class="ltx_Math" alttext="\mathbf{K},\mathbf{V}" display="inline"><semantics id="S1.F1.3.1.m1.2b"><mrow id="S1.F1.3.1.m1.2.3.2" xref="S1.F1.3.1.m1.2.3.1.cmml"><mi id="S1.F1.3.1.m1.1.1" xref="S1.F1.3.1.m1.1.1.cmml">𝐊</mi><mo id="S1.F1.3.1.m1.2.3.2.1" xref="S1.F1.3.1.m1.2.3.1.cmml">,</mo><mi id="S1.F1.3.1.m1.2.2" xref="S1.F1.3.1.m1.2.2.cmml">𝐕</mi></mrow><annotation-xml encoding="MathML-Content" id="S1.F1.3.1.m1.2c"><list id="S1.F1.3.1.m1.2.3.1.cmml" xref="S1.F1.3.1.m1.2.3.2"><ci id="S1.F1.3.1.m1.1.1.cmml" xref="S1.F1.3.1.m1.1.1">𝐊</ci><ci id="S1.F1.3.1.m1.2.2.cmml" xref="S1.F1.3.1.m1.2.2">𝐕</ci></list></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.3.1.m1.2d">\mathbf{K},\mathbf{V}</annotation></semantics></math> and aggregating local attention results using online softmax, BurstAttention avoids storing the intermediate result <math id="S1.F1.4.2.m2.1" class="ltx_Math" alttext="\mathbf{QK}^{T}" display="inline"><semantics id="S1.F1.4.2.m2.1b"><msup id="S1.F1.4.2.m2.1.1" xref="S1.F1.4.2.m2.1.1.cmml"><mi id="S1.F1.4.2.m2.1.1.2" xref="S1.F1.4.2.m2.1.1.2.cmml">𝐐𝐊</mi><mi id="S1.F1.4.2.m2.1.1.3" xref="S1.F1.4.2.m2.1.1.3.cmml">T</mi></msup><annotation-xml encoding="MathML-Content" id="S1.F1.4.2.m2.1c"><apply id="S1.F1.4.2.m2.1.1.cmml" xref="S1.F1.4.2.m2.1.1"><csymbol cd="ambiguous" id="S1.F1.4.2.m2.1.1.1.cmml" xref="S1.F1.4.2.m2.1.1">superscript</csymbol><ci id="S1.F1.4.2.m2.1.1.2.cmml" xref="S1.F1.4.2.m2.1.1.2">𝐐𝐊</ci><ci id="S1.F1.4.2.m2.1.1.3.cmml" xref="S1.F1.4.2.m2.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.4.2.m2.1d">\mathbf{QK}^{T}</annotation></semantics></math>, which has quadratic memory complexity, and instead recomputes it during the backward pass, which we call global attention optimization (GAO).
BurstAttention further partitions the subsequences into smaller tiles, aiming to perform block-wise computations within local attention. This can utilize the high bandwidth of SRAM while minimizing access to the lower bandwidth HBM, which we call local attention optimization (LAO). Also, by using double-buffer, the communication can be overlapped with computation in BurstAttention.
</span></figcaption>
</figure>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">All the above improvements orienting to speedup attention operation have achieved promising results, each targeting different bottleneck. However an intuitive problem is raised — whether we can combine these improvements to achieve a more efficient attention solution.
The concept is straightforward, yet in a distributed setting, simple combination of two methods may not benefit from their strength. Moreover , the RingAttention approach cannot directly incorporate with online softmax, and the FlashAttention implementation focuses exclusively on optimizing the computation of attention on a single device. To address these challenges,this paper introduces an efficient distributed attention framework to handle extremely long sequences named “BurstAttention”. BurstAttention can take full advantage of the power of both distributed clusters and single devices within clusters.
Specifically, given an extremely long sequence, BurstAttention first divides the sequence into partitions according to the number of devices in distributed clusters, and each partition is assigned to one of these devices. Then, each device projects the partitioned sequence into query, value, and key embedding partitions. The query partitions are pinned, and all key-value partitions are passed through all devices to compute their local attention scores with each pinned query partition. Based on the local attention scores, a global attention operation is adopted to aggregate the local results into the final global results.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">By fine-grained scheduling the computation and communication operations of devices during computing attention modules, as well as introducing online softmax <cite class="ltx_cite ltx_citemacro_citep">(Milakov &amp; Gimelshein, <a href="#bib.bib16" title="" class="ltx_ref">2018</a>)</cite>, BurstAttention proposes global attention optimization (GAO) and local attention optimization (LAO), which can fully optimize the input-output (I/O) and communication procedures in distributed clusters. These two strategies offer substantial benefits for computing local attention scores in each device and aggregating local results into global ones in the whole cluster, including improved memory consumption, reduced communication overhead, and enhanced cache utilization.
Owing to just splitting sequences, BurstAttention is orthogonal to other distributed methods and can be integrated with them for training and inferring Transformer-based LLMs, such as data parallelism <cite class="ltx_cite ltx_citemacro_citep">(Valiant, <a href="#bib.bib26" title="" class="ltx_ref">1990</a>)</cite>, tensor parallelism <cite class="ltx_cite ltx_citemacro_citep">(Narayanan et al., <a href="#bib.bib17" title="" class="ltx_ref">2021</a>)</cite>, pipeline parallelism <cite class="ltx_cite ltx_citemacro_citep">(Huang et al., <a href="#bib.bib10" title="" class="ltx_ref">2019</a>)</cite>, and zero redundancy optimizer <cite class="ltx_cite ltx_citemacro_citep">(Rajbhandari et al., <a href="#bib.bib22" title="" class="ltx_ref">2020</a>; Ren et al., <a href="#bib.bib23" title="" class="ltx_ref">2021</a>)</cite>.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.2" class="ltx_p">We evaluate BurstAttention and current competitive distributed attention solutions <cite class="ltx_cite ltx_citemacro_citep">(Dao et al., <a href="#bib.bib7" title="" class="ltx_ref">2022</a>; Li et al., <a href="#bib.bib15" title="" class="ltx_ref">2021</a>)</cite> under various sequence length settings. Comparing to tensor parallelism (Megatron-V3) with FlashAttention methods, our method reducing 40% communication overheads and achieving 2<math id="S1.p5.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S1.p5.1.m1.1a"><mo id="S1.p5.1.m1.1.1" xref="S1.p5.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.p5.1.m1.1b"><times id="S1.p5.1.m1.1.1.cmml" xref="S1.p5.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.p5.1.m1.1c">\times</annotation></semantics></math> speedup during training 128K sequence length on 8<math id="S1.p5.2.m2.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S1.p5.2.m2.1a"><mo id="S1.p5.2.m2.1.1" xref="S1.p5.2.m2.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.p5.2.m2.1b"><times id="S1.p5.2.m2.1.1.cmml" xref="S1.p5.2.m2.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.p5.2.m2.1c">\times</annotation></semantics></math>A100. The experimental results show that BurstAttention is a memory-efficient solution for attention modules to process long sequences and achieve good data throughputs. Moreover, since BurstAttention greatly optimizes the communication operations during the computation process of attention modules, BurstAttention makes it more difficult for device’s communication to become a bottleneck as the devices in distributed clusters increase, and thus can better utilize distributed clusters than other distributed solutions.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Methodology</h2>

<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Preliminary</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.9" class="ltx_p">As the key module in Transformers <cite class="ltx_cite ltx_citemacro_citep">(Vaswani et al., <a href="#bib.bib27" title="" class="ltx_ref">2017</a>)</cite>, an attention module can be formalized as</p>
<table id="S2.E1" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S2.E1X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.E1X.2.1.1.m1.2" class="ltx_Math" alttext="\displaystyle\mathbf{S}=\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d}},\quad\mathbf{P}=\text{softmax}(\mathbf{S}),\quad\mathbf{O}=\mathbf{P}\mathbf{V}," display="inline"><semantics id="S2.E1X.2.1.1.m1.2a"><mrow id="S2.E1X.2.1.1.m1.2.2.1"><mrow id="S2.E1X.2.1.1.m1.2.2.1.1.2" xref="S2.E1X.2.1.1.m1.2.2.1.1.3.cmml"><mrow id="S2.E1X.2.1.1.m1.2.2.1.1.1.1" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1.cmml"><mi mathsize="90%" id="S2.E1X.2.1.1.m1.2.2.1.1.1.1.2" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1.2.cmml">𝐒</mi><mo mathsize="90%" id="S2.E1X.2.1.1.m1.2.2.1.1.1.1.1" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1.1.cmml">=</mo><mstyle displaystyle="true" id="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.cmml"><mfrac id="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3a" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.cmml"><msup id="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.2" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.2.cmml"><mi mathsize="90%" id="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.2.2" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.2.2.cmml">𝐐𝐊</mi><mi mathsize="90%" id="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.2.3" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.2.3.cmml">T</mi></msup><msqrt id="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.3" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.3.cmml"><mi mathsize="90%" id="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.3.2" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.3.2.cmml">d</mi></msqrt></mfrac></mstyle></mrow><mo mathsize="90%" rspace="1.067em" id="S2.E1X.2.1.1.m1.2.2.1.1.2.3" xref="S2.E1X.2.1.1.m1.2.2.1.1.3a.cmml">,</mo><mrow id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.2" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.3.cmml"><mrow id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.cmml"><mi mathsize="90%" id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.2" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.2.cmml">𝐏</mi><mo mathsize="90%" id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.1" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.1.cmml">=</mo><mrow id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.3" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.3.cmml"><mtext mathsize="90%" id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.3.2" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.3.2a.cmml">softmax</mtext><mo lspace="0em" rspace="0em" id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.3.1" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.3.1.cmml">​</mo><mrow id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.3.3.2" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.3.cmml"><mo maxsize="90%" minsize="90%" id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.3.3.2.1" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.3.cmml">(</mo><mi mathsize="90%" id="S2.E1X.2.1.1.m1.1.1" xref="S2.E1X.2.1.1.m1.1.1.cmml">𝐒</mi><mo maxsize="90%" minsize="90%" id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.3.3.2.2" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.3.cmml">)</mo></mrow></mrow></mrow><mo mathsize="90%" rspace="1.067em" id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.2.3" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.3a.cmml">,</mo><mrow id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.2.2" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.2.2.cmml"><mi mathsize="90%" id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.2.2.2" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.2.2.2.cmml">𝐎</mi><mo mathsize="90%" id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.2.2.1" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.2.2.1.cmml">=</mo><mi mathsize="90%" id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.2.2.3" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.2.2.3.cmml">𝐏𝐕</mi></mrow></mrow></mrow><mo mathsize="90%" id="S2.E1X.2.1.1.m1.2.2.1.2">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E1X.2.1.1.m1.2b"><apply id="S2.E1X.2.1.1.m1.2.2.1.1.3.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.2"><csymbol cd="ambiguous" id="S2.E1X.2.1.1.m1.2.2.1.1.3a.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.3">formulae-sequence</csymbol><apply id="S2.E1X.2.1.1.m1.2.2.1.1.1.1.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1"><eq id="S2.E1X.2.1.1.m1.2.2.1.1.1.1.1.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1.1"></eq><ci id="S2.E1X.2.1.1.m1.2.2.1.1.1.1.2.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1.2">𝐒</ci><apply id="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3"><divide id="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.1.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3"></divide><apply id="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.2.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.2.1.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.2">superscript</csymbol><ci id="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.2.2.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.2.2">𝐐𝐊</ci><ci id="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.2.3.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.2.3">𝑇</ci></apply><apply id="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.3.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.3"><root id="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.3a.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.3"></root><ci id="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.3.2.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.1.1.3.3.2">𝑑</ci></apply></apply></apply><apply id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.3.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.2"><csymbol cd="ambiguous" id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.3a.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.2.3">formulae-sequence</csymbol><apply id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1"><eq id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.1.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.1"></eq><ci id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.2.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.2">𝐏</ci><apply id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.3.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.3"><times id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.3.1.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.3.1"></times><ci id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.3.2a.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.3.2"><mtext mathsize="90%" id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.3.2.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.1.1.3.2">softmax</mtext></ci><ci id="S2.E1X.2.1.1.m1.1.1.cmml" xref="S2.E1X.2.1.1.m1.1.1">𝐒</ci></apply></apply><apply id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.2.2.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.2.2"><eq id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.2.2.1.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.2.2.1"></eq><ci id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.2.2.2.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.2.2.2">𝐎</ci><ci id="S2.E1X.2.1.1.m1.2.2.1.1.2.2.2.2.3.cmml" xref="S2.E1X.2.1.1.m1.2.2.1.1.2.2.2.2.3">𝐏𝐕</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1X.2.1.1.m1.2c">\displaystyle\mathbf{S}=\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d}},\quad\mathbf{P}=\text{softmax}(\mathbf{S}),\quad\mathbf{O}=\mathbf{P}\mathbf{V},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(1)</span></td>
</tr>
</tbody>
</table>
<p id="S2.SS1.p1.8" class="ltx_p">where <math id="S2.SS1.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{Q}\in\mathbb{R}^{N\times d}" display="inline"><semantics id="S2.SS1.p1.1.m1.1a"><mrow id="S2.SS1.p1.1.m1.1.1" xref="S2.SS1.p1.1.m1.1.1.cmml"><mi id="S2.SS1.p1.1.m1.1.1.2" xref="S2.SS1.p1.1.m1.1.1.2.cmml">𝐐</mi><mo id="S2.SS1.p1.1.m1.1.1.1" xref="S2.SS1.p1.1.m1.1.1.1.cmml">∈</mo><msup id="S2.SS1.p1.1.m1.1.1.3" xref="S2.SS1.p1.1.m1.1.1.3.cmml"><mi id="S2.SS1.p1.1.m1.1.1.3.2" xref="S2.SS1.p1.1.m1.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS1.p1.1.m1.1.1.3.3" xref="S2.SS1.p1.1.m1.1.1.3.3.cmml"><mi id="S2.SS1.p1.1.m1.1.1.3.3.2" xref="S2.SS1.p1.1.m1.1.1.3.3.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.p1.1.m1.1.1.3.3.1" xref="S2.SS1.p1.1.m1.1.1.3.3.1.cmml">×</mo><mi id="S2.SS1.p1.1.m1.1.1.3.3.3" xref="S2.SS1.p1.1.m1.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.1.m1.1b"><apply id="S2.SS1.p1.1.m1.1.1.cmml" xref="S2.SS1.p1.1.m1.1.1"><in id="S2.SS1.p1.1.m1.1.1.1.cmml" xref="S2.SS1.p1.1.m1.1.1.1"></in><ci id="S2.SS1.p1.1.m1.1.1.2.cmml" xref="S2.SS1.p1.1.m1.1.1.2">𝐐</ci><apply id="S2.SS1.p1.1.m1.1.1.3.cmml" xref="S2.SS1.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p1.1.m1.1.1.3.1.cmml" xref="S2.SS1.p1.1.m1.1.1.3">superscript</csymbol><ci id="S2.SS1.p1.1.m1.1.1.3.2.cmml" xref="S2.SS1.p1.1.m1.1.1.3.2">ℝ</ci><apply id="S2.SS1.p1.1.m1.1.1.3.3.cmml" xref="S2.SS1.p1.1.m1.1.1.3.3"><times id="S2.SS1.p1.1.m1.1.1.3.3.1.cmml" xref="S2.SS1.p1.1.m1.1.1.3.3.1"></times><ci id="S2.SS1.p1.1.m1.1.1.3.3.2.cmml" xref="S2.SS1.p1.1.m1.1.1.3.3.2">𝑁</ci><ci id="S2.SS1.p1.1.m1.1.1.3.3.3.cmml" xref="S2.SS1.p1.1.m1.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.1.m1.1c">\mathbf{Q}\in\mathbb{R}^{N\times d}</annotation></semantics></math> indicates the embeddings of the query sequence, <math id="S2.SS1.p1.2.m2.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S2.SS1.p1.2.m2.1a"><mi id="S2.SS1.p1.2.m2.1.1" xref="S2.SS1.p1.2.m2.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.2.m2.1b"><ci id="S2.SS1.p1.2.m2.1.1.cmml" xref="S2.SS1.p1.2.m2.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.2.m2.1c">N</annotation></semantics></math> is the length of the query sequence, and <math id="S2.SS1.p1.3.m3.1" class="ltx_Math" alttext="d" display="inline"><semantics id="S2.SS1.p1.3.m3.1a"><mi id="S2.SS1.p1.3.m3.1.1" xref="S2.SS1.p1.3.m3.1.1.cmml">d</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.3.m3.1b"><ci id="S2.SS1.p1.3.m3.1.1.cmml" xref="S2.SS1.p1.3.m3.1.1">𝑑</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.3.m3.1c">d</annotation></semantics></math> is the embedding dimension.
<math id="S2.SS1.p1.4.m4.1" class="ltx_Math" alttext="\mathbf{K}\in\mathbb{R}^{N\times d}" display="inline"><semantics id="S2.SS1.p1.4.m4.1a"><mrow id="S2.SS1.p1.4.m4.1.1" xref="S2.SS1.p1.4.m4.1.1.cmml"><mi id="S2.SS1.p1.4.m4.1.1.2" xref="S2.SS1.p1.4.m4.1.1.2.cmml">𝐊</mi><mo id="S2.SS1.p1.4.m4.1.1.1" xref="S2.SS1.p1.4.m4.1.1.1.cmml">∈</mo><msup id="S2.SS1.p1.4.m4.1.1.3" xref="S2.SS1.p1.4.m4.1.1.3.cmml"><mi id="S2.SS1.p1.4.m4.1.1.3.2" xref="S2.SS1.p1.4.m4.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS1.p1.4.m4.1.1.3.3" xref="S2.SS1.p1.4.m4.1.1.3.3.cmml"><mi id="S2.SS1.p1.4.m4.1.1.3.3.2" xref="S2.SS1.p1.4.m4.1.1.3.3.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.p1.4.m4.1.1.3.3.1" xref="S2.SS1.p1.4.m4.1.1.3.3.1.cmml">×</mo><mi id="S2.SS1.p1.4.m4.1.1.3.3.3" xref="S2.SS1.p1.4.m4.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.4.m4.1b"><apply id="S2.SS1.p1.4.m4.1.1.cmml" xref="S2.SS1.p1.4.m4.1.1"><in id="S2.SS1.p1.4.m4.1.1.1.cmml" xref="S2.SS1.p1.4.m4.1.1.1"></in><ci id="S2.SS1.p1.4.m4.1.1.2.cmml" xref="S2.SS1.p1.4.m4.1.1.2">𝐊</ci><apply id="S2.SS1.p1.4.m4.1.1.3.cmml" xref="S2.SS1.p1.4.m4.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p1.4.m4.1.1.3.1.cmml" xref="S2.SS1.p1.4.m4.1.1.3">superscript</csymbol><ci id="S2.SS1.p1.4.m4.1.1.3.2.cmml" xref="S2.SS1.p1.4.m4.1.1.3.2">ℝ</ci><apply id="S2.SS1.p1.4.m4.1.1.3.3.cmml" xref="S2.SS1.p1.4.m4.1.1.3.3"><times id="S2.SS1.p1.4.m4.1.1.3.3.1.cmml" xref="S2.SS1.p1.4.m4.1.1.3.3.1"></times><ci id="S2.SS1.p1.4.m4.1.1.3.3.2.cmml" xref="S2.SS1.p1.4.m4.1.1.3.3.2">𝑁</ci><ci id="S2.SS1.p1.4.m4.1.1.3.3.3.cmml" xref="S2.SS1.p1.4.m4.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.4.m4.1c">\mathbf{K}\in\mathbb{R}^{N\times d}</annotation></semantics></math> and <math id="S2.SS1.p1.5.m5.1" class="ltx_Math" alttext="\mathbf{V}\in\mathbb{R}^{N\times d}" display="inline"><semantics id="S2.SS1.p1.5.m5.1a"><mrow id="S2.SS1.p1.5.m5.1.1" xref="S2.SS1.p1.5.m5.1.1.cmml"><mi id="S2.SS1.p1.5.m5.1.1.2" xref="S2.SS1.p1.5.m5.1.1.2.cmml">𝐕</mi><mo id="S2.SS1.p1.5.m5.1.1.1" xref="S2.SS1.p1.5.m5.1.1.1.cmml">∈</mo><msup id="S2.SS1.p1.5.m5.1.1.3" xref="S2.SS1.p1.5.m5.1.1.3.cmml"><mi id="S2.SS1.p1.5.m5.1.1.3.2" xref="S2.SS1.p1.5.m5.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS1.p1.5.m5.1.1.3.3" xref="S2.SS1.p1.5.m5.1.1.3.3.cmml"><mi id="S2.SS1.p1.5.m5.1.1.3.3.2" xref="S2.SS1.p1.5.m5.1.1.3.3.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.p1.5.m5.1.1.3.3.1" xref="S2.SS1.p1.5.m5.1.1.3.3.1.cmml">×</mo><mi id="S2.SS1.p1.5.m5.1.1.3.3.3" xref="S2.SS1.p1.5.m5.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.5.m5.1b"><apply id="S2.SS1.p1.5.m5.1.1.cmml" xref="S2.SS1.p1.5.m5.1.1"><in id="S2.SS1.p1.5.m5.1.1.1.cmml" xref="S2.SS1.p1.5.m5.1.1.1"></in><ci id="S2.SS1.p1.5.m5.1.1.2.cmml" xref="S2.SS1.p1.5.m5.1.1.2">𝐕</ci><apply id="S2.SS1.p1.5.m5.1.1.3.cmml" xref="S2.SS1.p1.5.m5.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p1.5.m5.1.1.3.1.cmml" xref="S2.SS1.p1.5.m5.1.1.3">superscript</csymbol><ci id="S2.SS1.p1.5.m5.1.1.3.2.cmml" xref="S2.SS1.p1.5.m5.1.1.3.2">ℝ</ci><apply id="S2.SS1.p1.5.m5.1.1.3.3.cmml" xref="S2.SS1.p1.5.m5.1.1.3.3"><times id="S2.SS1.p1.5.m5.1.1.3.3.1.cmml" xref="S2.SS1.p1.5.m5.1.1.3.3.1"></times><ci id="S2.SS1.p1.5.m5.1.1.3.3.2.cmml" xref="S2.SS1.p1.5.m5.1.1.3.3.2">𝑁</ci><ci id="S2.SS1.p1.5.m5.1.1.3.3.3.cmml" xref="S2.SS1.p1.5.m5.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.5.m5.1c">\mathbf{V}\in\mathbb{R}^{N\times d}</annotation></semantics></math> indicate the embeddings of the key sequence and the value sequence, respectively. <math id="S2.SS1.p1.6.m6.1" class="ltx_Math" alttext="\mathbf{S}\in\mathbb{R}^{N\times N}" display="inline"><semantics id="S2.SS1.p1.6.m6.1a"><mrow id="S2.SS1.p1.6.m6.1.1" xref="S2.SS1.p1.6.m6.1.1.cmml"><mi id="S2.SS1.p1.6.m6.1.1.2" xref="S2.SS1.p1.6.m6.1.1.2.cmml">𝐒</mi><mo id="S2.SS1.p1.6.m6.1.1.1" xref="S2.SS1.p1.6.m6.1.1.1.cmml">∈</mo><msup id="S2.SS1.p1.6.m6.1.1.3" xref="S2.SS1.p1.6.m6.1.1.3.cmml"><mi id="S2.SS1.p1.6.m6.1.1.3.2" xref="S2.SS1.p1.6.m6.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS1.p1.6.m6.1.1.3.3" xref="S2.SS1.p1.6.m6.1.1.3.3.cmml"><mi id="S2.SS1.p1.6.m6.1.1.3.3.2" xref="S2.SS1.p1.6.m6.1.1.3.3.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.p1.6.m6.1.1.3.3.1" xref="S2.SS1.p1.6.m6.1.1.3.3.1.cmml">×</mo><mi id="S2.SS1.p1.6.m6.1.1.3.3.3" xref="S2.SS1.p1.6.m6.1.1.3.3.3.cmml">N</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.6.m6.1b"><apply id="S2.SS1.p1.6.m6.1.1.cmml" xref="S2.SS1.p1.6.m6.1.1"><in id="S2.SS1.p1.6.m6.1.1.1.cmml" xref="S2.SS1.p1.6.m6.1.1.1"></in><ci id="S2.SS1.p1.6.m6.1.1.2.cmml" xref="S2.SS1.p1.6.m6.1.1.2">𝐒</ci><apply id="S2.SS1.p1.6.m6.1.1.3.cmml" xref="S2.SS1.p1.6.m6.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p1.6.m6.1.1.3.1.cmml" xref="S2.SS1.p1.6.m6.1.1.3">superscript</csymbol><ci id="S2.SS1.p1.6.m6.1.1.3.2.cmml" xref="S2.SS1.p1.6.m6.1.1.3.2">ℝ</ci><apply id="S2.SS1.p1.6.m6.1.1.3.3.cmml" xref="S2.SS1.p1.6.m6.1.1.3.3"><times id="S2.SS1.p1.6.m6.1.1.3.3.1.cmml" xref="S2.SS1.p1.6.m6.1.1.3.3.1"></times><ci id="S2.SS1.p1.6.m6.1.1.3.3.2.cmml" xref="S2.SS1.p1.6.m6.1.1.3.3.2">𝑁</ci><ci id="S2.SS1.p1.6.m6.1.1.3.3.3.cmml" xref="S2.SS1.p1.6.m6.1.1.3.3.3">𝑁</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.6.m6.1c">\mathbf{S}\in\mathbb{R}^{N\times N}</annotation></semantics></math> and <math id="S2.SS1.p1.7.m7.1" class="ltx_Math" alttext="\mathbf{P}\in\mathbb{R}^{N\times N}" display="inline"><semantics id="S2.SS1.p1.7.m7.1a"><mrow id="S2.SS1.p1.7.m7.1.1" xref="S2.SS1.p1.7.m7.1.1.cmml"><mi id="S2.SS1.p1.7.m7.1.1.2" xref="S2.SS1.p1.7.m7.1.1.2.cmml">𝐏</mi><mo id="S2.SS1.p1.7.m7.1.1.1" xref="S2.SS1.p1.7.m7.1.1.1.cmml">∈</mo><msup id="S2.SS1.p1.7.m7.1.1.3" xref="S2.SS1.p1.7.m7.1.1.3.cmml"><mi id="S2.SS1.p1.7.m7.1.1.3.2" xref="S2.SS1.p1.7.m7.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS1.p1.7.m7.1.1.3.3" xref="S2.SS1.p1.7.m7.1.1.3.3.cmml"><mi id="S2.SS1.p1.7.m7.1.1.3.3.2" xref="S2.SS1.p1.7.m7.1.1.3.3.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.p1.7.m7.1.1.3.3.1" xref="S2.SS1.p1.7.m7.1.1.3.3.1.cmml">×</mo><mi id="S2.SS1.p1.7.m7.1.1.3.3.3" xref="S2.SS1.p1.7.m7.1.1.3.3.3.cmml">N</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.7.m7.1b"><apply id="S2.SS1.p1.7.m7.1.1.cmml" xref="S2.SS1.p1.7.m7.1.1"><in id="S2.SS1.p1.7.m7.1.1.1.cmml" xref="S2.SS1.p1.7.m7.1.1.1"></in><ci id="S2.SS1.p1.7.m7.1.1.2.cmml" xref="S2.SS1.p1.7.m7.1.1.2">𝐏</ci><apply id="S2.SS1.p1.7.m7.1.1.3.cmml" xref="S2.SS1.p1.7.m7.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p1.7.m7.1.1.3.1.cmml" xref="S2.SS1.p1.7.m7.1.1.3">superscript</csymbol><ci id="S2.SS1.p1.7.m7.1.1.3.2.cmml" xref="S2.SS1.p1.7.m7.1.1.3.2">ℝ</ci><apply id="S2.SS1.p1.7.m7.1.1.3.3.cmml" xref="S2.SS1.p1.7.m7.1.1.3.3"><times id="S2.SS1.p1.7.m7.1.1.3.3.1.cmml" xref="S2.SS1.p1.7.m7.1.1.3.3.1"></times><ci id="S2.SS1.p1.7.m7.1.1.3.3.2.cmml" xref="S2.SS1.p1.7.m7.1.1.3.3.2">𝑁</ci><ci id="S2.SS1.p1.7.m7.1.1.3.3.3.cmml" xref="S2.SS1.p1.7.m7.1.1.3.3.3">𝑁</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.7.m7.1c">\mathbf{P}\in\mathbb{R}^{N\times N}</annotation></semantics></math> indicate the attention scores and the attention probabilities, respectively.
<math id="S2.SS1.p1.8.m8.1" class="ltx_Math" alttext="\mathbf{O}\in\mathbb{R}^{N\times d}" display="inline"><semantics id="S2.SS1.p1.8.m8.1a"><mrow id="S2.SS1.p1.8.m8.1.1" xref="S2.SS1.p1.8.m8.1.1.cmml"><mi id="S2.SS1.p1.8.m8.1.1.2" xref="S2.SS1.p1.8.m8.1.1.2.cmml">𝐎</mi><mo id="S2.SS1.p1.8.m8.1.1.1" xref="S2.SS1.p1.8.m8.1.1.1.cmml">∈</mo><msup id="S2.SS1.p1.8.m8.1.1.3" xref="S2.SS1.p1.8.m8.1.1.3.cmml"><mi id="S2.SS1.p1.8.m8.1.1.3.2" xref="S2.SS1.p1.8.m8.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS1.p1.8.m8.1.1.3.3" xref="S2.SS1.p1.8.m8.1.1.3.3.cmml"><mi id="S2.SS1.p1.8.m8.1.1.3.3.2" xref="S2.SS1.p1.8.m8.1.1.3.3.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS1.p1.8.m8.1.1.3.3.1" xref="S2.SS1.p1.8.m8.1.1.3.3.1.cmml">×</mo><mi id="S2.SS1.p1.8.m8.1.1.3.3.3" xref="S2.SS1.p1.8.m8.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.8.m8.1b"><apply id="S2.SS1.p1.8.m8.1.1.cmml" xref="S2.SS1.p1.8.m8.1.1"><in id="S2.SS1.p1.8.m8.1.1.1.cmml" xref="S2.SS1.p1.8.m8.1.1.1"></in><ci id="S2.SS1.p1.8.m8.1.1.2.cmml" xref="S2.SS1.p1.8.m8.1.1.2">𝐎</ci><apply id="S2.SS1.p1.8.m8.1.1.3.cmml" xref="S2.SS1.p1.8.m8.1.1.3"><csymbol cd="ambiguous" id="S2.SS1.p1.8.m8.1.1.3.1.cmml" xref="S2.SS1.p1.8.m8.1.1.3">superscript</csymbol><ci id="S2.SS1.p1.8.m8.1.1.3.2.cmml" xref="S2.SS1.p1.8.m8.1.1.3.2">ℝ</ci><apply id="S2.SS1.p1.8.m8.1.1.3.3.cmml" xref="S2.SS1.p1.8.m8.1.1.3.3"><times id="S2.SS1.p1.8.m8.1.1.3.3.1.cmml" xref="S2.SS1.p1.8.m8.1.1.3.3.1"></times><ci id="S2.SS1.p1.8.m8.1.1.3.3.2.cmml" xref="S2.SS1.p1.8.m8.1.1.3.3.2">𝑁</ci><ci id="S2.SS1.p1.8.m8.1.1.3.3.3.cmml" xref="S2.SS1.p1.8.m8.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.8.m8.1c">\mathbf{O}\in\mathbb{R}^{N\times d}</annotation></semantics></math> is the final attention result, which is the average of the value sequence embeddings weighted by the similarities between the query and key sequences. In this paper, we mainly use self-attention modules to illustrate BurstAttention, but BurstAttention can be easily extended to cross-attention modules.
For more details of various attention modules in the Transformer architecture, we recommend referring to the original paper of Transformers <cite class="ltx_cite ltx_citemacro_citep">(Vaswani et al., <a href="#bib.bib27" title="" class="ltx_ref">2017</a>)</cite>, and we will not go into details.</p>
</div>
<figure id="alg1" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<figcaption class="ltx_caption" style="font-size:80%;"><span class="ltx_tag ltx_tag_float"><span id="alg1.4.1.1" class="ltx_text ltx_font_bold">Algorithm 1</span> </span> The forward pass of GAO</figcaption>
<div id="alg1.5" class="ltx_listing ltx_listing">
<div id="alg1.l0" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l0.1.1.1" class="ltx_text" style="font-size:80%;">0:</span></span><span id="alg1.l0.2" class="ltx_text" style="font-size:80%;">  Matrices </span><math id="alg1.l0.m1.3" class="ltx_Math" alttext="\mathbf{Q}_{i},\mathbf{K}_{i},\mathbf{V}_{i}\in\mathbb{R}^{{\frac{N}{G}}\times d}" display="inline"><semantics id="alg1.l0.m1.3a"><mrow id="alg1.l0.m1.3.3" xref="alg1.l0.m1.3.3.cmml"><mrow id="alg1.l0.m1.3.3.3.3" xref="alg1.l0.m1.3.3.3.4.cmml"><msub id="alg1.l0.m1.1.1.1.1.1" xref="alg1.l0.m1.1.1.1.1.1.cmml"><mi mathsize="80%" id="alg1.l0.m1.1.1.1.1.1.2" xref="alg1.l0.m1.1.1.1.1.1.2.cmml">𝐐</mi><mi mathsize="80%" id="alg1.l0.m1.1.1.1.1.1.3" xref="alg1.l0.m1.1.1.1.1.1.3.cmml">i</mi></msub><mo mathsize="80%" id="alg1.l0.m1.3.3.3.3.4" xref="alg1.l0.m1.3.3.3.4.cmml">,</mo><msub id="alg1.l0.m1.2.2.2.2.2" xref="alg1.l0.m1.2.2.2.2.2.cmml"><mi mathsize="80%" id="alg1.l0.m1.2.2.2.2.2.2" xref="alg1.l0.m1.2.2.2.2.2.2.cmml">𝐊</mi><mi mathsize="80%" id="alg1.l0.m1.2.2.2.2.2.3" xref="alg1.l0.m1.2.2.2.2.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg1.l0.m1.3.3.3.3.5" xref="alg1.l0.m1.3.3.3.4.cmml">,</mo><msub id="alg1.l0.m1.3.3.3.3.3" xref="alg1.l0.m1.3.3.3.3.3.cmml"><mi mathsize="80%" id="alg1.l0.m1.3.3.3.3.3.2" xref="alg1.l0.m1.3.3.3.3.3.2.cmml">𝐕</mi><mi mathsize="80%" id="alg1.l0.m1.3.3.3.3.3.3" xref="alg1.l0.m1.3.3.3.3.3.3.cmml">i</mi></msub></mrow><mo mathsize="80%" id="alg1.l0.m1.3.3.4" xref="alg1.l0.m1.3.3.4.cmml">∈</mo><msup id="alg1.l0.m1.3.3.5" xref="alg1.l0.m1.3.3.5.cmml"><mi mathsize="80%" id="alg1.l0.m1.3.3.5.2" xref="alg1.l0.m1.3.3.5.2.cmml">ℝ</mi><mrow id="alg1.l0.m1.3.3.5.3" xref="alg1.l0.m1.3.3.5.3.cmml"><mfrac id="alg1.l0.m1.3.3.5.3.2" xref="alg1.l0.m1.3.3.5.3.2.cmml"><mi mathsize="80%" id="alg1.l0.m1.3.3.5.3.2.2" xref="alg1.l0.m1.3.3.5.3.2.2.cmml">N</mi><mi mathsize="80%" id="alg1.l0.m1.3.3.5.3.2.3" xref="alg1.l0.m1.3.3.5.3.2.3.cmml">G</mi></mfrac><mo lspace="0.222em" mathsize="80%" rspace="0.222em" id="alg1.l0.m1.3.3.5.3.1" xref="alg1.l0.m1.3.3.5.3.1.cmml">×</mo><mi mathsize="80%" id="alg1.l0.m1.3.3.5.3.3" xref="alg1.l0.m1.3.3.5.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="alg1.l0.m1.3b"><apply id="alg1.l0.m1.3.3.cmml" xref="alg1.l0.m1.3.3"><in id="alg1.l0.m1.3.3.4.cmml" xref="alg1.l0.m1.3.3.4"></in><list id="alg1.l0.m1.3.3.3.4.cmml" xref="alg1.l0.m1.3.3.3.3"><apply id="alg1.l0.m1.1.1.1.1.1.cmml" xref="alg1.l0.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l0.m1.1.1.1.1.1.1.cmml" xref="alg1.l0.m1.1.1.1.1.1">subscript</csymbol><ci id="alg1.l0.m1.1.1.1.1.1.2.cmml" xref="alg1.l0.m1.1.1.1.1.1.2">𝐐</ci><ci id="alg1.l0.m1.1.1.1.1.1.3.cmml" xref="alg1.l0.m1.1.1.1.1.1.3">𝑖</ci></apply><apply id="alg1.l0.m1.2.2.2.2.2.cmml" xref="alg1.l0.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="alg1.l0.m1.2.2.2.2.2.1.cmml" xref="alg1.l0.m1.2.2.2.2.2">subscript</csymbol><ci id="alg1.l0.m1.2.2.2.2.2.2.cmml" xref="alg1.l0.m1.2.2.2.2.2.2">𝐊</ci><ci id="alg1.l0.m1.2.2.2.2.2.3.cmml" xref="alg1.l0.m1.2.2.2.2.2.3">𝑖</ci></apply><apply id="alg1.l0.m1.3.3.3.3.3.cmml" xref="alg1.l0.m1.3.3.3.3.3"><csymbol cd="ambiguous" id="alg1.l0.m1.3.3.3.3.3.1.cmml" xref="alg1.l0.m1.3.3.3.3.3">subscript</csymbol><ci id="alg1.l0.m1.3.3.3.3.3.2.cmml" xref="alg1.l0.m1.3.3.3.3.3.2">𝐕</ci><ci id="alg1.l0.m1.3.3.3.3.3.3.cmml" xref="alg1.l0.m1.3.3.3.3.3.3">𝑖</ci></apply></list><apply id="alg1.l0.m1.3.3.5.cmml" xref="alg1.l0.m1.3.3.5"><csymbol cd="ambiguous" id="alg1.l0.m1.3.3.5.1.cmml" xref="alg1.l0.m1.3.3.5">superscript</csymbol><ci id="alg1.l0.m1.3.3.5.2.cmml" xref="alg1.l0.m1.3.3.5.2">ℝ</ci><apply id="alg1.l0.m1.3.3.5.3.cmml" xref="alg1.l0.m1.3.3.5.3"><times id="alg1.l0.m1.3.3.5.3.1.cmml" xref="alg1.l0.m1.3.3.5.3.1"></times><apply id="alg1.l0.m1.3.3.5.3.2.cmml" xref="alg1.l0.m1.3.3.5.3.2"><divide id="alg1.l0.m1.3.3.5.3.2.1.cmml" xref="alg1.l0.m1.3.3.5.3.2"></divide><ci id="alg1.l0.m1.3.3.5.3.2.2.cmml" xref="alg1.l0.m1.3.3.5.3.2.2">𝑁</ci><ci id="alg1.l0.m1.3.3.5.3.2.3.cmml" xref="alg1.l0.m1.3.3.5.3.2.3">𝐺</ci></apply><ci id="alg1.l0.m1.3.3.5.3.3.cmml" xref="alg1.l0.m1.3.3.5.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l0.m1.3c">\mathbf{Q}_{i},\mathbf{K}_{i},\mathbf{V}_{i}\in\mathbb{R}^{{\frac{N}{G}}\times d}</annotation></semantics></math><span id="alg1.l0.3" class="ltx_text" style="font-size:80%;"> on the </span><math id="alg1.l0.m2.1" class="ltx_Math" alttext="i" display="inline"><semantics id="alg1.l0.m2.1a"><mi mathsize="80%" id="alg1.l0.m2.1.1" xref="alg1.l0.m2.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="alg1.l0.m2.1b"><ci id="alg1.l0.m2.1.1.cmml" xref="alg1.l0.m2.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l0.m2.1c">i</annotation></semantics></math><span id="alg1.l0.4" class="ltx_text" style="font-size:80%;">-th device
</span>
</div>
<div id="alg1.l1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l1.1.1.1" class="ltx_text" style="font-size:80%;">1:</span></span><span id="alg1.l1.2" class="ltx_text" style="font-size:80%;">  Initialize </span><math id="alg1.l1.m1.4" class="ltx_Math" alttext="\mathbf{O}_{i}=(0)_{{\frac{N}{G}}\times d},l_{i}=(0)_{\frac{N}{G}},m_{i}=(-\infty)_{\frac{N}{G}}" display="inline"><semantics id="alg1.l1.m1.4a"><mrow id="alg1.l1.m1.4.4.2" xref="alg1.l1.m1.4.4.3.cmml"><mrow id="alg1.l1.m1.3.3.1.1" xref="alg1.l1.m1.3.3.1.1.cmml"><msub id="alg1.l1.m1.3.3.1.1.2" xref="alg1.l1.m1.3.3.1.1.2.cmml"><mi mathsize="80%" id="alg1.l1.m1.3.3.1.1.2.2" xref="alg1.l1.m1.3.3.1.1.2.2.cmml">𝐎</mi><mi mathsize="80%" id="alg1.l1.m1.3.3.1.1.2.3" xref="alg1.l1.m1.3.3.1.1.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg1.l1.m1.3.3.1.1.1" xref="alg1.l1.m1.3.3.1.1.1.cmml">=</mo><msub id="alg1.l1.m1.3.3.1.1.3" xref="alg1.l1.m1.3.3.1.1.3.cmml"><mrow id="alg1.l1.m1.3.3.1.1.3.2.2" xref="alg1.l1.m1.3.3.1.1.3.cmml"><mo maxsize="80%" minsize="80%" id="alg1.l1.m1.3.3.1.1.3.2.2.1" xref="alg1.l1.m1.3.3.1.1.3.cmml">(</mo><mn mathsize="80%" id="alg1.l1.m1.1.1" xref="alg1.l1.m1.1.1.cmml">0</mn><mo maxsize="80%" minsize="80%" id="alg1.l1.m1.3.3.1.1.3.2.2.2" xref="alg1.l1.m1.3.3.1.1.3.cmml">)</mo></mrow><mrow id="alg1.l1.m1.3.3.1.1.3.3" xref="alg1.l1.m1.3.3.1.1.3.3.cmml"><mfrac id="alg1.l1.m1.3.3.1.1.3.3.2" xref="alg1.l1.m1.3.3.1.1.3.3.2.cmml"><mi mathsize="80%" id="alg1.l1.m1.3.3.1.1.3.3.2.2" xref="alg1.l1.m1.3.3.1.1.3.3.2.2.cmml">N</mi><mi mathsize="80%" id="alg1.l1.m1.3.3.1.1.3.3.2.3" xref="alg1.l1.m1.3.3.1.1.3.3.2.3.cmml">G</mi></mfrac><mo lspace="0.222em" mathsize="80%" rspace="0.222em" id="alg1.l1.m1.3.3.1.1.3.3.1" xref="alg1.l1.m1.3.3.1.1.3.3.1.cmml">×</mo><mi mathsize="80%" id="alg1.l1.m1.3.3.1.1.3.3.3" xref="alg1.l1.m1.3.3.1.1.3.3.3.cmml">d</mi></mrow></msub></mrow><mo mathsize="80%" id="alg1.l1.m1.4.4.2.3" xref="alg1.l1.m1.4.4.3a.cmml">,</mo><mrow id="alg1.l1.m1.4.4.2.2.2" xref="alg1.l1.m1.4.4.2.2.3.cmml"><mrow id="alg1.l1.m1.4.4.2.2.1.1" xref="alg1.l1.m1.4.4.2.2.1.1.cmml"><msub id="alg1.l1.m1.4.4.2.2.1.1.2" xref="alg1.l1.m1.4.4.2.2.1.1.2.cmml"><mi mathsize="80%" id="alg1.l1.m1.4.4.2.2.1.1.2.2" xref="alg1.l1.m1.4.4.2.2.1.1.2.2.cmml">l</mi><mi mathsize="80%" id="alg1.l1.m1.4.4.2.2.1.1.2.3" xref="alg1.l1.m1.4.4.2.2.1.1.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg1.l1.m1.4.4.2.2.1.1.1" xref="alg1.l1.m1.4.4.2.2.1.1.1.cmml">=</mo><msub id="alg1.l1.m1.4.4.2.2.1.1.3" xref="alg1.l1.m1.4.4.2.2.1.1.3.cmml"><mrow id="alg1.l1.m1.4.4.2.2.1.1.3.2.2" xref="alg1.l1.m1.4.4.2.2.1.1.3.cmml"><mo maxsize="80%" minsize="80%" id="alg1.l1.m1.4.4.2.2.1.1.3.2.2.1" xref="alg1.l1.m1.4.4.2.2.1.1.3.cmml">(</mo><mn mathsize="80%" id="alg1.l1.m1.2.2" xref="alg1.l1.m1.2.2.cmml">0</mn><mo maxsize="80%" minsize="80%" id="alg1.l1.m1.4.4.2.2.1.1.3.2.2.2" xref="alg1.l1.m1.4.4.2.2.1.1.3.cmml">)</mo></mrow><mfrac id="alg1.l1.m1.4.4.2.2.1.1.3.3" xref="alg1.l1.m1.4.4.2.2.1.1.3.3.cmml"><mi mathsize="80%" id="alg1.l1.m1.4.4.2.2.1.1.3.3.2" xref="alg1.l1.m1.4.4.2.2.1.1.3.3.2.cmml">N</mi><mi mathsize="80%" id="alg1.l1.m1.4.4.2.2.1.1.3.3.3" xref="alg1.l1.m1.4.4.2.2.1.1.3.3.3.cmml">G</mi></mfrac></msub></mrow><mo mathsize="80%" id="alg1.l1.m1.4.4.2.2.2.3" xref="alg1.l1.m1.4.4.2.2.3a.cmml">,</mo><mrow id="alg1.l1.m1.4.4.2.2.2.2" xref="alg1.l1.m1.4.4.2.2.2.2.cmml"><msub id="alg1.l1.m1.4.4.2.2.2.2.3" xref="alg1.l1.m1.4.4.2.2.2.2.3.cmml"><mi mathsize="80%" id="alg1.l1.m1.4.4.2.2.2.2.3.2" xref="alg1.l1.m1.4.4.2.2.2.2.3.2.cmml">m</mi><mi mathsize="80%" id="alg1.l1.m1.4.4.2.2.2.2.3.3" xref="alg1.l1.m1.4.4.2.2.2.2.3.3.cmml">i</mi></msub><mo mathsize="80%" id="alg1.l1.m1.4.4.2.2.2.2.2" xref="alg1.l1.m1.4.4.2.2.2.2.2.cmml">=</mo><msub id="alg1.l1.m1.4.4.2.2.2.2.1" xref="alg1.l1.m1.4.4.2.2.2.2.1.cmml"><mrow id="alg1.l1.m1.4.4.2.2.2.2.1.1.1" xref="alg1.l1.m1.4.4.2.2.2.2.1.1.1.1.cmml"><mo maxsize="80%" minsize="80%" id="alg1.l1.m1.4.4.2.2.2.2.1.1.1.2" xref="alg1.l1.m1.4.4.2.2.2.2.1.1.1.1.cmml">(</mo><mrow id="alg1.l1.m1.4.4.2.2.2.2.1.1.1.1" xref="alg1.l1.m1.4.4.2.2.2.2.1.1.1.1.cmml"><mo mathsize="80%" id="alg1.l1.m1.4.4.2.2.2.2.1.1.1.1a" xref="alg1.l1.m1.4.4.2.2.2.2.1.1.1.1.cmml">−</mo><mi mathsize="80%" mathvariant="normal" id="alg1.l1.m1.4.4.2.2.2.2.1.1.1.1.2" xref="alg1.l1.m1.4.4.2.2.2.2.1.1.1.1.2.cmml">∞</mi></mrow><mo maxsize="80%" minsize="80%" id="alg1.l1.m1.4.4.2.2.2.2.1.1.1.3" xref="alg1.l1.m1.4.4.2.2.2.2.1.1.1.1.cmml">)</mo></mrow><mfrac id="alg1.l1.m1.4.4.2.2.2.2.1.3" xref="alg1.l1.m1.4.4.2.2.2.2.1.3.cmml"><mi mathsize="80%" id="alg1.l1.m1.4.4.2.2.2.2.1.3.2" xref="alg1.l1.m1.4.4.2.2.2.2.1.3.2.cmml">N</mi><mi mathsize="80%" id="alg1.l1.m1.4.4.2.2.2.2.1.3.3" xref="alg1.l1.m1.4.4.2.2.2.2.1.3.3.cmml">G</mi></mfrac></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l1.m1.4b"><apply id="alg1.l1.m1.4.4.3.cmml" xref="alg1.l1.m1.4.4.2"><csymbol cd="ambiguous" id="alg1.l1.m1.4.4.3a.cmml" xref="alg1.l1.m1.4.4.2.3">formulae-sequence</csymbol><apply id="alg1.l1.m1.3.3.1.1.cmml" xref="alg1.l1.m1.3.3.1.1"><eq id="alg1.l1.m1.3.3.1.1.1.cmml" xref="alg1.l1.m1.3.3.1.1.1"></eq><apply id="alg1.l1.m1.3.3.1.1.2.cmml" xref="alg1.l1.m1.3.3.1.1.2"><csymbol cd="ambiguous" id="alg1.l1.m1.3.3.1.1.2.1.cmml" xref="alg1.l1.m1.3.3.1.1.2">subscript</csymbol><ci id="alg1.l1.m1.3.3.1.1.2.2.cmml" xref="alg1.l1.m1.3.3.1.1.2.2">𝐎</ci><ci id="alg1.l1.m1.3.3.1.1.2.3.cmml" xref="alg1.l1.m1.3.3.1.1.2.3">𝑖</ci></apply><apply id="alg1.l1.m1.3.3.1.1.3.cmml" xref="alg1.l1.m1.3.3.1.1.3"><csymbol cd="ambiguous" id="alg1.l1.m1.3.3.1.1.3.1.cmml" xref="alg1.l1.m1.3.3.1.1.3">subscript</csymbol><cn type="integer" id="alg1.l1.m1.1.1.cmml" xref="alg1.l1.m1.1.1">0</cn><apply id="alg1.l1.m1.3.3.1.1.3.3.cmml" xref="alg1.l1.m1.3.3.1.1.3.3"><times id="alg1.l1.m1.3.3.1.1.3.3.1.cmml" xref="alg1.l1.m1.3.3.1.1.3.3.1"></times><apply id="alg1.l1.m1.3.3.1.1.3.3.2.cmml" xref="alg1.l1.m1.3.3.1.1.3.3.2"><divide id="alg1.l1.m1.3.3.1.1.3.3.2.1.cmml" xref="alg1.l1.m1.3.3.1.1.3.3.2"></divide><ci id="alg1.l1.m1.3.3.1.1.3.3.2.2.cmml" xref="alg1.l1.m1.3.3.1.1.3.3.2.2">𝑁</ci><ci id="alg1.l1.m1.3.3.1.1.3.3.2.3.cmml" xref="alg1.l1.m1.3.3.1.1.3.3.2.3">𝐺</ci></apply><ci id="alg1.l1.m1.3.3.1.1.3.3.3.cmml" xref="alg1.l1.m1.3.3.1.1.3.3.3">𝑑</ci></apply></apply></apply><apply id="alg1.l1.m1.4.4.2.2.3.cmml" xref="alg1.l1.m1.4.4.2.2.2"><csymbol cd="ambiguous" id="alg1.l1.m1.4.4.2.2.3a.cmml" xref="alg1.l1.m1.4.4.2.2.2.3">formulae-sequence</csymbol><apply id="alg1.l1.m1.4.4.2.2.1.1.cmml" xref="alg1.l1.m1.4.4.2.2.1.1"><eq id="alg1.l1.m1.4.4.2.2.1.1.1.cmml" xref="alg1.l1.m1.4.4.2.2.1.1.1"></eq><apply id="alg1.l1.m1.4.4.2.2.1.1.2.cmml" xref="alg1.l1.m1.4.4.2.2.1.1.2"><csymbol cd="ambiguous" id="alg1.l1.m1.4.4.2.2.1.1.2.1.cmml" xref="alg1.l1.m1.4.4.2.2.1.1.2">subscript</csymbol><ci id="alg1.l1.m1.4.4.2.2.1.1.2.2.cmml" xref="alg1.l1.m1.4.4.2.2.1.1.2.2">𝑙</ci><ci id="alg1.l1.m1.4.4.2.2.1.1.2.3.cmml" xref="alg1.l1.m1.4.4.2.2.1.1.2.3">𝑖</ci></apply><apply id="alg1.l1.m1.4.4.2.2.1.1.3.cmml" xref="alg1.l1.m1.4.4.2.2.1.1.3"><csymbol cd="ambiguous" id="alg1.l1.m1.4.4.2.2.1.1.3.1.cmml" xref="alg1.l1.m1.4.4.2.2.1.1.3">subscript</csymbol><cn type="integer" id="alg1.l1.m1.2.2.cmml" xref="alg1.l1.m1.2.2">0</cn><apply id="alg1.l1.m1.4.4.2.2.1.1.3.3.cmml" xref="alg1.l1.m1.4.4.2.2.1.1.3.3"><divide id="alg1.l1.m1.4.4.2.2.1.1.3.3.1.cmml" xref="alg1.l1.m1.4.4.2.2.1.1.3.3"></divide><ci id="alg1.l1.m1.4.4.2.2.1.1.3.3.2.cmml" xref="alg1.l1.m1.4.4.2.2.1.1.3.3.2">𝑁</ci><ci id="alg1.l1.m1.4.4.2.2.1.1.3.3.3.cmml" xref="alg1.l1.m1.4.4.2.2.1.1.3.3.3">𝐺</ci></apply></apply></apply><apply id="alg1.l1.m1.4.4.2.2.2.2.cmml" xref="alg1.l1.m1.4.4.2.2.2.2"><eq id="alg1.l1.m1.4.4.2.2.2.2.2.cmml" xref="alg1.l1.m1.4.4.2.2.2.2.2"></eq><apply id="alg1.l1.m1.4.4.2.2.2.2.3.cmml" xref="alg1.l1.m1.4.4.2.2.2.2.3"><csymbol cd="ambiguous" id="alg1.l1.m1.4.4.2.2.2.2.3.1.cmml" xref="alg1.l1.m1.4.4.2.2.2.2.3">subscript</csymbol><ci id="alg1.l1.m1.4.4.2.2.2.2.3.2.cmml" xref="alg1.l1.m1.4.4.2.2.2.2.3.2">𝑚</ci><ci id="alg1.l1.m1.4.4.2.2.2.2.3.3.cmml" xref="alg1.l1.m1.4.4.2.2.2.2.3.3">𝑖</ci></apply><apply id="alg1.l1.m1.4.4.2.2.2.2.1.cmml" xref="alg1.l1.m1.4.4.2.2.2.2.1"><csymbol cd="ambiguous" id="alg1.l1.m1.4.4.2.2.2.2.1.2.cmml" xref="alg1.l1.m1.4.4.2.2.2.2.1">subscript</csymbol><apply id="alg1.l1.m1.4.4.2.2.2.2.1.1.1.1.cmml" xref="alg1.l1.m1.4.4.2.2.2.2.1.1.1"><minus id="alg1.l1.m1.4.4.2.2.2.2.1.1.1.1.1.cmml" xref="alg1.l1.m1.4.4.2.2.2.2.1.1.1"></minus><infinity id="alg1.l1.m1.4.4.2.2.2.2.1.1.1.1.2.cmml" xref="alg1.l1.m1.4.4.2.2.2.2.1.1.1.1.2"></infinity></apply><apply id="alg1.l1.m1.4.4.2.2.2.2.1.3.cmml" xref="alg1.l1.m1.4.4.2.2.2.2.1.3"><divide id="alg1.l1.m1.4.4.2.2.2.2.1.3.1.cmml" xref="alg1.l1.m1.4.4.2.2.2.2.1.3"></divide><ci id="alg1.l1.m1.4.4.2.2.2.2.1.3.2.cmml" xref="alg1.l1.m1.4.4.2.2.2.2.1.3.2">𝑁</ci><ci id="alg1.l1.m1.4.4.2.2.2.2.1.3.3.cmml" xref="alg1.l1.m1.4.4.2.2.2.2.1.3.3">𝐺</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m1.4c">\mathbf{O}_{i}=(0)_{{\frac{N}{G}}\times d},l_{i}=(0)_{\frac{N}{G}},m_{i}=(-\infty)_{\frac{N}{G}}</annotation></semantics></math><span id="alg1.l1.3" class="ltx_text" style="font-size:80%;">
</span>
</div>
<div id="alg1.l2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l2.1.1.1" class="ltx_text" style="font-size:80%;">2:</span></span><span id="alg1.l2.2" class="ltx_text" style="font-size:80%;">  Put </span><math id="alg1.l2.m1.2" class="ltx_Math" alttext="\mathbf{K}_{i},\mathbf{V}_{i}" display="inline"><semantics id="alg1.l2.m1.2a"><mrow id="alg1.l2.m1.2.2.2" xref="alg1.l2.m1.2.2.3.cmml"><msub id="alg1.l2.m1.1.1.1.1" xref="alg1.l2.m1.1.1.1.1.cmml"><mi mathsize="80%" id="alg1.l2.m1.1.1.1.1.2" xref="alg1.l2.m1.1.1.1.1.2.cmml">𝐊</mi><mi mathsize="80%" id="alg1.l2.m1.1.1.1.1.3" xref="alg1.l2.m1.1.1.1.1.3.cmml">i</mi></msub><mo mathsize="80%" id="alg1.l2.m1.2.2.2.3" xref="alg1.l2.m1.2.2.3.cmml">,</mo><msub id="alg1.l2.m1.2.2.2.2" xref="alg1.l2.m1.2.2.2.2.cmml"><mi mathsize="80%" id="alg1.l2.m1.2.2.2.2.2" xref="alg1.l2.m1.2.2.2.2.2.cmml">𝐕</mi><mi mathsize="80%" id="alg1.l2.m1.2.2.2.2.3" xref="alg1.l2.m1.2.2.2.2.3.cmml">i</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="alg1.l2.m1.2b"><list id="alg1.l2.m1.2.2.3.cmml" xref="alg1.l2.m1.2.2.2"><apply id="alg1.l2.m1.1.1.1.1.cmml" xref="alg1.l2.m1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l2.m1.1.1.1.1.1.cmml" xref="alg1.l2.m1.1.1.1.1">subscript</csymbol><ci id="alg1.l2.m1.1.1.1.1.2.cmml" xref="alg1.l2.m1.1.1.1.1.2">𝐊</ci><ci id="alg1.l2.m1.1.1.1.1.3.cmml" xref="alg1.l2.m1.1.1.1.1.3">𝑖</ci></apply><apply id="alg1.l2.m1.2.2.2.2.cmml" xref="alg1.l2.m1.2.2.2.2"><csymbol cd="ambiguous" id="alg1.l2.m1.2.2.2.2.1.cmml" xref="alg1.l2.m1.2.2.2.2">subscript</csymbol><ci id="alg1.l2.m1.2.2.2.2.2.cmml" xref="alg1.l2.m1.2.2.2.2.2">𝐕</ci><ci id="alg1.l2.m1.2.2.2.2.3.cmml" xref="alg1.l2.m1.2.2.2.2.3">𝑖</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg1.l2.m1.2c">\mathbf{K}_{i},\mathbf{V}_{i}</annotation></semantics></math><span id="alg1.l2.3" class="ltx_text" style="font-size:80%;"> into communication ring
</span>
</div>
<div id="alg1.l3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l3.1.1.1" class="ltx_text" style="font-size:80%;">3:</span></span><span id="alg1.l3.2" class="ltx_text" style="font-size:80%;">  </span><span id="alg1.l3.3" class="ltx_text ltx_font_bold" style="font-size:80%;">for</span><span id="alg1.l3.4" class="ltx_text" style="font-size:80%;"> </span><math id="alg1.l3.m1.1" class="ltx_Math" alttext="j=1" display="inline"><semantics id="alg1.l3.m1.1a"><mrow id="alg1.l3.m1.1.1" xref="alg1.l3.m1.1.1.cmml"><mi mathsize="80%" id="alg1.l3.m1.1.1.2" xref="alg1.l3.m1.1.1.2.cmml">j</mi><mo mathsize="80%" id="alg1.l3.m1.1.1.1" xref="alg1.l3.m1.1.1.1.cmml">=</mo><mn mathsize="80%" id="alg1.l3.m1.1.1.3" xref="alg1.l3.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg1.l3.m1.1b"><apply id="alg1.l3.m1.1.1.cmml" xref="alg1.l3.m1.1.1"><eq id="alg1.l3.m1.1.1.1.cmml" xref="alg1.l3.m1.1.1.1"></eq><ci id="alg1.l3.m1.1.1.2.cmml" xref="alg1.l3.m1.1.1.2">𝑗</ci><cn type="integer" id="alg1.l3.m1.1.1.3.cmml" xref="alg1.l3.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l3.m1.1c">j=1</annotation></semantics></math><span id="alg1.l3.5" class="ltx_text" style="font-size:80%;"> </span><span id="alg1.l3.6" class="ltx_text ltx_font_bold" style="font-size:80%;">to</span><span id="alg1.l3.7" class="ltx_text" style="font-size:80%;"> </span><math id="alg1.l3.m2.1" class="ltx_Math" alttext="G" display="inline"><semantics id="alg1.l3.m2.1a"><mi mathsize="80%" id="alg1.l3.m2.1.1" xref="alg1.l3.m2.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="alg1.l3.m2.1b"><ci id="alg1.l3.m2.1.1.cmml" xref="alg1.l3.m2.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l3.m2.1c">G</annotation></semantics></math><span id="alg1.l3.8" class="ltx_text" style="font-size:80%;"> </span><span id="alg1.l3.9" class="ltx_text ltx_font_bold" style="font-size:80%;">do</span><span id="alg1.l3.10" class="ltx_text" style="font-size:80%;">


</span>
</div>
<div id="alg1.l4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l4.1.1.1" class="ltx_text" style="font-size:80%;">4:</span></span><span id="alg1.l4.2" class="ltx_text" style="font-size:80%;">    Conduct one step of ring communication;
</span>
</div>
<div id="alg1.l5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l5.1.1.1" class="ltx_text" style="font-size:80%;">5:</span></span><span id="alg1.l5.2" class="ltx_text" style="font-size:80%;">    Get </span><math id="alg1.l5.m1.2" class="ltx_Math" alttext="\mathbf{K}_{j},\mathbf{V}_{j}" display="inline"><semantics id="alg1.l5.m1.2a"><mrow id="alg1.l5.m1.2.2.2" xref="alg1.l5.m1.2.2.3.cmml"><msub id="alg1.l5.m1.1.1.1.1" xref="alg1.l5.m1.1.1.1.1.cmml"><mi mathsize="80%" id="alg1.l5.m1.1.1.1.1.2" xref="alg1.l5.m1.1.1.1.1.2.cmml">𝐊</mi><mi mathsize="80%" id="alg1.l5.m1.1.1.1.1.3" xref="alg1.l5.m1.1.1.1.1.3.cmml">j</mi></msub><mo mathsize="80%" id="alg1.l5.m1.2.2.2.3" xref="alg1.l5.m1.2.2.3.cmml">,</mo><msub id="alg1.l5.m1.2.2.2.2" xref="alg1.l5.m1.2.2.2.2.cmml"><mi mathsize="80%" id="alg1.l5.m1.2.2.2.2.2" xref="alg1.l5.m1.2.2.2.2.2.cmml">𝐕</mi><mi mathsize="80%" id="alg1.l5.m1.2.2.2.2.3" xref="alg1.l5.m1.2.2.2.2.3.cmml">j</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="alg1.l5.m1.2b"><list id="alg1.l5.m1.2.2.3.cmml" xref="alg1.l5.m1.2.2.2"><apply id="alg1.l5.m1.1.1.1.1.cmml" xref="alg1.l5.m1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l5.m1.1.1.1.1.1.cmml" xref="alg1.l5.m1.1.1.1.1">subscript</csymbol><ci id="alg1.l5.m1.1.1.1.1.2.cmml" xref="alg1.l5.m1.1.1.1.1.2">𝐊</ci><ci id="alg1.l5.m1.1.1.1.1.3.cmml" xref="alg1.l5.m1.1.1.1.1.3">𝑗</ci></apply><apply id="alg1.l5.m1.2.2.2.2.cmml" xref="alg1.l5.m1.2.2.2.2"><csymbol cd="ambiguous" id="alg1.l5.m1.2.2.2.2.1.cmml" xref="alg1.l5.m1.2.2.2.2">subscript</csymbol><ci id="alg1.l5.m1.2.2.2.2.2.cmml" xref="alg1.l5.m1.2.2.2.2.2">𝐕</ci><ci id="alg1.l5.m1.2.2.2.2.3.cmml" xref="alg1.l5.m1.2.2.2.2.3">𝑗</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m1.2c">\mathbf{K}_{j},\mathbf{V}_{j}</annotation></semantics></math><span id="alg1.l5.3" class="ltx_text" style="font-size:80%;"> from communication ring;
</span>
</div>
<div id="alg1.l6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l6.1.1.1" class="ltx_text" style="font-size:80%;">6:</span></span><span id="alg1.l6.2" class="ltx_text" style="font-size:80%;">    {The forward pass of local attention (w/o LAO).}
</span>
</div>
<div id="alg1.l7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l7.1.1.1" class="ltx_text" style="font-size:80%;">7:</span></span><span id="alg1.l7.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg1.l7.m1.2" class="ltx_Math" alttext="\mathbf{S}_{i,j}=\mathbf{Q}_{i}\mathbf{K}_{j}^{T}" display="inline"><semantics id="alg1.l7.m1.2a"><mrow id="alg1.l7.m1.2.3" xref="alg1.l7.m1.2.3.cmml"><msub id="alg1.l7.m1.2.3.2" xref="alg1.l7.m1.2.3.2.cmml"><mi mathsize="80%" id="alg1.l7.m1.2.3.2.2" xref="alg1.l7.m1.2.3.2.2.cmml">𝐒</mi><mrow id="alg1.l7.m1.2.2.2.4" xref="alg1.l7.m1.2.2.2.3.cmml"><mi mathsize="80%" id="alg1.l7.m1.1.1.1.1" xref="alg1.l7.m1.1.1.1.1.cmml">i</mi><mo mathsize="80%" id="alg1.l7.m1.2.2.2.4.1" xref="alg1.l7.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg1.l7.m1.2.2.2.2" xref="alg1.l7.m1.2.2.2.2.cmml">j</mi></mrow></msub><mo mathsize="80%" id="alg1.l7.m1.2.3.1" xref="alg1.l7.m1.2.3.1.cmml">=</mo><mrow id="alg1.l7.m1.2.3.3" xref="alg1.l7.m1.2.3.3.cmml"><msub id="alg1.l7.m1.2.3.3.2" xref="alg1.l7.m1.2.3.3.2.cmml"><mi mathsize="80%" id="alg1.l7.m1.2.3.3.2.2" xref="alg1.l7.m1.2.3.3.2.2.cmml">𝐐</mi><mi mathsize="80%" id="alg1.l7.m1.2.3.3.2.3" xref="alg1.l7.m1.2.3.3.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="alg1.l7.m1.2.3.3.1" xref="alg1.l7.m1.2.3.3.1.cmml">​</mo><msubsup id="alg1.l7.m1.2.3.3.3" xref="alg1.l7.m1.2.3.3.3.cmml"><mi mathsize="80%" id="alg1.l7.m1.2.3.3.3.2.2" xref="alg1.l7.m1.2.3.3.3.2.2.cmml">𝐊</mi><mi mathsize="80%" id="alg1.l7.m1.2.3.3.3.2.3" xref="alg1.l7.m1.2.3.3.3.2.3.cmml">j</mi><mi mathsize="80%" id="alg1.l7.m1.2.3.3.3.3" xref="alg1.l7.m1.2.3.3.3.3.cmml">T</mi></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l7.m1.2b"><apply id="alg1.l7.m1.2.3.cmml" xref="alg1.l7.m1.2.3"><eq id="alg1.l7.m1.2.3.1.cmml" xref="alg1.l7.m1.2.3.1"></eq><apply id="alg1.l7.m1.2.3.2.cmml" xref="alg1.l7.m1.2.3.2"><csymbol cd="ambiguous" id="alg1.l7.m1.2.3.2.1.cmml" xref="alg1.l7.m1.2.3.2">subscript</csymbol><ci id="alg1.l7.m1.2.3.2.2.cmml" xref="alg1.l7.m1.2.3.2.2">𝐒</ci><list id="alg1.l7.m1.2.2.2.3.cmml" xref="alg1.l7.m1.2.2.2.4"><ci id="alg1.l7.m1.1.1.1.1.cmml" xref="alg1.l7.m1.1.1.1.1">𝑖</ci><ci id="alg1.l7.m1.2.2.2.2.cmml" xref="alg1.l7.m1.2.2.2.2">𝑗</ci></list></apply><apply id="alg1.l7.m1.2.3.3.cmml" xref="alg1.l7.m1.2.3.3"><times id="alg1.l7.m1.2.3.3.1.cmml" xref="alg1.l7.m1.2.3.3.1"></times><apply id="alg1.l7.m1.2.3.3.2.cmml" xref="alg1.l7.m1.2.3.3.2"><csymbol cd="ambiguous" id="alg1.l7.m1.2.3.3.2.1.cmml" xref="alg1.l7.m1.2.3.3.2">subscript</csymbol><ci id="alg1.l7.m1.2.3.3.2.2.cmml" xref="alg1.l7.m1.2.3.3.2.2">𝐐</ci><ci id="alg1.l7.m1.2.3.3.2.3.cmml" xref="alg1.l7.m1.2.3.3.2.3">𝑖</ci></apply><apply id="alg1.l7.m1.2.3.3.3.cmml" xref="alg1.l7.m1.2.3.3.3"><csymbol cd="ambiguous" id="alg1.l7.m1.2.3.3.3.1.cmml" xref="alg1.l7.m1.2.3.3.3">superscript</csymbol><apply id="alg1.l7.m1.2.3.3.3.2.cmml" xref="alg1.l7.m1.2.3.3.3"><csymbol cd="ambiguous" id="alg1.l7.m1.2.3.3.3.2.1.cmml" xref="alg1.l7.m1.2.3.3.3">subscript</csymbol><ci id="alg1.l7.m1.2.3.3.3.2.2.cmml" xref="alg1.l7.m1.2.3.3.3.2.2">𝐊</ci><ci id="alg1.l7.m1.2.3.3.3.2.3.cmml" xref="alg1.l7.m1.2.3.3.3.2.3">𝑗</ci></apply><ci id="alg1.l7.m1.2.3.3.3.3.cmml" xref="alg1.l7.m1.2.3.3.3.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l7.m1.2c">\mathbf{S}_{i,j}=\mathbf{Q}_{i}\mathbf{K}_{j}^{T}</annotation></semantics></math><span id="alg1.l7.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg1.l8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l8.1.1.1" class="ltx_text" style="font-size:80%;">8:</span></span><span id="alg1.l8.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg1.l8.m1.5" class="ltx_Math" alttext="m_{i,j}=\text{rowmax}(\mathbf{S}_{i,j})" display="inline"><semantics id="alg1.l8.m1.5a"><mrow id="alg1.l8.m1.5.5" xref="alg1.l8.m1.5.5.cmml"><msub id="alg1.l8.m1.5.5.3" xref="alg1.l8.m1.5.5.3.cmml"><mi mathsize="80%" id="alg1.l8.m1.5.5.3.2" xref="alg1.l8.m1.5.5.3.2.cmml">m</mi><mrow id="alg1.l8.m1.2.2.2.4" xref="alg1.l8.m1.2.2.2.3.cmml"><mi mathsize="80%" id="alg1.l8.m1.1.1.1.1" xref="alg1.l8.m1.1.1.1.1.cmml">i</mi><mo mathsize="80%" id="alg1.l8.m1.2.2.2.4.1" xref="alg1.l8.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg1.l8.m1.2.2.2.2" xref="alg1.l8.m1.2.2.2.2.cmml">j</mi></mrow></msub><mo mathsize="80%" id="alg1.l8.m1.5.5.2" xref="alg1.l8.m1.5.5.2.cmml">=</mo><mrow id="alg1.l8.m1.5.5.1" xref="alg1.l8.m1.5.5.1.cmml"><mtext mathsize="80%" id="alg1.l8.m1.5.5.1.3" xref="alg1.l8.m1.5.5.1.3a.cmml">rowmax</mtext><mo lspace="0em" rspace="0em" id="alg1.l8.m1.5.5.1.2" xref="alg1.l8.m1.5.5.1.2.cmml">​</mo><mrow id="alg1.l8.m1.5.5.1.1.1" xref="alg1.l8.m1.5.5.1.1.1.1.cmml"><mo maxsize="80%" minsize="80%" id="alg1.l8.m1.5.5.1.1.1.2" xref="alg1.l8.m1.5.5.1.1.1.1.cmml">(</mo><msub id="alg1.l8.m1.5.5.1.1.1.1" xref="alg1.l8.m1.5.5.1.1.1.1.cmml"><mi mathsize="80%" id="alg1.l8.m1.5.5.1.1.1.1.2" xref="alg1.l8.m1.5.5.1.1.1.1.2.cmml">𝐒</mi><mrow id="alg1.l8.m1.4.4.2.4" xref="alg1.l8.m1.4.4.2.3.cmml"><mi mathsize="80%" id="alg1.l8.m1.3.3.1.1" xref="alg1.l8.m1.3.3.1.1.cmml">i</mi><mo mathsize="80%" id="alg1.l8.m1.4.4.2.4.1" xref="alg1.l8.m1.4.4.2.3.cmml">,</mo><mi mathsize="80%" id="alg1.l8.m1.4.4.2.2" xref="alg1.l8.m1.4.4.2.2.cmml">j</mi></mrow></msub><mo maxsize="80%" minsize="80%" id="alg1.l8.m1.5.5.1.1.1.3" xref="alg1.l8.m1.5.5.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l8.m1.5b"><apply id="alg1.l8.m1.5.5.cmml" xref="alg1.l8.m1.5.5"><eq id="alg1.l8.m1.5.5.2.cmml" xref="alg1.l8.m1.5.5.2"></eq><apply id="alg1.l8.m1.5.5.3.cmml" xref="alg1.l8.m1.5.5.3"><csymbol cd="ambiguous" id="alg1.l8.m1.5.5.3.1.cmml" xref="alg1.l8.m1.5.5.3">subscript</csymbol><ci id="alg1.l8.m1.5.5.3.2.cmml" xref="alg1.l8.m1.5.5.3.2">𝑚</ci><list id="alg1.l8.m1.2.2.2.3.cmml" xref="alg1.l8.m1.2.2.2.4"><ci id="alg1.l8.m1.1.1.1.1.cmml" xref="alg1.l8.m1.1.1.1.1">𝑖</ci><ci id="alg1.l8.m1.2.2.2.2.cmml" xref="alg1.l8.m1.2.2.2.2">𝑗</ci></list></apply><apply id="alg1.l8.m1.5.5.1.cmml" xref="alg1.l8.m1.5.5.1"><times id="alg1.l8.m1.5.5.1.2.cmml" xref="alg1.l8.m1.5.5.1.2"></times><ci id="alg1.l8.m1.5.5.1.3a.cmml" xref="alg1.l8.m1.5.5.1.3"><mtext mathsize="80%" id="alg1.l8.m1.5.5.1.3.cmml" xref="alg1.l8.m1.5.5.1.3">rowmax</mtext></ci><apply id="alg1.l8.m1.5.5.1.1.1.1.cmml" xref="alg1.l8.m1.5.5.1.1.1"><csymbol cd="ambiguous" id="alg1.l8.m1.5.5.1.1.1.1.1.cmml" xref="alg1.l8.m1.5.5.1.1.1">subscript</csymbol><ci id="alg1.l8.m1.5.5.1.1.1.1.2.cmml" xref="alg1.l8.m1.5.5.1.1.1.1.2">𝐒</ci><list id="alg1.l8.m1.4.4.2.3.cmml" xref="alg1.l8.m1.4.4.2.4"><ci id="alg1.l8.m1.3.3.1.1.cmml" xref="alg1.l8.m1.3.3.1.1">𝑖</ci><ci id="alg1.l8.m1.4.4.2.2.cmml" xref="alg1.l8.m1.4.4.2.2">𝑗</ci></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l8.m1.5c">m_{i,j}=\text{rowmax}(\mathbf{S}_{i,j})</annotation></semantics></math><span id="alg1.l8.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg1.l9" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l9.1.1.1" class="ltx_text" style="font-size:80%;">9:</span></span><span id="alg1.l9.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg1.l9.m1.7" class="ltx_Math" alttext="\mathbf{P}_{i,j}=\text{exp}(\mathbf{S}_{i,j}-m_{i,j})" display="inline"><semantics id="alg1.l9.m1.7a"><mrow id="alg1.l9.m1.7.7" xref="alg1.l9.m1.7.7.cmml"><msub id="alg1.l9.m1.7.7.3" xref="alg1.l9.m1.7.7.3.cmml"><mi mathsize="80%" id="alg1.l9.m1.7.7.3.2" xref="alg1.l9.m1.7.7.3.2.cmml">𝐏</mi><mrow id="alg1.l9.m1.2.2.2.4" xref="alg1.l9.m1.2.2.2.3.cmml"><mi mathsize="80%" id="alg1.l9.m1.1.1.1.1" xref="alg1.l9.m1.1.1.1.1.cmml">i</mi><mo mathsize="80%" id="alg1.l9.m1.2.2.2.4.1" xref="alg1.l9.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg1.l9.m1.2.2.2.2" xref="alg1.l9.m1.2.2.2.2.cmml">j</mi></mrow></msub><mo mathsize="80%" id="alg1.l9.m1.7.7.2" xref="alg1.l9.m1.7.7.2.cmml">=</mo><mrow id="alg1.l9.m1.7.7.1" xref="alg1.l9.m1.7.7.1.cmml"><mtext mathsize="80%" id="alg1.l9.m1.7.7.1.3" xref="alg1.l9.m1.7.7.1.3a.cmml">exp</mtext><mo lspace="0em" rspace="0em" id="alg1.l9.m1.7.7.1.2" xref="alg1.l9.m1.7.7.1.2.cmml">​</mo><mrow id="alg1.l9.m1.7.7.1.1.1" xref="alg1.l9.m1.7.7.1.1.1.1.cmml"><mo maxsize="80%" minsize="80%" id="alg1.l9.m1.7.7.1.1.1.2" xref="alg1.l9.m1.7.7.1.1.1.1.cmml">(</mo><mrow id="alg1.l9.m1.7.7.1.1.1.1" xref="alg1.l9.m1.7.7.1.1.1.1.cmml"><msub id="alg1.l9.m1.7.7.1.1.1.1.2" xref="alg1.l9.m1.7.7.1.1.1.1.2.cmml"><mi mathsize="80%" id="alg1.l9.m1.7.7.1.1.1.1.2.2" xref="alg1.l9.m1.7.7.1.1.1.1.2.2.cmml">𝐒</mi><mrow id="alg1.l9.m1.4.4.2.4" xref="alg1.l9.m1.4.4.2.3.cmml"><mi mathsize="80%" id="alg1.l9.m1.3.3.1.1" xref="alg1.l9.m1.3.3.1.1.cmml">i</mi><mo mathsize="80%" id="alg1.l9.m1.4.4.2.4.1" xref="alg1.l9.m1.4.4.2.3.cmml">,</mo><mi mathsize="80%" id="alg1.l9.m1.4.4.2.2" xref="alg1.l9.m1.4.4.2.2.cmml">j</mi></mrow></msub><mo mathsize="80%" id="alg1.l9.m1.7.7.1.1.1.1.1" xref="alg1.l9.m1.7.7.1.1.1.1.1.cmml">−</mo><msub id="alg1.l9.m1.7.7.1.1.1.1.3" xref="alg1.l9.m1.7.7.1.1.1.1.3.cmml"><mi mathsize="80%" id="alg1.l9.m1.7.7.1.1.1.1.3.2" xref="alg1.l9.m1.7.7.1.1.1.1.3.2.cmml">m</mi><mrow id="alg1.l9.m1.6.6.2.4" xref="alg1.l9.m1.6.6.2.3.cmml"><mi mathsize="80%" id="alg1.l9.m1.5.5.1.1" xref="alg1.l9.m1.5.5.1.1.cmml">i</mi><mo mathsize="80%" id="alg1.l9.m1.6.6.2.4.1" xref="alg1.l9.m1.6.6.2.3.cmml">,</mo><mi mathsize="80%" id="alg1.l9.m1.6.6.2.2" xref="alg1.l9.m1.6.6.2.2.cmml">j</mi></mrow></msub></mrow><mo maxsize="80%" minsize="80%" id="alg1.l9.m1.7.7.1.1.1.3" xref="alg1.l9.m1.7.7.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l9.m1.7b"><apply id="alg1.l9.m1.7.7.cmml" xref="alg1.l9.m1.7.7"><eq id="alg1.l9.m1.7.7.2.cmml" xref="alg1.l9.m1.7.7.2"></eq><apply id="alg1.l9.m1.7.7.3.cmml" xref="alg1.l9.m1.7.7.3"><csymbol cd="ambiguous" id="alg1.l9.m1.7.7.3.1.cmml" xref="alg1.l9.m1.7.7.3">subscript</csymbol><ci id="alg1.l9.m1.7.7.3.2.cmml" xref="alg1.l9.m1.7.7.3.2">𝐏</ci><list id="alg1.l9.m1.2.2.2.3.cmml" xref="alg1.l9.m1.2.2.2.4"><ci id="alg1.l9.m1.1.1.1.1.cmml" xref="alg1.l9.m1.1.1.1.1">𝑖</ci><ci id="alg1.l9.m1.2.2.2.2.cmml" xref="alg1.l9.m1.2.2.2.2">𝑗</ci></list></apply><apply id="alg1.l9.m1.7.7.1.cmml" xref="alg1.l9.m1.7.7.1"><times id="alg1.l9.m1.7.7.1.2.cmml" xref="alg1.l9.m1.7.7.1.2"></times><ci id="alg1.l9.m1.7.7.1.3a.cmml" xref="alg1.l9.m1.7.7.1.3"><mtext mathsize="80%" id="alg1.l9.m1.7.7.1.3.cmml" xref="alg1.l9.m1.7.7.1.3">exp</mtext></ci><apply id="alg1.l9.m1.7.7.1.1.1.1.cmml" xref="alg1.l9.m1.7.7.1.1.1"><minus id="alg1.l9.m1.7.7.1.1.1.1.1.cmml" xref="alg1.l9.m1.7.7.1.1.1.1.1"></minus><apply id="alg1.l9.m1.7.7.1.1.1.1.2.cmml" xref="alg1.l9.m1.7.7.1.1.1.1.2"><csymbol cd="ambiguous" id="alg1.l9.m1.7.7.1.1.1.1.2.1.cmml" xref="alg1.l9.m1.7.7.1.1.1.1.2">subscript</csymbol><ci id="alg1.l9.m1.7.7.1.1.1.1.2.2.cmml" xref="alg1.l9.m1.7.7.1.1.1.1.2.2">𝐒</ci><list id="alg1.l9.m1.4.4.2.3.cmml" xref="alg1.l9.m1.4.4.2.4"><ci id="alg1.l9.m1.3.3.1.1.cmml" xref="alg1.l9.m1.3.3.1.1">𝑖</ci><ci id="alg1.l9.m1.4.4.2.2.cmml" xref="alg1.l9.m1.4.4.2.2">𝑗</ci></list></apply><apply id="alg1.l9.m1.7.7.1.1.1.1.3.cmml" xref="alg1.l9.m1.7.7.1.1.1.1.3"><csymbol cd="ambiguous" id="alg1.l9.m1.7.7.1.1.1.1.3.1.cmml" xref="alg1.l9.m1.7.7.1.1.1.1.3">subscript</csymbol><ci id="alg1.l9.m1.7.7.1.1.1.1.3.2.cmml" xref="alg1.l9.m1.7.7.1.1.1.1.3.2">𝑚</ci><list id="alg1.l9.m1.6.6.2.3.cmml" xref="alg1.l9.m1.6.6.2.4"><ci id="alg1.l9.m1.5.5.1.1.cmml" xref="alg1.l9.m1.5.5.1.1">𝑖</ci><ci id="alg1.l9.m1.6.6.2.2.cmml" xref="alg1.l9.m1.6.6.2.2">𝑗</ci></list></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l9.m1.7c">\mathbf{P}_{i,j}=\text{exp}(\mathbf{S}_{i,j}-m_{i,j})</annotation></semantics></math><span id="alg1.l9.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg1.l10" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l10.1.1.1" class="ltx_text" style="font-size:80%;">10:</span></span><span id="alg1.l10.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg1.l10.m1.5" class="ltx_Math" alttext="l_{i,j}=\text{rowsum}(\mathbf{P}_{i,j})" display="inline"><semantics id="alg1.l10.m1.5a"><mrow id="alg1.l10.m1.5.5" xref="alg1.l10.m1.5.5.cmml"><msub id="alg1.l10.m1.5.5.3" xref="alg1.l10.m1.5.5.3.cmml"><mi mathsize="80%" id="alg1.l10.m1.5.5.3.2" xref="alg1.l10.m1.5.5.3.2.cmml">l</mi><mrow id="alg1.l10.m1.2.2.2.4" xref="alg1.l10.m1.2.2.2.3.cmml"><mi mathsize="80%" id="alg1.l10.m1.1.1.1.1" xref="alg1.l10.m1.1.1.1.1.cmml">i</mi><mo mathsize="80%" id="alg1.l10.m1.2.2.2.4.1" xref="alg1.l10.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg1.l10.m1.2.2.2.2" xref="alg1.l10.m1.2.2.2.2.cmml">j</mi></mrow></msub><mo mathsize="80%" id="alg1.l10.m1.5.5.2" xref="alg1.l10.m1.5.5.2.cmml">=</mo><mrow id="alg1.l10.m1.5.5.1" xref="alg1.l10.m1.5.5.1.cmml"><mtext mathsize="80%" id="alg1.l10.m1.5.5.1.3" xref="alg1.l10.m1.5.5.1.3a.cmml">rowsum</mtext><mo lspace="0em" rspace="0em" id="alg1.l10.m1.5.5.1.2" xref="alg1.l10.m1.5.5.1.2.cmml">​</mo><mrow id="alg1.l10.m1.5.5.1.1.1" xref="alg1.l10.m1.5.5.1.1.1.1.cmml"><mo maxsize="80%" minsize="80%" id="alg1.l10.m1.5.5.1.1.1.2" xref="alg1.l10.m1.5.5.1.1.1.1.cmml">(</mo><msub id="alg1.l10.m1.5.5.1.1.1.1" xref="alg1.l10.m1.5.5.1.1.1.1.cmml"><mi mathsize="80%" id="alg1.l10.m1.5.5.1.1.1.1.2" xref="alg1.l10.m1.5.5.1.1.1.1.2.cmml">𝐏</mi><mrow id="alg1.l10.m1.4.4.2.4" xref="alg1.l10.m1.4.4.2.3.cmml"><mi mathsize="80%" id="alg1.l10.m1.3.3.1.1" xref="alg1.l10.m1.3.3.1.1.cmml">i</mi><mo mathsize="80%" id="alg1.l10.m1.4.4.2.4.1" xref="alg1.l10.m1.4.4.2.3.cmml">,</mo><mi mathsize="80%" id="alg1.l10.m1.4.4.2.2" xref="alg1.l10.m1.4.4.2.2.cmml">j</mi></mrow></msub><mo maxsize="80%" minsize="80%" id="alg1.l10.m1.5.5.1.1.1.3" xref="alg1.l10.m1.5.5.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l10.m1.5b"><apply id="alg1.l10.m1.5.5.cmml" xref="alg1.l10.m1.5.5"><eq id="alg1.l10.m1.5.5.2.cmml" xref="alg1.l10.m1.5.5.2"></eq><apply id="alg1.l10.m1.5.5.3.cmml" xref="alg1.l10.m1.5.5.3"><csymbol cd="ambiguous" id="alg1.l10.m1.5.5.3.1.cmml" xref="alg1.l10.m1.5.5.3">subscript</csymbol><ci id="alg1.l10.m1.5.5.3.2.cmml" xref="alg1.l10.m1.5.5.3.2">𝑙</ci><list id="alg1.l10.m1.2.2.2.3.cmml" xref="alg1.l10.m1.2.2.2.4"><ci id="alg1.l10.m1.1.1.1.1.cmml" xref="alg1.l10.m1.1.1.1.1">𝑖</ci><ci id="alg1.l10.m1.2.2.2.2.cmml" xref="alg1.l10.m1.2.2.2.2">𝑗</ci></list></apply><apply id="alg1.l10.m1.5.5.1.cmml" xref="alg1.l10.m1.5.5.1"><times id="alg1.l10.m1.5.5.1.2.cmml" xref="alg1.l10.m1.5.5.1.2"></times><ci id="alg1.l10.m1.5.5.1.3a.cmml" xref="alg1.l10.m1.5.5.1.3"><mtext mathsize="80%" id="alg1.l10.m1.5.5.1.3.cmml" xref="alg1.l10.m1.5.5.1.3">rowsum</mtext></ci><apply id="alg1.l10.m1.5.5.1.1.1.1.cmml" xref="alg1.l10.m1.5.5.1.1.1"><csymbol cd="ambiguous" id="alg1.l10.m1.5.5.1.1.1.1.1.cmml" xref="alg1.l10.m1.5.5.1.1.1">subscript</csymbol><ci id="alg1.l10.m1.5.5.1.1.1.1.2.cmml" xref="alg1.l10.m1.5.5.1.1.1.1.2">𝐏</ci><list id="alg1.l10.m1.4.4.2.3.cmml" xref="alg1.l10.m1.4.4.2.4"><ci id="alg1.l10.m1.3.3.1.1.cmml" xref="alg1.l10.m1.3.3.1.1">𝑖</ci><ci id="alg1.l10.m1.4.4.2.2.cmml" xref="alg1.l10.m1.4.4.2.2">𝑗</ci></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l10.m1.5c">l_{i,j}=\text{rowsum}(\mathbf{P}_{i,j})</annotation></semantics></math><span id="alg1.l10.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg1.l11" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l11.1.1.1" class="ltx_text" style="font-size:80%;">11:</span></span><span id="alg1.l11.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg1.l11.m1.4" class="ltx_Math" alttext="\mathbf{O}_{i,j}=\mathbf{P}_{i,j}\mathbf{V}_{j}" display="inline"><semantics id="alg1.l11.m1.4a"><mrow id="alg1.l11.m1.4.5" xref="alg1.l11.m1.4.5.cmml"><msub id="alg1.l11.m1.4.5.2" xref="alg1.l11.m1.4.5.2.cmml"><mi mathsize="80%" id="alg1.l11.m1.4.5.2.2" xref="alg1.l11.m1.4.5.2.2.cmml">𝐎</mi><mrow id="alg1.l11.m1.2.2.2.4" xref="alg1.l11.m1.2.2.2.3.cmml"><mi mathsize="80%" id="alg1.l11.m1.1.1.1.1" xref="alg1.l11.m1.1.1.1.1.cmml">i</mi><mo mathsize="80%" id="alg1.l11.m1.2.2.2.4.1" xref="alg1.l11.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg1.l11.m1.2.2.2.2" xref="alg1.l11.m1.2.2.2.2.cmml">j</mi></mrow></msub><mo mathsize="80%" id="alg1.l11.m1.4.5.1" xref="alg1.l11.m1.4.5.1.cmml">=</mo><mrow id="alg1.l11.m1.4.5.3" xref="alg1.l11.m1.4.5.3.cmml"><msub id="alg1.l11.m1.4.5.3.2" xref="alg1.l11.m1.4.5.3.2.cmml"><mi mathsize="80%" id="alg1.l11.m1.4.5.3.2.2" xref="alg1.l11.m1.4.5.3.2.2.cmml">𝐏</mi><mrow id="alg1.l11.m1.4.4.2.4" xref="alg1.l11.m1.4.4.2.3.cmml"><mi mathsize="80%" id="alg1.l11.m1.3.3.1.1" xref="alg1.l11.m1.3.3.1.1.cmml">i</mi><mo mathsize="80%" id="alg1.l11.m1.4.4.2.4.1" xref="alg1.l11.m1.4.4.2.3.cmml">,</mo><mi mathsize="80%" id="alg1.l11.m1.4.4.2.2" xref="alg1.l11.m1.4.4.2.2.cmml">j</mi></mrow></msub><mo lspace="0em" rspace="0em" id="alg1.l11.m1.4.5.3.1" xref="alg1.l11.m1.4.5.3.1.cmml">​</mo><msub id="alg1.l11.m1.4.5.3.3" xref="alg1.l11.m1.4.5.3.3.cmml"><mi mathsize="80%" id="alg1.l11.m1.4.5.3.3.2" xref="alg1.l11.m1.4.5.3.3.2.cmml">𝐕</mi><mi mathsize="80%" id="alg1.l11.m1.4.5.3.3.3" xref="alg1.l11.m1.4.5.3.3.3.cmml">j</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l11.m1.4b"><apply id="alg1.l11.m1.4.5.cmml" xref="alg1.l11.m1.4.5"><eq id="alg1.l11.m1.4.5.1.cmml" xref="alg1.l11.m1.4.5.1"></eq><apply id="alg1.l11.m1.4.5.2.cmml" xref="alg1.l11.m1.4.5.2"><csymbol cd="ambiguous" id="alg1.l11.m1.4.5.2.1.cmml" xref="alg1.l11.m1.4.5.2">subscript</csymbol><ci id="alg1.l11.m1.4.5.2.2.cmml" xref="alg1.l11.m1.4.5.2.2">𝐎</ci><list id="alg1.l11.m1.2.2.2.3.cmml" xref="alg1.l11.m1.2.2.2.4"><ci id="alg1.l11.m1.1.1.1.1.cmml" xref="alg1.l11.m1.1.1.1.1">𝑖</ci><ci id="alg1.l11.m1.2.2.2.2.cmml" xref="alg1.l11.m1.2.2.2.2">𝑗</ci></list></apply><apply id="alg1.l11.m1.4.5.3.cmml" xref="alg1.l11.m1.4.5.3"><times id="alg1.l11.m1.4.5.3.1.cmml" xref="alg1.l11.m1.4.5.3.1"></times><apply id="alg1.l11.m1.4.5.3.2.cmml" xref="alg1.l11.m1.4.5.3.2"><csymbol cd="ambiguous" id="alg1.l11.m1.4.5.3.2.1.cmml" xref="alg1.l11.m1.4.5.3.2">subscript</csymbol><ci id="alg1.l11.m1.4.5.3.2.2.cmml" xref="alg1.l11.m1.4.5.3.2.2">𝐏</ci><list id="alg1.l11.m1.4.4.2.3.cmml" xref="alg1.l11.m1.4.4.2.4"><ci id="alg1.l11.m1.3.3.1.1.cmml" xref="alg1.l11.m1.3.3.1.1">𝑖</ci><ci id="alg1.l11.m1.4.4.2.2.cmml" xref="alg1.l11.m1.4.4.2.2">𝑗</ci></list></apply><apply id="alg1.l11.m1.4.5.3.3.cmml" xref="alg1.l11.m1.4.5.3.3"><csymbol cd="ambiguous" id="alg1.l11.m1.4.5.3.3.1.cmml" xref="alg1.l11.m1.4.5.3.3">subscript</csymbol><ci id="alg1.l11.m1.4.5.3.3.2.cmml" xref="alg1.l11.m1.4.5.3.3.2">𝐕</ci><ci id="alg1.l11.m1.4.5.3.3.3.cmml" xref="alg1.l11.m1.4.5.3.3.3">𝑗</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l11.m1.4c">\mathbf{O}_{i,j}=\mathbf{P}_{i,j}\mathbf{V}_{j}</annotation></semantics></math><span id="alg1.l11.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg1.l12" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l12.1.1.1" class="ltx_text" style="font-size:80%;">12:</span></span><span id="alg1.l12.2" class="ltx_text" style="font-size:80%;">    {The end of the forward pass of local attention.}
</span>
</div>
<div id="alg1.l13" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l13.1.1.1" class="ltx_text" style="font-size:80%;">13:</span></span><span id="alg1.l13.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg1.l13.m1.5" class="ltx_Math" alttext="m_{\text{new}}\leftarrow\max{\{m_{i},m_{i,j}\}}" display="inline"><semantics id="alg1.l13.m1.5a"><mrow id="alg1.l13.m1.5.5" xref="alg1.l13.m1.5.5.cmml"><msub id="alg1.l13.m1.5.5.4" xref="alg1.l13.m1.5.5.4.cmml"><mi mathsize="80%" id="alg1.l13.m1.5.5.4.2" xref="alg1.l13.m1.5.5.4.2.cmml">m</mi><mtext mathsize="80%" id="alg1.l13.m1.5.5.4.3" xref="alg1.l13.m1.5.5.4.3a.cmml">new</mtext></msub><mo mathsize="80%" stretchy="false" id="alg1.l13.m1.5.5.3" xref="alg1.l13.m1.5.5.3.cmml">←</mo><mrow id="alg1.l13.m1.5.5.2.2" xref="alg1.l13.m1.5.5.2.3.cmml"><mi mathsize="80%" id="alg1.l13.m1.3.3" xref="alg1.l13.m1.3.3.cmml">max</mi><mo id="alg1.l13.m1.5.5.2.2a" xref="alg1.l13.m1.5.5.2.3.cmml">⁡</mo><mrow id="alg1.l13.m1.5.5.2.2.2" xref="alg1.l13.m1.5.5.2.3.cmml"><mo maxsize="80%" minsize="80%" id="alg1.l13.m1.5.5.2.2.2.3" xref="alg1.l13.m1.5.5.2.3.cmml">{</mo><msub id="alg1.l13.m1.4.4.1.1.1.1" xref="alg1.l13.m1.4.4.1.1.1.1.cmml"><mi mathsize="80%" id="alg1.l13.m1.4.4.1.1.1.1.2" xref="alg1.l13.m1.4.4.1.1.1.1.2.cmml">m</mi><mi mathsize="80%" id="alg1.l13.m1.4.4.1.1.1.1.3" xref="alg1.l13.m1.4.4.1.1.1.1.3.cmml">i</mi></msub><mo mathsize="80%" id="alg1.l13.m1.5.5.2.2.2.4" xref="alg1.l13.m1.5.5.2.3.cmml">,</mo><msub id="alg1.l13.m1.5.5.2.2.2.2" xref="alg1.l13.m1.5.5.2.2.2.2.cmml"><mi mathsize="80%" id="alg1.l13.m1.5.5.2.2.2.2.2" xref="alg1.l13.m1.5.5.2.2.2.2.2.cmml">m</mi><mrow id="alg1.l13.m1.2.2.2.4" xref="alg1.l13.m1.2.2.2.3.cmml"><mi mathsize="80%" id="alg1.l13.m1.1.1.1.1" xref="alg1.l13.m1.1.1.1.1.cmml">i</mi><mo mathsize="80%" id="alg1.l13.m1.2.2.2.4.1" xref="alg1.l13.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg1.l13.m1.2.2.2.2" xref="alg1.l13.m1.2.2.2.2.cmml">j</mi></mrow></msub><mo maxsize="80%" minsize="80%" id="alg1.l13.m1.5.5.2.2.2.5" xref="alg1.l13.m1.5.5.2.3.cmml">}</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l13.m1.5b"><apply id="alg1.l13.m1.5.5.cmml" xref="alg1.l13.m1.5.5"><ci id="alg1.l13.m1.5.5.3.cmml" xref="alg1.l13.m1.5.5.3">←</ci><apply id="alg1.l13.m1.5.5.4.cmml" xref="alg1.l13.m1.5.5.4"><csymbol cd="ambiguous" id="alg1.l13.m1.5.5.4.1.cmml" xref="alg1.l13.m1.5.5.4">subscript</csymbol><ci id="alg1.l13.m1.5.5.4.2.cmml" xref="alg1.l13.m1.5.5.4.2">𝑚</ci><ci id="alg1.l13.m1.5.5.4.3a.cmml" xref="alg1.l13.m1.5.5.4.3"><mtext mathsize="56%" id="alg1.l13.m1.5.5.4.3.cmml" xref="alg1.l13.m1.5.5.4.3">new</mtext></ci></apply><apply id="alg1.l13.m1.5.5.2.3.cmml" xref="alg1.l13.m1.5.5.2.2"><max id="alg1.l13.m1.3.3.cmml" xref="alg1.l13.m1.3.3"></max><apply id="alg1.l13.m1.4.4.1.1.1.1.cmml" xref="alg1.l13.m1.4.4.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l13.m1.4.4.1.1.1.1.1.cmml" xref="alg1.l13.m1.4.4.1.1.1.1">subscript</csymbol><ci id="alg1.l13.m1.4.4.1.1.1.1.2.cmml" xref="alg1.l13.m1.4.4.1.1.1.1.2">𝑚</ci><ci id="alg1.l13.m1.4.4.1.1.1.1.3.cmml" xref="alg1.l13.m1.4.4.1.1.1.1.3">𝑖</ci></apply><apply id="alg1.l13.m1.5.5.2.2.2.2.cmml" xref="alg1.l13.m1.5.5.2.2.2.2"><csymbol cd="ambiguous" id="alg1.l13.m1.5.5.2.2.2.2.1.cmml" xref="alg1.l13.m1.5.5.2.2.2.2">subscript</csymbol><ci id="alg1.l13.m1.5.5.2.2.2.2.2.cmml" xref="alg1.l13.m1.5.5.2.2.2.2.2">𝑚</ci><list id="alg1.l13.m1.2.2.2.3.cmml" xref="alg1.l13.m1.2.2.2.4"><ci id="alg1.l13.m1.1.1.1.1.cmml" xref="alg1.l13.m1.1.1.1.1">𝑖</ci><ci id="alg1.l13.m1.2.2.2.2.cmml" xref="alg1.l13.m1.2.2.2.2">𝑗</ci></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l13.m1.5c">m_{\text{new}}\leftarrow\max{\{m_{i},m_{i,j}\}}</annotation></semantics></math><span id="alg1.l13.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg1.l14" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l14.1.1.1" class="ltx_text" style="font-size:80%;">14:</span></span><span id="alg1.l14.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg1.l14.m1.4" class="ltx_Math" alttext="\mathbf{l}_{i}=e^{m_{i}-m_{\text{new}}}l_{i}+e^{m_{i,j}-m_{\text{new}}}l_{i,j}" display="inline"><semantics id="alg1.l14.m1.4a"><mrow id="alg1.l14.m1.4.5" xref="alg1.l14.m1.4.5.cmml"><msub id="alg1.l14.m1.4.5.2" xref="alg1.l14.m1.4.5.2.cmml"><mi mathsize="80%" id="alg1.l14.m1.4.5.2.2" xref="alg1.l14.m1.4.5.2.2.cmml">𝐥</mi><mi mathsize="80%" id="alg1.l14.m1.4.5.2.3" xref="alg1.l14.m1.4.5.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg1.l14.m1.4.5.1" xref="alg1.l14.m1.4.5.1.cmml">=</mo><mrow id="alg1.l14.m1.4.5.3" xref="alg1.l14.m1.4.5.3.cmml"><mrow id="alg1.l14.m1.4.5.3.2" xref="alg1.l14.m1.4.5.3.2.cmml"><msup id="alg1.l14.m1.4.5.3.2.2" xref="alg1.l14.m1.4.5.3.2.2.cmml"><mi mathsize="80%" id="alg1.l14.m1.4.5.3.2.2.2" xref="alg1.l14.m1.4.5.3.2.2.2.cmml">e</mi><mrow id="alg1.l14.m1.4.5.3.2.2.3" xref="alg1.l14.m1.4.5.3.2.2.3.cmml"><msub id="alg1.l14.m1.4.5.3.2.2.3.2" xref="alg1.l14.m1.4.5.3.2.2.3.2.cmml"><mi mathsize="80%" id="alg1.l14.m1.4.5.3.2.2.3.2.2" xref="alg1.l14.m1.4.5.3.2.2.3.2.2.cmml">m</mi><mi mathsize="80%" id="alg1.l14.m1.4.5.3.2.2.3.2.3" xref="alg1.l14.m1.4.5.3.2.2.3.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg1.l14.m1.4.5.3.2.2.3.1" xref="alg1.l14.m1.4.5.3.2.2.3.1.cmml">−</mo><msub id="alg1.l14.m1.4.5.3.2.2.3.3" xref="alg1.l14.m1.4.5.3.2.2.3.3.cmml"><mi mathsize="80%" id="alg1.l14.m1.4.5.3.2.2.3.3.2" xref="alg1.l14.m1.4.5.3.2.2.3.3.2.cmml">m</mi><mtext mathsize="80%" id="alg1.l14.m1.4.5.3.2.2.3.3.3" xref="alg1.l14.m1.4.5.3.2.2.3.3.3a.cmml">new</mtext></msub></mrow></msup><mo lspace="0em" rspace="0em" id="alg1.l14.m1.4.5.3.2.1" xref="alg1.l14.m1.4.5.3.2.1.cmml">​</mo><msub id="alg1.l14.m1.4.5.3.2.3" xref="alg1.l14.m1.4.5.3.2.3.cmml"><mi mathsize="80%" id="alg1.l14.m1.4.5.3.2.3.2" xref="alg1.l14.m1.4.5.3.2.3.2.cmml">l</mi><mi mathsize="80%" id="alg1.l14.m1.4.5.3.2.3.3" xref="alg1.l14.m1.4.5.3.2.3.3.cmml">i</mi></msub></mrow><mo mathsize="80%" id="alg1.l14.m1.4.5.3.1" xref="alg1.l14.m1.4.5.3.1.cmml">+</mo><mrow id="alg1.l14.m1.4.5.3.3" xref="alg1.l14.m1.4.5.3.3.cmml"><msup id="alg1.l14.m1.4.5.3.3.2" xref="alg1.l14.m1.4.5.3.3.2.cmml"><mi mathsize="80%" id="alg1.l14.m1.4.5.3.3.2.2" xref="alg1.l14.m1.4.5.3.3.2.2.cmml">e</mi><mrow id="alg1.l14.m1.2.2.2" xref="alg1.l14.m1.2.2.2.cmml"><msub id="alg1.l14.m1.2.2.2.4" xref="alg1.l14.m1.2.2.2.4.cmml"><mi mathsize="80%" id="alg1.l14.m1.2.2.2.4.2" xref="alg1.l14.m1.2.2.2.4.2.cmml">m</mi><mrow id="alg1.l14.m1.2.2.2.2.2.4" xref="alg1.l14.m1.2.2.2.2.2.3.cmml"><mi mathsize="80%" id="alg1.l14.m1.1.1.1.1.1.1" xref="alg1.l14.m1.1.1.1.1.1.1.cmml">i</mi><mo mathsize="80%" id="alg1.l14.m1.2.2.2.2.2.4.1" xref="alg1.l14.m1.2.2.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg1.l14.m1.2.2.2.2.2.2" xref="alg1.l14.m1.2.2.2.2.2.2.cmml">j</mi></mrow></msub><mo mathsize="80%" id="alg1.l14.m1.2.2.2.3" xref="alg1.l14.m1.2.2.2.3.cmml">−</mo><msub id="alg1.l14.m1.2.2.2.5" xref="alg1.l14.m1.2.2.2.5.cmml"><mi mathsize="80%" id="alg1.l14.m1.2.2.2.5.2" xref="alg1.l14.m1.2.2.2.5.2.cmml">m</mi><mtext mathsize="80%" id="alg1.l14.m1.2.2.2.5.3" xref="alg1.l14.m1.2.2.2.5.3a.cmml">new</mtext></msub></mrow></msup><mo lspace="0em" rspace="0em" id="alg1.l14.m1.4.5.3.3.1" xref="alg1.l14.m1.4.5.3.3.1.cmml">​</mo><msub id="alg1.l14.m1.4.5.3.3.3" xref="alg1.l14.m1.4.5.3.3.3.cmml"><mi mathsize="80%" id="alg1.l14.m1.4.5.3.3.3.2" xref="alg1.l14.m1.4.5.3.3.3.2.cmml">l</mi><mrow id="alg1.l14.m1.4.4.2.4" xref="alg1.l14.m1.4.4.2.3.cmml"><mi mathsize="80%" id="alg1.l14.m1.3.3.1.1" xref="alg1.l14.m1.3.3.1.1.cmml">i</mi><mo mathsize="80%" id="alg1.l14.m1.4.4.2.4.1" xref="alg1.l14.m1.4.4.2.3.cmml">,</mo><mi mathsize="80%" id="alg1.l14.m1.4.4.2.2" xref="alg1.l14.m1.4.4.2.2.cmml">j</mi></mrow></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l14.m1.4b"><apply id="alg1.l14.m1.4.5.cmml" xref="alg1.l14.m1.4.5"><eq id="alg1.l14.m1.4.5.1.cmml" xref="alg1.l14.m1.4.5.1"></eq><apply id="alg1.l14.m1.4.5.2.cmml" xref="alg1.l14.m1.4.5.2"><csymbol cd="ambiguous" id="alg1.l14.m1.4.5.2.1.cmml" xref="alg1.l14.m1.4.5.2">subscript</csymbol><ci id="alg1.l14.m1.4.5.2.2.cmml" xref="alg1.l14.m1.4.5.2.2">𝐥</ci><ci id="alg1.l14.m1.4.5.2.3.cmml" xref="alg1.l14.m1.4.5.2.3">𝑖</ci></apply><apply id="alg1.l14.m1.4.5.3.cmml" xref="alg1.l14.m1.4.5.3"><plus id="alg1.l14.m1.4.5.3.1.cmml" xref="alg1.l14.m1.4.5.3.1"></plus><apply id="alg1.l14.m1.4.5.3.2.cmml" xref="alg1.l14.m1.4.5.3.2"><times id="alg1.l14.m1.4.5.3.2.1.cmml" xref="alg1.l14.m1.4.5.3.2.1"></times><apply id="alg1.l14.m1.4.5.3.2.2.cmml" xref="alg1.l14.m1.4.5.3.2.2"><csymbol cd="ambiguous" id="alg1.l14.m1.4.5.3.2.2.1.cmml" xref="alg1.l14.m1.4.5.3.2.2">superscript</csymbol><ci id="alg1.l14.m1.4.5.3.2.2.2.cmml" xref="alg1.l14.m1.4.5.3.2.2.2">𝑒</ci><apply id="alg1.l14.m1.4.5.3.2.2.3.cmml" xref="alg1.l14.m1.4.5.3.2.2.3"><minus id="alg1.l14.m1.4.5.3.2.2.3.1.cmml" xref="alg1.l14.m1.4.5.3.2.2.3.1"></minus><apply id="alg1.l14.m1.4.5.3.2.2.3.2.cmml" xref="alg1.l14.m1.4.5.3.2.2.3.2"><csymbol cd="ambiguous" id="alg1.l14.m1.4.5.3.2.2.3.2.1.cmml" xref="alg1.l14.m1.4.5.3.2.2.3.2">subscript</csymbol><ci id="alg1.l14.m1.4.5.3.2.2.3.2.2.cmml" xref="alg1.l14.m1.4.5.3.2.2.3.2.2">𝑚</ci><ci id="alg1.l14.m1.4.5.3.2.2.3.2.3.cmml" xref="alg1.l14.m1.4.5.3.2.2.3.2.3">𝑖</ci></apply><apply id="alg1.l14.m1.4.5.3.2.2.3.3.cmml" xref="alg1.l14.m1.4.5.3.2.2.3.3"><csymbol cd="ambiguous" id="alg1.l14.m1.4.5.3.2.2.3.3.1.cmml" xref="alg1.l14.m1.4.5.3.2.2.3.3">subscript</csymbol><ci id="alg1.l14.m1.4.5.3.2.2.3.3.2.cmml" xref="alg1.l14.m1.4.5.3.2.2.3.3.2">𝑚</ci><ci id="alg1.l14.m1.4.5.3.2.2.3.3.3a.cmml" xref="alg1.l14.m1.4.5.3.2.2.3.3.3"><mtext mathsize="40%" id="alg1.l14.m1.4.5.3.2.2.3.3.3.cmml" xref="alg1.l14.m1.4.5.3.2.2.3.3.3">new</mtext></ci></apply></apply></apply><apply id="alg1.l14.m1.4.5.3.2.3.cmml" xref="alg1.l14.m1.4.5.3.2.3"><csymbol cd="ambiguous" id="alg1.l14.m1.4.5.3.2.3.1.cmml" xref="alg1.l14.m1.4.5.3.2.3">subscript</csymbol><ci id="alg1.l14.m1.4.5.3.2.3.2.cmml" xref="alg1.l14.m1.4.5.3.2.3.2">𝑙</ci><ci id="alg1.l14.m1.4.5.3.2.3.3.cmml" xref="alg1.l14.m1.4.5.3.2.3.3">𝑖</ci></apply></apply><apply id="alg1.l14.m1.4.5.3.3.cmml" xref="alg1.l14.m1.4.5.3.3"><times id="alg1.l14.m1.4.5.3.3.1.cmml" xref="alg1.l14.m1.4.5.3.3.1"></times><apply id="alg1.l14.m1.4.5.3.3.2.cmml" xref="alg1.l14.m1.4.5.3.3.2"><csymbol cd="ambiguous" id="alg1.l14.m1.4.5.3.3.2.1.cmml" xref="alg1.l14.m1.4.5.3.3.2">superscript</csymbol><ci id="alg1.l14.m1.4.5.3.3.2.2.cmml" xref="alg1.l14.m1.4.5.3.3.2.2">𝑒</ci><apply id="alg1.l14.m1.2.2.2.cmml" xref="alg1.l14.m1.2.2.2"><minus id="alg1.l14.m1.2.2.2.3.cmml" xref="alg1.l14.m1.2.2.2.3"></minus><apply id="alg1.l14.m1.2.2.2.4.cmml" xref="alg1.l14.m1.2.2.2.4"><csymbol cd="ambiguous" id="alg1.l14.m1.2.2.2.4.1.cmml" xref="alg1.l14.m1.2.2.2.4">subscript</csymbol><ci id="alg1.l14.m1.2.2.2.4.2.cmml" xref="alg1.l14.m1.2.2.2.4.2">𝑚</ci><list id="alg1.l14.m1.2.2.2.2.2.3.cmml" xref="alg1.l14.m1.2.2.2.2.2.4"><ci id="alg1.l14.m1.1.1.1.1.1.1.cmml" xref="alg1.l14.m1.1.1.1.1.1.1">𝑖</ci><ci id="alg1.l14.m1.2.2.2.2.2.2.cmml" xref="alg1.l14.m1.2.2.2.2.2.2">𝑗</ci></list></apply><apply id="alg1.l14.m1.2.2.2.5.cmml" xref="alg1.l14.m1.2.2.2.5"><csymbol cd="ambiguous" id="alg1.l14.m1.2.2.2.5.1.cmml" xref="alg1.l14.m1.2.2.2.5">subscript</csymbol><ci id="alg1.l14.m1.2.2.2.5.2.cmml" xref="alg1.l14.m1.2.2.2.5.2">𝑚</ci><ci id="alg1.l14.m1.2.2.2.5.3a.cmml" xref="alg1.l14.m1.2.2.2.5.3"><mtext mathsize="40%" id="alg1.l14.m1.2.2.2.5.3.cmml" xref="alg1.l14.m1.2.2.2.5.3">new</mtext></ci></apply></apply></apply><apply id="alg1.l14.m1.4.5.3.3.3.cmml" xref="alg1.l14.m1.4.5.3.3.3"><csymbol cd="ambiguous" id="alg1.l14.m1.4.5.3.3.3.1.cmml" xref="alg1.l14.m1.4.5.3.3.3">subscript</csymbol><ci id="alg1.l14.m1.4.5.3.3.3.2.cmml" xref="alg1.l14.m1.4.5.3.3.3.2">𝑙</ci><list id="alg1.l14.m1.4.4.2.3.cmml" xref="alg1.l14.m1.4.4.2.4"><ci id="alg1.l14.m1.3.3.1.1.cmml" xref="alg1.l14.m1.3.3.1.1">𝑖</ci><ci id="alg1.l14.m1.4.4.2.2.cmml" xref="alg1.l14.m1.4.4.2.2">𝑗</ci></list></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l14.m1.4c">\mathbf{l}_{i}=e^{m_{i}-m_{\text{new}}}l_{i}+e^{m_{i,j}-m_{\text{new}}}l_{i,j}</annotation></semantics></math><span id="alg1.l14.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg1.l15" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l15.1.1.1" class="ltx_text" style="font-size:80%;">15:</span></span><span id="alg1.l15.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg1.l15.m1.4" class="ltx_Math" alttext="\mathbf{O}_{i}=e^{m_{i}-m_{\text{new}}}\mathbf{O}_{i}+e^{m_{i,j}-m_{\text{new}}}\mathbf{O}_{i,j}" display="inline"><semantics id="alg1.l15.m1.4a"><mrow id="alg1.l15.m1.4.5" xref="alg1.l15.m1.4.5.cmml"><msub id="alg1.l15.m1.4.5.2" xref="alg1.l15.m1.4.5.2.cmml"><mi mathsize="80%" id="alg1.l15.m1.4.5.2.2" xref="alg1.l15.m1.4.5.2.2.cmml">𝐎</mi><mi mathsize="80%" id="alg1.l15.m1.4.5.2.3" xref="alg1.l15.m1.4.5.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg1.l15.m1.4.5.1" xref="alg1.l15.m1.4.5.1.cmml">=</mo><mrow id="alg1.l15.m1.4.5.3" xref="alg1.l15.m1.4.5.3.cmml"><mrow id="alg1.l15.m1.4.5.3.2" xref="alg1.l15.m1.4.5.3.2.cmml"><msup id="alg1.l15.m1.4.5.3.2.2" xref="alg1.l15.m1.4.5.3.2.2.cmml"><mi mathsize="80%" id="alg1.l15.m1.4.5.3.2.2.2" xref="alg1.l15.m1.4.5.3.2.2.2.cmml">e</mi><mrow id="alg1.l15.m1.4.5.3.2.2.3" xref="alg1.l15.m1.4.5.3.2.2.3.cmml"><msub id="alg1.l15.m1.4.5.3.2.2.3.2" xref="alg1.l15.m1.4.5.3.2.2.3.2.cmml"><mi mathsize="80%" id="alg1.l15.m1.4.5.3.2.2.3.2.2" xref="alg1.l15.m1.4.5.3.2.2.3.2.2.cmml">m</mi><mi mathsize="80%" id="alg1.l15.m1.4.5.3.2.2.3.2.3" xref="alg1.l15.m1.4.5.3.2.2.3.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg1.l15.m1.4.5.3.2.2.3.1" xref="alg1.l15.m1.4.5.3.2.2.3.1.cmml">−</mo><msub id="alg1.l15.m1.4.5.3.2.2.3.3" xref="alg1.l15.m1.4.5.3.2.2.3.3.cmml"><mi mathsize="80%" id="alg1.l15.m1.4.5.3.2.2.3.3.2" xref="alg1.l15.m1.4.5.3.2.2.3.3.2.cmml">m</mi><mtext mathsize="80%" id="alg1.l15.m1.4.5.3.2.2.3.3.3" xref="alg1.l15.m1.4.5.3.2.2.3.3.3a.cmml">new</mtext></msub></mrow></msup><mo lspace="0em" rspace="0em" id="alg1.l15.m1.4.5.3.2.1" xref="alg1.l15.m1.4.5.3.2.1.cmml">​</mo><msub id="alg1.l15.m1.4.5.3.2.3" xref="alg1.l15.m1.4.5.3.2.3.cmml"><mi mathsize="80%" id="alg1.l15.m1.4.5.3.2.3.2" xref="alg1.l15.m1.4.5.3.2.3.2.cmml">𝐎</mi><mi mathsize="80%" id="alg1.l15.m1.4.5.3.2.3.3" xref="alg1.l15.m1.4.5.3.2.3.3.cmml">i</mi></msub></mrow><mo mathsize="80%" id="alg1.l15.m1.4.5.3.1" xref="alg1.l15.m1.4.5.3.1.cmml">+</mo><mrow id="alg1.l15.m1.4.5.3.3" xref="alg1.l15.m1.4.5.3.3.cmml"><msup id="alg1.l15.m1.4.5.3.3.2" xref="alg1.l15.m1.4.5.3.3.2.cmml"><mi mathsize="80%" id="alg1.l15.m1.4.5.3.3.2.2" xref="alg1.l15.m1.4.5.3.3.2.2.cmml">e</mi><mrow id="alg1.l15.m1.2.2.2" xref="alg1.l15.m1.2.2.2.cmml"><msub id="alg1.l15.m1.2.2.2.4" xref="alg1.l15.m1.2.2.2.4.cmml"><mi mathsize="80%" id="alg1.l15.m1.2.2.2.4.2" xref="alg1.l15.m1.2.2.2.4.2.cmml">m</mi><mrow id="alg1.l15.m1.2.2.2.2.2.4" xref="alg1.l15.m1.2.2.2.2.2.3.cmml"><mi mathsize="80%" id="alg1.l15.m1.1.1.1.1.1.1" xref="alg1.l15.m1.1.1.1.1.1.1.cmml">i</mi><mo mathsize="80%" id="alg1.l15.m1.2.2.2.2.2.4.1" xref="alg1.l15.m1.2.2.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg1.l15.m1.2.2.2.2.2.2" xref="alg1.l15.m1.2.2.2.2.2.2.cmml">j</mi></mrow></msub><mo mathsize="80%" id="alg1.l15.m1.2.2.2.3" xref="alg1.l15.m1.2.2.2.3.cmml">−</mo><msub id="alg1.l15.m1.2.2.2.5" xref="alg1.l15.m1.2.2.2.5.cmml"><mi mathsize="80%" id="alg1.l15.m1.2.2.2.5.2" xref="alg1.l15.m1.2.2.2.5.2.cmml">m</mi><mtext mathsize="80%" id="alg1.l15.m1.2.2.2.5.3" xref="alg1.l15.m1.2.2.2.5.3a.cmml">new</mtext></msub></mrow></msup><mo lspace="0em" rspace="0em" id="alg1.l15.m1.4.5.3.3.1" xref="alg1.l15.m1.4.5.3.3.1.cmml">​</mo><msub id="alg1.l15.m1.4.5.3.3.3" xref="alg1.l15.m1.4.5.3.3.3.cmml"><mi mathsize="80%" id="alg1.l15.m1.4.5.3.3.3.2" xref="alg1.l15.m1.4.5.3.3.3.2.cmml">𝐎</mi><mrow id="alg1.l15.m1.4.4.2.4" xref="alg1.l15.m1.4.4.2.3.cmml"><mi mathsize="80%" id="alg1.l15.m1.3.3.1.1" xref="alg1.l15.m1.3.3.1.1.cmml">i</mi><mo mathsize="80%" id="alg1.l15.m1.4.4.2.4.1" xref="alg1.l15.m1.4.4.2.3.cmml">,</mo><mi mathsize="80%" id="alg1.l15.m1.4.4.2.2" xref="alg1.l15.m1.4.4.2.2.cmml">j</mi></mrow></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l15.m1.4b"><apply id="alg1.l15.m1.4.5.cmml" xref="alg1.l15.m1.4.5"><eq id="alg1.l15.m1.4.5.1.cmml" xref="alg1.l15.m1.4.5.1"></eq><apply id="alg1.l15.m1.4.5.2.cmml" xref="alg1.l15.m1.4.5.2"><csymbol cd="ambiguous" id="alg1.l15.m1.4.5.2.1.cmml" xref="alg1.l15.m1.4.5.2">subscript</csymbol><ci id="alg1.l15.m1.4.5.2.2.cmml" xref="alg1.l15.m1.4.5.2.2">𝐎</ci><ci id="alg1.l15.m1.4.5.2.3.cmml" xref="alg1.l15.m1.4.5.2.3">𝑖</ci></apply><apply id="alg1.l15.m1.4.5.3.cmml" xref="alg1.l15.m1.4.5.3"><plus id="alg1.l15.m1.4.5.3.1.cmml" xref="alg1.l15.m1.4.5.3.1"></plus><apply id="alg1.l15.m1.4.5.3.2.cmml" xref="alg1.l15.m1.4.5.3.2"><times id="alg1.l15.m1.4.5.3.2.1.cmml" xref="alg1.l15.m1.4.5.3.2.1"></times><apply id="alg1.l15.m1.4.5.3.2.2.cmml" xref="alg1.l15.m1.4.5.3.2.2"><csymbol cd="ambiguous" id="alg1.l15.m1.4.5.3.2.2.1.cmml" xref="alg1.l15.m1.4.5.3.2.2">superscript</csymbol><ci id="alg1.l15.m1.4.5.3.2.2.2.cmml" xref="alg1.l15.m1.4.5.3.2.2.2">𝑒</ci><apply id="alg1.l15.m1.4.5.3.2.2.3.cmml" xref="alg1.l15.m1.4.5.3.2.2.3"><minus id="alg1.l15.m1.4.5.3.2.2.3.1.cmml" xref="alg1.l15.m1.4.5.3.2.2.3.1"></minus><apply id="alg1.l15.m1.4.5.3.2.2.3.2.cmml" xref="alg1.l15.m1.4.5.3.2.2.3.2"><csymbol cd="ambiguous" id="alg1.l15.m1.4.5.3.2.2.3.2.1.cmml" xref="alg1.l15.m1.4.5.3.2.2.3.2">subscript</csymbol><ci id="alg1.l15.m1.4.5.3.2.2.3.2.2.cmml" xref="alg1.l15.m1.4.5.3.2.2.3.2.2">𝑚</ci><ci id="alg1.l15.m1.4.5.3.2.2.3.2.3.cmml" xref="alg1.l15.m1.4.5.3.2.2.3.2.3">𝑖</ci></apply><apply id="alg1.l15.m1.4.5.3.2.2.3.3.cmml" xref="alg1.l15.m1.4.5.3.2.2.3.3"><csymbol cd="ambiguous" id="alg1.l15.m1.4.5.3.2.2.3.3.1.cmml" xref="alg1.l15.m1.4.5.3.2.2.3.3">subscript</csymbol><ci id="alg1.l15.m1.4.5.3.2.2.3.3.2.cmml" xref="alg1.l15.m1.4.5.3.2.2.3.3.2">𝑚</ci><ci id="alg1.l15.m1.4.5.3.2.2.3.3.3a.cmml" xref="alg1.l15.m1.4.5.3.2.2.3.3.3"><mtext mathsize="40%" id="alg1.l15.m1.4.5.3.2.2.3.3.3.cmml" xref="alg1.l15.m1.4.5.3.2.2.3.3.3">new</mtext></ci></apply></apply></apply><apply id="alg1.l15.m1.4.5.3.2.3.cmml" xref="alg1.l15.m1.4.5.3.2.3"><csymbol cd="ambiguous" id="alg1.l15.m1.4.5.3.2.3.1.cmml" xref="alg1.l15.m1.4.5.3.2.3">subscript</csymbol><ci id="alg1.l15.m1.4.5.3.2.3.2.cmml" xref="alg1.l15.m1.4.5.3.2.3.2">𝐎</ci><ci id="alg1.l15.m1.4.5.3.2.3.3.cmml" xref="alg1.l15.m1.4.5.3.2.3.3">𝑖</ci></apply></apply><apply id="alg1.l15.m1.4.5.3.3.cmml" xref="alg1.l15.m1.4.5.3.3"><times id="alg1.l15.m1.4.5.3.3.1.cmml" xref="alg1.l15.m1.4.5.3.3.1"></times><apply id="alg1.l15.m1.4.5.3.3.2.cmml" xref="alg1.l15.m1.4.5.3.3.2"><csymbol cd="ambiguous" id="alg1.l15.m1.4.5.3.3.2.1.cmml" xref="alg1.l15.m1.4.5.3.3.2">superscript</csymbol><ci id="alg1.l15.m1.4.5.3.3.2.2.cmml" xref="alg1.l15.m1.4.5.3.3.2.2">𝑒</ci><apply id="alg1.l15.m1.2.2.2.cmml" xref="alg1.l15.m1.2.2.2"><minus id="alg1.l15.m1.2.2.2.3.cmml" xref="alg1.l15.m1.2.2.2.3"></minus><apply id="alg1.l15.m1.2.2.2.4.cmml" xref="alg1.l15.m1.2.2.2.4"><csymbol cd="ambiguous" id="alg1.l15.m1.2.2.2.4.1.cmml" xref="alg1.l15.m1.2.2.2.4">subscript</csymbol><ci id="alg1.l15.m1.2.2.2.4.2.cmml" xref="alg1.l15.m1.2.2.2.4.2">𝑚</ci><list id="alg1.l15.m1.2.2.2.2.2.3.cmml" xref="alg1.l15.m1.2.2.2.2.2.4"><ci id="alg1.l15.m1.1.1.1.1.1.1.cmml" xref="alg1.l15.m1.1.1.1.1.1.1">𝑖</ci><ci id="alg1.l15.m1.2.2.2.2.2.2.cmml" xref="alg1.l15.m1.2.2.2.2.2.2">𝑗</ci></list></apply><apply id="alg1.l15.m1.2.2.2.5.cmml" xref="alg1.l15.m1.2.2.2.5"><csymbol cd="ambiguous" id="alg1.l15.m1.2.2.2.5.1.cmml" xref="alg1.l15.m1.2.2.2.5">subscript</csymbol><ci id="alg1.l15.m1.2.2.2.5.2.cmml" xref="alg1.l15.m1.2.2.2.5.2">𝑚</ci><ci id="alg1.l15.m1.2.2.2.5.3a.cmml" xref="alg1.l15.m1.2.2.2.5.3"><mtext mathsize="40%" id="alg1.l15.m1.2.2.2.5.3.cmml" xref="alg1.l15.m1.2.2.2.5.3">new</mtext></ci></apply></apply></apply><apply id="alg1.l15.m1.4.5.3.3.3.cmml" xref="alg1.l15.m1.4.5.3.3.3"><csymbol cd="ambiguous" id="alg1.l15.m1.4.5.3.3.3.1.cmml" xref="alg1.l15.m1.4.5.3.3.3">subscript</csymbol><ci id="alg1.l15.m1.4.5.3.3.3.2.cmml" xref="alg1.l15.m1.4.5.3.3.3.2">𝐎</ci><list id="alg1.l15.m1.4.4.2.3.cmml" xref="alg1.l15.m1.4.4.2.4"><ci id="alg1.l15.m1.3.3.1.1.cmml" xref="alg1.l15.m1.3.3.1.1">𝑖</ci><ci id="alg1.l15.m1.4.4.2.2.cmml" xref="alg1.l15.m1.4.4.2.2">𝑗</ci></list></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l15.m1.4c">\mathbf{O}_{i}=e^{m_{i}-m_{\text{new}}}\mathbf{O}_{i}+e^{m_{i,j}-m_{\text{new}}}\mathbf{O}_{i,j}</annotation></semantics></math><span id="alg1.l15.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg1.l16" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l16.1.1.1" class="ltx_text" style="font-size:80%;">16:</span></span><span id="alg1.l16.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg1.l16.m1.1" class="ltx_Math" alttext="m_{i}=m_{\text{new}}" display="inline"><semantics id="alg1.l16.m1.1a"><mrow id="alg1.l16.m1.1.1" xref="alg1.l16.m1.1.1.cmml"><msub id="alg1.l16.m1.1.1.2" xref="alg1.l16.m1.1.1.2.cmml"><mi mathsize="80%" id="alg1.l16.m1.1.1.2.2" xref="alg1.l16.m1.1.1.2.2.cmml">m</mi><mi mathsize="80%" id="alg1.l16.m1.1.1.2.3" xref="alg1.l16.m1.1.1.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg1.l16.m1.1.1.1" xref="alg1.l16.m1.1.1.1.cmml">=</mo><msub id="alg1.l16.m1.1.1.3" xref="alg1.l16.m1.1.1.3.cmml"><mi mathsize="80%" id="alg1.l16.m1.1.1.3.2" xref="alg1.l16.m1.1.1.3.2.cmml">m</mi><mtext mathsize="80%" id="alg1.l16.m1.1.1.3.3" xref="alg1.l16.m1.1.1.3.3a.cmml">new</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="alg1.l16.m1.1b"><apply id="alg1.l16.m1.1.1.cmml" xref="alg1.l16.m1.1.1"><eq id="alg1.l16.m1.1.1.1.cmml" xref="alg1.l16.m1.1.1.1"></eq><apply id="alg1.l16.m1.1.1.2.cmml" xref="alg1.l16.m1.1.1.2"><csymbol cd="ambiguous" id="alg1.l16.m1.1.1.2.1.cmml" xref="alg1.l16.m1.1.1.2">subscript</csymbol><ci id="alg1.l16.m1.1.1.2.2.cmml" xref="alg1.l16.m1.1.1.2.2">𝑚</ci><ci id="alg1.l16.m1.1.1.2.3.cmml" xref="alg1.l16.m1.1.1.2.3">𝑖</ci></apply><apply id="alg1.l16.m1.1.1.3.cmml" xref="alg1.l16.m1.1.1.3"><csymbol cd="ambiguous" id="alg1.l16.m1.1.1.3.1.cmml" xref="alg1.l16.m1.1.1.3">subscript</csymbol><ci id="alg1.l16.m1.1.1.3.2.cmml" xref="alg1.l16.m1.1.1.3.2">𝑚</ci><ci id="alg1.l16.m1.1.1.3.3a.cmml" xref="alg1.l16.m1.1.1.3.3"><mtext mathsize="56%" id="alg1.l16.m1.1.1.3.3.cmml" xref="alg1.l16.m1.1.1.3.3">new</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l16.m1.1c">m_{i}=m_{\text{new}}</annotation></semantics></math><span id="alg1.l16.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg1.l17" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l17.1.1.1" class="ltx_text" style="font-size:80%;">17:</span></span><span id="alg1.l17.2" class="ltx_text" style="font-size:80%;">    Put </span><math id="alg1.l17.m1.2" class="ltx_Math" alttext="\mathbf{K}_{j},\mathbf{V}_{j}" display="inline"><semantics id="alg1.l17.m1.2a"><mrow id="alg1.l17.m1.2.2.2" xref="alg1.l17.m1.2.2.3.cmml"><msub id="alg1.l17.m1.1.1.1.1" xref="alg1.l17.m1.1.1.1.1.cmml"><mi mathsize="80%" id="alg1.l17.m1.1.1.1.1.2" xref="alg1.l17.m1.1.1.1.1.2.cmml">𝐊</mi><mi mathsize="80%" id="alg1.l17.m1.1.1.1.1.3" xref="alg1.l17.m1.1.1.1.1.3.cmml">j</mi></msub><mo mathsize="80%" id="alg1.l17.m1.2.2.2.3" xref="alg1.l17.m1.2.2.3.cmml">,</mo><msub id="alg1.l17.m1.2.2.2.2" xref="alg1.l17.m1.2.2.2.2.cmml"><mi mathsize="80%" id="alg1.l17.m1.2.2.2.2.2" xref="alg1.l17.m1.2.2.2.2.2.cmml">𝐕</mi><mi mathsize="80%" id="alg1.l17.m1.2.2.2.2.3" xref="alg1.l17.m1.2.2.2.2.3.cmml">j</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="alg1.l17.m1.2b"><list id="alg1.l17.m1.2.2.3.cmml" xref="alg1.l17.m1.2.2.2"><apply id="alg1.l17.m1.1.1.1.1.cmml" xref="alg1.l17.m1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l17.m1.1.1.1.1.1.cmml" xref="alg1.l17.m1.1.1.1.1">subscript</csymbol><ci id="alg1.l17.m1.1.1.1.1.2.cmml" xref="alg1.l17.m1.1.1.1.1.2">𝐊</ci><ci id="alg1.l17.m1.1.1.1.1.3.cmml" xref="alg1.l17.m1.1.1.1.1.3">𝑗</ci></apply><apply id="alg1.l17.m1.2.2.2.2.cmml" xref="alg1.l17.m1.2.2.2.2"><csymbol cd="ambiguous" id="alg1.l17.m1.2.2.2.2.1.cmml" xref="alg1.l17.m1.2.2.2.2">subscript</csymbol><ci id="alg1.l17.m1.2.2.2.2.2.cmml" xref="alg1.l17.m1.2.2.2.2.2">𝐕</ci><ci id="alg1.l17.m1.2.2.2.2.3.cmml" xref="alg1.l17.m1.2.2.2.2.3">𝑗</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg1.l17.m1.2c">\mathbf{K}_{j},\mathbf{V}_{j}</annotation></semantics></math><span id="alg1.l17.3" class="ltx_text" style="font-size:80%;"> into communication ring;
</span>
</div>
<div id="alg1.l18" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l18.1.1.1" class="ltx_text" style="font-size:80%;">18:</span></span><span id="alg1.l18.2" class="ltx_text" style="font-size:80%;">  </span><span id="alg1.l18.3" class="ltx_text ltx_font_bold" style="font-size:80%;">end</span><span id="alg1.l18.4" class="ltx_text" style="font-size:80%;"> </span><span id="alg1.l18.5" class="ltx_text ltx_font_bold" style="font-size:80%;">for</span>
</div>
<div id="alg1.l19" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l19.1.1.1" class="ltx_text" style="font-size:80%;">19:</span></span><span id="alg1.l19.2" class="ltx_text" style="font-size:80%;">  </span><math id="alg1.l19.m1.1" class="ltx_Math" alttext="\mathbf{O}_{i}=\text{diag}(l_{i})^{-1}\mathbf{O}_{i}" display="inline"><semantics id="alg1.l19.m1.1a"><mrow id="alg1.l19.m1.1.1" xref="alg1.l19.m1.1.1.cmml"><msub id="alg1.l19.m1.1.1.3" xref="alg1.l19.m1.1.1.3.cmml"><mi mathsize="80%" id="alg1.l19.m1.1.1.3.2" xref="alg1.l19.m1.1.1.3.2.cmml">𝐎</mi><mi mathsize="80%" id="alg1.l19.m1.1.1.3.3" xref="alg1.l19.m1.1.1.3.3.cmml">i</mi></msub><mo mathsize="80%" id="alg1.l19.m1.1.1.2" xref="alg1.l19.m1.1.1.2.cmml">=</mo><mrow id="alg1.l19.m1.1.1.1" xref="alg1.l19.m1.1.1.1.cmml"><mtext mathsize="80%" id="alg1.l19.m1.1.1.1.3" xref="alg1.l19.m1.1.1.1.3a.cmml">diag</mtext><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.1.2" xref="alg1.l19.m1.1.1.1.2.cmml">​</mo><msup id="alg1.l19.m1.1.1.1.1" xref="alg1.l19.m1.1.1.1.1.cmml"><mrow id="alg1.l19.m1.1.1.1.1.1.1" xref="alg1.l19.m1.1.1.1.1.1.1.1.cmml"><mo maxsize="80%" minsize="80%" id="alg1.l19.m1.1.1.1.1.1.1.2" xref="alg1.l19.m1.1.1.1.1.1.1.1.cmml">(</mo><msub id="alg1.l19.m1.1.1.1.1.1.1.1" xref="alg1.l19.m1.1.1.1.1.1.1.1.cmml"><mi mathsize="80%" id="alg1.l19.m1.1.1.1.1.1.1.1.2" xref="alg1.l19.m1.1.1.1.1.1.1.1.2.cmml">l</mi><mi mathsize="80%" id="alg1.l19.m1.1.1.1.1.1.1.1.3" xref="alg1.l19.m1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo maxsize="80%" minsize="80%" id="alg1.l19.m1.1.1.1.1.1.1.3" xref="alg1.l19.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mrow id="alg1.l19.m1.1.1.1.1.3" xref="alg1.l19.m1.1.1.1.1.3.cmml"><mo mathsize="80%" id="alg1.l19.m1.1.1.1.1.3a" xref="alg1.l19.m1.1.1.1.1.3.cmml">−</mo><mn mathsize="80%" id="alg1.l19.m1.1.1.1.1.3.2" xref="alg1.l19.m1.1.1.1.1.3.2.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="alg1.l19.m1.1.1.1.2a" xref="alg1.l19.m1.1.1.1.2.cmml">​</mo><msub id="alg1.l19.m1.1.1.1.4" xref="alg1.l19.m1.1.1.1.4.cmml"><mi mathsize="80%" id="alg1.l19.m1.1.1.1.4.2" xref="alg1.l19.m1.1.1.1.4.2.cmml">𝐎</mi><mi mathsize="80%" id="alg1.l19.m1.1.1.1.4.3" xref="alg1.l19.m1.1.1.1.4.3.cmml">i</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l19.m1.1b"><apply id="alg1.l19.m1.1.1.cmml" xref="alg1.l19.m1.1.1"><eq id="alg1.l19.m1.1.1.2.cmml" xref="alg1.l19.m1.1.1.2"></eq><apply id="alg1.l19.m1.1.1.3.cmml" xref="alg1.l19.m1.1.1.3"><csymbol cd="ambiguous" id="alg1.l19.m1.1.1.3.1.cmml" xref="alg1.l19.m1.1.1.3">subscript</csymbol><ci id="alg1.l19.m1.1.1.3.2.cmml" xref="alg1.l19.m1.1.1.3.2">𝐎</ci><ci id="alg1.l19.m1.1.1.3.3.cmml" xref="alg1.l19.m1.1.1.3.3">𝑖</ci></apply><apply id="alg1.l19.m1.1.1.1.cmml" xref="alg1.l19.m1.1.1.1"><times id="alg1.l19.m1.1.1.1.2.cmml" xref="alg1.l19.m1.1.1.1.2"></times><ci id="alg1.l19.m1.1.1.1.3a.cmml" xref="alg1.l19.m1.1.1.1.3"><mtext mathsize="80%" id="alg1.l19.m1.1.1.1.3.cmml" xref="alg1.l19.m1.1.1.1.3">diag</mtext></ci><apply id="alg1.l19.m1.1.1.1.1.cmml" xref="alg1.l19.m1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l19.m1.1.1.1.1.2.cmml" xref="alg1.l19.m1.1.1.1.1">superscript</csymbol><apply id="alg1.l19.m1.1.1.1.1.1.1.1.cmml" xref="alg1.l19.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l19.m1.1.1.1.1.1.1.1.1.cmml" xref="alg1.l19.m1.1.1.1.1.1.1">subscript</csymbol><ci id="alg1.l19.m1.1.1.1.1.1.1.1.2.cmml" xref="alg1.l19.m1.1.1.1.1.1.1.1.2">𝑙</ci><ci id="alg1.l19.m1.1.1.1.1.1.1.1.3.cmml" xref="alg1.l19.m1.1.1.1.1.1.1.1.3">𝑖</ci></apply><apply id="alg1.l19.m1.1.1.1.1.3.cmml" xref="alg1.l19.m1.1.1.1.1.3"><minus id="alg1.l19.m1.1.1.1.1.3.1.cmml" xref="alg1.l19.m1.1.1.1.1.3"></minus><cn type="integer" id="alg1.l19.m1.1.1.1.1.3.2.cmml" xref="alg1.l19.m1.1.1.1.1.3.2">1</cn></apply></apply><apply id="alg1.l19.m1.1.1.1.4.cmml" xref="alg1.l19.m1.1.1.1.4"><csymbol cd="ambiguous" id="alg1.l19.m1.1.1.1.4.1.cmml" xref="alg1.l19.m1.1.1.1.4">subscript</csymbol><ci id="alg1.l19.m1.1.1.1.4.2.cmml" xref="alg1.l19.m1.1.1.1.4.2">𝐎</ci><ci id="alg1.l19.m1.1.1.1.4.3.cmml" xref="alg1.l19.m1.1.1.1.4.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l19.m1.1c">\mathbf{O}_{i}=\text{diag}(l_{i})^{-1}\mathbf{O}_{i}</annotation></semantics></math><span id="alg1.l19.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg1.l20" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l20.1.1.1" class="ltx_text" style="font-size:80%;">20:</span></span><span id="alg1.l20.2" class="ltx_text" style="font-size:80%;">  </span><math id="alg1.l20.m1.1" class="ltx_Math" alttext="lse_{i}=m_{i}+\log{l_{i}}" display="inline"><semantics id="alg1.l20.m1.1a"><mrow id="alg1.l20.m1.1.1" xref="alg1.l20.m1.1.1.cmml"><mrow id="alg1.l20.m1.1.1.2" xref="alg1.l20.m1.1.1.2.cmml"><mi mathsize="80%" id="alg1.l20.m1.1.1.2.2" xref="alg1.l20.m1.1.1.2.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l20.m1.1.1.2.1" xref="alg1.l20.m1.1.1.2.1.cmml">​</mo><mi mathsize="80%" id="alg1.l20.m1.1.1.2.3" xref="alg1.l20.m1.1.1.2.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l20.m1.1.1.2.1a" xref="alg1.l20.m1.1.1.2.1.cmml">​</mo><msub id="alg1.l20.m1.1.1.2.4" xref="alg1.l20.m1.1.1.2.4.cmml"><mi mathsize="80%" id="alg1.l20.m1.1.1.2.4.2" xref="alg1.l20.m1.1.1.2.4.2.cmml">e</mi><mi mathsize="80%" id="alg1.l20.m1.1.1.2.4.3" xref="alg1.l20.m1.1.1.2.4.3.cmml">i</mi></msub></mrow><mo mathsize="80%" id="alg1.l20.m1.1.1.1" xref="alg1.l20.m1.1.1.1.cmml">=</mo><mrow id="alg1.l20.m1.1.1.3" xref="alg1.l20.m1.1.1.3.cmml"><msub id="alg1.l20.m1.1.1.3.2" xref="alg1.l20.m1.1.1.3.2.cmml"><mi mathsize="80%" id="alg1.l20.m1.1.1.3.2.2" xref="alg1.l20.m1.1.1.3.2.2.cmml">m</mi><mi mathsize="80%" id="alg1.l20.m1.1.1.3.2.3" xref="alg1.l20.m1.1.1.3.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg1.l20.m1.1.1.3.1" xref="alg1.l20.m1.1.1.3.1.cmml">+</mo><mrow id="alg1.l20.m1.1.1.3.3" xref="alg1.l20.m1.1.1.3.3.cmml"><mi mathsize="80%" id="alg1.l20.m1.1.1.3.3.1" xref="alg1.l20.m1.1.1.3.3.1.cmml">log</mi><mo lspace="0.167em" id="alg1.l20.m1.1.1.3.3a" xref="alg1.l20.m1.1.1.3.3.cmml">⁡</mo><msub id="alg1.l20.m1.1.1.3.3.2" xref="alg1.l20.m1.1.1.3.3.2.cmml"><mi mathsize="80%" id="alg1.l20.m1.1.1.3.3.2.2" xref="alg1.l20.m1.1.1.3.3.2.2.cmml">l</mi><mi mathsize="80%" id="alg1.l20.m1.1.1.3.3.2.3" xref="alg1.l20.m1.1.1.3.3.2.3.cmml">i</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l20.m1.1b"><apply id="alg1.l20.m1.1.1.cmml" xref="alg1.l20.m1.1.1"><eq id="alg1.l20.m1.1.1.1.cmml" xref="alg1.l20.m1.1.1.1"></eq><apply id="alg1.l20.m1.1.1.2.cmml" xref="alg1.l20.m1.1.1.2"><times id="alg1.l20.m1.1.1.2.1.cmml" xref="alg1.l20.m1.1.1.2.1"></times><ci id="alg1.l20.m1.1.1.2.2.cmml" xref="alg1.l20.m1.1.1.2.2">𝑙</ci><ci id="alg1.l20.m1.1.1.2.3.cmml" xref="alg1.l20.m1.1.1.2.3">𝑠</ci><apply id="alg1.l20.m1.1.1.2.4.cmml" xref="alg1.l20.m1.1.1.2.4"><csymbol cd="ambiguous" id="alg1.l20.m1.1.1.2.4.1.cmml" xref="alg1.l20.m1.1.1.2.4">subscript</csymbol><ci id="alg1.l20.m1.1.1.2.4.2.cmml" xref="alg1.l20.m1.1.1.2.4.2">𝑒</ci><ci id="alg1.l20.m1.1.1.2.4.3.cmml" xref="alg1.l20.m1.1.1.2.4.3">𝑖</ci></apply></apply><apply id="alg1.l20.m1.1.1.3.cmml" xref="alg1.l20.m1.1.1.3"><plus id="alg1.l20.m1.1.1.3.1.cmml" xref="alg1.l20.m1.1.1.3.1"></plus><apply id="alg1.l20.m1.1.1.3.2.cmml" xref="alg1.l20.m1.1.1.3.2"><csymbol cd="ambiguous" id="alg1.l20.m1.1.1.3.2.1.cmml" xref="alg1.l20.m1.1.1.3.2">subscript</csymbol><ci id="alg1.l20.m1.1.1.3.2.2.cmml" xref="alg1.l20.m1.1.1.3.2.2">𝑚</ci><ci id="alg1.l20.m1.1.1.3.2.3.cmml" xref="alg1.l20.m1.1.1.3.2.3">𝑖</ci></apply><apply id="alg1.l20.m1.1.1.3.3.cmml" xref="alg1.l20.m1.1.1.3.3"><log id="alg1.l20.m1.1.1.3.3.1.cmml" xref="alg1.l20.m1.1.1.3.3.1"></log><apply id="alg1.l20.m1.1.1.3.3.2.cmml" xref="alg1.l20.m1.1.1.3.3.2"><csymbol cd="ambiguous" id="alg1.l20.m1.1.1.3.3.2.1.cmml" xref="alg1.l20.m1.1.1.3.3.2">subscript</csymbol><ci id="alg1.l20.m1.1.1.3.3.2.2.cmml" xref="alg1.l20.m1.1.1.3.3.2.2">𝑙</ci><ci id="alg1.l20.m1.1.1.3.3.2.3.cmml" xref="alg1.l20.m1.1.1.3.3.2.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l20.m1.1c">lse_{i}=m_{i}+\log{l_{i}}</annotation></semantics></math><span id="alg1.l20.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg1.l20a" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l20a.1.1.1" class="ltx_text" style="font-size:80%;">20:</span></span><span id="alg1.l20a.2" class="ltx_text" style="font-size:80%;">  </span><math id="alg1.l20a.m1.2" class="ltx_Math" alttext="\mathbf{O}_{i},lse_{i}" display="inline"><semantics id="alg1.l20a.m1.2a"><mrow id="alg1.l20a.m1.2.2.2" xref="alg1.l20a.m1.2.2.3.cmml"><msub id="alg1.l20a.m1.1.1.1.1" xref="alg1.l20a.m1.1.1.1.1.cmml"><mi mathsize="80%" id="alg1.l20a.m1.1.1.1.1.2" xref="alg1.l20a.m1.1.1.1.1.2.cmml">𝐎</mi><mi mathsize="80%" id="alg1.l20a.m1.1.1.1.1.3" xref="alg1.l20a.m1.1.1.1.1.3.cmml">i</mi></msub><mo mathsize="80%" id="alg1.l20a.m1.2.2.2.3" xref="alg1.l20a.m1.2.2.3.cmml">,</mo><mrow id="alg1.l20a.m1.2.2.2.2" xref="alg1.l20a.m1.2.2.2.2.cmml"><mi mathsize="80%" id="alg1.l20a.m1.2.2.2.2.2" xref="alg1.l20a.m1.2.2.2.2.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l20a.m1.2.2.2.2.1" xref="alg1.l20a.m1.2.2.2.2.1.cmml">​</mo><mi mathsize="80%" id="alg1.l20a.m1.2.2.2.2.3" xref="alg1.l20a.m1.2.2.2.2.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l20a.m1.2.2.2.2.1a" xref="alg1.l20a.m1.2.2.2.2.1.cmml">​</mo><msub id="alg1.l20a.m1.2.2.2.2.4" xref="alg1.l20a.m1.2.2.2.2.4.cmml"><mi mathsize="80%" id="alg1.l20a.m1.2.2.2.2.4.2" xref="alg1.l20a.m1.2.2.2.2.4.2.cmml">e</mi><mi mathsize="80%" id="alg1.l20a.m1.2.2.2.2.4.3" xref="alg1.l20a.m1.2.2.2.2.4.3.cmml">i</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l20a.m1.2b"><list id="alg1.l20a.m1.2.2.3.cmml" xref="alg1.l20a.m1.2.2.2"><apply id="alg1.l20a.m1.1.1.1.1.cmml" xref="alg1.l20a.m1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l20a.m1.1.1.1.1.1.cmml" xref="alg1.l20a.m1.1.1.1.1">subscript</csymbol><ci id="alg1.l20a.m1.1.1.1.1.2.cmml" xref="alg1.l20a.m1.1.1.1.1.2">𝐎</ci><ci id="alg1.l20a.m1.1.1.1.1.3.cmml" xref="alg1.l20a.m1.1.1.1.1.3">𝑖</ci></apply><apply id="alg1.l20a.m1.2.2.2.2.cmml" xref="alg1.l20a.m1.2.2.2.2"><times id="alg1.l20a.m1.2.2.2.2.1.cmml" xref="alg1.l20a.m1.2.2.2.2.1"></times><ci id="alg1.l20a.m1.2.2.2.2.2.cmml" xref="alg1.l20a.m1.2.2.2.2.2">𝑙</ci><ci id="alg1.l20a.m1.2.2.2.2.3.cmml" xref="alg1.l20a.m1.2.2.2.2.3">𝑠</ci><apply id="alg1.l20a.m1.2.2.2.2.4.cmml" xref="alg1.l20a.m1.2.2.2.2.4"><csymbol cd="ambiguous" id="alg1.l20a.m1.2.2.2.2.4.1.cmml" xref="alg1.l20a.m1.2.2.2.2.4">subscript</csymbol><ci id="alg1.l20a.m1.2.2.2.2.4.2.cmml" xref="alg1.l20a.m1.2.2.2.2.4.2">𝑒</ci><ci id="alg1.l20a.m1.2.2.2.2.4.3.cmml" xref="alg1.l20a.m1.2.2.2.2.4.3">𝑖</ci></apply></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg1.l20a.m1.2c">\mathbf{O}_{i},lse_{i}</annotation></semantics></math><span id="alg1.l20a.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
</div>
</figure>
<figure id="alg2" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<div id="alg2.2" class="ltx_listing ltx_listing">
<div id="alg2.l0" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l0.1.1.1" class="ltx_text" style="font-size:80%;">0:</span></span><span id="alg2.l0.2" class="ltx_text" style="font-size:80%;">  Matrices </span><math id="alg2.l0.m1.5" class="ltx_Math" alttext="\mathbf{Q}_{i},\mathbf{K}_{i},\mathbf{V}_{i},\mathbf{O}_{i},\mathbf{dO}_{i}\in\mathbb{R}^{{\frac{N}{G}}\times d}" display="inline"><semantics id="alg2.l0.m1.5a"><mrow id="alg2.l0.m1.5.5" xref="alg2.l0.m1.5.5.cmml"><mrow id="alg2.l0.m1.5.5.5.5" xref="alg2.l0.m1.5.5.5.6.cmml"><msub id="alg2.l0.m1.1.1.1.1.1" xref="alg2.l0.m1.1.1.1.1.1.cmml"><mi mathsize="80%" id="alg2.l0.m1.1.1.1.1.1.2" xref="alg2.l0.m1.1.1.1.1.1.2.cmml">𝐐</mi><mi mathsize="80%" id="alg2.l0.m1.1.1.1.1.1.3" xref="alg2.l0.m1.1.1.1.1.1.3.cmml">i</mi></msub><mo mathsize="80%" id="alg2.l0.m1.5.5.5.5.6" xref="alg2.l0.m1.5.5.5.6.cmml">,</mo><msub id="alg2.l0.m1.2.2.2.2.2" xref="alg2.l0.m1.2.2.2.2.2.cmml"><mi mathsize="80%" id="alg2.l0.m1.2.2.2.2.2.2" xref="alg2.l0.m1.2.2.2.2.2.2.cmml">𝐊</mi><mi mathsize="80%" id="alg2.l0.m1.2.2.2.2.2.3" xref="alg2.l0.m1.2.2.2.2.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg2.l0.m1.5.5.5.5.7" xref="alg2.l0.m1.5.5.5.6.cmml">,</mo><msub id="alg2.l0.m1.3.3.3.3.3" xref="alg2.l0.m1.3.3.3.3.3.cmml"><mi mathsize="80%" id="alg2.l0.m1.3.3.3.3.3.2" xref="alg2.l0.m1.3.3.3.3.3.2.cmml">𝐕</mi><mi mathsize="80%" id="alg2.l0.m1.3.3.3.3.3.3" xref="alg2.l0.m1.3.3.3.3.3.3.cmml">i</mi></msub><mo mathsize="80%" id="alg2.l0.m1.5.5.5.5.8" xref="alg2.l0.m1.5.5.5.6.cmml">,</mo><msub id="alg2.l0.m1.4.4.4.4.4" xref="alg2.l0.m1.4.4.4.4.4.cmml"><mi mathsize="80%" id="alg2.l0.m1.4.4.4.4.4.2" xref="alg2.l0.m1.4.4.4.4.4.2.cmml">𝐎</mi><mi mathsize="80%" id="alg2.l0.m1.4.4.4.4.4.3" xref="alg2.l0.m1.4.4.4.4.4.3.cmml">i</mi></msub><mo mathsize="80%" id="alg2.l0.m1.5.5.5.5.9" xref="alg2.l0.m1.5.5.5.6.cmml">,</mo><msub id="alg2.l0.m1.5.5.5.5.5" xref="alg2.l0.m1.5.5.5.5.5.cmml"><mi mathsize="80%" id="alg2.l0.m1.5.5.5.5.5.2" xref="alg2.l0.m1.5.5.5.5.5.2.cmml">𝐝𝐎</mi><mi mathsize="80%" id="alg2.l0.m1.5.5.5.5.5.3" xref="alg2.l0.m1.5.5.5.5.5.3.cmml">i</mi></msub></mrow><mo mathsize="80%" id="alg2.l0.m1.5.5.6" xref="alg2.l0.m1.5.5.6.cmml">∈</mo><msup id="alg2.l0.m1.5.5.7" xref="alg2.l0.m1.5.5.7.cmml"><mi mathsize="80%" id="alg2.l0.m1.5.5.7.2" xref="alg2.l0.m1.5.5.7.2.cmml">ℝ</mi><mrow id="alg2.l0.m1.5.5.7.3" xref="alg2.l0.m1.5.5.7.3.cmml"><mfrac id="alg2.l0.m1.5.5.7.3.2" xref="alg2.l0.m1.5.5.7.3.2.cmml"><mi mathsize="80%" id="alg2.l0.m1.5.5.7.3.2.2" xref="alg2.l0.m1.5.5.7.3.2.2.cmml">N</mi><mi mathsize="80%" id="alg2.l0.m1.5.5.7.3.2.3" xref="alg2.l0.m1.5.5.7.3.2.3.cmml">G</mi></mfrac><mo lspace="0.222em" mathsize="80%" rspace="0.222em" id="alg2.l0.m1.5.5.7.3.1" xref="alg2.l0.m1.5.5.7.3.1.cmml">×</mo><mi mathsize="80%" id="alg2.l0.m1.5.5.7.3.3" xref="alg2.l0.m1.5.5.7.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="alg2.l0.m1.5b"><apply id="alg2.l0.m1.5.5.cmml" xref="alg2.l0.m1.5.5"><in id="alg2.l0.m1.5.5.6.cmml" xref="alg2.l0.m1.5.5.6"></in><list id="alg2.l0.m1.5.5.5.6.cmml" xref="alg2.l0.m1.5.5.5.5"><apply id="alg2.l0.m1.1.1.1.1.1.cmml" xref="alg2.l0.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.l0.m1.1.1.1.1.1.1.cmml" xref="alg2.l0.m1.1.1.1.1.1">subscript</csymbol><ci id="alg2.l0.m1.1.1.1.1.1.2.cmml" xref="alg2.l0.m1.1.1.1.1.1.2">𝐐</ci><ci id="alg2.l0.m1.1.1.1.1.1.3.cmml" xref="alg2.l0.m1.1.1.1.1.1.3">𝑖</ci></apply><apply id="alg2.l0.m1.2.2.2.2.2.cmml" xref="alg2.l0.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="alg2.l0.m1.2.2.2.2.2.1.cmml" xref="alg2.l0.m1.2.2.2.2.2">subscript</csymbol><ci id="alg2.l0.m1.2.2.2.2.2.2.cmml" xref="alg2.l0.m1.2.2.2.2.2.2">𝐊</ci><ci id="alg2.l0.m1.2.2.2.2.2.3.cmml" xref="alg2.l0.m1.2.2.2.2.2.3">𝑖</ci></apply><apply id="alg2.l0.m1.3.3.3.3.3.cmml" xref="alg2.l0.m1.3.3.3.3.3"><csymbol cd="ambiguous" id="alg2.l0.m1.3.3.3.3.3.1.cmml" xref="alg2.l0.m1.3.3.3.3.3">subscript</csymbol><ci id="alg2.l0.m1.3.3.3.3.3.2.cmml" xref="alg2.l0.m1.3.3.3.3.3.2">𝐕</ci><ci id="alg2.l0.m1.3.3.3.3.3.3.cmml" xref="alg2.l0.m1.3.3.3.3.3.3">𝑖</ci></apply><apply id="alg2.l0.m1.4.4.4.4.4.cmml" xref="alg2.l0.m1.4.4.4.4.4"><csymbol cd="ambiguous" id="alg2.l0.m1.4.4.4.4.4.1.cmml" xref="alg2.l0.m1.4.4.4.4.4">subscript</csymbol><ci id="alg2.l0.m1.4.4.4.4.4.2.cmml" xref="alg2.l0.m1.4.4.4.4.4.2">𝐎</ci><ci id="alg2.l0.m1.4.4.4.4.4.3.cmml" xref="alg2.l0.m1.4.4.4.4.4.3">𝑖</ci></apply><apply id="alg2.l0.m1.5.5.5.5.5.cmml" xref="alg2.l0.m1.5.5.5.5.5"><csymbol cd="ambiguous" id="alg2.l0.m1.5.5.5.5.5.1.cmml" xref="alg2.l0.m1.5.5.5.5.5">subscript</csymbol><ci id="alg2.l0.m1.5.5.5.5.5.2.cmml" xref="alg2.l0.m1.5.5.5.5.5.2">𝐝𝐎</ci><ci id="alg2.l0.m1.5.5.5.5.5.3.cmml" xref="alg2.l0.m1.5.5.5.5.5.3">𝑖</ci></apply></list><apply id="alg2.l0.m1.5.5.7.cmml" xref="alg2.l0.m1.5.5.7"><csymbol cd="ambiguous" id="alg2.l0.m1.5.5.7.1.cmml" xref="alg2.l0.m1.5.5.7">superscript</csymbol><ci id="alg2.l0.m1.5.5.7.2.cmml" xref="alg2.l0.m1.5.5.7.2">ℝ</ci><apply id="alg2.l0.m1.5.5.7.3.cmml" xref="alg2.l0.m1.5.5.7.3"><times id="alg2.l0.m1.5.5.7.3.1.cmml" xref="alg2.l0.m1.5.5.7.3.1"></times><apply id="alg2.l0.m1.5.5.7.3.2.cmml" xref="alg2.l0.m1.5.5.7.3.2"><divide id="alg2.l0.m1.5.5.7.3.2.1.cmml" xref="alg2.l0.m1.5.5.7.3.2"></divide><ci id="alg2.l0.m1.5.5.7.3.2.2.cmml" xref="alg2.l0.m1.5.5.7.3.2.2">𝑁</ci><ci id="alg2.l0.m1.5.5.7.3.2.3.cmml" xref="alg2.l0.m1.5.5.7.3.2.3">𝐺</ci></apply><ci id="alg2.l0.m1.5.5.7.3.3.cmml" xref="alg2.l0.m1.5.5.7.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l0.m1.5c">\mathbf{Q}_{i},\mathbf{K}_{i},\mathbf{V}_{i},\mathbf{O}_{i},\mathbf{dO}_{i}\in\mathbb{R}^{{\frac{N}{G}}\times d}</annotation></semantics></math><span id="alg2.l0.3" class="ltx_text" style="font-size:80%;">, </span><math id="alg2.l0.m2.1" class="ltx_Math" alttext="lse_{i}\in\mathbb{R}^{\frac{N}{G}}" display="inline"><semantics id="alg2.l0.m2.1a"><mrow id="alg2.l0.m2.1.1" xref="alg2.l0.m2.1.1.cmml"><mrow id="alg2.l0.m2.1.1.2" xref="alg2.l0.m2.1.1.2.cmml"><mi mathsize="80%" id="alg2.l0.m2.1.1.2.2" xref="alg2.l0.m2.1.1.2.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg2.l0.m2.1.1.2.1" xref="alg2.l0.m2.1.1.2.1.cmml">​</mo><mi mathsize="80%" id="alg2.l0.m2.1.1.2.3" xref="alg2.l0.m2.1.1.2.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg2.l0.m2.1.1.2.1a" xref="alg2.l0.m2.1.1.2.1.cmml">​</mo><msub id="alg2.l0.m2.1.1.2.4" xref="alg2.l0.m2.1.1.2.4.cmml"><mi mathsize="80%" id="alg2.l0.m2.1.1.2.4.2" xref="alg2.l0.m2.1.1.2.4.2.cmml">e</mi><mi mathsize="80%" id="alg2.l0.m2.1.1.2.4.3" xref="alg2.l0.m2.1.1.2.4.3.cmml">i</mi></msub></mrow><mo mathsize="80%" id="alg2.l0.m2.1.1.1" xref="alg2.l0.m2.1.1.1.cmml">∈</mo><msup id="alg2.l0.m2.1.1.3" xref="alg2.l0.m2.1.1.3.cmml"><mi mathsize="80%" id="alg2.l0.m2.1.1.3.2" xref="alg2.l0.m2.1.1.3.2.cmml">ℝ</mi><mfrac id="alg2.l0.m2.1.1.3.3" xref="alg2.l0.m2.1.1.3.3.cmml"><mi mathsize="80%" id="alg2.l0.m2.1.1.3.3.2" xref="alg2.l0.m2.1.1.3.3.2.cmml">N</mi><mi mathsize="80%" id="alg2.l0.m2.1.1.3.3.3" xref="alg2.l0.m2.1.1.3.3.3.cmml">G</mi></mfrac></msup></mrow><annotation-xml encoding="MathML-Content" id="alg2.l0.m2.1b"><apply id="alg2.l0.m2.1.1.cmml" xref="alg2.l0.m2.1.1"><in id="alg2.l0.m2.1.1.1.cmml" xref="alg2.l0.m2.1.1.1"></in><apply id="alg2.l0.m2.1.1.2.cmml" xref="alg2.l0.m2.1.1.2"><times id="alg2.l0.m2.1.1.2.1.cmml" xref="alg2.l0.m2.1.1.2.1"></times><ci id="alg2.l0.m2.1.1.2.2.cmml" xref="alg2.l0.m2.1.1.2.2">𝑙</ci><ci id="alg2.l0.m2.1.1.2.3.cmml" xref="alg2.l0.m2.1.1.2.3">𝑠</ci><apply id="alg2.l0.m2.1.1.2.4.cmml" xref="alg2.l0.m2.1.1.2.4"><csymbol cd="ambiguous" id="alg2.l0.m2.1.1.2.4.1.cmml" xref="alg2.l0.m2.1.1.2.4">subscript</csymbol><ci id="alg2.l0.m2.1.1.2.4.2.cmml" xref="alg2.l0.m2.1.1.2.4.2">𝑒</ci><ci id="alg2.l0.m2.1.1.2.4.3.cmml" xref="alg2.l0.m2.1.1.2.4.3">𝑖</ci></apply></apply><apply id="alg2.l0.m2.1.1.3.cmml" xref="alg2.l0.m2.1.1.3"><csymbol cd="ambiguous" id="alg2.l0.m2.1.1.3.1.cmml" xref="alg2.l0.m2.1.1.3">superscript</csymbol><ci id="alg2.l0.m2.1.1.3.2.cmml" xref="alg2.l0.m2.1.1.3.2">ℝ</ci><apply id="alg2.l0.m2.1.1.3.3.cmml" xref="alg2.l0.m2.1.1.3.3"><divide id="alg2.l0.m2.1.1.3.3.1.cmml" xref="alg2.l0.m2.1.1.3.3"></divide><ci id="alg2.l0.m2.1.1.3.3.2.cmml" xref="alg2.l0.m2.1.1.3.3.2">𝑁</ci><ci id="alg2.l0.m2.1.1.3.3.3.cmml" xref="alg2.l0.m2.1.1.3.3.3">𝐺</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l0.m2.1c">lse_{i}\in\mathbb{R}^{\frac{N}{G}}</annotation></semantics></math><span id="alg2.l0.4" class="ltx_text" style="font-size:80%;"> on the </span><math id="alg2.l0.m3.1" class="ltx_Math" alttext="i" display="inline"><semantics id="alg2.l0.m3.1a"><mi mathsize="80%" id="alg2.l0.m3.1.1" xref="alg2.l0.m3.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="alg2.l0.m3.1b"><ci id="alg2.l0.m3.1.1.cmml" xref="alg2.l0.m3.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l0.m3.1c">i</annotation></semantics></math><span id="alg2.l0.5" class="ltx_text" style="font-size:80%;">-th device
</span>
</div>
<div id="alg2.l1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l1.1.1.1" class="ltx_text" style="font-size:80%;">1:</span></span><span id="alg2.l1.2" class="ltx_text" style="font-size:80%;">  Initialize </span><math id="alg2.l1.m1.4" class="ltx_Math" alttext="\mathbf{dQ}_{i},\mathbf{dK}_{i},\mathbf{dV}_{i}=(0)_{{\frac{N}{G}}\times d}" display="inline"><semantics id="alg2.l1.m1.4a"><mrow id="alg2.l1.m1.4.4" xref="alg2.l1.m1.4.4.cmml"><mrow id="alg2.l1.m1.4.4.3.3" xref="alg2.l1.m1.4.4.3.4.cmml"><msub id="alg2.l1.m1.2.2.1.1.1" xref="alg2.l1.m1.2.2.1.1.1.cmml"><mi mathsize="80%" id="alg2.l1.m1.2.2.1.1.1.2" xref="alg2.l1.m1.2.2.1.1.1.2.cmml">𝐝𝐐</mi><mi mathsize="80%" id="alg2.l1.m1.2.2.1.1.1.3" xref="alg2.l1.m1.2.2.1.1.1.3.cmml">i</mi></msub><mo mathsize="80%" id="alg2.l1.m1.4.4.3.3.4" xref="alg2.l1.m1.4.4.3.4.cmml">,</mo><msub id="alg2.l1.m1.3.3.2.2.2" xref="alg2.l1.m1.3.3.2.2.2.cmml"><mi mathsize="80%" id="alg2.l1.m1.3.3.2.2.2.2" xref="alg2.l1.m1.3.3.2.2.2.2.cmml">𝐝𝐊</mi><mi mathsize="80%" id="alg2.l1.m1.3.3.2.2.2.3" xref="alg2.l1.m1.3.3.2.2.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg2.l1.m1.4.4.3.3.5" xref="alg2.l1.m1.4.4.3.4.cmml">,</mo><msub id="alg2.l1.m1.4.4.3.3.3" xref="alg2.l1.m1.4.4.3.3.3.cmml"><mi mathsize="80%" id="alg2.l1.m1.4.4.3.3.3.2" xref="alg2.l1.m1.4.4.3.3.3.2.cmml">𝐝𝐕</mi><mi mathsize="80%" id="alg2.l1.m1.4.4.3.3.3.3" xref="alg2.l1.m1.4.4.3.3.3.3.cmml">i</mi></msub></mrow><mo mathsize="80%" id="alg2.l1.m1.4.4.4" xref="alg2.l1.m1.4.4.4.cmml">=</mo><msub id="alg2.l1.m1.4.4.5" xref="alg2.l1.m1.4.4.5.cmml"><mrow id="alg2.l1.m1.4.4.5.2.2" xref="alg2.l1.m1.4.4.5.cmml"><mo maxsize="80%" minsize="80%" id="alg2.l1.m1.4.4.5.2.2.1" xref="alg2.l1.m1.4.4.5.cmml">(</mo><mn mathsize="80%" id="alg2.l1.m1.1.1" xref="alg2.l1.m1.1.1.cmml">0</mn><mo maxsize="80%" minsize="80%" id="alg2.l1.m1.4.4.5.2.2.2" xref="alg2.l1.m1.4.4.5.cmml">)</mo></mrow><mrow id="alg2.l1.m1.4.4.5.3" xref="alg2.l1.m1.4.4.5.3.cmml"><mfrac id="alg2.l1.m1.4.4.5.3.2" xref="alg2.l1.m1.4.4.5.3.2.cmml"><mi mathsize="80%" id="alg2.l1.m1.4.4.5.3.2.2" xref="alg2.l1.m1.4.4.5.3.2.2.cmml">N</mi><mi mathsize="80%" id="alg2.l1.m1.4.4.5.3.2.3" xref="alg2.l1.m1.4.4.5.3.2.3.cmml">G</mi></mfrac><mo lspace="0.222em" mathsize="80%" rspace="0.222em" id="alg2.l1.m1.4.4.5.3.1" xref="alg2.l1.m1.4.4.5.3.1.cmml">×</mo><mi mathsize="80%" id="alg2.l1.m1.4.4.5.3.3" xref="alg2.l1.m1.4.4.5.3.3.cmml">d</mi></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="alg2.l1.m1.4b"><apply id="alg2.l1.m1.4.4.cmml" xref="alg2.l1.m1.4.4"><eq id="alg2.l1.m1.4.4.4.cmml" xref="alg2.l1.m1.4.4.4"></eq><list id="alg2.l1.m1.4.4.3.4.cmml" xref="alg2.l1.m1.4.4.3.3"><apply id="alg2.l1.m1.2.2.1.1.1.cmml" xref="alg2.l1.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="alg2.l1.m1.2.2.1.1.1.1.cmml" xref="alg2.l1.m1.2.2.1.1.1">subscript</csymbol><ci id="alg2.l1.m1.2.2.1.1.1.2.cmml" xref="alg2.l1.m1.2.2.1.1.1.2">𝐝𝐐</ci><ci id="alg2.l1.m1.2.2.1.1.1.3.cmml" xref="alg2.l1.m1.2.2.1.1.1.3">𝑖</ci></apply><apply id="alg2.l1.m1.3.3.2.2.2.cmml" xref="alg2.l1.m1.3.3.2.2.2"><csymbol cd="ambiguous" id="alg2.l1.m1.3.3.2.2.2.1.cmml" xref="alg2.l1.m1.3.3.2.2.2">subscript</csymbol><ci id="alg2.l1.m1.3.3.2.2.2.2.cmml" xref="alg2.l1.m1.3.3.2.2.2.2">𝐝𝐊</ci><ci id="alg2.l1.m1.3.3.2.2.2.3.cmml" xref="alg2.l1.m1.3.3.2.2.2.3">𝑖</ci></apply><apply id="alg2.l1.m1.4.4.3.3.3.cmml" xref="alg2.l1.m1.4.4.3.3.3"><csymbol cd="ambiguous" id="alg2.l1.m1.4.4.3.3.3.1.cmml" xref="alg2.l1.m1.4.4.3.3.3">subscript</csymbol><ci id="alg2.l1.m1.4.4.3.3.3.2.cmml" xref="alg2.l1.m1.4.4.3.3.3.2">𝐝𝐕</ci><ci id="alg2.l1.m1.4.4.3.3.3.3.cmml" xref="alg2.l1.m1.4.4.3.3.3.3">𝑖</ci></apply></list><apply id="alg2.l1.m1.4.4.5.cmml" xref="alg2.l1.m1.4.4.5"><csymbol cd="ambiguous" id="alg2.l1.m1.4.4.5.1.cmml" xref="alg2.l1.m1.4.4.5">subscript</csymbol><cn type="integer" id="alg2.l1.m1.1.1.cmml" xref="alg2.l1.m1.1.1">0</cn><apply id="alg2.l1.m1.4.4.5.3.cmml" xref="alg2.l1.m1.4.4.5.3"><times id="alg2.l1.m1.4.4.5.3.1.cmml" xref="alg2.l1.m1.4.4.5.3.1"></times><apply id="alg2.l1.m1.4.4.5.3.2.cmml" xref="alg2.l1.m1.4.4.5.3.2"><divide id="alg2.l1.m1.4.4.5.3.2.1.cmml" xref="alg2.l1.m1.4.4.5.3.2"></divide><ci id="alg2.l1.m1.4.4.5.3.2.2.cmml" xref="alg2.l1.m1.4.4.5.3.2.2">𝑁</ci><ci id="alg2.l1.m1.4.4.5.3.2.3.cmml" xref="alg2.l1.m1.4.4.5.3.2.3">𝐺</ci></apply><ci id="alg2.l1.m1.4.4.5.3.3.cmml" xref="alg2.l1.m1.4.4.5.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l1.m1.4c">\mathbf{dQ}_{i},\mathbf{dK}_{i},\mathbf{dV}_{i}=(0)_{{\frac{N}{G}}\times d}</annotation></semantics></math><span id="alg2.l1.3" class="ltx_text" style="font-size:80%;"> </span>
</div>
<div id="alg2.l2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l2.1.1.1" class="ltx_text" style="font-size:80%;">2:</span></span><span id="alg2.l2.2" class="ltx_text" style="font-size:80%;">  </span><math id="alg2.l2.m1.1" class="ltx_Math" alttext="D_{i}=\text{rowsum}(\mathbf{dO}_{i}\circ\mathbf{O}_{i})" display="inline"><semantics id="alg2.l2.m1.1a"><mrow id="alg2.l2.m1.1.1" xref="alg2.l2.m1.1.1.cmml"><msub id="alg2.l2.m1.1.1.3" xref="alg2.l2.m1.1.1.3.cmml"><mi mathsize="80%" id="alg2.l2.m1.1.1.3.2" xref="alg2.l2.m1.1.1.3.2.cmml">D</mi><mi mathsize="80%" id="alg2.l2.m1.1.1.3.3" xref="alg2.l2.m1.1.1.3.3.cmml">i</mi></msub><mo mathsize="80%" id="alg2.l2.m1.1.1.2" xref="alg2.l2.m1.1.1.2.cmml">=</mo><mrow id="alg2.l2.m1.1.1.1" xref="alg2.l2.m1.1.1.1.cmml"><mtext mathsize="80%" id="alg2.l2.m1.1.1.1.3" xref="alg2.l2.m1.1.1.1.3a.cmml">rowsum</mtext><mo lspace="0em" rspace="0em" id="alg2.l2.m1.1.1.1.2" xref="alg2.l2.m1.1.1.1.2.cmml">​</mo><mrow id="alg2.l2.m1.1.1.1.1.1" xref="alg2.l2.m1.1.1.1.1.1.1.cmml"><mo maxsize="80%" minsize="80%" id="alg2.l2.m1.1.1.1.1.1.2" xref="alg2.l2.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="alg2.l2.m1.1.1.1.1.1.1" xref="alg2.l2.m1.1.1.1.1.1.1.cmml"><msub id="alg2.l2.m1.1.1.1.1.1.1.2" xref="alg2.l2.m1.1.1.1.1.1.1.2.cmml"><mi mathsize="80%" id="alg2.l2.m1.1.1.1.1.1.1.2.2" xref="alg2.l2.m1.1.1.1.1.1.1.2.2.cmml">𝐝𝐎</mi><mi mathsize="80%" id="alg2.l2.m1.1.1.1.1.1.1.2.3" xref="alg2.l2.m1.1.1.1.1.1.1.2.3.cmml">i</mi></msub><mo lspace="0.222em" mathsize="80%" rspace="0.222em" id="alg2.l2.m1.1.1.1.1.1.1.1" xref="alg2.l2.m1.1.1.1.1.1.1.1.cmml">∘</mo><msub id="alg2.l2.m1.1.1.1.1.1.1.3" xref="alg2.l2.m1.1.1.1.1.1.1.3.cmml"><mi mathsize="80%" id="alg2.l2.m1.1.1.1.1.1.1.3.2" xref="alg2.l2.m1.1.1.1.1.1.1.3.2.cmml">𝐎</mi><mi mathsize="80%" id="alg2.l2.m1.1.1.1.1.1.1.3.3" xref="alg2.l2.m1.1.1.1.1.1.1.3.3.cmml">i</mi></msub></mrow><mo maxsize="80%" minsize="80%" id="alg2.l2.m1.1.1.1.1.1.3" xref="alg2.l2.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l2.m1.1b"><apply id="alg2.l2.m1.1.1.cmml" xref="alg2.l2.m1.1.1"><eq id="alg2.l2.m1.1.1.2.cmml" xref="alg2.l2.m1.1.1.2"></eq><apply id="alg2.l2.m1.1.1.3.cmml" xref="alg2.l2.m1.1.1.3"><csymbol cd="ambiguous" id="alg2.l2.m1.1.1.3.1.cmml" xref="alg2.l2.m1.1.1.3">subscript</csymbol><ci id="alg2.l2.m1.1.1.3.2.cmml" xref="alg2.l2.m1.1.1.3.2">𝐷</ci><ci id="alg2.l2.m1.1.1.3.3.cmml" xref="alg2.l2.m1.1.1.3.3">𝑖</ci></apply><apply id="alg2.l2.m1.1.1.1.cmml" xref="alg2.l2.m1.1.1.1"><times id="alg2.l2.m1.1.1.1.2.cmml" xref="alg2.l2.m1.1.1.1.2"></times><ci id="alg2.l2.m1.1.1.1.3a.cmml" xref="alg2.l2.m1.1.1.1.3"><mtext mathsize="80%" id="alg2.l2.m1.1.1.1.3.cmml" xref="alg2.l2.m1.1.1.1.3">rowsum</mtext></ci><apply id="alg2.l2.m1.1.1.1.1.1.1.cmml" xref="alg2.l2.m1.1.1.1.1.1"><compose id="alg2.l2.m1.1.1.1.1.1.1.1.cmml" xref="alg2.l2.m1.1.1.1.1.1.1.1"></compose><apply id="alg2.l2.m1.1.1.1.1.1.1.2.cmml" xref="alg2.l2.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="alg2.l2.m1.1.1.1.1.1.1.2.1.cmml" xref="alg2.l2.m1.1.1.1.1.1.1.2">subscript</csymbol><ci id="alg2.l2.m1.1.1.1.1.1.1.2.2.cmml" xref="alg2.l2.m1.1.1.1.1.1.1.2.2">𝐝𝐎</ci><ci id="alg2.l2.m1.1.1.1.1.1.1.2.3.cmml" xref="alg2.l2.m1.1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="alg2.l2.m1.1.1.1.1.1.1.3.cmml" xref="alg2.l2.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="alg2.l2.m1.1.1.1.1.1.1.3.1.cmml" xref="alg2.l2.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="alg2.l2.m1.1.1.1.1.1.1.3.2.cmml" xref="alg2.l2.m1.1.1.1.1.1.1.3.2">𝐎</ci><ci id="alg2.l2.m1.1.1.1.1.1.1.3.3.cmml" xref="alg2.l2.m1.1.1.1.1.1.1.3.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l2.m1.1c">D_{i}=\text{rowsum}(\mathbf{dO}_{i}\circ\mathbf{O}_{i})</annotation></semantics></math><span id="alg2.l2.3" class="ltx_text" style="font-size:80%;">
(elementwise multiplication)
</span>
</div>
<div id="alg2.l3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l3.1.1.1" class="ltx_text" style="font-size:80%;">3:</span></span><span id="alg2.l3.2" class="ltx_text" style="font-size:80%;">  Put </span><math id="alg2.l3.m1.5" class="ltx_Math" alttext="\mathbf{Q}_{i},\mathbf{dQ}_{i},\mathbf{dO}_{i},D_{i},lse_{i}" display="inline"><semantics id="alg2.l3.m1.5a"><mrow id="alg2.l3.m1.5.5.5" xref="alg2.l3.m1.5.5.6.cmml"><msub id="alg2.l3.m1.1.1.1.1" xref="alg2.l3.m1.1.1.1.1.cmml"><mi mathsize="80%" id="alg2.l3.m1.1.1.1.1.2" xref="alg2.l3.m1.1.1.1.1.2.cmml">𝐐</mi><mi mathsize="80%" id="alg2.l3.m1.1.1.1.1.3" xref="alg2.l3.m1.1.1.1.1.3.cmml">i</mi></msub><mo mathsize="80%" id="alg2.l3.m1.5.5.5.6" xref="alg2.l3.m1.5.5.6.cmml">,</mo><msub id="alg2.l3.m1.2.2.2.2" xref="alg2.l3.m1.2.2.2.2.cmml"><mi mathsize="80%" id="alg2.l3.m1.2.2.2.2.2" xref="alg2.l3.m1.2.2.2.2.2.cmml">𝐝𝐐</mi><mi mathsize="80%" id="alg2.l3.m1.2.2.2.2.3" xref="alg2.l3.m1.2.2.2.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg2.l3.m1.5.5.5.7" xref="alg2.l3.m1.5.5.6.cmml">,</mo><msub id="alg2.l3.m1.3.3.3.3" xref="alg2.l3.m1.3.3.3.3.cmml"><mi mathsize="80%" id="alg2.l3.m1.3.3.3.3.2" xref="alg2.l3.m1.3.3.3.3.2.cmml">𝐝𝐎</mi><mi mathsize="80%" id="alg2.l3.m1.3.3.3.3.3" xref="alg2.l3.m1.3.3.3.3.3.cmml">i</mi></msub><mo mathsize="80%" id="alg2.l3.m1.5.5.5.8" xref="alg2.l3.m1.5.5.6.cmml">,</mo><msub id="alg2.l3.m1.4.4.4.4" xref="alg2.l3.m1.4.4.4.4.cmml"><mi mathsize="80%" id="alg2.l3.m1.4.4.4.4.2" xref="alg2.l3.m1.4.4.4.4.2.cmml">D</mi><mi mathsize="80%" id="alg2.l3.m1.4.4.4.4.3" xref="alg2.l3.m1.4.4.4.4.3.cmml">i</mi></msub><mo mathsize="80%" id="alg2.l3.m1.5.5.5.9" xref="alg2.l3.m1.5.5.6.cmml">,</mo><mrow id="alg2.l3.m1.5.5.5.5" xref="alg2.l3.m1.5.5.5.5.cmml"><mi mathsize="80%" id="alg2.l3.m1.5.5.5.5.2" xref="alg2.l3.m1.5.5.5.5.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg2.l3.m1.5.5.5.5.1" xref="alg2.l3.m1.5.5.5.5.1.cmml">​</mo><mi mathsize="80%" id="alg2.l3.m1.5.5.5.5.3" xref="alg2.l3.m1.5.5.5.5.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg2.l3.m1.5.5.5.5.1a" xref="alg2.l3.m1.5.5.5.5.1.cmml">​</mo><msub id="alg2.l3.m1.5.5.5.5.4" xref="alg2.l3.m1.5.5.5.5.4.cmml"><mi mathsize="80%" id="alg2.l3.m1.5.5.5.5.4.2" xref="alg2.l3.m1.5.5.5.5.4.2.cmml">e</mi><mi mathsize="80%" id="alg2.l3.m1.5.5.5.5.4.3" xref="alg2.l3.m1.5.5.5.5.4.3.cmml">i</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l3.m1.5b"><list id="alg2.l3.m1.5.5.6.cmml" xref="alg2.l3.m1.5.5.5"><apply id="alg2.l3.m1.1.1.1.1.cmml" xref="alg2.l3.m1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.l3.m1.1.1.1.1.1.cmml" xref="alg2.l3.m1.1.1.1.1">subscript</csymbol><ci id="alg2.l3.m1.1.1.1.1.2.cmml" xref="alg2.l3.m1.1.1.1.1.2">𝐐</ci><ci id="alg2.l3.m1.1.1.1.1.3.cmml" xref="alg2.l3.m1.1.1.1.1.3">𝑖</ci></apply><apply id="alg2.l3.m1.2.2.2.2.cmml" xref="alg2.l3.m1.2.2.2.2"><csymbol cd="ambiguous" id="alg2.l3.m1.2.2.2.2.1.cmml" xref="alg2.l3.m1.2.2.2.2">subscript</csymbol><ci id="alg2.l3.m1.2.2.2.2.2.cmml" xref="alg2.l3.m1.2.2.2.2.2">𝐝𝐐</ci><ci id="alg2.l3.m1.2.2.2.2.3.cmml" xref="alg2.l3.m1.2.2.2.2.3">𝑖</ci></apply><apply id="alg2.l3.m1.3.3.3.3.cmml" xref="alg2.l3.m1.3.3.3.3"><csymbol cd="ambiguous" id="alg2.l3.m1.3.3.3.3.1.cmml" xref="alg2.l3.m1.3.3.3.3">subscript</csymbol><ci id="alg2.l3.m1.3.3.3.3.2.cmml" xref="alg2.l3.m1.3.3.3.3.2">𝐝𝐎</ci><ci id="alg2.l3.m1.3.3.3.3.3.cmml" xref="alg2.l3.m1.3.3.3.3.3">𝑖</ci></apply><apply id="alg2.l3.m1.4.4.4.4.cmml" xref="alg2.l3.m1.4.4.4.4"><csymbol cd="ambiguous" id="alg2.l3.m1.4.4.4.4.1.cmml" xref="alg2.l3.m1.4.4.4.4">subscript</csymbol><ci id="alg2.l3.m1.4.4.4.4.2.cmml" xref="alg2.l3.m1.4.4.4.4.2">𝐷</ci><ci id="alg2.l3.m1.4.4.4.4.3.cmml" xref="alg2.l3.m1.4.4.4.4.3">𝑖</ci></apply><apply id="alg2.l3.m1.5.5.5.5.cmml" xref="alg2.l3.m1.5.5.5.5"><times id="alg2.l3.m1.5.5.5.5.1.cmml" xref="alg2.l3.m1.5.5.5.5.1"></times><ci id="alg2.l3.m1.5.5.5.5.2.cmml" xref="alg2.l3.m1.5.5.5.5.2">𝑙</ci><ci id="alg2.l3.m1.5.5.5.5.3.cmml" xref="alg2.l3.m1.5.5.5.5.3">𝑠</ci><apply id="alg2.l3.m1.5.5.5.5.4.cmml" xref="alg2.l3.m1.5.5.5.5.4"><csymbol cd="ambiguous" id="alg2.l3.m1.5.5.5.5.4.1.cmml" xref="alg2.l3.m1.5.5.5.5.4">subscript</csymbol><ci id="alg2.l3.m1.5.5.5.5.4.2.cmml" xref="alg2.l3.m1.5.5.5.5.4.2">𝑒</ci><ci id="alg2.l3.m1.5.5.5.5.4.3.cmml" xref="alg2.l3.m1.5.5.5.5.4.3">𝑖</ci></apply></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg2.l3.m1.5c">\mathbf{Q}_{i},\mathbf{dQ}_{i},\mathbf{dO}_{i},D_{i},lse_{i}</annotation></semantics></math><span id="alg2.l3.3" class="ltx_text" style="font-size:80%;"> into communication ring

</span>
</div>
<div id="alg2.l4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l4.1.1.1" class="ltx_text" style="font-size:80%;">4:</span></span><span id="alg2.l4.2" class="ltx_text" style="font-size:80%;">  </span><span id="alg2.l4.3" class="ltx_text ltx_font_bold" style="font-size:80%;">for</span><span id="alg2.l4.4" class="ltx_text" style="font-size:80%;"> </span><math id="alg2.l4.m1.1" class="ltx_Math" alttext="j=1" display="inline"><semantics id="alg2.l4.m1.1a"><mrow id="alg2.l4.m1.1.1" xref="alg2.l4.m1.1.1.cmml"><mi mathsize="80%" id="alg2.l4.m1.1.1.2" xref="alg2.l4.m1.1.1.2.cmml">j</mi><mo mathsize="80%" id="alg2.l4.m1.1.1.1" xref="alg2.l4.m1.1.1.1.cmml">=</mo><mn mathsize="80%" id="alg2.l4.m1.1.1.3" xref="alg2.l4.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg2.l4.m1.1b"><apply id="alg2.l4.m1.1.1.cmml" xref="alg2.l4.m1.1.1"><eq id="alg2.l4.m1.1.1.1.cmml" xref="alg2.l4.m1.1.1.1"></eq><ci id="alg2.l4.m1.1.1.2.cmml" xref="alg2.l4.m1.1.1.2">𝑗</ci><cn type="integer" id="alg2.l4.m1.1.1.3.cmml" xref="alg2.l4.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l4.m1.1c">j=1</annotation></semantics></math><span id="alg2.l4.5" class="ltx_text" style="font-size:80%;"> to </span><math id="alg2.l4.m2.1" class="ltx_Math" alttext="G" display="inline"><semantics id="alg2.l4.m2.1a"><mi mathsize="80%" id="alg2.l4.m2.1.1" xref="alg2.l4.m2.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="alg2.l4.m2.1b"><ci id="alg2.l4.m2.1.1.cmml" xref="alg2.l4.m2.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l4.m2.1c">G</annotation></semantics></math><span id="alg2.l4.6" class="ltx_text" style="font-size:80%;"> </span><span id="alg2.l4.7" class="ltx_text ltx_font_bold" style="font-size:80%;">do</span><span id="alg2.l4.8" class="ltx_text" style="font-size:80%;">


</span>
</div>
<div id="alg2.l5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l5.1.1.1" class="ltx_text" style="font-size:80%;">5:</span></span><span id="alg2.l5.2" class="ltx_text" style="font-size:80%;">    Conduct one step of ring communication;
</span>
</div>
<div id="alg2.l6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l6.1.1.1" class="ltx_text" style="font-size:80%;">6:</span></span><span id="alg2.l6.2" class="ltx_text" style="font-size:80%;">    Get </span><math id="alg2.l6.m1.5" class="ltx_Math" alttext="\mathbf{Q}_{j},\mathbf{dQ}_{j},\mathbf{dO}_{j},D_{j},lse_{j}" display="inline"><semantics id="alg2.l6.m1.5a"><mrow id="alg2.l6.m1.5.5.5" xref="alg2.l6.m1.5.5.6.cmml"><msub id="alg2.l6.m1.1.1.1.1" xref="alg2.l6.m1.1.1.1.1.cmml"><mi mathsize="80%" id="alg2.l6.m1.1.1.1.1.2" xref="alg2.l6.m1.1.1.1.1.2.cmml">𝐐</mi><mi mathsize="80%" id="alg2.l6.m1.1.1.1.1.3" xref="alg2.l6.m1.1.1.1.1.3.cmml">j</mi></msub><mo mathsize="80%" id="alg2.l6.m1.5.5.5.6" xref="alg2.l6.m1.5.5.6.cmml">,</mo><msub id="alg2.l6.m1.2.2.2.2" xref="alg2.l6.m1.2.2.2.2.cmml"><mi mathsize="80%" id="alg2.l6.m1.2.2.2.2.2" xref="alg2.l6.m1.2.2.2.2.2.cmml">𝐝𝐐</mi><mi mathsize="80%" id="alg2.l6.m1.2.2.2.2.3" xref="alg2.l6.m1.2.2.2.2.3.cmml">j</mi></msub><mo mathsize="80%" id="alg2.l6.m1.5.5.5.7" xref="alg2.l6.m1.5.5.6.cmml">,</mo><msub id="alg2.l6.m1.3.3.3.3" xref="alg2.l6.m1.3.3.3.3.cmml"><mi mathsize="80%" id="alg2.l6.m1.3.3.3.3.2" xref="alg2.l6.m1.3.3.3.3.2.cmml">𝐝𝐎</mi><mi mathsize="80%" id="alg2.l6.m1.3.3.3.3.3" xref="alg2.l6.m1.3.3.3.3.3.cmml">j</mi></msub><mo mathsize="80%" id="alg2.l6.m1.5.5.5.8" xref="alg2.l6.m1.5.5.6.cmml">,</mo><msub id="alg2.l6.m1.4.4.4.4" xref="alg2.l6.m1.4.4.4.4.cmml"><mi mathsize="80%" id="alg2.l6.m1.4.4.4.4.2" xref="alg2.l6.m1.4.4.4.4.2.cmml">D</mi><mi mathsize="80%" id="alg2.l6.m1.4.4.4.4.3" xref="alg2.l6.m1.4.4.4.4.3.cmml">j</mi></msub><mo mathsize="80%" id="alg2.l6.m1.5.5.5.9" xref="alg2.l6.m1.5.5.6.cmml">,</mo><mrow id="alg2.l6.m1.5.5.5.5" xref="alg2.l6.m1.5.5.5.5.cmml"><mi mathsize="80%" id="alg2.l6.m1.5.5.5.5.2" xref="alg2.l6.m1.5.5.5.5.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg2.l6.m1.5.5.5.5.1" xref="alg2.l6.m1.5.5.5.5.1.cmml">​</mo><mi mathsize="80%" id="alg2.l6.m1.5.5.5.5.3" xref="alg2.l6.m1.5.5.5.5.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg2.l6.m1.5.5.5.5.1a" xref="alg2.l6.m1.5.5.5.5.1.cmml">​</mo><msub id="alg2.l6.m1.5.5.5.5.4" xref="alg2.l6.m1.5.5.5.5.4.cmml"><mi mathsize="80%" id="alg2.l6.m1.5.5.5.5.4.2" xref="alg2.l6.m1.5.5.5.5.4.2.cmml">e</mi><mi mathsize="80%" id="alg2.l6.m1.5.5.5.5.4.3" xref="alg2.l6.m1.5.5.5.5.4.3.cmml">j</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l6.m1.5b"><list id="alg2.l6.m1.5.5.6.cmml" xref="alg2.l6.m1.5.5.5"><apply id="alg2.l6.m1.1.1.1.1.cmml" xref="alg2.l6.m1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.l6.m1.1.1.1.1.1.cmml" xref="alg2.l6.m1.1.1.1.1">subscript</csymbol><ci id="alg2.l6.m1.1.1.1.1.2.cmml" xref="alg2.l6.m1.1.1.1.1.2">𝐐</ci><ci id="alg2.l6.m1.1.1.1.1.3.cmml" xref="alg2.l6.m1.1.1.1.1.3">𝑗</ci></apply><apply id="alg2.l6.m1.2.2.2.2.cmml" xref="alg2.l6.m1.2.2.2.2"><csymbol cd="ambiguous" id="alg2.l6.m1.2.2.2.2.1.cmml" xref="alg2.l6.m1.2.2.2.2">subscript</csymbol><ci id="alg2.l6.m1.2.2.2.2.2.cmml" xref="alg2.l6.m1.2.2.2.2.2">𝐝𝐐</ci><ci id="alg2.l6.m1.2.2.2.2.3.cmml" xref="alg2.l6.m1.2.2.2.2.3">𝑗</ci></apply><apply id="alg2.l6.m1.3.3.3.3.cmml" xref="alg2.l6.m1.3.3.3.3"><csymbol cd="ambiguous" id="alg2.l6.m1.3.3.3.3.1.cmml" xref="alg2.l6.m1.3.3.3.3">subscript</csymbol><ci id="alg2.l6.m1.3.3.3.3.2.cmml" xref="alg2.l6.m1.3.3.3.3.2">𝐝𝐎</ci><ci id="alg2.l6.m1.3.3.3.3.3.cmml" xref="alg2.l6.m1.3.3.3.3.3">𝑗</ci></apply><apply id="alg2.l6.m1.4.4.4.4.cmml" xref="alg2.l6.m1.4.4.4.4"><csymbol cd="ambiguous" id="alg2.l6.m1.4.4.4.4.1.cmml" xref="alg2.l6.m1.4.4.4.4">subscript</csymbol><ci id="alg2.l6.m1.4.4.4.4.2.cmml" xref="alg2.l6.m1.4.4.4.4.2">𝐷</ci><ci id="alg2.l6.m1.4.4.4.4.3.cmml" xref="alg2.l6.m1.4.4.4.4.3">𝑗</ci></apply><apply id="alg2.l6.m1.5.5.5.5.cmml" xref="alg2.l6.m1.5.5.5.5"><times id="alg2.l6.m1.5.5.5.5.1.cmml" xref="alg2.l6.m1.5.5.5.5.1"></times><ci id="alg2.l6.m1.5.5.5.5.2.cmml" xref="alg2.l6.m1.5.5.5.5.2">𝑙</ci><ci id="alg2.l6.m1.5.5.5.5.3.cmml" xref="alg2.l6.m1.5.5.5.5.3">𝑠</ci><apply id="alg2.l6.m1.5.5.5.5.4.cmml" xref="alg2.l6.m1.5.5.5.5.4"><csymbol cd="ambiguous" id="alg2.l6.m1.5.5.5.5.4.1.cmml" xref="alg2.l6.m1.5.5.5.5.4">subscript</csymbol><ci id="alg2.l6.m1.5.5.5.5.4.2.cmml" xref="alg2.l6.m1.5.5.5.5.4.2">𝑒</ci><ci id="alg2.l6.m1.5.5.5.5.4.3.cmml" xref="alg2.l6.m1.5.5.5.5.4.3">𝑗</ci></apply></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg2.l6.m1.5c">\mathbf{Q}_{j},\mathbf{dQ}_{j},\mathbf{dO}_{j},D_{j},lse_{j}</annotation></semantics></math><span id="alg2.l6.3" class="ltx_text" style="font-size:80%;"> from communication ring;
</span>
</div>
<div id="alg2.l7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l7.1.1.1" class="ltx_text" style="font-size:80%;">7:</span></span><span id="alg2.l7.2" class="ltx_text" style="font-size:80%;">    {The backward pass of local attention (w/o LAO).}
</span>
</div>
<div id="alg2.l8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l8.1.1.1" class="ltx_text" style="font-size:80%;">8:</span></span><span id="alg2.l8.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg2.l8.m1.2" class="ltx_Math" alttext="\mathbf{S}_{j,i}=\mathbf{Q}_{j}\mathbf{K}_{i}^{T}" display="inline"><semantics id="alg2.l8.m1.2a"><mrow id="alg2.l8.m1.2.3" xref="alg2.l8.m1.2.3.cmml"><msub id="alg2.l8.m1.2.3.2" xref="alg2.l8.m1.2.3.2.cmml"><mi mathsize="80%" id="alg2.l8.m1.2.3.2.2" xref="alg2.l8.m1.2.3.2.2.cmml">𝐒</mi><mrow id="alg2.l8.m1.2.2.2.4" xref="alg2.l8.m1.2.2.2.3.cmml"><mi mathsize="80%" id="alg2.l8.m1.1.1.1.1" xref="alg2.l8.m1.1.1.1.1.cmml">j</mi><mo mathsize="80%" id="alg2.l8.m1.2.2.2.4.1" xref="alg2.l8.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg2.l8.m1.2.2.2.2" xref="alg2.l8.m1.2.2.2.2.cmml">i</mi></mrow></msub><mo mathsize="80%" id="alg2.l8.m1.2.3.1" xref="alg2.l8.m1.2.3.1.cmml">=</mo><mrow id="alg2.l8.m1.2.3.3" xref="alg2.l8.m1.2.3.3.cmml"><msub id="alg2.l8.m1.2.3.3.2" xref="alg2.l8.m1.2.3.3.2.cmml"><mi mathsize="80%" id="alg2.l8.m1.2.3.3.2.2" xref="alg2.l8.m1.2.3.3.2.2.cmml">𝐐</mi><mi mathsize="80%" id="alg2.l8.m1.2.3.3.2.3" xref="alg2.l8.m1.2.3.3.2.3.cmml">j</mi></msub><mo lspace="0em" rspace="0em" id="alg2.l8.m1.2.3.3.1" xref="alg2.l8.m1.2.3.3.1.cmml">​</mo><msubsup id="alg2.l8.m1.2.3.3.3" xref="alg2.l8.m1.2.3.3.3.cmml"><mi mathsize="80%" id="alg2.l8.m1.2.3.3.3.2.2" xref="alg2.l8.m1.2.3.3.3.2.2.cmml">𝐊</mi><mi mathsize="80%" id="alg2.l8.m1.2.3.3.3.2.3" xref="alg2.l8.m1.2.3.3.3.2.3.cmml">i</mi><mi mathsize="80%" id="alg2.l8.m1.2.3.3.3.3" xref="alg2.l8.m1.2.3.3.3.3.cmml">T</mi></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l8.m1.2b"><apply id="alg2.l8.m1.2.3.cmml" xref="alg2.l8.m1.2.3"><eq id="alg2.l8.m1.2.3.1.cmml" xref="alg2.l8.m1.2.3.1"></eq><apply id="alg2.l8.m1.2.3.2.cmml" xref="alg2.l8.m1.2.3.2"><csymbol cd="ambiguous" id="alg2.l8.m1.2.3.2.1.cmml" xref="alg2.l8.m1.2.3.2">subscript</csymbol><ci id="alg2.l8.m1.2.3.2.2.cmml" xref="alg2.l8.m1.2.3.2.2">𝐒</ci><list id="alg2.l8.m1.2.2.2.3.cmml" xref="alg2.l8.m1.2.2.2.4"><ci id="alg2.l8.m1.1.1.1.1.cmml" xref="alg2.l8.m1.1.1.1.1">𝑗</ci><ci id="alg2.l8.m1.2.2.2.2.cmml" xref="alg2.l8.m1.2.2.2.2">𝑖</ci></list></apply><apply id="alg2.l8.m1.2.3.3.cmml" xref="alg2.l8.m1.2.3.3"><times id="alg2.l8.m1.2.3.3.1.cmml" xref="alg2.l8.m1.2.3.3.1"></times><apply id="alg2.l8.m1.2.3.3.2.cmml" xref="alg2.l8.m1.2.3.3.2"><csymbol cd="ambiguous" id="alg2.l8.m1.2.3.3.2.1.cmml" xref="alg2.l8.m1.2.3.3.2">subscript</csymbol><ci id="alg2.l8.m1.2.3.3.2.2.cmml" xref="alg2.l8.m1.2.3.3.2.2">𝐐</ci><ci id="alg2.l8.m1.2.3.3.2.3.cmml" xref="alg2.l8.m1.2.3.3.2.3">𝑗</ci></apply><apply id="alg2.l8.m1.2.3.3.3.cmml" xref="alg2.l8.m1.2.3.3.3"><csymbol cd="ambiguous" id="alg2.l8.m1.2.3.3.3.1.cmml" xref="alg2.l8.m1.2.3.3.3">superscript</csymbol><apply id="alg2.l8.m1.2.3.3.3.2.cmml" xref="alg2.l8.m1.2.3.3.3"><csymbol cd="ambiguous" id="alg2.l8.m1.2.3.3.3.2.1.cmml" xref="alg2.l8.m1.2.3.3.3">subscript</csymbol><ci id="alg2.l8.m1.2.3.3.3.2.2.cmml" xref="alg2.l8.m1.2.3.3.3.2.2">𝐊</ci><ci id="alg2.l8.m1.2.3.3.3.2.3.cmml" xref="alg2.l8.m1.2.3.3.3.2.3">𝑖</ci></apply><ci id="alg2.l8.m1.2.3.3.3.3.cmml" xref="alg2.l8.m1.2.3.3.3.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l8.m1.2c">\mathbf{S}_{j,i}=\mathbf{Q}_{j}\mathbf{K}_{i}^{T}</annotation></semantics></math><span id="alg2.l8.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg2.l9" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l9.1.1.1" class="ltx_text" style="font-size:80%;">9:</span></span><span id="alg2.l9.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg2.l9.m1.5" class="ltx_Math" alttext="\mathbf{P}_{j,i}=\text{exp}(\mathbf{S}_{j,i}-lse_{j})" display="inline"><semantics id="alg2.l9.m1.5a"><mrow id="alg2.l9.m1.5.5" xref="alg2.l9.m1.5.5.cmml"><msub id="alg2.l9.m1.5.5.3" xref="alg2.l9.m1.5.5.3.cmml"><mi mathsize="80%" id="alg2.l9.m1.5.5.3.2" xref="alg2.l9.m1.5.5.3.2.cmml">𝐏</mi><mrow id="alg2.l9.m1.2.2.2.4" xref="alg2.l9.m1.2.2.2.3.cmml"><mi mathsize="80%" id="alg2.l9.m1.1.1.1.1" xref="alg2.l9.m1.1.1.1.1.cmml">j</mi><mo mathsize="80%" id="alg2.l9.m1.2.2.2.4.1" xref="alg2.l9.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg2.l9.m1.2.2.2.2" xref="alg2.l9.m1.2.2.2.2.cmml">i</mi></mrow></msub><mo mathsize="80%" id="alg2.l9.m1.5.5.2" xref="alg2.l9.m1.5.5.2.cmml">=</mo><mrow id="alg2.l9.m1.5.5.1" xref="alg2.l9.m1.5.5.1.cmml"><mtext mathsize="80%" id="alg2.l9.m1.5.5.1.3" xref="alg2.l9.m1.5.5.1.3a.cmml">exp</mtext><mo lspace="0em" rspace="0em" id="alg2.l9.m1.5.5.1.2" xref="alg2.l9.m1.5.5.1.2.cmml">​</mo><mrow id="alg2.l9.m1.5.5.1.1.1" xref="alg2.l9.m1.5.5.1.1.1.1.cmml"><mo maxsize="80%" minsize="80%" id="alg2.l9.m1.5.5.1.1.1.2" xref="alg2.l9.m1.5.5.1.1.1.1.cmml">(</mo><mrow id="alg2.l9.m1.5.5.1.1.1.1" xref="alg2.l9.m1.5.5.1.1.1.1.cmml"><msub id="alg2.l9.m1.5.5.1.1.1.1.2" xref="alg2.l9.m1.5.5.1.1.1.1.2.cmml"><mi mathsize="80%" id="alg2.l9.m1.5.5.1.1.1.1.2.2" xref="alg2.l9.m1.5.5.1.1.1.1.2.2.cmml">𝐒</mi><mrow id="alg2.l9.m1.4.4.2.4" xref="alg2.l9.m1.4.4.2.3.cmml"><mi mathsize="80%" id="alg2.l9.m1.3.3.1.1" xref="alg2.l9.m1.3.3.1.1.cmml">j</mi><mo mathsize="80%" id="alg2.l9.m1.4.4.2.4.1" xref="alg2.l9.m1.4.4.2.3.cmml">,</mo><mi mathsize="80%" id="alg2.l9.m1.4.4.2.2" xref="alg2.l9.m1.4.4.2.2.cmml">i</mi></mrow></msub><mo mathsize="80%" id="alg2.l9.m1.5.5.1.1.1.1.1" xref="alg2.l9.m1.5.5.1.1.1.1.1.cmml">−</mo><mrow id="alg2.l9.m1.5.5.1.1.1.1.3" xref="alg2.l9.m1.5.5.1.1.1.1.3.cmml"><mi mathsize="80%" id="alg2.l9.m1.5.5.1.1.1.1.3.2" xref="alg2.l9.m1.5.5.1.1.1.1.3.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg2.l9.m1.5.5.1.1.1.1.3.1" xref="alg2.l9.m1.5.5.1.1.1.1.3.1.cmml">​</mo><mi mathsize="80%" id="alg2.l9.m1.5.5.1.1.1.1.3.3" xref="alg2.l9.m1.5.5.1.1.1.1.3.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg2.l9.m1.5.5.1.1.1.1.3.1a" xref="alg2.l9.m1.5.5.1.1.1.1.3.1.cmml">​</mo><msub id="alg2.l9.m1.5.5.1.1.1.1.3.4" xref="alg2.l9.m1.5.5.1.1.1.1.3.4.cmml"><mi mathsize="80%" id="alg2.l9.m1.5.5.1.1.1.1.3.4.2" xref="alg2.l9.m1.5.5.1.1.1.1.3.4.2.cmml">e</mi><mi mathsize="80%" id="alg2.l9.m1.5.5.1.1.1.1.3.4.3" xref="alg2.l9.m1.5.5.1.1.1.1.3.4.3.cmml">j</mi></msub></mrow></mrow><mo maxsize="80%" minsize="80%" id="alg2.l9.m1.5.5.1.1.1.3" xref="alg2.l9.m1.5.5.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l9.m1.5b"><apply id="alg2.l9.m1.5.5.cmml" xref="alg2.l9.m1.5.5"><eq id="alg2.l9.m1.5.5.2.cmml" xref="alg2.l9.m1.5.5.2"></eq><apply id="alg2.l9.m1.5.5.3.cmml" xref="alg2.l9.m1.5.5.3"><csymbol cd="ambiguous" id="alg2.l9.m1.5.5.3.1.cmml" xref="alg2.l9.m1.5.5.3">subscript</csymbol><ci id="alg2.l9.m1.5.5.3.2.cmml" xref="alg2.l9.m1.5.5.3.2">𝐏</ci><list id="alg2.l9.m1.2.2.2.3.cmml" xref="alg2.l9.m1.2.2.2.4"><ci id="alg2.l9.m1.1.1.1.1.cmml" xref="alg2.l9.m1.1.1.1.1">𝑗</ci><ci id="alg2.l9.m1.2.2.2.2.cmml" xref="alg2.l9.m1.2.2.2.2">𝑖</ci></list></apply><apply id="alg2.l9.m1.5.5.1.cmml" xref="alg2.l9.m1.5.5.1"><times id="alg2.l9.m1.5.5.1.2.cmml" xref="alg2.l9.m1.5.5.1.2"></times><ci id="alg2.l9.m1.5.5.1.3a.cmml" xref="alg2.l9.m1.5.5.1.3"><mtext mathsize="80%" id="alg2.l9.m1.5.5.1.3.cmml" xref="alg2.l9.m1.5.5.1.3">exp</mtext></ci><apply id="alg2.l9.m1.5.5.1.1.1.1.cmml" xref="alg2.l9.m1.5.5.1.1.1"><minus id="alg2.l9.m1.5.5.1.1.1.1.1.cmml" xref="alg2.l9.m1.5.5.1.1.1.1.1"></minus><apply id="alg2.l9.m1.5.5.1.1.1.1.2.cmml" xref="alg2.l9.m1.5.5.1.1.1.1.2"><csymbol cd="ambiguous" id="alg2.l9.m1.5.5.1.1.1.1.2.1.cmml" xref="alg2.l9.m1.5.5.1.1.1.1.2">subscript</csymbol><ci id="alg2.l9.m1.5.5.1.1.1.1.2.2.cmml" xref="alg2.l9.m1.5.5.1.1.1.1.2.2">𝐒</ci><list id="alg2.l9.m1.4.4.2.3.cmml" xref="alg2.l9.m1.4.4.2.4"><ci id="alg2.l9.m1.3.3.1.1.cmml" xref="alg2.l9.m1.3.3.1.1">𝑗</ci><ci id="alg2.l9.m1.4.4.2.2.cmml" xref="alg2.l9.m1.4.4.2.2">𝑖</ci></list></apply><apply id="alg2.l9.m1.5.5.1.1.1.1.3.cmml" xref="alg2.l9.m1.5.5.1.1.1.1.3"><times id="alg2.l9.m1.5.5.1.1.1.1.3.1.cmml" xref="alg2.l9.m1.5.5.1.1.1.1.3.1"></times><ci id="alg2.l9.m1.5.5.1.1.1.1.3.2.cmml" xref="alg2.l9.m1.5.5.1.1.1.1.3.2">𝑙</ci><ci id="alg2.l9.m1.5.5.1.1.1.1.3.3.cmml" xref="alg2.l9.m1.5.5.1.1.1.1.3.3">𝑠</ci><apply id="alg2.l9.m1.5.5.1.1.1.1.3.4.cmml" xref="alg2.l9.m1.5.5.1.1.1.1.3.4"><csymbol cd="ambiguous" id="alg2.l9.m1.5.5.1.1.1.1.3.4.1.cmml" xref="alg2.l9.m1.5.5.1.1.1.1.3.4">subscript</csymbol><ci id="alg2.l9.m1.5.5.1.1.1.1.3.4.2.cmml" xref="alg2.l9.m1.5.5.1.1.1.1.3.4.2">𝑒</ci><ci id="alg2.l9.m1.5.5.1.1.1.1.3.4.3.cmml" xref="alg2.l9.m1.5.5.1.1.1.1.3.4.3">𝑗</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l9.m1.5c">\mathbf{P}_{j,i}=\text{exp}(\mathbf{S}_{j,i}-lse_{j})</annotation></semantics></math><span id="alg2.l9.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg2.l10" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l10.1.1.1" class="ltx_text" style="font-size:80%;">10:</span></span><span id="alg2.l10.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg2.l10.m1.2" class="ltx_Math" alttext="\mathbf{dV}_{i}=\mathbf{dV}_{i}+\mathbf{P}_{j,i}^{T}\mathbf{dO}_{j}" display="inline"><semantics id="alg2.l10.m1.2a"><mrow id="alg2.l10.m1.2.3" xref="alg2.l10.m1.2.3.cmml"><msub id="alg2.l10.m1.2.3.2" xref="alg2.l10.m1.2.3.2.cmml"><mi mathsize="80%" id="alg2.l10.m1.2.3.2.2" xref="alg2.l10.m1.2.3.2.2.cmml">𝐝𝐕</mi><mi mathsize="80%" id="alg2.l10.m1.2.3.2.3" xref="alg2.l10.m1.2.3.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg2.l10.m1.2.3.1" xref="alg2.l10.m1.2.3.1.cmml">=</mo><mrow id="alg2.l10.m1.2.3.3" xref="alg2.l10.m1.2.3.3.cmml"><msub id="alg2.l10.m1.2.3.3.2" xref="alg2.l10.m1.2.3.3.2.cmml"><mi mathsize="80%" id="alg2.l10.m1.2.3.3.2.2" xref="alg2.l10.m1.2.3.3.2.2.cmml">𝐝𝐕</mi><mi mathsize="80%" id="alg2.l10.m1.2.3.3.2.3" xref="alg2.l10.m1.2.3.3.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg2.l10.m1.2.3.3.1" xref="alg2.l10.m1.2.3.3.1.cmml">+</mo><mrow id="alg2.l10.m1.2.3.3.3" xref="alg2.l10.m1.2.3.3.3.cmml"><msubsup id="alg2.l10.m1.2.3.3.3.2" xref="alg2.l10.m1.2.3.3.3.2.cmml"><mi mathsize="80%" id="alg2.l10.m1.2.3.3.3.2.2.2" xref="alg2.l10.m1.2.3.3.3.2.2.2.cmml">𝐏</mi><mrow id="alg2.l10.m1.2.2.2.4" xref="alg2.l10.m1.2.2.2.3.cmml"><mi mathsize="80%" id="alg2.l10.m1.1.1.1.1" xref="alg2.l10.m1.1.1.1.1.cmml">j</mi><mo mathsize="80%" id="alg2.l10.m1.2.2.2.4.1" xref="alg2.l10.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg2.l10.m1.2.2.2.2" xref="alg2.l10.m1.2.2.2.2.cmml">i</mi></mrow><mi mathsize="80%" id="alg2.l10.m1.2.3.3.3.2.3" xref="alg2.l10.m1.2.3.3.3.2.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="alg2.l10.m1.2.3.3.3.1" xref="alg2.l10.m1.2.3.3.3.1.cmml">​</mo><msub id="alg2.l10.m1.2.3.3.3.3" xref="alg2.l10.m1.2.3.3.3.3.cmml"><mi mathsize="80%" id="alg2.l10.m1.2.3.3.3.3.2" xref="alg2.l10.m1.2.3.3.3.3.2.cmml">𝐝𝐎</mi><mi mathsize="80%" id="alg2.l10.m1.2.3.3.3.3.3" xref="alg2.l10.m1.2.3.3.3.3.3.cmml">j</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l10.m1.2b"><apply id="alg2.l10.m1.2.3.cmml" xref="alg2.l10.m1.2.3"><eq id="alg2.l10.m1.2.3.1.cmml" xref="alg2.l10.m1.2.3.1"></eq><apply id="alg2.l10.m1.2.3.2.cmml" xref="alg2.l10.m1.2.3.2"><csymbol cd="ambiguous" id="alg2.l10.m1.2.3.2.1.cmml" xref="alg2.l10.m1.2.3.2">subscript</csymbol><ci id="alg2.l10.m1.2.3.2.2.cmml" xref="alg2.l10.m1.2.3.2.2">𝐝𝐕</ci><ci id="alg2.l10.m1.2.3.2.3.cmml" xref="alg2.l10.m1.2.3.2.3">𝑖</ci></apply><apply id="alg2.l10.m1.2.3.3.cmml" xref="alg2.l10.m1.2.3.3"><plus id="alg2.l10.m1.2.3.3.1.cmml" xref="alg2.l10.m1.2.3.3.1"></plus><apply id="alg2.l10.m1.2.3.3.2.cmml" xref="alg2.l10.m1.2.3.3.2"><csymbol cd="ambiguous" id="alg2.l10.m1.2.3.3.2.1.cmml" xref="alg2.l10.m1.2.3.3.2">subscript</csymbol><ci id="alg2.l10.m1.2.3.3.2.2.cmml" xref="alg2.l10.m1.2.3.3.2.2">𝐝𝐕</ci><ci id="alg2.l10.m1.2.3.3.2.3.cmml" xref="alg2.l10.m1.2.3.3.2.3">𝑖</ci></apply><apply id="alg2.l10.m1.2.3.3.3.cmml" xref="alg2.l10.m1.2.3.3.3"><times id="alg2.l10.m1.2.3.3.3.1.cmml" xref="alg2.l10.m1.2.3.3.3.1"></times><apply id="alg2.l10.m1.2.3.3.3.2.cmml" xref="alg2.l10.m1.2.3.3.3.2"><csymbol cd="ambiguous" id="alg2.l10.m1.2.3.3.3.2.1.cmml" xref="alg2.l10.m1.2.3.3.3.2">superscript</csymbol><apply id="alg2.l10.m1.2.3.3.3.2.2.cmml" xref="alg2.l10.m1.2.3.3.3.2"><csymbol cd="ambiguous" id="alg2.l10.m1.2.3.3.3.2.2.1.cmml" xref="alg2.l10.m1.2.3.3.3.2">subscript</csymbol><ci id="alg2.l10.m1.2.3.3.3.2.2.2.cmml" xref="alg2.l10.m1.2.3.3.3.2.2.2">𝐏</ci><list id="alg2.l10.m1.2.2.2.3.cmml" xref="alg2.l10.m1.2.2.2.4"><ci id="alg2.l10.m1.1.1.1.1.cmml" xref="alg2.l10.m1.1.1.1.1">𝑗</ci><ci id="alg2.l10.m1.2.2.2.2.cmml" xref="alg2.l10.m1.2.2.2.2">𝑖</ci></list></apply><ci id="alg2.l10.m1.2.3.3.3.2.3.cmml" xref="alg2.l10.m1.2.3.3.3.2.3">𝑇</ci></apply><apply id="alg2.l10.m1.2.3.3.3.3.cmml" xref="alg2.l10.m1.2.3.3.3.3"><csymbol cd="ambiguous" id="alg2.l10.m1.2.3.3.3.3.1.cmml" xref="alg2.l10.m1.2.3.3.3.3">subscript</csymbol><ci id="alg2.l10.m1.2.3.3.3.3.2.cmml" xref="alg2.l10.m1.2.3.3.3.3.2">𝐝𝐎</ci><ci id="alg2.l10.m1.2.3.3.3.3.3.cmml" xref="alg2.l10.m1.2.3.3.3.3.3">𝑗</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l10.m1.2c">\mathbf{dV}_{i}=\mathbf{dV}_{i}+\mathbf{P}_{j,i}^{T}\mathbf{dO}_{j}</annotation></semantics></math><span id="alg2.l10.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg2.l11" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l11.1.1.1" class="ltx_text" style="font-size:80%;">11:</span></span><span id="alg2.l11.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg2.l11.m1.2" class="ltx_Math" alttext="\mathbf{dP}_{j,i}=\mathbf{dO}_{j}~{}\mathbf{V}_{i}^{T}" display="inline"><semantics id="alg2.l11.m1.2a"><mrow id="alg2.l11.m1.2.3" xref="alg2.l11.m1.2.3.cmml"><msub id="alg2.l11.m1.2.3.2" xref="alg2.l11.m1.2.3.2.cmml"><mi mathsize="80%" id="alg2.l11.m1.2.3.2.2" xref="alg2.l11.m1.2.3.2.2.cmml">𝐝𝐏</mi><mrow id="alg2.l11.m1.2.2.2.4" xref="alg2.l11.m1.2.2.2.3.cmml"><mi mathsize="80%" id="alg2.l11.m1.1.1.1.1" xref="alg2.l11.m1.1.1.1.1.cmml">j</mi><mo mathsize="80%" id="alg2.l11.m1.2.2.2.4.1" xref="alg2.l11.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg2.l11.m1.2.2.2.2" xref="alg2.l11.m1.2.2.2.2.cmml">i</mi></mrow></msub><mo mathsize="80%" id="alg2.l11.m1.2.3.1" xref="alg2.l11.m1.2.3.1.cmml">=</mo><mrow id="alg2.l11.m1.2.3.3" xref="alg2.l11.m1.2.3.3.cmml"><msub id="alg2.l11.m1.2.3.3.2" xref="alg2.l11.m1.2.3.3.2.cmml"><mi mathsize="80%" id="alg2.l11.m1.2.3.3.2.2" xref="alg2.l11.m1.2.3.3.2.2.cmml">𝐝𝐎</mi><mi mathsize="80%" id="alg2.l11.m1.2.3.3.2.3" xref="alg2.l11.m1.2.3.3.2.3.cmml">j</mi></msub><mo lspace="0em" rspace="0em" id="alg2.l11.m1.2.3.3.1" xref="alg2.l11.m1.2.3.3.1.cmml">​</mo><msubsup id="alg2.l11.m1.2.3.3.3" xref="alg2.l11.m1.2.3.3.3.cmml"><mi mathsize="80%" id="alg2.l11.m1.2.3.3.3.2.2" xref="alg2.l11.m1.2.3.3.3.2.2.cmml">𝐕</mi><mi mathsize="80%" id="alg2.l11.m1.2.3.3.3.2.3" xref="alg2.l11.m1.2.3.3.3.2.3.cmml">i</mi><mi mathsize="80%" id="alg2.l11.m1.2.3.3.3.3" xref="alg2.l11.m1.2.3.3.3.3.cmml">T</mi></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l11.m1.2b"><apply id="alg2.l11.m1.2.3.cmml" xref="alg2.l11.m1.2.3"><eq id="alg2.l11.m1.2.3.1.cmml" xref="alg2.l11.m1.2.3.1"></eq><apply id="alg2.l11.m1.2.3.2.cmml" xref="alg2.l11.m1.2.3.2"><csymbol cd="ambiguous" id="alg2.l11.m1.2.3.2.1.cmml" xref="alg2.l11.m1.2.3.2">subscript</csymbol><ci id="alg2.l11.m1.2.3.2.2.cmml" xref="alg2.l11.m1.2.3.2.2">𝐝𝐏</ci><list id="alg2.l11.m1.2.2.2.3.cmml" xref="alg2.l11.m1.2.2.2.4"><ci id="alg2.l11.m1.1.1.1.1.cmml" xref="alg2.l11.m1.1.1.1.1">𝑗</ci><ci id="alg2.l11.m1.2.2.2.2.cmml" xref="alg2.l11.m1.2.2.2.2">𝑖</ci></list></apply><apply id="alg2.l11.m1.2.3.3.cmml" xref="alg2.l11.m1.2.3.3"><times id="alg2.l11.m1.2.3.3.1.cmml" xref="alg2.l11.m1.2.3.3.1"></times><apply id="alg2.l11.m1.2.3.3.2.cmml" xref="alg2.l11.m1.2.3.3.2"><csymbol cd="ambiguous" id="alg2.l11.m1.2.3.3.2.1.cmml" xref="alg2.l11.m1.2.3.3.2">subscript</csymbol><ci id="alg2.l11.m1.2.3.3.2.2.cmml" xref="alg2.l11.m1.2.3.3.2.2">𝐝𝐎</ci><ci id="alg2.l11.m1.2.3.3.2.3.cmml" xref="alg2.l11.m1.2.3.3.2.3">𝑗</ci></apply><apply id="alg2.l11.m1.2.3.3.3.cmml" xref="alg2.l11.m1.2.3.3.3"><csymbol cd="ambiguous" id="alg2.l11.m1.2.3.3.3.1.cmml" xref="alg2.l11.m1.2.3.3.3">superscript</csymbol><apply id="alg2.l11.m1.2.3.3.3.2.cmml" xref="alg2.l11.m1.2.3.3.3"><csymbol cd="ambiguous" id="alg2.l11.m1.2.3.3.3.2.1.cmml" xref="alg2.l11.m1.2.3.3.3">subscript</csymbol><ci id="alg2.l11.m1.2.3.3.3.2.2.cmml" xref="alg2.l11.m1.2.3.3.3.2.2">𝐕</ci><ci id="alg2.l11.m1.2.3.3.3.2.3.cmml" xref="alg2.l11.m1.2.3.3.3.2.3">𝑖</ci></apply><ci id="alg2.l11.m1.2.3.3.3.3.cmml" xref="alg2.l11.m1.2.3.3.3.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l11.m1.2c">\mathbf{dP}_{j,i}=\mathbf{dO}_{j}~{}\mathbf{V}_{i}^{T}</annotation></semantics></math><span id="alg2.l11.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg2.l12" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l12.1.1.1" class="ltx_text" style="font-size:80%;">12:</span></span><span id="alg2.l12.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg2.l12.m1.7" class="ltx_Math" alttext="\mathbf{dS}_{j,i}=\mathbf{P}_{j,i}\circ(\mathbf{dP}_{j,i}-D_{j})" display="inline"><semantics id="alg2.l12.m1.7a"><mrow id="alg2.l12.m1.7.7" xref="alg2.l12.m1.7.7.cmml"><msub id="alg2.l12.m1.7.7.3" xref="alg2.l12.m1.7.7.3.cmml"><mi mathsize="80%" id="alg2.l12.m1.7.7.3.2" xref="alg2.l12.m1.7.7.3.2.cmml">𝐝𝐒</mi><mrow id="alg2.l12.m1.2.2.2.4" xref="alg2.l12.m1.2.2.2.3.cmml"><mi mathsize="80%" id="alg2.l12.m1.1.1.1.1" xref="alg2.l12.m1.1.1.1.1.cmml">j</mi><mo mathsize="80%" id="alg2.l12.m1.2.2.2.4.1" xref="alg2.l12.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg2.l12.m1.2.2.2.2" xref="alg2.l12.m1.2.2.2.2.cmml">i</mi></mrow></msub><mo mathsize="80%" id="alg2.l12.m1.7.7.2" xref="alg2.l12.m1.7.7.2.cmml">=</mo><mrow id="alg2.l12.m1.7.7.1" xref="alg2.l12.m1.7.7.1.cmml"><msub id="alg2.l12.m1.7.7.1.3" xref="alg2.l12.m1.7.7.1.3.cmml"><mi mathsize="80%" id="alg2.l12.m1.7.7.1.3.2" xref="alg2.l12.m1.7.7.1.3.2.cmml">𝐏</mi><mrow id="alg2.l12.m1.4.4.2.4" xref="alg2.l12.m1.4.4.2.3.cmml"><mi mathsize="80%" id="alg2.l12.m1.3.3.1.1" xref="alg2.l12.m1.3.3.1.1.cmml">j</mi><mo mathsize="80%" id="alg2.l12.m1.4.4.2.4.1" xref="alg2.l12.m1.4.4.2.3.cmml">,</mo><mi mathsize="80%" id="alg2.l12.m1.4.4.2.2" xref="alg2.l12.m1.4.4.2.2.cmml">i</mi></mrow></msub><mo lspace="0.222em" mathsize="80%" rspace="0.222em" id="alg2.l12.m1.7.7.1.2" xref="alg2.l12.m1.7.7.1.2.cmml">∘</mo><mrow id="alg2.l12.m1.7.7.1.1.1" xref="alg2.l12.m1.7.7.1.1.1.1.cmml"><mo maxsize="80%" minsize="80%" id="alg2.l12.m1.7.7.1.1.1.2" xref="alg2.l12.m1.7.7.1.1.1.1.cmml">(</mo><mrow id="alg2.l12.m1.7.7.1.1.1.1" xref="alg2.l12.m1.7.7.1.1.1.1.cmml"><msub id="alg2.l12.m1.7.7.1.1.1.1.2" xref="alg2.l12.m1.7.7.1.1.1.1.2.cmml"><mi mathsize="80%" id="alg2.l12.m1.7.7.1.1.1.1.2.2" xref="alg2.l12.m1.7.7.1.1.1.1.2.2.cmml">𝐝𝐏</mi><mrow id="alg2.l12.m1.6.6.2.4" xref="alg2.l12.m1.6.6.2.3.cmml"><mi mathsize="80%" id="alg2.l12.m1.5.5.1.1" xref="alg2.l12.m1.5.5.1.1.cmml">j</mi><mo mathsize="80%" id="alg2.l12.m1.6.6.2.4.1" xref="alg2.l12.m1.6.6.2.3.cmml">,</mo><mi mathsize="80%" id="alg2.l12.m1.6.6.2.2" xref="alg2.l12.m1.6.6.2.2.cmml">i</mi></mrow></msub><mo mathsize="80%" id="alg2.l12.m1.7.7.1.1.1.1.1" xref="alg2.l12.m1.7.7.1.1.1.1.1.cmml">−</mo><msub id="alg2.l12.m1.7.7.1.1.1.1.3" xref="alg2.l12.m1.7.7.1.1.1.1.3.cmml"><mi mathsize="80%" id="alg2.l12.m1.7.7.1.1.1.1.3.2" xref="alg2.l12.m1.7.7.1.1.1.1.3.2.cmml">D</mi><mi mathsize="80%" id="alg2.l12.m1.7.7.1.1.1.1.3.3" xref="alg2.l12.m1.7.7.1.1.1.1.3.3.cmml">j</mi></msub></mrow><mo maxsize="80%" minsize="80%" id="alg2.l12.m1.7.7.1.1.1.3" xref="alg2.l12.m1.7.7.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l12.m1.7b"><apply id="alg2.l12.m1.7.7.cmml" xref="alg2.l12.m1.7.7"><eq id="alg2.l12.m1.7.7.2.cmml" xref="alg2.l12.m1.7.7.2"></eq><apply id="alg2.l12.m1.7.7.3.cmml" xref="alg2.l12.m1.7.7.3"><csymbol cd="ambiguous" id="alg2.l12.m1.7.7.3.1.cmml" xref="alg2.l12.m1.7.7.3">subscript</csymbol><ci id="alg2.l12.m1.7.7.3.2.cmml" xref="alg2.l12.m1.7.7.3.2">𝐝𝐒</ci><list id="alg2.l12.m1.2.2.2.3.cmml" xref="alg2.l12.m1.2.2.2.4"><ci id="alg2.l12.m1.1.1.1.1.cmml" xref="alg2.l12.m1.1.1.1.1">𝑗</ci><ci id="alg2.l12.m1.2.2.2.2.cmml" xref="alg2.l12.m1.2.2.2.2">𝑖</ci></list></apply><apply id="alg2.l12.m1.7.7.1.cmml" xref="alg2.l12.m1.7.7.1"><compose id="alg2.l12.m1.7.7.1.2.cmml" xref="alg2.l12.m1.7.7.1.2"></compose><apply id="alg2.l12.m1.7.7.1.3.cmml" xref="alg2.l12.m1.7.7.1.3"><csymbol cd="ambiguous" id="alg2.l12.m1.7.7.1.3.1.cmml" xref="alg2.l12.m1.7.7.1.3">subscript</csymbol><ci id="alg2.l12.m1.7.7.1.3.2.cmml" xref="alg2.l12.m1.7.7.1.3.2">𝐏</ci><list id="alg2.l12.m1.4.4.2.3.cmml" xref="alg2.l12.m1.4.4.2.4"><ci id="alg2.l12.m1.3.3.1.1.cmml" xref="alg2.l12.m1.3.3.1.1">𝑗</ci><ci id="alg2.l12.m1.4.4.2.2.cmml" xref="alg2.l12.m1.4.4.2.2">𝑖</ci></list></apply><apply id="alg2.l12.m1.7.7.1.1.1.1.cmml" xref="alg2.l12.m1.7.7.1.1.1"><minus id="alg2.l12.m1.7.7.1.1.1.1.1.cmml" xref="alg2.l12.m1.7.7.1.1.1.1.1"></minus><apply id="alg2.l12.m1.7.7.1.1.1.1.2.cmml" xref="alg2.l12.m1.7.7.1.1.1.1.2"><csymbol cd="ambiguous" id="alg2.l12.m1.7.7.1.1.1.1.2.1.cmml" xref="alg2.l12.m1.7.7.1.1.1.1.2">subscript</csymbol><ci id="alg2.l12.m1.7.7.1.1.1.1.2.2.cmml" xref="alg2.l12.m1.7.7.1.1.1.1.2.2">𝐝𝐏</ci><list id="alg2.l12.m1.6.6.2.3.cmml" xref="alg2.l12.m1.6.6.2.4"><ci id="alg2.l12.m1.5.5.1.1.cmml" xref="alg2.l12.m1.5.5.1.1">𝑗</ci><ci id="alg2.l12.m1.6.6.2.2.cmml" xref="alg2.l12.m1.6.6.2.2">𝑖</ci></list></apply><apply id="alg2.l12.m1.7.7.1.1.1.1.3.cmml" xref="alg2.l12.m1.7.7.1.1.1.1.3"><csymbol cd="ambiguous" id="alg2.l12.m1.7.7.1.1.1.1.3.1.cmml" xref="alg2.l12.m1.7.7.1.1.1.1.3">subscript</csymbol><ci id="alg2.l12.m1.7.7.1.1.1.1.3.2.cmml" xref="alg2.l12.m1.7.7.1.1.1.1.3.2">𝐷</ci><ci id="alg2.l12.m1.7.7.1.1.1.1.3.3.cmml" xref="alg2.l12.m1.7.7.1.1.1.1.3.3">𝑗</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l12.m1.7c">\mathbf{dS}_{j,i}=\mathbf{P}_{j,i}\circ(\mathbf{dP}_{j,i}-D_{j})</annotation></semantics></math><span id="alg2.l12.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg2.l13" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l13.1.1.1" class="ltx_text" style="font-size:80%;">13:</span></span><span id="alg2.l13.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg2.l13.m1.2" class="ltx_Math" alttext="\mathbf{dK}_{i}=\mathbf{dK}_{i}+\mathbf{dS}_{j,i}^{T}\mathbf{Q}_{j}" display="inline"><semantics id="alg2.l13.m1.2a"><mrow id="alg2.l13.m1.2.3" xref="alg2.l13.m1.2.3.cmml"><msub id="alg2.l13.m1.2.3.2" xref="alg2.l13.m1.2.3.2.cmml"><mi mathsize="80%" id="alg2.l13.m1.2.3.2.2" xref="alg2.l13.m1.2.3.2.2.cmml">𝐝𝐊</mi><mi mathsize="80%" id="alg2.l13.m1.2.3.2.3" xref="alg2.l13.m1.2.3.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg2.l13.m1.2.3.1" xref="alg2.l13.m1.2.3.1.cmml">=</mo><mrow id="alg2.l13.m1.2.3.3" xref="alg2.l13.m1.2.3.3.cmml"><msub id="alg2.l13.m1.2.3.3.2" xref="alg2.l13.m1.2.3.3.2.cmml"><mi mathsize="80%" id="alg2.l13.m1.2.3.3.2.2" xref="alg2.l13.m1.2.3.3.2.2.cmml">𝐝𝐊</mi><mi mathsize="80%" id="alg2.l13.m1.2.3.3.2.3" xref="alg2.l13.m1.2.3.3.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg2.l13.m1.2.3.3.1" xref="alg2.l13.m1.2.3.3.1.cmml">+</mo><mrow id="alg2.l13.m1.2.3.3.3" xref="alg2.l13.m1.2.3.3.3.cmml"><msubsup id="alg2.l13.m1.2.3.3.3.2" xref="alg2.l13.m1.2.3.3.3.2.cmml"><mi mathsize="80%" id="alg2.l13.m1.2.3.3.3.2.2.2" xref="alg2.l13.m1.2.3.3.3.2.2.2.cmml">𝐝𝐒</mi><mrow id="alg2.l13.m1.2.2.2.4" xref="alg2.l13.m1.2.2.2.3.cmml"><mi mathsize="80%" id="alg2.l13.m1.1.1.1.1" xref="alg2.l13.m1.1.1.1.1.cmml">j</mi><mo mathsize="80%" id="alg2.l13.m1.2.2.2.4.1" xref="alg2.l13.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg2.l13.m1.2.2.2.2" xref="alg2.l13.m1.2.2.2.2.cmml">i</mi></mrow><mi mathsize="80%" id="alg2.l13.m1.2.3.3.3.2.3" xref="alg2.l13.m1.2.3.3.3.2.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="alg2.l13.m1.2.3.3.3.1" xref="alg2.l13.m1.2.3.3.3.1.cmml">​</mo><msub id="alg2.l13.m1.2.3.3.3.3" xref="alg2.l13.m1.2.3.3.3.3.cmml"><mi mathsize="80%" id="alg2.l13.m1.2.3.3.3.3.2" xref="alg2.l13.m1.2.3.3.3.3.2.cmml">𝐐</mi><mi mathsize="80%" id="alg2.l13.m1.2.3.3.3.3.3" xref="alg2.l13.m1.2.3.3.3.3.3.cmml">j</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l13.m1.2b"><apply id="alg2.l13.m1.2.3.cmml" xref="alg2.l13.m1.2.3"><eq id="alg2.l13.m1.2.3.1.cmml" xref="alg2.l13.m1.2.3.1"></eq><apply id="alg2.l13.m1.2.3.2.cmml" xref="alg2.l13.m1.2.3.2"><csymbol cd="ambiguous" id="alg2.l13.m1.2.3.2.1.cmml" xref="alg2.l13.m1.2.3.2">subscript</csymbol><ci id="alg2.l13.m1.2.3.2.2.cmml" xref="alg2.l13.m1.2.3.2.2">𝐝𝐊</ci><ci id="alg2.l13.m1.2.3.2.3.cmml" xref="alg2.l13.m1.2.3.2.3">𝑖</ci></apply><apply id="alg2.l13.m1.2.3.3.cmml" xref="alg2.l13.m1.2.3.3"><plus id="alg2.l13.m1.2.3.3.1.cmml" xref="alg2.l13.m1.2.3.3.1"></plus><apply id="alg2.l13.m1.2.3.3.2.cmml" xref="alg2.l13.m1.2.3.3.2"><csymbol cd="ambiguous" id="alg2.l13.m1.2.3.3.2.1.cmml" xref="alg2.l13.m1.2.3.3.2">subscript</csymbol><ci id="alg2.l13.m1.2.3.3.2.2.cmml" xref="alg2.l13.m1.2.3.3.2.2">𝐝𝐊</ci><ci id="alg2.l13.m1.2.3.3.2.3.cmml" xref="alg2.l13.m1.2.3.3.2.3">𝑖</ci></apply><apply id="alg2.l13.m1.2.3.3.3.cmml" xref="alg2.l13.m1.2.3.3.3"><times id="alg2.l13.m1.2.3.3.3.1.cmml" xref="alg2.l13.m1.2.3.3.3.1"></times><apply id="alg2.l13.m1.2.3.3.3.2.cmml" xref="alg2.l13.m1.2.3.3.3.2"><csymbol cd="ambiguous" id="alg2.l13.m1.2.3.3.3.2.1.cmml" xref="alg2.l13.m1.2.3.3.3.2">superscript</csymbol><apply id="alg2.l13.m1.2.3.3.3.2.2.cmml" xref="alg2.l13.m1.2.3.3.3.2"><csymbol cd="ambiguous" id="alg2.l13.m1.2.3.3.3.2.2.1.cmml" xref="alg2.l13.m1.2.3.3.3.2">subscript</csymbol><ci id="alg2.l13.m1.2.3.3.3.2.2.2.cmml" xref="alg2.l13.m1.2.3.3.3.2.2.2">𝐝𝐒</ci><list id="alg2.l13.m1.2.2.2.3.cmml" xref="alg2.l13.m1.2.2.2.4"><ci id="alg2.l13.m1.1.1.1.1.cmml" xref="alg2.l13.m1.1.1.1.1">𝑗</ci><ci id="alg2.l13.m1.2.2.2.2.cmml" xref="alg2.l13.m1.2.2.2.2">𝑖</ci></list></apply><ci id="alg2.l13.m1.2.3.3.3.2.3.cmml" xref="alg2.l13.m1.2.3.3.3.2.3">𝑇</ci></apply><apply id="alg2.l13.m1.2.3.3.3.3.cmml" xref="alg2.l13.m1.2.3.3.3.3"><csymbol cd="ambiguous" id="alg2.l13.m1.2.3.3.3.3.1.cmml" xref="alg2.l13.m1.2.3.3.3.3">subscript</csymbol><ci id="alg2.l13.m1.2.3.3.3.3.2.cmml" xref="alg2.l13.m1.2.3.3.3.3.2">𝐐</ci><ci id="alg2.l13.m1.2.3.3.3.3.3.cmml" xref="alg2.l13.m1.2.3.3.3.3.3">𝑗</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l13.m1.2c">\mathbf{dK}_{i}=\mathbf{dK}_{i}+\mathbf{dS}_{j,i}^{T}\mathbf{Q}_{j}</annotation></semantics></math><span id="alg2.l13.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg2.l14" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l14.1.1.1" class="ltx_text" style="font-size:80%;">14:</span></span><span id="alg2.l14.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg2.l14.m1.2" class="ltx_Math" alttext="\mathbf{dQ}_{j}=\mathbf{dQ}_{j}+\mathbf{dS}_{j,i}~{}\mathbf{K}_{i}" display="inline"><semantics id="alg2.l14.m1.2a"><mrow id="alg2.l14.m1.2.3" xref="alg2.l14.m1.2.3.cmml"><msub id="alg2.l14.m1.2.3.2" xref="alg2.l14.m1.2.3.2.cmml"><mi mathsize="80%" id="alg2.l14.m1.2.3.2.2" xref="alg2.l14.m1.2.3.2.2.cmml">𝐝𝐐</mi><mi mathsize="80%" id="alg2.l14.m1.2.3.2.3" xref="alg2.l14.m1.2.3.2.3.cmml">j</mi></msub><mo mathsize="80%" id="alg2.l14.m1.2.3.1" xref="alg2.l14.m1.2.3.1.cmml">=</mo><mrow id="alg2.l14.m1.2.3.3" xref="alg2.l14.m1.2.3.3.cmml"><msub id="alg2.l14.m1.2.3.3.2" xref="alg2.l14.m1.2.3.3.2.cmml"><mi mathsize="80%" id="alg2.l14.m1.2.3.3.2.2" xref="alg2.l14.m1.2.3.3.2.2.cmml">𝐝𝐐</mi><mi mathsize="80%" id="alg2.l14.m1.2.3.3.2.3" xref="alg2.l14.m1.2.3.3.2.3.cmml">j</mi></msub><mo mathsize="80%" id="alg2.l14.m1.2.3.3.1" xref="alg2.l14.m1.2.3.3.1.cmml">+</mo><mrow id="alg2.l14.m1.2.3.3.3" xref="alg2.l14.m1.2.3.3.3.cmml"><msub id="alg2.l14.m1.2.3.3.3.2" xref="alg2.l14.m1.2.3.3.3.2.cmml"><mi mathsize="80%" id="alg2.l14.m1.2.3.3.3.2.2" xref="alg2.l14.m1.2.3.3.3.2.2.cmml">𝐝𝐒</mi><mrow id="alg2.l14.m1.2.2.2.4" xref="alg2.l14.m1.2.2.2.3.cmml"><mi mathsize="80%" id="alg2.l14.m1.1.1.1.1" xref="alg2.l14.m1.1.1.1.1.cmml">j</mi><mo mathsize="80%" id="alg2.l14.m1.2.2.2.4.1" xref="alg2.l14.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg2.l14.m1.2.2.2.2" xref="alg2.l14.m1.2.2.2.2.cmml">i</mi></mrow></msub><mo lspace="0em" rspace="0em" id="alg2.l14.m1.2.3.3.3.1" xref="alg2.l14.m1.2.3.3.3.1.cmml">​</mo><msub id="alg2.l14.m1.2.3.3.3.3" xref="alg2.l14.m1.2.3.3.3.3.cmml"><mi mathsize="80%" id="alg2.l14.m1.2.3.3.3.3.2" xref="alg2.l14.m1.2.3.3.3.3.2.cmml">𝐊</mi><mi mathsize="80%" id="alg2.l14.m1.2.3.3.3.3.3" xref="alg2.l14.m1.2.3.3.3.3.3.cmml">i</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l14.m1.2b"><apply id="alg2.l14.m1.2.3.cmml" xref="alg2.l14.m1.2.3"><eq id="alg2.l14.m1.2.3.1.cmml" xref="alg2.l14.m1.2.3.1"></eq><apply id="alg2.l14.m1.2.3.2.cmml" xref="alg2.l14.m1.2.3.2"><csymbol cd="ambiguous" id="alg2.l14.m1.2.3.2.1.cmml" xref="alg2.l14.m1.2.3.2">subscript</csymbol><ci id="alg2.l14.m1.2.3.2.2.cmml" xref="alg2.l14.m1.2.3.2.2">𝐝𝐐</ci><ci id="alg2.l14.m1.2.3.2.3.cmml" xref="alg2.l14.m1.2.3.2.3">𝑗</ci></apply><apply id="alg2.l14.m1.2.3.3.cmml" xref="alg2.l14.m1.2.3.3"><plus id="alg2.l14.m1.2.3.3.1.cmml" xref="alg2.l14.m1.2.3.3.1"></plus><apply id="alg2.l14.m1.2.3.3.2.cmml" xref="alg2.l14.m1.2.3.3.2"><csymbol cd="ambiguous" id="alg2.l14.m1.2.3.3.2.1.cmml" xref="alg2.l14.m1.2.3.3.2">subscript</csymbol><ci id="alg2.l14.m1.2.3.3.2.2.cmml" xref="alg2.l14.m1.2.3.3.2.2">𝐝𝐐</ci><ci id="alg2.l14.m1.2.3.3.2.3.cmml" xref="alg2.l14.m1.2.3.3.2.3">𝑗</ci></apply><apply id="alg2.l14.m1.2.3.3.3.cmml" xref="alg2.l14.m1.2.3.3.3"><times id="alg2.l14.m1.2.3.3.3.1.cmml" xref="alg2.l14.m1.2.3.3.3.1"></times><apply id="alg2.l14.m1.2.3.3.3.2.cmml" xref="alg2.l14.m1.2.3.3.3.2"><csymbol cd="ambiguous" id="alg2.l14.m1.2.3.3.3.2.1.cmml" xref="alg2.l14.m1.2.3.3.3.2">subscript</csymbol><ci id="alg2.l14.m1.2.3.3.3.2.2.cmml" xref="alg2.l14.m1.2.3.3.3.2.2">𝐝𝐒</ci><list id="alg2.l14.m1.2.2.2.3.cmml" xref="alg2.l14.m1.2.2.2.4"><ci id="alg2.l14.m1.1.1.1.1.cmml" xref="alg2.l14.m1.1.1.1.1">𝑗</ci><ci id="alg2.l14.m1.2.2.2.2.cmml" xref="alg2.l14.m1.2.2.2.2">𝑖</ci></list></apply><apply id="alg2.l14.m1.2.3.3.3.3.cmml" xref="alg2.l14.m1.2.3.3.3.3"><csymbol cd="ambiguous" id="alg2.l14.m1.2.3.3.3.3.1.cmml" xref="alg2.l14.m1.2.3.3.3.3">subscript</csymbol><ci id="alg2.l14.m1.2.3.3.3.3.2.cmml" xref="alg2.l14.m1.2.3.3.3.3.2">𝐊</ci><ci id="alg2.l14.m1.2.3.3.3.3.3.cmml" xref="alg2.l14.m1.2.3.3.3.3.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l14.m1.2c">\mathbf{dQ}_{j}=\mathbf{dQ}_{j}+\mathbf{dS}_{j,i}~{}\mathbf{K}_{i}</annotation></semantics></math><span id="alg2.l14.3" class="ltx_text" style="font-size:80%;"> ;
</span>
</div>
<div id="alg2.l15" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l15.1.1.1" class="ltx_text" style="font-size:80%;">15:</span></span><span id="alg2.l15.2" class="ltx_text" style="font-size:80%;">    {The end of the backward pass of local attention.}
</span>
</div>
<div id="alg2.l16" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l16.1.1.1" class="ltx_text" style="font-size:80%;">16:</span></span><span id="alg2.l16.2" class="ltx_text" style="font-size:80%;">    Put </span><math id="alg2.l16.m1.5" class="ltx_Math" alttext="\mathbf{Q}_{j},\mathbf{dQ}_{j},\mathbf{dO}_{j},D_{j},lse_{j}" display="inline"><semantics id="alg2.l16.m1.5a"><mrow id="alg2.l16.m1.5.5.5" xref="alg2.l16.m1.5.5.6.cmml"><msub id="alg2.l16.m1.1.1.1.1" xref="alg2.l16.m1.1.1.1.1.cmml"><mi mathsize="80%" id="alg2.l16.m1.1.1.1.1.2" xref="alg2.l16.m1.1.1.1.1.2.cmml">𝐐</mi><mi mathsize="80%" id="alg2.l16.m1.1.1.1.1.3" xref="alg2.l16.m1.1.1.1.1.3.cmml">j</mi></msub><mo mathsize="80%" id="alg2.l16.m1.5.5.5.6" xref="alg2.l16.m1.5.5.6.cmml">,</mo><msub id="alg2.l16.m1.2.2.2.2" xref="alg2.l16.m1.2.2.2.2.cmml"><mi mathsize="80%" id="alg2.l16.m1.2.2.2.2.2" xref="alg2.l16.m1.2.2.2.2.2.cmml">𝐝𝐐</mi><mi mathsize="80%" id="alg2.l16.m1.2.2.2.2.3" xref="alg2.l16.m1.2.2.2.2.3.cmml">j</mi></msub><mo mathsize="80%" id="alg2.l16.m1.5.5.5.7" xref="alg2.l16.m1.5.5.6.cmml">,</mo><msub id="alg2.l16.m1.3.3.3.3" xref="alg2.l16.m1.3.3.3.3.cmml"><mi mathsize="80%" id="alg2.l16.m1.3.3.3.3.2" xref="alg2.l16.m1.3.3.3.3.2.cmml">𝐝𝐎</mi><mi mathsize="80%" id="alg2.l16.m1.3.3.3.3.3" xref="alg2.l16.m1.3.3.3.3.3.cmml">j</mi></msub><mo mathsize="80%" id="alg2.l16.m1.5.5.5.8" xref="alg2.l16.m1.5.5.6.cmml">,</mo><msub id="alg2.l16.m1.4.4.4.4" xref="alg2.l16.m1.4.4.4.4.cmml"><mi mathsize="80%" id="alg2.l16.m1.4.4.4.4.2" xref="alg2.l16.m1.4.4.4.4.2.cmml">D</mi><mi mathsize="80%" id="alg2.l16.m1.4.4.4.4.3" xref="alg2.l16.m1.4.4.4.4.3.cmml">j</mi></msub><mo mathsize="80%" id="alg2.l16.m1.5.5.5.9" xref="alg2.l16.m1.5.5.6.cmml">,</mo><mrow id="alg2.l16.m1.5.5.5.5" xref="alg2.l16.m1.5.5.5.5.cmml"><mi mathsize="80%" id="alg2.l16.m1.5.5.5.5.2" xref="alg2.l16.m1.5.5.5.5.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg2.l16.m1.5.5.5.5.1" xref="alg2.l16.m1.5.5.5.5.1.cmml">​</mo><mi mathsize="80%" id="alg2.l16.m1.5.5.5.5.3" xref="alg2.l16.m1.5.5.5.5.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg2.l16.m1.5.5.5.5.1a" xref="alg2.l16.m1.5.5.5.5.1.cmml">​</mo><msub id="alg2.l16.m1.5.5.5.5.4" xref="alg2.l16.m1.5.5.5.5.4.cmml"><mi mathsize="80%" id="alg2.l16.m1.5.5.5.5.4.2" xref="alg2.l16.m1.5.5.5.5.4.2.cmml">e</mi><mi mathsize="80%" id="alg2.l16.m1.5.5.5.5.4.3" xref="alg2.l16.m1.5.5.5.5.4.3.cmml">j</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l16.m1.5b"><list id="alg2.l16.m1.5.5.6.cmml" xref="alg2.l16.m1.5.5.5"><apply id="alg2.l16.m1.1.1.1.1.cmml" xref="alg2.l16.m1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.l16.m1.1.1.1.1.1.cmml" xref="alg2.l16.m1.1.1.1.1">subscript</csymbol><ci id="alg2.l16.m1.1.1.1.1.2.cmml" xref="alg2.l16.m1.1.1.1.1.2">𝐐</ci><ci id="alg2.l16.m1.1.1.1.1.3.cmml" xref="alg2.l16.m1.1.1.1.1.3">𝑗</ci></apply><apply id="alg2.l16.m1.2.2.2.2.cmml" xref="alg2.l16.m1.2.2.2.2"><csymbol cd="ambiguous" id="alg2.l16.m1.2.2.2.2.1.cmml" xref="alg2.l16.m1.2.2.2.2">subscript</csymbol><ci id="alg2.l16.m1.2.2.2.2.2.cmml" xref="alg2.l16.m1.2.2.2.2.2">𝐝𝐐</ci><ci id="alg2.l16.m1.2.2.2.2.3.cmml" xref="alg2.l16.m1.2.2.2.2.3">𝑗</ci></apply><apply id="alg2.l16.m1.3.3.3.3.cmml" xref="alg2.l16.m1.3.3.3.3"><csymbol cd="ambiguous" id="alg2.l16.m1.3.3.3.3.1.cmml" xref="alg2.l16.m1.3.3.3.3">subscript</csymbol><ci id="alg2.l16.m1.3.3.3.3.2.cmml" xref="alg2.l16.m1.3.3.3.3.2">𝐝𝐎</ci><ci id="alg2.l16.m1.3.3.3.3.3.cmml" xref="alg2.l16.m1.3.3.3.3.3">𝑗</ci></apply><apply id="alg2.l16.m1.4.4.4.4.cmml" xref="alg2.l16.m1.4.4.4.4"><csymbol cd="ambiguous" id="alg2.l16.m1.4.4.4.4.1.cmml" xref="alg2.l16.m1.4.4.4.4">subscript</csymbol><ci id="alg2.l16.m1.4.4.4.4.2.cmml" xref="alg2.l16.m1.4.4.4.4.2">𝐷</ci><ci id="alg2.l16.m1.4.4.4.4.3.cmml" xref="alg2.l16.m1.4.4.4.4.3">𝑗</ci></apply><apply id="alg2.l16.m1.5.5.5.5.cmml" xref="alg2.l16.m1.5.5.5.5"><times id="alg2.l16.m1.5.5.5.5.1.cmml" xref="alg2.l16.m1.5.5.5.5.1"></times><ci id="alg2.l16.m1.5.5.5.5.2.cmml" xref="alg2.l16.m1.5.5.5.5.2">𝑙</ci><ci id="alg2.l16.m1.5.5.5.5.3.cmml" xref="alg2.l16.m1.5.5.5.5.3">𝑠</ci><apply id="alg2.l16.m1.5.5.5.5.4.cmml" xref="alg2.l16.m1.5.5.5.5.4"><csymbol cd="ambiguous" id="alg2.l16.m1.5.5.5.5.4.1.cmml" xref="alg2.l16.m1.5.5.5.5.4">subscript</csymbol><ci id="alg2.l16.m1.5.5.5.5.4.2.cmml" xref="alg2.l16.m1.5.5.5.5.4.2">𝑒</ci><ci id="alg2.l16.m1.5.5.5.5.4.3.cmml" xref="alg2.l16.m1.5.5.5.5.4.3">𝑗</ci></apply></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg2.l16.m1.5c">\mathbf{Q}_{j},\mathbf{dQ}_{j},\mathbf{dO}_{j},D_{j},lse_{j}</annotation></semantics></math><span id="alg2.l16.3" class="ltx_text" style="font-size:80%;"> into communication ring;
</span>
</div>
<div id="alg2.l17" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l17.1.1.1" class="ltx_text" style="font-size:80%;">17:</span></span><span id="alg2.l17.2" class="ltx_text" style="font-size:80%;">  </span><span id="alg2.l17.3" class="ltx_text ltx_font_bold" style="font-size:80%;">end</span><span id="alg2.l17.4" class="ltx_text" style="font-size:80%;"> </span><span id="alg2.l17.5" class="ltx_text ltx_font_bold" style="font-size:80%;">for</span>
</div>
<div id="alg2.l17a" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg2.l17a.1.1.1" class="ltx_text" style="font-size:80%;">17:</span></span><span id="alg2.l17a.2" class="ltx_text" style="font-size:80%;">  </span><math id="alg2.l17a.m1.3" class="ltx_Math" alttext="\mathbf{dQ}_{i},\mathbf{dK}_{i},\mathbf{dV}_{i}" display="inline"><semantics id="alg2.l17a.m1.3a"><mrow id="alg2.l17a.m1.3.3.3" xref="alg2.l17a.m1.3.3.4.cmml"><msub id="alg2.l17a.m1.1.1.1.1" xref="alg2.l17a.m1.1.1.1.1.cmml"><mi mathsize="80%" id="alg2.l17a.m1.1.1.1.1.2" xref="alg2.l17a.m1.1.1.1.1.2.cmml">𝐝𝐐</mi><mi mathsize="80%" id="alg2.l17a.m1.1.1.1.1.3" xref="alg2.l17a.m1.1.1.1.1.3.cmml">i</mi></msub><mo mathsize="80%" id="alg2.l17a.m1.3.3.3.4" xref="alg2.l17a.m1.3.3.4.cmml">,</mo><msub id="alg2.l17a.m1.2.2.2.2" xref="alg2.l17a.m1.2.2.2.2.cmml"><mi mathsize="80%" id="alg2.l17a.m1.2.2.2.2.2" xref="alg2.l17a.m1.2.2.2.2.2.cmml">𝐝𝐊</mi><mi mathsize="80%" id="alg2.l17a.m1.2.2.2.2.3" xref="alg2.l17a.m1.2.2.2.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg2.l17a.m1.3.3.3.5" xref="alg2.l17a.m1.3.3.4.cmml">,</mo><msub id="alg2.l17a.m1.3.3.3.3" xref="alg2.l17a.m1.3.3.3.3.cmml"><mi mathsize="80%" id="alg2.l17a.m1.3.3.3.3.2" xref="alg2.l17a.m1.3.3.3.3.2.cmml">𝐝𝐕</mi><mi mathsize="80%" id="alg2.l17a.m1.3.3.3.3.3" xref="alg2.l17a.m1.3.3.3.3.3.cmml">i</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="alg2.l17a.m1.3b"><list id="alg2.l17a.m1.3.3.4.cmml" xref="alg2.l17a.m1.3.3.3"><apply id="alg2.l17a.m1.1.1.1.1.cmml" xref="alg2.l17a.m1.1.1.1.1"><csymbol cd="ambiguous" id="alg2.l17a.m1.1.1.1.1.1.cmml" xref="alg2.l17a.m1.1.1.1.1">subscript</csymbol><ci id="alg2.l17a.m1.1.1.1.1.2.cmml" xref="alg2.l17a.m1.1.1.1.1.2">𝐝𝐐</ci><ci id="alg2.l17a.m1.1.1.1.1.3.cmml" xref="alg2.l17a.m1.1.1.1.1.3">𝑖</ci></apply><apply id="alg2.l17a.m1.2.2.2.2.cmml" xref="alg2.l17a.m1.2.2.2.2"><csymbol cd="ambiguous" id="alg2.l17a.m1.2.2.2.2.1.cmml" xref="alg2.l17a.m1.2.2.2.2">subscript</csymbol><ci id="alg2.l17a.m1.2.2.2.2.2.cmml" xref="alg2.l17a.m1.2.2.2.2.2">𝐝𝐊</ci><ci id="alg2.l17a.m1.2.2.2.2.3.cmml" xref="alg2.l17a.m1.2.2.2.2.3">𝑖</ci></apply><apply id="alg2.l17a.m1.3.3.3.3.cmml" xref="alg2.l17a.m1.3.3.3.3"><csymbol cd="ambiguous" id="alg2.l17a.m1.3.3.3.3.1.cmml" xref="alg2.l17a.m1.3.3.3.3">subscript</csymbol><ci id="alg2.l17a.m1.3.3.3.3.2.cmml" xref="alg2.l17a.m1.3.3.3.3.2">𝐝𝐕</ci><ci id="alg2.l17a.m1.3.3.3.3.3.cmml" xref="alg2.l17a.m1.3.3.3.3.3">𝑖</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg2.l17a.m1.3c">\mathbf{dQ}_{i},\mathbf{dK}_{i},\mathbf{dV}_{i}</annotation></semantics></math><span id="alg2.l17a.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
</div>
<figcaption class="ltx_caption" style="font-size:80%;"><span class="ltx_tag ltx_tag_float"><span id="alg2.5.1.1" class="ltx_text ltx_font_bold">Algorithm 2</span> </span> The backward pass of GAO</figcaption>
</figure>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>The Whole Framework of BurstAttention</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.14" class="ltx_p">In BurstAttention, <math id="S2.SS2.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{Q}" display="inline"><semantics id="S2.SS2.p1.1.m1.1a"><mi id="S2.SS2.p1.1.m1.1.1" xref="S2.SS2.p1.1.m1.1.1.cmml">𝐐</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.1.m1.1b"><ci id="S2.SS2.p1.1.m1.1.1.cmml" xref="S2.SS2.p1.1.m1.1.1">𝐐</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.1.m1.1c">\mathbf{Q}</annotation></semantics></math>, <math id="S2.SS2.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{K}" display="inline"><semantics id="S2.SS2.p1.2.m2.1a"><mi id="S2.SS2.p1.2.m2.1.1" xref="S2.SS2.p1.2.m2.1.1.cmml">𝐊</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.2.m2.1b"><ci id="S2.SS2.p1.2.m2.1.1.cmml" xref="S2.SS2.p1.2.m2.1.1">𝐊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.2.m2.1c">\mathbf{K}</annotation></semantics></math> and <math id="S2.SS2.p1.3.m3.1" class="ltx_Math" alttext="\mathbf{V}" display="inline"><semantics id="S2.SS2.p1.3.m3.1a"><mi id="S2.SS2.p1.3.m3.1.1" xref="S2.SS2.p1.3.m3.1.1.cmml">𝐕</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.3.m3.1b"><ci id="S2.SS2.p1.3.m3.1.1.cmml" xref="S2.SS2.p1.3.m3.1.1">𝐕</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.3.m3.1c">\mathbf{V}</annotation></semantics></math> are divided into multiple partitions along the sequence dimension according to the number of devices (e.g., GPUs) in a distributed cluster. Each device in the cluster will be assigned a query partition, a key partition, and a value partition.
Formally, given the device number <math id="S2.SS2.p1.4.m4.1" class="ltx_Math" alttext="G" display="inline"><semantics id="S2.SS2.p1.4.m4.1a"><mi id="S2.SS2.p1.4.m4.1.1" xref="S2.SS2.p1.4.m4.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.4.m4.1b"><ci id="S2.SS2.p1.4.m4.1.1.cmml" xref="S2.SS2.p1.4.m4.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.4.m4.1c">G</annotation></semantics></math>, the <math id="S2.SS2.p1.5.m5.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S2.SS2.p1.5.m5.1a"><mi id="S2.SS2.p1.5.m5.1.1" xref="S2.SS2.p1.5.m5.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.5.m5.1b"><ci id="S2.SS2.p1.5.m5.1.1.cmml" xref="S2.SS2.p1.5.m5.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.5.m5.1c">i</annotation></semantics></math>-th device will be assigned <math id="S2.SS2.p1.6.m6.3" class="ltx_Math" alttext="\mathbf{Q}_{i},\mathbf{K}_{i},\mathbf{V}_{i}\in\mathbb{R}^{\frac{N}{G}\times d}" display="inline"><semantics id="S2.SS2.p1.6.m6.3a"><mrow id="S2.SS2.p1.6.m6.3.3" xref="S2.SS2.p1.6.m6.3.3.cmml"><mrow id="S2.SS2.p1.6.m6.3.3.3.3" xref="S2.SS2.p1.6.m6.3.3.3.4.cmml"><msub id="S2.SS2.p1.6.m6.1.1.1.1.1" xref="S2.SS2.p1.6.m6.1.1.1.1.1.cmml"><mi id="S2.SS2.p1.6.m6.1.1.1.1.1.2" xref="S2.SS2.p1.6.m6.1.1.1.1.1.2.cmml">𝐐</mi><mi id="S2.SS2.p1.6.m6.1.1.1.1.1.3" xref="S2.SS2.p1.6.m6.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S2.SS2.p1.6.m6.3.3.3.3.4" xref="S2.SS2.p1.6.m6.3.3.3.4.cmml">,</mo><msub id="S2.SS2.p1.6.m6.2.2.2.2.2" xref="S2.SS2.p1.6.m6.2.2.2.2.2.cmml"><mi id="S2.SS2.p1.6.m6.2.2.2.2.2.2" xref="S2.SS2.p1.6.m6.2.2.2.2.2.2.cmml">𝐊</mi><mi id="S2.SS2.p1.6.m6.2.2.2.2.2.3" xref="S2.SS2.p1.6.m6.2.2.2.2.2.3.cmml">i</mi></msub><mo id="S2.SS2.p1.6.m6.3.3.3.3.5" xref="S2.SS2.p1.6.m6.3.3.3.4.cmml">,</mo><msub id="S2.SS2.p1.6.m6.3.3.3.3.3" xref="S2.SS2.p1.6.m6.3.3.3.3.3.cmml"><mi id="S2.SS2.p1.6.m6.3.3.3.3.3.2" xref="S2.SS2.p1.6.m6.3.3.3.3.3.2.cmml">𝐕</mi><mi id="S2.SS2.p1.6.m6.3.3.3.3.3.3" xref="S2.SS2.p1.6.m6.3.3.3.3.3.3.cmml">i</mi></msub></mrow><mo id="S2.SS2.p1.6.m6.3.3.4" xref="S2.SS2.p1.6.m6.3.3.4.cmml">∈</mo><msup id="S2.SS2.p1.6.m6.3.3.5" xref="S2.SS2.p1.6.m6.3.3.5.cmml"><mi id="S2.SS2.p1.6.m6.3.3.5.2" xref="S2.SS2.p1.6.m6.3.3.5.2.cmml">ℝ</mi><mrow id="S2.SS2.p1.6.m6.3.3.5.3" xref="S2.SS2.p1.6.m6.3.3.5.3.cmml"><mfrac id="S2.SS2.p1.6.m6.3.3.5.3.2" xref="S2.SS2.p1.6.m6.3.3.5.3.2.cmml"><mi id="S2.SS2.p1.6.m6.3.3.5.3.2.2" xref="S2.SS2.p1.6.m6.3.3.5.3.2.2.cmml">N</mi><mi id="S2.SS2.p1.6.m6.3.3.5.3.2.3" xref="S2.SS2.p1.6.m6.3.3.5.3.2.3.cmml">G</mi></mfrac><mo lspace="0.222em" rspace="0.222em" id="S2.SS2.p1.6.m6.3.3.5.3.1" xref="S2.SS2.p1.6.m6.3.3.5.3.1.cmml">×</mo><mi id="S2.SS2.p1.6.m6.3.3.5.3.3" xref="S2.SS2.p1.6.m6.3.3.5.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.6.m6.3b"><apply id="S2.SS2.p1.6.m6.3.3.cmml" xref="S2.SS2.p1.6.m6.3.3"><in id="S2.SS2.p1.6.m6.3.3.4.cmml" xref="S2.SS2.p1.6.m6.3.3.4"></in><list id="S2.SS2.p1.6.m6.3.3.3.4.cmml" xref="S2.SS2.p1.6.m6.3.3.3.3"><apply id="S2.SS2.p1.6.m6.1.1.1.1.1.cmml" xref="S2.SS2.p1.6.m6.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.6.m6.1.1.1.1.1.1.cmml" xref="S2.SS2.p1.6.m6.1.1.1.1.1">subscript</csymbol><ci id="S2.SS2.p1.6.m6.1.1.1.1.1.2.cmml" xref="S2.SS2.p1.6.m6.1.1.1.1.1.2">𝐐</ci><ci id="S2.SS2.p1.6.m6.1.1.1.1.1.3.cmml" xref="S2.SS2.p1.6.m6.1.1.1.1.1.3">𝑖</ci></apply><apply id="S2.SS2.p1.6.m6.2.2.2.2.2.cmml" xref="S2.SS2.p1.6.m6.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS2.p1.6.m6.2.2.2.2.2.1.cmml" xref="S2.SS2.p1.6.m6.2.2.2.2.2">subscript</csymbol><ci id="S2.SS2.p1.6.m6.2.2.2.2.2.2.cmml" xref="S2.SS2.p1.6.m6.2.2.2.2.2.2">𝐊</ci><ci id="S2.SS2.p1.6.m6.2.2.2.2.2.3.cmml" xref="S2.SS2.p1.6.m6.2.2.2.2.2.3">𝑖</ci></apply><apply id="S2.SS2.p1.6.m6.3.3.3.3.3.cmml" xref="S2.SS2.p1.6.m6.3.3.3.3.3"><csymbol cd="ambiguous" id="S2.SS2.p1.6.m6.3.3.3.3.3.1.cmml" xref="S2.SS2.p1.6.m6.3.3.3.3.3">subscript</csymbol><ci id="S2.SS2.p1.6.m6.3.3.3.3.3.2.cmml" xref="S2.SS2.p1.6.m6.3.3.3.3.3.2">𝐕</ci><ci id="S2.SS2.p1.6.m6.3.3.3.3.3.3.cmml" xref="S2.SS2.p1.6.m6.3.3.3.3.3.3">𝑖</ci></apply></list><apply id="S2.SS2.p1.6.m6.3.3.5.cmml" xref="S2.SS2.p1.6.m6.3.3.5"><csymbol cd="ambiguous" id="S2.SS2.p1.6.m6.3.3.5.1.cmml" xref="S2.SS2.p1.6.m6.3.3.5">superscript</csymbol><ci id="S2.SS2.p1.6.m6.3.3.5.2.cmml" xref="S2.SS2.p1.6.m6.3.3.5.2">ℝ</ci><apply id="S2.SS2.p1.6.m6.3.3.5.3.cmml" xref="S2.SS2.p1.6.m6.3.3.5.3"><times id="S2.SS2.p1.6.m6.3.3.5.3.1.cmml" xref="S2.SS2.p1.6.m6.3.3.5.3.1"></times><apply id="S2.SS2.p1.6.m6.3.3.5.3.2.cmml" xref="S2.SS2.p1.6.m6.3.3.5.3.2"><divide id="S2.SS2.p1.6.m6.3.3.5.3.2.1.cmml" xref="S2.SS2.p1.6.m6.3.3.5.3.2"></divide><ci id="S2.SS2.p1.6.m6.3.3.5.3.2.2.cmml" xref="S2.SS2.p1.6.m6.3.3.5.3.2.2">𝑁</ci><ci id="S2.SS2.p1.6.m6.3.3.5.3.2.3.cmml" xref="S2.SS2.p1.6.m6.3.3.5.3.2.3">𝐺</ci></apply><ci id="S2.SS2.p1.6.m6.3.3.5.3.3.cmml" xref="S2.SS2.p1.6.m6.3.3.5.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.6.m6.3c">\mathbf{Q}_{i},\mathbf{K}_{i},\mathbf{V}_{i}\in\mathbb{R}^{\frac{N}{G}\times d}</annotation></semantics></math>. As shown in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, at each step,
the <math id="S2.SS2.p1.7.m7.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S2.SS2.p1.7.m7.1a"><mi id="S2.SS2.p1.7.m7.1.1" xref="S2.SS2.p1.7.m7.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.7.m7.1b"><ci id="S2.SS2.p1.7.m7.1.1.cmml" xref="S2.SS2.p1.7.m7.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.7.m7.1c">i</annotation></semantics></math>-th device receives a key partition <math id="S2.SS2.p1.8.m8.1" class="ltx_Math" alttext="\mathbf{K}_{j}" display="inline"><semantics id="S2.SS2.p1.8.m8.1a"><msub id="S2.SS2.p1.8.m8.1.1" xref="S2.SS2.p1.8.m8.1.1.cmml"><mi id="S2.SS2.p1.8.m8.1.1.2" xref="S2.SS2.p1.8.m8.1.1.2.cmml">𝐊</mi><mi id="S2.SS2.p1.8.m8.1.1.3" xref="S2.SS2.p1.8.m8.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.8.m8.1b"><apply id="S2.SS2.p1.8.m8.1.1.cmml" xref="S2.SS2.p1.8.m8.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.8.m8.1.1.1.cmml" xref="S2.SS2.p1.8.m8.1.1">subscript</csymbol><ci id="S2.SS2.p1.8.m8.1.1.2.cmml" xref="S2.SS2.p1.8.m8.1.1.2">𝐊</ci><ci id="S2.SS2.p1.8.m8.1.1.3.cmml" xref="S2.SS2.p1.8.m8.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.8.m8.1c">\mathbf{K}_{j}</annotation></semantics></math> and a value partition <math id="S2.SS2.p1.9.m9.1" class="ltx_Math" alttext="\mathbf{V}_{j}" display="inline"><semantics id="S2.SS2.p1.9.m9.1a"><msub id="S2.SS2.p1.9.m9.1.1" xref="S2.SS2.p1.9.m9.1.1.cmml"><mi id="S2.SS2.p1.9.m9.1.1.2" xref="S2.SS2.p1.9.m9.1.1.2.cmml">𝐕</mi><mi id="S2.SS2.p1.9.m9.1.1.3" xref="S2.SS2.p1.9.m9.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.9.m9.1b"><apply id="S2.SS2.p1.9.m9.1.1.cmml" xref="S2.SS2.p1.9.m9.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.9.m9.1.1.1.cmml" xref="S2.SS2.p1.9.m9.1.1">subscript</csymbol><ci id="S2.SS2.p1.9.m9.1.1.2.cmml" xref="S2.SS2.p1.9.m9.1.1.2">𝐕</ci><ci id="S2.SS2.p1.9.m9.1.1.3.cmml" xref="S2.SS2.p1.9.m9.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.9.m9.1c">\mathbf{V}_{j}</annotation></semantics></math> from its previous neighbor and performs local attention operations. After that, the <math id="S2.SS2.p1.10.m10.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S2.SS2.p1.10.m10.1a"><mi id="S2.SS2.p1.10.m10.1.1" xref="S2.SS2.p1.10.m10.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.10.m10.1b"><ci id="S2.SS2.p1.10.m10.1.1.cmml" xref="S2.SS2.p1.10.m10.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.10.m10.1c">i</annotation></semantics></math>-th device sends its received key and value partitions <math id="S2.SS2.p1.11.m11.1" class="ltx_Math" alttext="\mathbf{K}_{j}" display="inline"><semantics id="S2.SS2.p1.11.m11.1a"><msub id="S2.SS2.p1.11.m11.1.1" xref="S2.SS2.p1.11.m11.1.1.cmml"><mi id="S2.SS2.p1.11.m11.1.1.2" xref="S2.SS2.p1.11.m11.1.1.2.cmml">𝐊</mi><mi id="S2.SS2.p1.11.m11.1.1.3" xref="S2.SS2.p1.11.m11.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.11.m11.1b"><apply id="S2.SS2.p1.11.m11.1.1.cmml" xref="S2.SS2.p1.11.m11.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.11.m11.1.1.1.cmml" xref="S2.SS2.p1.11.m11.1.1">subscript</csymbol><ci id="S2.SS2.p1.11.m11.1.1.2.cmml" xref="S2.SS2.p1.11.m11.1.1.2">𝐊</ci><ci id="S2.SS2.p1.11.m11.1.1.3.cmml" xref="S2.SS2.p1.11.m11.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.11.m11.1c">\mathbf{K}_{j}</annotation></semantics></math> and <math id="S2.SS2.p1.12.m12.1" class="ltx_Math" alttext="\mathbf{V}_{j}" display="inline"><semantics id="S2.SS2.p1.12.m12.1a"><msub id="S2.SS2.p1.12.m12.1.1" xref="S2.SS2.p1.12.m12.1.1.cmml"><mi id="S2.SS2.p1.12.m12.1.1.2" xref="S2.SS2.p1.12.m12.1.1.2.cmml">𝐕</mi><mi id="S2.SS2.p1.12.m12.1.1.3" xref="S2.SS2.p1.12.m12.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.12.m12.1b"><apply id="S2.SS2.p1.12.m12.1.1.cmml" xref="S2.SS2.p1.12.m12.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.12.m12.1.1.1.cmml" xref="S2.SS2.p1.12.m12.1.1">subscript</csymbol><ci id="S2.SS2.p1.12.m12.1.1.2.cmml" xref="S2.SS2.p1.12.m12.1.1.2">𝐕</ci><ci id="S2.SS2.p1.12.m12.1.1.3.cmml" xref="S2.SS2.p1.12.m12.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.12.m12.1c">\mathbf{V}_{j}</annotation></semantics></math> to its next neighbor for the use of the next step, which forms a ring-style communication process. This ring-style communication process continues until all <math id="S2.SS2.p1.13.m13.1" class="ltx_Math" alttext="\mathbf{K}" display="inline"><semantics id="S2.SS2.p1.13.m13.1a"><mi id="S2.SS2.p1.13.m13.1.1" xref="S2.SS2.p1.13.m13.1.1.cmml">𝐊</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.13.m13.1b"><ci id="S2.SS2.p1.13.m13.1.1.cmml" xref="S2.SS2.p1.13.m13.1.1">𝐊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.13.m13.1c">\mathbf{K}</annotation></semantics></math> and <math id="S2.SS2.p1.14.m14.1" class="ltx_Math" alttext="\mathbf{V}" display="inline"><semantics id="S2.SS2.p1.14.m14.1a"><mi id="S2.SS2.p1.14.m14.1.1" xref="S2.SS2.p1.14.m14.1.1.cmml">𝐕</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.14.m14.1b"><ci id="S2.SS2.p1.14.m14.1.1.cmml" xref="S2.SS2.p1.14.m14.1.1">𝐕</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.14.m14.1c">\mathbf{V}</annotation></semantics></math> partitions have made a full circle around the ring, completing local attention operations on all devices.
The local attention operations can be formalized as</p>
<table id="S2.E2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E2.m1.11" class="ltx_Math" alttext="\small\mathbf{S}_{i,j}=\frac{\mathbf{Q}_{i}\mathbf{K}_{j}^{T}}{\sqrt{d}},\;\mathbf{P}_{i,j}=\text{softmax}(\mathbf{S}_{i,j}),\;\mathbf{O}_{i,j}=\mathbf{P}_{i,j}\mathbf{V}_{j}," display="block"><semantics id="S2.E2.m1.11a"><mrow id="S2.E2.m1.11.11.1"><mrow id="S2.E2.m1.11.11.1.1.2" xref="S2.E2.m1.11.11.1.1.3.cmml"><mrow id="S2.E2.m1.11.11.1.1.1.1" xref="S2.E2.m1.11.11.1.1.1.1.cmml"><msub id="S2.E2.m1.11.11.1.1.1.1.2" xref="S2.E2.m1.11.11.1.1.1.1.2.cmml"><mi mathsize="90%" id="S2.E2.m1.11.11.1.1.1.1.2.2" xref="S2.E2.m1.11.11.1.1.1.1.2.2.cmml">𝐒</mi><mrow id="S2.E2.m1.2.2.2.4" xref="S2.E2.m1.2.2.2.3.cmml"><mi mathsize="90%" id="S2.E2.m1.1.1.1.1" xref="S2.E2.m1.1.1.1.1.cmml">i</mi><mo mathsize="90%" id="S2.E2.m1.2.2.2.4.1" xref="S2.E2.m1.2.2.2.3.cmml">,</mo><mi mathsize="90%" id="S2.E2.m1.2.2.2.2" xref="S2.E2.m1.2.2.2.2.cmml">j</mi></mrow></msub><mo mathsize="90%" id="S2.E2.m1.11.11.1.1.1.1.1" xref="S2.E2.m1.11.11.1.1.1.1.1.cmml">=</mo><mfrac id="S2.E2.m1.11.11.1.1.1.1.3" xref="S2.E2.m1.11.11.1.1.1.1.3.cmml"><mrow id="S2.E2.m1.11.11.1.1.1.1.3.2" xref="S2.E2.m1.11.11.1.1.1.1.3.2.cmml"><msub id="S2.E2.m1.11.11.1.1.1.1.3.2.2" xref="S2.E2.m1.11.11.1.1.1.1.3.2.2.cmml"><mi mathsize="90%" id="S2.E2.m1.11.11.1.1.1.1.3.2.2.2" xref="S2.E2.m1.11.11.1.1.1.1.3.2.2.2.cmml">𝐐</mi><mi mathsize="90%" id="S2.E2.m1.11.11.1.1.1.1.3.2.2.3" xref="S2.E2.m1.11.11.1.1.1.1.3.2.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S2.E2.m1.11.11.1.1.1.1.3.2.1" xref="S2.E2.m1.11.11.1.1.1.1.3.2.1.cmml">​</mo><msubsup id="S2.E2.m1.11.11.1.1.1.1.3.2.3" xref="S2.E2.m1.11.11.1.1.1.1.3.2.3.cmml"><mi mathsize="90%" id="S2.E2.m1.11.11.1.1.1.1.3.2.3.2.2" xref="S2.E2.m1.11.11.1.1.1.1.3.2.3.2.2.cmml">𝐊</mi><mi mathsize="90%" id="S2.E2.m1.11.11.1.1.1.1.3.2.3.2.3" xref="S2.E2.m1.11.11.1.1.1.1.3.2.3.2.3.cmml">j</mi><mi mathsize="90%" id="S2.E2.m1.11.11.1.1.1.1.3.2.3.3" xref="S2.E2.m1.11.11.1.1.1.1.3.2.3.3.cmml">T</mi></msubsup></mrow><msqrt id="S2.E2.m1.11.11.1.1.1.1.3.3" xref="S2.E2.m1.11.11.1.1.1.1.3.3.cmml"><mi mathsize="90%" id="S2.E2.m1.11.11.1.1.1.1.3.3.2" xref="S2.E2.m1.11.11.1.1.1.1.3.3.2.cmml">d</mi></msqrt></mfrac></mrow><mo mathsize="90%" rspace="0.447em" id="S2.E2.m1.11.11.1.1.2.3" xref="S2.E2.m1.11.11.1.1.3a.cmml">,</mo><mrow id="S2.E2.m1.11.11.1.1.2.2.2" xref="S2.E2.m1.11.11.1.1.2.2.3.cmml"><mrow id="S2.E2.m1.11.11.1.1.2.2.1.1" xref="S2.E2.m1.11.11.1.1.2.2.1.1.cmml"><msub id="S2.E2.m1.11.11.1.1.2.2.1.1.3" xref="S2.E2.m1.11.11.1.1.2.2.1.1.3.cmml"><mi mathsize="90%" id="S2.E2.m1.11.11.1.1.2.2.1.1.3.2" xref="S2.E2.m1.11.11.1.1.2.2.1.1.3.2.cmml">𝐏</mi><mrow id="S2.E2.m1.4.4.2.4" xref="S2.E2.m1.4.4.2.3.cmml"><mi mathsize="90%" id="S2.E2.m1.3.3.1.1" xref="S2.E2.m1.3.3.1.1.cmml">i</mi><mo mathsize="90%" id="S2.E2.m1.4.4.2.4.1" xref="S2.E2.m1.4.4.2.3.cmml">,</mo><mi mathsize="90%" id="S2.E2.m1.4.4.2.2" xref="S2.E2.m1.4.4.2.2.cmml">j</mi></mrow></msub><mo mathsize="90%" id="S2.E2.m1.11.11.1.1.2.2.1.1.2" xref="S2.E2.m1.11.11.1.1.2.2.1.1.2.cmml">=</mo><mrow id="S2.E2.m1.11.11.1.1.2.2.1.1.1" xref="S2.E2.m1.11.11.1.1.2.2.1.1.1.cmml"><mtext mathsize="90%" id="S2.E2.m1.11.11.1.1.2.2.1.1.1.3" xref="S2.E2.m1.11.11.1.1.2.2.1.1.1.3a.cmml">softmax</mtext><mo lspace="0em" rspace="0em" id="S2.E2.m1.11.11.1.1.2.2.1.1.1.2" xref="S2.E2.m1.11.11.1.1.2.2.1.1.1.2.cmml">​</mo><mrow id="S2.E2.m1.11.11.1.1.2.2.1.1.1.1.1" xref="S2.E2.m1.11.11.1.1.2.2.1.1.1.1.1.1.cmml"><mo maxsize="90%" minsize="90%" id="S2.E2.m1.11.11.1.1.2.2.1.1.1.1.1.2" xref="S2.E2.m1.11.11.1.1.2.2.1.1.1.1.1.1.cmml">(</mo><msub id="S2.E2.m1.11.11.1.1.2.2.1.1.1.1.1.1" xref="S2.E2.m1.11.11.1.1.2.2.1.1.1.1.1.1.cmml"><mi mathsize="90%" id="S2.E2.m1.11.11.1.1.2.2.1.1.1.1.1.1.2" xref="S2.E2.m1.11.11.1.1.2.2.1.1.1.1.1.1.2.cmml">𝐒</mi><mrow id="S2.E2.m1.6.6.2.4" xref="S2.E2.m1.6.6.2.3.cmml"><mi mathsize="90%" id="S2.E2.m1.5.5.1.1" xref="S2.E2.m1.5.5.1.1.cmml">i</mi><mo mathsize="90%" id="S2.E2.m1.6.6.2.4.1" xref="S2.E2.m1.6.6.2.3.cmml">,</mo><mi mathsize="90%" id="S2.E2.m1.6.6.2.2" xref="S2.E2.m1.6.6.2.2.cmml">j</mi></mrow></msub><mo maxsize="90%" minsize="90%" id="S2.E2.m1.11.11.1.1.2.2.1.1.1.1.1.3" xref="S2.E2.m1.11.11.1.1.2.2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo mathsize="90%" rspace="0.447em" id="S2.E2.m1.11.11.1.1.2.2.2.3" xref="S2.E2.m1.11.11.1.1.2.2.3a.cmml">,</mo><mrow id="S2.E2.m1.11.11.1.1.2.2.2.2" xref="S2.E2.m1.11.11.1.1.2.2.2.2.cmml"><msub id="S2.E2.m1.11.11.1.1.2.2.2.2.2" xref="S2.E2.m1.11.11.1.1.2.2.2.2.2.cmml"><mi mathsize="90%" id="S2.E2.m1.11.11.1.1.2.2.2.2.2.2" xref="S2.E2.m1.11.11.1.1.2.2.2.2.2.2.cmml">𝐎</mi><mrow id="S2.E2.m1.8.8.2.4" xref="S2.E2.m1.8.8.2.3.cmml"><mi mathsize="90%" id="S2.E2.m1.7.7.1.1" xref="S2.E2.m1.7.7.1.1.cmml">i</mi><mo mathsize="90%" id="S2.E2.m1.8.8.2.4.1" xref="S2.E2.m1.8.8.2.3.cmml">,</mo><mi mathsize="90%" id="S2.E2.m1.8.8.2.2" xref="S2.E2.m1.8.8.2.2.cmml">j</mi></mrow></msub><mo mathsize="90%" id="S2.E2.m1.11.11.1.1.2.2.2.2.1" xref="S2.E2.m1.11.11.1.1.2.2.2.2.1.cmml">=</mo><mrow id="S2.E2.m1.11.11.1.1.2.2.2.2.3" xref="S2.E2.m1.11.11.1.1.2.2.2.2.3.cmml"><msub id="S2.E2.m1.11.11.1.1.2.2.2.2.3.2" xref="S2.E2.m1.11.11.1.1.2.2.2.2.3.2.cmml"><mi mathsize="90%" id="S2.E2.m1.11.11.1.1.2.2.2.2.3.2.2" xref="S2.E2.m1.11.11.1.1.2.2.2.2.3.2.2.cmml">𝐏</mi><mrow id="S2.E2.m1.10.10.2.4" xref="S2.E2.m1.10.10.2.3.cmml"><mi mathsize="90%" id="S2.E2.m1.9.9.1.1" xref="S2.E2.m1.9.9.1.1.cmml">i</mi><mo mathsize="90%" id="S2.E2.m1.10.10.2.4.1" xref="S2.E2.m1.10.10.2.3.cmml">,</mo><mi mathsize="90%" id="S2.E2.m1.10.10.2.2" xref="S2.E2.m1.10.10.2.2.cmml">j</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S2.E2.m1.11.11.1.1.2.2.2.2.3.1" xref="S2.E2.m1.11.11.1.1.2.2.2.2.3.1.cmml">​</mo><msub id="S2.E2.m1.11.11.1.1.2.2.2.2.3.3" xref="S2.E2.m1.11.11.1.1.2.2.2.2.3.3.cmml"><mi mathsize="90%" id="S2.E2.m1.11.11.1.1.2.2.2.2.3.3.2" xref="S2.E2.m1.11.11.1.1.2.2.2.2.3.3.2.cmml">𝐕</mi><mi mathsize="90%" id="S2.E2.m1.11.11.1.1.2.2.2.2.3.3.3" xref="S2.E2.m1.11.11.1.1.2.2.2.2.3.3.3.cmml">j</mi></msub></mrow></mrow></mrow></mrow><mo mathsize="90%" id="S2.E2.m1.11.11.1.2">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E2.m1.11b"><apply id="S2.E2.m1.11.11.1.1.3.cmml" xref="S2.E2.m1.11.11.1.1.2"><csymbol cd="ambiguous" id="S2.E2.m1.11.11.1.1.3a.cmml" xref="S2.E2.m1.11.11.1.1.2.3">formulae-sequence</csymbol><apply id="S2.E2.m1.11.11.1.1.1.1.cmml" xref="S2.E2.m1.11.11.1.1.1.1"><eq id="S2.E2.m1.11.11.1.1.1.1.1.cmml" xref="S2.E2.m1.11.11.1.1.1.1.1"></eq><apply id="S2.E2.m1.11.11.1.1.1.1.2.cmml" xref="S2.E2.m1.11.11.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E2.m1.11.11.1.1.1.1.2.1.cmml" xref="S2.E2.m1.11.11.1.1.1.1.2">subscript</csymbol><ci id="S2.E2.m1.11.11.1.1.1.1.2.2.cmml" xref="S2.E2.m1.11.11.1.1.1.1.2.2">𝐒</ci><list id="S2.E2.m1.2.2.2.3.cmml" xref="S2.E2.m1.2.2.2.4"><ci id="S2.E2.m1.1.1.1.1.cmml" xref="S2.E2.m1.1.1.1.1">𝑖</ci><ci id="S2.E2.m1.2.2.2.2.cmml" xref="S2.E2.m1.2.2.2.2">𝑗</ci></list></apply><apply id="S2.E2.m1.11.11.1.1.1.1.3.cmml" xref="S2.E2.m1.11.11.1.1.1.1.3"><divide id="S2.E2.m1.11.11.1.1.1.1.3.1.cmml" xref="S2.E2.m1.11.11.1.1.1.1.3"></divide><apply id="S2.E2.m1.11.11.1.1.1.1.3.2.cmml" xref="S2.E2.m1.11.11.1.1.1.1.3.2"><times id="S2.E2.m1.11.11.1.1.1.1.3.2.1.cmml" xref="S2.E2.m1.11.11.1.1.1.1.3.2.1"></times><apply id="S2.E2.m1.11.11.1.1.1.1.3.2.2.cmml" xref="S2.E2.m1.11.11.1.1.1.1.3.2.2"><csymbol cd="ambiguous" id="S2.E2.m1.11.11.1.1.1.1.3.2.2.1.cmml" xref="S2.E2.m1.11.11.1.1.1.1.3.2.2">subscript</csymbol><ci id="S2.E2.m1.11.11.1.1.1.1.3.2.2.2.cmml" xref="S2.E2.m1.11.11.1.1.1.1.3.2.2.2">𝐐</ci><ci id="S2.E2.m1.11.11.1.1.1.1.3.2.2.3.cmml" xref="S2.E2.m1.11.11.1.1.1.1.3.2.2.3">𝑖</ci></apply><apply id="S2.E2.m1.11.11.1.1.1.1.3.2.3.cmml" xref="S2.E2.m1.11.11.1.1.1.1.3.2.3"><csymbol cd="ambiguous" id="S2.E2.m1.11.11.1.1.1.1.3.2.3.1.cmml" xref="S2.E2.m1.11.11.1.1.1.1.3.2.3">superscript</csymbol><apply id="S2.E2.m1.11.11.1.1.1.1.3.2.3.2.cmml" xref="S2.E2.m1.11.11.1.1.1.1.3.2.3"><csymbol cd="ambiguous" id="S2.E2.m1.11.11.1.1.1.1.3.2.3.2.1.cmml" xref="S2.E2.m1.11.11.1.1.1.1.3.2.3">subscript</csymbol><ci id="S2.E2.m1.11.11.1.1.1.1.3.2.3.2.2.cmml" xref="S2.E2.m1.11.11.1.1.1.1.3.2.3.2.2">𝐊</ci><ci id="S2.E2.m1.11.11.1.1.1.1.3.2.3.2.3.cmml" xref="S2.E2.m1.11.11.1.1.1.1.3.2.3.2.3">𝑗</ci></apply><ci id="S2.E2.m1.11.11.1.1.1.1.3.2.3.3.cmml" xref="S2.E2.m1.11.11.1.1.1.1.3.2.3.3">𝑇</ci></apply></apply><apply id="S2.E2.m1.11.11.1.1.1.1.3.3.cmml" xref="S2.E2.m1.11.11.1.1.1.1.3.3"><root id="S2.E2.m1.11.11.1.1.1.1.3.3a.cmml" xref="S2.E2.m1.11.11.1.1.1.1.3.3"></root><ci id="S2.E2.m1.11.11.1.1.1.1.3.3.2.cmml" xref="S2.E2.m1.11.11.1.1.1.1.3.3.2">𝑑</ci></apply></apply></apply><apply id="S2.E2.m1.11.11.1.1.2.2.3.cmml" xref="S2.E2.m1.11.11.1.1.2.2.2"><csymbol cd="ambiguous" id="S2.E2.m1.11.11.1.1.2.2.3a.cmml" xref="S2.E2.m1.11.11.1.1.2.2.2.3">formulae-sequence</csymbol><apply id="S2.E2.m1.11.11.1.1.2.2.1.1.cmml" xref="S2.E2.m1.11.11.1.1.2.2.1.1"><eq id="S2.E2.m1.11.11.1.1.2.2.1.1.2.cmml" xref="S2.E2.m1.11.11.1.1.2.2.1.1.2"></eq><apply id="S2.E2.m1.11.11.1.1.2.2.1.1.3.cmml" xref="S2.E2.m1.11.11.1.1.2.2.1.1.3"><csymbol cd="ambiguous" id="S2.E2.m1.11.11.1.1.2.2.1.1.3.1.cmml" xref="S2.E2.m1.11.11.1.1.2.2.1.1.3">subscript</csymbol><ci id="S2.E2.m1.11.11.1.1.2.2.1.1.3.2.cmml" xref="S2.E2.m1.11.11.1.1.2.2.1.1.3.2">𝐏</ci><list id="S2.E2.m1.4.4.2.3.cmml" xref="S2.E2.m1.4.4.2.4"><ci id="S2.E2.m1.3.3.1.1.cmml" xref="S2.E2.m1.3.3.1.1">𝑖</ci><ci id="S2.E2.m1.4.4.2.2.cmml" xref="S2.E2.m1.4.4.2.2">𝑗</ci></list></apply><apply id="S2.E2.m1.11.11.1.1.2.2.1.1.1.cmml" xref="S2.E2.m1.11.11.1.1.2.2.1.1.1"><times id="S2.E2.m1.11.11.1.1.2.2.1.1.1.2.cmml" xref="S2.E2.m1.11.11.1.1.2.2.1.1.1.2"></times><ci id="S2.E2.m1.11.11.1.1.2.2.1.1.1.3a.cmml" xref="S2.E2.m1.11.11.1.1.2.2.1.1.1.3"><mtext mathsize="90%" id="S2.E2.m1.11.11.1.1.2.2.1.1.1.3.cmml" xref="S2.E2.m1.11.11.1.1.2.2.1.1.1.3">softmax</mtext></ci><apply id="S2.E2.m1.11.11.1.1.2.2.1.1.1.1.1.1.cmml" xref="S2.E2.m1.11.11.1.1.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E2.m1.11.11.1.1.2.2.1.1.1.1.1.1.1.cmml" xref="S2.E2.m1.11.11.1.1.2.2.1.1.1.1.1">subscript</csymbol><ci id="S2.E2.m1.11.11.1.1.2.2.1.1.1.1.1.1.2.cmml" xref="S2.E2.m1.11.11.1.1.2.2.1.1.1.1.1.1.2">𝐒</ci><list id="S2.E2.m1.6.6.2.3.cmml" xref="S2.E2.m1.6.6.2.4"><ci id="S2.E2.m1.5.5.1.1.cmml" xref="S2.E2.m1.5.5.1.1">𝑖</ci><ci id="S2.E2.m1.6.6.2.2.cmml" xref="S2.E2.m1.6.6.2.2">𝑗</ci></list></apply></apply></apply><apply id="S2.E2.m1.11.11.1.1.2.2.2.2.cmml" xref="S2.E2.m1.11.11.1.1.2.2.2.2"><eq id="S2.E2.m1.11.11.1.1.2.2.2.2.1.cmml" xref="S2.E2.m1.11.11.1.1.2.2.2.2.1"></eq><apply id="S2.E2.m1.11.11.1.1.2.2.2.2.2.cmml" xref="S2.E2.m1.11.11.1.1.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.E2.m1.11.11.1.1.2.2.2.2.2.1.cmml" xref="S2.E2.m1.11.11.1.1.2.2.2.2.2">subscript</csymbol><ci id="S2.E2.m1.11.11.1.1.2.2.2.2.2.2.cmml" xref="S2.E2.m1.11.11.1.1.2.2.2.2.2.2">𝐎</ci><list id="S2.E2.m1.8.8.2.3.cmml" xref="S2.E2.m1.8.8.2.4"><ci id="S2.E2.m1.7.7.1.1.cmml" xref="S2.E2.m1.7.7.1.1">𝑖</ci><ci id="S2.E2.m1.8.8.2.2.cmml" xref="S2.E2.m1.8.8.2.2">𝑗</ci></list></apply><apply id="S2.E2.m1.11.11.1.1.2.2.2.2.3.cmml" xref="S2.E2.m1.11.11.1.1.2.2.2.2.3"><times id="S2.E2.m1.11.11.1.1.2.2.2.2.3.1.cmml" xref="S2.E2.m1.11.11.1.1.2.2.2.2.3.1"></times><apply id="S2.E2.m1.11.11.1.1.2.2.2.2.3.2.cmml" xref="S2.E2.m1.11.11.1.1.2.2.2.2.3.2"><csymbol cd="ambiguous" id="S2.E2.m1.11.11.1.1.2.2.2.2.3.2.1.cmml" xref="S2.E2.m1.11.11.1.1.2.2.2.2.3.2">subscript</csymbol><ci id="S2.E2.m1.11.11.1.1.2.2.2.2.3.2.2.cmml" xref="S2.E2.m1.11.11.1.1.2.2.2.2.3.2.2">𝐏</ci><list id="S2.E2.m1.10.10.2.3.cmml" xref="S2.E2.m1.10.10.2.4"><ci id="S2.E2.m1.9.9.1.1.cmml" xref="S2.E2.m1.9.9.1.1">𝑖</ci><ci id="S2.E2.m1.10.10.2.2.cmml" xref="S2.E2.m1.10.10.2.2">𝑗</ci></list></apply><apply id="S2.E2.m1.11.11.1.1.2.2.2.2.3.3.cmml" xref="S2.E2.m1.11.11.1.1.2.2.2.2.3.3"><csymbol cd="ambiguous" id="S2.E2.m1.11.11.1.1.2.2.2.2.3.3.1.cmml" xref="S2.E2.m1.11.11.1.1.2.2.2.2.3.3">subscript</csymbol><ci id="S2.E2.m1.11.11.1.1.2.2.2.2.3.3.2.cmml" xref="S2.E2.m1.11.11.1.1.2.2.2.2.3.3.2">𝐕</ci><ci id="S2.E2.m1.11.11.1.1.2.2.2.2.3.3.3.cmml" xref="S2.E2.m1.11.11.1.1.2.2.2.2.3.3.3">𝑗</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E2.m1.11c">\small\mathbf{S}_{i,j}=\frac{\mathbf{Q}_{i}\mathbf{K}_{j}^{T}}{\sqrt{d}},\;\mathbf{P}_{i,j}=\text{softmax}(\mathbf{S}_{i,j}),\;\mathbf{O}_{i,j}=\mathbf{P}_{i,j}\mathbf{V}_{j},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
</div>
<div id="S2.SS2.p2" class="ltx_para">
<p id="S2.SS2.p2.6" class="ltx_p">where <math id="S2.SS2.p2.1.m1.2" class="ltx_Math" alttext="\mathbf{O}_{i,j}\in\mathbb{R}^{\frac{N}{G}\times d}" display="inline"><semantics id="S2.SS2.p2.1.m1.2a"><mrow id="S2.SS2.p2.1.m1.2.3" xref="S2.SS2.p2.1.m1.2.3.cmml"><msub id="S2.SS2.p2.1.m1.2.3.2" xref="S2.SS2.p2.1.m1.2.3.2.cmml"><mi id="S2.SS2.p2.1.m1.2.3.2.2" xref="S2.SS2.p2.1.m1.2.3.2.2.cmml">𝐎</mi><mrow id="S2.SS2.p2.1.m1.2.2.2.4" xref="S2.SS2.p2.1.m1.2.2.2.3.cmml"><mi id="S2.SS2.p2.1.m1.1.1.1.1" xref="S2.SS2.p2.1.m1.1.1.1.1.cmml">i</mi><mo id="S2.SS2.p2.1.m1.2.2.2.4.1" xref="S2.SS2.p2.1.m1.2.2.2.3.cmml">,</mo><mi id="S2.SS2.p2.1.m1.2.2.2.2" xref="S2.SS2.p2.1.m1.2.2.2.2.cmml">j</mi></mrow></msub><mo id="S2.SS2.p2.1.m1.2.3.1" xref="S2.SS2.p2.1.m1.2.3.1.cmml">∈</mo><msup id="S2.SS2.p2.1.m1.2.3.3" xref="S2.SS2.p2.1.m1.2.3.3.cmml"><mi id="S2.SS2.p2.1.m1.2.3.3.2" xref="S2.SS2.p2.1.m1.2.3.3.2.cmml">ℝ</mi><mrow id="S2.SS2.p2.1.m1.2.3.3.3" xref="S2.SS2.p2.1.m1.2.3.3.3.cmml"><mfrac id="S2.SS2.p2.1.m1.2.3.3.3.2" xref="S2.SS2.p2.1.m1.2.3.3.3.2.cmml"><mi id="S2.SS2.p2.1.m1.2.3.3.3.2.2" xref="S2.SS2.p2.1.m1.2.3.3.3.2.2.cmml">N</mi><mi id="S2.SS2.p2.1.m1.2.3.3.3.2.3" xref="S2.SS2.p2.1.m1.2.3.3.3.2.3.cmml">G</mi></mfrac><mo lspace="0.222em" rspace="0.222em" id="S2.SS2.p2.1.m1.2.3.3.3.1" xref="S2.SS2.p2.1.m1.2.3.3.3.1.cmml">×</mo><mi id="S2.SS2.p2.1.m1.2.3.3.3.3" xref="S2.SS2.p2.1.m1.2.3.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.1.m1.2b"><apply id="S2.SS2.p2.1.m1.2.3.cmml" xref="S2.SS2.p2.1.m1.2.3"><in id="S2.SS2.p2.1.m1.2.3.1.cmml" xref="S2.SS2.p2.1.m1.2.3.1"></in><apply id="S2.SS2.p2.1.m1.2.3.2.cmml" xref="S2.SS2.p2.1.m1.2.3.2"><csymbol cd="ambiguous" id="S2.SS2.p2.1.m1.2.3.2.1.cmml" xref="S2.SS2.p2.1.m1.2.3.2">subscript</csymbol><ci id="S2.SS2.p2.1.m1.2.3.2.2.cmml" xref="S2.SS2.p2.1.m1.2.3.2.2">𝐎</ci><list id="S2.SS2.p2.1.m1.2.2.2.3.cmml" xref="S2.SS2.p2.1.m1.2.2.2.4"><ci id="S2.SS2.p2.1.m1.1.1.1.1.cmml" xref="S2.SS2.p2.1.m1.1.1.1.1">𝑖</ci><ci id="S2.SS2.p2.1.m1.2.2.2.2.cmml" xref="S2.SS2.p2.1.m1.2.2.2.2">𝑗</ci></list></apply><apply id="S2.SS2.p2.1.m1.2.3.3.cmml" xref="S2.SS2.p2.1.m1.2.3.3"><csymbol cd="ambiguous" id="S2.SS2.p2.1.m1.2.3.3.1.cmml" xref="S2.SS2.p2.1.m1.2.3.3">superscript</csymbol><ci id="S2.SS2.p2.1.m1.2.3.3.2.cmml" xref="S2.SS2.p2.1.m1.2.3.3.2">ℝ</ci><apply id="S2.SS2.p2.1.m1.2.3.3.3.cmml" xref="S2.SS2.p2.1.m1.2.3.3.3"><times id="S2.SS2.p2.1.m1.2.3.3.3.1.cmml" xref="S2.SS2.p2.1.m1.2.3.3.3.1"></times><apply id="S2.SS2.p2.1.m1.2.3.3.3.2.cmml" xref="S2.SS2.p2.1.m1.2.3.3.3.2"><divide id="S2.SS2.p2.1.m1.2.3.3.3.2.1.cmml" xref="S2.SS2.p2.1.m1.2.3.3.3.2"></divide><ci id="S2.SS2.p2.1.m1.2.3.3.3.2.2.cmml" xref="S2.SS2.p2.1.m1.2.3.3.3.2.2">𝑁</ci><ci id="S2.SS2.p2.1.m1.2.3.3.3.2.3.cmml" xref="S2.SS2.p2.1.m1.2.3.3.3.2.3">𝐺</ci></apply><ci id="S2.SS2.p2.1.m1.2.3.3.3.3.cmml" xref="S2.SS2.p2.1.m1.2.3.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.1.m1.2c">\mathbf{O}_{i,j}\in\mathbb{R}^{\frac{N}{G}\times d}</annotation></semantics></math> indicates the local attention results between the device-assigned query partition <math id="S2.SS2.p2.2.m2.1" class="ltx_Math" alttext="\mathbf{Q}_{i}" display="inline"><semantics id="S2.SS2.p2.2.m2.1a"><msub id="S2.SS2.p2.2.m2.1.1" xref="S2.SS2.p2.2.m2.1.1.cmml"><mi id="S2.SS2.p2.2.m2.1.1.2" xref="S2.SS2.p2.2.m2.1.1.2.cmml">𝐐</mi><mi id="S2.SS2.p2.2.m2.1.1.3" xref="S2.SS2.p2.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.2.m2.1b"><apply id="S2.SS2.p2.2.m2.1.1.cmml" xref="S2.SS2.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS2.p2.2.m2.1.1.1.cmml" xref="S2.SS2.p2.2.m2.1.1">subscript</csymbol><ci id="S2.SS2.p2.2.m2.1.1.2.cmml" xref="S2.SS2.p2.2.m2.1.1.2">𝐐</ci><ci id="S2.SS2.p2.2.m2.1.1.3.cmml" xref="S2.SS2.p2.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.2.m2.1c">\mathbf{Q}_{i}</annotation></semantics></math> and the device-received partitions <math id="S2.SS2.p2.3.m3.1" class="ltx_Math" alttext="\mathbf{K}_{j}" display="inline"><semantics id="S2.SS2.p2.3.m3.1a"><msub id="S2.SS2.p2.3.m3.1.1" xref="S2.SS2.p2.3.m3.1.1.cmml"><mi id="S2.SS2.p2.3.m3.1.1.2" xref="S2.SS2.p2.3.m3.1.1.2.cmml">𝐊</mi><mi id="S2.SS2.p2.3.m3.1.1.3" xref="S2.SS2.p2.3.m3.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.3.m3.1b"><apply id="S2.SS2.p2.3.m3.1.1.cmml" xref="S2.SS2.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS2.p2.3.m3.1.1.1.cmml" xref="S2.SS2.p2.3.m3.1.1">subscript</csymbol><ci id="S2.SS2.p2.3.m3.1.1.2.cmml" xref="S2.SS2.p2.3.m3.1.1.2">𝐊</ci><ci id="S2.SS2.p2.3.m3.1.1.3.cmml" xref="S2.SS2.p2.3.m3.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.3.m3.1c">\mathbf{K}_{j}</annotation></semantics></math> and <math id="S2.SS2.p2.4.m4.1" class="ltx_Math" alttext="\mathbf{V}_{j}" display="inline"><semantics id="S2.SS2.p2.4.m4.1a"><msub id="S2.SS2.p2.4.m4.1.1" xref="S2.SS2.p2.4.m4.1.1.cmml"><mi id="S2.SS2.p2.4.m4.1.1.2" xref="S2.SS2.p2.4.m4.1.1.2.cmml">𝐕</mi><mi id="S2.SS2.p2.4.m4.1.1.3" xref="S2.SS2.p2.4.m4.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.4.m4.1b"><apply id="S2.SS2.p2.4.m4.1.1.cmml" xref="S2.SS2.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS2.p2.4.m4.1.1.1.cmml" xref="S2.SS2.p2.4.m4.1.1">subscript</csymbol><ci id="S2.SS2.p2.4.m4.1.1.2.cmml" xref="S2.SS2.p2.4.m4.1.1.2">𝐕</ci><ci id="S2.SS2.p2.4.m4.1.1.3.cmml" xref="S2.SS2.p2.4.m4.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.4.m4.1c">\mathbf{V}_{j}</annotation></semantics></math>.
<math id="S2.SS2.p2.5.m5.2" class="ltx_Math" alttext="\mathbf{S}_{i,j}\in\mathbb{R}^{\frac{N}{G}\times\frac{N}{G}}" display="inline"><semantics id="S2.SS2.p2.5.m5.2a"><mrow id="S2.SS2.p2.5.m5.2.3" xref="S2.SS2.p2.5.m5.2.3.cmml"><msub id="S2.SS2.p2.5.m5.2.3.2" xref="S2.SS2.p2.5.m5.2.3.2.cmml"><mi id="S2.SS2.p2.5.m5.2.3.2.2" xref="S2.SS2.p2.5.m5.2.3.2.2.cmml">𝐒</mi><mrow id="S2.SS2.p2.5.m5.2.2.2.4" xref="S2.SS2.p2.5.m5.2.2.2.3.cmml"><mi id="S2.SS2.p2.5.m5.1.1.1.1" xref="S2.SS2.p2.5.m5.1.1.1.1.cmml">i</mi><mo id="S2.SS2.p2.5.m5.2.2.2.4.1" xref="S2.SS2.p2.5.m5.2.2.2.3.cmml">,</mo><mi id="S2.SS2.p2.5.m5.2.2.2.2" xref="S2.SS2.p2.5.m5.2.2.2.2.cmml">j</mi></mrow></msub><mo id="S2.SS2.p2.5.m5.2.3.1" xref="S2.SS2.p2.5.m5.2.3.1.cmml">∈</mo><msup id="S2.SS2.p2.5.m5.2.3.3" xref="S2.SS2.p2.5.m5.2.3.3.cmml"><mi id="S2.SS2.p2.5.m5.2.3.3.2" xref="S2.SS2.p2.5.m5.2.3.3.2.cmml">ℝ</mi><mrow id="S2.SS2.p2.5.m5.2.3.3.3" xref="S2.SS2.p2.5.m5.2.3.3.3.cmml"><mfrac id="S2.SS2.p2.5.m5.2.3.3.3.2" xref="S2.SS2.p2.5.m5.2.3.3.3.2.cmml"><mi id="S2.SS2.p2.5.m5.2.3.3.3.2.2" xref="S2.SS2.p2.5.m5.2.3.3.3.2.2.cmml">N</mi><mi id="S2.SS2.p2.5.m5.2.3.3.3.2.3" xref="S2.SS2.p2.5.m5.2.3.3.3.2.3.cmml">G</mi></mfrac><mo lspace="0.222em" rspace="0.222em" id="S2.SS2.p2.5.m5.2.3.3.3.1" xref="S2.SS2.p2.5.m5.2.3.3.3.1.cmml">×</mo><mfrac id="S2.SS2.p2.5.m5.2.3.3.3.3" xref="S2.SS2.p2.5.m5.2.3.3.3.3.cmml"><mi id="S2.SS2.p2.5.m5.2.3.3.3.3.2" xref="S2.SS2.p2.5.m5.2.3.3.3.3.2.cmml">N</mi><mi id="S2.SS2.p2.5.m5.2.3.3.3.3.3" xref="S2.SS2.p2.5.m5.2.3.3.3.3.3.cmml">G</mi></mfrac></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.5.m5.2b"><apply id="S2.SS2.p2.5.m5.2.3.cmml" xref="S2.SS2.p2.5.m5.2.3"><in id="S2.SS2.p2.5.m5.2.3.1.cmml" xref="S2.SS2.p2.5.m5.2.3.1"></in><apply id="S2.SS2.p2.5.m5.2.3.2.cmml" xref="S2.SS2.p2.5.m5.2.3.2"><csymbol cd="ambiguous" id="S2.SS2.p2.5.m5.2.3.2.1.cmml" xref="S2.SS2.p2.5.m5.2.3.2">subscript</csymbol><ci id="S2.SS2.p2.5.m5.2.3.2.2.cmml" xref="S2.SS2.p2.5.m5.2.3.2.2">𝐒</ci><list id="S2.SS2.p2.5.m5.2.2.2.3.cmml" xref="S2.SS2.p2.5.m5.2.2.2.4"><ci id="S2.SS2.p2.5.m5.1.1.1.1.cmml" xref="S2.SS2.p2.5.m5.1.1.1.1">𝑖</ci><ci id="S2.SS2.p2.5.m5.2.2.2.2.cmml" xref="S2.SS2.p2.5.m5.2.2.2.2">𝑗</ci></list></apply><apply id="S2.SS2.p2.5.m5.2.3.3.cmml" xref="S2.SS2.p2.5.m5.2.3.3"><csymbol cd="ambiguous" id="S2.SS2.p2.5.m5.2.3.3.1.cmml" xref="S2.SS2.p2.5.m5.2.3.3">superscript</csymbol><ci id="S2.SS2.p2.5.m5.2.3.3.2.cmml" xref="S2.SS2.p2.5.m5.2.3.3.2">ℝ</ci><apply id="S2.SS2.p2.5.m5.2.3.3.3.cmml" xref="S2.SS2.p2.5.m5.2.3.3.3"><times id="S2.SS2.p2.5.m5.2.3.3.3.1.cmml" xref="S2.SS2.p2.5.m5.2.3.3.3.1"></times><apply id="S2.SS2.p2.5.m5.2.3.3.3.2.cmml" xref="S2.SS2.p2.5.m5.2.3.3.3.2"><divide id="S2.SS2.p2.5.m5.2.3.3.3.2.1.cmml" xref="S2.SS2.p2.5.m5.2.3.3.3.2"></divide><ci id="S2.SS2.p2.5.m5.2.3.3.3.2.2.cmml" xref="S2.SS2.p2.5.m5.2.3.3.3.2.2">𝑁</ci><ci id="S2.SS2.p2.5.m5.2.3.3.3.2.3.cmml" xref="S2.SS2.p2.5.m5.2.3.3.3.2.3">𝐺</ci></apply><apply id="S2.SS2.p2.5.m5.2.3.3.3.3.cmml" xref="S2.SS2.p2.5.m5.2.3.3.3.3"><divide id="S2.SS2.p2.5.m5.2.3.3.3.3.1.cmml" xref="S2.SS2.p2.5.m5.2.3.3.3.3"></divide><ci id="S2.SS2.p2.5.m5.2.3.3.3.3.2.cmml" xref="S2.SS2.p2.5.m5.2.3.3.3.3.2">𝑁</ci><ci id="S2.SS2.p2.5.m5.2.3.3.3.3.3.cmml" xref="S2.SS2.p2.5.m5.2.3.3.3.3.3">𝐺</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.5.m5.2c">\mathbf{S}_{i,j}\in\mathbb{R}^{\frac{N}{G}\times\frac{N}{G}}</annotation></semantics></math> and <math id="S2.SS2.p2.6.m6.2" class="ltx_Math" alttext="\mathbf{P}_{i,j}\in\mathbb{R}^{\frac{N}{G}\times\frac{N}{G}}" display="inline"><semantics id="S2.SS2.p2.6.m6.2a"><mrow id="S2.SS2.p2.6.m6.2.3" xref="S2.SS2.p2.6.m6.2.3.cmml"><msub id="S2.SS2.p2.6.m6.2.3.2" xref="S2.SS2.p2.6.m6.2.3.2.cmml"><mi id="S2.SS2.p2.6.m6.2.3.2.2" xref="S2.SS2.p2.6.m6.2.3.2.2.cmml">𝐏</mi><mrow id="S2.SS2.p2.6.m6.2.2.2.4" xref="S2.SS2.p2.6.m6.2.2.2.3.cmml"><mi id="S2.SS2.p2.6.m6.1.1.1.1" xref="S2.SS2.p2.6.m6.1.1.1.1.cmml">i</mi><mo id="S2.SS2.p2.6.m6.2.2.2.4.1" xref="S2.SS2.p2.6.m6.2.2.2.3.cmml">,</mo><mi id="S2.SS2.p2.6.m6.2.2.2.2" xref="S2.SS2.p2.6.m6.2.2.2.2.cmml">j</mi></mrow></msub><mo id="S2.SS2.p2.6.m6.2.3.1" xref="S2.SS2.p2.6.m6.2.3.1.cmml">∈</mo><msup id="S2.SS2.p2.6.m6.2.3.3" xref="S2.SS2.p2.6.m6.2.3.3.cmml"><mi id="S2.SS2.p2.6.m6.2.3.3.2" xref="S2.SS2.p2.6.m6.2.3.3.2.cmml">ℝ</mi><mrow id="S2.SS2.p2.6.m6.2.3.3.3" xref="S2.SS2.p2.6.m6.2.3.3.3.cmml"><mfrac id="S2.SS2.p2.6.m6.2.3.3.3.2" xref="S2.SS2.p2.6.m6.2.3.3.3.2.cmml"><mi id="S2.SS2.p2.6.m6.2.3.3.3.2.2" xref="S2.SS2.p2.6.m6.2.3.3.3.2.2.cmml">N</mi><mi id="S2.SS2.p2.6.m6.2.3.3.3.2.3" xref="S2.SS2.p2.6.m6.2.3.3.3.2.3.cmml">G</mi></mfrac><mo lspace="0.222em" rspace="0.222em" id="S2.SS2.p2.6.m6.2.3.3.3.1" xref="S2.SS2.p2.6.m6.2.3.3.3.1.cmml">×</mo><mfrac id="S2.SS2.p2.6.m6.2.3.3.3.3" xref="S2.SS2.p2.6.m6.2.3.3.3.3.cmml"><mi id="S2.SS2.p2.6.m6.2.3.3.3.3.2" xref="S2.SS2.p2.6.m6.2.3.3.3.3.2.cmml">N</mi><mi id="S2.SS2.p2.6.m6.2.3.3.3.3.3" xref="S2.SS2.p2.6.m6.2.3.3.3.3.3.cmml">G</mi></mfrac></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.6.m6.2b"><apply id="S2.SS2.p2.6.m6.2.3.cmml" xref="S2.SS2.p2.6.m6.2.3"><in id="S2.SS2.p2.6.m6.2.3.1.cmml" xref="S2.SS2.p2.6.m6.2.3.1"></in><apply id="S2.SS2.p2.6.m6.2.3.2.cmml" xref="S2.SS2.p2.6.m6.2.3.2"><csymbol cd="ambiguous" id="S2.SS2.p2.6.m6.2.3.2.1.cmml" xref="S2.SS2.p2.6.m6.2.3.2">subscript</csymbol><ci id="S2.SS2.p2.6.m6.2.3.2.2.cmml" xref="S2.SS2.p2.6.m6.2.3.2.2">𝐏</ci><list id="S2.SS2.p2.6.m6.2.2.2.3.cmml" xref="S2.SS2.p2.6.m6.2.2.2.4"><ci id="S2.SS2.p2.6.m6.1.1.1.1.cmml" xref="S2.SS2.p2.6.m6.1.1.1.1">𝑖</ci><ci id="S2.SS2.p2.6.m6.2.2.2.2.cmml" xref="S2.SS2.p2.6.m6.2.2.2.2">𝑗</ci></list></apply><apply id="S2.SS2.p2.6.m6.2.3.3.cmml" xref="S2.SS2.p2.6.m6.2.3.3"><csymbol cd="ambiguous" id="S2.SS2.p2.6.m6.2.3.3.1.cmml" xref="S2.SS2.p2.6.m6.2.3.3">superscript</csymbol><ci id="S2.SS2.p2.6.m6.2.3.3.2.cmml" xref="S2.SS2.p2.6.m6.2.3.3.2">ℝ</ci><apply id="S2.SS2.p2.6.m6.2.3.3.3.cmml" xref="S2.SS2.p2.6.m6.2.3.3.3"><times id="S2.SS2.p2.6.m6.2.3.3.3.1.cmml" xref="S2.SS2.p2.6.m6.2.3.3.3.1"></times><apply id="S2.SS2.p2.6.m6.2.3.3.3.2.cmml" xref="S2.SS2.p2.6.m6.2.3.3.3.2"><divide id="S2.SS2.p2.6.m6.2.3.3.3.2.1.cmml" xref="S2.SS2.p2.6.m6.2.3.3.3.2"></divide><ci id="S2.SS2.p2.6.m6.2.3.3.3.2.2.cmml" xref="S2.SS2.p2.6.m6.2.3.3.3.2.2">𝑁</ci><ci id="S2.SS2.p2.6.m6.2.3.3.3.2.3.cmml" xref="S2.SS2.p2.6.m6.2.3.3.3.2.3">𝐺</ci></apply><apply id="S2.SS2.p2.6.m6.2.3.3.3.3.cmml" xref="S2.SS2.p2.6.m6.2.3.3.3.3"><divide id="S2.SS2.p2.6.m6.2.3.3.3.3.1.cmml" xref="S2.SS2.p2.6.m6.2.3.3.3.3"></divide><ci id="S2.SS2.p2.6.m6.2.3.3.3.3.2.cmml" xref="S2.SS2.p2.6.m6.2.3.3.3.3.2">𝑁</ci><ci id="S2.SS2.p2.6.m6.2.3.3.3.3.3.cmml" xref="S2.SS2.p2.6.m6.2.3.3.3.3.3">𝐺</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.6.m6.2c">\mathbf{P}_{i,j}\in\mathbb{R}^{\frac{N}{G}\times\frac{N}{G}}</annotation></semantics></math> indicate the local attention scores and the local attention probabilities, respectively.</p>
</div>
<div id="S2.SS2.p3" class="ltx_para">
<p id="S2.SS2.p3.3" class="ltx_p">Obviously, Eq. (<a href="#S2.E1" title="Equation 1 ‣ 2.1 Preliminary ‣ 2 Methodology ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>) and Eq. (<a href="#S2.E2" title="Equation 2 ‣ 2.2 The Whole Framework of BurstAttention ‣ 2 Methodology ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>) are not equivalent, we thus introduce global attention operations to aggregate all local attention results
<math id="S2.SS2.p3.1.m1.7" class="ltx_Math" alttext="\{\mathbf{O}_{i,j}\}_{i=1,j=1}^{\frac{N}{G},\frac{N}{G}}" display="inline"><semantics id="S2.SS2.p3.1.m1.7a"><msubsup id="S2.SS2.p3.1.m1.7.7" xref="S2.SS2.p3.1.m1.7.7.cmml"><mrow id="S2.SS2.p3.1.m1.7.7.1.1.1" xref="S2.SS2.p3.1.m1.7.7.1.1.2.cmml"><mo stretchy="false" id="S2.SS2.p3.1.m1.7.7.1.1.1.2" xref="S2.SS2.p3.1.m1.7.7.1.1.2.cmml">{</mo><msub id="S2.SS2.p3.1.m1.7.7.1.1.1.1" xref="S2.SS2.p3.1.m1.7.7.1.1.1.1.cmml"><mi id="S2.SS2.p3.1.m1.7.7.1.1.1.1.2" xref="S2.SS2.p3.1.m1.7.7.1.1.1.1.2.cmml">𝐎</mi><mrow id="S2.SS2.p3.1.m1.2.2.2.4" xref="S2.SS2.p3.1.m1.2.2.2.3.cmml"><mi id="S2.SS2.p3.1.m1.1.1.1.1" xref="S2.SS2.p3.1.m1.1.1.1.1.cmml">i</mi><mo id="S2.SS2.p3.1.m1.2.2.2.4.1" xref="S2.SS2.p3.1.m1.2.2.2.3.cmml">,</mo><mi id="S2.SS2.p3.1.m1.2.2.2.2" xref="S2.SS2.p3.1.m1.2.2.2.2.cmml">j</mi></mrow></msub><mo stretchy="false" id="S2.SS2.p3.1.m1.7.7.1.1.1.3" xref="S2.SS2.p3.1.m1.7.7.1.1.2.cmml">}</mo></mrow><mrow id="S2.SS2.p3.1.m1.4.4.2.2" xref="S2.SS2.p3.1.m1.4.4.2.3.cmml"><mrow id="S2.SS2.p3.1.m1.3.3.1.1.1" xref="S2.SS2.p3.1.m1.3.3.1.1.1.cmml"><mi id="S2.SS2.p3.1.m1.3.3.1.1.1.2" xref="S2.SS2.p3.1.m1.3.3.1.1.1.2.cmml">i</mi><mo id="S2.SS2.p3.1.m1.3.3.1.1.1.1" xref="S2.SS2.p3.1.m1.3.3.1.1.1.1.cmml">=</mo><mn id="S2.SS2.p3.1.m1.3.3.1.1.1.3" xref="S2.SS2.p3.1.m1.3.3.1.1.1.3.cmml">1</mn></mrow><mo id="S2.SS2.p3.1.m1.4.4.2.2.3" xref="S2.SS2.p3.1.m1.4.4.2.3a.cmml">,</mo><mrow id="S2.SS2.p3.1.m1.4.4.2.2.2" xref="S2.SS2.p3.1.m1.4.4.2.2.2.cmml"><mi id="S2.SS2.p3.1.m1.4.4.2.2.2.2" xref="S2.SS2.p3.1.m1.4.4.2.2.2.2.cmml">j</mi><mo id="S2.SS2.p3.1.m1.4.4.2.2.2.1" xref="S2.SS2.p3.1.m1.4.4.2.2.2.1.cmml">=</mo><mn id="S2.SS2.p3.1.m1.4.4.2.2.2.3" xref="S2.SS2.p3.1.m1.4.4.2.2.2.3.cmml">1</mn></mrow></mrow><mrow id="S2.SS2.p3.1.m1.6.6.2.4" xref="S2.SS2.p3.1.m1.6.6.2.3.cmml"><mfrac id="S2.SS2.p3.1.m1.5.5.1.1" xref="S2.SS2.p3.1.m1.5.5.1.1.cmml"><mi id="S2.SS2.p3.1.m1.5.5.1.1.2" xref="S2.SS2.p3.1.m1.5.5.1.1.2.cmml">N</mi><mi id="S2.SS2.p3.1.m1.5.5.1.1.3" xref="S2.SS2.p3.1.m1.5.5.1.1.3.cmml">G</mi></mfrac><mo id="S2.SS2.p3.1.m1.6.6.2.4.1" xref="S2.SS2.p3.1.m1.6.6.2.3.cmml">,</mo><mfrac id="S2.SS2.p3.1.m1.6.6.2.2" xref="S2.SS2.p3.1.m1.6.6.2.2.cmml"><mi id="S2.SS2.p3.1.m1.6.6.2.2.2" xref="S2.SS2.p3.1.m1.6.6.2.2.2.cmml">N</mi><mi id="S2.SS2.p3.1.m1.6.6.2.2.3" xref="S2.SS2.p3.1.m1.6.6.2.2.3.cmml">G</mi></mfrac></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.1.m1.7b"><apply id="S2.SS2.p3.1.m1.7.7.cmml" xref="S2.SS2.p3.1.m1.7.7"><csymbol cd="ambiguous" id="S2.SS2.p3.1.m1.7.7.2.cmml" xref="S2.SS2.p3.1.m1.7.7">superscript</csymbol><apply id="S2.SS2.p3.1.m1.7.7.1.cmml" xref="S2.SS2.p3.1.m1.7.7"><csymbol cd="ambiguous" id="S2.SS2.p3.1.m1.7.7.1.2.cmml" xref="S2.SS2.p3.1.m1.7.7">subscript</csymbol><set id="S2.SS2.p3.1.m1.7.7.1.1.2.cmml" xref="S2.SS2.p3.1.m1.7.7.1.1.1"><apply id="S2.SS2.p3.1.m1.7.7.1.1.1.1.cmml" xref="S2.SS2.p3.1.m1.7.7.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.1.m1.7.7.1.1.1.1.1.cmml" xref="S2.SS2.p3.1.m1.7.7.1.1.1.1">subscript</csymbol><ci id="S2.SS2.p3.1.m1.7.7.1.1.1.1.2.cmml" xref="S2.SS2.p3.1.m1.7.7.1.1.1.1.2">𝐎</ci><list id="S2.SS2.p3.1.m1.2.2.2.3.cmml" xref="S2.SS2.p3.1.m1.2.2.2.4"><ci id="S2.SS2.p3.1.m1.1.1.1.1.cmml" xref="S2.SS2.p3.1.m1.1.1.1.1">𝑖</ci><ci id="S2.SS2.p3.1.m1.2.2.2.2.cmml" xref="S2.SS2.p3.1.m1.2.2.2.2">𝑗</ci></list></apply></set><apply id="S2.SS2.p3.1.m1.4.4.2.3.cmml" xref="S2.SS2.p3.1.m1.4.4.2.2"><csymbol cd="ambiguous" id="S2.SS2.p3.1.m1.4.4.2.3a.cmml" xref="S2.SS2.p3.1.m1.4.4.2.2.3">formulae-sequence</csymbol><apply id="S2.SS2.p3.1.m1.3.3.1.1.1.cmml" xref="S2.SS2.p3.1.m1.3.3.1.1.1"><eq id="S2.SS2.p3.1.m1.3.3.1.1.1.1.cmml" xref="S2.SS2.p3.1.m1.3.3.1.1.1.1"></eq><ci id="S2.SS2.p3.1.m1.3.3.1.1.1.2.cmml" xref="S2.SS2.p3.1.m1.3.3.1.1.1.2">𝑖</ci><cn type="integer" id="S2.SS2.p3.1.m1.3.3.1.1.1.3.cmml" xref="S2.SS2.p3.1.m1.3.3.1.1.1.3">1</cn></apply><apply id="S2.SS2.p3.1.m1.4.4.2.2.2.cmml" xref="S2.SS2.p3.1.m1.4.4.2.2.2"><eq id="S2.SS2.p3.1.m1.4.4.2.2.2.1.cmml" xref="S2.SS2.p3.1.m1.4.4.2.2.2.1"></eq><ci id="S2.SS2.p3.1.m1.4.4.2.2.2.2.cmml" xref="S2.SS2.p3.1.m1.4.4.2.2.2.2">𝑗</ci><cn type="integer" id="S2.SS2.p3.1.m1.4.4.2.2.2.3.cmml" xref="S2.SS2.p3.1.m1.4.4.2.2.2.3">1</cn></apply></apply></apply><list id="S2.SS2.p3.1.m1.6.6.2.3.cmml" xref="S2.SS2.p3.1.m1.6.6.2.4"><apply id="S2.SS2.p3.1.m1.5.5.1.1.cmml" xref="S2.SS2.p3.1.m1.5.5.1.1"><divide id="S2.SS2.p3.1.m1.5.5.1.1.1.cmml" xref="S2.SS2.p3.1.m1.5.5.1.1"></divide><ci id="S2.SS2.p3.1.m1.5.5.1.1.2.cmml" xref="S2.SS2.p3.1.m1.5.5.1.1.2">𝑁</ci><ci id="S2.SS2.p3.1.m1.5.5.1.1.3.cmml" xref="S2.SS2.p3.1.m1.5.5.1.1.3">𝐺</ci></apply><apply id="S2.SS2.p3.1.m1.6.6.2.2.cmml" xref="S2.SS2.p3.1.m1.6.6.2.2"><divide id="S2.SS2.p3.1.m1.6.6.2.2.1.cmml" xref="S2.SS2.p3.1.m1.6.6.2.2"></divide><ci id="S2.SS2.p3.1.m1.6.6.2.2.2.cmml" xref="S2.SS2.p3.1.m1.6.6.2.2.2">𝑁</ci><ci id="S2.SS2.p3.1.m1.6.6.2.2.3.cmml" xref="S2.SS2.p3.1.m1.6.6.2.2.3">𝐺</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.1.m1.7c">\{\mathbf{O}_{i,j}\}_{i=1,j=1}^{\frac{N}{G},\frac{N}{G}}</annotation></semantics></math> into the final partitioned attention results <math id="S2.SS2.p3.2.m2.1" class="ltx_Math" alttext="\mathbf{O}_{i}\in\mathbb{R}^{\frac{N}{G}\times d}" display="inline"><semantics id="S2.SS2.p3.2.m2.1a"><mrow id="S2.SS2.p3.2.m2.1.1" xref="S2.SS2.p3.2.m2.1.1.cmml"><msub id="S2.SS2.p3.2.m2.1.1.2" xref="S2.SS2.p3.2.m2.1.1.2.cmml"><mi id="S2.SS2.p3.2.m2.1.1.2.2" xref="S2.SS2.p3.2.m2.1.1.2.2.cmml">𝐎</mi><mi id="S2.SS2.p3.2.m2.1.1.2.3" xref="S2.SS2.p3.2.m2.1.1.2.3.cmml">i</mi></msub><mo id="S2.SS2.p3.2.m2.1.1.1" xref="S2.SS2.p3.2.m2.1.1.1.cmml">∈</mo><msup id="S2.SS2.p3.2.m2.1.1.3" xref="S2.SS2.p3.2.m2.1.1.3.cmml"><mi id="S2.SS2.p3.2.m2.1.1.3.2" xref="S2.SS2.p3.2.m2.1.1.3.2.cmml">ℝ</mi><mrow id="S2.SS2.p3.2.m2.1.1.3.3" xref="S2.SS2.p3.2.m2.1.1.3.3.cmml"><mfrac id="S2.SS2.p3.2.m2.1.1.3.3.2" xref="S2.SS2.p3.2.m2.1.1.3.3.2.cmml"><mi id="S2.SS2.p3.2.m2.1.1.3.3.2.2" xref="S2.SS2.p3.2.m2.1.1.3.3.2.2.cmml">N</mi><mi id="S2.SS2.p3.2.m2.1.1.3.3.2.3" xref="S2.SS2.p3.2.m2.1.1.3.3.2.3.cmml">G</mi></mfrac><mo lspace="0.222em" rspace="0.222em" id="S2.SS2.p3.2.m2.1.1.3.3.1" xref="S2.SS2.p3.2.m2.1.1.3.3.1.cmml">×</mo><mi id="S2.SS2.p3.2.m2.1.1.3.3.3" xref="S2.SS2.p3.2.m2.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.2.m2.1b"><apply id="S2.SS2.p3.2.m2.1.1.cmml" xref="S2.SS2.p3.2.m2.1.1"><in id="S2.SS2.p3.2.m2.1.1.1.cmml" xref="S2.SS2.p3.2.m2.1.1.1"></in><apply id="S2.SS2.p3.2.m2.1.1.2.cmml" xref="S2.SS2.p3.2.m2.1.1.2"><csymbol cd="ambiguous" id="S2.SS2.p3.2.m2.1.1.2.1.cmml" xref="S2.SS2.p3.2.m2.1.1.2">subscript</csymbol><ci id="S2.SS2.p3.2.m2.1.1.2.2.cmml" xref="S2.SS2.p3.2.m2.1.1.2.2">𝐎</ci><ci id="S2.SS2.p3.2.m2.1.1.2.3.cmml" xref="S2.SS2.p3.2.m2.1.1.2.3">𝑖</ci></apply><apply id="S2.SS2.p3.2.m2.1.1.3.cmml" xref="S2.SS2.p3.2.m2.1.1.3"><csymbol cd="ambiguous" id="S2.SS2.p3.2.m2.1.1.3.1.cmml" xref="S2.SS2.p3.2.m2.1.1.3">superscript</csymbol><ci id="S2.SS2.p3.2.m2.1.1.3.2.cmml" xref="S2.SS2.p3.2.m2.1.1.3.2">ℝ</ci><apply id="S2.SS2.p3.2.m2.1.1.3.3.cmml" xref="S2.SS2.p3.2.m2.1.1.3.3"><times id="S2.SS2.p3.2.m2.1.1.3.3.1.cmml" xref="S2.SS2.p3.2.m2.1.1.3.3.1"></times><apply id="S2.SS2.p3.2.m2.1.1.3.3.2.cmml" xref="S2.SS2.p3.2.m2.1.1.3.3.2"><divide id="S2.SS2.p3.2.m2.1.1.3.3.2.1.cmml" xref="S2.SS2.p3.2.m2.1.1.3.3.2"></divide><ci id="S2.SS2.p3.2.m2.1.1.3.3.2.2.cmml" xref="S2.SS2.p3.2.m2.1.1.3.3.2.2">𝑁</ci><ci id="S2.SS2.p3.2.m2.1.1.3.3.2.3.cmml" xref="S2.SS2.p3.2.m2.1.1.3.3.2.3">𝐺</ci></apply><ci id="S2.SS2.p3.2.m2.1.1.3.3.3.cmml" xref="S2.SS2.p3.2.m2.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.2.m2.1c">\mathbf{O}_{i}\in\mathbb{R}^{\frac{N}{G}\times d}</annotation></semantics></math>, and <math id="S2.SS2.p3.3.m3.1" class="ltx_Math" alttext="\{\mathbf{O}_{i}\}_{i=1}^{\frac{N}{G}}" display="inline"><semantics id="S2.SS2.p3.3.m3.1a"><msubsup id="S2.SS2.p3.3.m3.1.1" xref="S2.SS2.p3.3.m3.1.1.cmml"><mrow id="S2.SS2.p3.3.m3.1.1.1.1.1" xref="S2.SS2.p3.3.m3.1.1.1.1.2.cmml"><mo stretchy="false" id="S2.SS2.p3.3.m3.1.1.1.1.1.2" xref="S2.SS2.p3.3.m3.1.1.1.1.2.cmml">{</mo><msub id="S2.SS2.p3.3.m3.1.1.1.1.1.1" xref="S2.SS2.p3.3.m3.1.1.1.1.1.1.cmml"><mi id="S2.SS2.p3.3.m3.1.1.1.1.1.1.2" xref="S2.SS2.p3.3.m3.1.1.1.1.1.1.2.cmml">𝐎</mi><mi id="S2.SS2.p3.3.m3.1.1.1.1.1.1.3" xref="S2.SS2.p3.3.m3.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S2.SS2.p3.3.m3.1.1.1.1.1.3" xref="S2.SS2.p3.3.m3.1.1.1.1.2.cmml">}</mo></mrow><mrow id="S2.SS2.p3.3.m3.1.1.1.3" xref="S2.SS2.p3.3.m3.1.1.1.3.cmml"><mi id="S2.SS2.p3.3.m3.1.1.1.3.2" xref="S2.SS2.p3.3.m3.1.1.1.3.2.cmml">i</mi><mo id="S2.SS2.p3.3.m3.1.1.1.3.1" xref="S2.SS2.p3.3.m3.1.1.1.3.1.cmml">=</mo><mn id="S2.SS2.p3.3.m3.1.1.1.3.3" xref="S2.SS2.p3.3.m3.1.1.1.3.3.cmml">1</mn></mrow><mfrac id="S2.SS2.p3.3.m3.1.1.3" xref="S2.SS2.p3.3.m3.1.1.3.cmml"><mi id="S2.SS2.p3.3.m3.1.1.3.2" xref="S2.SS2.p3.3.m3.1.1.3.2.cmml">N</mi><mi id="S2.SS2.p3.3.m3.1.1.3.3" xref="S2.SS2.p3.3.m3.1.1.3.3.cmml">G</mi></mfrac></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.3.m3.1b"><apply id="S2.SS2.p3.3.m3.1.1.cmml" xref="S2.SS2.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.3.m3.1.1.2.cmml" xref="S2.SS2.p3.3.m3.1.1">superscript</csymbol><apply id="S2.SS2.p3.3.m3.1.1.1.cmml" xref="S2.SS2.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.3.m3.1.1.1.2.cmml" xref="S2.SS2.p3.3.m3.1.1">subscript</csymbol><set id="S2.SS2.p3.3.m3.1.1.1.1.2.cmml" xref="S2.SS2.p3.3.m3.1.1.1.1.1"><apply id="S2.SS2.p3.3.m3.1.1.1.1.1.1.cmml" xref="S2.SS2.p3.3.m3.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.3.m3.1.1.1.1.1.1.1.cmml" xref="S2.SS2.p3.3.m3.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS2.p3.3.m3.1.1.1.1.1.1.2.cmml" xref="S2.SS2.p3.3.m3.1.1.1.1.1.1.2">𝐎</ci><ci id="S2.SS2.p3.3.m3.1.1.1.1.1.1.3.cmml" xref="S2.SS2.p3.3.m3.1.1.1.1.1.1.3">𝑖</ci></apply></set><apply id="S2.SS2.p3.3.m3.1.1.1.3.cmml" xref="S2.SS2.p3.3.m3.1.1.1.3"><eq id="S2.SS2.p3.3.m3.1.1.1.3.1.cmml" xref="S2.SS2.p3.3.m3.1.1.1.3.1"></eq><ci id="S2.SS2.p3.3.m3.1.1.1.3.2.cmml" xref="S2.SS2.p3.3.m3.1.1.1.3.2">𝑖</ci><cn type="integer" id="S2.SS2.p3.3.m3.1.1.1.3.3.cmml" xref="S2.SS2.p3.3.m3.1.1.1.3.3">1</cn></apply></apply><apply id="S2.SS2.p3.3.m3.1.1.3.cmml" xref="S2.SS2.p3.3.m3.1.1.3"><divide id="S2.SS2.p3.3.m3.1.1.3.1.cmml" xref="S2.SS2.p3.3.m3.1.1.3"></divide><ci id="S2.SS2.p3.3.m3.1.1.3.2.cmml" xref="S2.SS2.p3.3.m3.1.1.3.2">𝑁</ci><ci id="S2.SS2.p3.3.m3.1.1.3.3.cmml" xref="S2.SS2.p3.3.m3.1.1.3.3">𝐺</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.3.m3.1c">\{\mathbf{O}_{i}\}_{i=1}^{\frac{N}{G}}</annotation></semantics></math> is the final global attention results. To make both the global and local attention operations more efficient, we introduce Global Attention Optimization (GAO) and Local Attention Optimization (LAO), respectively. Next, we will introduce these attention optimization strategies in detail.</p>
</div>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3 </span>Global Attention Optimization (GAO)</h3>

<div id="S2.SS3.p1" class="ltx_para">
<p id="S2.SS3.p1.6" class="ltx_p">Global attention operations are to aggregate <math id="S2.SS3.p1.1.m1.2" class="ltx_Math" alttext="\mathbf{O}_{i,j}" display="inline"><semantics id="S2.SS3.p1.1.m1.2a"><msub id="S2.SS3.p1.1.m1.2.3" xref="S2.SS3.p1.1.m1.2.3.cmml"><mi id="S2.SS3.p1.1.m1.2.3.2" xref="S2.SS3.p1.1.m1.2.3.2.cmml">𝐎</mi><mrow id="S2.SS3.p1.1.m1.2.2.2.4" xref="S2.SS3.p1.1.m1.2.2.2.3.cmml"><mi id="S2.SS3.p1.1.m1.1.1.1.1" xref="S2.SS3.p1.1.m1.1.1.1.1.cmml">i</mi><mo id="S2.SS3.p1.1.m1.2.2.2.4.1" xref="S2.SS3.p1.1.m1.2.2.2.3.cmml">,</mo><mi id="S2.SS3.p1.1.m1.2.2.2.2" xref="S2.SS3.p1.1.m1.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p1.1.m1.2b"><apply id="S2.SS3.p1.1.m1.2.3.cmml" xref="S2.SS3.p1.1.m1.2.3"><csymbol cd="ambiguous" id="S2.SS3.p1.1.m1.2.3.1.cmml" xref="S2.SS3.p1.1.m1.2.3">subscript</csymbol><ci id="S2.SS3.p1.1.m1.2.3.2.cmml" xref="S2.SS3.p1.1.m1.2.3.2">𝐎</ci><list id="S2.SS3.p1.1.m1.2.2.2.3.cmml" xref="S2.SS3.p1.1.m1.2.2.2.4"><ci id="S2.SS3.p1.1.m1.1.1.1.1.cmml" xref="S2.SS3.p1.1.m1.1.1.1.1">𝑖</ci><ci id="S2.SS3.p1.1.m1.2.2.2.2.cmml" xref="S2.SS3.p1.1.m1.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p1.1.m1.2c">\mathbf{O}_{i,j}</annotation></semantics></math> in Eq. (<a href="#S2.E2" title="Equation 2 ‣ 2.2 The Whole Framework of BurstAttention ‣ 2 Methodology ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>) into <math id="S2.SS3.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{O}_{i}" display="inline"><semantics id="S2.SS3.p1.2.m2.1a"><msub id="S2.SS3.p1.2.m2.1.1" xref="S2.SS3.p1.2.m2.1.1.cmml"><mi id="S2.SS3.p1.2.m2.1.1.2" xref="S2.SS3.p1.2.m2.1.1.2.cmml">𝐎</mi><mi id="S2.SS3.p1.2.m2.1.1.3" xref="S2.SS3.p1.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p1.2.m2.1b"><apply id="S2.SS3.p1.2.m2.1.1.cmml" xref="S2.SS3.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS3.p1.2.m2.1.1.1.cmml" xref="S2.SS3.p1.2.m2.1.1">subscript</csymbol><ci id="S2.SS3.p1.2.m2.1.1.2.cmml" xref="S2.SS3.p1.2.m2.1.1.2">𝐎</ci><ci id="S2.SS3.p1.2.m2.1.1.3.cmml" xref="S2.SS3.p1.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p1.2.m2.1c">\mathbf{O}_{i}</annotation></semantics></math>.
For some conventional methods such as RingAttention <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib15" title="" class="ltx_ref">2021</a>)</cite>, for the <math id="S2.SS3.p1.3.m3.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S2.SS3.p1.3.m3.1a"><mi id="S2.SS3.p1.3.m3.1.1" xref="S2.SS3.p1.3.m3.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p1.3.m3.1b"><ci id="S2.SS3.p1.3.m3.1.1.cmml" xref="S2.SS3.p1.3.m3.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p1.3.m3.1c">i</annotation></semantics></math>-th query partition, they store the intermediate results <math id="S2.SS3.p1.4.m4.2" class="ltx_Math" alttext="\mathbf{S}_{i,j}" display="inline"><semantics id="S2.SS3.p1.4.m4.2a"><msub id="S2.SS3.p1.4.m4.2.3" xref="S2.SS3.p1.4.m4.2.3.cmml"><mi id="S2.SS3.p1.4.m4.2.3.2" xref="S2.SS3.p1.4.m4.2.3.2.cmml">𝐒</mi><mrow id="S2.SS3.p1.4.m4.2.2.2.4" xref="S2.SS3.p1.4.m4.2.2.2.3.cmml"><mi id="S2.SS3.p1.4.m4.1.1.1.1" xref="S2.SS3.p1.4.m4.1.1.1.1.cmml">i</mi><mo id="S2.SS3.p1.4.m4.2.2.2.4.1" xref="S2.SS3.p1.4.m4.2.2.2.3.cmml">,</mo><mi id="S2.SS3.p1.4.m4.2.2.2.2" xref="S2.SS3.p1.4.m4.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p1.4.m4.2b"><apply id="S2.SS3.p1.4.m4.2.3.cmml" xref="S2.SS3.p1.4.m4.2.3"><csymbol cd="ambiguous" id="S2.SS3.p1.4.m4.2.3.1.cmml" xref="S2.SS3.p1.4.m4.2.3">subscript</csymbol><ci id="S2.SS3.p1.4.m4.2.3.2.cmml" xref="S2.SS3.p1.4.m4.2.3.2">𝐒</ci><list id="S2.SS3.p1.4.m4.2.2.2.3.cmml" xref="S2.SS3.p1.4.m4.2.2.2.4"><ci id="S2.SS3.p1.4.m4.1.1.1.1.cmml" xref="S2.SS3.p1.4.m4.1.1.1.1">𝑖</ci><ci id="S2.SS3.p1.4.m4.2.2.2.2.cmml" xref="S2.SS3.p1.4.m4.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p1.4.m4.2c">\mathbf{S}_{i,j}</annotation></semantics></math> and <math id="S2.SS3.p1.5.m5.2" class="ltx_Math" alttext="\mathbf{P}_{i,j}" display="inline"><semantics id="S2.SS3.p1.5.m5.2a"><msub id="S2.SS3.p1.5.m5.2.3" xref="S2.SS3.p1.5.m5.2.3.cmml"><mi id="S2.SS3.p1.5.m5.2.3.2" xref="S2.SS3.p1.5.m5.2.3.2.cmml">𝐏</mi><mrow id="S2.SS3.p1.5.m5.2.2.2.4" xref="S2.SS3.p1.5.m5.2.2.2.3.cmml"><mi id="S2.SS3.p1.5.m5.1.1.1.1" xref="S2.SS3.p1.5.m5.1.1.1.1.cmml">i</mi><mo id="S2.SS3.p1.5.m5.2.2.2.4.1" xref="S2.SS3.p1.5.m5.2.2.2.3.cmml">,</mo><mi id="S2.SS3.p1.5.m5.2.2.2.2" xref="S2.SS3.p1.5.m5.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p1.5.m5.2b"><apply id="S2.SS3.p1.5.m5.2.3.cmml" xref="S2.SS3.p1.5.m5.2.3"><csymbol cd="ambiguous" id="S2.SS3.p1.5.m5.2.3.1.cmml" xref="S2.SS3.p1.5.m5.2.3">subscript</csymbol><ci id="S2.SS3.p1.5.m5.2.3.2.cmml" xref="S2.SS3.p1.5.m5.2.3.2">𝐏</ci><list id="S2.SS3.p1.5.m5.2.2.2.3.cmml" xref="S2.SS3.p1.5.m5.2.2.2.4"><ci id="S2.SS3.p1.5.m5.1.1.1.1.cmml" xref="S2.SS3.p1.5.m5.1.1.1.1">𝑖</ci><ci id="S2.SS3.p1.5.m5.2.2.2.2.cmml" xref="S2.SS3.p1.5.m5.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p1.5.m5.2c">\mathbf{P}_{i,j}</annotation></semantics></math> for every <math id="S2.SS3.p1.6.m6.1" class="ltx_Math" alttext="j" display="inline"><semantics id="S2.SS3.p1.6.m6.1a"><mi id="S2.SS3.p1.6.m6.1.1" xref="S2.SS3.p1.6.m6.1.1.cmml">j</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p1.6.m6.1b"><ci id="S2.SS3.p1.6.m6.1.1.cmml" xref="S2.SS3.p1.6.m6.1.1">𝑗</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p1.6.m6.1c">j</annotation></semantics></math>. This introduces a non-negligible memory overhead.
To get rid of this memory overhead, we introduce GAO.</p>
</div>
<div id="S2.SS3.p2" class="ltx_para">
<p id="S2.SS3.p2.10" class="ltx_p">As shown in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, GAO consists of two main steps. First, devices are organized in a ring for communication. Each round, <math id="S2.SS3.p2.1.m1.2" class="ltx_Math" alttext="\mathbf{K},\mathbf{V}" display="inline"><semantics id="S2.SS3.p2.1.m1.2a"><mrow id="S2.SS3.p2.1.m1.2.3.2" xref="S2.SS3.p2.1.m1.2.3.1.cmml"><mi id="S2.SS3.p2.1.m1.1.1" xref="S2.SS3.p2.1.m1.1.1.cmml">𝐊</mi><mo id="S2.SS3.p2.1.m1.2.3.2.1" xref="S2.SS3.p2.1.m1.2.3.1.cmml">,</mo><mi id="S2.SS3.p2.1.m1.2.2" xref="S2.SS3.p2.1.m1.2.2.cmml">𝐕</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.1.m1.2b"><list id="S2.SS3.p2.1.m1.2.3.1.cmml" xref="S2.SS3.p2.1.m1.2.3.2"><ci id="S2.SS3.p2.1.m1.1.1.cmml" xref="S2.SS3.p2.1.m1.1.1">𝐊</ci><ci id="S2.SS3.p2.1.m1.2.2.cmml" xref="S2.SS3.p2.1.m1.2.2">𝐕</ci></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.1.m1.2c">\mathbf{K},\mathbf{V}</annotation></semantics></math> partitions are shifted along the ring to the next adjacent device. Second, after each round of <math id="S2.SS3.p2.2.m2.2" class="ltx_Math" alttext="\mathbf{K},\mathbf{V}" display="inline"><semantics id="S2.SS3.p2.2.m2.2a"><mrow id="S2.SS3.p2.2.m2.2.3.2" xref="S2.SS3.p2.2.m2.2.3.1.cmml"><mi id="S2.SS3.p2.2.m2.1.1" xref="S2.SS3.p2.2.m2.1.1.cmml">𝐊</mi><mo id="S2.SS3.p2.2.m2.2.3.2.1" xref="S2.SS3.p2.2.m2.2.3.1.cmml">,</mo><mi id="S2.SS3.p2.2.m2.2.2" xref="S2.SS3.p2.2.m2.2.2.cmml">𝐕</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.2.m2.2b"><list id="S2.SS3.p2.2.m2.2.3.1.cmml" xref="S2.SS3.p2.2.m2.2.3.2"><ci id="S2.SS3.p2.2.m2.1.1.cmml" xref="S2.SS3.p2.2.m2.1.1">𝐊</ci><ci id="S2.SS3.p2.2.m2.2.2.cmml" xref="S2.SS3.p2.2.m2.2.2">𝐕</ci></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.2.m2.2c">\mathbf{K},\mathbf{V}</annotation></semantics></math> transmission, each device <math id="S2.SS3.p2.3.m3.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S2.SS3.p2.3.m3.1a"><mi id="S2.SS3.p2.3.m3.1.1" xref="S2.SS3.p2.3.m3.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.3.m3.1b"><ci id="S2.SS3.p2.3.m3.1.1.cmml" xref="S2.SS3.p2.3.m3.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.3.m3.1c">i</annotation></semantics></math> performs a local attention operation using the partitions <math id="S2.SS3.p2.4.m4.1" class="ltx_Math" alttext="\mathbf{Q}_{i}" display="inline"><semantics id="S2.SS3.p2.4.m4.1a"><msub id="S2.SS3.p2.4.m4.1.1" xref="S2.SS3.p2.4.m4.1.1.cmml"><mi id="S2.SS3.p2.4.m4.1.1.2" xref="S2.SS3.p2.4.m4.1.1.2.cmml">𝐐</mi><mi id="S2.SS3.p2.4.m4.1.1.3" xref="S2.SS3.p2.4.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.4.m4.1b"><apply id="S2.SS3.p2.4.m4.1.1.cmml" xref="S2.SS3.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.4.m4.1.1.1.cmml" xref="S2.SS3.p2.4.m4.1.1">subscript</csymbol><ci id="S2.SS3.p2.4.m4.1.1.2.cmml" xref="S2.SS3.p2.4.m4.1.1.2">𝐐</ci><ci id="S2.SS3.p2.4.m4.1.1.3.cmml" xref="S2.SS3.p2.4.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.4.m4.1c">\mathbf{Q}_{i}</annotation></semantics></math> and its received partition <math id="S2.SS3.p2.5.m5.1" class="ltx_Math" alttext="\mathbf{K}_{j}" display="inline"><semantics id="S2.SS3.p2.5.m5.1a"><msub id="S2.SS3.p2.5.m5.1.1" xref="S2.SS3.p2.5.m5.1.1.cmml"><mi id="S2.SS3.p2.5.m5.1.1.2" xref="S2.SS3.p2.5.m5.1.1.2.cmml">𝐊</mi><mi id="S2.SS3.p2.5.m5.1.1.3" xref="S2.SS3.p2.5.m5.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.5.m5.1b"><apply id="S2.SS3.p2.5.m5.1.1.cmml" xref="S2.SS3.p2.5.m5.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.5.m5.1.1.1.cmml" xref="S2.SS3.p2.5.m5.1.1">subscript</csymbol><ci id="S2.SS3.p2.5.m5.1.1.2.cmml" xref="S2.SS3.p2.5.m5.1.1.2">𝐊</ci><ci id="S2.SS3.p2.5.m5.1.1.3.cmml" xref="S2.SS3.p2.5.m5.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.5.m5.1c">\mathbf{K}_{j}</annotation></semantics></math>, and <math id="S2.SS3.p2.6.m6.1" class="ltx_Math" alttext="\mathbf{V}_{j}" display="inline"><semantics id="S2.SS3.p2.6.m6.1a"><msub id="S2.SS3.p2.6.m6.1.1" xref="S2.SS3.p2.6.m6.1.1.cmml"><mi id="S2.SS3.p2.6.m6.1.1.2" xref="S2.SS3.p2.6.m6.1.1.2.cmml">𝐕</mi><mi id="S2.SS3.p2.6.m6.1.1.3" xref="S2.SS3.p2.6.m6.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.6.m6.1b"><apply id="S2.SS3.p2.6.m6.1.1.cmml" xref="S2.SS3.p2.6.m6.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.6.m6.1.1.1.cmml" xref="S2.SS3.p2.6.m6.1.1">subscript</csymbol><ci id="S2.SS3.p2.6.m6.1.1.2.cmml" xref="S2.SS3.p2.6.m6.1.1.2">𝐕</ci><ci id="S2.SS3.p2.6.m6.1.1.3.cmml" xref="S2.SS3.p2.6.m6.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.6.m6.1c">\mathbf{V}_{j}</annotation></semantics></math>, as described in Eq. (<a href="#S2.E2" title="Equation 2 ‣ 2.2 The Whole Framework of BurstAttention ‣ 2 Methodology ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>). The local attention result <math id="S2.SS3.p2.7.m7.2" class="ltx_Math" alttext="\mathbf{O}_{i,j}" display="inline"><semantics id="S2.SS3.p2.7.m7.2a"><msub id="S2.SS3.p2.7.m7.2.3" xref="S2.SS3.p2.7.m7.2.3.cmml"><mi id="S2.SS3.p2.7.m7.2.3.2" xref="S2.SS3.p2.7.m7.2.3.2.cmml">𝐎</mi><mrow id="S2.SS3.p2.7.m7.2.2.2.4" xref="S2.SS3.p2.7.m7.2.2.2.3.cmml"><mi id="S2.SS3.p2.7.m7.1.1.1.1" xref="S2.SS3.p2.7.m7.1.1.1.1.cmml">i</mi><mo id="S2.SS3.p2.7.m7.2.2.2.4.1" xref="S2.SS3.p2.7.m7.2.2.2.3.cmml">,</mo><mi id="S2.SS3.p2.7.m7.2.2.2.2" xref="S2.SS3.p2.7.m7.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.7.m7.2b"><apply id="S2.SS3.p2.7.m7.2.3.cmml" xref="S2.SS3.p2.7.m7.2.3"><csymbol cd="ambiguous" id="S2.SS3.p2.7.m7.2.3.1.cmml" xref="S2.SS3.p2.7.m7.2.3">subscript</csymbol><ci id="S2.SS3.p2.7.m7.2.3.2.cmml" xref="S2.SS3.p2.7.m7.2.3.2">𝐎</ci><list id="S2.SS3.p2.7.m7.2.2.2.3.cmml" xref="S2.SS3.p2.7.m7.2.2.2.4"><ci id="S2.SS3.p2.7.m7.1.1.1.1.cmml" xref="S2.SS3.p2.7.m7.1.1.1.1">𝑖</ci><ci id="S2.SS3.p2.7.m7.2.2.2.2.cmml" xref="S2.SS3.p2.7.m7.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.7.m7.2c">\mathbf{O}_{i,j}</annotation></semantics></math> are then dynamically accumulated into global attention result <math id="S2.SS3.p2.8.m8.1" class="ltx_Math" alttext="\mathbf{O}_{i}" display="inline"><semantics id="S2.SS3.p2.8.m8.1a"><msub id="S2.SS3.p2.8.m8.1.1" xref="S2.SS3.p2.8.m8.1.1.cmml"><mi id="S2.SS3.p2.8.m8.1.1.2" xref="S2.SS3.p2.8.m8.1.1.2.cmml">𝐎</mi><mi id="S2.SS3.p2.8.m8.1.1.3" xref="S2.SS3.p2.8.m8.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.8.m8.1b"><apply id="S2.SS3.p2.8.m8.1.1.cmml" xref="S2.SS3.p2.8.m8.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.8.m8.1.1.1.cmml" xref="S2.SS3.p2.8.m8.1.1">subscript</csymbol><ci id="S2.SS3.p2.8.m8.1.1.2.cmml" xref="S2.SS3.p2.8.m8.1.1.2">𝐎</ci><ci id="S2.SS3.p2.8.m8.1.1.3.cmml" xref="S2.SS3.p2.8.m8.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.8.m8.1c">\mathbf{O}_{i}</annotation></semantics></math> by employing online softmax <cite class="ltx_cite ltx_citemacro_citep">(Milakov &amp; Gimelshein, <a href="#bib.bib16" title="" class="ltx_ref">2018</a>)</cite>, which eliminates the need to store intermediate results <math id="S2.SS3.p2.9.m9.2" class="ltx_Math" alttext="\mathbf{S}_{i,j}" display="inline"><semantics id="S2.SS3.p2.9.m9.2a"><msub id="S2.SS3.p2.9.m9.2.3" xref="S2.SS3.p2.9.m9.2.3.cmml"><mi id="S2.SS3.p2.9.m9.2.3.2" xref="S2.SS3.p2.9.m9.2.3.2.cmml">𝐒</mi><mrow id="S2.SS3.p2.9.m9.2.2.2.4" xref="S2.SS3.p2.9.m9.2.2.2.3.cmml"><mi id="S2.SS3.p2.9.m9.1.1.1.1" xref="S2.SS3.p2.9.m9.1.1.1.1.cmml">i</mi><mo id="S2.SS3.p2.9.m9.2.2.2.4.1" xref="S2.SS3.p2.9.m9.2.2.2.3.cmml">,</mo><mi id="S2.SS3.p2.9.m9.2.2.2.2" xref="S2.SS3.p2.9.m9.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.9.m9.2b"><apply id="S2.SS3.p2.9.m9.2.3.cmml" xref="S2.SS3.p2.9.m9.2.3"><csymbol cd="ambiguous" id="S2.SS3.p2.9.m9.2.3.1.cmml" xref="S2.SS3.p2.9.m9.2.3">subscript</csymbol><ci id="S2.SS3.p2.9.m9.2.3.2.cmml" xref="S2.SS3.p2.9.m9.2.3.2">𝐒</ci><list id="S2.SS3.p2.9.m9.2.2.2.3.cmml" xref="S2.SS3.p2.9.m9.2.2.2.4"><ci id="S2.SS3.p2.9.m9.1.1.1.1.cmml" xref="S2.SS3.p2.9.m9.1.1.1.1">𝑖</ci><ci id="S2.SS3.p2.9.m9.2.2.2.2.cmml" xref="S2.SS3.p2.9.m9.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.9.m9.2c">\mathbf{S}_{i,j}</annotation></semantics></math> and <math id="S2.SS3.p2.10.m10.2" class="ltx_Math" alttext="\mathbf{P}_{i,j}" display="inline"><semantics id="S2.SS3.p2.10.m10.2a"><msub id="S2.SS3.p2.10.m10.2.3" xref="S2.SS3.p2.10.m10.2.3.cmml"><mi id="S2.SS3.p2.10.m10.2.3.2" xref="S2.SS3.p2.10.m10.2.3.2.cmml">𝐏</mi><mrow id="S2.SS3.p2.10.m10.2.2.2.4" xref="S2.SS3.p2.10.m10.2.2.2.3.cmml"><mi id="S2.SS3.p2.10.m10.1.1.1.1" xref="S2.SS3.p2.10.m10.1.1.1.1.cmml">i</mi><mo id="S2.SS3.p2.10.m10.2.2.2.4.1" xref="S2.SS3.p2.10.m10.2.2.2.3.cmml">,</mo><mi id="S2.SS3.p2.10.m10.2.2.2.2" xref="S2.SS3.p2.10.m10.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.10.m10.2b"><apply id="S2.SS3.p2.10.m10.2.3.cmml" xref="S2.SS3.p2.10.m10.2.3"><csymbol cd="ambiguous" id="S2.SS3.p2.10.m10.2.3.1.cmml" xref="S2.SS3.p2.10.m10.2.3">subscript</csymbol><ci id="S2.SS3.p2.10.m10.2.3.2.cmml" xref="S2.SS3.p2.10.m10.2.3.2">𝐏</ci><list id="S2.SS3.p2.10.m10.2.2.2.3.cmml" xref="S2.SS3.p2.10.m10.2.2.2.4"><ci id="S2.SS3.p2.10.m10.1.1.1.1.cmml" xref="S2.SS3.p2.10.m10.1.1.1.1">𝑖</ci><ci id="S2.SS3.p2.10.m10.2.2.2.2.cmml" xref="S2.SS3.p2.10.m10.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.10.m10.2c">\mathbf{P}_{i,j}</annotation></semantics></math>.</p>
</div>
<div id="S2.SS3.p3" class="ltx_para">
<p id="S2.SS3.p3.11" class="ltx_p">As depicted in Algorithm <a href="#alg1" title="Algorithm 1 ‣ 2.1 Preliminary ‣ 2 Methodology ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, in the forward pass, we dynamically maintain the row-wise maximum value <math id="S2.SS3.p3.1.m1.1" class="ltx_Math" alttext="m_{i}" display="inline"><semantics id="S2.SS3.p3.1.m1.1a"><msub id="S2.SS3.p3.1.m1.1.1" xref="S2.SS3.p3.1.m1.1.1.cmml"><mi id="S2.SS3.p3.1.m1.1.1.2" xref="S2.SS3.p3.1.m1.1.1.2.cmml">m</mi><mi id="S2.SS3.p3.1.m1.1.1.3" xref="S2.SS3.p3.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.1.m1.1b"><apply id="S2.SS3.p3.1.m1.1.1.cmml" xref="S2.SS3.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p3.1.m1.1.1.1.cmml" xref="S2.SS3.p3.1.m1.1.1">subscript</csymbol><ci id="S2.SS3.p3.1.m1.1.1.2.cmml" xref="S2.SS3.p3.1.m1.1.1.2">𝑚</ci><ci id="S2.SS3.p3.1.m1.1.1.3.cmml" xref="S2.SS3.p3.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.1.m1.1c">m_{i}</annotation></semantics></math> of <math id="S2.SS3.p3.2.m2.2" class="ltx_Math" alttext="\mathbf{S_{i,j}}" display="inline"><semantics id="S2.SS3.p3.2.m2.2a"><msub id="S2.SS3.p3.2.m2.2.3" xref="S2.SS3.p3.2.m2.2.3.cmml"><mi id="S2.SS3.p3.2.m2.2.3.2" xref="S2.SS3.p3.2.m2.2.3.2.cmml">𝐒</mi><mrow id="S2.SS3.p3.2.m2.2.2.2.4" xref="S2.SS3.p3.2.m2.2.2.2.3.cmml"><mi id="S2.SS3.p3.2.m2.1.1.1.1" xref="S2.SS3.p3.2.m2.1.1.1.1.cmml">𝐢</mi><mo id="S2.SS3.p3.2.m2.2.2.2.4.1" xref="S2.SS3.p3.2.m2.2.2.2.3.cmml">,</mo><mi id="S2.SS3.p3.2.m2.2.2.2.2" xref="S2.SS3.p3.2.m2.2.2.2.2.cmml">𝐣</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.2.m2.2b"><apply id="S2.SS3.p3.2.m2.2.3.cmml" xref="S2.SS3.p3.2.m2.2.3"><csymbol cd="ambiguous" id="S2.SS3.p3.2.m2.2.3.1.cmml" xref="S2.SS3.p3.2.m2.2.3">subscript</csymbol><ci id="S2.SS3.p3.2.m2.2.3.2.cmml" xref="S2.SS3.p3.2.m2.2.3.2">𝐒</ci><list id="S2.SS3.p3.2.m2.2.2.2.3.cmml" xref="S2.SS3.p3.2.m2.2.2.2.4"><ci id="S2.SS3.p3.2.m2.1.1.1.1.cmml" xref="S2.SS3.p3.2.m2.1.1.1.1">𝐢</ci><ci id="S2.SS3.p3.2.m2.2.2.2.2.cmml" xref="S2.SS3.p3.2.m2.2.2.2.2">𝐣</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.2.m2.2c">\mathbf{S_{i,j}}</annotation></semantics></math> as in Line <a href="#alg1.l13" title="In Algorithm 1 ‣ 2.1 Preliminary ‣ 2 Methodology ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13</span></a> and the row-wise sum <math id="S2.SS3.p3.3.m3.1" class="ltx_Math" alttext="l" display="inline"><semantics id="S2.SS3.p3.3.m3.1a"><mi id="S2.SS3.p3.3.m3.1.1" xref="S2.SS3.p3.3.m3.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.3.m3.1b"><ci id="S2.SS3.p3.3.m3.1.1.cmml" xref="S2.SS3.p3.3.m3.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.3.m3.1c">l</annotation></semantics></math> of <math id="S2.SS3.p3.4.m4.2" class="ltx_Math" alttext="\mathbf{P_{i,j}}" display="inline"><semantics id="S2.SS3.p3.4.m4.2a"><msub id="S2.SS3.p3.4.m4.2.3" xref="S2.SS3.p3.4.m4.2.3.cmml"><mi id="S2.SS3.p3.4.m4.2.3.2" xref="S2.SS3.p3.4.m4.2.3.2.cmml">𝐏</mi><mrow id="S2.SS3.p3.4.m4.2.2.2.4" xref="S2.SS3.p3.4.m4.2.2.2.3.cmml"><mi id="S2.SS3.p3.4.m4.1.1.1.1" xref="S2.SS3.p3.4.m4.1.1.1.1.cmml">𝐢</mi><mo id="S2.SS3.p3.4.m4.2.2.2.4.1" xref="S2.SS3.p3.4.m4.2.2.2.3.cmml">,</mo><mi id="S2.SS3.p3.4.m4.2.2.2.2" xref="S2.SS3.p3.4.m4.2.2.2.2.cmml">𝐣</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.4.m4.2b"><apply id="S2.SS3.p3.4.m4.2.3.cmml" xref="S2.SS3.p3.4.m4.2.3"><csymbol cd="ambiguous" id="S2.SS3.p3.4.m4.2.3.1.cmml" xref="S2.SS3.p3.4.m4.2.3">subscript</csymbol><ci id="S2.SS3.p3.4.m4.2.3.2.cmml" xref="S2.SS3.p3.4.m4.2.3.2">𝐏</ci><list id="S2.SS3.p3.4.m4.2.2.2.3.cmml" xref="S2.SS3.p3.4.m4.2.2.2.4"><ci id="S2.SS3.p3.4.m4.1.1.1.1.cmml" xref="S2.SS3.p3.4.m4.1.1.1.1">𝐢</ci><ci id="S2.SS3.p3.4.m4.2.2.2.2.cmml" xref="S2.SS3.p3.4.m4.2.2.2.2">𝐣</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.4.m4.2c">\mathbf{P_{i,j}}</annotation></semantics></math> as in Line <a href="#alg1.l14" title="In Algorithm 1 ‣ 2.1 Preliminary ‣ 2 Methodology ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">14</span></a> to avoid storing <math id="S2.SS3.p3.5.m5.1" class="ltx_Math" alttext="\mathbf{S}" display="inline"><semantics id="S2.SS3.p3.5.m5.1a"><mi id="S2.SS3.p3.5.m5.1.1" xref="S2.SS3.p3.5.m5.1.1.cmml">𝐒</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.5.m5.1b"><ci id="S2.SS3.p3.5.m5.1.1.cmml" xref="S2.SS3.p3.5.m5.1.1">𝐒</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.5.m5.1c">\mathbf{S}</annotation></semantics></math> and <math id="S2.SS3.p3.6.m6.1" class="ltx_Math" alttext="\mathbf{P}" display="inline"><semantics id="S2.SS3.p3.6.m6.1a"><mi id="S2.SS3.p3.6.m6.1.1" xref="S2.SS3.p3.6.m6.1.1.cmml">𝐏</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.6.m6.1b"><ci id="S2.SS3.p3.6.m6.1.1.cmml" xref="S2.SS3.p3.6.m6.1.1">𝐏</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.6.m6.1c">\mathbf{P}</annotation></semantics></math>, and use <math id="S2.SS3.p3.7.m7.1" class="ltx_Math" alttext="m_{i}" display="inline"><semantics id="S2.SS3.p3.7.m7.1a"><msub id="S2.SS3.p3.7.m7.1.1" xref="S2.SS3.p3.7.m7.1.1.cmml"><mi id="S2.SS3.p3.7.m7.1.1.2" xref="S2.SS3.p3.7.m7.1.1.2.cmml">m</mi><mi id="S2.SS3.p3.7.m7.1.1.3" xref="S2.SS3.p3.7.m7.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.7.m7.1b"><apply id="S2.SS3.p3.7.m7.1.1.cmml" xref="S2.SS3.p3.7.m7.1.1"><csymbol cd="ambiguous" id="S2.SS3.p3.7.m7.1.1.1.cmml" xref="S2.SS3.p3.7.m7.1.1">subscript</csymbol><ci id="S2.SS3.p3.7.m7.1.1.2.cmml" xref="S2.SS3.p3.7.m7.1.1.2">𝑚</ci><ci id="S2.SS3.p3.7.m7.1.1.3.cmml" xref="S2.SS3.p3.7.m7.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.7.m7.1c">m_{i}</annotation></semantics></math> and <math id="S2.SS3.p3.8.m8.1" class="ltx_Math" alttext="l_{i}" display="inline"><semantics id="S2.SS3.p3.8.m8.1a"><msub id="S2.SS3.p3.8.m8.1.1" xref="S2.SS3.p3.8.m8.1.1.cmml"><mi id="S2.SS3.p3.8.m8.1.1.2" xref="S2.SS3.p3.8.m8.1.1.2.cmml">l</mi><mi id="S2.SS3.p3.8.m8.1.1.3" xref="S2.SS3.p3.8.m8.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.8.m8.1b"><apply id="S2.SS3.p3.8.m8.1.1.cmml" xref="S2.SS3.p3.8.m8.1.1"><csymbol cd="ambiguous" id="S2.SS3.p3.8.m8.1.1.1.cmml" xref="S2.SS3.p3.8.m8.1.1">subscript</csymbol><ci id="S2.SS3.p3.8.m8.1.1.2.cmml" xref="S2.SS3.p3.8.m8.1.1.2">𝑙</ci><ci id="S2.SS3.p3.8.m8.1.1.3.cmml" xref="S2.SS3.p3.8.m8.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.8.m8.1c">l_{i}</annotation></semantics></math> for scaling during the aggregation of <math id="S2.SS3.p3.9.m9.1" class="ltx_Math" alttext="\mathbf{O}_{i}" display="inline"><semantics id="S2.SS3.p3.9.m9.1a"><msub id="S2.SS3.p3.9.m9.1.1" xref="S2.SS3.p3.9.m9.1.1.cmml"><mi id="S2.SS3.p3.9.m9.1.1.2" xref="S2.SS3.p3.9.m9.1.1.2.cmml">𝐎</mi><mi id="S2.SS3.p3.9.m9.1.1.3" xref="S2.SS3.p3.9.m9.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.9.m9.1b"><apply id="S2.SS3.p3.9.m9.1.1.cmml" xref="S2.SS3.p3.9.m9.1.1"><csymbol cd="ambiguous" id="S2.SS3.p3.9.m9.1.1.1.cmml" xref="S2.SS3.p3.9.m9.1.1">subscript</csymbol><ci id="S2.SS3.p3.9.m9.1.1.2.cmml" xref="S2.SS3.p3.9.m9.1.1.2">𝐎</ci><ci id="S2.SS3.p3.9.m9.1.1.3.cmml" xref="S2.SS3.p3.9.m9.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.9.m9.1c">\mathbf{O}_{i}</annotation></semantics></math> as in Line <a href="#alg1.l15" title="In Algorithm 1 ‣ 2.1 Preliminary ‣ 2 Methodology ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">15</span></a>. Note that, the functions <math id="S2.SS3.p3.10.m10.1" class="ltx_Math" alttext="\text{rowmax}(\cdot)" display="inline"><semantics id="S2.SS3.p3.10.m10.1a"><mrow id="S2.SS3.p3.10.m10.1.2" xref="S2.SS3.p3.10.m10.1.2.cmml"><mtext id="S2.SS3.p3.10.m10.1.2.2" xref="S2.SS3.p3.10.m10.1.2.2a.cmml">rowmax</mtext><mo lspace="0em" rspace="0em" id="S2.SS3.p3.10.m10.1.2.1" xref="S2.SS3.p3.10.m10.1.2.1.cmml">​</mo><mrow id="S2.SS3.p3.10.m10.1.2.3.2" xref="S2.SS3.p3.10.m10.1.2.cmml"><mo stretchy="false" id="S2.SS3.p3.10.m10.1.2.3.2.1" xref="S2.SS3.p3.10.m10.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S2.SS3.p3.10.m10.1.1" xref="S2.SS3.p3.10.m10.1.1.cmml">⋅</mo><mo stretchy="false" id="S2.SS3.p3.10.m10.1.2.3.2.2" xref="S2.SS3.p3.10.m10.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.10.m10.1b"><apply id="S2.SS3.p3.10.m10.1.2.cmml" xref="S2.SS3.p3.10.m10.1.2"><times id="S2.SS3.p3.10.m10.1.2.1.cmml" xref="S2.SS3.p3.10.m10.1.2.1"></times><ci id="S2.SS3.p3.10.m10.1.2.2a.cmml" xref="S2.SS3.p3.10.m10.1.2.2"><mtext id="S2.SS3.p3.10.m10.1.2.2.cmml" xref="S2.SS3.p3.10.m10.1.2.2">rowmax</mtext></ci><ci id="S2.SS3.p3.10.m10.1.1.cmml" xref="S2.SS3.p3.10.m10.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.10.m10.1c">\text{rowmax}(\cdot)</annotation></semantics></math> and <math id="S2.SS3.p3.11.m11.1" class="ltx_Math" alttext="\text{rowsum}(\cdot)" display="inline"><semantics id="S2.SS3.p3.11.m11.1a"><mrow id="S2.SS3.p3.11.m11.1.2" xref="S2.SS3.p3.11.m11.1.2.cmml"><mtext id="S2.SS3.p3.11.m11.1.2.2" xref="S2.SS3.p3.11.m11.1.2.2a.cmml">rowsum</mtext><mo lspace="0em" rspace="0em" id="S2.SS3.p3.11.m11.1.2.1" xref="S2.SS3.p3.11.m11.1.2.1.cmml">​</mo><mrow id="S2.SS3.p3.11.m11.1.2.3.2" xref="S2.SS3.p3.11.m11.1.2.cmml"><mo stretchy="false" id="S2.SS3.p3.11.m11.1.2.3.2.1" xref="S2.SS3.p3.11.m11.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S2.SS3.p3.11.m11.1.1" xref="S2.SS3.p3.11.m11.1.1.cmml">⋅</mo><mo stretchy="false" id="S2.SS3.p3.11.m11.1.2.3.2.2" xref="S2.SS3.p3.11.m11.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.11.m11.1b"><apply id="S2.SS3.p3.11.m11.1.2.cmml" xref="S2.SS3.p3.11.m11.1.2"><times id="S2.SS3.p3.11.m11.1.2.1.cmml" xref="S2.SS3.p3.11.m11.1.2.1"></times><ci id="S2.SS3.p3.11.m11.1.2.2a.cmml" xref="S2.SS3.p3.11.m11.1.2.2"><mtext id="S2.SS3.p3.11.m11.1.2.2.cmml" xref="S2.SS3.p3.11.m11.1.2.2">rowsum</mtext></ci><ci id="S2.SS3.p3.11.m11.1.1.cmml" xref="S2.SS3.p3.11.m11.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.11.m11.1c">\text{rowsum}(\cdot)</annotation></semantics></math> can be formalized as</p>
<table id="S2.E3" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S2.E3X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><sub id="S2.E3X.2.1.1.1" class="ltx_sub"><span id="S2.E3X.2.1.1.1.1" class="ltx_text ltx_font_italic">i</span></sub></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.E3X.3.2.2.m1.4" class="ltx_Math" alttext="\displaystyle=\max_{j}\{[\mathbf{W}]_{i,j}\}," display="inline"><semantics id="S2.E3X.3.2.2.m1.4a"><mrow id="S2.E3X.3.2.2.m1.4.4.1" xref="S2.E3X.3.2.2.m1.4.4.1.1.cmml"><mrow id="S2.E3X.3.2.2.m1.4.4.1.1" xref="S2.E3X.3.2.2.m1.4.4.1.1.cmml"><mi id="S2.E3X.3.2.2.m1.4.4.1.1.4" xref="S2.E3X.3.2.2.m1.4.4.1.1.4.cmml"></mi><mo mathsize="90%" id="S2.E3X.3.2.2.m1.4.4.1.1.3" xref="S2.E3X.3.2.2.m1.4.4.1.1.3.cmml">=</mo><mrow id="S2.E3X.3.2.2.m1.4.4.1.1.2.2" xref="S2.E3X.3.2.2.m1.4.4.1.1.2.3.cmml"><munder id="S2.E3X.3.2.2.m1.4.4.1.1.1.1.1" xref="S2.E3X.3.2.2.m1.4.4.1.1.1.1.1.cmml"><mi mathsize="90%" id="S2.E3X.3.2.2.m1.4.4.1.1.1.1.1.2" xref="S2.E3X.3.2.2.m1.4.4.1.1.1.1.1.2.cmml">max</mi><mi mathsize="90%" id="S2.E3X.3.2.2.m1.4.4.1.1.1.1.1.3" xref="S2.E3X.3.2.2.m1.4.4.1.1.1.1.1.3.cmml">j</mi></munder><mo id="S2.E3X.3.2.2.m1.4.4.1.1.2.2a" xref="S2.E3X.3.2.2.m1.4.4.1.1.2.3.cmml">⁡</mo><mrow id="S2.E3X.3.2.2.m1.4.4.1.1.2.2.2" xref="S2.E3X.3.2.2.m1.4.4.1.1.2.3.cmml"><mo maxsize="90%" minsize="90%" id="S2.E3X.3.2.2.m1.4.4.1.1.2.2.2.2" xref="S2.E3X.3.2.2.m1.4.4.1.1.2.3.cmml">{</mo><msub id="S2.E3X.3.2.2.m1.4.4.1.1.2.2.2.1" xref="S2.E3X.3.2.2.m1.4.4.1.1.2.2.2.1.cmml"><mrow id="S2.E3X.3.2.2.m1.4.4.1.1.2.2.2.1.2.2" xref="S2.E3X.3.2.2.m1.4.4.1.1.2.2.2.1.2.1.cmml"><mo maxsize="90%" minsize="90%" id="S2.E3X.3.2.2.m1.4.4.1.1.2.2.2.1.2.2.1" xref="S2.E3X.3.2.2.m1.4.4.1.1.2.2.2.1.2.1.1.cmml">[</mo><mi mathsize="90%" id="S2.E3X.3.2.2.m1.3.3" xref="S2.E3X.3.2.2.m1.3.3.cmml">𝐖</mi><mo maxsize="90%" minsize="90%" id="S2.E3X.3.2.2.m1.4.4.1.1.2.2.2.1.2.2.2" xref="S2.E3X.3.2.2.m1.4.4.1.1.2.2.2.1.2.1.1.cmml">]</mo></mrow><mrow id="S2.E3X.3.2.2.m1.2.2.2.4" xref="S2.E3X.3.2.2.m1.2.2.2.3.cmml"><mi mathsize="90%" id="S2.E3X.3.2.2.m1.1.1.1.1" xref="S2.E3X.3.2.2.m1.1.1.1.1.cmml">i</mi><mo mathsize="90%" id="S2.E3X.3.2.2.m1.2.2.2.4.1" xref="S2.E3X.3.2.2.m1.2.2.2.3.cmml">,</mo><mi mathsize="90%" id="S2.E3X.3.2.2.m1.2.2.2.2" xref="S2.E3X.3.2.2.m1.2.2.2.2.cmml">j</mi></mrow></msub><mo maxsize="90%" minsize="90%" id="S2.E3X.3.2.2.m1.4.4.1.1.2.2.2.3" xref="S2.E3X.3.2.2.m1.4.4.1.1.2.3.cmml">}</mo></mrow></mrow></mrow><mo mathsize="90%" id="S2.E3X.3.2.2.m1.4.4.1.2" xref="S2.E3X.3.2.2.m1.4.4.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E3X.3.2.2.m1.4b"><apply id="S2.E3X.3.2.2.m1.4.4.1.1.cmml" xref="S2.E3X.3.2.2.m1.4.4.1"><eq id="S2.E3X.3.2.2.m1.4.4.1.1.3.cmml" xref="S2.E3X.3.2.2.m1.4.4.1.1.3"></eq><csymbol cd="latexml" id="S2.E3X.3.2.2.m1.4.4.1.1.4.cmml" xref="S2.E3X.3.2.2.m1.4.4.1.1.4">absent</csymbol><apply id="S2.E3X.3.2.2.m1.4.4.1.1.2.3.cmml" xref="S2.E3X.3.2.2.m1.4.4.1.1.2.2"><apply id="S2.E3X.3.2.2.m1.4.4.1.1.1.1.1.cmml" xref="S2.E3X.3.2.2.m1.4.4.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E3X.3.2.2.m1.4.4.1.1.1.1.1.1.cmml" xref="S2.E3X.3.2.2.m1.4.4.1.1.1.1.1">subscript</csymbol><max id="S2.E3X.3.2.2.m1.4.4.1.1.1.1.1.2.cmml" xref="S2.E3X.3.2.2.m1.4.4.1.1.1.1.1.2"></max><ci id="S2.E3X.3.2.2.m1.4.4.1.1.1.1.1.3.cmml" xref="S2.E3X.3.2.2.m1.4.4.1.1.1.1.1.3">𝑗</ci></apply><apply id="S2.E3X.3.2.2.m1.4.4.1.1.2.2.2.1.cmml" xref="S2.E3X.3.2.2.m1.4.4.1.1.2.2.2.1"><csymbol cd="ambiguous" id="S2.E3X.3.2.2.m1.4.4.1.1.2.2.2.1.1.cmml" xref="S2.E3X.3.2.2.m1.4.4.1.1.2.2.2.1">subscript</csymbol><apply id="S2.E3X.3.2.2.m1.4.4.1.1.2.2.2.1.2.1.cmml" xref="S2.E3X.3.2.2.m1.4.4.1.1.2.2.2.1.2.2"><csymbol cd="latexml" id="S2.E3X.3.2.2.m1.4.4.1.1.2.2.2.1.2.1.1.cmml" xref="S2.E3X.3.2.2.m1.4.4.1.1.2.2.2.1.2.2.1">delimited-[]</csymbol><ci id="S2.E3X.3.2.2.m1.3.3.cmml" xref="S2.E3X.3.2.2.m1.3.3">𝐖</ci></apply><list id="S2.E3X.3.2.2.m1.2.2.2.3.cmml" xref="S2.E3X.3.2.2.m1.2.2.2.4"><ci id="S2.E3X.3.2.2.m1.1.1.1.1.cmml" xref="S2.E3X.3.2.2.m1.1.1.1.1">𝑖</ci><ci id="S2.E3X.3.2.2.m1.2.2.2.2.cmml" xref="S2.E3X.3.2.2.m1.2.2.2.2">𝑗</ci></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E3X.3.2.2.m1.4c">\displaystyle=\max_{j}\{[\mathbf{W}]_{i,j}\},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="2" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(3)</span></td>
</tr>
<tr id="S2.E3Xa" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S2.E3Xa.2.1.1.m1.2" class="ltx_Math" alttext="\displaystyle[\text{rowsum}(\mathbf{W})]_{i}" display="inline"><semantics id="S2.E3Xa.2.1.1.m1.2a"><msub id="S2.E3Xa.2.1.1.m1.2.2" xref="S2.E3Xa.2.1.1.m1.2.2.cmml"><mrow id="S2.E3Xa.2.1.1.m1.2.2.1.1" xref="S2.E3Xa.2.1.1.m1.2.2.1.2.cmml"><mo maxsize="90%" minsize="90%" id="S2.E3Xa.2.1.1.m1.2.2.1.1.2" xref="S2.E3Xa.2.1.1.m1.2.2.1.2.1.cmml">[</mo><mrow id="S2.E3Xa.2.1.1.m1.2.2.1.1.1" xref="S2.E3Xa.2.1.1.m1.2.2.1.1.1.cmml"><mtext mathsize="90%" id="S2.E3Xa.2.1.1.m1.2.2.1.1.1.2" xref="S2.E3Xa.2.1.1.m1.2.2.1.1.1.2a.cmml">rowsum</mtext><mo lspace="0em" rspace="0em" id="S2.E3Xa.2.1.1.m1.2.2.1.1.1.1" xref="S2.E3Xa.2.1.1.m1.2.2.1.1.1.1.cmml">​</mo><mrow id="S2.E3Xa.2.1.1.m1.2.2.1.1.1.3.2" xref="S2.E3Xa.2.1.1.m1.2.2.1.1.1.cmml"><mo maxsize="90%" minsize="90%" id="S2.E3Xa.2.1.1.m1.2.2.1.1.1.3.2.1" xref="S2.E3Xa.2.1.1.m1.2.2.1.1.1.cmml">(</mo><mi mathsize="90%" id="S2.E3Xa.2.1.1.m1.1.1" xref="S2.E3Xa.2.1.1.m1.1.1.cmml">𝐖</mi><mo maxsize="90%" minsize="90%" id="S2.E3Xa.2.1.1.m1.2.2.1.1.1.3.2.2" xref="S2.E3Xa.2.1.1.m1.2.2.1.1.1.cmml">)</mo></mrow></mrow><mo maxsize="90%" minsize="90%" id="S2.E3Xa.2.1.1.m1.2.2.1.1.3" xref="S2.E3Xa.2.1.1.m1.2.2.1.2.1.cmml">]</mo></mrow><mi mathsize="90%" id="S2.E3Xa.2.1.1.m1.2.2.3" xref="S2.E3Xa.2.1.1.m1.2.2.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.E3Xa.2.1.1.m1.2b"><apply id="S2.E3Xa.2.1.1.m1.2.2.cmml" xref="S2.E3Xa.2.1.1.m1.2.2"><csymbol cd="ambiguous" id="S2.E3Xa.2.1.1.m1.2.2.2.cmml" xref="S2.E3Xa.2.1.1.m1.2.2">subscript</csymbol><apply id="S2.E3Xa.2.1.1.m1.2.2.1.2.cmml" xref="S2.E3Xa.2.1.1.m1.2.2.1.1"><csymbol cd="latexml" id="S2.E3Xa.2.1.1.m1.2.2.1.2.1.cmml" xref="S2.E3Xa.2.1.1.m1.2.2.1.1.2">delimited-[]</csymbol><apply id="S2.E3Xa.2.1.1.m1.2.2.1.1.1.cmml" xref="S2.E3Xa.2.1.1.m1.2.2.1.1.1"><times id="S2.E3Xa.2.1.1.m1.2.2.1.1.1.1.cmml" xref="S2.E3Xa.2.1.1.m1.2.2.1.1.1.1"></times><ci id="S2.E3Xa.2.1.1.m1.2.2.1.1.1.2a.cmml" xref="S2.E3Xa.2.1.1.m1.2.2.1.1.1.2"><mtext mathsize="90%" id="S2.E3Xa.2.1.1.m1.2.2.1.1.1.2.cmml" xref="S2.E3Xa.2.1.1.m1.2.2.1.1.1.2">rowsum</mtext></ci><ci id="S2.E3Xa.2.1.1.m1.1.1.cmml" xref="S2.E3Xa.2.1.1.m1.1.1">𝐖</ci></apply></apply><ci id="S2.E3Xa.2.1.1.m1.2.2.3.cmml" xref="S2.E3Xa.2.1.1.m1.2.2.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E3Xa.2.1.1.m1.2c">\displaystyle[\text{rowsum}(\mathbf{W})]_{i}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S2.E3Xa.3.2.2.m1.4" class="ltx_Math" alttext="\displaystyle=\sum_{j}[\mathbf{W}]_{i,j}," display="inline"><semantics id="S2.E3Xa.3.2.2.m1.4a"><mrow id="S2.E3Xa.3.2.2.m1.4.4.1" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.cmml"><mrow id="S2.E3Xa.3.2.2.m1.4.4.1.1" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.cmml"><mi id="S2.E3Xa.3.2.2.m1.4.4.1.1.2" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.2.cmml"></mi><mo mathsize="90%" id="S2.E3Xa.3.2.2.m1.4.4.1.1.1" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.1.cmml">=</mo><mrow id="S2.E3Xa.3.2.2.m1.4.4.1.1.3" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.3.cmml"><mstyle displaystyle="true" id="S2.E3Xa.3.2.2.m1.4.4.1.1.3.1" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.3.1.cmml"><munder id="S2.E3Xa.3.2.2.m1.4.4.1.1.3.1a" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.3.1.cmml"><mo maxsize="90%" minsize="90%" movablelimits="false" stretchy="true" id="S2.E3Xa.3.2.2.m1.4.4.1.1.3.1.2" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.3.1.2.cmml">∑</mo><mi mathsize="90%" id="S2.E3Xa.3.2.2.m1.4.4.1.1.3.1.3" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.3.1.3.cmml">j</mi></munder></mstyle><msub id="S2.E3Xa.3.2.2.m1.4.4.1.1.3.2" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.3.2.cmml"><mrow id="S2.E3Xa.3.2.2.m1.4.4.1.1.3.2.2.2" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.3.2.2.1.cmml"><mo maxsize="90%" minsize="90%" id="S2.E3Xa.3.2.2.m1.4.4.1.1.3.2.2.2.1" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.3.2.2.1.1.cmml">[</mo><mi mathsize="90%" id="S2.E3Xa.3.2.2.m1.3.3" xref="S2.E3Xa.3.2.2.m1.3.3.cmml">𝐖</mi><mo maxsize="90%" minsize="90%" id="S2.E3Xa.3.2.2.m1.4.4.1.1.3.2.2.2.2" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.3.2.2.1.1.cmml">]</mo></mrow><mrow id="S2.E3Xa.3.2.2.m1.2.2.2.4" xref="S2.E3Xa.3.2.2.m1.2.2.2.3.cmml"><mi mathsize="90%" id="S2.E3Xa.3.2.2.m1.1.1.1.1" xref="S2.E3Xa.3.2.2.m1.1.1.1.1.cmml">i</mi><mo mathsize="90%" id="S2.E3Xa.3.2.2.m1.2.2.2.4.1" xref="S2.E3Xa.3.2.2.m1.2.2.2.3.cmml">,</mo><mi mathsize="90%" id="S2.E3Xa.3.2.2.m1.2.2.2.2" xref="S2.E3Xa.3.2.2.m1.2.2.2.2.cmml">j</mi></mrow></msub></mrow></mrow><mo mathsize="90%" id="S2.E3Xa.3.2.2.m1.4.4.1.2" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E3Xa.3.2.2.m1.4b"><apply id="S2.E3Xa.3.2.2.m1.4.4.1.1.cmml" xref="S2.E3Xa.3.2.2.m1.4.4.1"><eq id="S2.E3Xa.3.2.2.m1.4.4.1.1.1.cmml" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.1"></eq><csymbol cd="latexml" id="S2.E3Xa.3.2.2.m1.4.4.1.1.2.cmml" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.2">absent</csymbol><apply id="S2.E3Xa.3.2.2.m1.4.4.1.1.3.cmml" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.3"><apply id="S2.E3Xa.3.2.2.m1.4.4.1.1.3.1.cmml" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.3.1"><csymbol cd="ambiguous" id="S2.E3Xa.3.2.2.m1.4.4.1.1.3.1.1.cmml" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.3.1">subscript</csymbol><sum id="S2.E3Xa.3.2.2.m1.4.4.1.1.3.1.2.cmml" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.3.1.2"></sum><ci id="S2.E3Xa.3.2.2.m1.4.4.1.1.3.1.3.cmml" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.3.1.3">𝑗</ci></apply><apply id="S2.E3Xa.3.2.2.m1.4.4.1.1.3.2.cmml" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.3.2"><csymbol cd="ambiguous" id="S2.E3Xa.3.2.2.m1.4.4.1.1.3.2.1.cmml" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.3.2">subscript</csymbol><apply id="S2.E3Xa.3.2.2.m1.4.4.1.1.3.2.2.1.cmml" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.3.2.2.2"><csymbol cd="latexml" id="S2.E3Xa.3.2.2.m1.4.4.1.1.3.2.2.1.1.cmml" xref="S2.E3Xa.3.2.2.m1.4.4.1.1.3.2.2.2.1">delimited-[]</csymbol><ci id="S2.E3Xa.3.2.2.m1.3.3.cmml" xref="S2.E3Xa.3.2.2.m1.3.3">𝐖</ci></apply><list id="S2.E3Xa.3.2.2.m1.2.2.2.3.cmml" xref="S2.E3Xa.3.2.2.m1.2.2.2.4"><ci id="S2.E3Xa.3.2.2.m1.1.1.1.1.cmml" xref="S2.E3Xa.3.2.2.m1.1.1.1.1">𝑖</ci><ci id="S2.E3Xa.3.2.2.m1.2.2.2.2.cmml" xref="S2.E3Xa.3.2.2.m1.2.2.2.2">𝑗</ci></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E3Xa.3.2.2.m1.4c">\displaystyle=\sum_{j}[\mathbf{W}]_{i,j},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
</tbody>
</table>
<p id="S2.SS3.p3.20" class="ltx_p">where <math id="S2.SS3.p3.12.m1.1" class="ltx_Math" alttext="[\cdot]_{i}" display="inline"><semantics id="S2.SS3.p3.12.m1.1a"><msub id="S2.SS3.p3.12.m1.1.2" xref="S2.SS3.p3.12.m1.1.2.cmml"><mrow id="S2.SS3.p3.12.m1.1.2.2.2" xref="S2.SS3.p3.12.m1.1.2.2.1.cmml"><mo stretchy="false" id="S2.SS3.p3.12.m1.1.2.2.2.1" xref="S2.SS3.p3.12.m1.1.2.2.1.1.cmml">[</mo><mo lspace="0em" rspace="0em" id="S2.SS3.p3.12.m1.1.1" xref="S2.SS3.p3.12.m1.1.1.cmml">⋅</mo><mo stretchy="false" id="S2.SS3.p3.12.m1.1.2.2.2.2" xref="S2.SS3.p3.12.m1.1.2.2.1.1.cmml">]</mo></mrow><mi id="S2.SS3.p3.12.m1.1.2.3" xref="S2.SS3.p3.12.m1.1.2.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.12.m1.1b"><apply id="S2.SS3.p3.12.m1.1.2.cmml" xref="S2.SS3.p3.12.m1.1.2"><csymbol cd="ambiguous" id="S2.SS3.p3.12.m1.1.2.1.cmml" xref="S2.SS3.p3.12.m1.1.2">subscript</csymbol><apply id="S2.SS3.p3.12.m1.1.2.2.1.cmml" xref="S2.SS3.p3.12.m1.1.2.2.2"><csymbol cd="latexml" id="S2.SS3.p3.12.m1.1.2.2.1.1.cmml" xref="S2.SS3.p3.12.m1.1.2.2.2.1">delimited-[]</csymbol><ci id="S2.SS3.p3.12.m1.1.1.cmml" xref="S2.SS3.p3.12.m1.1.1">⋅</ci></apply><ci id="S2.SS3.p3.12.m1.1.2.3.cmml" xref="S2.SS3.p3.12.m1.1.2.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.12.m1.1c">[\cdot]_{i}</annotation></semantics></math> is the <math id="S2.SS3.p3.13.m2.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S2.SS3.p3.13.m2.1a"><mi id="S2.SS3.p3.13.m2.1.1" xref="S2.SS3.p3.13.m2.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.13.m2.1b"><ci id="S2.SS3.p3.13.m2.1.1.cmml" xref="S2.SS3.p3.13.m2.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.13.m2.1c">i</annotation></semantics></math>-th element of the vector, <math id="S2.SS3.p3.14.m3.3" class="ltx_Math" alttext="[\cdot]_{i,j}" display="inline"><semantics id="S2.SS3.p3.14.m3.3a"><msub id="S2.SS3.p3.14.m3.3.4" xref="S2.SS3.p3.14.m3.3.4.cmml"><mrow id="S2.SS3.p3.14.m3.3.4.2.2" xref="S2.SS3.p3.14.m3.3.4.2.1.cmml"><mo stretchy="false" id="S2.SS3.p3.14.m3.3.4.2.2.1" xref="S2.SS3.p3.14.m3.3.4.2.1.1.cmml">[</mo><mo lspace="0em" rspace="0em" id="S2.SS3.p3.14.m3.3.3" xref="S2.SS3.p3.14.m3.3.3.cmml">⋅</mo><mo stretchy="false" id="S2.SS3.p3.14.m3.3.4.2.2.2" xref="S2.SS3.p3.14.m3.3.4.2.1.1.cmml">]</mo></mrow><mrow id="S2.SS3.p3.14.m3.2.2.2.4" xref="S2.SS3.p3.14.m3.2.2.2.3.cmml"><mi id="S2.SS3.p3.14.m3.1.1.1.1" xref="S2.SS3.p3.14.m3.1.1.1.1.cmml">i</mi><mo id="S2.SS3.p3.14.m3.2.2.2.4.1" xref="S2.SS3.p3.14.m3.2.2.2.3.cmml">,</mo><mi id="S2.SS3.p3.14.m3.2.2.2.2" xref="S2.SS3.p3.14.m3.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.14.m3.3b"><apply id="S2.SS3.p3.14.m3.3.4.cmml" xref="S2.SS3.p3.14.m3.3.4"><csymbol cd="ambiguous" id="S2.SS3.p3.14.m3.3.4.1.cmml" xref="S2.SS3.p3.14.m3.3.4">subscript</csymbol><apply id="S2.SS3.p3.14.m3.3.4.2.1.cmml" xref="S2.SS3.p3.14.m3.3.4.2.2"><csymbol cd="latexml" id="S2.SS3.p3.14.m3.3.4.2.1.1.cmml" xref="S2.SS3.p3.14.m3.3.4.2.2.1">delimited-[]</csymbol><ci id="S2.SS3.p3.14.m3.3.3.cmml" xref="S2.SS3.p3.14.m3.3.3">⋅</ci></apply><list id="S2.SS3.p3.14.m3.2.2.2.3.cmml" xref="S2.SS3.p3.14.m3.2.2.2.4"><ci id="S2.SS3.p3.14.m3.1.1.1.1.cmml" xref="S2.SS3.p3.14.m3.1.1.1.1">𝑖</ci><ci id="S2.SS3.p3.14.m3.2.2.2.2.cmml" xref="S2.SS3.p3.14.m3.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.14.m3.3c">[\cdot]_{i,j}</annotation></semantics></math> is the element in the <math id="S2.SS3.p3.15.m4.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S2.SS3.p3.15.m4.1a"><mi id="S2.SS3.p3.15.m4.1.1" xref="S2.SS3.p3.15.m4.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.15.m4.1b"><ci id="S2.SS3.p3.15.m4.1.1.cmml" xref="S2.SS3.p3.15.m4.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.15.m4.1c">i</annotation></semantics></math>-th row and <math id="S2.SS3.p3.16.m5.1" class="ltx_Math" alttext="j" display="inline"><semantics id="S2.SS3.p3.16.m5.1a"><mi id="S2.SS3.p3.16.m5.1.1" xref="S2.SS3.p3.16.m5.1.1.cmml">j</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.16.m5.1b"><ci id="S2.SS3.p3.16.m5.1.1.cmml" xref="S2.SS3.p3.16.m5.1.1">𝑗</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.16.m5.1c">j</annotation></semantics></math>-th column of the matrix.
To make the subsequent backward pass more efficient, we store <math id="S2.SS3.p3.17.m6.1" class="ltx_Math" alttext="lse_{i}" display="inline"><semantics id="S2.SS3.p3.17.m6.1a"><mrow id="S2.SS3.p3.17.m6.1.1" xref="S2.SS3.p3.17.m6.1.1.cmml"><mi id="S2.SS3.p3.17.m6.1.1.2" xref="S2.SS3.p3.17.m6.1.1.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="S2.SS3.p3.17.m6.1.1.1" xref="S2.SS3.p3.17.m6.1.1.1.cmml">​</mo><mi id="S2.SS3.p3.17.m6.1.1.3" xref="S2.SS3.p3.17.m6.1.1.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="S2.SS3.p3.17.m6.1.1.1a" xref="S2.SS3.p3.17.m6.1.1.1.cmml">​</mo><msub id="S2.SS3.p3.17.m6.1.1.4" xref="S2.SS3.p3.17.m6.1.1.4.cmml"><mi id="S2.SS3.p3.17.m6.1.1.4.2" xref="S2.SS3.p3.17.m6.1.1.4.2.cmml">e</mi><mi id="S2.SS3.p3.17.m6.1.1.4.3" xref="S2.SS3.p3.17.m6.1.1.4.3.cmml">i</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.17.m6.1b"><apply id="S2.SS3.p3.17.m6.1.1.cmml" xref="S2.SS3.p3.17.m6.1.1"><times id="S2.SS3.p3.17.m6.1.1.1.cmml" xref="S2.SS3.p3.17.m6.1.1.1"></times><ci id="S2.SS3.p3.17.m6.1.1.2.cmml" xref="S2.SS3.p3.17.m6.1.1.2">𝑙</ci><ci id="S2.SS3.p3.17.m6.1.1.3.cmml" xref="S2.SS3.p3.17.m6.1.1.3">𝑠</ci><apply id="S2.SS3.p3.17.m6.1.1.4.cmml" xref="S2.SS3.p3.17.m6.1.1.4"><csymbol cd="ambiguous" id="S2.SS3.p3.17.m6.1.1.4.1.cmml" xref="S2.SS3.p3.17.m6.1.1.4">subscript</csymbol><ci id="S2.SS3.p3.17.m6.1.1.4.2.cmml" xref="S2.SS3.p3.17.m6.1.1.4.2">𝑒</ci><ci id="S2.SS3.p3.17.m6.1.1.4.3.cmml" xref="S2.SS3.p3.17.m6.1.1.4.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.17.m6.1c">lse_{i}</annotation></semantics></math> besides the global results <math id="S2.SS3.p3.18.m7.1" class="ltx_Math" alttext="\mathbf{O}_{i}" display="inline"><semantics id="S2.SS3.p3.18.m7.1a"><msub id="S2.SS3.p3.18.m7.1.1" xref="S2.SS3.p3.18.m7.1.1.cmml"><mi id="S2.SS3.p3.18.m7.1.1.2" xref="S2.SS3.p3.18.m7.1.1.2.cmml">𝐎</mi><mi id="S2.SS3.p3.18.m7.1.1.3" xref="S2.SS3.p3.18.m7.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.18.m7.1b"><apply id="S2.SS3.p3.18.m7.1.1.cmml" xref="S2.SS3.p3.18.m7.1.1"><csymbol cd="ambiguous" id="S2.SS3.p3.18.m7.1.1.1.cmml" xref="S2.SS3.p3.18.m7.1.1">subscript</csymbol><ci id="S2.SS3.p3.18.m7.1.1.2.cmml" xref="S2.SS3.p3.18.m7.1.1.2">𝐎</ci><ci id="S2.SS3.p3.18.m7.1.1.3.cmml" xref="S2.SS3.p3.18.m7.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.18.m7.1c">\mathbf{O}_{i}</annotation></semantics></math> after the forward pass. During the backward pass, as depicted in Algorithm <a href="#alg2" title="Algorithm 2 ‣ 2.1 Preliminary ‣ 2 Methodology ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>,
we employ the same strategy for the forward pass to obtain gradients based only on recomputed <math id="S2.SS3.p3.19.m8.1" class="ltx_Math" alttext="\mathbf{S}" display="inline"><semantics id="S2.SS3.p3.19.m8.1a"><mi id="S2.SS3.p3.19.m8.1.1" xref="S2.SS3.p3.19.m8.1.1.cmml">𝐒</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.19.m8.1b"><ci id="S2.SS3.p3.19.m8.1.1.cmml" xref="S2.SS3.p3.19.m8.1.1">𝐒</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.19.m8.1c">\mathbf{S}</annotation></semantics></math> and <math id="S2.SS3.p3.20.m9.1" class="ltx_Math" alttext="\mathbf{P}" display="inline"><semantics id="S2.SS3.p3.20.m9.1a"><mi id="S2.SS3.p3.20.m9.1.1" xref="S2.SS3.p3.20.m9.1.1.cmml">𝐏</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.20.m9.1b"><ci id="S2.SS3.p3.20.m9.1.1.cmml" xref="S2.SS3.p3.20.m9.1.1">𝐏</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.20.m9.1c">\mathbf{P}</annotation></semantics></math>.</p>
</div>
</section>
<section id="S2.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.4 </span>Local Attention Optimization (LAO)</h3>

<div id="S2.SS4.p1" class="ltx_para">
<p id="S2.SS4.p1.14" class="ltx_p">Given <math id="S2.SS4.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{Q}_{i}" display="inline"><semantics id="S2.SS4.p1.1.m1.1a"><msub id="S2.SS4.p1.1.m1.1.1" xref="S2.SS4.p1.1.m1.1.1.cmml"><mi id="S2.SS4.p1.1.m1.1.1.2" xref="S2.SS4.p1.1.m1.1.1.2.cmml">𝐐</mi><mi id="S2.SS4.p1.1.m1.1.1.3" xref="S2.SS4.p1.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.1.m1.1b"><apply id="S2.SS4.p1.1.m1.1.1.cmml" xref="S2.SS4.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS4.p1.1.m1.1.1.1.cmml" xref="S2.SS4.p1.1.m1.1.1">subscript</csymbol><ci id="S2.SS4.p1.1.m1.1.1.2.cmml" xref="S2.SS4.p1.1.m1.1.1.2">𝐐</ci><ci id="S2.SS4.p1.1.m1.1.1.3.cmml" xref="S2.SS4.p1.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.1.m1.1c">\mathbf{Q}_{i}</annotation></semantics></math>, <math id="S2.SS4.p1.2.m2.1" class="ltx_Math" alttext="\mathbf{K}_{j}" display="inline"><semantics id="S2.SS4.p1.2.m2.1a"><msub id="S2.SS4.p1.2.m2.1.1" xref="S2.SS4.p1.2.m2.1.1.cmml"><mi id="S2.SS4.p1.2.m2.1.1.2" xref="S2.SS4.p1.2.m2.1.1.2.cmml">𝐊</mi><mi id="S2.SS4.p1.2.m2.1.1.3" xref="S2.SS4.p1.2.m2.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.2.m2.1b"><apply id="S2.SS4.p1.2.m2.1.1.cmml" xref="S2.SS4.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS4.p1.2.m2.1.1.1.cmml" xref="S2.SS4.p1.2.m2.1.1">subscript</csymbol><ci id="S2.SS4.p1.2.m2.1.1.2.cmml" xref="S2.SS4.p1.2.m2.1.1.2">𝐊</ci><ci id="S2.SS4.p1.2.m2.1.1.3.cmml" xref="S2.SS4.p1.2.m2.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.2.m2.1c">\mathbf{K}_{j}</annotation></semantics></math>, and <math id="S2.SS4.p1.3.m3.1" class="ltx_Math" alttext="\mathbf{V}_{j}" display="inline"><semantics id="S2.SS4.p1.3.m3.1a"><msub id="S2.SS4.p1.3.m3.1.1" xref="S2.SS4.p1.3.m3.1.1.cmml"><mi id="S2.SS4.p1.3.m3.1.1.2" xref="S2.SS4.p1.3.m3.1.1.2.cmml">𝐕</mi><mi id="S2.SS4.p1.3.m3.1.1.3" xref="S2.SS4.p1.3.m3.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.3.m3.1b"><apply id="S2.SS4.p1.3.m3.1.1.cmml" xref="S2.SS4.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS4.p1.3.m3.1.1.1.cmml" xref="S2.SS4.p1.3.m3.1.1">subscript</csymbol><ci id="S2.SS4.p1.3.m3.1.1.2.cmml" xref="S2.SS4.p1.3.m3.1.1.2">𝐕</ci><ci id="S2.SS4.p1.3.m3.1.1.3.cmml" xref="S2.SS4.p1.3.m3.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.3.m3.1c">\mathbf{V}_{j}</annotation></semantics></math>, the local attention operations that involve these partitions are performed only on a single device (e.g., a GPU).
When computing <math id="S2.SS4.p1.4.m4.2" class="ltx_Math" alttext="\mathbf{O}_{i,j}" display="inline"><semantics id="S2.SS4.p1.4.m4.2a"><msub id="S2.SS4.p1.4.m4.2.3" xref="S2.SS4.p1.4.m4.2.3.cmml"><mi id="S2.SS4.p1.4.m4.2.3.2" xref="S2.SS4.p1.4.m4.2.3.2.cmml">𝐎</mi><mrow id="S2.SS4.p1.4.m4.2.2.2.4" xref="S2.SS4.p1.4.m4.2.2.2.3.cmml"><mi id="S2.SS4.p1.4.m4.1.1.1.1" xref="S2.SS4.p1.4.m4.1.1.1.1.cmml">i</mi><mo id="S2.SS4.p1.4.m4.2.2.2.4.1" xref="S2.SS4.p1.4.m4.2.2.2.3.cmml">,</mo><mi id="S2.SS4.p1.4.m4.2.2.2.2" xref="S2.SS4.p1.4.m4.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.4.m4.2b"><apply id="S2.SS4.p1.4.m4.2.3.cmml" xref="S2.SS4.p1.4.m4.2.3"><csymbol cd="ambiguous" id="S2.SS4.p1.4.m4.2.3.1.cmml" xref="S2.SS4.p1.4.m4.2.3">subscript</csymbol><ci id="S2.SS4.p1.4.m4.2.3.2.cmml" xref="S2.SS4.p1.4.m4.2.3.2">𝐎</ci><list id="S2.SS4.p1.4.m4.2.2.2.3.cmml" xref="S2.SS4.p1.4.m4.2.2.2.4"><ci id="S2.SS4.p1.4.m4.1.1.1.1.cmml" xref="S2.SS4.p1.4.m4.1.1.1.1">𝑖</ci><ci id="S2.SS4.p1.4.m4.2.2.2.2.cmml" xref="S2.SS4.p1.4.m4.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.4.m4.2c">\mathbf{O}_{i,j}</annotation></semantics></math> in Eq. (<a href="#S2.E2" title="Equation 2 ‣ 2.2 The Whole Framework of BurstAttention ‣ 2 Methodology ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>), <math id="S2.SS4.p1.5.m5.2" class="ltx_Math" alttext="\mathbf{S}_{i,j}" display="inline"><semantics id="S2.SS4.p1.5.m5.2a"><msub id="S2.SS4.p1.5.m5.2.3" xref="S2.SS4.p1.5.m5.2.3.cmml"><mi id="S2.SS4.p1.5.m5.2.3.2" xref="S2.SS4.p1.5.m5.2.3.2.cmml">𝐒</mi><mrow id="S2.SS4.p1.5.m5.2.2.2.4" xref="S2.SS4.p1.5.m5.2.2.2.3.cmml"><mi id="S2.SS4.p1.5.m5.1.1.1.1" xref="S2.SS4.p1.5.m5.1.1.1.1.cmml">i</mi><mo id="S2.SS4.p1.5.m5.2.2.2.4.1" xref="S2.SS4.p1.5.m5.2.2.2.3.cmml">,</mo><mi id="S2.SS4.p1.5.m5.2.2.2.2" xref="S2.SS4.p1.5.m5.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.5.m5.2b"><apply id="S2.SS4.p1.5.m5.2.3.cmml" xref="S2.SS4.p1.5.m5.2.3"><csymbol cd="ambiguous" id="S2.SS4.p1.5.m5.2.3.1.cmml" xref="S2.SS4.p1.5.m5.2.3">subscript</csymbol><ci id="S2.SS4.p1.5.m5.2.3.2.cmml" xref="S2.SS4.p1.5.m5.2.3.2">𝐒</ci><list id="S2.SS4.p1.5.m5.2.2.2.3.cmml" xref="S2.SS4.p1.5.m5.2.2.2.4"><ci id="S2.SS4.p1.5.m5.1.1.1.1.cmml" xref="S2.SS4.p1.5.m5.1.1.1.1">𝑖</ci><ci id="S2.SS4.p1.5.m5.2.2.2.2.cmml" xref="S2.SS4.p1.5.m5.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.5.m5.2c">\mathbf{S}_{i,j}</annotation></semantics></math> and <math id="S2.SS4.p1.6.m6.2" class="ltx_Math" alttext="\mathbf{P}_{i,j}" display="inline"><semantics id="S2.SS4.p1.6.m6.2a"><msub id="S2.SS4.p1.6.m6.2.3" xref="S2.SS4.p1.6.m6.2.3.cmml"><mi id="S2.SS4.p1.6.m6.2.3.2" xref="S2.SS4.p1.6.m6.2.3.2.cmml">𝐏</mi><mrow id="S2.SS4.p1.6.m6.2.2.2.4" xref="S2.SS4.p1.6.m6.2.2.2.3.cmml"><mi id="S2.SS4.p1.6.m6.1.1.1.1" xref="S2.SS4.p1.6.m6.1.1.1.1.cmml">i</mi><mo id="S2.SS4.p1.6.m6.2.2.2.4.1" xref="S2.SS4.p1.6.m6.2.2.2.3.cmml">,</mo><mi id="S2.SS4.p1.6.m6.2.2.2.2" xref="S2.SS4.p1.6.m6.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.6.m6.2b"><apply id="S2.SS4.p1.6.m6.2.3.cmml" xref="S2.SS4.p1.6.m6.2.3"><csymbol cd="ambiguous" id="S2.SS4.p1.6.m6.2.3.1.cmml" xref="S2.SS4.p1.6.m6.2.3">subscript</csymbol><ci id="S2.SS4.p1.6.m6.2.3.2.cmml" xref="S2.SS4.p1.6.m6.2.3.2">𝐏</ci><list id="S2.SS4.p1.6.m6.2.2.2.3.cmml" xref="S2.SS4.p1.6.m6.2.2.2.4"><ci id="S2.SS4.p1.6.m6.1.1.1.1.cmml" xref="S2.SS4.p1.6.m6.1.1.1.1">𝑖</ci><ci id="S2.SS4.p1.6.m6.2.2.2.2.cmml" xref="S2.SS4.p1.6.m6.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.6.m6.2c">\mathbf{P}_{i,j}</annotation></semantics></math> are computed and stored on the HBM of the device.
To avoid frequent I/O operations of <math id="S2.SS4.p1.7.m7.2" class="ltx_Math" alttext="\mathbf{S}_{i,j}" display="inline"><semantics id="S2.SS4.p1.7.m7.2a"><msub id="S2.SS4.p1.7.m7.2.3" xref="S2.SS4.p1.7.m7.2.3.cmml"><mi id="S2.SS4.p1.7.m7.2.3.2" xref="S2.SS4.p1.7.m7.2.3.2.cmml">𝐒</mi><mrow id="S2.SS4.p1.7.m7.2.2.2.4" xref="S2.SS4.p1.7.m7.2.2.2.3.cmml"><mi id="S2.SS4.p1.7.m7.1.1.1.1" xref="S2.SS4.p1.7.m7.1.1.1.1.cmml">i</mi><mo id="S2.SS4.p1.7.m7.2.2.2.4.1" xref="S2.SS4.p1.7.m7.2.2.2.3.cmml">,</mo><mi id="S2.SS4.p1.7.m7.2.2.2.2" xref="S2.SS4.p1.7.m7.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.7.m7.2b"><apply id="S2.SS4.p1.7.m7.2.3.cmml" xref="S2.SS4.p1.7.m7.2.3"><csymbol cd="ambiguous" id="S2.SS4.p1.7.m7.2.3.1.cmml" xref="S2.SS4.p1.7.m7.2.3">subscript</csymbol><ci id="S2.SS4.p1.7.m7.2.3.2.cmml" xref="S2.SS4.p1.7.m7.2.3.2">𝐒</ci><list id="S2.SS4.p1.7.m7.2.2.2.3.cmml" xref="S2.SS4.p1.7.m7.2.2.2.4"><ci id="S2.SS4.p1.7.m7.1.1.1.1.cmml" xref="S2.SS4.p1.7.m7.1.1.1.1">𝑖</ci><ci id="S2.SS4.p1.7.m7.2.2.2.2.cmml" xref="S2.SS4.p1.7.m7.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.7.m7.2c">\mathbf{S}_{i,j}</annotation></semantics></math> and <math id="S2.SS4.p1.8.m8.2" class="ltx_Math" alttext="\mathbf{P}_{i,j}" display="inline"><semantics id="S2.SS4.p1.8.m8.2a"><msub id="S2.SS4.p1.8.m8.2.3" xref="S2.SS4.p1.8.m8.2.3.cmml"><mi id="S2.SS4.p1.8.m8.2.3.2" xref="S2.SS4.p1.8.m8.2.3.2.cmml">𝐏</mi><mrow id="S2.SS4.p1.8.m8.2.2.2.4" xref="S2.SS4.p1.8.m8.2.2.2.3.cmml"><mi id="S2.SS4.p1.8.m8.1.1.1.1" xref="S2.SS4.p1.8.m8.1.1.1.1.cmml">i</mi><mo id="S2.SS4.p1.8.m8.2.2.2.4.1" xref="S2.SS4.p1.8.m8.2.2.2.3.cmml">,</mo><mi id="S2.SS4.p1.8.m8.2.2.2.2" xref="S2.SS4.p1.8.m8.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.8.m8.2b"><apply id="S2.SS4.p1.8.m8.2.3.cmml" xref="S2.SS4.p1.8.m8.2.3"><csymbol cd="ambiguous" id="S2.SS4.p1.8.m8.2.3.1.cmml" xref="S2.SS4.p1.8.m8.2.3">subscript</csymbol><ci id="S2.SS4.p1.8.m8.2.3.2.cmml" xref="S2.SS4.p1.8.m8.2.3.2">𝐏</ci><list id="S2.SS4.p1.8.m8.2.2.2.3.cmml" xref="S2.SS4.p1.8.m8.2.2.2.4"><ci id="S2.SS4.p1.8.m8.1.1.1.1.cmml" xref="S2.SS4.p1.8.m8.1.1.1.1">𝑖</ci><ci id="S2.SS4.p1.8.m8.2.2.2.2.cmml" xref="S2.SS4.p1.8.m8.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.8.m8.2c">\mathbf{P}_{i,j}</annotation></semantics></math> on the HBM, the local attention operations of BurstAttention further divide <math id="S2.SS4.p1.9.m9.1" class="ltx_Math" alttext="\mathbf{Q}_{i}" display="inline"><semantics id="S2.SS4.p1.9.m9.1a"><msub id="S2.SS4.p1.9.m9.1.1" xref="S2.SS4.p1.9.m9.1.1.cmml"><mi id="S2.SS4.p1.9.m9.1.1.2" xref="S2.SS4.p1.9.m9.1.1.2.cmml">𝐐</mi><mi id="S2.SS4.p1.9.m9.1.1.3" xref="S2.SS4.p1.9.m9.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.9.m9.1b"><apply id="S2.SS4.p1.9.m9.1.1.cmml" xref="S2.SS4.p1.9.m9.1.1"><csymbol cd="ambiguous" id="S2.SS4.p1.9.m9.1.1.1.cmml" xref="S2.SS4.p1.9.m9.1.1">subscript</csymbol><ci id="S2.SS4.p1.9.m9.1.1.2.cmml" xref="S2.SS4.p1.9.m9.1.1.2">𝐐</ci><ci id="S2.SS4.p1.9.m9.1.1.3.cmml" xref="S2.SS4.p1.9.m9.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.9.m9.1c">\mathbf{Q}_{i}</annotation></semantics></math>, <math id="S2.SS4.p1.10.m10.1" class="ltx_Math" alttext="\mathbf{K}_{j}" display="inline"><semantics id="S2.SS4.p1.10.m10.1a"><msub id="S2.SS4.p1.10.m10.1.1" xref="S2.SS4.p1.10.m10.1.1.cmml"><mi id="S2.SS4.p1.10.m10.1.1.2" xref="S2.SS4.p1.10.m10.1.1.2.cmml">𝐊</mi><mi id="S2.SS4.p1.10.m10.1.1.3" xref="S2.SS4.p1.10.m10.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.10.m10.1b"><apply id="S2.SS4.p1.10.m10.1.1.cmml" xref="S2.SS4.p1.10.m10.1.1"><csymbol cd="ambiguous" id="S2.SS4.p1.10.m10.1.1.1.cmml" xref="S2.SS4.p1.10.m10.1.1">subscript</csymbol><ci id="S2.SS4.p1.10.m10.1.1.2.cmml" xref="S2.SS4.p1.10.m10.1.1.2">𝐊</ci><ci id="S2.SS4.p1.10.m10.1.1.3.cmml" xref="S2.SS4.p1.10.m10.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.10.m10.1c">\mathbf{K}_{j}</annotation></semantics></math>, and <math id="S2.SS4.p1.11.m11.1" class="ltx_Math" alttext="\mathbf{V}_{j}" display="inline"><semantics id="S2.SS4.p1.11.m11.1a"><msub id="S2.SS4.p1.11.m11.1.1" xref="S2.SS4.p1.11.m11.1.1.cmml"><mi id="S2.SS4.p1.11.m11.1.1.2" xref="S2.SS4.p1.11.m11.1.1.2.cmml">𝐕</mi><mi id="S2.SS4.p1.11.m11.1.1.3" xref="S2.SS4.p1.11.m11.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.11.m11.1b"><apply id="S2.SS4.p1.11.m11.1.1.cmml" xref="S2.SS4.p1.11.m11.1.1"><csymbol cd="ambiguous" id="S2.SS4.p1.11.m11.1.1.1.cmml" xref="S2.SS4.p1.11.m11.1.1">subscript</csymbol><ci id="S2.SS4.p1.11.m11.1.1.2.cmml" xref="S2.SS4.p1.11.m11.1.1.2">𝐕</ci><ci id="S2.SS4.p1.11.m11.1.1.3.cmml" xref="S2.SS4.p1.11.m11.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.11.m11.1c">\mathbf{V}_{j}</annotation></semantics></math> into tiles along the sequence dimension, with each tile <math id="S2.SS4.p1.12.m12.1" class="ltx_Math" alttext="\frac{M}{4d}" display="inline"><semantics id="S2.SS4.p1.12.m12.1a"><mfrac id="S2.SS4.p1.12.m12.1.1" xref="S2.SS4.p1.12.m12.1.1.cmml"><mi id="S2.SS4.p1.12.m12.1.1.2" xref="S2.SS4.p1.12.m12.1.1.2.cmml">M</mi><mrow id="S2.SS4.p1.12.m12.1.1.3" xref="S2.SS4.p1.12.m12.1.1.3.cmml"><mn id="S2.SS4.p1.12.m12.1.1.3.2" xref="S2.SS4.p1.12.m12.1.1.3.2.cmml">4</mn><mo lspace="0em" rspace="0em" id="S2.SS4.p1.12.m12.1.1.3.1" xref="S2.SS4.p1.12.m12.1.1.3.1.cmml">​</mo><mi id="S2.SS4.p1.12.m12.1.1.3.3" xref="S2.SS4.p1.12.m12.1.1.3.3.cmml">d</mi></mrow></mfrac><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.12.m12.1b"><apply id="S2.SS4.p1.12.m12.1.1.cmml" xref="S2.SS4.p1.12.m12.1.1"><divide id="S2.SS4.p1.12.m12.1.1.1.cmml" xref="S2.SS4.p1.12.m12.1.1"></divide><ci id="S2.SS4.p1.12.m12.1.1.2.cmml" xref="S2.SS4.p1.12.m12.1.1.2">𝑀</ci><apply id="S2.SS4.p1.12.m12.1.1.3.cmml" xref="S2.SS4.p1.12.m12.1.1.3"><times id="S2.SS4.p1.12.m12.1.1.3.1.cmml" xref="S2.SS4.p1.12.m12.1.1.3.1"></times><cn type="integer" id="S2.SS4.p1.12.m12.1.1.3.2.cmml" xref="S2.SS4.p1.12.m12.1.1.3.2">4</cn><ci id="S2.SS4.p1.12.m12.1.1.3.3.cmml" xref="S2.SS4.p1.12.m12.1.1.3.3">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.12.m12.1c">\frac{M}{4d}</annotation></semantics></math> sequence length, where <math id="S2.SS4.p1.13.m13.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S2.SS4.p1.13.m13.1a"><mi id="S2.SS4.p1.13.m13.1.1" xref="S2.SS4.p1.13.m13.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.13.m13.1b"><ci id="S2.SS4.p1.13.m13.1.1.cmml" xref="S2.SS4.p1.13.m13.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.13.m13.1c">M</annotation></semantics></math> represents the SRAM size of the device, <math id="S2.SS4.p1.14.m14.1" class="ltx_Math" alttext="d" display="inline"><semantics id="S2.SS4.p1.14.m14.1a"><mi id="S2.SS4.p1.14.m14.1.1" xref="S2.SS4.p1.14.m14.1.1.cmml">d</mi><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.14.m14.1b"><ci id="S2.SS4.p1.14.m14.1.1.cmml" xref="S2.SS4.p1.14.m14.1.1">𝑑</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.14.m14.1c">d</annotation></semantics></math> represents the attention head dimension.</p>
</div>
<div id="S2.SS4.p2" class="ltx_para">
<p id="S2.SS4.p2.10" class="ltx_p">As shown in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, during computing <math id="S2.SS4.p2.1.m1.2" class="ltx_Math" alttext="\mathbf{O}_{i,j}" display="inline"><semantics id="S2.SS4.p2.1.m1.2a"><msub id="S2.SS4.p2.1.m1.2.3" xref="S2.SS4.p2.1.m1.2.3.cmml"><mi id="S2.SS4.p2.1.m1.2.3.2" xref="S2.SS4.p2.1.m1.2.3.2.cmml">𝐎</mi><mrow id="S2.SS4.p2.1.m1.2.2.2.4" xref="S2.SS4.p2.1.m1.2.2.2.3.cmml"><mi id="S2.SS4.p2.1.m1.1.1.1.1" xref="S2.SS4.p2.1.m1.1.1.1.1.cmml">i</mi><mo id="S2.SS4.p2.1.m1.2.2.2.4.1" xref="S2.SS4.p2.1.m1.2.2.2.3.cmml">,</mo><mi id="S2.SS4.p2.1.m1.2.2.2.2" xref="S2.SS4.p2.1.m1.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.1.m1.2b"><apply id="S2.SS4.p2.1.m1.2.3.cmml" xref="S2.SS4.p2.1.m1.2.3"><csymbol cd="ambiguous" id="S2.SS4.p2.1.m1.2.3.1.cmml" xref="S2.SS4.p2.1.m1.2.3">subscript</csymbol><ci id="S2.SS4.p2.1.m1.2.3.2.cmml" xref="S2.SS4.p2.1.m1.2.3.2">𝐎</ci><list id="S2.SS4.p2.1.m1.2.2.2.3.cmml" xref="S2.SS4.p2.1.m1.2.2.2.4"><ci id="S2.SS4.p2.1.m1.1.1.1.1.cmml" xref="S2.SS4.p2.1.m1.1.1.1.1">𝑖</ci><ci id="S2.SS4.p2.1.m1.2.2.2.2.cmml" xref="S2.SS4.p2.1.m1.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.1.m1.2c">\mathbf{O}_{i,j}</annotation></semantics></math>, each thread block reads the tiles of <math id="S2.SS4.p2.2.m2.3" class="ltx_Math" alttext="\mathbf{Q}_{i},\mathbf{K}_{j},\mathbf{V}_{j}" display="inline"><semantics id="S2.SS4.p2.2.m2.3a"><mrow id="S2.SS4.p2.2.m2.3.3.3" xref="S2.SS4.p2.2.m2.3.3.4.cmml"><msub id="S2.SS4.p2.2.m2.1.1.1.1" xref="S2.SS4.p2.2.m2.1.1.1.1.cmml"><mi id="S2.SS4.p2.2.m2.1.1.1.1.2" xref="S2.SS4.p2.2.m2.1.1.1.1.2.cmml">𝐐</mi><mi id="S2.SS4.p2.2.m2.1.1.1.1.3" xref="S2.SS4.p2.2.m2.1.1.1.1.3.cmml">i</mi></msub><mo id="S2.SS4.p2.2.m2.3.3.3.4" xref="S2.SS4.p2.2.m2.3.3.4.cmml">,</mo><msub id="S2.SS4.p2.2.m2.2.2.2.2" xref="S2.SS4.p2.2.m2.2.2.2.2.cmml"><mi id="S2.SS4.p2.2.m2.2.2.2.2.2" xref="S2.SS4.p2.2.m2.2.2.2.2.2.cmml">𝐊</mi><mi id="S2.SS4.p2.2.m2.2.2.2.2.3" xref="S2.SS4.p2.2.m2.2.2.2.2.3.cmml">j</mi></msub><mo id="S2.SS4.p2.2.m2.3.3.3.5" xref="S2.SS4.p2.2.m2.3.3.4.cmml">,</mo><msub id="S2.SS4.p2.2.m2.3.3.3.3" xref="S2.SS4.p2.2.m2.3.3.3.3.cmml"><mi id="S2.SS4.p2.2.m2.3.3.3.3.2" xref="S2.SS4.p2.2.m2.3.3.3.3.2.cmml">𝐕</mi><mi id="S2.SS4.p2.2.m2.3.3.3.3.3" xref="S2.SS4.p2.2.m2.3.3.3.3.3.cmml">j</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.2.m2.3b"><list id="S2.SS4.p2.2.m2.3.3.4.cmml" xref="S2.SS4.p2.2.m2.3.3.3"><apply id="S2.SS4.p2.2.m2.1.1.1.1.cmml" xref="S2.SS4.p2.2.m2.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS4.p2.2.m2.1.1.1.1.1.cmml" xref="S2.SS4.p2.2.m2.1.1.1.1">subscript</csymbol><ci id="S2.SS4.p2.2.m2.1.1.1.1.2.cmml" xref="S2.SS4.p2.2.m2.1.1.1.1.2">𝐐</ci><ci id="S2.SS4.p2.2.m2.1.1.1.1.3.cmml" xref="S2.SS4.p2.2.m2.1.1.1.1.3">𝑖</ci></apply><apply id="S2.SS4.p2.2.m2.2.2.2.2.cmml" xref="S2.SS4.p2.2.m2.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS4.p2.2.m2.2.2.2.2.1.cmml" xref="S2.SS4.p2.2.m2.2.2.2.2">subscript</csymbol><ci id="S2.SS4.p2.2.m2.2.2.2.2.2.cmml" xref="S2.SS4.p2.2.m2.2.2.2.2.2">𝐊</ci><ci id="S2.SS4.p2.2.m2.2.2.2.2.3.cmml" xref="S2.SS4.p2.2.m2.2.2.2.2.3">𝑗</ci></apply><apply id="S2.SS4.p2.2.m2.3.3.3.3.cmml" xref="S2.SS4.p2.2.m2.3.3.3.3"><csymbol cd="ambiguous" id="S2.SS4.p2.2.m2.3.3.3.3.1.cmml" xref="S2.SS4.p2.2.m2.3.3.3.3">subscript</csymbol><ci id="S2.SS4.p2.2.m2.3.3.3.3.2.cmml" xref="S2.SS4.p2.2.m2.3.3.3.3.2">𝐕</ci><ci id="S2.SS4.p2.2.m2.3.3.3.3.3.cmml" xref="S2.SS4.p2.2.m2.3.3.3.3.3">𝑗</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.2.m2.3c">\mathbf{Q}_{i},\mathbf{K}_{j},\mathbf{V}_{j}</annotation></semantics></math> from the HBM to SRAM, the tiles of <math id="S2.SS4.p2.3.m3.2" class="ltx_Math" alttext="\mathbf{S}_{i,j}" display="inline"><semantics id="S2.SS4.p2.3.m3.2a"><msub id="S2.SS4.p2.3.m3.2.3" xref="S2.SS4.p2.3.m3.2.3.cmml"><mi id="S2.SS4.p2.3.m3.2.3.2" xref="S2.SS4.p2.3.m3.2.3.2.cmml">𝐒</mi><mrow id="S2.SS4.p2.3.m3.2.2.2.4" xref="S2.SS4.p2.3.m3.2.2.2.3.cmml"><mi id="S2.SS4.p2.3.m3.1.1.1.1" xref="S2.SS4.p2.3.m3.1.1.1.1.cmml">i</mi><mo id="S2.SS4.p2.3.m3.2.2.2.4.1" xref="S2.SS4.p2.3.m3.2.2.2.3.cmml">,</mo><mi id="S2.SS4.p2.3.m3.2.2.2.2" xref="S2.SS4.p2.3.m3.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.3.m3.2b"><apply id="S2.SS4.p2.3.m3.2.3.cmml" xref="S2.SS4.p2.3.m3.2.3"><csymbol cd="ambiguous" id="S2.SS4.p2.3.m3.2.3.1.cmml" xref="S2.SS4.p2.3.m3.2.3">subscript</csymbol><ci id="S2.SS4.p2.3.m3.2.3.2.cmml" xref="S2.SS4.p2.3.m3.2.3.2">𝐒</ci><list id="S2.SS4.p2.3.m3.2.2.2.3.cmml" xref="S2.SS4.p2.3.m3.2.2.2.4"><ci id="S2.SS4.p2.3.m3.1.1.1.1.cmml" xref="S2.SS4.p2.3.m3.1.1.1.1">𝑖</ci><ci id="S2.SS4.p2.3.m3.2.2.2.2.cmml" xref="S2.SS4.p2.3.m3.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.3.m3.2c">\mathbf{S}_{i,j}</annotation></semantics></math> and <math id="S2.SS4.p2.4.m4.2" class="ltx_Math" alttext="\mathbf{P}_{i,j}" display="inline"><semantics id="S2.SS4.p2.4.m4.2a"><msub id="S2.SS4.p2.4.m4.2.3" xref="S2.SS4.p2.4.m4.2.3.cmml"><mi id="S2.SS4.p2.4.m4.2.3.2" xref="S2.SS4.p2.4.m4.2.3.2.cmml">𝐏</mi><mrow id="S2.SS4.p2.4.m4.2.2.2.4" xref="S2.SS4.p2.4.m4.2.2.2.3.cmml"><mi id="S2.SS4.p2.4.m4.1.1.1.1" xref="S2.SS4.p2.4.m4.1.1.1.1.cmml">i</mi><mo id="S2.SS4.p2.4.m4.2.2.2.4.1" xref="S2.SS4.p2.4.m4.2.2.2.3.cmml">,</mo><mi id="S2.SS4.p2.4.m4.2.2.2.2" xref="S2.SS4.p2.4.m4.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.4.m4.2b"><apply id="S2.SS4.p2.4.m4.2.3.cmml" xref="S2.SS4.p2.4.m4.2.3"><csymbol cd="ambiguous" id="S2.SS4.p2.4.m4.2.3.1.cmml" xref="S2.SS4.p2.4.m4.2.3">subscript</csymbol><ci id="S2.SS4.p2.4.m4.2.3.2.cmml" xref="S2.SS4.p2.4.m4.2.3.2">𝐏</ci><list id="S2.SS4.p2.4.m4.2.2.2.3.cmml" xref="S2.SS4.p2.4.m4.2.2.2.4"><ci id="S2.SS4.p2.4.m4.1.1.1.1.cmml" xref="S2.SS4.p2.4.m4.1.1.1.1">𝑖</ci><ci id="S2.SS4.p2.4.m4.2.2.2.2.cmml" xref="S2.SS4.p2.4.m4.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.4.m4.2c">\mathbf{P}_{i,j}</annotation></semantics></math> are computed and then written on the SRAM instead of the HBM, <math id="S2.SS4.p2.5.m5.2" class="ltx_Math" alttext="\mathbf{O}_{i,j}" display="inline"><semantics id="S2.SS4.p2.5.m5.2a"><msub id="S2.SS4.p2.5.m5.2.3" xref="S2.SS4.p2.5.m5.2.3.cmml"><mi id="S2.SS4.p2.5.m5.2.3.2" xref="S2.SS4.p2.5.m5.2.3.2.cmml">𝐎</mi><mrow id="S2.SS4.p2.5.m5.2.2.2.4" xref="S2.SS4.p2.5.m5.2.2.2.3.cmml"><mi id="S2.SS4.p2.5.m5.1.1.1.1" xref="S2.SS4.p2.5.m5.1.1.1.1.cmml">i</mi><mo id="S2.SS4.p2.5.m5.2.2.2.4.1" xref="S2.SS4.p2.5.m5.2.2.2.3.cmml">,</mo><mi id="S2.SS4.p2.5.m5.2.2.2.2" xref="S2.SS4.p2.5.m5.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.5.m5.2b"><apply id="S2.SS4.p2.5.m5.2.3.cmml" xref="S2.SS4.p2.5.m5.2.3"><csymbol cd="ambiguous" id="S2.SS4.p2.5.m5.2.3.1.cmml" xref="S2.SS4.p2.5.m5.2.3">subscript</csymbol><ci id="S2.SS4.p2.5.m5.2.3.2.cmml" xref="S2.SS4.p2.5.m5.2.3.2">𝐎</ci><list id="S2.SS4.p2.5.m5.2.2.2.3.cmml" xref="S2.SS4.p2.5.m5.2.2.2.4"><ci id="S2.SS4.p2.5.m5.1.1.1.1.cmml" xref="S2.SS4.p2.5.m5.1.1.1.1">𝑖</ci><ci id="S2.SS4.p2.5.m5.2.2.2.2.cmml" xref="S2.SS4.p2.5.m5.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.5.m5.2c">\mathbf{O}_{i,j}</annotation></semantics></math> are dynamically accumulated based on online softmax operations and written back to the HBM. Since the SRAM has a much higher I/O bandwidth than the HBM, the above optimization can make local attention operations more efficient.
Although the memory of the SRAM is tiny, further dividing <math id="S2.SS4.p2.6.m6.1" class="ltx_Math" alttext="\mathbf{Q}_{i}" display="inline"><semantics id="S2.SS4.p2.6.m6.1a"><msub id="S2.SS4.p2.6.m6.1.1" xref="S2.SS4.p2.6.m6.1.1.cmml"><mi id="S2.SS4.p2.6.m6.1.1.2" xref="S2.SS4.p2.6.m6.1.1.2.cmml">𝐐</mi><mi id="S2.SS4.p2.6.m6.1.1.3" xref="S2.SS4.p2.6.m6.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.6.m6.1b"><apply id="S2.SS4.p2.6.m6.1.1.cmml" xref="S2.SS4.p2.6.m6.1.1"><csymbol cd="ambiguous" id="S2.SS4.p2.6.m6.1.1.1.cmml" xref="S2.SS4.p2.6.m6.1.1">subscript</csymbol><ci id="S2.SS4.p2.6.m6.1.1.2.cmml" xref="S2.SS4.p2.6.m6.1.1.2">𝐐</ci><ci id="S2.SS4.p2.6.m6.1.1.3.cmml" xref="S2.SS4.p2.6.m6.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.6.m6.1c">\mathbf{Q}_{i}</annotation></semantics></math>, <math id="S2.SS4.p2.7.m7.1" class="ltx_Math" alttext="\mathbf{K}_{j}" display="inline"><semantics id="S2.SS4.p2.7.m7.1a"><msub id="S2.SS4.p2.7.m7.1.1" xref="S2.SS4.p2.7.m7.1.1.cmml"><mi id="S2.SS4.p2.7.m7.1.1.2" xref="S2.SS4.p2.7.m7.1.1.2.cmml">𝐊</mi><mi id="S2.SS4.p2.7.m7.1.1.3" xref="S2.SS4.p2.7.m7.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.7.m7.1b"><apply id="S2.SS4.p2.7.m7.1.1.cmml" xref="S2.SS4.p2.7.m7.1.1"><csymbol cd="ambiguous" id="S2.SS4.p2.7.m7.1.1.1.cmml" xref="S2.SS4.p2.7.m7.1.1">subscript</csymbol><ci id="S2.SS4.p2.7.m7.1.1.2.cmml" xref="S2.SS4.p2.7.m7.1.1.2">𝐊</ci><ci id="S2.SS4.p2.7.m7.1.1.3.cmml" xref="S2.SS4.p2.7.m7.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.7.m7.1c">\mathbf{K}_{j}</annotation></semantics></math>, and <math id="S2.SS4.p2.8.m8.1" class="ltx_Math" alttext="\mathbf{V}_{j}" display="inline"><semantics id="S2.SS4.p2.8.m8.1a"><msub id="S2.SS4.p2.8.m8.1.1" xref="S2.SS4.p2.8.m8.1.1.cmml"><mi id="S2.SS4.p2.8.m8.1.1.2" xref="S2.SS4.p2.8.m8.1.1.2.cmml">𝐕</mi><mi id="S2.SS4.p2.8.m8.1.1.3" xref="S2.SS4.p2.8.m8.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.8.m8.1b"><apply id="S2.SS4.p2.8.m8.1.1.cmml" xref="S2.SS4.p2.8.m8.1.1"><csymbol cd="ambiguous" id="S2.SS4.p2.8.m8.1.1.1.cmml" xref="S2.SS4.p2.8.m8.1.1">subscript</csymbol><ci id="S2.SS4.p2.8.m8.1.1.2.cmml" xref="S2.SS4.p2.8.m8.1.1.2">𝐕</ci><ci id="S2.SS4.p2.8.m8.1.1.3.cmml" xref="S2.SS4.p2.8.m8.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.8.m8.1c">\mathbf{V}_{j}</annotation></semantics></math> into many fine-grained tiles ensure the intermediate results <math id="S2.SS4.p2.9.m9.2" class="ltx_Math" alttext="\mathbf{S}_{i,j}" display="inline"><semantics id="S2.SS4.p2.9.m9.2a"><msub id="S2.SS4.p2.9.m9.2.3" xref="S2.SS4.p2.9.m9.2.3.cmml"><mi id="S2.SS4.p2.9.m9.2.3.2" xref="S2.SS4.p2.9.m9.2.3.2.cmml">𝐒</mi><mrow id="S2.SS4.p2.9.m9.2.2.2.4" xref="S2.SS4.p2.9.m9.2.2.2.3.cmml"><mi id="S2.SS4.p2.9.m9.1.1.1.1" xref="S2.SS4.p2.9.m9.1.1.1.1.cmml">i</mi><mo id="S2.SS4.p2.9.m9.2.2.2.4.1" xref="S2.SS4.p2.9.m9.2.2.2.3.cmml">,</mo><mi id="S2.SS4.p2.9.m9.2.2.2.2" xref="S2.SS4.p2.9.m9.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.9.m9.2b"><apply id="S2.SS4.p2.9.m9.2.3.cmml" xref="S2.SS4.p2.9.m9.2.3"><csymbol cd="ambiguous" id="S2.SS4.p2.9.m9.2.3.1.cmml" xref="S2.SS4.p2.9.m9.2.3">subscript</csymbol><ci id="S2.SS4.p2.9.m9.2.3.2.cmml" xref="S2.SS4.p2.9.m9.2.3.2">𝐒</ci><list id="S2.SS4.p2.9.m9.2.2.2.3.cmml" xref="S2.SS4.p2.9.m9.2.2.2.4"><ci id="S2.SS4.p2.9.m9.1.1.1.1.cmml" xref="S2.SS4.p2.9.m9.1.1.1.1">𝑖</ci><ci id="S2.SS4.p2.9.m9.2.2.2.2.cmml" xref="S2.SS4.p2.9.m9.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.9.m9.2c">\mathbf{S}_{i,j}</annotation></semantics></math> and <math id="S2.SS4.p2.10.m10.2" class="ltx_Math" alttext="\mathbf{P}_{i,j}" display="inline"><semantics id="S2.SS4.p2.10.m10.2a"><msub id="S2.SS4.p2.10.m10.2.3" xref="S2.SS4.p2.10.m10.2.3.cmml"><mi id="S2.SS4.p2.10.m10.2.3.2" xref="S2.SS4.p2.10.m10.2.3.2.cmml">𝐏</mi><mrow id="S2.SS4.p2.10.m10.2.2.2.4" xref="S2.SS4.p2.10.m10.2.2.2.3.cmml"><mi id="S2.SS4.p2.10.m10.1.1.1.1" xref="S2.SS4.p2.10.m10.1.1.1.1.cmml">i</mi><mo id="S2.SS4.p2.10.m10.2.2.2.4.1" xref="S2.SS4.p2.10.m10.2.2.2.3.cmml">,</mo><mi id="S2.SS4.p2.10.m10.2.2.2.2" xref="S2.SS4.p2.10.m10.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.10.m10.2b"><apply id="S2.SS4.p2.10.m10.2.3.cmml" xref="S2.SS4.p2.10.m10.2.3"><csymbol cd="ambiguous" id="S2.SS4.p2.10.m10.2.3.1.cmml" xref="S2.SS4.p2.10.m10.2.3">subscript</csymbol><ci id="S2.SS4.p2.10.m10.2.3.2.cmml" xref="S2.SS4.p2.10.m10.2.3.2">𝐏</ci><list id="S2.SS4.p2.10.m10.2.2.2.3.cmml" xref="S2.SS4.p2.10.m10.2.2.2.4"><ci id="S2.SS4.p2.10.m10.1.1.1.1.cmml" xref="S2.SS4.p2.10.m10.1.1.1.1">𝑖</ci><ci id="S2.SS4.p2.10.m10.2.2.2.2.cmml" xref="S2.SS4.p2.10.m10.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.10.m10.2c">\mathbf{P}_{i,j}</annotation></semantics></math> can be entirely stored into the SRAM.</p>
</div>
<div id="S2.SS4.p3" class="ltx_para">
<p id="S2.SS4.p3.1" class="ltx_p">Intuitively, when BurstAttention is running on a single device rather than a distributed cluster, there is no need to use GAO, and LAO will play the same role as FlashAttention <cite class="ltx_cite ltx_citemacro_citep">(Dao et al., <a href="#bib.bib7" title="" class="ltx_ref">2022</a>)</cite>, i.e., FlashAttention can be viewed as a specialization of BurstAttention using a single device.</p>
</div>
<figure id="S2.T1" class="ltx_table">
<div id="S2.T1.18" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:411.5pt;height:129.6pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-22.9pt,7.2pt) scale(0.9,0.9) ;">
<table id="S2.T1.18.18" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S2.T1.18.18.19.1" class="ltx_tr">
<td id="S2.T1.18.18.19.1.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_tt" rowspan="2"><span id="S2.T1.18.18.19.1.1.1" class="ltx_text" style="font-size:90%;">Method</span></td>
<td id="S2.T1.18.18.19.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt" rowspan="2"><span id="S2.T1.18.18.19.1.2.1" class="ltx_text" style="font-size:90%;">FlashATT/LAO</span></td>
<td id="S2.T1.18.18.19.1.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="2"><span id="S2.T1.18.18.19.1.3.1" class="ltx_text" style="font-size:90%;">Memory Overheads</span></td>
<td id="S2.T1.18.18.19.1.4" class="ltx_td ltx_align_center ltx_border_l ltx_border_tt" colspan="2"><span id="S2.T1.18.18.19.1.4.1" class="ltx_text" style="font-size:90%;">Communication Overheads</span></td>
</tr>
<tr id="S2.T1.18.18.20.2" class="ltx_tr">
<td id="S2.T1.18.18.20.2.1" class="ltx_td ltx_align_center ltx_border_l"><span id="S2.T1.18.18.20.2.1.1" class="ltx_text" style="font-size:90%;">Parameter</span></td>
<td id="S2.T1.18.18.20.2.2" class="ltx_td ltx_align_center"><span id="S2.T1.18.18.20.2.2.1" class="ltx_text" style="font-size:90%;">Activation</span></td>
<td id="S2.T1.18.18.20.2.3" class="ltx_td ltx_align_center ltx_border_l"><span id="S2.T1.18.18.20.2.3.1" class="ltx_text" style="font-size:90%;">Forward</span></td>
<td id="S2.T1.18.18.20.2.4" class="ltx_td ltx_align_center"><span id="S2.T1.18.18.20.2.4.1" class="ltx_text" style="font-size:90%;">Backward</span></td>
</tr>
<tr id="S2.T1.4.4.4" class="ltx_tr">
<td id="S2.T1.4.4.4.5" class="ltx_td ltx_align_left ltx_border_r ltx_border_t"><span id="S2.T1.4.4.4.5.1" class="ltx_text" style="font-size:90%;">RingAttention</span></td>
<td id="S2.T1.4.4.4.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S2.T1.4.4.4.6.1" class="ltx_text" style="font-size:90%;">w/o</span></td>
<td id="S2.T1.1.1.1.1" class="ltx_td ltx_align_center ltx_border_t"><math id="S2.T1.1.1.1.1.m1.1" class="ltx_Math" alttext="4HZd" display="inline"><semantics id="S2.T1.1.1.1.1.m1.1a"><mrow id="S2.T1.1.1.1.1.m1.1.1" xref="S2.T1.1.1.1.1.m1.1.1.cmml"><mn mathsize="90%" id="S2.T1.1.1.1.1.m1.1.1.2" xref="S2.T1.1.1.1.1.m1.1.1.2.cmml">4</mn><mo lspace="0em" rspace="0em" id="S2.T1.1.1.1.1.m1.1.1.1" xref="S2.T1.1.1.1.1.m1.1.1.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.1.1.1.1.m1.1.1.3" xref="S2.T1.1.1.1.1.m1.1.1.3.cmml">H</mi><mo lspace="0em" rspace="0em" id="S2.T1.1.1.1.1.m1.1.1.1a" xref="S2.T1.1.1.1.1.m1.1.1.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.1.1.1.1.m1.1.1.4" xref="S2.T1.1.1.1.1.m1.1.1.4.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S2.T1.1.1.1.1.m1.1.1.1b" xref="S2.T1.1.1.1.1.m1.1.1.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.1.1.1.1.m1.1.1.5" xref="S2.T1.1.1.1.1.m1.1.1.5.cmml">d</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.T1.1.1.1.1.m1.1b"><apply id="S2.T1.1.1.1.1.m1.1.1.cmml" xref="S2.T1.1.1.1.1.m1.1.1"><times id="S2.T1.1.1.1.1.m1.1.1.1.cmml" xref="S2.T1.1.1.1.1.m1.1.1.1"></times><cn type="integer" id="S2.T1.1.1.1.1.m1.1.1.2.cmml" xref="S2.T1.1.1.1.1.m1.1.1.2">4</cn><ci id="S2.T1.1.1.1.1.m1.1.1.3.cmml" xref="S2.T1.1.1.1.1.m1.1.1.3">𝐻</ci><ci id="S2.T1.1.1.1.1.m1.1.1.4.cmml" xref="S2.T1.1.1.1.1.m1.1.1.4">𝑍</ci><ci id="S2.T1.1.1.1.1.m1.1.1.5.cmml" xref="S2.T1.1.1.1.1.m1.1.1.5">𝑑</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.1.1.1.1.m1.1c">4HZd</annotation></semantics></math></td>
<td id="S2.T1.2.2.2.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><math id="S2.T1.2.2.2.2.m1.1" class="ltx_Math" alttext="{4\frac{BZNd}{G}}+\frac{BZN^{2}}{G}+{\frac{BNH}{G}}" display="inline"><semantics id="S2.T1.2.2.2.2.m1.1a"><mrow id="S2.T1.2.2.2.2.m1.1.1" xref="S2.T1.2.2.2.2.m1.1.1.cmml"><mrow id="S2.T1.2.2.2.2.m1.1.1.2" xref="S2.T1.2.2.2.2.m1.1.1.2.cmml"><mn mathsize="90%" id="S2.T1.2.2.2.2.m1.1.1.2.2" xref="S2.T1.2.2.2.2.m1.1.1.2.2.cmml">4</mn><mo lspace="0em" rspace="0em" id="S2.T1.2.2.2.2.m1.1.1.2.1" xref="S2.T1.2.2.2.2.m1.1.1.2.1.cmml">​</mo><mfrac id="S2.T1.2.2.2.2.m1.1.1.2.3" xref="S2.T1.2.2.2.2.m1.1.1.2.3.cmml"><mrow id="S2.T1.2.2.2.2.m1.1.1.2.3.2" xref="S2.T1.2.2.2.2.m1.1.1.2.3.2.cmml"><mi mathsize="90%" id="S2.T1.2.2.2.2.m1.1.1.2.3.2.2" xref="S2.T1.2.2.2.2.m1.1.1.2.3.2.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.2.2.2.2.m1.1.1.2.3.2.1" xref="S2.T1.2.2.2.2.m1.1.1.2.3.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.2.2.2.2.m1.1.1.2.3.2.3" xref="S2.T1.2.2.2.2.m1.1.1.2.3.2.3.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S2.T1.2.2.2.2.m1.1.1.2.3.2.1a" xref="S2.T1.2.2.2.2.m1.1.1.2.3.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.2.2.2.2.m1.1.1.2.3.2.4" xref="S2.T1.2.2.2.2.m1.1.1.2.3.2.4.cmml">N</mi><mo lspace="0em" rspace="0em" id="S2.T1.2.2.2.2.m1.1.1.2.3.2.1b" xref="S2.T1.2.2.2.2.m1.1.1.2.3.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.2.2.2.2.m1.1.1.2.3.2.5" xref="S2.T1.2.2.2.2.m1.1.1.2.3.2.5.cmml">d</mi></mrow><mi mathsize="90%" id="S2.T1.2.2.2.2.m1.1.1.2.3.3" xref="S2.T1.2.2.2.2.m1.1.1.2.3.3.cmml">G</mi></mfrac></mrow><mo mathsize="90%" id="S2.T1.2.2.2.2.m1.1.1.1" xref="S2.T1.2.2.2.2.m1.1.1.1.cmml">+</mo><mfrac id="S2.T1.2.2.2.2.m1.1.1.3" xref="S2.T1.2.2.2.2.m1.1.1.3.cmml"><mrow id="S2.T1.2.2.2.2.m1.1.1.3.2" xref="S2.T1.2.2.2.2.m1.1.1.3.2.cmml"><mi mathsize="90%" id="S2.T1.2.2.2.2.m1.1.1.3.2.2" xref="S2.T1.2.2.2.2.m1.1.1.3.2.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.2.2.2.2.m1.1.1.3.2.1" xref="S2.T1.2.2.2.2.m1.1.1.3.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.2.2.2.2.m1.1.1.3.2.3" xref="S2.T1.2.2.2.2.m1.1.1.3.2.3.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S2.T1.2.2.2.2.m1.1.1.3.2.1a" xref="S2.T1.2.2.2.2.m1.1.1.3.2.1.cmml">​</mo><msup id="S2.T1.2.2.2.2.m1.1.1.3.2.4" xref="S2.T1.2.2.2.2.m1.1.1.3.2.4.cmml"><mi mathsize="90%" id="S2.T1.2.2.2.2.m1.1.1.3.2.4.2" xref="S2.T1.2.2.2.2.m1.1.1.3.2.4.2.cmml">N</mi><mn mathsize="90%" id="S2.T1.2.2.2.2.m1.1.1.3.2.4.3" xref="S2.T1.2.2.2.2.m1.1.1.3.2.4.3.cmml">2</mn></msup></mrow><mi mathsize="90%" id="S2.T1.2.2.2.2.m1.1.1.3.3" xref="S2.T1.2.2.2.2.m1.1.1.3.3.cmml">G</mi></mfrac><mo mathsize="90%" id="S2.T1.2.2.2.2.m1.1.1.1a" xref="S2.T1.2.2.2.2.m1.1.1.1.cmml">+</mo><mfrac id="S2.T1.2.2.2.2.m1.1.1.4" xref="S2.T1.2.2.2.2.m1.1.1.4.cmml"><mrow id="S2.T1.2.2.2.2.m1.1.1.4.2" xref="S2.T1.2.2.2.2.m1.1.1.4.2.cmml"><mi mathsize="90%" id="S2.T1.2.2.2.2.m1.1.1.4.2.2" xref="S2.T1.2.2.2.2.m1.1.1.4.2.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.2.2.2.2.m1.1.1.4.2.1" xref="S2.T1.2.2.2.2.m1.1.1.4.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.2.2.2.2.m1.1.1.4.2.3" xref="S2.T1.2.2.2.2.m1.1.1.4.2.3.cmml">N</mi><mo lspace="0em" rspace="0em" id="S2.T1.2.2.2.2.m1.1.1.4.2.1a" xref="S2.T1.2.2.2.2.m1.1.1.4.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.2.2.2.2.m1.1.1.4.2.4" xref="S2.T1.2.2.2.2.m1.1.1.4.2.4.cmml">H</mi></mrow><mi mathsize="90%" id="S2.T1.2.2.2.2.m1.1.1.4.3" xref="S2.T1.2.2.2.2.m1.1.1.4.3.cmml">G</mi></mfrac></mrow><annotation-xml encoding="MathML-Content" id="S2.T1.2.2.2.2.m1.1b"><apply id="S2.T1.2.2.2.2.m1.1.1.cmml" xref="S2.T1.2.2.2.2.m1.1.1"><plus id="S2.T1.2.2.2.2.m1.1.1.1.cmml" xref="S2.T1.2.2.2.2.m1.1.1.1"></plus><apply id="S2.T1.2.2.2.2.m1.1.1.2.cmml" xref="S2.T1.2.2.2.2.m1.1.1.2"><times id="S2.T1.2.2.2.2.m1.1.1.2.1.cmml" xref="S2.T1.2.2.2.2.m1.1.1.2.1"></times><cn type="integer" id="S2.T1.2.2.2.2.m1.1.1.2.2.cmml" xref="S2.T1.2.2.2.2.m1.1.1.2.2">4</cn><apply id="S2.T1.2.2.2.2.m1.1.1.2.3.cmml" xref="S2.T1.2.2.2.2.m1.1.1.2.3"><divide id="S2.T1.2.2.2.2.m1.1.1.2.3.1.cmml" xref="S2.T1.2.2.2.2.m1.1.1.2.3"></divide><apply id="S2.T1.2.2.2.2.m1.1.1.2.3.2.cmml" xref="S2.T1.2.2.2.2.m1.1.1.2.3.2"><times id="S2.T1.2.2.2.2.m1.1.1.2.3.2.1.cmml" xref="S2.T1.2.2.2.2.m1.1.1.2.3.2.1"></times><ci id="S2.T1.2.2.2.2.m1.1.1.2.3.2.2.cmml" xref="S2.T1.2.2.2.2.m1.1.1.2.3.2.2">𝐵</ci><ci id="S2.T1.2.2.2.2.m1.1.1.2.3.2.3.cmml" xref="S2.T1.2.2.2.2.m1.1.1.2.3.2.3">𝑍</ci><ci id="S2.T1.2.2.2.2.m1.1.1.2.3.2.4.cmml" xref="S2.T1.2.2.2.2.m1.1.1.2.3.2.4">𝑁</ci><ci id="S2.T1.2.2.2.2.m1.1.1.2.3.2.5.cmml" xref="S2.T1.2.2.2.2.m1.1.1.2.3.2.5">𝑑</ci></apply><ci id="S2.T1.2.2.2.2.m1.1.1.2.3.3.cmml" xref="S2.T1.2.2.2.2.m1.1.1.2.3.3">𝐺</ci></apply></apply><apply id="S2.T1.2.2.2.2.m1.1.1.3.cmml" xref="S2.T1.2.2.2.2.m1.1.1.3"><divide id="S2.T1.2.2.2.2.m1.1.1.3.1.cmml" xref="S2.T1.2.2.2.2.m1.1.1.3"></divide><apply id="S2.T1.2.2.2.2.m1.1.1.3.2.cmml" xref="S2.T1.2.2.2.2.m1.1.1.3.2"><times id="S2.T1.2.2.2.2.m1.1.1.3.2.1.cmml" xref="S2.T1.2.2.2.2.m1.1.1.3.2.1"></times><ci id="S2.T1.2.2.2.2.m1.1.1.3.2.2.cmml" xref="S2.T1.2.2.2.2.m1.1.1.3.2.2">𝐵</ci><ci id="S2.T1.2.2.2.2.m1.1.1.3.2.3.cmml" xref="S2.T1.2.2.2.2.m1.1.1.3.2.3">𝑍</ci><apply id="S2.T1.2.2.2.2.m1.1.1.3.2.4.cmml" xref="S2.T1.2.2.2.2.m1.1.1.3.2.4"><csymbol cd="ambiguous" id="S2.T1.2.2.2.2.m1.1.1.3.2.4.1.cmml" xref="S2.T1.2.2.2.2.m1.1.1.3.2.4">superscript</csymbol><ci id="S2.T1.2.2.2.2.m1.1.1.3.2.4.2.cmml" xref="S2.T1.2.2.2.2.m1.1.1.3.2.4.2">𝑁</ci><cn type="integer" id="S2.T1.2.2.2.2.m1.1.1.3.2.4.3.cmml" xref="S2.T1.2.2.2.2.m1.1.1.3.2.4.3">2</cn></apply></apply><ci id="S2.T1.2.2.2.2.m1.1.1.3.3.cmml" xref="S2.T1.2.2.2.2.m1.1.1.3.3">𝐺</ci></apply><apply id="S2.T1.2.2.2.2.m1.1.1.4.cmml" xref="S2.T1.2.2.2.2.m1.1.1.4"><divide id="S2.T1.2.2.2.2.m1.1.1.4.1.cmml" xref="S2.T1.2.2.2.2.m1.1.1.4"></divide><apply id="S2.T1.2.2.2.2.m1.1.1.4.2.cmml" xref="S2.T1.2.2.2.2.m1.1.1.4.2"><times id="S2.T1.2.2.2.2.m1.1.1.4.2.1.cmml" xref="S2.T1.2.2.2.2.m1.1.1.4.2.1"></times><ci id="S2.T1.2.2.2.2.m1.1.1.4.2.2.cmml" xref="S2.T1.2.2.2.2.m1.1.1.4.2.2">𝐵</ci><ci id="S2.T1.2.2.2.2.m1.1.1.4.2.3.cmml" xref="S2.T1.2.2.2.2.m1.1.1.4.2.3">𝑁</ci><ci id="S2.T1.2.2.2.2.m1.1.1.4.2.4.cmml" xref="S2.T1.2.2.2.2.m1.1.1.4.2.4">𝐻</ci></apply><ci id="S2.T1.2.2.2.2.m1.1.1.4.3.cmml" xref="S2.T1.2.2.2.2.m1.1.1.4.3">𝐺</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.2.2.2.2.m1.1c">{4\frac{BZNd}{G}}+\frac{BZN^{2}}{G}+{\frac{BNH}{G}}</annotation></semantics></math></td>
<td id="S2.T1.3.3.3.3" class="ltx_td ltx_align_left ltx_border_l ltx_border_t" rowspan="2"><span id="S2.T1.3.3.3.3.1" class="ltx_text" style="font-size:90%;"><math id="S2.T1.3.3.3.3.1.m1.1" class="ltx_Math" alttext="2BZNd" display="inline"><semantics id="S2.T1.3.3.3.3.1.m1.1a"><mrow id="S2.T1.3.3.3.3.1.m1.1.1" xref="S2.T1.3.3.3.3.1.m1.1.1.cmml"><mn id="S2.T1.3.3.3.3.1.m1.1.1.2" xref="S2.T1.3.3.3.3.1.m1.1.1.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S2.T1.3.3.3.3.1.m1.1.1.1" xref="S2.T1.3.3.3.3.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.3.3.3.3.1.m1.1.1.3" xref="S2.T1.3.3.3.3.1.m1.1.1.3.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.3.3.3.3.1.m1.1.1.1a" xref="S2.T1.3.3.3.3.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.3.3.3.3.1.m1.1.1.4" xref="S2.T1.3.3.3.3.1.m1.1.1.4.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S2.T1.3.3.3.3.1.m1.1.1.1b" xref="S2.T1.3.3.3.3.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.3.3.3.3.1.m1.1.1.5" xref="S2.T1.3.3.3.3.1.m1.1.1.5.cmml">N</mi><mo lspace="0em" rspace="0em" id="S2.T1.3.3.3.3.1.m1.1.1.1c" xref="S2.T1.3.3.3.3.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.3.3.3.3.1.m1.1.1.6" xref="S2.T1.3.3.3.3.1.m1.1.1.6.cmml">d</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.T1.3.3.3.3.1.m1.1b"><apply id="S2.T1.3.3.3.3.1.m1.1.1.cmml" xref="S2.T1.3.3.3.3.1.m1.1.1"><times id="S2.T1.3.3.3.3.1.m1.1.1.1.cmml" xref="S2.T1.3.3.3.3.1.m1.1.1.1"></times><cn type="integer" id="S2.T1.3.3.3.3.1.m1.1.1.2.cmml" xref="S2.T1.3.3.3.3.1.m1.1.1.2">2</cn><ci id="S2.T1.3.3.3.3.1.m1.1.1.3.cmml" xref="S2.T1.3.3.3.3.1.m1.1.1.3">𝐵</ci><ci id="S2.T1.3.3.3.3.1.m1.1.1.4.cmml" xref="S2.T1.3.3.3.3.1.m1.1.1.4">𝑍</ci><ci id="S2.T1.3.3.3.3.1.m1.1.1.5.cmml" xref="S2.T1.3.3.3.3.1.m1.1.1.5">𝑁</ci><ci id="S2.T1.3.3.3.3.1.m1.1.1.6.cmml" xref="S2.T1.3.3.3.3.1.m1.1.1.6">𝑑</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.3.3.3.3.1.m1.1c">2BZNd</annotation></semantics></math></span></td>
<td id="S2.T1.4.4.4.4" class="ltx_td ltx_align_center ltx_border_l ltx_border_t" rowspan="2"><span id="S2.T1.4.4.4.4.1" class="ltx_text" style="font-size:90%;"><math id="S2.T1.4.4.4.4.1.m1.1" class="ltx_Math" alttext="6BZNd" display="inline"><semantics id="S2.T1.4.4.4.4.1.m1.1a"><mrow id="S2.T1.4.4.4.4.1.m1.1.1" xref="S2.T1.4.4.4.4.1.m1.1.1.cmml"><mn id="S2.T1.4.4.4.4.1.m1.1.1.2" xref="S2.T1.4.4.4.4.1.m1.1.1.2.cmml">6</mn><mo lspace="0em" rspace="0em" id="S2.T1.4.4.4.4.1.m1.1.1.1" xref="S2.T1.4.4.4.4.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.4.4.4.4.1.m1.1.1.3" xref="S2.T1.4.4.4.4.1.m1.1.1.3.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.4.4.4.4.1.m1.1.1.1a" xref="S2.T1.4.4.4.4.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.4.4.4.4.1.m1.1.1.4" xref="S2.T1.4.4.4.4.1.m1.1.1.4.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S2.T1.4.4.4.4.1.m1.1.1.1b" xref="S2.T1.4.4.4.4.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.4.4.4.4.1.m1.1.1.5" xref="S2.T1.4.4.4.4.1.m1.1.1.5.cmml">N</mi><mo lspace="0em" rspace="0em" id="S2.T1.4.4.4.4.1.m1.1.1.1c" xref="S2.T1.4.4.4.4.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.4.4.4.4.1.m1.1.1.6" xref="S2.T1.4.4.4.4.1.m1.1.1.6.cmml">d</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.T1.4.4.4.4.1.m1.1b"><apply id="S2.T1.4.4.4.4.1.m1.1.1.cmml" xref="S2.T1.4.4.4.4.1.m1.1.1"><times id="S2.T1.4.4.4.4.1.m1.1.1.1.cmml" xref="S2.T1.4.4.4.4.1.m1.1.1.1"></times><cn type="integer" id="S2.T1.4.4.4.4.1.m1.1.1.2.cmml" xref="S2.T1.4.4.4.4.1.m1.1.1.2">6</cn><ci id="S2.T1.4.4.4.4.1.m1.1.1.3.cmml" xref="S2.T1.4.4.4.4.1.m1.1.1.3">𝐵</ci><ci id="S2.T1.4.4.4.4.1.m1.1.1.4.cmml" xref="S2.T1.4.4.4.4.1.m1.1.1.4">𝑍</ci><ci id="S2.T1.4.4.4.4.1.m1.1.1.5.cmml" xref="S2.T1.4.4.4.4.1.m1.1.1.5">𝑁</ci><ci id="S2.T1.4.4.4.4.1.m1.1.1.6.cmml" xref="S2.T1.4.4.4.4.1.m1.1.1.6">𝑑</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.4.4.4.4.1.m1.1c">6BZNd</annotation></semantics></math></span></td>
</tr>
<tr id="S2.T1.8.8.8" class="ltx_tr">
<td id="S2.T1.5.5.5.1" class="ltx_td ltx_align_left ltx_border_r">
<span id="S2.T1.5.5.5.1.1" class="ltx_text" style="font-size:90%;">RingAttention</span><sup id="S2.T1.5.5.5.1.2" class="ltx_sup"><span id="S2.T1.5.5.5.1.2.1" class="ltx_text ltx_font_italic" style="font-size:90%;">†</span></sup>
</td>
<td id="S2.T1.6.6.6.2" class="ltx_td ltx_align_center ltx_border_r"><math id="S2.T1.6.6.6.2.m1.1" class="ltx_Math" alttext="-" display="inline"><semantics id="S2.T1.6.6.6.2.m1.1a"><mo mathsize="90%" id="S2.T1.6.6.6.2.m1.1.1" xref="S2.T1.6.6.6.2.m1.1.1.cmml">−</mo><annotation-xml encoding="MathML-Content" id="S2.T1.6.6.6.2.m1.1b"><minus id="S2.T1.6.6.6.2.m1.1.1.cmml" xref="S2.T1.6.6.6.2.m1.1.1"></minus></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.6.6.6.2.m1.1c">-</annotation></semantics></math></td>
<td id="S2.T1.7.7.7.3" class="ltx_td ltx_align_center"><math id="S2.T1.7.7.7.3.m1.1" class="ltx_Math" alttext="-" display="inline"><semantics id="S2.T1.7.7.7.3.m1.1a"><mo mathsize="90%" id="S2.T1.7.7.7.3.m1.1.1" xref="S2.T1.7.7.7.3.m1.1.1.cmml">−</mo><annotation-xml encoding="MathML-Content" id="S2.T1.7.7.7.3.m1.1b"><minus id="S2.T1.7.7.7.3.m1.1.1.cmml" xref="S2.T1.7.7.7.3.m1.1.1"></minus></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.7.7.7.3.m1.1c">-</annotation></semantics></math></td>
<td id="S2.T1.8.8.8.4" class="ltx_td ltx_align_center ltx_border_r"><math id="S2.T1.8.8.8.4.m1.1" class="ltx_Math" alttext="-" display="inline"><semantics id="S2.T1.8.8.8.4.m1.1a"><mo mathsize="90%" id="S2.T1.8.8.8.4.m1.1.1" xref="S2.T1.8.8.8.4.m1.1.1.cmml">−</mo><annotation-xml encoding="MathML-Content" id="S2.T1.8.8.8.4.m1.1b"><minus id="S2.T1.8.8.8.4.m1.1.1.cmml" xref="S2.T1.8.8.8.4.m1.1.1"></minus></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.8.8.8.4.m1.1c">-</annotation></semantics></math></td>
</tr>
<tr id="S2.T1.12.12.12" class="ltx_tr">
<td id="S2.T1.12.12.12.5" class="ltx_td ltx_align_left ltx_border_r ltx_border_t"><span id="S2.T1.12.12.12.5.1" class="ltx_text" style="font-size:90%;">Tensor Parallelism</span></td>
<td id="S2.T1.12.12.12.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S2.T1.12.12.12.6.1" class="ltx_text" style="font-size:90%;">w/o</span></td>
<td id="S2.T1.9.9.9.1" class="ltx_td ltx_align_center ltx_border_t" rowspan="2"><span id="S2.T1.9.9.9.1.1" class="ltx_text" style="font-size:90%;"><math id="S2.T1.9.9.9.1.1.m1.1" class="ltx_Math" alttext="{4\frac{HZd}{G}}" display="inline"><semantics id="S2.T1.9.9.9.1.1.m1.1a"><mrow id="S2.T1.9.9.9.1.1.m1.1.1" xref="S2.T1.9.9.9.1.1.m1.1.1.cmml"><mn id="S2.T1.9.9.9.1.1.m1.1.1.2" xref="S2.T1.9.9.9.1.1.m1.1.1.2.cmml">4</mn><mo lspace="0em" rspace="0em" id="S2.T1.9.9.9.1.1.m1.1.1.1" xref="S2.T1.9.9.9.1.1.m1.1.1.1.cmml">​</mo><mfrac id="S2.T1.9.9.9.1.1.m1.1.1.3" xref="S2.T1.9.9.9.1.1.m1.1.1.3.cmml"><mrow id="S2.T1.9.9.9.1.1.m1.1.1.3.2" xref="S2.T1.9.9.9.1.1.m1.1.1.3.2.cmml"><mi id="S2.T1.9.9.9.1.1.m1.1.1.3.2.2" xref="S2.T1.9.9.9.1.1.m1.1.1.3.2.2.cmml">H</mi><mo lspace="0em" rspace="0em" id="S2.T1.9.9.9.1.1.m1.1.1.3.2.1" xref="S2.T1.9.9.9.1.1.m1.1.1.3.2.1.cmml">​</mo><mi id="S2.T1.9.9.9.1.1.m1.1.1.3.2.3" xref="S2.T1.9.9.9.1.1.m1.1.1.3.2.3.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S2.T1.9.9.9.1.1.m1.1.1.3.2.1a" xref="S2.T1.9.9.9.1.1.m1.1.1.3.2.1.cmml">​</mo><mi id="S2.T1.9.9.9.1.1.m1.1.1.3.2.4" xref="S2.T1.9.9.9.1.1.m1.1.1.3.2.4.cmml">d</mi></mrow><mi id="S2.T1.9.9.9.1.1.m1.1.1.3.3" xref="S2.T1.9.9.9.1.1.m1.1.1.3.3.cmml">G</mi></mfrac></mrow><annotation-xml encoding="MathML-Content" id="S2.T1.9.9.9.1.1.m1.1b"><apply id="S2.T1.9.9.9.1.1.m1.1.1.cmml" xref="S2.T1.9.9.9.1.1.m1.1.1"><times id="S2.T1.9.9.9.1.1.m1.1.1.1.cmml" xref="S2.T1.9.9.9.1.1.m1.1.1.1"></times><cn type="integer" id="S2.T1.9.9.9.1.1.m1.1.1.2.cmml" xref="S2.T1.9.9.9.1.1.m1.1.1.2">4</cn><apply id="S2.T1.9.9.9.1.1.m1.1.1.3.cmml" xref="S2.T1.9.9.9.1.1.m1.1.1.3"><divide id="S2.T1.9.9.9.1.1.m1.1.1.3.1.cmml" xref="S2.T1.9.9.9.1.1.m1.1.1.3"></divide><apply id="S2.T1.9.9.9.1.1.m1.1.1.3.2.cmml" xref="S2.T1.9.9.9.1.1.m1.1.1.3.2"><times id="S2.T1.9.9.9.1.1.m1.1.1.3.2.1.cmml" xref="S2.T1.9.9.9.1.1.m1.1.1.3.2.1"></times><ci id="S2.T1.9.9.9.1.1.m1.1.1.3.2.2.cmml" xref="S2.T1.9.9.9.1.1.m1.1.1.3.2.2">𝐻</ci><ci id="S2.T1.9.9.9.1.1.m1.1.1.3.2.3.cmml" xref="S2.T1.9.9.9.1.1.m1.1.1.3.2.3">𝑍</ci><ci id="S2.T1.9.9.9.1.1.m1.1.1.3.2.4.cmml" xref="S2.T1.9.9.9.1.1.m1.1.1.3.2.4">𝑑</ci></apply><ci id="S2.T1.9.9.9.1.1.m1.1.1.3.3.cmml" xref="S2.T1.9.9.9.1.1.m1.1.1.3.3">𝐺</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.9.9.9.1.1.m1.1c">{4\frac{HZd}{G}}</annotation></semantics></math></span></td>
<td id="S2.T1.10.10.10.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><math id="S2.T1.10.10.10.2.m1.1" class="ltx_Math" alttext="{4\frac{BZNd}{G}}+\frac{BZN^{2}}{G}+BNH" display="inline"><semantics id="S2.T1.10.10.10.2.m1.1a"><mrow id="S2.T1.10.10.10.2.m1.1.1" xref="S2.T1.10.10.10.2.m1.1.1.cmml"><mrow id="S2.T1.10.10.10.2.m1.1.1.2" xref="S2.T1.10.10.10.2.m1.1.1.2.cmml"><mn mathsize="90%" id="S2.T1.10.10.10.2.m1.1.1.2.2" xref="S2.T1.10.10.10.2.m1.1.1.2.2.cmml">4</mn><mo lspace="0em" rspace="0em" id="S2.T1.10.10.10.2.m1.1.1.2.1" xref="S2.T1.10.10.10.2.m1.1.1.2.1.cmml">​</mo><mfrac id="S2.T1.10.10.10.2.m1.1.1.2.3" xref="S2.T1.10.10.10.2.m1.1.1.2.3.cmml"><mrow id="S2.T1.10.10.10.2.m1.1.1.2.3.2" xref="S2.T1.10.10.10.2.m1.1.1.2.3.2.cmml"><mi mathsize="90%" id="S2.T1.10.10.10.2.m1.1.1.2.3.2.2" xref="S2.T1.10.10.10.2.m1.1.1.2.3.2.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.10.10.10.2.m1.1.1.2.3.2.1" xref="S2.T1.10.10.10.2.m1.1.1.2.3.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.10.10.10.2.m1.1.1.2.3.2.3" xref="S2.T1.10.10.10.2.m1.1.1.2.3.2.3.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S2.T1.10.10.10.2.m1.1.1.2.3.2.1a" xref="S2.T1.10.10.10.2.m1.1.1.2.3.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.10.10.10.2.m1.1.1.2.3.2.4" xref="S2.T1.10.10.10.2.m1.1.1.2.3.2.4.cmml">N</mi><mo lspace="0em" rspace="0em" id="S2.T1.10.10.10.2.m1.1.1.2.3.2.1b" xref="S2.T1.10.10.10.2.m1.1.1.2.3.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.10.10.10.2.m1.1.1.2.3.2.5" xref="S2.T1.10.10.10.2.m1.1.1.2.3.2.5.cmml">d</mi></mrow><mi mathsize="90%" id="S2.T1.10.10.10.2.m1.1.1.2.3.3" xref="S2.T1.10.10.10.2.m1.1.1.2.3.3.cmml">G</mi></mfrac></mrow><mo mathsize="90%" id="S2.T1.10.10.10.2.m1.1.1.1" xref="S2.T1.10.10.10.2.m1.1.1.1.cmml">+</mo><mfrac id="S2.T1.10.10.10.2.m1.1.1.3" xref="S2.T1.10.10.10.2.m1.1.1.3.cmml"><mrow id="S2.T1.10.10.10.2.m1.1.1.3.2" xref="S2.T1.10.10.10.2.m1.1.1.3.2.cmml"><mi mathsize="90%" id="S2.T1.10.10.10.2.m1.1.1.3.2.2" xref="S2.T1.10.10.10.2.m1.1.1.3.2.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.10.10.10.2.m1.1.1.3.2.1" xref="S2.T1.10.10.10.2.m1.1.1.3.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.10.10.10.2.m1.1.1.3.2.3" xref="S2.T1.10.10.10.2.m1.1.1.3.2.3.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S2.T1.10.10.10.2.m1.1.1.3.2.1a" xref="S2.T1.10.10.10.2.m1.1.1.3.2.1.cmml">​</mo><msup id="S2.T1.10.10.10.2.m1.1.1.3.2.4" xref="S2.T1.10.10.10.2.m1.1.1.3.2.4.cmml"><mi mathsize="90%" id="S2.T1.10.10.10.2.m1.1.1.3.2.4.2" xref="S2.T1.10.10.10.2.m1.1.1.3.2.4.2.cmml">N</mi><mn mathsize="90%" id="S2.T1.10.10.10.2.m1.1.1.3.2.4.3" xref="S2.T1.10.10.10.2.m1.1.1.3.2.4.3.cmml">2</mn></msup></mrow><mi mathsize="90%" id="S2.T1.10.10.10.2.m1.1.1.3.3" xref="S2.T1.10.10.10.2.m1.1.1.3.3.cmml">G</mi></mfrac><mo mathsize="90%" id="S2.T1.10.10.10.2.m1.1.1.1a" xref="S2.T1.10.10.10.2.m1.1.1.1.cmml">+</mo><mrow id="S2.T1.10.10.10.2.m1.1.1.4" xref="S2.T1.10.10.10.2.m1.1.1.4.cmml"><mi mathsize="90%" id="S2.T1.10.10.10.2.m1.1.1.4.2" xref="S2.T1.10.10.10.2.m1.1.1.4.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.10.10.10.2.m1.1.1.4.1" xref="S2.T1.10.10.10.2.m1.1.1.4.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.10.10.10.2.m1.1.1.4.3" xref="S2.T1.10.10.10.2.m1.1.1.4.3.cmml">N</mi><mo lspace="0em" rspace="0em" id="S2.T1.10.10.10.2.m1.1.1.4.1a" xref="S2.T1.10.10.10.2.m1.1.1.4.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.10.10.10.2.m1.1.1.4.4" xref="S2.T1.10.10.10.2.m1.1.1.4.4.cmml">H</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.T1.10.10.10.2.m1.1b"><apply id="S2.T1.10.10.10.2.m1.1.1.cmml" xref="S2.T1.10.10.10.2.m1.1.1"><plus id="S2.T1.10.10.10.2.m1.1.1.1.cmml" xref="S2.T1.10.10.10.2.m1.1.1.1"></plus><apply id="S2.T1.10.10.10.2.m1.1.1.2.cmml" xref="S2.T1.10.10.10.2.m1.1.1.2"><times id="S2.T1.10.10.10.2.m1.1.1.2.1.cmml" xref="S2.T1.10.10.10.2.m1.1.1.2.1"></times><cn type="integer" id="S2.T1.10.10.10.2.m1.1.1.2.2.cmml" xref="S2.T1.10.10.10.2.m1.1.1.2.2">4</cn><apply id="S2.T1.10.10.10.2.m1.1.1.2.3.cmml" xref="S2.T1.10.10.10.2.m1.1.1.2.3"><divide id="S2.T1.10.10.10.2.m1.1.1.2.3.1.cmml" xref="S2.T1.10.10.10.2.m1.1.1.2.3"></divide><apply id="S2.T1.10.10.10.2.m1.1.1.2.3.2.cmml" xref="S2.T1.10.10.10.2.m1.1.1.2.3.2"><times id="S2.T1.10.10.10.2.m1.1.1.2.3.2.1.cmml" xref="S2.T1.10.10.10.2.m1.1.1.2.3.2.1"></times><ci id="S2.T1.10.10.10.2.m1.1.1.2.3.2.2.cmml" xref="S2.T1.10.10.10.2.m1.1.1.2.3.2.2">𝐵</ci><ci id="S2.T1.10.10.10.2.m1.1.1.2.3.2.3.cmml" xref="S2.T1.10.10.10.2.m1.1.1.2.3.2.3">𝑍</ci><ci id="S2.T1.10.10.10.2.m1.1.1.2.3.2.4.cmml" xref="S2.T1.10.10.10.2.m1.1.1.2.3.2.4">𝑁</ci><ci id="S2.T1.10.10.10.2.m1.1.1.2.3.2.5.cmml" xref="S2.T1.10.10.10.2.m1.1.1.2.3.2.5">𝑑</ci></apply><ci id="S2.T1.10.10.10.2.m1.1.1.2.3.3.cmml" xref="S2.T1.10.10.10.2.m1.1.1.2.3.3">𝐺</ci></apply></apply><apply id="S2.T1.10.10.10.2.m1.1.1.3.cmml" xref="S2.T1.10.10.10.2.m1.1.1.3"><divide id="S2.T1.10.10.10.2.m1.1.1.3.1.cmml" xref="S2.T1.10.10.10.2.m1.1.1.3"></divide><apply id="S2.T1.10.10.10.2.m1.1.1.3.2.cmml" xref="S2.T1.10.10.10.2.m1.1.1.3.2"><times id="S2.T1.10.10.10.2.m1.1.1.3.2.1.cmml" xref="S2.T1.10.10.10.2.m1.1.1.3.2.1"></times><ci id="S2.T1.10.10.10.2.m1.1.1.3.2.2.cmml" xref="S2.T1.10.10.10.2.m1.1.1.3.2.2">𝐵</ci><ci id="S2.T1.10.10.10.2.m1.1.1.3.2.3.cmml" xref="S2.T1.10.10.10.2.m1.1.1.3.2.3">𝑍</ci><apply id="S2.T1.10.10.10.2.m1.1.1.3.2.4.cmml" xref="S2.T1.10.10.10.2.m1.1.1.3.2.4"><csymbol cd="ambiguous" id="S2.T1.10.10.10.2.m1.1.1.3.2.4.1.cmml" xref="S2.T1.10.10.10.2.m1.1.1.3.2.4">superscript</csymbol><ci id="S2.T1.10.10.10.2.m1.1.1.3.2.4.2.cmml" xref="S2.T1.10.10.10.2.m1.1.1.3.2.4.2">𝑁</ci><cn type="integer" id="S2.T1.10.10.10.2.m1.1.1.3.2.4.3.cmml" xref="S2.T1.10.10.10.2.m1.1.1.3.2.4.3">2</cn></apply></apply><ci id="S2.T1.10.10.10.2.m1.1.1.3.3.cmml" xref="S2.T1.10.10.10.2.m1.1.1.3.3">𝐺</ci></apply><apply id="S2.T1.10.10.10.2.m1.1.1.4.cmml" xref="S2.T1.10.10.10.2.m1.1.1.4"><times id="S2.T1.10.10.10.2.m1.1.1.4.1.cmml" xref="S2.T1.10.10.10.2.m1.1.1.4.1"></times><ci id="S2.T1.10.10.10.2.m1.1.1.4.2.cmml" xref="S2.T1.10.10.10.2.m1.1.1.4.2">𝐵</ci><ci id="S2.T1.10.10.10.2.m1.1.1.4.3.cmml" xref="S2.T1.10.10.10.2.m1.1.1.4.3">𝑁</ci><ci id="S2.T1.10.10.10.2.m1.1.1.4.4.cmml" xref="S2.T1.10.10.10.2.m1.1.1.4.4">𝐻</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.10.10.10.2.m1.1c">{4\frac{BZNd}{G}}+\frac{BZN^{2}}{G}+BNH</annotation></semantics></math></td>
<td id="S2.T1.11.11.11.3" class="ltx_td ltx_align_left ltx_border_l ltx_border_t" rowspan="2"><span id="S2.T1.11.11.11.3.1" class="ltx_text" style="font-size:90%;"><math id="S2.T1.11.11.11.3.1.m1.1" class="ltx_Math" alttext="4BZNd" display="inline"><semantics id="S2.T1.11.11.11.3.1.m1.1a"><mrow id="S2.T1.11.11.11.3.1.m1.1.1" xref="S2.T1.11.11.11.3.1.m1.1.1.cmml"><mn id="S2.T1.11.11.11.3.1.m1.1.1.2" xref="S2.T1.11.11.11.3.1.m1.1.1.2.cmml">4</mn><mo lspace="0em" rspace="0em" id="S2.T1.11.11.11.3.1.m1.1.1.1" xref="S2.T1.11.11.11.3.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.11.11.11.3.1.m1.1.1.3" xref="S2.T1.11.11.11.3.1.m1.1.1.3.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.11.11.11.3.1.m1.1.1.1a" xref="S2.T1.11.11.11.3.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.11.11.11.3.1.m1.1.1.4" xref="S2.T1.11.11.11.3.1.m1.1.1.4.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S2.T1.11.11.11.3.1.m1.1.1.1b" xref="S2.T1.11.11.11.3.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.11.11.11.3.1.m1.1.1.5" xref="S2.T1.11.11.11.3.1.m1.1.1.5.cmml">N</mi><mo lspace="0em" rspace="0em" id="S2.T1.11.11.11.3.1.m1.1.1.1c" xref="S2.T1.11.11.11.3.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.11.11.11.3.1.m1.1.1.6" xref="S2.T1.11.11.11.3.1.m1.1.1.6.cmml">d</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.T1.11.11.11.3.1.m1.1b"><apply id="S2.T1.11.11.11.3.1.m1.1.1.cmml" xref="S2.T1.11.11.11.3.1.m1.1.1"><times id="S2.T1.11.11.11.3.1.m1.1.1.1.cmml" xref="S2.T1.11.11.11.3.1.m1.1.1.1"></times><cn type="integer" id="S2.T1.11.11.11.3.1.m1.1.1.2.cmml" xref="S2.T1.11.11.11.3.1.m1.1.1.2">4</cn><ci id="S2.T1.11.11.11.3.1.m1.1.1.3.cmml" xref="S2.T1.11.11.11.3.1.m1.1.1.3">𝐵</ci><ci id="S2.T1.11.11.11.3.1.m1.1.1.4.cmml" xref="S2.T1.11.11.11.3.1.m1.1.1.4">𝑍</ci><ci id="S2.T1.11.11.11.3.1.m1.1.1.5.cmml" xref="S2.T1.11.11.11.3.1.m1.1.1.5">𝑁</ci><ci id="S2.T1.11.11.11.3.1.m1.1.1.6.cmml" xref="S2.T1.11.11.11.3.1.m1.1.1.6">𝑑</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.11.11.11.3.1.m1.1c">4BZNd</annotation></semantics></math></span></td>
<td id="S2.T1.12.12.12.4" class="ltx_td ltx_align_center ltx_border_l ltx_border_t" rowspan="2"><span id="S2.T1.12.12.12.4.1" class="ltx_text" style="font-size:90%;"><math id="S2.T1.12.12.12.4.1.m1.1" class="ltx_Math" alttext="4BZNd" display="inline"><semantics id="S2.T1.12.12.12.4.1.m1.1a"><mrow id="S2.T1.12.12.12.4.1.m1.1.1" xref="S2.T1.12.12.12.4.1.m1.1.1.cmml"><mn id="S2.T1.12.12.12.4.1.m1.1.1.2" xref="S2.T1.12.12.12.4.1.m1.1.1.2.cmml">4</mn><mo lspace="0em" rspace="0em" id="S2.T1.12.12.12.4.1.m1.1.1.1" xref="S2.T1.12.12.12.4.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.12.12.12.4.1.m1.1.1.3" xref="S2.T1.12.12.12.4.1.m1.1.1.3.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.12.12.12.4.1.m1.1.1.1a" xref="S2.T1.12.12.12.4.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.12.12.12.4.1.m1.1.1.4" xref="S2.T1.12.12.12.4.1.m1.1.1.4.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S2.T1.12.12.12.4.1.m1.1.1.1b" xref="S2.T1.12.12.12.4.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.12.12.12.4.1.m1.1.1.5" xref="S2.T1.12.12.12.4.1.m1.1.1.5.cmml">N</mi><mo lspace="0em" rspace="0em" id="S2.T1.12.12.12.4.1.m1.1.1.1c" xref="S2.T1.12.12.12.4.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.12.12.12.4.1.m1.1.1.6" xref="S2.T1.12.12.12.4.1.m1.1.1.6.cmml">d</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.T1.12.12.12.4.1.m1.1b"><apply id="S2.T1.12.12.12.4.1.m1.1.1.cmml" xref="S2.T1.12.12.12.4.1.m1.1.1"><times id="S2.T1.12.12.12.4.1.m1.1.1.1.cmml" xref="S2.T1.12.12.12.4.1.m1.1.1.1"></times><cn type="integer" id="S2.T1.12.12.12.4.1.m1.1.1.2.cmml" xref="S2.T1.12.12.12.4.1.m1.1.1.2">4</cn><ci id="S2.T1.12.12.12.4.1.m1.1.1.3.cmml" xref="S2.T1.12.12.12.4.1.m1.1.1.3">𝐵</ci><ci id="S2.T1.12.12.12.4.1.m1.1.1.4.cmml" xref="S2.T1.12.12.12.4.1.m1.1.1.4">𝑍</ci><ci id="S2.T1.12.12.12.4.1.m1.1.1.5.cmml" xref="S2.T1.12.12.12.4.1.m1.1.1.5">𝑁</ci><ci id="S2.T1.12.12.12.4.1.m1.1.1.6.cmml" xref="S2.T1.12.12.12.4.1.m1.1.1.6">𝑑</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.12.12.12.4.1.m1.1c">4BZNd</annotation></semantics></math></span></td>
</tr>
<tr id="S2.T1.13.13.13" class="ltx_tr">
<td id="S2.T1.13.13.13.2" class="ltx_td ltx_align_left ltx_border_r"><span id="S2.T1.13.13.13.2.1" class="ltx_text" style="font-size:90%;">Tensor Parallelism</span></td>
<td id="S2.T1.13.13.13.3" class="ltx_td ltx_align_center ltx_border_r"><span id="S2.T1.13.13.13.3.1" class="ltx_text" style="font-size:90%;">w/ FlashATT</span></td>
<td id="S2.T1.13.13.13.1" class="ltx_td ltx_align_center ltx_border_r"><math id="S2.T1.13.13.13.1.m1.1" class="ltx_Math" alttext="{4\frac{BZNd}{G}}+\frac{BZN^{2}}{(M/4d)^{2}G}+BNH" display="inline"><semantics id="S2.T1.13.13.13.1.m1.1a"><mrow id="S2.T1.13.13.13.1.m1.1.2" xref="S2.T1.13.13.13.1.m1.1.2.cmml"><mrow id="S2.T1.13.13.13.1.m1.1.2.2" xref="S2.T1.13.13.13.1.m1.1.2.2.cmml"><mn mathsize="90%" id="S2.T1.13.13.13.1.m1.1.2.2.2" xref="S2.T1.13.13.13.1.m1.1.2.2.2.cmml">4</mn><mo lspace="0em" rspace="0em" id="S2.T1.13.13.13.1.m1.1.2.2.1" xref="S2.T1.13.13.13.1.m1.1.2.2.1.cmml">​</mo><mfrac id="S2.T1.13.13.13.1.m1.1.2.2.3" xref="S2.T1.13.13.13.1.m1.1.2.2.3.cmml"><mrow id="S2.T1.13.13.13.1.m1.1.2.2.3.2" xref="S2.T1.13.13.13.1.m1.1.2.2.3.2.cmml"><mi mathsize="90%" id="S2.T1.13.13.13.1.m1.1.2.2.3.2.2" xref="S2.T1.13.13.13.1.m1.1.2.2.3.2.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.13.13.13.1.m1.1.2.2.3.2.1" xref="S2.T1.13.13.13.1.m1.1.2.2.3.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.13.13.13.1.m1.1.2.2.3.2.3" xref="S2.T1.13.13.13.1.m1.1.2.2.3.2.3.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S2.T1.13.13.13.1.m1.1.2.2.3.2.1a" xref="S2.T1.13.13.13.1.m1.1.2.2.3.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.13.13.13.1.m1.1.2.2.3.2.4" xref="S2.T1.13.13.13.1.m1.1.2.2.3.2.4.cmml">N</mi><mo lspace="0em" rspace="0em" id="S2.T1.13.13.13.1.m1.1.2.2.3.2.1b" xref="S2.T1.13.13.13.1.m1.1.2.2.3.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.13.13.13.1.m1.1.2.2.3.2.5" xref="S2.T1.13.13.13.1.m1.1.2.2.3.2.5.cmml">d</mi></mrow><mi mathsize="90%" id="S2.T1.13.13.13.1.m1.1.2.2.3.3" xref="S2.T1.13.13.13.1.m1.1.2.2.3.3.cmml">G</mi></mfrac></mrow><mo mathsize="90%" id="S2.T1.13.13.13.1.m1.1.2.1" xref="S2.T1.13.13.13.1.m1.1.2.1.cmml">+</mo><mfrac id="S2.T1.13.13.13.1.m1.1.1" xref="S2.T1.13.13.13.1.m1.1.1.cmml"><mrow id="S2.T1.13.13.13.1.m1.1.1.3" xref="S2.T1.13.13.13.1.m1.1.1.3.cmml"><mi mathsize="90%" id="S2.T1.13.13.13.1.m1.1.1.3.2" xref="S2.T1.13.13.13.1.m1.1.1.3.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.13.13.13.1.m1.1.1.3.1" xref="S2.T1.13.13.13.1.m1.1.1.3.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.13.13.13.1.m1.1.1.3.3" xref="S2.T1.13.13.13.1.m1.1.1.3.3.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S2.T1.13.13.13.1.m1.1.1.3.1a" xref="S2.T1.13.13.13.1.m1.1.1.3.1.cmml">​</mo><msup id="S2.T1.13.13.13.1.m1.1.1.3.4" xref="S2.T1.13.13.13.1.m1.1.1.3.4.cmml"><mi mathsize="90%" id="S2.T1.13.13.13.1.m1.1.1.3.4.2" xref="S2.T1.13.13.13.1.m1.1.1.3.4.2.cmml">N</mi><mn mathsize="90%" id="S2.T1.13.13.13.1.m1.1.1.3.4.3" xref="S2.T1.13.13.13.1.m1.1.1.3.4.3.cmml">2</mn></msup></mrow><mrow id="S2.T1.13.13.13.1.m1.1.1.1" xref="S2.T1.13.13.13.1.m1.1.1.1.cmml"><msup id="S2.T1.13.13.13.1.m1.1.1.1.1" xref="S2.T1.13.13.13.1.m1.1.1.1.1.cmml"><mrow id="S2.T1.13.13.13.1.m1.1.1.1.1.1.1" xref="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.cmml"><mo maxsize="90%" minsize="90%" id="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.2" xref="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1" xref="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.cmml"><mrow id="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.2" xref="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.2.cmml"><mi mathsize="90%" id="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.2.2" xref="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.2.2.cmml">M</mi><mo maxsize="90%" minsize="90%" stretchy="true" symmetric="true" id="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.2.1" xref="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.2.1.cmml">/</mo><mn mathsize="90%" id="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.2.3" xref="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.2.3.cmml">4</mn></mrow><mo lspace="0em" rspace="0em" id="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.1" xref="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.3" xref="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.3.cmml">d</mi></mrow><mo maxsize="90%" minsize="90%" id="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.3" xref="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mn mathsize="90%" id="S2.T1.13.13.13.1.m1.1.1.1.1.3" xref="S2.T1.13.13.13.1.m1.1.1.1.1.3.cmml">2</mn></msup><mo lspace="0em" rspace="0em" id="S2.T1.13.13.13.1.m1.1.1.1.2" xref="S2.T1.13.13.13.1.m1.1.1.1.2.cmml">​</mo><mi mathsize="90%" id="S2.T1.13.13.13.1.m1.1.1.1.3" xref="S2.T1.13.13.13.1.m1.1.1.1.3.cmml">G</mi></mrow></mfrac><mo mathsize="90%" id="S2.T1.13.13.13.1.m1.1.2.1a" xref="S2.T1.13.13.13.1.m1.1.2.1.cmml">+</mo><mrow id="S2.T1.13.13.13.1.m1.1.2.3" xref="S2.T1.13.13.13.1.m1.1.2.3.cmml"><mi mathsize="90%" id="S2.T1.13.13.13.1.m1.1.2.3.2" xref="S2.T1.13.13.13.1.m1.1.2.3.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.13.13.13.1.m1.1.2.3.1" xref="S2.T1.13.13.13.1.m1.1.2.3.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.13.13.13.1.m1.1.2.3.3" xref="S2.T1.13.13.13.1.m1.1.2.3.3.cmml">N</mi><mo lspace="0em" rspace="0em" id="S2.T1.13.13.13.1.m1.1.2.3.1a" xref="S2.T1.13.13.13.1.m1.1.2.3.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.13.13.13.1.m1.1.2.3.4" xref="S2.T1.13.13.13.1.m1.1.2.3.4.cmml">H</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.T1.13.13.13.1.m1.1b"><apply id="S2.T1.13.13.13.1.m1.1.2.cmml" xref="S2.T1.13.13.13.1.m1.1.2"><plus id="S2.T1.13.13.13.1.m1.1.2.1.cmml" xref="S2.T1.13.13.13.1.m1.1.2.1"></plus><apply id="S2.T1.13.13.13.1.m1.1.2.2.cmml" xref="S2.T1.13.13.13.1.m1.1.2.2"><times id="S2.T1.13.13.13.1.m1.1.2.2.1.cmml" xref="S2.T1.13.13.13.1.m1.1.2.2.1"></times><cn type="integer" id="S2.T1.13.13.13.1.m1.1.2.2.2.cmml" xref="S2.T1.13.13.13.1.m1.1.2.2.2">4</cn><apply id="S2.T1.13.13.13.1.m1.1.2.2.3.cmml" xref="S2.T1.13.13.13.1.m1.1.2.2.3"><divide id="S2.T1.13.13.13.1.m1.1.2.2.3.1.cmml" xref="S2.T1.13.13.13.1.m1.1.2.2.3"></divide><apply id="S2.T1.13.13.13.1.m1.1.2.2.3.2.cmml" xref="S2.T1.13.13.13.1.m1.1.2.2.3.2"><times id="S2.T1.13.13.13.1.m1.1.2.2.3.2.1.cmml" xref="S2.T1.13.13.13.1.m1.1.2.2.3.2.1"></times><ci id="S2.T1.13.13.13.1.m1.1.2.2.3.2.2.cmml" xref="S2.T1.13.13.13.1.m1.1.2.2.3.2.2">𝐵</ci><ci id="S2.T1.13.13.13.1.m1.1.2.2.3.2.3.cmml" xref="S2.T1.13.13.13.1.m1.1.2.2.3.2.3">𝑍</ci><ci id="S2.T1.13.13.13.1.m1.1.2.2.3.2.4.cmml" xref="S2.T1.13.13.13.1.m1.1.2.2.3.2.4">𝑁</ci><ci id="S2.T1.13.13.13.1.m1.1.2.2.3.2.5.cmml" xref="S2.T1.13.13.13.1.m1.1.2.2.3.2.5">𝑑</ci></apply><ci id="S2.T1.13.13.13.1.m1.1.2.2.3.3.cmml" xref="S2.T1.13.13.13.1.m1.1.2.2.3.3">𝐺</ci></apply></apply><apply id="S2.T1.13.13.13.1.m1.1.1.cmml" xref="S2.T1.13.13.13.1.m1.1.1"><divide id="S2.T1.13.13.13.1.m1.1.1.2.cmml" xref="S2.T1.13.13.13.1.m1.1.1"></divide><apply id="S2.T1.13.13.13.1.m1.1.1.3.cmml" xref="S2.T1.13.13.13.1.m1.1.1.3"><times id="S2.T1.13.13.13.1.m1.1.1.3.1.cmml" xref="S2.T1.13.13.13.1.m1.1.1.3.1"></times><ci id="S2.T1.13.13.13.1.m1.1.1.3.2.cmml" xref="S2.T1.13.13.13.1.m1.1.1.3.2">𝐵</ci><ci id="S2.T1.13.13.13.1.m1.1.1.3.3.cmml" xref="S2.T1.13.13.13.1.m1.1.1.3.3">𝑍</ci><apply id="S2.T1.13.13.13.1.m1.1.1.3.4.cmml" xref="S2.T1.13.13.13.1.m1.1.1.3.4"><csymbol cd="ambiguous" id="S2.T1.13.13.13.1.m1.1.1.3.4.1.cmml" xref="S2.T1.13.13.13.1.m1.1.1.3.4">superscript</csymbol><ci id="S2.T1.13.13.13.1.m1.1.1.3.4.2.cmml" xref="S2.T1.13.13.13.1.m1.1.1.3.4.2">𝑁</ci><cn type="integer" id="S2.T1.13.13.13.1.m1.1.1.3.4.3.cmml" xref="S2.T1.13.13.13.1.m1.1.1.3.4.3">2</cn></apply></apply><apply id="S2.T1.13.13.13.1.m1.1.1.1.cmml" xref="S2.T1.13.13.13.1.m1.1.1.1"><times id="S2.T1.13.13.13.1.m1.1.1.1.2.cmml" xref="S2.T1.13.13.13.1.m1.1.1.1.2"></times><apply id="S2.T1.13.13.13.1.m1.1.1.1.1.cmml" xref="S2.T1.13.13.13.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S2.T1.13.13.13.1.m1.1.1.1.1.2.cmml" xref="S2.T1.13.13.13.1.m1.1.1.1.1">superscript</csymbol><apply id="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.cmml" xref="S2.T1.13.13.13.1.m1.1.1.1.1.1.1"><times id="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.1.cmml" xref="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.1"></times><apply id="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.2.cmml" xref="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.2"><divide id="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.2.1"></divide><ci id="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.2.2.cmml" xref="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.2.2">𝑀</ci><cn type="integer" id="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.2.3.cmml" xref="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.2.3">4</cn></apply><ci id="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.3.cmml" xref="S2.T1.13.13.13.1.m1.1.1.1.1.1.1.1.3">𝑑</ci></apply><cn type="integer" id="S2.T1.13.13.13.1.m1.1.1.1.1.3.cmml" xref="S2.T1.13.13.13.1.m1.1.1.1.1.3">2</cn></apply><ci id="S2.T1.13.13.13.1.m1.1.1.1.3.cmml" xref="S2.T1.13.13.13.1.m1.1.1.1.3">𝐺</ci></apply></apply><apply id="S2.T1.13.13.13.1.m1.1.2.3.cmml" xref="S2.T1.13.13.13.1.m1.1.2.3"><times id="S2.T1.13.13.13.1.m1.1.2.3.1.cmml" xref="S2.T1.13.13.13.1.m1.1.2.3.1"></times><ci id="S2.T1.13.13.13.1.m1.1.2.3.2.cmml" xref="S2.T1.13.13.13.1.m1.1.2.3.2">𝐵</ci><ci id="S2.T1.13.13.13.1.m1.1.2.3.3.cmml" xref="S2.T1.13.13.13.1.m1.1.2.3.3">𝑁</ci><ci id="S2.T1.13.13.13.1.m1.1.2.3.4.cmml" xref="S2.T1.13.13.13.1.m1.1.2.3.4">𝐻</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.13.13.13.1.m1.1c">{4\frac{BZNd}{G}}+\frac{BZN^{2}}{(M/4d)^{2}G}+BNH</annotation></semantics></math></td>
</tr>
<tr id="S2.T1.17.17.17" class="ltx_tr">
<td id="S2.T1.17.17.17.5" class="ltx_td ltx_align_left ltx_border_r ltx_border_t"><span id="S2.T1.17.17.17.5.1" class="ltx_text" style="font-size:90%;">BurstAttention</span></td>
<td id="S2.T1.17.17.17.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="S2.T1.17.17.17.6.1" class="ltx_text" style="font-size:90%;">w/o</span></td>
<td id="S2.T1.14.14.14.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" rowspan="2"><span id="S2.T1.14.14.14.1.1" class="ltx_text" style="font-size:90%;"><math id="S2.T1.14.14.14.1.1.m1.1" class="ltx_Math" alttext="4HZd" display="inline"><semantics id="S2.T1.14.14.14.1.1.m1.1a"><mrow id="S2.T1.14.14.14.1.1.m1.1.1" xref="S2.T1.14.14.14.1.1.m1.1.1.cmml"><mn id="S2.T1.14.14.14.1.1.m1.1.1.2" xref="S2.T1.14.14.14.1.1.m1.1.1.2.cmml">4</mn><mo lspace="0em" rspace="0em" id="S2.T1.14.14.14.1.1.m1.1.1.1" xref="S2.T1.14.14.14.1.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.14.14.14.1.1.m1.1.1.3" xref="S2.T1.14.14.14.1.1.m1.1.1.3.cmml">H</mi><mo lspace="0em" rspace="0em" id="S2.T1.14.14.14.1.1.m1.1.1.1a" xref="S2.T1.14.14.14.1.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.14.14.14.1.1.m1.1.1.4" xref="S2.T1.14.14.14.1.1.m1.1.1.4.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S2.T1.14.14.14.1.1.m1.1.1.1b" xref="S2.T1.14.14.14.1.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.14.14.14.1.1.m1.1.1.5" xref="S2.T1.14.14.14.1.1.m1.1.1.5.cmml">d</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.T1.14.14.14.1.1.m1.1b"><apply id="S2.T1.14.14.14.1.1.m1.1.1.cmml" xref="S2.T1.14.14.14.1.1.m1.1.1"><times id="S2.T1.14.14.14.1.1.m1.1.1.1.cmml" xref="S2.T1.14.14.14.1.1.m1.1.1.1"></times><cn type="integer" id="S2.T1.14.14.14.1.1.m1.1.1.2.cmml" xref="S2.T1.14.14.14.1.1.m1.1.1.2">4</cn><ci id="S2.T1.14.14.14.1.1.m1.1.1.3.cmml" xref="S2.T1.14.14.14.1.1.m1.1.1.3">𝐻</ci><ci id="S2.T1.14.14.14.1.1.m1.1.1.4.cmml" xref="S2.T1.14.14.14.1.1.m1.1.1.4">𝑍</ci><ci id="S2.T1.14.14.14.1.1.m1.1.1.5.cmml" xref="S2.T1.14.14.14.1.1.m1.1.1.5">𝑑</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.14.14.14.1.1.m1.1c">4HZd</annotation></semantics></math></span></td>
<td id="S2.T1.15.15.15.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><math id="S2.T1.15.15.15.2.m1.1" class="ltx_Math" alttext="{4\frac{BZNd}{G}}+\frac{BZN^{2}}{G^{2}}+{\frac{BNH}{G}}" display="inline"><semantics id="S2.T1.15.15.15.2.m1.1a"><mrow id="S2.T1.15.15.15.2.m1.1.1" xref="S2.T1.15.15.15.2.m1.1.1.cmml"><mrow id="S2.T1.15.15.15.2.m1.1.1.2" xref="S2.T1.15.15.15.2.m1.1.1.2.cmml"><mn mathsize="90%" id="S2.T1.15.15.15.2.m1.1.1.2.2" xref="S2.T1.15.15.15.2.m1.1.1.2.2.cmml">4</mn><mo lspace="0em" rspace="0em" id="S2.T1.15.15.15.2.m1.1.1.2.1" xref="S2.T1.15.15.15.2.m1.1.1.2.1.cmml">​</mo><mfrac id="S2.T1.15.15.15.2.m1.1.1.2.3" xref="S2.T1.15.15.15.2.m1.1.1.2.3.cmml"><mrow id="S2.T1.15.15.15.2.m1.1.1.2.3.2" xref="S2.T1.15.15.15.2.m1.1.1.2.3.2.cmml"><mi mathsize="90%" id="S2.T1.15.15.15.2.m1.1.1.2.3.2.2" xref="S2.T1.15.15.15.2.m1.1.1.2.3.2.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.15.15.15.2.m1.1.1.2.3.2.1" xref="S2.T1.15.15.15.2.m1.1.1.2.3.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.15.15.15.2.m1.1.1.2.3.2.3" xref="S2.T1.15.15.15.2.m1.1.1.2.3.2.3.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S2.T1.15.15.15.2.m1.1.1.2.3.2.1a" xref="S2.T1.15.15.15.2.m1.1.1.2.3.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.15.15.15.2.m1.1.1.2.3.2.4" xref="S2.T1.15.15.15.2.m1.1.1.2.3.2.4.cmml">N</mi><mo lspace="0em" rspace="0em" id="S2.T1.15.15.15.2.m1.1.1.2.3.2.1b" xref="S2.T1.15.15.15.2.m1.1.1.2.3.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.15.15.15.2.m1.1.1.2.3.2.5" xref="S2.T1.15.15.15.2.m1.1.1.2.3.2.5.cmml">d</mi></mrow><mi mathsize="90%" id="S2.T1.15.15.15.2.m1.1.1.2.3.3" xref="S2.T1.15.15.15.2.m1.1.1.2.3.3.cmml">G</mi></mfrac></mrow><mo mathsize="90%" id="S2.T1.15.15.15.2.m1.1.1.1" xref="S2.T1.15.15.15.2.m1.1.1.1.cmml">+</mo><mfrac id="S2.T1.15.15.15.2.m1.1.1.3" xref="S2.T1.15.15.15.2.m1.1.1.3.cmml"><mrow id="S2.T1.15.15.15.2.m1.1.1.3.2" xref="S2.T1.15.15.15.2.m1.1.1.3.2.cmml"><mi mathsize="90%" id="S2.T1.15.15.15.2.m1.1.1.3.2.2" xref="S2.T1.15.15.15.2.m1.1.1.3.2.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.15.15.15.2.m1.1.1.3.2.1" xref="S2.T1.15.15.15.2.m1.1.1.3.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.15.15.15.2.m1.1.1.3.2.3" xref="S2.T1.15.15.15.2.m1.1.1.3.2.3.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S2.T1.15.15.15.2.m1.1.1.3.2.1a" xref="S2.T1.15.15.15.2.m1.1.1.3.2.1.cmml">​</mo><msup id="S2.T1.15.15.15.2.m1.1.1.3.2.4" xref="S2.T1.15.15.15.2.m1.1.1.3.2.4.cmml"><mi mathsize="90%" id="S2.T1.15.15.15.2.m1.1.1.3.2.4.2" xref="S2.T1.15.15.15.2.m1.1.1.3.2.4.2.cmml">N</mi><mn mathsize="90%" id="S2.T1.15.15.15.2.m1.1.1.3.2.4.3" xref="S2.T1.15.15.15.2.m1.1.1.3.2.4.3.cmml">2</mn></msup></mrow><msup id="S2.T1.15.15.15.2.m1.1.1.3.3" xref="S2.T1.15.15.15.2.m1.1.1.3.3.cmml"><mi mathsize="90%" id="S2.T1.15.15.15.2.m1.1.1.3.3.2" xref="S2.T1.15.15.15.2.m1.1.1.3.3.2.cmml">G</mi><mn mathsize="90%" id="S2.T1.15.15.15.2.m1.1.1.3.3.3" xref="S2.T1.15.15.15.2.m1.1.1.3.3.3.cmml">2</mn></msup></mfrac><mo mathsize="90%" id="S2.T1.15.15.15.2.m1.1.1.1a" xref="S2.T1.15.15.15.2.m1.1.1.1.cmml">+</mo><mfrac id="S2.T1.15.15.15.2.m1.1.1.4" xref="S2.T1.15.15.15.2.m1.1.1.4.cmml"><mrow id="S2.T1.15.15.15.2.m1.1.1.4.2" xref="S2.T1.15.15.15.2.m1.1.1.4.2.cmml"><mi mathsize="90%" id="S2.T1.15.15.15.2.m1.1.1.4.2.2" xref="S2.T1.15.15.15.2.m1.1.1.4.2.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.15.15.15.2.m1.1.1.4.2.1" xref="S2.T1.15.15.15.2.m1.1.1.4.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.15.15.15.2.m1.1.1.4.2.3" xref="S2.T1.15.15.15.2.m1.1.1.4.2.3.cmml">N</mi><mo lspace="0em" rspace="0em" id="S2.T1.15.15.15.2.m1.1.1.4.2.1a" xref="S2.T1.15.15.15.2.m1.1.1.4.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.15.15.15.2.m1.1.1.4.2.4" xref="S2.T1.15.15.15.2.m1.1.1.4.2.4.cmml">H</mi></mrow><mi mathsize="90%" id="S2.T1.15.15.15.2.m1.1.1.4.3" xref="S2.T1.15.15.15.2.m1.1.1.4.3.cmml">G</mi></mfrac></mrow><annotation-xml encoding="MathML-Content" id="S2.T1.15.15.15.2.m1.1b"><apply id="S2.T1.15.15.15.2.m1.1.1.cmml" xref="S2.T1.15.15.15.2.m1.1.1"><plus id="S2.T1.15.15.15.2.m1.1.1.1.cmml" xref="S2.T1.15.15.15.2.m1.1.1.1"></plus><apply id="S2.T1.15.15.15.2.m1.1.1.2.cmml" xref="S2.T1.15.15.15.2.m1.1.1.2"><times id="S2.T1.15.15.15.2.m1.1.1.2.1.cmml" xref="S2.T1.15.15.15.2.m1.1.1.2.1"></times><cn type="integer" id="S2.T1.15.15.15.2.m1.1.1.2.2.cmml" xref="S2.T1.15.15.15.2.m1.1.1.2.2">4</cn><apply id="S2.T1.15.15.15.2.m1.1.1.2.3.cmml" xref="S2.T1.15.15.15.2.m1.1.1.2.3"><divide id="S2.T1.15.15.15.2.m1.1.1.2.3.1.cmml" xref="S2.T1.15.15.15.2.m1.1.1.2.3"></divide><apply id="S2.T1.15.15.15.2.m1.1.1.2.3.2.cmml" xref="S2.T1.15.15.15.2.m1.1.1.2.3.2"><times id="S2.T1.15.15.15.2.m1.1.1.2.3.2.1.cmml" xref="S2.T1.15.15.15.2.m1.1.1.2.3.2.1"></times><ci id="S2.T1.15.15.15.2.m1.1.1.2.3.2.2.cmml" xref="S2.T1.15.15.15.2.m1.1.1.2.3.2.2">𝐵</ci><ci id="S2.T1.15.15.15.2.m1.1.1.2.3.2.3.cmml" xref="S2.T1.15.15.15.2.m1.1.1.2.3.2.3">𝑍</ci><ci id="S2.T1.15.15.15.2.m1.1.1.2.3.2.4.cmml" xref="S2.T1.15.15.15.2.m1.1.1.2.3.2.4">𝑁</ci><ci id="S2.T1.15.15.15.2.m1.1.1.2.3.2.5.cmml" xref="S2.T1.15.15.15.2.m1.1.1.2.3.2.5">𝑑</ci></apply><ci id="S2.T1.15.15.15.2.m1.1.1.2.3.3.cmml" xref="S2.T1.15.15.15.2.m1.1.1.2.3.3">𝐺</ci></apply></apply><apply id="S2.T1.15.15.15.2.m1.1.1.3.cmml" xref="S2.T1.15.15.15.2.m1.1.1.3"><divide id="S2.T1.15.15.15.2.m1.1.1.3.1.cmml" xref="S2.T1.15.15.15.2.m1.1.1.3"></divide><apply id="S2.T1.15.15.15.2.m1.1.1.3.2.cmml" xref="S2.T1.15.15.15.2.m1.1.1.3.2"><times id="S2.T1.15.15.15.2.m1.1.1.3.2.1.cmml" xref="S2.T1.15.15.15.2.m1.1.1.3.2.1"></times><ci id="S2.T1.15.15.15.2.m1.1.1.3.2.2.cmml" xref="S2.T1.15.15.15.2.m1.1.1.3.2.2">𝐵</ci><ci id="S2.T1.15.15.15.2.m1.1.1.3.2.3.cmml" xref="S2.T1.15.15.15.2.m1.1.1.3.2.3">𝑍</ci><apply id="S2.T1.15.15.15.2.m1.1.1.3.2.4.cmml" xref="S2.T1.15.15.15.2.m1.1.1.3.2.4"><csymbol cd="ambiguous" id="S2.T1.15.15.15.2.m1.1.1.3.2.4.1.cmml" xref="S2.T1.15.15.15.2.m1.1.1.3.2.4">superscript</csymbol><ci id="S2.T1.15.15.15.2.m1.1.1.3.2.4.2.cmml" xref="S2.T1.15.15.15.2.m1.1.1.3.2.4.2">𝑁</ci><cn type="integer" id="S2.T1.15.15.15.2.m1.1.1.3.2.4.3.cmml" xref="S2.T1.15.15.15.2.m1.1.1.3.2.4.3">2</cn></apply></apply><apply id="S2.T1.15.15.15.2.m1.1.1.3.3.cmml" xref="S2.T1.15.15.15.2.m1.1.1.3.3"><csymbol cd="ambiguous" id="S2.T1.15.15.15.2.m1.1.1.3.3.1.cmml" xref="S2.T1.15.15.15.2.m1.1.1.3.3">superscript</csymbol><ci id="S2.T1.15.15.15.2.m1.1.1.3.3.2.cmml" xref="S2.T1.15.15.15.2.m1.1.1.3.3.2">𝐺</ci><cn type="integer" id="S2.T1.15.15.15.2.m1.1.1.3.3.3.cmml" xref="S2.T1.15.15.15.2.m1.1.1.3.3.3">2</cn></apply></apply><apply id="S2.T1.15.15.15.2.m1.1.1.4.cmml" xref="S2.T1.15.15.15.2.m1.1.1.4"><divide id="S2.T1.15.15.15.2.m1.1.1.4.1.cmml" xref="S2.T1.15.15.15.2.m1.1.1.4"></divide><apply id="S2.T1.15.15.15.2.m1.1.1.4.2.cmml" xref="S2.T1.15.15.15.2.m1.1.1.4.2"><times id="S2.T1.15.15.15.2.m1.1.1.4.2.1.cmml" xref="S2.T1.15.15.15.2.m1.1.1.4.2.1"></times><ci id="S2.T1.15.15.15.2.m1.1.1.4.2.2.cmml" xref="S2.T1.15.15.15.2.m1.1.1.4.2.2">𝐵</ci><ci id="S2.T1.15.15.15.2.m1.1.1.4.2.3.cmml" xref="S2.T1.15.15.15.2.m1.1.1.4.2.3">𝑁</ci><ci id="S2.T1.15.15.15.2.m1.1.1.4.2.4.cmml" xref="S2.T1.15.15.15.2.m1.1.1.4.2.4">𝐻</ci></apply><ci id="S2.T1.15.15.15.2.m1.1.1.4.3.cmml" xref="S2.T1.15.15.15.2.m1.1.1.4.3">𝐺</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.15.15.15.2.m1.1c">{4\frac{BZNd}{G}}+\frac{BZN^{2}}{G^{2}}+{\frac{BNH}{G}}</annotation></semantics></math></td>
<td id="S2.T1.16.16.16.3" class="ltx_td ltx_align_left ltx_border_bb ltx_border_l ltx_border_t" rowspan="2"><span id="S2.T1.16.16.16.3.1" class="ltx_text" style="font-size:90%;"><math id="S2.T1.16.16.16.3.1.m1.1" class="ltx_Math" alttext="2BZNd" display="inline"><semantics id="S2.T1.16.16.16.3.1.m1.1a"><mrow id="S2.T1.16.16.16.3.1.m1.1.1" xref="S2.T1.16.16.16.3.1.m1.1.1.cmml"><mn id="S2.T1.16.16.16.3.1.m1.1.1.2" xref="S2.T1.16.16.16.3.1.m1.1.1.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S2.T1.16.16.16.3.1.m1.1.1.1" xref="S2.T1.16.16.16.3.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.16.16.16.3.1.m1.1.1.3" xref="S2.T1.16.16.16.3.1.m1.1.1.3.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.16.16.16.3.1.m1.1.1.1a" xref="S2.T1.16.16.16.3.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.16.16.16.3.1.m1.1.1.4" xref="S2.T1.16.16.16.3.1.m1.1.1.4.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S2.T1.16.16.16.3.1.m1.1.1.1b" xref="S2.T1.16.16.16.3.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.16.16.16.3.1.m1.1.1.5" xref="S2.T1.16.16.16.3.1.m1.1.1.5.cmml">N</mi><mo lspace="0em" rspace="0em" id="S2.T1.16.16.16.3.1.m1.1.1.1c" xref="S2.T1.16.16.16.3.1.m1.1.1.1.cmml">​</mo><mi id="S2.T1.16.16.16.3.1.m1.1.1.6" xref="S2.T1.16.16.16.3.1.m1.1.1.6.cmml">d</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.T1.16.16.16.3.1.m1.1b"><apply id="S2.T1.16.16.16.3.1.m1.1.1.cmml" xref="S2.T1.16.16.16.3.1.m1.1.1"><times id="S2.T1.16.16.16.3.1.m1.1.1.1.cmml" xref="S2.T1.16.16.16.3.1.m1.1.1.1"></times><cn type="integer" id="S2.T1.16.16.16.3.1.m1.1.1.2.cmml" xref="S2.T1.16.16.16.3.1.m1.1.1.2">2</cn><ci id="S2.T1.16.16.16.3.1.m1.1.1.3.cmml" xref="S2.T1.16.16.16.3.1.m1.1.1.3">𝐵</ci><ci id="S2.T1.16.16.16.3.1.m1.1.1.4.cmml" xref="S2.T1.16.16.16.3.1.m1.1.1.4">𝑍</ci><ci id="S2.T1.16.16.16.3.1.m1.1.1.5.cmml" xref="S2.T1.16.16.16.3.1.m1.1.1.5">𝑁</ci><ci id="S2.T1.16.16.16.3.1.m1.1.1.6.cmml" xref="S2.T1.16.16.16.3.1.m1.1.1.6">𝑑</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.16.16.16.3.1.m1.1c">2BZNd</annotation></semantics></math></span></td>
<td id="S2.T1.17.17.17.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_l ltx_border_t" rowspan="2"><span id="S2.T1.17.17.17.4.1" class="ltx_text" style="font-size:90%;"><math id="S2.T1.17.17.17.4.1.m1.1" class="ltx_Math" alttext="3BZNd+2BZN" display="inline"><semantics id="S2.T1.17.17.17.4.1.m1.1a"><mrow id="S2.T1.17.17.17.4.1.m1.1.1" xref="S2.T1.17.17.17.4.1.m1.1.1.cmml"><mrow id="S2.T1.17.17.17.4.1.m1.1.1.2" xref="S2.T1.17.17.17.4.1.m1.1.1.2.cmml"><mn id="S2.T1.17.17.17.4.1.m1.1.1.2.2" xref="S2.T1.17.17.17.4.1.m1.1.1.2.2.cmml">3</mn><mo lspace="0em" rspace="0em" id="S2.T1.17.17.17.4.1.m1.1.1.2.1" xref="S2.T1.17.17.17.4.1.m1.1.1.2.1.cmml">​</mo><mi id="S2.T1.17.17.17.4.1.m1.1.1.2.3" xref="S2.T1.17.17.17.4.1.m1.1.1.2.3.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.17.17.17.4.1.m1.1.1.2.1a" xref="S2.T1.17.17.17.4.1.m1.1.1.2.1.cmml">​</mo><mi id="S2.T1.17.17.17.4.1.m1.1.1.2.4" xref="S2.T1.17.17.17.4.1.m1.1.1.2.4.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S2.T1.17.17.17.4.1.m1.1.1.2.1b" xref="S2.T1.17.17.17.4.1.m1.1.1.2.1.cmml">​</mo><mi id="S2.T1.17.17.17.4.1.m1.1.1.2.5" xref="S2.T1.17.17.17.4.1.m1.1.1.2.5.cmml">N</mi><mo lspace="0em" rspace="0em" id="S2.T1.17.17.17.4.1.m1.1.1.2.1c" xref="S2.T1.17.17.17.4.1.m1.1.1.2.1.cmml">​</mo><mi id="S2.T1.17.17.17.4.1.m1.1.1.2.6" xref="S2.T1.17.17.17.4.1.m1.1.1.2.6.cmml">d</mi></mrow><mo id="S2.T1.17.17.17.4.1.m1.1.1.1" xref="S2.T1.17.17.17.4.1.m1.1.1.1.cmml">+</mo><mrow id="S2.T1.17.17.17.4.1.m1.1.1.3" xref="S2.T1.17.17.17.4.1.m1.1.1.3.cmml"><mn id="S2.T1.17.17.17.4.1.m1.1.1.3.2" xref="S2.T1.17.17.17.4.1.m1.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S2.T1.17.17.17.4.1.m1.1.1.3.1" xref="S2.T1.17.17.17.4.1.m1.1.1.3.1.cmml">​</mo><mi id="S2.T1.17.17.17.4.1.m1.1.1.3.3" xref="S2.T1.17.17.17.4.1.m1.1.1.3.3.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.17.17.17.4.1.m1.1.1.3.1a" xref="S2.T1.17.17.17.4.1.m1.1.1.3.1.cmml">​</mo><mi id="S2.T1.17.17.17.4.1.m1.1.1.3.4" xref="S2.T1.17.17.17.4.1.m1.1.1.3.4.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S2.T1.17.17.17.4.1.m1.1.1.3.1b" xref="S2.T1.17.17.17.4.1.m1.1.1.3.1.cmml">​</mo><mi id="S2.T1.17.17.17.4.1.m1.1.1.3.5" xref="S2.T1.17.17.17.4.1.m1.1.1.3.5.cmml">N</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.T1.17.17.17.4.1.m1.1b"><apply id="S2.T1.17.17.17.4.1.m1.1.1.cmml" xref="S2.T1.17.17.17.4.1.m1.1.1"><plus id="S2.T1.17.17.17.4.1.m1.1.1.1.cmml" xref="S2.T1.17.17.17.4.1.m1.1.1.1"></plus><apply id="S2.T1.17.17.17.4.1.m1.1.1.2.cmml" xref="S2.T1.17.17.17.4.1.m1.1.1.2"><times id="S2.T1.17.17.17.4.1.m1.1.1.2.1.cmml" xref="S2.T1.17.17.17.4.1.m1.1.1.2.1"></times><cn type="integer" id="S2.T1.17.17.17.4.1.m1.1.1.2.2.cmml" xref="S2.T1.17.17.17.4.1.m1.1.1.2.2">3</cn><ci id="S2.T1.17.17.17.4.1.m1.1.1.2.3.cmml" xref="S2.T1.17.17.17.4.1.m1.1.1.2.3">𝐵</ci><ci id="S2.T1.17.17.17.4.1.m1.1.1.2.4.cmml" xref="S2.T1.17.17.17.4.1.m1.1.1.2.4">𝑍</ci><ci id="S2.T1.17.17.17.4.1.m1.1.1.2.5.cmml" xref="S2.T1.17.17.17.4.1.m1.1.1.2.5">𝑁</ci><ci id="S2.T1.17.17.17.4.1.m1.1.1.2.6.cmml" xref="S2.T1.17.17.17.4.1.m1.1.1.2.6">𝑑</ci></apply><apply id="S2.T1.17.17.17.4.1.m1.1.1.3.cmml" xref="S2.T1.17.17.17.4.1.m1.1.1.3"><times id="S2.T1.17.17.17.4.1.m1.1.1.3.1.cmml" xref="S2.T1.17.17.17.4.1.m1.1.1.3.1"></times><cn type="integer" id="S2.T1.17.17.17.4.1.m1.1.1.3.2.cmml" xref="S2.T1.17.17.17.4.1.m1.1.1.3.2">2</cn><ci id="S2.T1.17.17.17.4.1.m1.1.1.3.3.cmml" xref="S2.T1.17.17.17.4.1.m1.1.1.3.3">𝐵</ci><ci id="S2.T1.17.17.17.4.1.m1.1.1.3.4.cmml" xref="S2.T1.17.17.17.4.1.m1.1.1.3.4">𝑍</ci><ci id="S2.T1.17.17.17.4.1.m1.1.1.3.5.cmml" xref="S2.T1.17.17.17.4.1.m1.1.1.3.5">𝑁</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.17.17.17.4.1.m1.1c">3BZNd+2BZN</annotation></semantics></math></span></td>
</tr>
<tr id="S2.T1.18.18.18" class="ltx_tr">
<td id="S2.T1.18.18.18.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r"><span id="S2.T1.18.18.18.2.1" class="ltx_text" style="font-size:90%;">BurstAttention</span></td>
<td id="S2.T1.18.18.18.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r"><span id="S2.T1.18.18.18.3.1" class="ltx_text" style="font-size:90%;">w/ LAO</span></td>
<td id="S2.T1.18.18.18.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r"><math id="S2.T1.18.18.18.1.m1.1" class="ltx_Math" alttext="{4\frac{BZNd}{G}}+{\frac{BZN^{2}}{(M/4d)^{2}G^{2}}}+{\frac{BNH}{G}}" display="inline"><semantics id="S2.T1.18.18.18.1.m1.1a"><mrow id="S2.T1.18.18.18.1.m1.1.2" xref="S2.T1.18.18.18.1.m1.1.2.cmml"><mrow id="S2.T1.18.18.18.1.m1.1.2.2" xref="S2.T1.18.18.18.1.m1.1.2.2.cmml"><mn mathsize="90%" id="S2.T1.18.18.18.1.m1.1.2.2.2" xref="S2.T1.18.18.18.1.m1.1.2.2.2.cmml">4</mn><mo lspace="0em" rspace="0em" id="S2.T1.18.18.18.1.m1.1.2.2.1" xref="S2.T1.18.18.18.1.m1.1.2.2.1.cmml">​</mo><mfrac id="S2.T1.18.18.18.1.m1.1.2.2.3" xref="S2.T1.18.18.18.1.m1.1.2.2.3.cmml"><mrow id="S2.T1.18.18.18.1.m1.1.2.2.3.2" xref="S2.T1.18.18.18.1.m1.1.2.2.3.2.cmml"><mi mathsize="90%" id="S2.T1.18.18.18.1.m1.1.2.2.3.2.2" xref="S2.T1.18.18.18.1.m1.1.2.2.3.2.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.18.18.18.1.m1.1.2.2.3.2.1" xref="S2.T1.18.18.18.1.m1.1.2.2.3.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.18.18.18.1.m1.1.2.2.3.2.3" xref="S2.T1.18.18.18.1.m1.1.2.2.3.2.3.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S2.T1.18.18.18.1.m1.1.2.2.3.2.1a" xref="S2.T1.18.18.18.1.m1.1.2.2.3.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.18.18.18.1.m1.1.2.2.3.2.4" xref="S2.T1.18.18.18.1.m1.1.2.2.3.2.4.cmml">N</mi><mo lspace="0em" rspace="0em" id="S2.T1.18.18.18.1.m1.1.2.2.3.2.1b" xref="S2.T1.18.18.18.1.m1.1.2.2.3.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.18.18.18.1.m1.1.2.2.3.2.5" xref="S2.T1.18.18.18.1.m1.1.2.2.3.2.5.cmml">d</mi></mrow><mi mathsize="90%" id="S2.T1.18.18.18.1.m1.1.2.2.3.3" xref="S2.T1.18.18.18.1.m1.1.2.2.3.3.cmml">G</mi></mfrac></mrow><mo mathsize="90%" id="S2.T1.18.18.18.1.m1.1.2.1" xref="S2.T1.18.18.18.1.m1.1.2.1.cmml">+</mo><mfrac id="S2.T1.18.18.18.1.m1.1.1" xref="S2.T1.18.18.18.1.m1.1.1.cmml"><mrow id="S2.T1.18.18.18.1.m1.1.1.3" xref="S2.T1.18.18.18.1.m1.1.1.3.cmml"><mi mathsize="90%" id="S2.T1.18.18.18.1.m1.1.1.3.2" xref="S2.T1.18.18.18.1.m1.1.1.3.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.18.18.18.1.m1.1.1.3.1" xref="S2.T1.18.18.18.1.m1.1.1.3.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.18.18.18.1.m1.1.1.3.3" xref="S2.T1.18.18.18.1.m1.1.1.3.3.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S2.T1.18.18.18.1.m1.1.1.3.1a" xref="S2.T1.18.18.18.1.m1.1.1.3.1.cmml">​</mo><msup id="S2.T1.18.18.18.1.m1.1.1.3.4" xref="S2.T1.18.18.18.1.m1.1.1.3.4.cmml"><mi mathsize="90%" id="S2.T1.18.18.18.1.m1.1.1.3.4.2" xref="S2.T1.18.18.18.1.m1.1.1.3.4.2.cmml">N</mi><mn mathsize="90%" id="S2.T1.18.18.18.1.m1.1.1.3.4.3" xref="S2.T1.18.18.18.1.m1.1.1.3.4.3.cmml">2</mn></msup></mrow><mrow id="S2.T1.18.18.18.1.m1.1.1.1" xref="S2.T1.18.18.18.1.m1.1.1.1.cmml"><msup id="S2.T1.18.18.18.1.m1.1.1.1.1" xref="S2.T1.18.18.18.1.m1.1.1.1.1.cmml"><mrow id="S2.T1.18.18.18.1.m1.1.1.1.1.1.1" xref="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.cmml"><mo maxsize="90%" minsize="90%" id="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.2" xref="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1" xref="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.cmml"><mrow id="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.2" xref="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.2.cmml"><mi mathsize="90%" id="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.2.2" xref="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.2.2.cmml">M</mi><mo maxsize="90%" minsize="90%" stretchy="true" symmetric="true" id="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.2.1" xref="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.2.1.cmml">/</mo><mn mathsize="90%" id="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.2.3" xref="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.2.3.cmml">4</mn></mrow><mo lspace="0em" rspace="0em" id="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.1" xref="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.3" xref="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.3.cmml">d</mi></mrow><mo maxsize="90%" minsize="90%" id="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.3" xref="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mn mathsize="90%" id="S2.T1.18.18.18.1.m1.1.1.1.1.3" xref="S2.T1.18.18.18.1.m1.1.1.1.1.3.cmml">2</mn></msup><mo lspace="0em" rspace="0em" id="S2.T1.18.18.18.1.m1.1.1.1.2" xref="S2.T1.18.18.18.1.m1.1.1.1.2.cmml">​</mo><msup id="S2.T1.18.18.18.1.m1.1.1.1.3" xref="S2.T1.18.18.18.1.m1.1.1.1.3.cmml"><mi mathsize="90%" id="S2.T1.18.18.18.1.m1.1.1.1.3.2" xref="S2.T1.18.18.18.1.m1.1.1.1.3.2.cmml">G</mi><mn mathsize="90%" id="S2.T1.18.18.18.1.m1.1.1.1.3.3" xref="S2.T1.18.18.18.1.m1.1.1.1.3.3.cmml">2</mn></msup></mrow></mfrac><mo mathsize="90%" id="S2.T1.18.18.18.1.m1.1.2.1a" xref="S2.T1.18.18.18.1.m1.1.2.1.cmml">+</mo><mfrac id="S2.T1.18.18.18.1.m1.1.2.3" xref="S2.T1.18.18.18.1.m1.1.2.3.cmml"><mrow id="S2.T1.18.18.18.1.m1.1.2.3.2" xref="S2.T1.18.18.18.1.m1.1.2.3.2.cmml"><mi mathsize="90%" id="S2.T1.18.18.18.1.m1.1.2.3.2.2" xref="S2.T1.18.18.18.1.m1.1.2.3.2.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S2.T1.18.18.18.1.m1.1.2.3.2.1" xref="S2.T1.18.18.18.1.m1.1.2.3.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.18.18.18.1.m1.1.2.3.2.3" xref="S2.T1.18.18.18.1.m1.1.2.3.2.3.cmml">N</mi><mo lspace="0em" rspace="0em" id="S2.T1.18.18.18.1.m1.1.2.3.2.1a" xref="S2.T1.18.18.18.1.m1.1.2.3.2.1.cmml">​</mo><mi mathsize="90%" id="S2.T1.18.18.18.1.m1.1.2.3.2.4" xref="S2.T1.18.18.18.1.m1.1.2.3.2.4.cmml">H</mi></mrow><mi mathsize="90%" id="S2.T1.18.18.18.1.m1.1.2.3.3" xref="S2.T1.18.18.18.1.m1.1.2.3.3.cmml">G</mi></mfrac></mrow><annotation-xml encoding="MathML-Content" id="S2.T1.18.18.18.1.m1.1b"><apply id="S2.T1.18.18.18.1.m1.1.2.cmml" xref="S2.T1.18.18.18.1.m1.1.2"><plus id="S2.T1.18.18.18.1.m1.1.2.1.cmml" xref="S2.T1.18.18.18.1.m1.1.2.1"></plus><apply id="S2.T1.18.18.18.1.m1.1.2.2.cmml" xref="S2.T1.18.18.18.1.m1.1.2.2"><times id="S2.T1.18.18.18.1.m1.1.2.2.1.cmml" xref="S2.T1.18.18.18.1.m1.1.2.2.1"></times><cn type="integer" id="S2.T1.18.18.18.1.m1.1.2.2.2.cmml" xref="S2.T1.18.18.18.1.m1.1.2.2.2">4</cn><apply id="S2.T1.18.18.18.1.m1.1.2.2.3.cmml" xref="S2.T1.18.18.18.1.m1.1.2.2.3"><divide id="S2.T1.18.18.18.1.m1.1.2.2.3.1.cmml" xref="S2.T1.18.18.18.1.m1.1.2.2.3"></divide><apply id="S2.T1.18.18.18.1.m1.1.2.2.3.2.cmml" xref="S2.T1.18.18.18.1.m1.1.2.2.3.2"><times id="S2.T1.18.18.18.1.m1.1.2.2.3.2.1.cmml" xref="S2.T1.18.18.18.1.m1.1.2.2.3.2.1"></times><ci id="S2.T1.18.18.18.1.m1.1.2.2.3.2.2.cmml" xref="S2.T1.18.18.18.1.m1.1.2.2.3.2.2">𝐵</ci><ci id="S2.T1.18.18.18.1.m1.1.2.2.3.2.3.cmml" xref="S2.T1.18.18.18.1.m1.1.2.2.3.2.3">𝑍</ci><ci id="S2.T1.18.18.18.1.m1.1.2.2.3.2.4.cmml" xref="S2.T1.18.18.18.1.m1.1.2.2.3.2.4">𝑁</ci><ci id="S2.T1.18.18.18.1.m1.1.2.2.3.2.5.cmml" xref="S2.T1.18.18.18.1.m1.1.2.2.3.2.5">𝑑</ci></apply><ci id="S2.T1.18.18.18.1.m1.1.2.2.3.3.cmml" xref="S2.T1.18.18.18.1.m1.1.2.2.3.3">𝐺</ci></apply></apply><apply id="S2.T1.18.18.18.1.m1.1.1.cmml" xref="S2.T1.18.18.18.1.m1.1.1"><divide id="S2.T1.18.18.18.1.m1.1.1.2.cmml" xref="S2.T1.18.18.18.1.m1.1.1"></divide><apply id="S2.T1.18.18.18.1.m1.1.1.3.cmml" xref="S2.T1.18.18.18.1.m1.1.1.3"><times id="S2.T1.18.18.18.1.m1.1.1.3.1.cmml" xref="S2.T1.18.18.18.1.m1.1.1.3.1"></times><ci id="S2.T1.18.18.18.1.m1.1.1.3.2.cmml" xref="S2.T1.18.18.18.1.m1.1.1.3.2">𝐵</ci><ci id="S2.T1.18.18.18.1.m1.1.1.3.3.cmml" xref="S2.T1.18.18.18.1.m1.1.1.3.3">𝑍</ci><apply id="S2.T1.18.18.18.1.m1.1.1.3.4.cmml" xref="S2.T1.18.18.18.1.m1.1.1.3.4"><csymbol cd="ambiguous" id="S2.T1.18.18.18.1.m1.1.1.3.4.1.cmml" xref="S2.T1.18.18.18.1.m1.1.1.3.4">superscript</csymbol><ci id="S2.T1.18.18.18.1.m1.1.1.3.4.2.cmml" xref="S2.T1.18.18.18.1.m1.1.1.3.4.2">𝑁</ci><cn type="integer" id="S2.T1.18.18.18.1.m1.1.1.3.4.3.cmml" xref="S2.T1.18.18.18.1.m1.1.1.3.4.3">2</cn></apply></apply><apply id="S2.T1.18.18.18.1.m1.1.1.1.cmml" xref="S2.T1.18.18.18.1.m1.1.1.1"><times id="S2.T1.18.18.18.1.m1.1.1.1.2.cmml" xref="S2.T1.18.18.18.1.m1.1.1.1.2"></times><apply id="S2.T1.18.18.18.1.m1.1.1.1.1.cmml" xref="S2.T1.18.18.18.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S2.T1.18.18.18.1.m1.1.1.1.1.2.cmml" xref="S2.T1.18.18.18.1.m1.1.1.1.1">superscript</csymbol><apply id="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.cmml" xref="S2.T1.18.18.18.1.m1.1.1.1.1.1.1"><times id="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.1.cmml" xref="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.1"></times><apply id="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.2.cmml" xref="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.2"><divide id="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.2.1"></divide><ci id="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.2.2.cmml" xref="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.2.2">𝑀</ci><cn type="integer" id="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.2.3.cmml" xref="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.2.3">4</cn></apply><ci id="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.3.cmml" xref="S2.T1.18.18.18.1.m1.1.1.1.1.1.1.1.3">𝑑</ci></apply><cn type="integer" id="S2.T1.18.18.18.1.m1.1.1.1.1.3.cmml" xref="S2.T1.18.18.18.1.m1.1.1.1.1.3">2</cn></apply><apply id="S2.T1.18.18.18.1.m1.1.1.1.3.cmml" xref="S2.T1.18.18.18.1.m1.1.1.1.3"><csymbol cd="ambiguous" id="S2.T1.18.18.18.1.m1.1.1.1.3.1.cmml" xref="S2.T1.18.18.18.1.m1.1.1.1.3">superscript</csymbol><ci id="S2.T1.18.18.18.1.m1.1.1.1.3.2.cmml" xref="S2.T1.18.18.18.1.m1.1.1.1.3.2">𝐺</ci><cn type="integer" id="S2.T1.18.18.18.1.m1.1.1.1.3.3.cmml" xref="S2.T1.18.18.18.1.m1.1.1.1.3.3">2</cn></apply></apply></apply><apply id="S2.T1.18.18.18.1.m1.1.2.3.cmml" xref="S2.T1.18.18.18.1.m1.1.2.3"><divide id="S2.T1.18.18.18.1.m1.1.2.3.1.cmml" xref="S2.T1.18.18.18.1.m1.1.2.3"></divide><apply id="S2.T1.18.18.18.1.m1.1.2.3.2.cmml" xref="S2.T1.18.18.18.1.m1.1.2.3.2"><times id="S2.T1.18.18.18.1.m1.1.2.3.2.1.cmml" xref="S2.T1.18.18.18.1.m1.1.2.3.2.1"></times><ci id="S2.T1.18.18.18.1.m1.1.2.3.2.2.cmml" xref="S2.T1.18.18.18.1.m1.1.2.3.2.2">𝐵</ci><ci id="S2.T1.18.18.18.1.m1.1.2.3.2.3.cmml" xref="S2.T1.18.18.18.1.m1.1.2.3.2.3">𝑁</ci><ci id="S2.T1.18.18.18.1.m1.1.2.3.2.4.cmml" xref="S2.T1.18.18.18.1.m1.1.2.3.2.4">𝐻</ci></apply><ci id="S2.T1.18.18.18.1.m1.1.2.3.3.cmml" xref="S2.T1.18.18.18.1.m1.1.2.3.3">𝐺</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.18.18.18.1.m1.1c">{4\frac{BZNd}{G}}+{\frac{BZN^{2}}{(M/4d)^{2}G^{2}}}+{\frac{BNH}{G}}</annotation></semantics></math></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering" style="font-size:90%;"><span class="ltx_tag ltx_tag_table">Table 1: </span>The overheads of different distributed attention solutions. <math id="S2.T1.29.m1.1" class="ltx_Math" alttext="G" display="inline"><semantics id="S2.T1.29.m1.1b"><mi id="S2.T1.29.m1.1.1" xref="S2.T1.29.m1.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="S2.T1.29.m1.1c"><ci id="S2.T1.29.m1.1.1.cmml" xref="S2.T1.29.m1.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.29.m1.1d">G</annotation></semantics></math> is the device number, <math id="S2.T1.30.m2.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S2.T1.30.m2.1b"><mi id="S2.T1.30.m2.1.1" xref="S2.T1.30.m2.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S2.T1.30.m2.1c"><ci id="S2.T1.30.m2.1.1.cmml" xref="S2.T1.30.m2.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.30.m2.1d">B</annotation></semantics></math> denotes the batch size, <math id="S2.T1.31.m3.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S2.T1.31.m3.1b"><mi id="S2.T1.31.m3.1.1" xref="S2.T1.31.m3.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S2.T1.31.m3.1c"><ci id="S2.T1.31.m3.1.1.cmml" xref="S2.T1.31.m3.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.31.m3.1d">N</annotation></semantics></math> represents the sequence length, <math id="S2.T1.32.m4.1" class="ltx_Math" alttext="Z" display="inline"><semantics id="S2.T1.32.m4.1b"><mi id="S2.T1.32.m4.1.1" xref="S2.T1.32.m4.1.1.cmml">Z</mi><annotation-xml encoding="MathML-Content" id="S2.T1.32.m4.1c"><ci id="S2.T1.32.m4.1.1.cmml" xref="S2.T1.32.m4.1.1">𝑍</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.32.m4.1d">Z</annotation></semantics></math> signifies the number of attention heads, <math id="S2.T1.33.m5.1" class="ltx_Math" alttext="d" display="inline"><semantics id="S2.T1.33.m5.1b"><mi id="S2.T1.33.m5.1.1" xref="S2.T1.33.m5.1.1.cmml">d</mi><annotation-xml encoding="MathML-Content" id="S2.T1.33.m5.1c"><ci id="S2.T1.33.m5.1.1.cmml" xref="S2.T1.33.m5.1.1">𝑑</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.33.m5.1d">d</annotation></semantics></math> corresponds to the hidden dimension per head, <math id="S2.T1.34.m6.1" class="ltx_Math" alttext="H" display="inline"><semantics id="S2.T1.34.m6.1b"><mi id="S2.T1.34.m6.1.1" xref="S2.T1.34.m6.1.1.cmml">H</mi><annotation-xml encoding="MathML-Content" id="S2.T1.34.m6.1c"><ci id="S2.T1.34.m6.1.1.cmml" xref="S2.T1.34.m6.1.1">𝐻</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.34.m6.1d">H</annotation></semantics></math> represents the model dimension of Transformers, and <math id="S2.T1.35.m7.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S2.T1.35.m7.1b"><mi id="S2.T1.35.m7.1.1" xref="S2.T1.35.m7.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S2.T1.35.m7.1c"><ci id="S2.T1.35.m7.1.1.cmml" xref="S2.T1.35.m7.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.35.m7.1d">M</annotation></semantics></math> represents the device SRAM size. <sup id="S2.T1.53.1" class="ltx_sup"><span id="S2.T1.53.1.1" class="ltx_text ltx_font_italic">†</span></sup> means from an implementation perspective, RingAttention’s separating <math id="S2.T1.37.m9.1" class="ltx_Math" alttext="\mathbf{K}" display="inline"><semantics id="S2.T1.37.m9.1b"><mi id="S2.T1.37.m9.1.1" xref="S2.T1.37.m9.1.1.cmml">𝐊</mi><annotation-xml encoding="MathML-Content" id="S2.T1.37.m9.1c"><ci id="S2.T1.37.m9.1.1.cmml" xref="S2.T1.37.m9.1.1">𝐊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.37.m9.1d">\mathbf{K}</annotation></semantics></math> and <math id="S2.T1.38.m10.1" class="ltx_Math" alttext="\mathbf{V}" display="inline"><semantics id="S2.T1.38.m10.1b"><mi id="S2.T1.38.m10.1.1" xref="S2.T1.38.m10.1.1.cmml">𝐕</mi><annotation-xml encoding="MathML-Content" id="S2.T1.38.m10.1c"><ci id="S2.T1.38.m10.1.1.cmml" xref="S2.T1.38.m10.1.1">𝐕</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.38.m10.1d">\mathbf{V}</annotation></semantics></math> into two independent rounds of communication cannot be combined with FlashAttention to improve efficiency.
</figcaption>
</figure>
</section>
<section id="S2.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.5 </span>Overlapping Communication and Computation</h3>

<div id="S2.SS5.p1" class="ltx_para">
<p id="S2.SS5.p1.1" class="ltx_p">Although splitting sequences can efficiently utilize distributed clusters to handle the long-sequence attention, this also inevitably introduces additional time costs to transmit partitions between devices. To this end, BurstAttention leverages the potential of devices (e.g., GPUs) for overlapping communication and computation.
This contrasts with some other typical distributed methods like tensor parallelism <cite class="ltx_cite ltx_citemacro_citep">(Narayanan et al., <a href="#bib.bib17" title="" class="ltx_ref">2021</a>)</cite>, where such overlapping is not feasible due to the dependency of subsequent layers’ computations on preceding layers’ outputs.</p>
</div>
<div id="S2.SS5.p2" class="ltx_para">
<p id="S2.SS5.p2.1" class="ltx_p">To address this, BurstAttention adopts a double-buffer technique, enabling concurrent execution of communication and computation.
The technique designs two buffers for each device, one is used as input to local attention operations, and the other is used to receive data from other devices. As depicted in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, each element (query, key, or value) involved in the ring-style communication process is allocated a dedicated buffer. Concurrent with the initiation of each local attention round, the double-buffer technique triggers the transmission of the corresponding buffer tensor. This preemptive action ensures that, by the commencement of the subsequent local attention round, the required data is already available on each device, having been carried over by the buffer. The process is then repeated until all local attention operations are completed, with each round of local attention operations initiating the transmission of data required for the next round of local attention operations. More details can be found in our appendix<a href="#alg3" title="Algorithm 3 ‣ Appendix A BurstAttention Algorithm with Double-buffer ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>.</p>
</div>
</section>
<section id="S2.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.6 </span>Integrating Sparse Attention Methods</h3>

<div id="S2.SS6.p1" class="ltx_para">
<p id="S2.SS6.p1.1" class="ltx_p">Various sparse attention methods, including low-rank methods <cite class="ltx_cite ltx_citemacro_citep">(Winata et al., <a href="#bib.bib29" title="" class="ltx_ref">2020</a>; Wang et al., <a href="#bib.bib28" title="" class="ltx_ref">2020</a>)</cite>, kernel-based methods <cite class="ltx_cite ltx_citemacro_citep">(Katharopoulos et al., <a href="#bib.bib12" title="" class="ltx_ref">2020</a>; Choromanski et al., <a href="#bib.bib5" title="" class="ltx_ref">2020</a>; Qin et al., <a href="#bib.bib19" title="" class="ltx_ref">2022</a>)</cite> and downsampling methods <cite class="ltx_cite ltx_citemacro_citep">(Lee et al., <a href="#bib.bib14" title="" class="ltx_ref">2019</a>; Jaegle et al., <a href="#bib.bib11" title="" class="ltx_ref">2021</a>)</cite> are also widely explored. These methods reduce the time and memory costs of attention modules by computing a limited selection of similarity scores from a sequence rather than all possible pairs, resulting in sparse attention softmax logits rather than dense ones. Recently, <cite class="ltx_cite ltx_citemacro_citet">Ding et al. (<a href="#bib.bib8" title="" class="ltx_ref">2023</a>)</cite> have explored sparse attention based on distributed clusters and achieved promising results.</p>
</div>
<div id="S2.SS6.p2" class="ltx_para">
<p id="S2.SS6.p2.6" class="ltx_p">The sequence parallelism mechanism makes BurstAttention easy to cooperate with sparse attention methods. During the computation process of BurstAttention, given <math id="S2.SS6.p2.1.m1.1" class="ltx_Math" alttext="\mathbf{Q}_{i}" display="inline"><semantics id="S2.SS6.p2.1.m1.1a"><msub id="S2.SS6.p2.1.m1.1.1" xref="S2.SS6.p2.1.m1.1.1.cmml"><mi id="S2.SS6.p2.1.m1.1.1.2" xref="S2.SS6.p2.1.m1.1.1.2.cmml">𝐐</mi><mi id="S2.SS6.p2.1.m1.1.1.3" xref="S2.SS6.p2.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS6.p2.1.m1.1b"><apply id="S2.SS6.p2.1.m1.1.1.cmml" xref="S2.SS6.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS6.p2.1.m1.1.1.1.cmml" xref="S2.SS6.p2.1.m1.1.1">subscript</csymbol><ci id="S2.SS6.p2.1.m1.1.1.2.cmml" xref="S2.SS6.p2.1.m1.1.1.2">𝐐</ci><ci id="S2.SS6.p2.1.m1.1.1.3.cmml" xref="S2.SS6.p2.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.p2.1.m1.1c">\mathbf{Q}_{i}</annotation></semantics></math>, <math id="S2.SS6.p2.2.m2.1" class="ltx_Math" alttext="\mathbf{K}_{j}" display="inline"><semantics id="S2.SS6.p2.2.m2.1a"><msub id="S2.SS6.p2.2.m2.1.1" xref="S2.SS6.p2.2.m2.1.1.cmml"><mi id="S2.SS6.p2.2.m2.1.1.2" xref="S2.SS6.p2.2.m2.1.1.2.cmml">𝐊</mi><mi id="S2.SS6.p2.2.m2.1.1.3" xref="S2.SS6.p2.2.m2.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS6.p2.2.m2.1b"><apply id="S2.SS6.p2.2.m2.1.1.cmml" xref="S2.SS6.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS6.p2.2.m2.1.1.1.cmml" xref="S2.SS6.p2.2.m2.1.1">subscript</csymbol><ci id="S2.SS6.p2.2.m2.1.1.2.cmml" xref="S2.SS6.p2.2.m2.1.1.2">𝐊</ci><ci id="S2.SS6.p2.2.m2.1.1.3.cmml" xref="S2.SS6.p2.2.m2.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.p2.2.m2.1c">\mathbf{K}_{j}</annotation></semantics></math>, <math id="S2.SS6.p2.3.m3.1" class="ltx_Math" alttext="\mathbf{V}_{j}" display="inline"><semantics id="S2.SS6.p2.3.m3.1a"><msub id="S2.SS6.p2.3.m3.1.1" xref="S2.SS6.p2.3.m3.1.1.cmml"><mi id="S2.SS6.p2.3.m3.1.1.2" xref="S2.SS6.p2.3.m3.1.1.2.cmml">𝐕</mi><mi id="S2.SS6.p2.3.m3.1.1.3" xref="S2.SS6.p2.3.m3.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS6.p2.3.m3.1b"><apply id="S2.SS6.p2.3.m3.1.1.cmml" xref="S2.SS6.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS6.p2.3.m3.1.1.1.cmml" xref="S2.SS6.p2.3.m3.1.1">subscript</csymbol><ci id="S2.SS6.p2.3.m3.1.1.2.cmml" xref="S2.SS6.p2.3.m3.1.1.2">𝐕</ci><ci id="S2.SS6.p2.3.m3.1.1.3.cmml" xref="S2.SS6.p2.3.m3.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.p2.3.m3.1c">\mathbf{V}_{j}</annotation></semantics></math>, if there is no need to compute the similarities between these partitions, then the local attention operations on these partitions can be skipped directly. If just some tokens in <math id="S2.SS6.p2.4.m4.1" class="ltx_Math" alttext="\mathbf{Q}_{i}" display="inline"><semantics id="S2.SS6.p2.4.m4.1a"><msub id="S2.SS6.p2.4.m4.1.1" xref="S2.SS6.p2.4.m4.1.1.cmml"><mi id="S2.SS6.p2.4.m4.1.1.2" xref="S2.SS6.p2.4.m4.1.1.2.cmml">𝐐</mi><mi id="S2.SS6.p2.4.m4.1.1.3" xref="S2.SS6.p2.4.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS6.p2.4.m4.1b"><apply id="S2.SS6.p2.4.m4.1.1.cmml" xref="S2.SS6.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS6.p2.4.m4.1.1.1.cmml" xref="S2.SS6.p2.4.m4.1.1">subscript</csymbol><ci id="S2.SS6.p2.4.m4.1.1.2.cmml" xref="S2.SS6.p2.4.m4.1.1.2">𝐐</ci><ci id="S2.SS6.p2.4.m4.1.1.3.cmml" xref="S2.SS6.p2.4.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.p2.4.m4.1c">\mathbf{Q}_{i}</annotation></semantics></math>, <math id="S2.SS6.p2.5.m5.1" class="ltx_Math" alttext="\mathbf{K}_{j}" display="inline"><semantics id="S2.SS6.p2.5.m5.1a"><msub id="S2.SS6.p2.5.m5.1.1" xref="S2.SS6.p2.5.m5.1.1.cmml"><mi id="S2.SS6.p2.5.m5.1.1.2" xref="S2.SS6.p2.5.m5.1.1.2.cmml">𝐊</mi><mi id="S2.SS6.p2.5.m5.1.1.3" xref="S2.SS6.p2.5.m5.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS6.p2.5.m5.1b"><apply id="S2.SS6.p2.5.m5.1.1.cmml" xref="S2.SS6.p2.5.m5.1.1"><csymbol cd="ambiguous" id="S2.SS6.p2.5.m5.1.1.1.cmml" xref="S2.SS6.p2.5.m5.1.1">subscript</csymbol><ci id="S2.SS6.p2.5.m5.1.1.2.cmml" xref="S2.SS6.p2.5.m5.1.1.2">𝐊</ci><ci id="S2.SS6.p2.5.m5.1.1.3.cmml" xref="S2.SS6.p2.5.m5.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.p2.5.m5.1c">\mathbf{K}_{j}</annotation></semantics></math> and <math id="S2.SS6.p2.6.m6.1" class="ltx_Math" alttext="\mathbf{V}_{j}" display="inline"><semantics id="S2.SS6.p2.6.m6.1a"><msub id="S2.SS6.p2.6.m6.1.1" xref="S2.SS6.p2.6.m6.1.1.cmml"><mi id="S2.SS6.p2.6.m6.1.1.2" xref="S2.SS6.p2.6.m6.1.1.2.cmml">𝐕</mi><mi id="S2.SS6.p2.6.m6.1.1.3" xref="S2.SS6.p2.6.m6.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS6.p2.6.m6.1b"><apply id="S2.SS6.p2.6.m6.1.1.cmml" xref="S2.SS6.p2.6.m6.1.1"><csymbol cd="ambiguous" id="S2.SS6.p2.6.m6.1.1.1.cmml" xref="S2.SS6.p2.6.m6.1.1">subscript</csymbol><ci id="S2.SS6.p2.6.m6.1.1.2.cmml" xref="S2.SS6.p2.6.m6.1.1.2">𝐕</ci><ci id="S2.SS6.p2.6.m6.1.1.3.cmml" xref="S2.SS6.p2.6.m6.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.p2.6.m6.1c">\mathbf{V}_{j}</annotation></semantics></math> are required to compute their similarities for final attention results, we can similarly skip unnecessary operations in local attention operations. Note that these sparse attention methods inevitably lead to significant performance degradation, along with reducing the time and memory overheads. Although BurstAttention is well compatible with sparse attention methods, in the actual processing of long sequences, the use of these lossy methods needs to be cautious.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Overhead Analysis</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">In this section, we will analyze the memory, I/O, and communication overheads of BurstAttention as compared to existing competitive distributed attention solutions.
As data parallelism and pipeline parallelism are often used as the most basic distributed strategies and cannot reduce the cost of long sequence processing, we focus here on comparing BurstAttention, tensor parallelism <cite class="ltx_cite ltx_citemacro_citep">(Narayanan et al., <a href="#bib.bib17" title="" class="ltx_ref">2021</a>)</cite>, and the typical sequence parallelism method RingAttention <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib15" title="" class="ltx_ref">2021</a>)</cite>.</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Memory and I/O Overheads</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.4" class="ltx_p">When we split the input along the sequence dimension across devices for global operations and further split them in each device for local operations, the memory overheads caused by <math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{QK}^{T}" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><msup id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml"><mi id="S3.SS1.p1.1.m1.1.1.2" xref="S3.SS1.p1.1.m1.1.1.2.cmml">𝐐𝐊</mi><mi id="S3.SS1.p1.1.m1.1.1.3" xref="S3.SS1.p1.1.m1.1.1.3.cmml">T</mi></msup><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><apply id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.1.m1.1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">superscript</csymbol><ci id="S3.SS1.p1.1.m1.1.1.2.cmml" xref="S3.SS1.p1.1.m1.1.1.2">𝐐𝐊</ci><ci id="S3.SS1.p1.1.m1.1.1.3.cmml" xref="S3.SS1.p1.1.m1.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">\mathbf{QK}^{T}</annotation></semantics></math> will be reduced to <math id="S3.SS1.p1.2.m2.1" class="ltx_Math" alttext="\frac{1}{(M/d)^{2}G^{2}}" display="inline"><semantics id="S3.SS1.p1.2.m2.1a"><mfrac id="S3.SS1.p1.2.m2.1.1" xref="S3.SS1.p1.2.m2.1.1.cmml"><mn id="S3.SS1.p1.2.m2.1.1.3" xref="S3.SS1.p1.2.m2.1.1.3.cmml">1</mn><mrow id="S3.SS1.p1.2.m2.1.1.1" xref="S3.SS1.p1.2.m2.1.1.1.cmml"><msup id="S3.SS1.p1.2.m2.1.1.1.1" xref="S3.SS1.p1.2.m2.1.1.1.1.cmml"><mrow id="S3.SS1.p1.2.m2.1.1.1.1.1.1" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p1.2.m2.1.1.1.1.1.1.2" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.cmml"><mi id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.2" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.2.cmml">M</mi><mo id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.cmml">/</mo><mi id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.cmml">d</mi></mrow><mo stretchy="false" id="S3.SS1.p1.2.m2.1.1.1.1.1.1.3" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.cmml">)</mo></mrow><mn id="S3.SS1.p1.2.m2.1.1.1.1.3" xref="S3.SS1.p1.2.m2.1.1.1.1.3.cmml">2</mn></msup><mo lspace="0em" rspace="0em" id="S3.SS1.p1.2.m2.1.1.1.2" xref="S3.SS1.p1.2.m2.1.1.1.2.cmml">​</mo><msup id="S3.SS1.p1.2.m2.1.1.1.3" xref="S3.SS1.p1.2.m2.1.1.1.3.cmml"><mi id="S3.SS1.p1.2.m2.1.1.1.3.2" xref="S3.SS1.p1.2.m2.1.1.1.3.2.cmml">G</mi><mn id="S3.SS1.p1.2.m2.1.1.1.3.3" xref="S3.SS1.p1.2.m2.1.1.1.3.3.cmml">2</mn></msup></mrow></mfrac><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.2.m2.1b"><apply id="S3.SS1.p1.2.m2.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1"><divide id="S3.SS1.p1.2.m2.1.1.2.cmml" xref="S3.SS1.p1.2.m2.1.1"></divide><cn type="integer" id="S3.SS1.p1.2.m2.1.1.3.cmml" xref="S3.SS1.p1.2.m2.1.1.3">1</cn><apply id="S3.SS1.p1.2.m2.1.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1.1"><times id="S3.SS1.p1.2.m2.1.1.1.2.cmml" xref="S3.SS1.p1.2.m2.1.1.1.2"></times><apply id="S3.SS1.p1.2.m2.1.1.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.2.m2.1.1.1.1.2.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1">superscript</csymbol><apply id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1"><divide id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.1"></divide><ci id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.2">𝑀</ci><ci id="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.1.1.1.3">𝑑</ci></apply><cn type="integer" id="S3.SS1.p1.2.m2.1.1.1.1.3.cmml" xref="S3.SS1.p1.2.m2.1.1.1.1.3">2</cn></apply><apply id="S3.SS1.p1.2.m2.1.1.1.3.cmml" xref="S3.SS1.p1.2.m2.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p1.2.m2.1.1.1.3.1.cmml" xref="S3.SS1.p1.2.m2.1.1.1.3">superscript</csymbol><ci id="S3.SS1.p1.2.m2.1.1.1.3.2.cmml" xref="S3.SS1.p1.2.m2.1.1.1.3.2">𝐺</ci><cn type="integer" id="S3.SS1.p1.2.m2.1.1.1.3.3.cmml" xref="S3.SS1.p1.2.m2.1.1.1.3.3">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.2.m2.1c">\frac{1}{(M/d)^{2}G^{2}}</annotation></semantics></math> of the original ones. Table <a href="#S2.T1" title="Table 1 ‣ 2.4 Local Attention Optimization (LAO) ‣ 2 Methodology ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> shows the memory overheads of various distributed attention solutions.
The table shows that BurstAttention has lower activation memory while tensor parallelism has lower parameter memory.
This means that the longer the sequence, the more pronounced the advantage of BurstAttention. Moreover, by combining BurstAttention with some parallelism strategies like zero redundancy optimizer <cite class="ltx_cite ltx_citemacro_citep">(Rajbhandari et al., <a href="#bib.bib22" title="" class="ltx_ref">2020</a>; Ren et al., <a href="#bib.bib23" title="" class="ltx_ref">2021</a>)</cite> to partition parameters, BurstAttention can easily obtain the same parameter memory overheads as tensor parallelism.
In terms of I/O overheads, RingAttention requires <math id="S3.SS1.p1.3.m3.1" class="ltx_Math" alttext="\Theta(\frac{BZN^{2}}{G}+BZNd)" display="inline"><semantics id="S3.SS1.p1.3.m3.1a"><mrow id="S3.SS1.p1.3.m3.1.1" xref="S3.SS1.p1.3.m3.1.1.cmml"><mi mathvariant="normal" id="S3.SS1.p1.3.m3.1.1.3" xref="S3.SS1.p1.3.m3.1.1.3.cmml">Θ</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.3.m3.1.1.2" xref="S3.SS1.p1.3.m3.1.1.2.cmml">​</mo><mrow id="S3.SS1.p1.3.m3.1.1.1.1" xref="S3.SS1.p1.3.m3.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p1.3.m3.1.1.1.1.2" xref="S3.SS1.p1.3.m3.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS1.p1.3.m3.1.1.1.1.1" xref="S3.SS1.p1.3.m3.1.1.1.1.1.cmml"><mfrac id="S3.SS1.p1.3.m3.1.1.1.1.1.2" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2.cmml"><mrow id="S3.SS1.p1.3.m3.1.1.1.1.1.2.2" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.cmml"><mi id="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.2" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.1" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.1.cmml">​</mo><mi id="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.3" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.3.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.1a" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.1.cmml">​</mo><msup id="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.4" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.4.cmml"><mi id="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.4.2" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.4.2.cmml">N</mi><mn id="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.4.3" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.4.3.cmml">2</mn></msup></mrow><mi id="S3.SS1.p1.3.m3.1.1.1.1.1.2.3" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2.3.cmml">G</mi></mfrac><mo id="S3.SS1.p1.3.m3.1.1.1.1.1.1" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1.cmml">+</mo><mrow id="S3.SS1.p1.3.m3.1.1.1.1.1.3" xref="S3.SS1.p1.3.m3.1.1.1.1.1.3.cmml"><mi id="S3.SS1.p1.3.m3.1.1.1.1.1.3.2" xref="S3.SS1.p1.3.m3.1.1.1.1.1.3.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.3.m3.1.1.1.1.1.3.1" xref="S3.SS1.p1.3.m3.1.1.1.1.1.3.1.cmml">​</mo><mi id="S3.SS1.p1.3.m3.1.1.1.1.1.3.3" xref="S3.SS1.p1.3.m3.1.1.1.1.1.3.3.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.3.m3.1.1.1.1.1.3.1a" xref="S3.SS1.p1.3.m3.1.1.1.1.1.3.1.cmml">​</mo><mi id="S3.SS1.p1.3.m3.1.1.1.1.1.3.4" xref="S3.SS1.p1.3.m3.1.1.1.1.1.3.4.cmml">N</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.3.m3.1.1.1.1.1.3.1b" xref="S3.SS1.p1.3.m3.1.1.1.1.1.3.1.cmml">​</mo><mi id="S3.SS1.p1.3.m3.1.1.1.1.1.3.5" xref="S3.SS1.p1.3.m3.1.1.1.1.1.3.5.cmml">d</mi></mrow></mrow><mo stretchy="false" id="S3.SS1.p1.3.m3.1.1.1.1.3" xref="S3.SS1.p1.3.m3.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.3.m3.1b"><apply id="S3.SS1.p1.3.m3.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1"><times id="S3.SS1.p1.3.m3.1.1.2.cmml" xref="S3.SS1.p1.3.m3.1.1.2"></times><ci id="S3.SS1.p1.3.m3.1.1.3.cmml" xref="S3.SS1.p1.3.m3.1.1.3">Θ</ci><apply id="S3.SS1.p1.3.m3.1.1.1.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1"><plus id="S3.SS1.p1.3.m3.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.1"></plus><apply id="S3.SS1.p1.3.m3.1.1.1.1.1.2.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2"><divide id="S3.SS1.p1.3.m3.1.1.1.1.1.2.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2"></divide><apply id="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2.2"><times id="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.1"></times><ci id="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.2.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.2">𝐵</ci><ci id="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.3.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.3">𝑍</ci><apply id="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.4.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.4"><csymbol cd="ambiguous" id="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.4.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.4">superscript</csymbol><ci id="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.4.2.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.4.2">𝑁</ci><cn type="integer" id="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.4.3.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2.2.4.3">2</cn></apply></apply><ci id="S3.SS1.p1.3.m3.1.1.1.1.1.2.3.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.2.3">𝐺</ci></apply><apply id="S3.SS1.p1.3.m3.1.1.1.1.1.3.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.3"><times id="S3.SS1.p1.3.m3.1.1.1.1.1.3.1.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.3.1"></times><ci id="S3.SS1.p1.3.m3.1.1.1.1.1.3.2.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.3.2">𝐵</ci><ci id="S3.SS1.p1.3.m3.1.1.1.1.1.3.3.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.3.3">𝑍</ci><ci id="S3.SS1.p1.3.m3.1.1.1.1.1.3.4.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.3.4">𝑁</ci><ci id="S3.SS1.p1.3.m3.1.1.1.1.1.3.5.cmml" xref="S3.SS1.p1.3.m3.1.1.1.1.1.3.5">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.3.m3.1c">\Theta(\frac{BZN^{2}}{G}+BZNd)</annotation></semantics></math> memory accesses on every single device of the whole cluster; tensor parallelism and BurstAttention only require <math id="S3.SS1.p1.4.m4.1" class="ltx_Math" alttext="\Theta(\frac{BZN^{2}}{(M/d^{2})G})" display="inline"><semantics id="S3.SS1.p1.4.m4.1a"><mrow id="S3.SS1.p1.4.m4.1.2" xref="S3.SS1.p1.4.m4.1.2.cmml"><mi mathvariant="normal" id="S3.SS1.p1.4.m4.1.2.2" xref="S3.SS1.p1.4.m4.1.2.2.cmml">Θ</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.4.m4.1.2.1" xref="S3.SS1.p1.4.m4.1.2.1.cmml">​</mo><mrow id="S3.SS1.p1.4.m4.1.2.3.2" xref="S3.SS1.p1.4.m4.1.1.cmml"><mo stretchy="false" id="S3.SS1.p1.4.m4.1.2.3.2.1" xref="S3.SS1.p1.4.m4.1.1.cmml">(</mo><mfrac id="S3.SS1.p1.4.m4.1.1" xref="S3.SS1.p1.4.m4.1.1.cmml"><mrow id="S3.SS1.p1.4.m4.1.1.3" xref="S3.SS1.p1.4.m4.1.1.3.cmml"><mi id="S3.SS1.p1.4.m4.1.1.3.2" xref="S3.SS1.p1.4.m4.1.1.3.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.4.m4.1.1.3.1" xref="S3.SS1.p1.4.m4.1.1.3.1.cmml">​</mo><mi id="S3.SS1.p1.4.m4.1.1.3.3" xref="S3.SS1.p1.4.m4.1.1.3.3.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.4.m4.1.1.3.1a" xref="S3.SS1.p1.4.m4.1.1.3.1.cmml">​</mo><msup id="S3.SS1.p1.4.m4.1.1.3.4" xref="S3.SS1.p1.4.m4.1.1.3.4.cmml"><mi id="S3.SS1.p1.4.m4.1.1.3.4.2" xref="S3.SS1.p1.4.m4.1.1.3.4.2.cmml">N</mi><mn id="S3.SS1.p1.4.m4.1.1.3.4.3" xref="S3.SS1.p1.4.m4.1.1.3.4.3.cmml">2</mn></msup></mrow><mrow id="S3.SS1.p1.4.m4.1.1.1" xref="S3.SS1.p1.4.m4.1.1.1.cmml"><mrow id="S3.SS1.p1.4.m4.1.1.1.1.1" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p1.4.m4.1.1.1.1.1.2" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS1.p1.4.m4.1.1.1.1.1.1" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.cmml"><mi id="S3.SS1.p1.4.m4.1.1.1.1.1.1.2" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.2.cmml">M</mi><mo id="S3.SS1.p1.4.m4.1.1.1.1.1.1.1" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.cmml">/</mo><msup id="S3.SS1.p1.4.m4.1.1.1.1.1.1.3" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.3.cmml"><mi id="S3.SS1.p1.4.m4.1.1.1.1.1.1.3.2" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.3.2.cmml">d</mi><mn id="S3.SS1.p1.4.m4.1.1.1.1.1.1.3.3" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.3.3.cmml">2</mn></msup></mrow><mo stretchy="false" id="S3.SS1.p1.4.m4.1.1.1.1.1.3" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S3.SS1.p1.4.m4.1.1.1.2" xref="S3.SS1.p1.4.m4.1.1.1.2.cmml">​</mo><mi id="S3.SS1.p1.4.m4.1.1.1.3" xref="S3.SS1.p1.4.m4.1.1.1.3.cmml">G</mi></mrow></mfrac><mo stretchy="false" id="S3.SS1.p1.4.m4.1.2.3.2.2" xref="S3.SS1.p1.4.m4.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.4.m4.1b"><apply id="S3.SS1.p1.4.m4.1.2.cmml" xref="S3.SS1.p1.4.m4.1.2"><times id="S3.SS1.p1.4.m4.1.2.1.cmml" xref="S3.SS1.p1.4.m4.1.2.1"></times><ci id="S3.SS1.p1.4.m4.1.2.2.cmml" xref="S3.SS1.p1.4.m4.1.2.2">Θ</ci><apply id="S3.SS1.p1.4.m4.1.1.cmml" xref="S3.SS1.p1.4.m4.1.2.3.2"><divide id="S3.SS1.p1.4.m4.1.1.2.cmml" xref="S3.SS1.p1.4.m4.1.2.3.2"></divide><apply id="S3.SS1.p1.4.m4.1.1.3.cmml" xref="S3.SS1.p1.4.m4.1.1.3"><times id="S3.SS1.p1.4.m4.1.1.3.1.cmml" xref="S3.SS1.p1.4.m4.1.1.3.1"></times><ci id="S3.SS1.p1.4.m4.1.1.3.2.cmml" xref="S3.SS1.p1.4.m4.1.1.3.2">𝐵</ci><ci id="S3.SS1.p1.4.m4.1.1.3.3.cmml" xref="S3.SS1.p1.4.m4.1.1.3.3">𝑍</ci><apply id="S3.SS1.p1.4.m4.1.1.3.4.cmml" xref="S3.SS1.p1.4.m4.1.1.3.4"><csymbol cd="ambiguous" id="S3.SS1.p1.4.m4.1.1.3.4.1.cmml" xref="S3.SS1.p1.4.m4.1.1.3.4">superscript</csymbol><ci id="S3.SS1.p1.4.m4.1.1.3.4.2.cmml" xref="S3.SS1.p1.4.m4.1.1.3.4.2">𝑁</ci><cn type="integer" id="S3.SS1.p1.4.m4.1.1.3.4.3.cmml" xref="S3.SS1.p1.4.m4.1.1.3.4.3">2</cn></apply></apply><apply id="S3.SS1.p1.4.m4.1.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1.1"><times id="S3.SS1.p1.4.m4.1.1.1.2.cmml" xref="S3.SS1.p1.4.m4.1.1.1.2"></times><apply id="S3.SS1.p1.4.m4.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1"><divide id="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.1"></divide><ci id="S3.SS1.p1.4.m4.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.2">𝑀</ci><apply id="S3.SS1.p1.4.m4.1.1.1.1.1.1.3.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p1.4.m4.1.1.1.1.1.1.3.1.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.3">superscript</csymbol><ci id="S3.SS1.p1.4.m4.1.1.1.1.1.1.3.2.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.3.2">𝑑</ci><cn type="integer" id="S3.SS1.p1.4.m4.1.1.1.1.1.1.3.3.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.3.3">2</cn></apply></apply><ci id="S3.SS1.p1.4.m4.1.1.1.3.cmml" xref="S3.SS1.p1.4.m4.1.1.1.3">𝐺</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.4.m4.1c">\Theta(\frac{BZN^{2}}{(M/d^{2})G})</annotation></semantics></math> memory accesses.
This indicates that BurstAttention can significantly reduce I/O time costs compared to other distributed attention baselines.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Communication Overheads</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.9" class="ltx_p">In the forward pass, BurstAttention involves one round of ring-style peer-to-peer communications on the <math id="S3.SS2.p1.1.m1.2" class="ltx_Math" alttext="\mathbf{K},\mathbf{V}\in\mathbb{R}^{B\times Z\times\frac{N}{G}\times d}" display="inline"><semantics id="S3.SS2.p1.1.m1.2a"><mrow id="S3.SS2.p1.1.m1.2.3" xref="S3.SS2.p1.1.m1.2.3.cmml"><mrow id="S3.SS2.p1.1.m1.2.3.2.2" xref="S3.SS2.p1.1.m1.2.3.2.1.cmml"><mi id="S3.SS2.p1.1.m1.1.1" xref="S3.SS2.p1.1.m1.1.1.cmml">𝐊</mi><mo id="S3.SS2.p1.1.m1.2.3.2.2.1" xref="S3.SS2.p1.1.m1.2.3.2.1.cmml">,</mo><mi id="S3.SS2.p1.1.m1.2.2" xref="S3.SS2.p1.1.m1.2.2.cmml">𝐕</mi></mrow><mo id="S3.SS2.p1.1.m1.2.3.1" xref="S3.SS2.p1.1.m1.2.3.1.cmml">∈</mo><msup id="S3.SS2.p1.1.m1.2.3.3" xref="S3.SS2.p1.1.m1.2.3.3.cmml"><mi id="S3.SS2.p1.1.m1.2.3.3.2" xref="S3.SS2.p1.1.m1.2.3.3.2.cmml">ℝ</mi><mrow id="S3.SS2.p1.1.m1.2.3.3.3" xref="S3.SS2.p1.1.m1.2.3.3.3.cmml"><mi id="S3.SS2.p1.1.m1.2.3.3.3.2" xref="S3.SS2.p1.1.m1.2.3.3.3.2.cmml">B</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.p1.1.m1.2.3.3.3.1" xref="S3.SS2.p1.1.m1.2.3.3.3.1.cmml">×</mo><mi id="S3.SS2.p1.1.m1.2.3.3.3.3" xref="S3.SS2.p1.1.m1.2.3.3.3.3.cmml">Z</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.p1.1.m1.2.3.3.3.1a" xref="S3.SS2.p1.1.m1.2.3.3.3.1.cmml">×</mo><mfrac id="S3.SS2.p1.1.m1.2.3.3.3.4" xref="S3.SS2.p1.1.m1.2.3.3.3.4.cmml"><mi id="S3.SS2.p1.1.m1.2.3.3.3.4.2" xref="S3.SS2.p1.1.m1.2.3.3.3.4.2.cmml">N</mi><mi id="S3.SS2.p1.1.m1.2.3.3.3.4.3" xref="S3.SS2.p1.1.m1.2.3.3.3.4.3.cmml">G</mi></mfrac><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.p1.1.m1.2.3.3.3.1b" xref="S3.SS2.p1.1.m1.2.3.3.3.1.cmml">×</mo><mi id="S3.SS2.p1.1.m1.2.3.3.3.5" xref="S3.SS2.p1.1.m1.2.3.3.3.5.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.1.m1.2b"><apply id="S3.SS2.p1.1.m1.2.3.cmml" xref="S3.SS2.p1.1.m1.2.3"><in id="S3.SS2.p1.1.m1.2.3.1.cmml" xref="S3.SS2.p1.1.m1.2.3.1"></in><list id="S3.SS2.p1.1.m1.2.3.2.1.cmml" xref="S3.SS2.p1.1.m1.2.3.2.2"><ci id="S3.SS2.p1.1.m1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1">𝐊</ci><ci id="S3.SS2.p1.1.m1.2.2.cmml" xref="S3.SS2.p1.1.m1.2.2">𝐕</ci></list><apply id="S3.SS2.p1.1.m1.2.3.3.cmml" xref="S3.SS2.p1.1.m1.2.3.3"><csymbol cd="ambiguous" id="S3.SS2.p1.1.m1.2.3.3.1.cmml" xref="S3.SS2.p1.1.m1.2.3.3">superscript</csymbol><ci id="S3.SS2.p1.1.m1.2.3.3.2.cmml" xref="S3.SS2.p1.1.m1.2.3.3.2">ℝ</ci><apply id="S3.SS2.p1.1.m1.2.3.3.3.cmml" xref="S3.SS2.p1.1.m1.2.3.3.3"><times id="S3.SS2.p1.1.m1.2.3.3.3.1.cmml" xref="S3.SS2.p1.1.m1.2.3.3.3.1"></times><ci id="S3.SS2.p1.1.m1.2.3.3.3.2.cmml" xref="S3.SS2.p1.1.m1.2.3.3.3.2">𝐵</ci><ci id="S3.SS2.p1.1.m1.2.3.3.3.3.cmml" xref="S3.SS2.p1.1.m1.2.3.3.3.3">𝑍</ci><apply id="S3.SS2.p1.1.m1.2.3.3.3.4.cmml" xref="S3.SS2.p1.1.m1.2.3.3.3.4"><divide id="S3.SS2.p1.1.m1.2.3.3.3.4.1.cmml" xref="S3.SS2.p1.1.m1.2.3.3.3.4"></divide><ci id="S3.SS2.p1.1.m1.2.3.3.3.4.2.cmml" xref="S3.SS2.p1.1.m1.2.3.3.3.4.2">𝑁</ci><ci id="S3.SS2.p1.1.m1.2.3.3.3.4.3.cmml" xref="S3.SS2.p1.1.m1.2.3.3.3.4.3">𝐺</ci></apply><ci id="S3.SS2.p1.1.m1.2.3.3.3.5.cmml" xref="S3.SS2.p1.1.m1.2.3.3.3.5">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.1.m1.2c">\mathbf{K},\mathbf{V}\in\mathbb{R}^{B\times Z\times\frac{N}{G}\times d}</annotation></semantics></math>, with a total cost of <math id="S3.SS2.p1.2.m2.1" class="ltx_Math" alttext="\Theta(2BZNd)" display="inline"><semantics id="S3.SS2.p1.2.m2.1a"><mrow id="S3.SS2.p1.2.m2.1.1" xref="S3.SS2.p1.2.m2.1.1.cmml"><mi mathvariant="normal" id="S3.SS2.p1.2.m2.1.1.3" xref="S3.SS2.p1.2.m2.1.1.3.cmml">Θ</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.1.1.2" xref="S3.SS2.p1.2.m2.1.1.2.cmml">​</mo><mrow id="S3.SS2.p1.2.m2.1.1.1.1" xref="S3.SS2.p1.2.m2.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p1.2.m2.1.1.1.1.2" xref="S3.SS2.p1.2.m2.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p1.2.m2.1.1.1.1.1" xref="S3.SS2.p1.2.m2.1.1.1.1.1.cmml"><mn id="S3.SS2.p1.2.m2.1.1.1.1.1.2" xref="S3.SS2.p1.2.m2.1.1.1.1.1.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.1.1.1.1.1.1" xref="S3.SS2.p1.2.m2.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.2.m2.1.1.1.1.1.3" xref="S3.SS2.p1.2.m2.1.1.1.1.1.3.cmml">B</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.1.1.1.1.1.1a" xref="S3.SS2.p1.2.m2.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.2.m2.1.1.1.1.1.4" xref="S3.SS2.p1.2.m2.1.1.1.1.1.4.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.1.1.1.1.1.1b" xref="S3.SS2.p1.2.m2.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.2.m2.1.1.1.1.1.5" xref="S3.SS2.p1.2.m2.1.1.1.1.1.5.cmml">N</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.1.1.1.1.1.1c" xref="S3.SS2.p1.2.m2.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.2.m2.1.1.1.1.1.6" xref="S3.SS2.p1.2.m2.1.1.1.1.1.6.cmml">d</mi></mrow><mo stretchy="false" id="S3.SS2.p1.2.m2.1.1.1.1.3" xref="S3.SS2.p1.2.m2.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.2.m2.1b"><apply id="S3.SS2.p1.2.m2.1.1.cmml" xref="S3.SS2.p1.2.m2.1.1"><times id="S3.SS2.p1.2.m2.1.1.2.cmml" xref="S3.SS2.p1.2.m2.1.1.2"></times><ci id="S3.SS2.p1.2.m2.1.1.3.cmml" xref="S3.SS2.p1.2.m2.1.1.3">Θ</ci><apply id="S3.SS2.p1.2.m2.1.1.1.1.1.cmml" xref="S3.SS2.p1.2.m2.1.1.1.1"><times id="S3.SS2.p1.2.m2.1.1.1.1.1.1.cmml" xref="S3.SS2.p1.2.m2.1.1.1.1.1.1"></times><cn type="integer" id="S3.SS2.p1.2.m2.1.1.1.1.1.2.cmml" xref="S3.SS2.p1.2.m2.1.1.1.1.1.2">2</cn><ci id="S3.SS2.p1.2.m2.1.1.1.1.1.3.cmml" xref="S3.SS2.p1.2.m2.1.1.1.1.1.3">𝐵</ci><ci id="S3.SS2.p1.2.m2.1.1.1.1.1.4.cmml" xref="S3.SS2.p1.2.m2.1.1.1.1.1.4">𝑍</ci><ci id="S3.SS2.p1.2.m2.1.1.1.1.1.5.cmml" xref="S3.SS2.p1.2.m2.1.1.1.1.1.5">𝑁</ci><ci id="S3.SS2.p1.2.m2.1.1.1.1.1.6.cmml" xref="S3.SS2.p1.2.m2.1.1.1.1.1.6">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.2.m2.1c">\Theta(2BZNd)</annotation></semantics></math>.
In the backward pass, BurstAttention requires one round of ring-style communication on tensors <math id="S3.SS2.p1.3.m3.3" class="ltx_Math" alttext="\mathbf{Q},\mathbf{dQ},\mathbf{dO}\in\mathbb{R}^{B\times\frac{N}{G}\times Z\times d}" display="inline"><semantics id="S3.SS2.p1.3.m3.3a"><mrow id="S3.SS2.p1.3.m3.3.4" xref="S3.SS2.p1.3.m3.3.4.cmml"><mrow id="S3.SS2.p1.3.m3.3.4.2.2" xref="S3.SS2.p1.3.m3.3.4.2.1.cmml"><mi id="S3.SS2.p1.3.m3.1.1" xref="S3.SS2.p1.3.m3.1.1.cmml">𝐐</mi><mo id="S3.SS2.p1.3.m3.3.4.2.2.1" xref="S3.SS2.p1.3.m3.3.4.2.1.cmml">,</mo><mi id="S3.SS2.p1.3.m3.2.2" xref="S3.SS2.p1.3.m3.2.2.cmml">𝐝𝐐</mi><mo id="S3.SS2.p1.3.m3.3.4.2.2.2" xref="S3.SS2.p1.3.m3.3.4.2.1.cmml">,</mo><mi id="S3.SS2.p1.3.m3.3.3" xref="S3.SS2.p1.3.m3.3.3.cmml">𝐝𝐎</mi></mrow><mo id="S3.SS2.p1.3.m3.3.4.1" xref="S3.SS2.p1.3.m3.3.4.1.cmml">∈</mo><msup id="S3.SS2.p1.3.m3.3.4.3" xref="S3.SS2.p1.3.m3.3.4.3.cmml"><mi id="S3.SS2.p1.3.m3.3.4.3.2" xref="S3.SS2.p1.3.m3.3.4.3.2.cmml">ℝ</mi><mrow id="S3.SS2.p1.3.m3.3.4.3.3" xref="S3.SS2.p1.3.m3.3.4.3.3.cmml"><mi id="S3.SS2.p1.3.m3.3.4.3.3.2" xref="S3.SS2.p1.3.m3.3.4.3.3.2.cmml">B</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.p1.3.m3.3.4.3.3.1" xref="S3.SS2.p1.3.m3.3.4.3.3.1.cmml">×</mo><mfrac id="S3.SS2.p1.3.m3.3.4.3.3.3" xref="S3.SS2.p1.3.m3.3.4.3.3.3.cmml"><mi id="S3.SS2.p1.3.m3.3.4.3.3.3.2" xref="S3.SS2.p1.3.m3.3.4.3.3.3.2.cmml">N</mi><mi id="S3.SS2.p1.3.m3.3.4.3.3.3.3" xref="S3.SS2.p1.3.m3.3.4.3.3.3.3.cmml">G</mi></mfrac><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.p1.3.m3.3.4.3.3.1a" xref="S3.SS2.p1.3.m3.3.4.3.3.1.cmml">×</mo><mi id="S3.SS2.p1.3.m3.3.4.3.3.4" xref="S3.SS2.p1.3.m3.3.4.3.3.4.cmml">Z</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.p1.3.m3.3.4.3.3.1b" xref="S3.SS2.p1.3.m3.3.4.3.3.1.cmml">×</mo><mi id="S3.SS2.p1.3.m3.3.4.3.3.5" xref="S3.SS2.p1.3.m3.3.4.3.3.5.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.3.m3.3b"><apply id="S3.SS2.p1.3.m3.3.4.cmml" xref="S3.SS2.p1.3.m3.3.4"><in id="S3.SS2.p1.3.m3.3.4.1.cmml" xref="S3.SS2.p1.3.m3.3.4.1"></in><list id="S3.SS2.p1.3.m3.3.4.2.1.cmml" xref="S3.SS2.p1.3.m3.3.4.2.2"><ci id="S3.SS2.p1.3.m3.1.1.cmml" xref="S3.SS2.p1.3.m3.1.1">𝐐</ci><ci id="S3.SS2.p1.3.m3.2.2.cmml" xref="S3.SS2.p1.3.m3.2.2">𝐝𝐐</ci><ci id="S3.SS2.p1.3.m3.3.3.cmml" xref="S3.SS2.p1.3.m3.3.3">𝐝𝐎</ci></list><apply id="S3.SS2.p1.3.m3.3.4.3.cmml" xref="S3.SS2.p1.3.m3.3.4.3"><csymbol cd="ambiguous" id="S3.SS2.p1.3.m3.3.4.3.1.cmml" xref="S3.SS2.p1.3.m3.3.4.3">superscript</csymbol><ci id="S3.SS2.p1.3.m3.3.4.3.2.cmml" xref="S3.SS2.p1.3.m3.3.4.3.2">ℝ</ci><apply id="S3.SS2.p1.3.m3.3.4.3.3.cmml" xref="S3.SS2.p1.3.m3.3.4.3.3"><times id="S3.SS2.p1.3.m3.3.4.3.3.1.cmml" xref="S3.SS2.p1.3.m3.3.4.3.3.1"></times><ci id="S3.SS2.p1.3.m3.3.4.3.3.2.cmml" xref="S3.SS2.p1.3.m3.3.4.3.3.2">𝐵</ci><apply id="S3.SS2.p1.3.m3.3.4.3.3.3.cmml" xref="S3.SS2.p1.3.m3.3.4.3.3.3"><divide id="S3.SS2.p1.3.m3.3.4.3.3.3.1.cmml" xref="S3.SS2.p1.3.m3.3.4.3.3.3"></divide><ci id="S3.SS2.p1.3.m3.3.4.3.3.3.2.cmml" xref="S3.SS2.p1.3.m3.3.4.3.3.3.2">𝑁</ci><ci id="S3.SS2.p1.3.m3.3.4.3.3.3.3.cmml" xref="S3.SS2.p1.3.m3.3.4.3.3.3.3">𝐺</ci></apply><ci id="S3.SS2.p1.3.m3.3.4.3.3.4.cmml" xref="S3.SS2.p1.3.m3.3.4.3.3.4">𝑍</ci><ci id="S3.SS2.p1.3.m3.3.4.3.3.5.cmml" xref="S3.SS2.p1.3.m3.3.4.3.3.5">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.3.m3.3c">\mathbf{Q},\mathbf{dQ},\mathbf{dO}\in\mathbb{R}^{B\times\frac{N}{G}\times Z\times d}</annotation></semantics></math> and <math id="S3.SS2.p1.4.m4.2" class="ltx_Math" alttext="D,lse\in\mathbb{R}^{B\times\frac{N}{G}\times Z}" display="inline"><semantics id="S3.SS2.p1.4.m4.2a"><mrow id="S3.SS2.p1.4.m4.2.2" xref="S3.SS2.p1.4.m4.2.2.cmml"><mrow id="S3.SS2.p1.4.m4.2.2.1.1" xref="S3.SS2.p1.4.m4.2.2.1.2.cmml"><mi id="S3.SS2.p1.4.m4.1.1" xref="S3.SS2.p1.4.m4.1.1.cmml">D</mi><mo id="S3.SS2.p1.4.m4.2.2.1.1.2" xref="S3.SS2.p1.4.m4.2.2.1.2.cmml">,</mo><mrow id="S3.SS2.p1.4.m4.2.2.1.1.1" xref="S3.SS2.p1.4.m4.2.2.1.1.1.cmml"><mi id="S3.SS2.p1.4.m4.2.2.1.1.1.2" xref="S3.SS2.p1.4.m4.2.2.1.1.1.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.4.m4.2.2.1.1.1.1" xref="S3.SS2.p1.4.m4.2.2.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.4.m4.2.2.1.1.1.3" xref="S3.SS2.p1.4.m4.2.2.1.1.1.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.4.m4.2.2.1.1.1.1a" xref="S3.SS2.p1.4.m4.2.2.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.4.m4.2.2.1.1.1.4" xref="S3.SS2.p1.4.m4.2.2.1.1.1.4.cmml">e</mi></mrow></mrow><mo id="S3.SS2.p1.4.m4.2.2.2" xref="S3.SS2.p1.4.m4.2.2.2.cmml">∈</mo><msup id="S3.SS2.p1.4.m4.2.2.3" xref="S3.SS2.p1.4.m4.2.2.3.cmml"><mi id="S3.SS2.p1.4.m4.2.2.3.2" xref="S3.SS2.p1.4.m4.2.2.3.2.cmml">ℝ</mi><mrow id="S3.SS2.p1.4.m4.2.2.3.3" xref="S3.SS2.p1.4.m4.2.2.3.3.cmml"><mi id="S3.SS2.p1.4.m4.2.2.3.3.2" xref="S3.SS2.p1.4.m4.2.2.3.3.2.cmml">B</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.p1.4.m4.2.2.3.3.1" xref="S3.SS2.p1.4.m4.2.2.3.3.1.cmml">×</mo><mfrac id="S3.SS2.p1.4.m4.2.2.3.3.3" xref="S3.SS2.p1.4.m4.2.2.3.3.3.cmml"><mi id="S3.SS2.p1.4.m4.2.2.3.3.3.2" xref="S3.SS2.p1.4.m4.2.2.3.3.3.2.cmml">N</mi><mi id="S3.SS2.p1.4.m4.2.2.3.3.3.3" xref="S3.SS2.p1.4.m4.2.2.3.3.3.3.cmml">G</mi></mfrac><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.p1.4.m4.2.2.3.3.1a" xref="S3.SS2.p1.4.m4.2.2.3.3.1.cmml">×</mo><mi id="S3.SS2.p1.4.m4.2.2.3.3.4" xref="S3.SS2.p1.4.m4.2.2.3.3.4.cmml">Z</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.4.m4.2b"><apply id="S3.SS2.p1.4.m4.2.2.cmml" xref="S3.SS2.p1.4.m4.2.2"><in id="S3.SS2.p1.4.m4.2.2.2.cmml" xref="S3.SS2.p1.4.m4.2.2.2"></in><list id="S3.SS2.p1.4.m4.2.2.1.2.cmml" xref="S3.SS2.p1.4.m4.2.2.1.1"><ci id="S3.SS2.p1.4.m4.1.1.cmml" xref="S3.SS2.p1.4.m4.1.1">𝐷</ci><apply id="S3.SS2.p1.4.m4.2.2.1.1.1.cmml" xref="S3.SS2.p1.4.m4.2.2.1.1.1"><times id="S3.SS2.p1.4.m4.2.2.1.1.1.1.cmml" xref="S3.SS2.p1.4.m4.2.2.1.1.1.1"></times><ci id="S3.SS2.p1.4.m4.2.2.1.1.1.2.cmml" xref="S3.SS2.p1.4.m4.2.2.1.1.1.2">𝑙</ci><ci id="S3.SS2.p1.4.m4.2.2.1.1.1.3.cmml" xref="S3.SS2.p1.4.m4.2.2.1.1.1.3">𝑠</ci><ci id="S3.SS2.p1.4.m4.2.2.1.1.1.4.cmml" xref="S3.SS2.p1.4.m4.2.2.1.1.1.4">𝑒</ci></apply></list><apply id="S3.SS2.p1.4.m4.2.2.3.cmml" xref="S3.SS2.p1.4.m4.2.2.3"><csymbol cd="ambiguous" id="S3.SS2.p1.4.m4.2.2.3.1.cmml" xref="S3.SS2.p1.4.m4.2.2.3">superscript</csymbol><ci id="S3.SS2.p1.4.m4.2.2.3.2.cmml" xref="S3.SS2.p1.4.m4.2.2.3.2">ℝ</ci><apply id="S3.SS2.p1.4.m4.2.2.3.3.cmml" xref="S3.SS2.p1.4.m4.2.2.3.3"><times id="S3.SS2.p1.4.m4.2.2.3.3.1.cmml" xref="S3.SS2.p1.4.m4.2.2.3.3.1"></times><ci id="S3.SS2.p1.4.m4.2.2.3.3.2.cmml" xref="S3.SS2.p1.4.m4.2.2.3.3.2">𝐵</ci><apply id="S3.SS2.p1.4.m4.2.2.3.3.3.cmml" xref="S3.SS2.p1.4.m4.2.2.3.3.3"><divide id="S3.SS2.p1.4.m4.2.2.3.3.3.1.cmml" xref="S3.SS2.p1.4.m4.2.2.3.3.3"></divide><ci id="S3.SS2.p1.4.m4.2.2.3.3.3.2.cmml" xref="S3.SS2.p1.4.m4.2.2.3.3.3.2">𝑁</ci><ci id="S3.SS2.p1.4.m4.2.2.3.3.3.3.cmml" xref="S3.SS2.p1.4.m4.2.2.3.3.3.3">𝐺</ci></apply><ci id="S3.SS2.p1.4.m4.2.2.3.3.4.cmml" xref="S3.SS2.p1.4.m4.2.2.3.3.4">𝑍</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.4.m4.2c">D,lse\in\mathbb{R}^{B\times\frac{N}{G}\times Z}</annotation></semantics></math>, with a total cost of <math id="S3.SS2.p1.5.m5.1" class="ltx_Math" alttext="\Theta(3BZNd+2\frac{BNZ}{G})" display="inline"><semantics id="S3.SS2.p1.5.m5.1a"><mrow id="S3.SS2.p1.5.m5.1.1" xref="S3.SS2.p1.5.m5.1.1.cmml"><mi mathvariant="normal" id="S3.SS2.p1.5.m5.1.1.3" xref="S3.SS2.p1.5.m5.1.1.3.cmml">Θ</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.5.m5.1.1.2" xref="S3.SS2.p1.5.m5.1.1.2.cmml">​</mo><mrow id="S3.SS2.p1.5.m5.1.1.1.1" xref="S3.SS2.p1.5.m5.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p1.5.m5.1.1.1.1.2" xref="S3.SS2.p1.5.m5.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p1.5.m5.1.1.1.1.1" xref="S3.SS2.p1.5.m5.1.1.1.1.1.cmml"><mrow id="S3.SS2.p1.5.m5.1.1.1.1.1.2" xref="S3.SS2.p1.5.m5.1.1.1.1.1.2.cmml"><mn id="S3.SS2.p1.5.m5.1.1.1.1.1.2.2" xref="S3.SS2.p1.5.m5.1.1.1.1.1.2.2.cmml">3</mn><mo lspace="0em" rspace="0em" id="S3.SS2.p1.5.m5.1.1.1.1.1.2.1" xref="S3.SS2.p1.5.m5.1.1.1.1.1.2.1.cmml">​</mo><mi id="S3.SS2.p1.5.m5.1.1.1.1.1.2.3" xref="S3.SS2.p1.5.m5.1.1.1.1.1.2.3.cmml">B</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.5.m5.1.1.1.1.1.2.1a" xref="S3.SS2.p1.5.m5.1.1.1.1.1.2.1.cmml">​</mo><mi id="S3.SS2.p1.5.m5.1.1.1.1.1.2.4" xref="S3.SS2.p1.5.m5.1.1.1.1.1.2.4.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.5.m5.1.1.1.1.1.2.1b" xref="S3.SS2.p1.5.m5.1.1.1.1.1.2.1.cmml">​</mo><mi id="S3.SS2.p1.5.m5.1.1.1.1.1.2.5" xref="S3.SS2.p1.5.m5.1.1.1.1.1.2.5.cmml">N</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.5.m5.1.1.1.1.1.2.1c" xref="S3.SS2.p1.5.m5.1.1.1.1.1.2.1.cmml">​</mo><mi id="S3.SS2.p1.5.m5.1.1.1.1.1.2.6" xref="S3.SS2.p1.5.m5.1.1.1.1.1.2.6.cmml">d</mi></mrow><mo id="S3.SS2.p1.5.m5.1.1.1.1.1.1" xref="S3.SS2.p1.5.m5.1.1.1.1.1.1.cmml">+</mo><mrow id="S3.SS2.p1.5.m5.1.1.1.1.1.3" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3.cmml"><mn id="S3.SS2.p1.5.m5.1.1.1.1.1.3.2" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S3.SS2.p1.5.m5.1.1.1.1.1.3.1" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3.1.cmml">​</mo><mfrac id="S3.SS2.p1.5.m5.1.1.1.1.1.3.3" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.cmml"><mrow id="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2.cmml"><mi id="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2.2" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2.1" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2.1.cmml">​</mo><mi id="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2.3" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2.3.cmml">N</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2.1a" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2.1.cmml">​</mo><mi id="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2.4" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2.4.cmml">Z</mi></mrow><mi id="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.3" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.3.cmml">G</mi></mfrac></mrow></mrow><mo stretchy="false" id="S3.SS2.p1.5.m5.1.1.1.1.3" xref="S3.SS2.p1.5.m5.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.5.m5.1b"><apply id="S3.SS2.p1.5.m5.1.1.cmml" xref="S3.SS2.p1.5.m5.1.1"><times id="S3.SS2.p1.5.m5.1.1.2.cmml" xref="S3.SS2.p1.5.m5.1.1.2"></times><ci id="S3.SS2.p1.5.m5.1.1.3.cmml" xref="S3.SS2.p1.5.m5.1.1.3">Θ</ci><apply id="S3.SS2.p1.5.m5.1.1.1.1.1.cmml" xref="S3.SS2.p1.5.m5.1.1.1.1"><plus id="S3.SS2.p1.5.m5.1.1.1.1.1.1.cmml" xref="S3.SS2.p1.5.m5.1.1.1.1.1.1"></plus><apply id="S3.SS2.p1.5.m5.1.1.1.1.1.2.cmml" xref="S3.SS2.p1.5.m5.1.1.1.1.1.2"><times id="S3.SS2.p1.5.m5.1.1.1.1.1.2.1.cmml" xref="S3.SS2.p1.5.m5.1.1.1.1.1.2.1"></times><cn type="integer" id="S3.SS2.p1.5.m5.1.1.1.1.1.2.2.cmml" xref="S3.SS2.p1.5.m5.1.1.1.1.1.2.2">3</cn><ci id="S3.SS2.p1.5.m5.1.1.1.1.1.2.3.cmml" xref="S3.SS2.p1.5.m5.1.1.1.1.1.2.3">𝐵</ci><ci id="S3.SS2.p1.5.m5.1.1.1.1.1.2.4.cmml" xref="S3.SS2.p1.5.m5.1.1.1.1.1.2.4">𝑍</ci><ci id="S3.SS2.p1.5.m5.1.1.1.1.1.2.5.cmml" xref="S3.SS2.p1.5.m5.1.1.1.1.1.2.5">𝑁</ci><ci id="S3.SS2.p1.5.m5.1.1.1.1.1.2.6.cmml" xref="S3.SS2.p1.5.m5.1.1.1.1.1.2.6">𝑑</ci></apply><apply id="S3.SS2.p1.5.m5.1.1.1.1.1.3.cmml" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3"><times id="S3.SS2.p1.5.m5.1.1.1.1.1.3.1.cmml" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3.1"></times><cn type="integer" id="S3.SS2.p1.5.m5.1.1.1.1.1.3.2.cmml" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3.2">2</cn><apply id="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.cmml" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3.3"><divide id="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.1.cmml" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3.3"></divide><apply id="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2.cmml" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2"><times id="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2.1.cmml" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2.1"></times><ci id="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2.2.cmml" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2.2">𝐵</ci><ci id="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2.3.cmml" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2.3">𝑁</ci><ci id="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2.4.cmml" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.2.4">𝑍</ci></apply><ci id="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.3.cmml" xref="S3.SS2.p1.5.m5.1.1.1.1.1.3.3.3">𝐺</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.5.m5.1c">\Theta(3BZNd+2\frac{BNZ}{G})</annotation></semantics></math>.
Table <a href="#S2.T1" title="Table 1 ‣ 2.4 Local Attention Optimization (LAO) ‣ 2 Methodology ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> shows the communication overheads of various distributed attention solutions. The forward communication of RingAttention is the same as BurstAttention, which is <math id="S3.SS2.p1.6.m6.1" class="ltx_Math" alttext="\Theta(2BZNd)" display="inline"><semantics id="S3.SS2.p1.6.m6.1a"><mrow id="S3.SS2.p1.6.m6.1.1" xref="S3.SS2.p1.6.m6.1.1.cmml"><mi mathvariant="normal" id="S3.SS2.p1.6.m6.1.1.3" xref="S3.SS2.p1.6.m6.1.1.3.cmml">Θ</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.1.1.2" xref="S3.SS2.p1.6.m6.1.1.2.cmml">​</mo><mrow id="S3.SS2.p1.6.m6.1.1.1.1" xref="S3.SS2.p1.6.m6.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p1.6.m6.1.1.1.1.2" xref="S3.SS2.p1.6.m6.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p1.6.m6.1.1.1.1.1" xref="S3.SS2.p1.6.m6.1.1.1.1.1.cmml"><mn id="S3.SS2.p1.6.m6.1.1.1.1.1.2" xref="S3.SS2.p1.6.m6.1.1.1.1.1.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.1.1.1.1.1.1" xref="S3.SS2.p1.6.m6.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.6.m6.1.1.1.1.1.3" xref="S3.SS2.p1.6.m6.1.1.1.1.1.3.cmml">B</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.1.1.1.1.1.1a" xref="S3.SS2.p1.6.m6.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.6.m6.1.1.1.1.1.4" xref="S3.SS2.p1.6.m6.1.1.1.1.1.4.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.1.1.1.1.1.1b" xref="S3.SS2.p1.6.m6.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.6.m6.1.1.1.1.1.5" xref="S3.SS2.p1.6.m6.1.1.1.1.1.5.cmml">N</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.6.m6.1.1.1.1.1.1c" xref="S3.SS2.p1.6.m6.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.6.m6.1.1.1.1.1.6" xref="S3.SS2.p1.6.m6.1.1.1.1.1.6.cmml">d</mi></mrow><mo stretchy="false" id="S3.SS2.p1.6.m6.1.1.1.1.3" xref="S3.SS2.p1.6.m6.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.6.m6.1b"><apply id="S3.SS2.p1.6.m6.1.1.cmml" xref="S3.SS2.p1.6.m6.1.1"><times id="S3.SS2.p1.6.m6.1.1.2.cmml" xref="S3.SS2.p1.6.m6.1.1.2"></times><ci id="S3.SS2.p1.6.m6.1.1.3.cmml" xref="S3.SS2.p1.6.m6.1.1.3">Θ</ci><apply id="S3.SS2.p1.6.m6.1.1.1.1.1.cmml" xref="S3.SS2.p1.6.m6.1.1.1.1"><times id="S3.SS2.p1.6.m6.1.1.1.1.1.1.cmml" xref="S3.SS2.p1.6.m6.1.1.1.1.1.1"></times><cn type="integer" id="S3.SS2.p1.6.m6.1.1.1.1.1.2.cmml" xref="S3.SS2.p1.6.m6.1.1.1.1.1.2">2</cn><ci id="S3.SS2.p1.6.m6.1.1.1.1.1.3.cmml" xref="S3.SS2.p1.6.m6.1.1.1.1.1.3">𝐵</ci><ci id="S3.SS2.p1.6.m6.1.1.1.1.1.4.cmml" xref="S3.SS2.p1.6.m6.1.1.1.1.1.4">𝑍</ci><ci id="S3.SS2.p1.6.m6.1.1.1.1.1.5.cmml" xref="S3.SS2.p1.6.m6.1.1.1.1.1.5">𝑁</ci><ci id="S3.SS2.p1.6.m6.1.1.1.1.1.6.cmml" xref="S3.SS2.p1.6.m6.1.1.1.1.1.6">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.6.m6.1c">\Theta(2BZNd)</annotation></semantics></math>, but without GAO and LAO, RingAttention requires a total cost of <math id="S3.SS2.p1.7.m7.1" class="ltx_Math" alttext="\Theta(6BZNd)" display="inline"><semantics id="S3.SS2.p1.7.m7.1a"><mrow id="S3.SS2.p1.7.m7.1.1" xref="S3.SS2.p1.7.m7.1.1.cmml"><mi mathvariant="normal" id="S3.SS2.p1.7.m7.1.1.3" xref="S3.SS2.p1.7.m7.1.1.3.cmml">Θ</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.7.m7.1.1.2" xref="S3.SS2.p1.7.m7.1.1.2.cmml">​</mo><mrow id="S3.SS2.p1.7.m7.1.1.1.1" xref="S3.SS2.p1.7.m7.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p1.7.m7.1.1.1.1.2" xref="S3.SS2.p1.7.m7.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p1.7.m7.1.1.1.1.1" xref="S3.SS2.p1.7.m7.1.1.1.1.1.cmml"><mn id="S3.SS2.p1.7.m7.1.1.1.1.1.2" xref="S3.SS2.p1.7.m7.1.1.1.1.1.2.cmml">6</mn><mo lspace="0em" rspace="0em" id="S3.SS2.p1.7.m7.1.1.1.1.1.1" xref="S3.SS2.p1.7.m7.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.7.m7.1.1.1.1.1.3" xref="S3.SS2.p1.7.m7.1.1.1.1.1.3.cmml">B</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.7.m7.1.1.1.1.1.1a" xref="S3.SS2.p1.7.m7.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.7.m7.1.1.1.1.1.4" xref="S3.SS2.p1.7.m7.1.1.1.1.1.4.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.7.m7.1.1.1.1.1.1b" xref="S3.SS2.p1.7.m7.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.7.m7.1.1.1.1.1.5" xref="S3.SS2.p1.7.m7.1.1.1.1.1.5.cmml">N</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.7.m7.1.1.1.1.1.1c" xref="S3.SS2.p1.7.m7.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.7.m7.1.1.1.1.1.6" xref="S3.SS2.p1.7.m7.1.1.1.1.1.6.cmml">d</mi></mrow><mo stretchy="false" id="S3.SS2.p1.7.m7.1.1.1.1.3" xref="S3.SS2.p1.7.m7.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.7.m7.1b"><apply id="S3.SS2.p1.7.m7.1.1.cmml" xref="S3.SS2.p1.7.m7.1.1"><times id="S3.SS2.p1.7.m7.1.1.2.cmml" xref="S3.SS2.p1.7.m7.1.1.2"></times><ci id="S3.SS2.p1.7.m7.1.1.3.cmml" xref="S3.SS2.p1.7.m7.1.1.3">Θ</ci><apply id="S3.SS2.p1.7.m7.1.1.1.1.1.cmml" xref="S3.SS2.p1.7.m7.1.1.1.1"><times id="S3.SS2.p1.7.m7.1.1.1.1.1.1.cmml" xref="S3.SS2.p1.7.m7.1.1.1.1.1.1"></times><cn type="integer" id="S3.SS2.p1.7.m7.1.1.1.1.1.2.cmml" xref="S3.SS2.p1.7.m7.1.1.1.1.1.2">6</cn><ci id="S3.SS2.p1.7.m7.1.1.1.1.1.3.cmml" xref="S3.SS2.p1.7.m7.1.1.1.1.1.3">𝐵</ci><ci id="S3.SS2.p1.7.m7.1.1.1.1.1.4.cmml" xref="S3.SS2.p1.7.m7.1.1.1.1.1.4">𝑍</ci><ci id="S3.SS2.p1.7.m7.1.1.1.1.1.5.cmml" xref="S3.SS2.p1.7.m7.1.1.1.1.1.5">𝑁</ci><ci id="S3.SS2.p1.7.m7.1.1.1.1.1.6.cmml" xref="S3.SS2.p1.7.m7.1.1.1.1.1.6">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.7.m7.1c">\Theta(6BZNd)</annotation></semantics></math> in the backward pass, which is about twice that of BurstAttention. Therefore, BurstAttention has great advantage of communication overheads during training than RingAttention.
The forward communication of tensor parallelism is <math id="S3.SS2.p1.8.m8.1" class="ltx_Math" alttext="\Theta(4BZNd)" display="inline"><semantics id="S3.SS2.p1.8.m8.1a"><mrow id="S3.SS2.p1.8.m8.1.1" xref="S3.SS2.p1.8.m8.1.1.cmml"><mi mathvariant="normal" id="S3.SS2.p1.8.m8.1.1.3" xref="S3.SS2.p1.8.m8.1.1.3.cmml">Θ</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.8.m8.1.1.2" xref="S3.SS2.p1.8.m8.1.1.2.cmml">​</mo><mrow id="S3.SS2.p1.8.m8.1.1.1.1" xref="S3.SS2.p1.8.m8.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p1.8.m8.1.1.1.1.2" xref="S3.SS2.p1.8.m8.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p1.8.m8.1.1.1.1.1" xref="S3.SS2.p1.8.m8.1.1.1.1.1.cmml"><mn id="S3.SS2.p1.8.m8.1.1.1.1.1.2" xref="S3.SS2.p1.8.m8.1.1.1.1.1.2.cmml">4</mn><mo lspace="0em" rspace="0em" id="S3.SS2.p1.8.m8.1.1.1.1.1.1" xref="S3.SS2.p1.8.m8.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.8.m8.1.1.1.1.1.3" xref="S3.SS2.p1.8.m8.1.1.1.1.1.3.cmml">B</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.8.m8.1.1.1.1.1.1a" xref="S3.SS2.p1.8.m8.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.8.m8.1.1.1.1.1.4" xref="S3.SS2.p1.8.m8.1.1.1.1.1.4.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.8.m8.1.1.1.1.1.1b" xref="S3.SS2.p1.8.m8.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.8.m8.1.1.1.1.1.5" xref="S3.SS2.p1.8.m8.1.1.1.1.1.5.cmml">N</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.8.m8.1.1.1.1.1.1c" xref="S3.SS2.p1.8.m8.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.8.m8.1.1.1.1.1.6" xref="S3.SS2.p1.8.m8.1.1.1.1.1.6.cmml">d</mi></mrow><mo stretchy="false" id="S3.SS2.p1.8.m8.1.1.1.1.3" xref="S3.SS2.p1.8.m8.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.8.m8.1b"><apply id="S3.SS2.p1.8.m8.1.1.cmml" xref="S3.SS2.p1.8.m8.1.1"><times id="S3.SS2.p1.8.m8.1.1.2.cmml" xref="S3.SS2.p1.8.m8.1.1.2"></times><ci id="S3.SS2.p1.8.m8.1.1.3.cmml" xref="S3.SS2.p1.8.m8.1.1.3">Θ</ci><apply id="S3.SS2.p1.8.m8.1.1.1.1.1.cmml" xref="S3.SS2.p1.8.m8.1.1.1.1"><times id="S3.SS2.p1.8.m8.1.1.1.1.1.1.cmml" xref="S3.SS2.p1.8.m8.1.1.1.1.1.1"></times><cn type="integer" id="S3.SS2.p1.8.m8.1.1.1.1.1.2.cmml" xref="S3.SS2.p1.8.m8.1.1.1.1.1.2">4</cn><ci id="S3.SS2.p1.8.m8.1.1.1.1.1.3.cmml" xref="S3.SS2.p1.8.m8.1.1.1.1.1.3">𝐵</ci><ci id="S3.SS2.p1.8.m8.1.1.1.1.1.4.cmml" xref="S3.SS2.p1.8.m8.1.1.1.1.1.4">𝑍</ci><ci id="S3.SS2.p1.8.m8.1.1.1.1.1.5.cmml" xref="S3.SS2.p1.8.m8.1.1.1.1.1.5">𝑁</ci><ci id="S3.SS2.p1.8.m8.1.1.1.1.1.6.cmml" xref="S3.SS2.p1.8.m8.1.1.1.1.1.6">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.8.m8.1c">\Theta(4BZNd)</annotation></semantics></math> and the total communication is <math id="S3.SS2.p1.9.m9.1" class="ltx_Math" alttext="\Theta(8BZNd)" display="inline"><semantics id="S3.SS2.p1.9.m9.1a"><mrow id="S3.SS2.p1.9.m9.1.1" xref="S3.SS2.p1.9.m9.1.1.cmml"><mi mathvariant="normal" id="S3.SS2.p1.9.m9.1.1.3" xref="S3.SS2.p1.9.m9.1.1.3.cmml">Θ</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.9.m9.1.1.2" xref="S3.SS2.p1.9.m9.1.1.2.cmml">​</mo><mrow id="S3.SS2.p1.9.m9.1.1.1.1" xref="S3.SS2.p1.9.m9.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p1.9.m9.1.1.1.1.2" xref="S3.SS2.p1.9.m9.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p1.9.m9.1.1.1.1.1" xref="S3.SS2.p1.9.m9.1.1.1.1.1.cmml"><mn id="S3.SS2.p1.9.m9.1.1.1.1.1.2" xref="S3.SS2.p1.9.m9.1.1.1.1.1.2.cmml">8</mn><mo lspace="0em" rspace="0em" id="S3.SS2.p1.9.m9.1.1.1.1.1.1" xref="S3.SS2.p1.9.m9.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.9.m9.1.1.1.1.1.3" xref="S3.SS2.p1.9.m9.1.1.1.1.1.3.cmml">B</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.9.m9.1.1.1.1.1.1a" xref="S3.SS2.p1.9.m9.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.9.m9.1.1.1.1.1.4" xref="S3.SS2.p1.9.m9.1.1.1.1.1.4.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.9.m9.1.1.1.1.1.1b" xref="S3.SS2.p1.9.m9.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.9.m9.1.1.1.1.1.5" xref="S3.SS2.p1.9.m9.1.1.1.1.1.5.cmml">N</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.9.m9.1.1.1.1.1.1c" xref="S3.SS2.p1.9.m9.1.1.1.1.1.1.cmml">​</mo><mi id="S3.SS2.p1.9.m9.1.1.1.1.1.6" xref="S3.SS2.p1.9.m9.1.1.1.1.1.6.cmml">d</mi></mrow><mo stretchy="false" id="S3.SS2.p1.9.m9.1.1.1.1.3" xref="S3.SS2.p1.9.m9.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.9.m9.1b"><apply id="S3.SS2.p1.9.m9.1.1.cmml" xref="S3.SS2.p1.9.m9.1.1"><times id="S3.SS2.p1.9.m9.1.1.2.cmml" xref="S3.SS2.p1.9.m9.1.1.2"></times><ci id="S3.SS2.p1.9.m9.1.1.3.cmml" xref="S3.SS2.p1.9.m9.1.1.3">Θ</ci><apply id="S3.SS2.p1.9.m9.1.1.1.1.1.cmml" xref="S3.SS2.p1.9.m9.1.1.1.1"><times id="S3.SS2.p1.9.m9.1.1.1.1.1.1.cmml" xref="S3.SS2.p1.9.m9.1.1.1.1.1.1"></times><cn type="integer" id="S3.SS2.p1.9.m9.1.1.1.1.1.2.cmml" xref="S3.SS2.p1.9.m9.1.1.1.1.1.2">8</cn><ci id="S3.SS2.p1.9.m9.1.1.1.1.1.3.cmml" xref="S3.SS2.p1.9.m9.1.1.1.1.1.3">𝐵</ci><ci id="S3.SS2.p1.9.m9.1.1.1.1.1.4.cmml" xref="S3.SS2.p1.9.m9.1.1.1.1.1.4">𝑍</ci><ci id="S3.SS2.p1.9.m9.1.1.1.1.1.5.cmml" xref="S3.SS2.p1.9.m9.1.1.1.1.1.5">𝑁</ci><ci id="S3.SS2.p1.9.m9.1.1.1.1.1.6.cmml" xref="S3.SS2.p1.9.m9.1.1.1.1.1.6">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.9.m9.1c">\Theta(8BZNd)</annotation></semantics></math>, thus BurstAttention also has higher communication efficiency during both inferring and training than tensor parallelism.</p>
</div>
<figure id="S3.T2" class="ltx_table">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_table">Table 2: </span>The first token latency of the LLaMA-7b inference (s).</figcaption>
<div id="S3.T2.4" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:467.6pt;height:98.1pt;vertical-align:-0.9pt;"><span class="ltx_transformed_inner" style="transform:translate(-26.0pt,5.4pt) scale(0.9,0.9) ;">
<table id="S3.T2.4.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S3.T2.4.1.1.1" class="ltx_tr">
<th id="S3.T2.4.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><span id="S3.T2.4.1.1.1.1.1" class="ltx_text" style="font-size:90%;">Sequence Length</span></th>
<th id="S3.T2.4.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T2.4.1.1.1.2.1" class="ltx_text" style="font-size:90%;">4,096</span></th>
<th id="S3.T2.4.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T2.4.1.1.1.3.1" class="ltx_text" style="font-size:90%;">8,192</span></th>
<th id="S3.T2.4.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T2.4.1.1.1.4.1" class="ltx_text" style="font-size:90%;">16,384</span></th>
<th id="S3.T2.4.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T2.4.1.1.1.5.1" class="ltx_text" style="font-size:90%;">32,768</span></th>
<th id="S3.T2.4.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T2.4.1.1.1.6.1" class="ltx_text" style="font-size:90%;">65,536</span></th>
<th id="S3.T2.4.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T2.4.1.1.1.7.1" class="ltx_text" style="font-size:90%;">131,072</span></th>
<th id="S3.T2.4.1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T2.4.1.1.1.8.1" class="ltx_text" style="font-size:90%;">262,144</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S3.T2.4.1.2.1" class="ltx_tr">
<th id="S3.T2.4.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t"><span id="S3.T2.4.1.2.1.1.1" class="ltx_text" style="font-size:90%;">RingAttention</span></th>
<td id="S3.T2.4.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.1.2.1.2.1" class="ltx_text" style="font-size:90%;">0.42±0.01</span></td>
<td id="S3.T2.4.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.1.2.1.3.1" class="ltx_text" style="font-size:90%;">0.87±0.01</span></td>
<td id="S3.T2.4.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.1.2.1.4.1" class="ltx_text" style="font-size:90%;">2.00±0.01</span></td>
<td id="S3.T2.4.1.2.1.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.1.2.1.5.1" class="ltx_text" style="font-size:90%;">5.13±0.05</span></td>
<td id="S3.T2.4.1.2.1.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.1.2.1.6.1" class="ltx_text" style="font-size:90%;">OOM</span></td>
<td id="S3.T2.4.1.2.1.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.1.2.1.7.1" class="ltx_text" style="font-size:90%;">OOM</span></td>
<td id="S3.T2.4.1.2.1.8" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T2.4.1.2.1.8.1" class="ltx_text" style="font-size:90%;">OOM</span></td>
</tr>
<tr id="S3.T2.4.1.3.2" class="ltx_tr">
<th id="S3.T2.4.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S3.T2.4.1.3.2.1.1" class="ltx_text" style="font-size:90%;">TP(Megatron V1) w/ Flash</span></th>
<td id="S3.T2.4.1.3.2.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.1.3.2.2.1" class="ltx_text" style="font-size:90%;">0.67±0.01</span></td>
<td id="S3.T2.4.1.3.2.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.1.3.2.3.1" class="ltx_text" style="font-size:90%;">1.29±0.01</span></td>
<td id="S3.T2.4.1.3.2.4" class="ltx_td ltx_align_center"><span id="S3.T2.4.1.3.2.4.1" class="ltx_text" style="font-size:90%;">2.58±0.01</span></td>
<td id="S3.T2.4.1.3.2.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.1.3.2.5.1" class="ltx_text" style="font-size:90%;">5.27±0.01</span></td>
<td id="S3.T2.4.1.3.2.6" class="ltx_td ltx_align_center"><span id="S3.T2.4.1.3.2.6.1" class="ltx_text" style="font-size:90%;">11.63±0.02</span></td>
<td id="S3.T2.4.1.3.2.7" class="ltx_td ltx_align_center"><span id="S3.T2.4.1.3.2.7.1" class="ltx_text" style="font-size:90%;">27.54±0.01</span></td>
<td id="S3.T2.4.1.3.2.8" class="ltx_td ltx_align_center"><span id="S3.T2.4.1.3.2.8.1" class="ltx_text" style="font-size:90%;">71.52±0.06</span></td>
</tr>
<tr id="S3.T2.4.1.4.3" class="ltx_tr">
<th id="S3.T2.4.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S3.T2.4.1.4.3.1.1" class="ltx_text" style="font-size:90%;">TP(Megatron V3) w/ Flash</span></th>
<td id="S3.T2.4.1.4.3.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.1.4.3.2.1" class="ltx_text" style="font-size:90%;">0.73±0.02</span></td>
<td id="S3.T2.4.1.4.3.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.1.4.3.3.1" class="ltx_text" style="font-size:90%;">1.36±0.01</span></td>
<td id="S3.T2.4.1.4.3.4" class="ltx_td ltx_align_center"><span id="S3.T2.4.1.4.3.4.1" class="ltx_text" style="font-size:90%;">2.68±0.01</span></td>
<td id="S3.T2.4.1.4.3.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.1.4.3.5.1" class="ltx_text" style="font-size:90%;">5.67±0.01</span></td>
<td id="S3.T2.4.1.4.3.6" class="ltx_td ltx_align_center"><span id="S3.T2.4.1.4.3.6.1" class="ltx_text" style="font-size:90%;">12.25±0.01</span></td>
<td id="S3.T2.4.1.4.3.7" class="ltx_td ltx_align_center"><span id="S3.T2.4.1.4.3.7.1" class="ltx_text" style="font-size:90%;">28.73±0.03</span></td>
<td id="S3.T2.4.1.4.3.8" class="ltx_td ltx_align_center"><span id="S3.T2.4.1.4.3.8.1" class="ltx_text" style="font-size:90%;">75.52±0.05</span></td>
</tr>
<tr id="S3.T2.4.1.5.4" class="ltx_tr">
<th id="S3.T2.4.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S3.T2.4.1.5.4.1.1" class="ltx_text" style="font-size:90%;">BurstAttention w/o LAO</span></th>
<td id="S3.T2.4.1.5.4.2" class="ltx_td ltx_align_center"><span id="S3.T2.4.1.5.4.2.1" class="ltx_text" style="font-size:90%;">0.46±0.01</span></td>
<td id="S3.T2.4.1.5.4.3" class="ltx_td ltx_align_center"><span id="S3.T2.4.1.5.4.3.1" class="ltx_text" style="font-size:90%;">0.88±0.01</span></td>
<td id="S3.T2.4.1.5.4.4" class="ltx_td ltx_align_center"><span id="S3.T2.4.1.5.4.4.1" class="ltx_text" style="font-size:90%;">1.79±0.01</span></td>
<td id="S3.T2.4.1.5.4.5" class="ltx_td ltx_align_center"><span id="S3.T2.4.1.5.4.5.1" class="ltx_text" style="font-size:90%;">3.88±0.01</span></td>
<td id="S3.T2.4.1.5.4.6" class="ltx_td ltx_align_center"><span id="S3.T2.4.1.5.4.6.1" class="ltx_text" style="font-size:90%;">10.78±0.01</span></td>
<td id="S3.T2.4.1.5.4.7" class="ltx_td ltx_align_center"><span id="S3.T2.4.1.5.4.7.1" class="ltx_text" style="font-size:90%;">OOM</span></td>
<td id="S3.T2.4.1.5.4.8" class="ltx_td ltx_align_center"><span id="S3.T2.4.1.5.4.8.1" class="ltx_text" style="font-size:90%;">OOM</span></td>
</tr>
<tr id="S3.T2.4.1.6.5" class="ltx_tr">
<th id="S3.T2.4.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="S3.T2.4.1.6.5.1.1" class="ltx_text" style="font-size:90%;">BurstAttention</span></th>
<td id="S3.T2.4.1.6.5.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T2.4.1.6.5.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">0.44±0.01</span></td>
<td id="S3.T2.4.1.6.5.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T2.4.1.6.5.3.1" class="ltx_text ltx_font_bold" style="font-size:90%;">0.84±0.01</span></td>
<td id="S3.T2.4.1.6.5.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T2.4.1.6.5.4.1" class="ltx_text ltx_font_bold" style="font-size:90%;">1.68±0.01</span></td>
<td id="S3.T2.4.1.6.5.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T2.4.1.6.5.5.1" class="ltx_text ltx_font_bold" style="font-size:90%;">3.27±0.01</span></td>
<td id="S3.T2.4.1.6.5.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T2.4.1.6.5.6.1" class="ltx_text ltx_font_bold" style="font-size:90%;">6.49±0.01</span></td>
<td id="S3.T2.4.1.6.5.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T2.4.1.6.5.7.1" class="ltx_text ltx_font_bold" style="font-size:90%;">16.01±0.01</span></td>
<td id="S3.T2.4.1.6.5.8" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T2.4.1.6.5.8.1" class="ltx_text ltx_font_bold" style="font-size:90%;">49.32±0.11</span></td>
</tr>
</tbody>
</table>
</span></div>
</figure>
<figure id="S3.T3" class="ltx_table">
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_table">Table 3: </span>The first token latency of the LLaMA-13b inference (s).</figcaption>
<div id="S3.T3.4" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:471.7pt;height:98.1pt;vertical-align:-0.9pt;"><span class="ltx_transformed_inner" style="transform:translate(-26.2pt,5.4pt) scale(0.9,0.9) ;">
<table id="S3.T3.4.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S3.T3.4.1.1.1" class="ltx_tr">
<th id="S3.T3.4.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><span id="S3.T3.4.1.1.1.1.1" class="ltx_text" style="font-size:90%;">Sequence Length</span></th>
<th id="S3.T3.4.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T3.4.1.1.1.2.1" class="ltx_text" style="font-size:90%;">4,096</span></th>
<th id="S3.T3.4.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T3.4.1.1.1.3.1" class="ltx_text" style="font-size:90%;">8,192</span></th>
<th id="S3.T3.4.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T3.4.1.1.1.4.1" class="ltx_text" style="font-size:90%;">16,384</span></th>
<th id="S3.T3.4.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T3.4.1.1.1.5.1" class="ltx_text" style="font-size:90%;">32,768</span></th>
<th id="S3.T3.4.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T3.4.1.1.1.6.1" class="ltx_text" style="font-size:90%;">65,536</span></th>
<th id="S3.T3.4.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T3.4.1.1.1.7.1" class="ltx_text" style="font-size:90%;">131,072</span></th>
<th id="S3.T3.4.1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S3.T3.4.1.1.1.8.1" class="ltx_text" style="font-size:90%;">262,144</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S3.T3.4.1.2.1" class="ltx_tr">
<th id="S3.T3.4.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t"><span id="S3.T3.4.1.2.1.1.1" class="ltx_text" style="font-size:90%;">RingAttention</span></th>
<td id="S3.T3.4.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T3.4.1.2.1.2.1" class="ltx_text" style="font-size:90%;">0.66±0.01</span></td>
<td id="S3.T3.4.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T3.4.1.2.1.3.1" class="ltx_text" style="font-size:90%;">1.36±0.01</span></td>
<td id="S3.T3.4.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T3.4.1.2.1.4.1" class="ltx_text" style="font-size:90%;">3.08±0.01</span></td>
<td id="S3.T3.4.1.2.1.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T3.4.1.2.1.5.1" class="ltx_text" style="font-size:90%;">7.98±0.02</span></td>
<td id="S3.T3.4.1.2.1.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T3.4.1.2.1.6.1" class="ltx_text" style="font-size:90%;">OOM</span></td>
<td id="S3.T3.4.1.2.1.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T3.4.1.2.1.7.1" class="ltx_text" style="font-size:90%;">OOM</span></td>
<td id="S3.T3.4.1.2.1.8" class="ltx_td ltx_align_center ltx_border_t"><span id="S3.T3.4.1.2.1.8.1" class="ltx_text" style="font-size:90%;">OOM</span></td>
</tr>
<tr id="S3.T3.4.1.3.2" class="ltx_tr">
<th id="S3.T3.4.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S3.T3.4.1.3.2.1.1" class="ltx_text" style="font-size:90%;">TP(Megatron V1) w/ Flash</span></th>
<td id="S3.T3.4.1.3.2.2" class="ltx_td ltx_align_center"><span id="S3.T3.4.1.3.2.2.1" class="ltx_text" style="font-size:90%;">1.05±0.01</span></td>
<td id="S3.T3.4.1.3.2.3" class="ltx_td ltx_align_center"><span id="S3.T3.4.1.3.2.3.1" class="ltx_text" style="font-size:90%;">2.01±0.01</span></td>
<td id="S3.T3.4.1.3.2.4" class="ltx_td ltx_align_center"><span id="S3.T3.4.1.3.2.4.1" class="ltx_text" style="font-size:90%;">4.03±0.01</span></td>
<td id="S3.T3.4.1.3.2.5" class="ltx_td ltx_align_center"><span id="S3.T3.4.1.3.2.5.1" class="ltx_text" style="font-size:90%;">8.41±0.01</span></td>
<td id="S3.T3.4.1.3.2.6" class="ltx_td ltx_align_center"><span id="S3.T3.4.1.3.2.6.1" class="ltx_text" style="font-size:90%;">18.56±0.02</span></td>
<td id="S3.T3.4.1.3.2.7" class="ltx_td ltx_align_center"><span id="S3.T3.4.1.3.2.7.1" class="ltx_text" style="font-size:90%;">44.39±0.04</span></td>
<td id="S3.T3.4.1.3.2.8" class="ltx_td ltx_align_center"><span id="S3.T3.4.1.3.2.8.1" class="ltx_text" style="font-size:90%;">OOM</span></td>
</tr>
<tr id="S3.T3.4.1.4.3" class="ltx_tr">
<th id="S3.T3.4.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S3.T3.4.1.4.3.1.1" class="ltx_text" style="font-size:90%;">TP(Megatron V3) w/ Flash</span></th>
<td id="S3.T3.4.1.4.3.2" class="ltx_td ltx_align_center"><span id="S3.T3.4.1.4.3.2.1" class="ltx_text" style="font-size:90%;">1.07±0.01</span></td>
<td id="S3.T3.4.1.4.3.3" class="ltx_td ltx_align_center"><span id="S3.T3.4.1.4.3.3.1" class="ltx_text" style="font-size:90%;">2.09±0.01</span></td>
<td id="S3.T3.4.1.4.3.4" class="ltx_td ltx_align_center"><span id="S3.T3.4.1.4.3.4.1" class="ltx_text" style="font-size:90%;">4.20±0.01</span></td>
<td id="S3.T3.4.1.4.3.5" class="ltx_td ltx_align_center"><span id="S3.T3.4.1.4.3.5.1" class="ltx_text" style="font-size:90%;">8.76±0.01</span></td>
<td id="S3.T3.4.1.4.3.6" class="ltx_td ltx_align_center"><span id="S3.T3.4.1.4.3.6.1" class="ltx_text" style="font-size:90%;">19.06±0.06</span></td>
<td id="S3.T3.4.1.4.3.7" class="ltx_td ltx_align_center"><span id="S3.T3.4.1.4.3.7.1" class="ltx_text" style="font-size:90%;">45.46±0.03</span></td>
<td id="S3.T3.4.1.4.3.8" class="ltx_td ltx_align_center"><span id="S3.T3.4.1.4.3.8.1" class="ltx_text" style="font-size:90%;">119.03±0.04</span></td>
</tr>
<tr id="S3.T3.4.1.5.4" class="ltx_tr">
<th id="S3.T3.4.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S3.T3.4.1.5.4.1.1" class="ltx_text" style="font-size:90%;">BurstAttention w/o LAO</span></th>
<td id="S3.T3.4.1.5.4.2" class="ltx_td ltx_align_center"><span id="S3.T3.4.1.5.4.2.1" class="ltx_text" style="font-size:90%;">0.72±0.01</span></td>
<td id="S3.T3.4.1.5.4.3" class="ltx_td ltx_align_center"><span id="S3.T3.4.1.5.4.3.1" class="ltx_text" style="font-size:90%;">1.39±0.01</span></td>
<td id="S3.T3.4.1.5.4.4" class="ltx_td ltx_align_center"><span id="S3.T3.4.1.5.4.4.1" class="ltx_text" style="font-size:90%;">2.77±0.05</span></td>
<td id="S3.T3.4.1.5.4.5" class="ltx_td ltx_align_center"><span id="S3.T3.4.1.5.4.5.1" class="ltx_text" style="font-size:90%;">5.99±0.01</span></td>
<td id="S3.T3.4.1.5.4.6" class="ltx_td ltx_align_center"><span id="S3.T3.4.1.5.4.6.1" class="ltx_text" style="font-size:90%;">16.95±0.01</span></td>
<td id="S3.T3.4.1.5.4.7" class="ltx_td ltx_align_center"><span id="S3.T3.4.1.5.4.7.1" class="ltx_text" style="font-size:90%;">OOM</span></td>
<td id="S3.T3.4.1.5.4.8" class="ltx_td ltx_align_center"><span id="S3.T3.4.1.5.4.8.1" class="ltx_text" style="font-size:90%;">OOM</span></td>
</tr>
<tr id="S3.T3.4.1.6.5" class="ltx_tr">
<th id="S3.T3.4.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="S3.T3.4.1.6.5.1.1" class="ltx_text" style="font-size:90%;">BurstAttention</span></th>
<td id="S3.T3.4.1.6.5.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T3.4.1.6.5.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">0.69±0.01</span></td>
<td id="S3.T3.4.1.6.5.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T3.4.1.6.5.3.1" class="ltx_text ltx_font_bold" style="font-size:90%;">1.40±0.05</span></td>
<td id="S3.T3.4.1.6.5.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T3.4.1.6.5.4.1" class="ltx_text ltx_font_bold" style="font-size:90%;">2.57±0.03</span></td>
<td id="S3.T3.4.1.6.5.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T3.4.1.6.5.5.1" class="ltx_text ltx_font_bold" style="font-size:90%;">5.08±0.02</span></td>
<td id="S3.T3.4.1.6.5.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T3.4.1.6.5.6.1" class="ltx_text ltx_font_bold" style="font-size:90%;">9.92±0.01</span></td>
<td id="S3.T3.4.1.6.5.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T3.4.1.6.5.7.1" class="ltx_text ltx_font_bold" style="font-size:90%;">25.91±0.01</span></td>
<td id="S3.T3.4.1.6.5.8" class="ltx_td ltx_align_center ltx_border_bb"><span id="S3.T3.4.1.6.5.8.1" class="ltx_text ltx_font_bold" style="font-size:90%;">78.80±0.07</span></td>
</tr>
</tbody>
</table>
</span></div>
</figure>
<figure id="S3.F2" class="ltx_figure">
<div id="S3.F2.4" class="ltx_block">
<figure id="S3.F2.sf1" class="ltx_figure ltx_align_center"><img src="/html/2403.09347/assets/x2.png" id="S3.F2.sf1.g1" class="ltx_graphics ltx_img_landscape" width="175" height="112" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S3.F2.sf1.2.1.1" class="ltx_text" style="font-size:90%;">(a)</span> </span><span id="S3.F2.sf1.3.2" class="ltx_text" style="font-size:90%;">Training time</span></figcaption>
</figure>
<figure id="S3.F2.sf2" class="ltx_figure ltx_align_center"><img src="/html/2403.09347/assets/x3.png" id="S3.F2.sf2.g1" class="ltx_graphics ltx_img_landscape" width="175" height="112" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S3.F2.sf2.2.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span><span id="S3.F2.sf2.3.2" class="ltx_text" style="font-size:90%;">Training memory</span></figcaption>
</figure>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S3.F2.5.2.1" class="ltx_text" style="font-size:90%;">Figure 2</span>: </span><span id="S3.F2.2.1" class="ltx_text" style="font-size:90%;">The training time and memory of LLaMA-7b on 8<math id="S3.F2.2.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S3.F2.2.1.m1.1b"><mo id="S3.F2.2.1.m1.1.1" xref="S3.F2.2.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S3.F2.2.1.m1.1c"><times id="S3.F2.2.1.m1.1.1.cmml" xref="S3.F2.2.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S3.F2.2.1.m1.1d">\times</annotation></semantics></math>A100.</span></figcaption>
</figure>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Experiments</h2>

<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Experimental Settings</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">We perform experiments in two configurations: one involves a single node equipped with 8 A100 GPUs linked via PCI-E, and the other is a distributed setup comprising four identical nodes, each with the same 8 A100 GPU configuration, interconnected by a 600 Gb/s RoCE network. We adopts two LLMs’ settings in our experiments, LLaMA-2 with 7 billion parameters (7b) and LLaMA-2 with 13 billion parameters (13b) <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib25" title="" class="ltx_ref">2023b</a>)</cite>.
Our experiments consist of the following methods:</p>
</div>
<div id="S4.SS1.p2" class="ltx_para">
<p id="S4.SS1.p2.1" class="ltx_p">(1) <span id="S4.SS1.p2.1.1" class="ltx_text ltx_font_bold">TP</span>, which refers to tensor parallelism <cite class="ltx_cite ltx_citemacro_citep">(Narayanan et al., <a href="#bib.bib17" title="" class="ltx_ref">2021</a>)</cite>, a commonly used distributed strategy in the stages of both training and inference. Note that here we futher classify TP into <span id="S4.SS1.p2.1.2" class="ltx_text ltx_font_bold">TP (Megatron V1)</span> and <span id="S4.SS1.p2.1.3" class="ltx_text ltx_font_bold">TP (Megatron V3)</span> based on the detail communication operations (Megatron V1 uses the all-reduce operation while Megatron V3 uses the combination of the all-gather and reduce-scatter operations).
(2) <span id="S4.SS1.p2.1.4" class="ltx_text ltx_font_bold">TP w/ FlashAttention</span>, which combines FlashAttention <cite class="ltx_cite ltx_citemacro_citep">(Dao et al., <a href="#bib.bib7" title="" class="ltx_ref">2022</a>)</cite> with tensor parallelism as a strong baseline. <span id="S4.SS1.p2.1.5" class="ltx_text ltx_font_bold">Note that this is a commonly used strategy in current LLM pre-training and inference.</span>
(3) <span id="S4.SS1.p2.1.6" class="ltx_text ltx_font_bold">RingAttention</span>, a typical sequence parallelism baseline.
(4) <span id="S4.SS1.p2.1.7" class="ltx_text ltx_font_bold">BurstAttention</span>, our distributed attention method includes both GAO and LAO strategies.
(5) <span id="S4.SS1.p2.1.8" class="ltx_text ltx_font_bold">BurstAttention w/o LAO</span>, where we remove the LAO strategy for ablation studies.
(6) <span id="S4.SS1.p2.1.9" class="ltx_text ltx_font_bold">BurstAttention+ZeRO</span> , where we futher optimize the memory overhead of BurstAttention by adopting the ZeRO<cite class="ltx_cite ltx_citemacro_citep">(Rajbhandari et al., <a href="#bib.bib22" title="" class="ltx_ref">2020</a>)</cite> technique to shard model parameters across devices.</p>
</div>
<div id="S4.SS1.p3" class="ltx_para">
<p id="S4.SS1.p3.1" class="ltx_p">As we mentioned before, data parallelism and pipeline parallelism cannot effectively reduce the cost of long sequence processing, and we do not use them as baselines. In fact, we conduct some experiments to adapt data parallelism and pipeline parallelism for long-sequence attention, but unfortunately, these two parallelism methods cannot process extremely long sequences. <span id="S4.SS1.p3.1.1" class="ltx_text ltx_font_bold">From our pilot experiments, directly adopting data parallelism or pipeline parallelism can only handle sequences shorter than 8192, much shorter than RingAttention and TP.</span></p>
</div>
<div id="S4.SS1.p4" class="ltx_para">
<p id="S4.SS1.p4.1" class="ltx_p">Our experiments does not specifically focus on any particular attention masking mechanism. However, for the methods we compared against, such as Tensor Parallelism (Megatron V3) with FlashAttention, we adopt its causal implementation in these experiments. This means that our baselines can bypass half of the attention computations owing to the causal attention structure. We observe that this approach yields only a marginal improvement, as communication remains the bottleneck in our experimental environment. Notably, in our implementation of BurstAttention, the computation is overlapped by the communication, which is a key factor in the observed performance gains. This distinction is crucial to understand the context and the specific conditions under which our method demonstrates its advantages.</p>
</div>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Inference Latency</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">In this section, we focus on the latency needed for generating the first token (i.e., the first token latency) in the inference process. We concentrate on the time of the first token generation because the long-sequence attention computation mainly exists in the inference encoding process. Since the first token latency is much higher than the latency of generating subsequent tokens, the first token latency thus becomes one of the most critical targets existing works seek to optimize.</p>
</div>
<div id="S4.SS2.p2" class="ltx_para">
<p id="S4.SS2.p2.1" class="ltx_p">In real-time AI services such as ChatGPT, the system’s responsiveness significantly impacts the user experience, and these applications usually output results in a streaming manner to improve responsiveness. Since the first token latency is the longest, the first token latency directly influences the perceived responsiveness and efficiency of the model in these streaming scenarios.</p>
</div>
<div id="S4.SS2.p3" class="ltx_para">
<p id="S4.SS2.p3.1" class="ltx_p">As shown in Table <a href="#S3.T2" title="Table 2 ‣ 3.2 Communication Overheads ‣ 3 Overhead Analysis ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> and Table <a href="#S3.T3" title="Table 3 ‣ 3.2 Communication Overheads ‣ 3 Overhead Analysis ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, we can see that, compared with tensor parallelism, sequence parallelism methods are more suitable to infer long sequences. Compared with the RingAttention method, by using GAO, BurstAttention can support longer sequences. By further using LAO, BurstAttention can achieve more latency improvements and support much longer sequences. Note that, although TP (Megatron V3) is more memory efficient than TP (Megatron V1), the all-reduce operation used by TP (Megatron V1) is better optimized than the reduce-scatter and all-gather operations used by TP(Megatron V3). In the actual inference, TP(Megatron V1) is slightly faster than TP (Megatron V3). Since TP (Megatron V3) has a similar time to TP (Megatron V1) but better memory efficiency, we mainly compare our method with TP (Megatron V3) in subsequent experiments.</p>
</div>
<figure id="S4.F3" class="ltx_figure">
<div id="S4.F3.4" class="ltx_block">
<figure id="S4.F3.sf1" class="ltx_figure ltx_align_center"><img src="/html/2403.09347/assets/x4.png" id="S4.F3.sf1.g1" class="ltx_graphics ltx_img_landscape" width="175" height="112" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S4.F3.sf1.2.1.1" class="ltx_text" style="font-size:90%;">(a)</span> </span><span id="S4.F3.sf1.3.2" class="ltx_text" style="font-size:90%;">Training time</span></figcaption>
</figure>
<figure id="S4.F3.sf2" class="ltx_figure ltx_align_center"><img src="/html/2403.09347/assets/x5.png" id="S4.F3.sf2.g1" class="ltx_graphics ltx_img_landscape" width="175" height="112" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S4.F3.sf2.2.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span><span id="S4.F3.sf2.3.2" class="ltx_text" style="font-size:90%;">Training memory</span></figcaption>
</figure>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F3.5.2.1" class="ltx_text" style="font-size:90%;">Figure 3</span>: </span><span id="S4.F3.2.1" class="ltx_text" style="font-size:90%;">The training time and memory of LLaMA-7b on 32<math id="S4.F3.2.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.F3.2.1.m1.1b"><mo id="S4.F3.2.1.m1.1.1" xref="S4.F3.2.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.F3.2.1.m1.1c"><times id="S4.F3.2.1.m1.1.1.cmml" xref="S4.F3.2.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.F3.2.1.m1.1d">\times</annotation></semantics></math>A100.</span></figcaption>
</figure>
<figure id="S4.F4" class="ltx_figure">
<div id="S4.F4.2" class="ltx_block">
<figure id="S4.F4.sf1" class="ltx_figure ltx_align_center"><img src="/html/2403.09347/assets/x6.png" id="S4.F4.sf1.g1" class="ltx_graphics ltx_img_landscape" width="175" height="108" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S4.F4.sf1.2.1.1" class="ltx_text" style="font-size:90%;">(a)</span> </span><span id="S4.F4.sf1.3.2" class="ltx_text" style="font-size:90%;">LLaMA-13b latency - GPU number</span></figcaption>
</figure>
<figure id="S4.F4.sf2" class="ltx_figure ltx_align_center"><img src="/html/2403.09347/assets/x7.png" id="S4.F4.sf2.g1" class="ltx_graphics ltx_img_landscape" width="175" height="108" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S4.F4.sf2.2.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span><span id="S4.F4.sf2.3.2" class="ltx_text" style="font-size:90%;">LLaMA-7b throughput - batch size </span></figcaption>
</figure>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F4.3.1.1" class="ltx_text" style="font-size:90%;">Figure 4</span>: </span><span id="S4.F4.4.2" class="ltx_text" style="font-size:90%;">Scaling abilities on different GPU numbers and batch sizes.</span></figcaption>
</figure>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Training Performance</h3>

<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p">For training LLMs, a batch is required to have 2 to 4 million tokens, otherwise, the model performance may be degraded, i.e., the longer the sequence length is, the smaller the batch size is. Due to this, several GPUs may need to process one example together.
For example, using 2048 GPUs to train 128-layer GPT-3, the sequence length is 4096, the batch size is 1024, data parallelism is 16, pipeline parallelism is 32, and tensor parallelism is 4. In this scenario, the optimal setup is to divide a batch into 64 micro-batches with a micro-batch size of 1.
In this case, four GPUs under the same tensor parallelism group are inevitably required to process one piece of data together. In view of this, we fix the batch size to 1 for experimental convenience and vary the input sequence length from 1K to 32K.
</p>
</div>
<div id="S4.SS3.p2" class="ltx_para">
<p id="S4.SS3.p2.1" class="ltx_p">As can be seen from Figure <a href="#S3.F2.sf1" title="Figure 2(a) ‣ Figure 2 ‣ 3.2 Communication Overheads ‣ 3 Overhead Analysis ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2(a)</span></a>, although tensor parallelism adopts FlashAttention to improve its processing of long sequences, both RingAttention and BurstAttention have better training time than tensor parallelism when processing long sequences. This is also why existing works using tensor parallelism to train LLMs usually set the training length between 2048 and 4096. Compared with BurstAttention, RingAttention is limited by the sequence length since it stores too many intermediate states, but BurstAttention can support the longest input length. On the other hand, BurstAttention without LAO has a similar trend of training time as RingAttention and tensor parallelism.</p>
</div>
<div id="S4.SS3.p3" class="ltx_para">
<p id="S4.SS3.p3.1" class="ltx_p">From Figure <a href="#S4.F3" title="Figure 3 ‣ 4.2 Inference Latency ‣ 4 Experiments ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, BurstAttention achieves nearly 2.0<math id="S4.SS3.p3.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.SS3.p3.1.m1.1a"><mo id="S4.SS3.p3.1.m1.1.1" xref="S4.SS3.p3.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.1.m1.1b"><times id="S4.SS3.p3.1.m1.1.1.cmml" xref="S4.SS3.p3.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.1.m1.1c">\times</annotation></semantics></math> speedup when the sequence is longer than 128K. Also combining BurstAttention with ZeRO optimization brings significant improvements in memory efficiency. Although BurstAttention+ZeRO brings little additional communication overheads, BurstAttention+ZeRO still achieves memory efficiency comparable to Megatron V3 and demonstrates superior speed in both multi-node and single-node setups than Megatron V3. This suggests that BurstAttention, with its current optimizations, offers a more efficient solution in terms of speed, even when faced with a memory-efficient competitor like Megatron V3.</p>
</div>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.4 </span>Scaling Ability</h3>

<div id="S4.SS4.p1" class="ltx_para">
<p id="S4.SS4.p1.1" class="ltx_p">In this section, we further verify the scaling ability of BurstAttention.
In Figure <a href="#S4.F4.sf1" title="Figure 4(a) ‣ Figure 4 ‣ 4.2 Inference Latency ‣ 4 Experiments ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4(a)</span></a>, we set batch size to 1 and sequence length to 65,536, and then evaluate the latency changes with increasing GPU numbers. As shown in the figure, in the single-GPU scenario, BurstAttention with LAO is equivalent to FlashAttention, and its inference latency is on par with the baseline using FlashAttention. Tensor parallelism cannot further decrease the latency when the number of GPUs increases from 4 to 8 due to the communication overhead with increased batch-size, while BurstAttention can achieve better scaling trends. Note that RingAttention requires storing <math id="S4.SS4.p1.1.m1.1" class="ltx_Math" alttext="\Theta(\frac{BZN^{2}}{G})" display="inline"><semantics id="S4.SS4.p1.1.m1.1a"><mrow id="S4.SS4.p1.1.m1.1.2" xref="S4.SS4.p1.1.m1.1.2.cmml"><mi mathvariant="normal" id="S4.SS4.p1.1.m1.1.2.2" xref="S4.SS4.p1.1.m1.1.2.2.cmml">Θ</mi><mo lspace="0em" rspace="0em" id="S4.SS4.p1.1.m1.1.2.1" xref="S4.SS4.p1.1.m1.1.2.1.cmml">​</mo><mrow id="S4.SS4.p1.1.m1.1.2.3.2" xref="S4.SS4.p1.1.m1.1.1.cmml"><mo stretchy="false" id="S4.SS4.p1.1.m1.1.2.3.2.1" xref="S4.SS4.p1.1.m1.1.1.cmml">(</mo><mfrac id="S4.SS4.p1.1.m1.1.1" xref="S4.SS4.p1.1.m1.1.1.cmml"><mrow id="S4.SS4.p1.1.m1.1.1.2" xref="S4.SS4.p1.1.m1.1.1.2.cmml"><mi id="S4.SS4.p1.1.m1.1.1.2.2" xref="S4.SS4.p1.1.m1.1.1.2.2.cmml">B</mi><mo lspace="0em" rspace="0em" id="S4.SS4.p1.1.m1.1.1.2.1" xref="S4.SS4.p1.1.m1.1.1.2.1.cmml">​</mo><mi id="S4.SS4.p1.1.m1.1.1.2.3" xref="S4.SS4.p1.1.m1.1.1.2.3.cmml">Z</mi><mo lspace="0em" rspace="0em" id="S4.SS4.p1.1.m1.1.1.2.1a" xref="S4.SS4.p1.1.m1.1.1.2.1.cmml">​</mo><msup id="S4.SS4.p1.1.m1.1.1.2.4" xref="S4.SS4.p1.1.m1.1.1.2.4.cmml"><mi id="S4.SS4.p1.1.m1.1.1.2.4.2" xref="S4.SS4.p1.1.m1.1.1.2.4.2.cmml">N</mi><mn id="S4.SS4.p1.1.m1.1.1.2.4.3" xref="S4.SS4.p1.1.m1.1.1.2.4.3.cmml">2</mn></msup></mrow><mi id="S4.SS4.p1.1.m1.1.1.3" xref="S4.SS4.p1.1.m1.1.1.3.cmml">G</mi></mfrac><mo stretchy="false" id="S4.SS4.p1.1.m1.1.2.3.2.2" xref="S4.SS4.p1.1.m1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS4.p1.1.m1.1b"><apply id="S4.SS4.p1.1.m1.1.2.cmml" xref="S4.SS4.p1.1.m1.1.2"><times id="S4.SS4.p1.1.m1.1.2.1.cmml" xref="S4.SS4.p1.1.m1.1.2.1"></times><ci id="S4.SS4.p1.1.m1.1.2.2.cmml" xref="S4.SS4.p1.1.m1.1.2.2">Θ</ci><apply id="S4.SS4.p1.1.m1.1.1.cmml" xref="S4.SS4.p1.1.m1.1.2.3.2"><divide id="S4.SS4.p1.1.m1.1.1.1.cmml" xref="S4.SS4.p1.1.m1.1.2.3.2"></divide><apply id="S4.SS4.p1.1.m1.1.1.2.cmml" xref="S4.SS4.p1.1.m1.1.1.2"><times id="S4.SS4.p1.1.m1.1.1.2.1.cmml" xref="S4.SS4.p1.1.m1.1.1.2.1"></times><ci id="S4.SS4.p1.1.m1.1.1.2.2.cmml" xref="S4.SS4.p1.1.m1.1.1.2.2">𝐵</ci><ci id="S4.SS4.p1.1.m1.1.1.2.3.cmml" xref="S4.SS4.p1.1.m1.1.1.2.3">𝑍</ci><apply id="S4.SS4.p1.1.m1.1.1.2.4.cmml" xref="S4.SS4.p1.1.m1.1.1.2.4"><csymbol cd="ambiguous" id="S4.SS4.p1.1.m1.1.1.2.4.1.cmml" xref="S4.SS4.p1.1.m1.1.1.2.4">superscript</csymbol><ci id="S4.SS4.p1.1.m1.1.1.2.4.2.cmml" xref="S4.SS4.p1.1.m1.1.1.2.4.2">𝑁</ci><cn type="integer" id="S4.SS4.p1.1.m1.1.1.2.4.3.cmml" xref="S4.SS4.p1.1.m1.1.1.2.4.3">2</cn></apply></apply><ci id="S4.SS4.p1.1.m1.1.1.3.cmml" xref="S4.SS4.p1.1.m1.1.1.3">𝐺</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p1.1.m1.1c">\Theta(\frac{BZN^{2}}{G})</annotation></semantics></math> memory for each layer, which is extremely large and cannot fit into GPUs even sharded on 8 GPUs. In Figure <a href="#S4.F4.sf2" title="Figure 4(b) ‣ Figure 4 ‣ 4.2 Inference Latency ‣ 4 Experiments ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4(b)</span></a>, we fix the sequence length to 4096 and the number of GPUs to 8 to evaluate the training throughput changes with increasing batch sizes. The experimental results show that BurstAttention can support a larger batch size, and the throughput grows with the increase of batch sizes in training scenario.</p>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Related Work</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">Transformer-based LLMs such as GPT <cite class="ltx_cite ltx_citemacro_citep">(Brown et al., <a href="#bib.bib3" title="" class="ltx_ref">2020</a>; Ouyang et al., <a href="#bib.bib18" title="" class="ltx_ref">2022</a>)</cite>, LLaMA <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib24" title="" class="ltx_ref">2023a</a>, <a href="#bib.bib25" title="" class="ltx_ref">b</a>)</cite>, and PaLM <cite class="ltx_cite ltx_citemacro_citep">(Chowdhery et al., <a href="#bib.bib6" title="" class="ltx_ref">2022</a>; Anil et al., <a href="#bib.bib1" title="" class="ltx_ref">2023</a>)</cite> have achieved great success <cite class="ltx_cite ltx_citemacro_citep">(Han et al., <a href="#bib.bib9" title="" class="ltx_ref">2021</a>; Bommasani et al., <a href="#bib.bib2" title="" class="ltx_ref">2021</a>; Zhao et al., <a href="#bib.bib30" title="" class="ltx_ref">2023</a>)</cite>. Despite the success of these LLMs, they still face efficiency challenges: one is that as these models continue to grow in size, the time and memory costs associated with training and inference have become bottlenecks. Another is that the quadratic attention computational complexity of the Transformer architecture makes these LLMs difficult to handle long sequences. Up to now, various parallelism strategies <cite class="ltx_cite ltx_citemacro_citep">(Valiant, <a href="#bib.bib26" title="" class="ltx_ref">1990</a>; Huang et al., <a href="#bib.bib10" title="" class="ltx_ref">2019</a>; Rajbhandari et al., <a href="#bib.bib22" title="" class="ltx_ref">2020</a>; Narayanan et al., <a href="#bib.bib17" title="" class="ltx_ref">2021</a>)</cite> and memory optimization strategies <cite class="ltx_cite ltx_citemacro_citep">(Ren et al., <a href="#bib.bib23" title="" class="ltx_ref">2021</a>; Chen et al., <a href="#bib.bib4" title="" class="ltx_ref">2016</a>; Korthikanti et al., <a href="#bib.bib13" title="" class="ltx_ref">2023</a>)</cite>,
have well solved the bottleneck caused by the model size growth, but it is still challenging to solve the efficiency issue caused by the sequence growth.</p>
</div>
<div id="S5.p2" class="ltx_para">
<p id="S5.p2.1" class="ltx_p">To enable LLMs to process longer sequences more efficiently, several attention solutions have been proposed. <cite class="ltx_cite ltx_citemacro_citet">Korthikanti et al. (<a href="#bib.bib13" title="" class="ltx_ref">2023</a>)</cite> adopt selective activation recomputation to avoid storing attention softmax logits during the forward pass, and then recompute these logits during the backward pass to build a computation graph for backpropagation, significantly reducing memory overheads of attention modules to process long sequences. <cite class="ltx_cite ltx_citemacro_citet">Rabe &amp; Staats (<a href="#bib.bib20" title="" class="ltx_ref">2021</a>)</cite> formalize the computation of attention modules at the block level and make each thread block in devices handle the attention computation of a subsequence, further reducing temporary memory consumptions and achieving a logarithmic memory complexity relative to the sequence length. Based on these works, <cite class="ltx_cite ltx_citemacro_citet">Dao et al. (<a href="#bib.bib7" title="" class="ltx_ref">2022</a>)</cite> introduce FlashAttention, a CUDA implementation of attention modules that leverages the fast I/O capabilities of the SRAM in devices for further speedup. FlashAttention optimizes the attention algorithm by introducing I/O complexity analysis and minimizing the I/O costs on the HBM in devices, offering a new perspective on attention optimization.</p>
</div>
<div id="S5.p3" class="ltx_para">
<p id="S5.p3.1" class="ltx_p">While the above solutions focus on optimizing the long-sequence attention problem using a single device, they still struggle to handle extremely long sequences due to the limitations of a single device’s performance. Some efforts have therefore aimed to address this long-sequence challenge using distributed clusters.
Adopting general parallelism strategies is most straightforward method, such as data parallelism <cite class="ltx_cite ltx_citemacro_citep">(Valiant, <a href="#bib.bib26" title="" class="ltx_ref">1990</a>)</cite>, tensor parallelism <cite class="ltx_cite ltx_citemacro_citep">(Narayanan et al., <a href="#bib.bib17" title="" class="ltx_ref">2021</a>)</cite>, pipeline parallelism <cite class="ltx_cite ltx_citemacro_citep">(Huang et al., <a href="#bib.bib10" title="" class="ltx_ref">2019</a>)</cite>, and zero redundancy optimizer <cite class="ltx_cite ltx_citemacro_citep">(Rajbhandari et al., <a href="#bib.bib22" title="" class="ltx_ref">2020</a>; Ren et al., <a href="#bib.bib23" title="" class="ltx_ref">2021</a>)</cite>. To better process long sequences using distributed clusters, <cite class="ltx_cite ltx_citemacro_citet">Li et al. (<a href="#bib.bib15" title="" class="ltx_ref">2021</a>)</cite> propose the sequence parallelism method RingAttention, which splits the computation and memory overheads of attention modules across multiple devices following the sequence dimension.</p>
</div>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Conclusion</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">We present an efficient distributed attention framework named BurstAttention, which can enhance performance in terms of memory consumption and running speed when processing extremely long sequences. When running on a single device, BurstAttention can achieve comparable efficiency to FlashAttention. When running on a distributed cluster, BurstAttention can outperform existing competitive distributed attention solutions, including RingAttention and tensor parallelism. Moreover, the experimental results show that BurstAttention also has greater scaling abilities than existing solutions as increasing devices and batch sizes.</p>
</div>
</section>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Acknowledgements</h2>

<div id="Sx1.p1" class="ltx_para">
<p id="Sx1.p1.1" class="ltx_p">Thanks for the hardware support of OpenBMB and technique support of Huawei.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Anil et al. (2023)</span>
<span class="ltx_bibblock">
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al.

</span>
<span class="ltx_bibblock">PaLM 2 technical report.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.10403</em>, 2023.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bommasani et al. (2021)</span>
<span class="ltx_bibblock">
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al.

</span>
<span class="ltx_bibblock">On the opportunities and risks of foundation models.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2108.07258</em>, 2021.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al. (2020)</span>
<span class="ltx_bibblock">
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.

</span>
<span class="ltx_bibblock">Language models are few-shot learners.

</span>
<span class="ltx_bibblock">In <em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">Proceedings of NeurIPS</em>, pp.  1877–1901, 2020.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2016)</span>
<span class="ltx_bibblock">
Chen, T., Xu, B., Zhang, C., and Guestrin, C.

</span>
<span class="ltx_bibblock">Training deep nets with sublinear memory cost.

</span>
<span class="ltx_bibblock"><em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1604.06174</em>, 2016.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Choromanski et al. (2020)</span>
<span class="ltx_bibblock">
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al.

</span>
<span class="ltx_bibblock">Rethinking attention with performers.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2009.14794</em>, 2020.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chowdhery et al. (2022)</span>
<span class="ltx_bibblock">
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al.

</span>
<span class="ltx_bibblock">PaLM: Scaling language modeling with pathways.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2204.02311</em>, 2022.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dao et al. (2022)</span>
<span class="ltx_bibblock">
Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C.

</span>
<span class="ltx_bibblock">FlashAttention: Fast and memory-efficient exact attention with io-awareness.

</span>
<span class="ltx_bibblock">In <em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">Proceedings of NeurIPS</em>, pp.  16344–16359, 2022.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ding et al. (2023)</span>
<span class="ltx_bibblock">
Ding, J., Ma, S., Dong, L., Zhang, X., Huang, S., Wang, W., and Wei, F.

</span>
<span class="ltx_bibblock">LongNet: Scaling transformers to 1,000,000,000 tokens.

</span>
<span class="ltx_bibblock"><em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2307.02486</em>, 2023.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Han et al. (2021)</span>
<span class="ltx_bibblock">
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al.

</span>
<span class="ltx_bibblock">Pre-trained models: Past, present and future.

</span>
<span class="ltx_bibblock"><em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">AI Open</em>, 2:225–250, 2021.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Huang et al. (2019)</span>
<span class="ltx_bibblock">
Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, M. X., Chen, D., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al.

</span>
<span class="ltx_bibblock">GPipe: efficient training of giant neural networks using pipeline parallelism.

</span>
<span class="ltx_bibblock">In <em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">Proceedings of NuerIPS</em>, pp.  103–112, 2019.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jaegle et al. (2021)</span>
<span class="ltx_bibblock">
Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., and Carreira, J.

</span>
<span class="ltx_bibblock">Perceiver: General perception with iterative attention.

</span>
<span class="ltx_bibblock">In <em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">Proceedings of ICML</em>, pp.  4651–4664, 2021.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Katharopoulos et al. (2020)</span>
<span class="ltx_bibblock">
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F.

</span>
<span class="ltx_bibblock">Transformers are RNNs: Fast autoregressive transformers with linear attention.

</span>
<span class="ltx_bibblock">In <em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">Proceedings of ICML</em>, pp.  5156–5165, 2020.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Korthikanti et al. (2023)</span>
<span class="ltx_bibblock">
Korthikanti, V. A., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B.

</span>
<span class="ltx_bibblock">Reducing activation recomputation in large transformer models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">Proceedings of MLSYS</em>, 2023.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lee et al. (2019)</span>
<span class="ltx_bibblock">
Lee, J., Lee, Y., Kim, J., Kosiorek, A., Choi, S., and Teh, Y. W.

</span>
<span class="ltx_bibblock">Set Transformer: A framework for attention-based permutation-invariant neural networks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">Proceedings of ICML</em>, pp.  3744–3753, 2019.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2021)</span>
<span class="ltx_bibblock">
Li, S., Xue, F., Baranwal, C., Li, Y., and You, Y.

</span>
<span class="ltx_bibblock">Sequence parallelism: Long sequence training from system perspective.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2105.13120</em>, 2021.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Milakov &amp; Gimelshein (2018)</span>
<span class="ltx_bibblock">
Milakov, M. and Gimelshein, N.

</span>
<span class="ltx_bibblock">Online normalizer calculation for softmax.

</span>
<span class="ltx_bibblock"><em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1805.02867</em>, 2018.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Narayanan et al. (2021)</span>
<span class="ltx_bibblock">
Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., et al.

</span>
<span class="ltx_bibblock">Efficient large-scale language model training on gpu clusters using Megatron-LM.

</span>
<span class="ltx_bibblock">In <em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">Proceedings of SC</em>, 2021.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ouyang et al. (2022)</span>
<span class="ltx_bibblock">
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.

</span>
<span class="ltx_bibblock">Training language models to follow instructions with human feedback.

</span>
<span class="ltx_bibblock">pp.  27730–27744, 2022.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Qin et al. (2022)</span>
<span class="ltx_bibblock">
Qin, Z., Han, X., Sun, W., Li, D., Kong, L., Barnes, N., and Zhong, Y.

</span>
<span class="ltx_bibblock">The devil in linear transformer.

</span>
<span class="ltx_bibblock">In <em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">Proceedings of EMNLP</em>, pp.  7025–7041, 2022.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rabe &amp; Staats (2021)</span>
<span class="ltx_bibblock">
Rabe, M. N. and Staats, C.

</span>
<span class="ltx_bibblock">Self-attention does not need <math id="bib.bib20.1.m1.1" class="ltx_Math" alttext="o(n^{2})" display="inline"><semantics id="bib.bib20.1.m1.1a"><mrow id="bib.bib20.1.m1.1.1" xref="bib.bib20.1.m1.1.1.cmml"><mi id="bib.bib20.1.m1.1.1.3" xref="bib.bib20.1.m1.1.1.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="bib.bib20.1.m1.1.1.2" xref="bib.bib20.1.m1.1.1.2.cmml">​</mo><mrow id="bib.bib20.1.m1.1.1.1.1" xref="bib.bib20.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="bib.bib20.1.m1.1.1.1.1.2" xref="bib.bib20.1.m1.1.1.1.1.1.cmml">(</mo><msup id="bib.bib20.1.m1.1.1.1.1.1" xref="bib.bib20.1.m1.1.1.1.1.1.cmml"><mi id="bib.bib20.1.m1.1.1.1.1.1.2" xref="bib.bib20.1.m1.1.1.1.1.1.2.cmml">n</mi><mn id="bib.bib20.1.m1.1.1.1.1.1.3" xref="bib.bib20.1.m1.1.1.1.1.1.3.cmml">2</mn></msup><mo stretchy="false" id="bib.bib20.1.m1.1.1.1.1.3" xref="bib.bib20.1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="bib.bib20.1.m1.1b"><apply id="bib.bib20.1.m1.1.1.cmml" xref="bib.bib20.1.m1.1.1"><times id="bib.bib20.1.m1.1.1.2.cmml" xref="bib.bib20.1.m1.1.1.2"></times><ci id="bib.bib20.1.m1.1.1.3.cmml" xref="bib.bib20.1.m1.1.1.3">𝑜</ci><apply id="bib.bib20.1.m1.1.1.1.1.1.cmml" xref="bib.bib20.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="bib.bib20.1.m1.1.1.1.1.1.1.cmml" xref="bib.bib20.1.m1.1.1.1.1">superscript</csymbol><ci id="bib.bib20.1.m1.1.1.1.1.1.2.cmml" xref="bib.bib20.1.m1.1.1.1.1.1.2">𝑛</ci><cn type="integer" id="bib.bib20.1.m1.1.1.1.1.1.3.cmml" xref="bib.bib20.1.m1.1.1.1.1.1.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="bib.bib20.1.m1.1c">o(n^{2})</annotation></semantics></math> memory.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.2.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2112.05682</em>, 2021.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Raffel et al. (2020)</span>
<span class="ltx_bibblock">
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J.

</span>
<span class="ltx_bibblock">Exploring the limits of transfer learning with a unified text-to-text transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">The Journal of Machine Learning Research</em>, 21:5485–5551, 2020.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rajbhandari et al. (2020)</span>
<span class="ltx_bibblock">
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y.

</span>
<span class="ltx_bibblock">ZeRO: Memory optimizations toward training trillion parameter models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">Proceedings of SC</em>, 2020.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ren et al. (2021)</span>
<span class="ltx_bibblock">
Ren, J., Rajbhandari, S., Aminabadi, R. Y., Ruwase, O., Yang, S., Zhang, M., Li, D., and He, Y.

</span>
<span class="ltx_bibblock">ZeRO-Offload: Democratizing billion-scale model training.

</span>
<span class="ltx_bibblock">In <em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">Proceedings of ATC</em>, pp.  551–564, 2021.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023a)</span>
<span class="ltx_bibblock">
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.

</span>
<span class="ltx_bibblock">LLaMA: Open and efficient foundation language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2302.13971</em>, 2023a.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023b)</span>
<span class="ltx_bibblock">
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.

</span>
<span class="ltx_bibblock">LLaMA 2: Open foundation and fine-tuned chat models.

</span>
<span class="ltx_bibblock"><em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2307.09288</em>, 2023b.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Valiant (1990)</span>
<span class="ltx_bibblock">
Valiant, L. G.

</span>
<span class="ltx_bibblock">A bridging model for parallel computation.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">Communications of the ACM</em>, pp.  103–111, 1990.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vaswani et al. (2017)</span>
<span class="ltx_bibblock">
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.

</span>
<span class="ltx_bibblock">Attention is all you need.

</span>
<span class="ltx_bibblock">In <em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Proceedings of NeurIPS</em>, 2017.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2020)</span>
<span class="ltx_bibblock">
Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H.

</span>
<span class="ltx_bibblock">Linformer: Self-attention with linear complexity.

</span>
<span class="ltx_bibblock"><em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2006.04768</em>, 2020.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Winata et al. (2020)</span>
<span class="ltx_bibblock">
Winata, G. I., Cahyawijaya, S., Lin, Z., Liu, Z., and Fung, P.

</span>
<span class="ltx_bibblock">Lightweight and efficient end-to-end speech recognition using low-rank transformer.

</span>
<span class="ltx_bibblock">In <em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">Proceedings of ICASSP</em>, pp.  6144–6148, 2020.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhao et al. (2023)</span>
<span class="ltx_bibblock">
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al.

</span>
<span class="ltx_bibblock">A survey of large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2303.18223</em>, 2023.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>BurstAttention Algorithm with Double-buffer</h2>

<figure id="alg3" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="alg3.2" class="ltx_p ltx_figure_panel">!ht
<span id="alg3.2.1" class="ltx_text" style="font-size:80%;"></span></p>
</div>
</div>
<figcaption class="ltx_caption" style="font-size:80%;"><span class="ltx_tag ltx_tag_float"><span id="alg3.5.1.1" class="ltx_text ltx_font_bold">Algorithm 3</span> </span> The forward pass of GAO with overlapping</figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="alg3.6" class="ltx_listing ltx_figure_panel ltx_listing">
<div id="algx1.l0" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l0.1.1.1" class="ltx_text" style="font-size:80%;">0:</span></span><span id="algx1.l0.2" class="ltx_text" style="font-size:80%;">  Matrices </span><math id="algx1.l0.m1.3" class="ltx_Math" alttext="\mathbf{Q}_{i},\mathbf{K}_{i},\mathbf{V}_{i}\in\mathbb{R}^{{\frac{N}{G}}\times d}" display="inline"><semantics id="algx1.l0.m1.3a"><mrow id="algx1.l0.m1.3.3" xref="algx1.l0.m1.3.3.cmml"><mrow id="algx1.l0.m1.3.3.3.3" xref="algx1.l0.m1.3.3.3.4.cmml"><msub id="algx1.l0.m1.1.1.1.1.1" xref="algx1.l0.m1.1.1.1.1.1.cmml"><mi mathsize="80%" id="algx1.l0.m1.1.1.1.1.1.2" xref="algx1.l0.m1.1.1.1.1.1.2.cmml">𝐐</mi><mi mathsize="80%" id="algx1.l0.m1.1.1.1.1.1.3" xref="algx1.l0.m1.1.1.1.1.1.3.cmml">i</mi></msub><mo mathsize="80%" id="algx1.l0.m1.3.3.3.3.4" xref="algx1.l0.m1.3.3.3.4.cmml">,</mo><msub id="algx1.l0.m1.2.2.2.2.2" xref="algx1.l0.m1.2.2.2.2.2.cmml"><mi mathsize="80%" id="algx1.l0.m1.2.2.2.2.2.2" xref="algx1.l0.m1.2.2.2.2.2.2.cmml">𝐊</mi><mi mathsize="80%" id="algx1.l0.m1.2.2.2.2.2.3" xref="algx1.l0.m1.2.2.2.2.2.3.cmml">i</mi></msub><mo mathsize="80%" id="algx1.l0.m1.3.3.3.3.5" xref="algx1.l0.m1.3.3.3.4.cmml">,</mo><msub id="algx1.l0.m1.3.3.3.3.3" xref="algx1.l0.m1.3.3.3.3.3.cmml"><mi mathsize="80%" id="algx1.l0.m1.3.3.3.3.3.2" xref="algx1.l0.m1.3.3.3.3.3.2.cmml">𝐕</mi><mi mathsize="80%" id="algx1.l0.m1.3.3.3.3.3.3" xref="algx1.l0.m1.3.3.3.3.3.3.cmml">i</mi></msub></mrow><mo mathsize="80%" id="algx1.l0.m1.3.3.4" xref="algx1.l0.m1.3.3.4.cmml">∈</mo><msup id="algx1.l0.m1.3.3.5" xref="algx1.l0.m1.3.3.5.cmml"><mi mathsize="80%" id="algx1.l0.m1.3.3.5.2" xref="algx1.l0.m1.3.3.5.2.cmml">ℝ</mi><mrow id="algx1.l0.m1.3.3.5.3" xref="algx1.l0.m1.3.3.5.3.cmml"><mfrac id="algx1.l0.m1.3.3.5.3.2" xref="algx1.l0.m1.3.3.5.3.2.cmml"><mi mathsize="80%" id="algx1.l0.m1.3.3.5.3.2.2" xref="algx1.l0.m1.3.3.5.3.2.2.cmml">N</mi><mi mathsize="80%" id="algx1.l0.m1.3.3.5.3.2.3" xref="algx1.l0.m1.3.3.5.3.2.3.cmml">G</mi></mfrac><mo lspace="0.222em" mathsize="80%" rspace="0.222em" id="algx1.l0.m1.3.3.5.3.1" xref="algx1.l0.m1.3.3.5.3.1.cmml">×</mo><mi mathsize="80%" id="algx1.l0.m1.3.3.5.3.3" xref="algx1.l0.m1.3.3.5.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="algx1.l0.m1.3b"><apply id="algx1.l0.m1.3.3.cmml" xref="algx1.l0.m1.3.3"><in id="algx1.l0.m1.3.3.4.cmml" xref="algx1.l0.m1.3.3.4"></in><list id="algx1.l0.m1.3.3.3.4.cmml" xref="algx1.l0.m1.3.3.3.3"><apply id="algx1.l0.m1.1.1.1.1.1.cmml" xref="algx1.l0.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="algx1.l0.m1.1.1.1.1.1.1.cmml" xref="algx1.l0.m1.1.1.1.1.1">subscript</csymbol><ci id="algx1.l0.m1.1.1.1.1.1.2.cmml" xref="algx1.l0.m1.1.1.1.1.1.2">𝐐</ci><ci id="algx1.l0.m1.1.1.1.1.1.3.cmml" xref="algx1.l0.m1.1.1.1.1.1.3">𝑖</ci></apply><apply id="algx1.l0.m1.2.2.2.2.2.cmml" xref="algx1.l0.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="algx1.l0.m1.2.2.2.2.2.1.cmml" xref="algx1.l0.m1.2.2.2.2.2">subscript</csymbol><ci id="algx1.l0.m1.2.2.2.2.2.2.cmml" xref="algx1.l0.m1.2.2.2.2.2.2">𝐊</ci><ci id="algx1.l0.m1.2.2.2.2.2.3.cmml" xref="algx1.l0.m1.2.2.2.2.2.3">𝑖</ci></apply><apply id="algx1.l0.m1.3.3.3.3.3.cmml" xref="algx1.l0.m1.3.3.3.3.3"><csymbol cd="ambiguous" id="algx1.l0.m1.3.3.3.3.3.1.cmml" xref="algx1.l0.m1.3.3.3.3.3">subscript</csymbol><ci id="algx1.l0.m1.3.3.3.3.3.2.cmml" xref="algx1.l0.m1.3.3.3.3.3.2">𝐕</ci><ci id="algx1.l0.m1.3.3.3.3.3.3.cmml" xref="algx1.l0.m1.3.3.3.3.3.3">𝑖</ci></apply></list><apply id="algx1.l0.m1.3.3.5.cmml" xref="algx1.l0.m1.3.3.5"><csymbol cd="ambiguous" id="algx1.l0.m1.3.3.5.1.cmml" xref="algx1.l0.m1.3.3.5">superscript</csymbol><ci id="algx1.l0.m1.3.3.5.2.cmml" xref="algx1.l0.m1.3.3.5.2">ℝ</ci><apply id="algx1.l0.m1.3.3.5.3.cmml" xref="algx1.l0.m1.3.3.5.3"><times id="algx1.l0.m1.3.3.5.3.1.cmml" xref="algx1.l0.m1.3.3.5.3.1"></times><apply id="algx1.l0.m1.3.3.5.3.2.cmml" xref="algx1.l0.m1.3.3.5.3.2"><divide id="algx1.l0.m1.3.3.5.3.2.1.cmml" xref="algx1.l0.m1.3.3.5.3.2"></divide><ci id="algx1.l0.m1.3.3.5.3.2.2.cmml" xref="algx1.l0.m1.3.3.5.3.2.2">𝑁</ci><ci id="algx1.l0.m1.3.3.5.3.2.3.cmml" xref="algx1.l0.m1.3.3.5.3.2.3">𝐺</ci></apply><ci id="algx1.l0.m1.3.3.5.3.3.cmml" xref="algx1.l0.m1.3.3.5.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="algx1.l0.m1.3c">\mathbf{Q}_{i},\mathbf{K}_{i},\mathbf{V}_{i}\in\mathbb{R}^{{\frac{N}{G}}\times d}</annotation></semantics></math><span id="algx1.l0.3" class="ltx_text" style="font-size:80%;"> on the </span><math id="algx1.l0.m2.1" class="ltx_Math" alttext="i" display="inline"><semantics id="algx1.l0.m2.1a"><mi mathsize="80%" id="algx1.l0.m2.1.1" xref="algx1.l0.m2.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="algx1.l0.m2.1b"><ci id="algx1.l0.m2.1.1.cmml" xref="algx1.l0.m2.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="algx1.l0.m2.1c">i</annotation></semantics></math><span id="algx1.l0.4" class="ltx_text" style="font-size:80%;">-th device
</span>
</div>
<div id="algx1.l1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l1.1.1.1" class="ltx_text" style="font-size:80%;">1:</span></span><span id="algx1.l1.2" class="ltx_text" style="font-size:80%;">  Initialize </span><math id="algx1.l1.m1.5" class="ltx_Math" alttext="\mathbf{O}_{i}=(0)_{{\frac{N}{G}}\times d}\in\mathbb{R}^{{\frac{N}{G}}\times d},l_{i}=(0)_{\frac{N}{G}}\in\mathbb{R}^{\frac{N}{G}},m_{i}=(-\infty)_{\frac{N}{G}}\in\mathbb{R}^{\frac{N}{G}}" display="inline"><semantics id="algx1.l1.m1.5a"><mrow id="algx1.l1.m1.5.5.3" xref="algx1.l1.m1.5.5.4.cmml"><mrow id="algx1.l1.m1.3.3.1.1" xref="algx1.l1.m1.3.3.1.1.cmml"><msub id="algx1.l1.m1.3.3.1.1.2" xref="algx1.l1.m1.3.3.1.1.2.cmml"><mi mathsize="80%" id="algx1.l1.m1.3.3.1.1.2.2" xref="algx1.l1.m1.3.3.1.1.2.2.cmml">𝐎</mi><mi mathsize="80%" id="algx1.l1.m1.3.3.1.1.2.3" xref="algx1.l1.m1.3.3.1.1.2.3.cmml">i</mi></msub><mo mathsize="80%" id="algx1.l1.m1.3.3.1.1.3" xref="algx1.l1.m1.3.3.1.1.3.cmml">=</mo><msub id="algx1.l1.m1.3.3.1.1.4" xref="algx1.l1.m1.3.3.1.1.4.cmml"><mrow id="algx1.l1.m1.3.3.1.1.4.2.2" xref="algx1.l1.m1.3.3.1.1.4.cmml"><mo maxsize="80%" minsize="80%" id="algx1.l1.m1.3.3.1.1.4.2.2.1" xref="algx1.l1.m1.3.3.1.1.4.cmml">(</mo><mn mathsize="80%" id="algx1.l1.m1.1.1" xref="algx1.l1.m1.1.1.cmml">0</mn><mo maxsize="80%" minsize="80%" id="algx1.l1.m1.3.3.1.1.4.2.2.2" xref="algx1.l1.m1.3.3.1.1.4.cmml">)</mo></mrow><mrow id="algx1.l1.m1.3.3.1.1.4.3" xref="algx1.l1.m1.3.3.1.1.4.3.cmml"><mfrac id="algx1.l1.m1.3.3.1.1.4.3.2" xref="algx1.l1.m1.3.3.1.1.4.3.2.cmml"><mi mathsize="80%" id="algx1.l1.m1.3.3.1.1.4.3.2.2" xref="algx1.l1.m1.3.3.1.1.4.3.2.2.cmml">N</mi><mi mathsize="80%" id="algx1.l1.m1.3.3.1.1.4.3.2.3" xref="algx1.l1.m1.3.3.1.1.4.3.2.3.cmml">G</mi></mfrac><mo lspace="0.222em" mathsize="80%" rspace="0.222em" id="algx1.l1.m1.3.3.1.1.4.3.1" xref="algx1.l1.m1.3.3.1.1.4.3.1.cmml">×</mo><mi mathsize="80%" id="algx1.l1.m1.3.3.1.1.4.3.3" xref="algx1.l1.m1.3.3.1.1.4.3.3.cmml">d</mi></mrow></msub><mo mathsize="80%" id="algx1.l1.m1.3.3.1.1.5" xref="algx1.l1.m1.3.3.1.1.5.cmml">∈</mo><msup id="algx1.l1.m1.3.3.1.1.6" xref="algx1.l1.m1.3.3.1.1.6.cmml"><mi mathsize="80%" id="algx1.l1.m1.3.3.1.1.6.2" xref="algx1.l1.m1.3.3.1.1.6.2.cmml">ℝ</mi><mrow id="algx1.l1.m1.3.3.1.1.6.3" xref="algx1.l1.m1.3.3.1.1.6.3.cmml"><mfrac id="algx1.l1.m1.3.3.1.1.6.3.2" xref="algx1.l1.m1.3.3.1.1.6.3.2.cmml"><mi mathsize="80%" id="algx1.l1.m1.3.3.1.1.6.3.2.2" xref="algx1.l1.m1.3.3.1.1.6.3.2.2.cmml">N</mi><mi mathsize="80%" id="algx1.l1.m1.3.3.1.1.6.3.2.3" xref="algx1.l1.m1.3.3.1.1.6.3.2.3.cmml">G</mi></mfrac><mo lspace="0.222em" mathsize="80%" rspace="0.222em" id="algx1.l1.m1.3.3.1.1.6.3.1" xref="algx1.l1.m1.3.3.1.1.6.3.1.cmml">×</mo><mi mathsize="80%" id="algx1.l1.m1.3.3.1.1.6.3.3" xref="algx1.l1.m1.3.3.1.1.6.3.3.cmml">d</mi></mrow></msup></mrow><mo mathsize="80%" id="algx1.l1.m1.5.5.3.4" xref="algx1.l1.m1.5.5.4a.cmml">,</mo><mrow id="algx1.l1.m1.4.4.2.2" xref="algx1.l1.m1.4.4.2.2.cmml"><msub id="algx1.l1.m1.4.4.2.2.2" xref="algx1.l1.m1.4.4.2.2.2.cmml"><mi mathsize="80%" id="algx1.l1.m1.4.4.2.2.2.2" xref="algx1.l1.m1.4.4.2.2.2.2.cmml">l</mi><mi mathsize="80%" id="algx1.l1.m1.4.4.2.2.2.3" xref="algx1.l1.m1.4.4.2.2.2.3.cmml">i</mi></msub><mo mathsize="80%" id="algx1.l1.m1.4.4.2.2.3" xref="algx1.l1.m1.4.4.2.2.3.cmml">=</mo><msub id="algx1.l1.m1.4.4.2.2.4" xref="algx1.l1.m1.4.4.2.2.4.cmml"><mrow id="algx1.l1.m1.4.4.2.2.4.2.2" xref="algx1.l1.m1.4.4.2.2.4.cmml"><mo maxsize="80%" minsize="80%" id="algx1.l1.m1.4.4.2.2.4.2.2.1" xref="algx1.l1.m1.4.4.2.2.4.cmml">(</mo><mn mathsize="80%" id="algx1.l1.m1.2.2" xref="algx1.l1.m1.2.2.cmml">0</mn><mo maxsize="80%" minsize="80%" id="algx1.l1.m1.4.4.2.2.4.2.2.2" xref="algx1.l1.m1.4.4.2.2.4.cmml">)</mo></mrow><mfrac id="algx1.l1.m1.4.4.2.2.4.3" xref="algx1.l1.m1.4.4.2.2.4.3.cmml"><mi mathsize="80%" id="algx1.l1.m1.4.4.2.2.4.3.2" xref="algx1.l1.m1.4.4.2.2.4.3.2.cmml">N</mi><mi mathsize="80%" id="algx1.l1.m1.4.4.2.2.4.3.3" xref="algx1.l1.m1.4.4.2.2.4.3.3.cmml">G</mi></mfrac></msub><mo mathsize="80%" id="algx1.l1.m1.4.4.2.2.5" xref="algx1.l1.m1.4.4.2.2.5.cmml">∈</mo><msup id="algx1.l1.m1.4.4.2.2.6" xref="algx1.l1.m1.4.4.2.2.6.cmml"><mi mathsize="80%" id="algx1.l1.m1.4.4.2.2.6.2" xref="algx1.l1.m1.4.4.2.2.6.2.cmml">ℝ</mi><mfrac id="algx1.l1.m1.4.4.2.2.6.3" xref="algx1.l1.m1.4.4.2.2.6.3.cmml"><mi mathsize="80%" id="algx1.l1.m1.4.4.2.2.6.3.2" xref="algx1.l1.m1.4.4.2.2.6.3.2.cmml">N</mi><mi mathsize="80%" id="algx1.l1.m1.4.4.2.2.6.3.3" xref="algx1.l1.m1.4.4.2.2.6.3.3.cmml">G</mi></mfrac></msup></mrow><mo mathsize="80%" id="algx1.l1.m1.5.5.3.5" xref="algx1.l1.m1.5.5.4a.cmml">,</mo><mrow id="algx1.l1.m1.5.5.3.3" xref="algx1.l1.m1.5.5.3.3.cmml"><msub id="algx1.l1.m1.5.5.3.3.3" xref="algx1.l1.m1.5.5.3.3.3.cmml"><mi mathsize="80%" id="algx1.l1.m1.5.5.3.3.3.2" xref="algx1.l1.m1.5.5.3.3.3.2.cmml">m</mi><mi mathsize="80%" id="algx1.l1.m1.5.5.3.3.3.3" xref="algx1.l1.m1.5.5.3.3.3.3.cmml">i</mi></msub><mo mathsize="80%" id="algx1.l1.m1.5.5.3.3.4" xref="algx1.l1.m1.5.5.3.3.4.cmml">=</mo><msub id="algx1.l1.m1.5.5.3.3.1" xref="algx1.l1.m1.5.5.3.3.1.cmml"><mrow id="algx1.l1.m1.5.5.3.3.1.1.1" xref="algx1.l1.m1.5.5.3.3.1.1.1.1.cmml"><mo maxsize="80%" minsize="80%" id="algx1.l1.m1.5.5.3.3.1.1.1.2" xref="algx1.l1.m1.5.5.3.3.1.1.1.1.cmml">(</mo><mrow id="algx1.l1.m1.5.5.3.3.1.1.1.1" xref="algx1.l1.m1.5.5.3.3.1.1.1.1.cmml"><mo mathsize="80%" id="algx1.l1.m1.5.5.3.3.1.1.1.1a" xref="algx1.l1.m1.5.5.3.3.1.1.1.1.cmml">−</mo><mi mathsize="80%" mathvariant="normal" id="algx1.l1.m1.5.5.3.3.1.1.1.1.2" xref="algx1.l1.m1.5.5.3.3.1.1.1.1.2.cmml">∞</mi></mrow><mo maxsize="80%" minsize="80%" id="algx1.l1.m1.5.5.3.3.1.1.1.3" xref="algx1.l1.m1.5.5.3.3.1.1.1.1.cmml">)</mo></mrow><mfrac id="algx1.l1.m1.5.5.3.3.1.3" xref="algx1.l1.m1.5.5.3.3.1.3.cmml"><mi mathsize="80%" id="algx1.l1.m1.5.5.3.3.1.3.2" xref="algx1.l1.m1.5.5.3.3.1.3.2.cmml">N</mi><mi mathsize="80%" id="algx1.l1.m1.5.5.3.3.1.3.3" xref="algx1.l1.m1.5.5.3.3.1.3.3.cmml">G</mi></mfrac></msub><mo mathsize="80%" id="algx1.l1.m1.5.5.3.3.5" xref="algx1.l1.m1.5.5.3.3.5.cmml">∈</mo><msup id="algx1.l1.m1.5.5.3.3.6" xref="algx1.l1.m1.5.5.3.3.6.cmml"><mi mathsize="80%" id="algx1.l1.m1.5.5.3.3.6.2" xref="algx1.l1.m1.5.5.3.3.6.2.cmml">ℝ</mi><mfrac id="algx1.l1.m1.5.5.3.3.6.3" xref="algx1.l1.m1.5.5.3.3.6.3.cmml"><mi mathsize="80%" id="algx1.l1.m1.5.5.3.3.6.3.2" xref="algx1.l1.m1.5.5.3.3.6.3.2.cmml">N</mi><mi mathsize="80%" id="algx1.l1.m1.5.5.3.3.6.3.3" xref="algx1.l1.m1.5.5.3.3.6.3.3.cmml">G</mi></mfrac></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="algx1.l1.m1.5b"><apply id="algx1.l1.m1.5.5.4.cmml" xref="algx1.l1.m1.5.5.3"><csymbol cd="ambiguous" id="algx1.l1.m1.5.5.4a.cmml" xref="algx1.l1.m1.5.5.3.4">formulae-sequence</csymbol><apply id="algx1.l1.m1.3.3.1.1.cmml" xref="algx1.l1.m1.3.3.1.1"><and id="algx1.l1.m1.3.3.1.1a.cmml" xref="algx1.l1.m1.3.3.1.1"></and><apply id="algx1.l1.m1.3.3.1.1b.cmml" xref="algx1.l1.m1.3.3.1.1"><eq id="algx1.l1.m1.3.3.1.1.3.cmml" xref="algx1.l1.m1.3.3.1.1.3"></eq><apply id="algx1.l1.m1.3.3.1.1.2.cmml" xref="algx1.l1.m1.3.3.1.1.2"><csymbol cd="ambiguous" id="algx1.l1.m1.3.3.1.1.2.1.cmml" xref="algx1.l1.m1.3.3.1.1.2">subscript</csymbol><ci id="algx1.l1.m1.3.3.1.1.2.2.cmml" xref="algx1.l1.m1.3.3.1.1.2.2">𝐎</ci><ci id="algx1.l1.m1.3.3.1.1.2.3.cmml" xref="algx1.l1.m1.3.3.1.1.2.3">𝑖</ci></apply><apply id="algx1.l1.m1.3.3.1.1.4.cmml" xref="algx1.l1.m1.3.3.1.1.4"><csymbol cd="ambiguous" id="algx1.l1.m1.3.3.1.1.4.1.cmml" xref="algx1.l1.m1.3.3.1.1.4">subscript</csymbol><cn type="integer" id="algx1.l1.m1.1.1.cmml" xref="algx1.l1.m1.1.1">0</cn><apply id="algx1.l1.m1.3.3.1.1.4.3.cmml" xref="algx1.l1.m1.3.3.1.1.4.3"><times id="algx1.l1.m1.3.3.1.1.4.3.1.cmml" xref="algx1.l1.m1.3.3.1.1.4.3.1"></times><apply id="algx1.l1.m1.3.3.1.1.4.3.2.cmml" xref="algx1.l1.m1.3.3.1.1.4.3.2"><divide id="algx1.l1.m1.3.3.1.1.4.3.2.1.cmml" xref="algx1.l1.m1.3.3.1.1.4.3.2"></divide><ci id="algx1.l1.m1.3.3.1.1.4.3.2.2.cmml" xref="algx1.l1.m1.3.3.1.1.4.3.2.2">𝑁</ci><ci id="algx1.l1.m1.3.3.1.1.4.3.2.3.cmml" xref="algx1.l1.m1.3.3.1.1.4.3.2.3">𝐺</ci></apply><ci id="algx1.l1.m1.3.3.1.1.4.3.3.cmml" xref="algx1.l1.m1.3.3.1.1.4.3.3">𝑑</ci></apply></apply></apply><apply id="algx1.l1.m1.3.3.1.1c.cmml" xref="algx1.l1.m1.3.3.1.1"><in id="algx1.l1.m1.3.3.1.1.5.cmml" xref="algx1.l1.m1.3.3.1.1.5"></in><share href="#algx1.l1.m1.3.3.1.1.4.cmml" id="algx1.l1.m1.3.3.1.1d.cmml" xref="algx1.l1.m1.3.3.1.1"></share><apply id="algx1.l1.m1.3.3.1.1.6.cmml" xref="algx1.l1.m1.3.3.1.1.6"><csymbol cd="ambiguous" id="algx1.l1.m1.3.3.1.1.6.1.cmml" xref="algx1.l1.m1.3.3.1.1.6">superscript</csymbol><ci id="algx1.l1.m1.3.3.1.1.6.2.cmml" xref="algx1.l1.m1.3.3.1.1.6.2">ℝ</ci><apply id="algx1.l1.m1.3.3.1.1.6.3.cmml" xref="algx1.l1.m1.3.3.1.1.6.3"><times id="algx1.l1.m1.3.3.1.1.6.3.1.cmml" xref="algx1.l1.m1.3.3.1.1.6.3.1"></times><apply id="algx1.l1.m1.3.3.1.1.6.3.2.cmml" xref="algx1.l1.m1.3.3.1.1.6.3.2"><divide id="algx1.l1.m1.3.3.1.1.6.3.2.1.cmml" xref="algx1.l1.m1.3.3.1.1.6.3.2"></divide><ci id="algx1.l1.m1.3.3.1.1.6.3.2.2.cmml" xref="algx1.l1.m1.3.3.1.1.6.3.2.2">𝑁</ci><ci id="algx1.l1.m1.3.3.1.1.6.3.2.3.cmml" xref="algx1.l1.m1.3.3.1.1.6.3.2.3">𝐺</ci></apply><ci id="algx1.l1.m1.3.3.1.1.6.3.3.cmml" xref="algx1.l1.m1.3.3.1.1.6.3.3">𝑑</ci></apply></apply></apply></apply><apply id="algx1.l1.m1.4.4.2.2.cmml" xref="algx1.l1.m1.4.4.2.2"><and id="algx1.l1.m1.4.4.2.2a.cmml" xref="algx1.l1.m1.4.4.2.2"></and><apply id="algx1.l1.m1.4.4.2.2b.cmml" xref="algx1.l1.m1.4.4.2.2"><eq id="algx1.l1.m1.4.4.2.2.3.cmml" xref="algx1.l1.m1.4.4.2.2.3"></eq><apply id="algx1.l1.m1.4.4.2.2.2.cmml" xref="algx1.l1.m1.4.4.2.2.2"><csymbol cd="ambiguous" id="algx1.l1.m1.4.4.2.2.2.1.cmml" xref="algx1.l1.m1.4.4.2.2.2">subscript</csymbol><ci id="algx1.l1.m1.4.4.2.2.2.2.cmml" xref="algx1.l1.m1.4.4.2.2.2.2">𝑙</ci><ci id="algx1.l1.m1.4.4.2.2.2.3.cmml" xref="algx1.l1.m1.4.4.2.2.2.3">𝑖</ci></apply><apply id="algx1.l1.m1.4.4.2.2.4.cmml" xref="algx1.l1.m1.4.4.2.2.4"><csymbol cd="ambiguous" id="algx1.l1.m1.4.4.2.2.4.1.cmml" xref="algx1.l1.m1.4.4.2.2.4">subscript</csymbol><cn type="integer" id="algx1.l1.m1.2.2.cmml" xref="algx1.l1.m1.2.2">0</cn><apply id="algx1.l1.m1.4.4.2.2.4.3.cmml" xref="algx1.l1.m1.4.4.2.2.4.3"><divide id="algx1.l1.m1.4.4.2.2.4.3.1.cmml" xref="algx1.l1.m1.4.4.2.2.4.3"></divide><ci id="algx1.l1.m1.4.4.2.2.4.3.2.cmml" xref="algx1.l1.m1.4.4.2.2.4.3.2">𝑁</ci><ci id="algx1.l1.m1.4.4.2.2.4.3.3.cmml" xref="algx1.l1.m1.4.4.2.2.4.3.3">𝐺</ci></apply></apply></apply><apply id="algx1.l1.m1.4.4.2.2c.cmml" xref="algx1.l1.m1.4.4.2.2"><in id="algx1.l1.m1.4.4.2.2.5.cmml" xref="algx1.l1.m1.4.4.2.2.5"></in><share href="#algx1.l1.m1.4.4.2.2.4.cmml" id="algx1.l1.m1.4.4.2.2d.cmml" xref="algx1.l1.m1.4.4.2.2"></share><apply id="algx1.l1.m1.4.4.2.2.6.cmml" xref="algx1.l1.m1.4.4.2.2.6"><csymbol cd="ambiguous" id="algx1.l1.m1.4.4.2.2.6.1.cmml" xref="algx1.l1.m1.4.4.2.2.6">superscript</csymbol><ci id="algx1.l1.m1.4.4.2.2.6.2.cmml" xref="algx1.l1.m1.4.4.2.2.6.2">ℝ</ci><apply id="algx1.l1.m1.4.4.2.2.6.3.cmml" xref="algx1.l1.m1.4.4.2.2.6.3"><divide id="algx1.l1.m1.4.4.2.2.6.3.1.cmml" xref="algx1.l1.m1.4.4.2.2.6.3"></divide><ci id="algx1.l1.m1.4.4.2.2.6.3.2.cmml" xref="algx1.l1.m1.4.4.2.2.6.3.2">𝑁</ci><ci id="algx1.l1.m1.4.4.2.2.6.3.3.cmml" xref="algx1.l1.m1.4.4.2.2.6.3.3">𝐺</ci></apply></apply></apply></apply><apply id="algx1.l1.m1.5.5.3.3.cmml" xref="algx1.l1.m1.5.5.3.3"><and id="algx1.l1.m1.5.5.3.3a.cmml" xref="algx1.l1.m1.5.5.3.3"></and><apply id="algx1.l1.m1.5.5.3.3b.cmml" xref="algx1.l1.m1.5.5.3.3"><eq id="algx1.l1.m1.5.5.3.3.4.cmml" xref="algx1.l1.m1.5.5.3.3.4"></eq><apply id="algx1.l1.m1.5.5.3.3.3.cmml" xref="algx1.l1.m1.5.5.3.3.3"><csymbol cd="ambiguous" id="algx1.l1.m1.5.5.3.3.3.1.cmml" xref="algx1.l1.m1.5.5.3.3.3">subscript</csymbol><ci id="algx1.l1.m1.5.5.3.3.3.2.cmml" xref="algx1.l1.m1.5.5.3.3.3.2">𝑚</ci><ci id="algx1.l1.m1.5.5.3.3.3.3.cmml" xref="algx1.l1.m1.5.5.3.3.3.3">𝑖</ci></apply><apply id="algx1.l1.m1.5.5.3.3.1.cmml" xref="algx1.l1.m1.5.5.3.3.1"><csymbol cd="ambiguous" id="algx1.l1.m1.5.5.3.3.1.2.cmml" xref="algx1.l1.m1.5.5.3.3.1">subscript</csymbol><apply id="algx1.l1.m1.5.5.3.3.1.1.1.1.cmml" xref="algx1.l1.m1.5.5.3.3.1.1.1"><minus id="algx1.l1.m1.5.5.3.3.1.1.1.1.1.cmml" xref="algx1.l1.m1.5.5.3.3.1.1.1"></minus><infinity id="algx1.l1.m1.5.5.3.3.1.1.1.1.2.cmml" xref="algx1.l1.m1.5.5.3.3.1.1.1.1.2"></infinity></apply><apply id="algx1.l1.m1.5.5.3.3.1.3.cmml" xref="algx1.l1.m1.5.5.3.3.1.3"><divide id="algx1.l1.m1.5.5.3.3.1.3.1.cmml" xref="algx1.l1.m1.5.5.3.3.1.3"></divide><ci id="algx1.l1.m1.5.5.3.3.1.3.2.cmml" xref="algx1.l1.m1.5.5.3.3.1.3.2">𝑁</ci><ci id="algx1.l1.m1.5.5.3.3.1.3.3.cmml" xref="algx1.l1.m1.5.5.3.3.1.3.3">𝐺</ci></apply></apply></apply><apply id="algx1.l1.m1.5.5.3.3c.cmml" xref="algx1.l1.m1.5.5.3.3"><in id="algx1.l1.m1.5.5.3.3.5.cmml" xref="algx1.l1.m1.5.5.3.3.5"></in><share href="#algx1.l1.m1.5.5.3.3.1.cmml" id="algx1.l1.m1.5.5.3.3d.cmml" xref="algx1.l1.m1.5.5.3.3"></share><apply id="algx1.l1.m1.5.5.3.3.6.cmml" xref="algx1.l1.m1.5.5.3.3.6"><csymbol cd="ambiguous" id="algx1.l1.m1.5.5.3.3.6.1.cmml" xref="algx1.l1.m1.5.5.3.3.6">superscript</csymbol><ci id="algx1.l1.m1.5.5.3.3.6.2.cmml" xref="algx1.l1.m1.5.5.3.3.6.2">ℝ</ci><apply id="algx1.l1.m1.5.5.3.3.6.3.cmml" xref="algx1.l1.m1.5.5.3.3.6.3"><divide id="algx1.l1.m1.5.5.3.3.6.3.1.cmml" xref="algx1.l1.m1.5.5.3.3.6.3"></divide><ci id="algx1.l1.m1.5.5.3.3.6.3.2.cmml" xref="algx1.l1.m1.5.5.3.3.6.3.2">𝑁</ci><ci id="algx1.l1.m1.5.5.3.3.6.3.3.cmml" xref="algx1.l1.m1.5.5.3.3.6.3.3">𝐺</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="algx1.l1.m1.5c">\mathbf{O}_{i}=(0)_{{\frac{N}{G}}\times d}\in\mathbb{R}^{{\frac{N}{G}}\times d},l_{i}=(0)_{\frac{N}{G}}\in\mathbb{R}^{\frac{N}{G}},m_{i}=(-\infty)_{\frac{N}{G}}\in\mathbb{R}^{\frac{N}{G}}</annotation></semantics></math><span id="algx1.l1.3" class="ltx_text" style="font-size:80%;">
</span>
</div>
<div id="algx1.l2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l2.1.1.1" class="ltx_text" style="font-size:80%;">2:</span></span><span id="algx1.l2.2" class="ltx_text" style="font-size:80%;">  Initialize Buffer </span><math id="algx1.l2.m1.1" class="ltx_Math" alttext="K_{buf}" display="inline"><semantics id="algx1.l2.m1.1a"><msub id="algx1.l2.m1.1.1" xref="algx1.l2.m1.1.1.cmml"><mi mathsize="80%" id="algx1.l2.m1.1.1.2" xref="algx1.l2.m1.1.1.2.cmml">K</mi><mrow id="algx1.l2.m1.1.1.3" xref="algx1.l2.m1.1.1.3.cmml"><mi mathsize="80%" id="algx1.l2.m1.1.1.3.2" xref="algx1.l2.m1.1.1.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="algx1.l2.m1.1.1.3.1" xref="algx1.l2.m1.1.1.3.1.cmml">​</mo><mi mathsize="80%" id="algx1.l2.m1.1.1.3.3" xref="algx1.l2.m1.1.1.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="algx1.l2.m1.1.1.3.1a" xref="algx1.l2.m1.1.1.3.1.cmml">​</mo><mi mathsize="80%" id="algx1.l2.m1.1.1.3.4" xref="algx1.l2.m1.1.1.3.4.cmml">f</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="algx1.l2.m1.1b"><apply id="algx1.l2.m1.1.1.cmml" xref="algx1.l2.m1.1.1"><csymbol cd="ambiguous" id="algx1.l2.m1.1.1.1.cmml" xref="algx1.l2.m1.1.1">subscript</csymbol><ci id="algx1.l2.m1.1.1.2.cmml" xref="algx1.l2.m1.1.1.2">𝐾</ci><apply id="algx1.l2.m1.1.1.3.cmml" xref="algx1.l2.m1.1.1.3"><times id="algx1.l2.m1.1.1.3.1.cmml" xref="algx1.l2.m1.1.1.3.1"></times><ci id="algx1.l2.m1.1.1.3.2.cmml" xref="algx1.l2.m1.1.1.3.2">𝑏</ci><ci id="algx1.l2.m1.1.1.3.3.cmml" xref="algx1.l2.m1.1.1.3.3">𝑢</ci><ci id="algx1.l2.m1.1.1.3.4.cmml" xref="algx1.l2.m1.1.1.3.4">𝑓</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="algx1.l2.m1.1c">K_{buf}</annotation></semantics></math><span id="algx1.l2.3" class="ltx_text" style="font-size:80%;"> with </span><math id="algx1.l2.m2.1" class="ltx_Math" alttext="\mathbf{K}_{i}," display="inline"><semantics id="algx1.l2.m2.1a"><mrow id="algx1.l2.m2.1.1.1" xref="algx1.l2.m2.1.1.1.1.cmml"><msub id="algx1.l2.m2.1.1.1.1" xref="algx1.l2.m2.1.1.1.1.cmml"><mi mathsize="80%" id="algx1.l2.m2.1.1.1.1.2" xref="algx1.l2.m2.1.1.1.1.2.cmml">𝐊</mi><mi mathsize="80%" id="algx1.l2.m2.1.1.1.1.3" xref="algx1.l2.m2.1.1.1.1.3.cmml">i</mi></msub><mo mathsize="80%" id="algx1.l2.m2.1.1.1.2" xref="algx1.l2.m2.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="algx1.l2.m2.1b"><apply id="algx1.l2.m2.1.1.1.1.cmml" xref="algx1.l2.m2.1.1.1"><csymbol cd="ambiguous" id="algx1.l2.m2.1.1.1.1.1.cmml" xref="algx1.l2.m2.1.1.1">subscript</csymbol><ci id="algx1.l2.m2.1.1.1.1.2.cmml" xref="algx1.l2.m2.1.1.1.1.2">𝐊</ci><ci id="algx1.l2.m2.1.1.1.1.3.cmml" xref="algx1.l2.m2.1.1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="algx1.l2.m2.1c">\mathbf{K}_{i},</annotation></semantics></math><span id="algx1.l2.4" class="ltx_text" style="font-size:80%;">Buffer </span><math id="algx1.l2.m3.1" class="ltx_Math" alttext="V_{buf}" display="inline"><semantics id="algx1.l2.m3.1a"><msub id="algx1.l2.m3.1.1" xref="algx1.l2.m3.1.1.cmml"><mi mathsize="80%" id="algx1.l2.m3.1.1.2" xref="algx1.l2.m3.1.1.2.cmml">V</mi><mrow id="algx1.l2.m3.1.1.3" xref="algx1.l2.m3.1.1.3.cmml"><mi mathsize="80%" id="algx1.l2.m3.1.1.3.2" xref="algx1.l2.m3.1.1.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="algx1.l2.m3.1.1.3.1" xref="algx1.l2.m3.1.1.3.1.cmml">​</mo><mi mathsize="80%" id="algx1.l2.m3.1.1.3.3" xref="algx1.l2.m3.1.1.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="algx1.l2.m3.1.1.3.1a" xref="algx1.l2.m3.1.1.3.1.cmml">​</mo><mi mathsize="80%" id="algx1.l2.m3.1.1.3.4" xref="algx1.l2.m3.1.1.3.4.cmml">f</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="algx1.l2.m3.1b"><apply id="algx1.l2.m3.1.1.cmml" xref="algx1.l2.m3.1.1"><csymbol cd="ambiguous" id="algx1.l2.m3.1.1.1.cmml" xref="algx1.l2.m3.1.1">subscript</csymbol><ci id="algx1.l2.m3.1.1.2.cmml" xref="algx1.l2.m3.1.1.2">𝑉</ci><apply id="algx1.l2.m3.1.1.3.cmml" xref="algx1.l2.m3.1.1.3"><times id="algx1.l2.m3.1.1.3.1.cmml" xref="algx1.l2.m3.1.1.3.1"></times><ci id="algx1.l2.m3.1.1.3.2.cmml" xref="algx1.l2.m3.1.1.3.2">𝑏</ci><ci id="algx1.l2.m3.1.1.3.3.cmml" xref="algx1.l2.m3.1.1.3.3">𝑢</ci><ci id="algx1.l2.m3.1.1.3.4.cmml" xref="algx1.l2.m3.1.1.3.4">𝑓</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="algx1.l2.m3.1c">V_{buf}</annotation></semantics></math><span id="algx1.l2.5" class="ltx_text" style="font-size:80%;"> with </span><math id="algx1.l2.m4.1" class="ltx_Math" alttext="\mathbf{V}_{i}" display="inline"><semantics id="algx1.l2.m4.1a"><msub id="algx1.l2.m4.1.1" xref="algx1.l2.m4.1.1.cmml"><mi mathsize="80%" id="algx1.l2.m4.1.1.2" xref="algx1.l2.m4.1.1.2.cmml">𝐕</mi><mi mathsize="80%" id="algx1.l2.m4.1.1.3" xref="algx1.l2.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="algx1.l2.m4.1b"><apply id="algx1.l2.m4.1.1.cmml" xref="algx1.l2.m4.1.1"><csymbol cd="ambiguous" id="algx1.l2.m4.1.1.1.cmml" xref="algx1.l2.m4.1.1">subscript</csymbol><ci id="algx1.l2.m4.1.1.2.cmml" xref="algx1.l2.m4.1.1.2">𝐕</ci><ci id="algx1.l2.m4.1.1.3.cmml" xref="algx1.l2.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="algx1.l2.m4.1c">\mathbf{V}_{i}</annotation></semantics></math><span id="algx1.l2.6" class="ltx_text" style="font-size:80%;">.
</span>
</div>
<div id="algx1.l3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l3.1.1.1" class="ltx_text" style="font-size:80%;">3:</span></span><span id="algx1.l3.2" class="ltx_text" style="font-size:80%;">  </span><span id="algx1.l3.3" class="ltx_text ltx_font_bold" style="font-size:80%;">for</span><span id="algx1.l3.4" class="ltx_text" style="font-size:80%;"> </span><math id="algx1.l3.m1.1" class="ltx_Math" alttext="j=1" display="inline"><semantics id="algx1.l3.m1.1a"><mrow id="algx1.l3.m1.1.1" xref="algx1.l3.m1.1.1.cmml"><mi mathsize="80%" id="algx1.l3.m1.1.1.2" xref="algx1.l3.m1.1.1.2.cmml">j</mi><mo mathsize="80%" id="algx1.l3.m1.1.1.1" xref="algx1.l3.m1.1.1.1.cmml">=</mo><mn mathsize="80%" id="algx1.l3.m1.1.1.3" xref="algx1.l3.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="algx1.l3.m1.1b"><apply id="algx1.l3.m1.1.1.cmml" xref="algx1.l3.m1.1.1"><eq id="algx1.l3.m1.1.1.1.cmml" xref="algx1.l3.m1.1.1.1"></eq><ci id="algx1.l3.m1.1.1.2.cmml" xref="algx1.l3.m1.1.1.2">𝑗</ci><cn type="integer" id="algx1.l3.m1.1.1.3.cmml" xref="algx1.l3.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="algx1.l3.m1.1c">j=1</annotation></semantics></math><span id="algx1.l3.5" class="ltx_text" style="font-size:80%;"> </span><span id="algx1.l3.6" class="ltx_text ltx_font_bold" style="font-size:80%;">to</span><span id="algx1.l3.7" class="ltx_text" style="font-size:80%;"> </span><math id="algx1.l3.m2.1" class="ltx_Math" alttext="G" display="inline"><semantics id="algx1.l3.m2.1a"><mi mathsize="80%" id="algx1.l3.m2.1.1" xref="algx1.l3.m2.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="algx1.l3.m2.1b"><ci id="algx1.l3.m2.1.1.cmml" xref="algx1.l3.m2.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="algx1.l3.m2.1c">G</annotation></semantics></math><span id="algx1.l3.8" class="ltx_text" style="font-size:80%;"> </span><span id="algx1.l3.9" class="ltx_text ltx_font_bold" style="font-size:80%;">do</span><span id="algx1.l3.10" class="ltx_text" style="font-size:80%;">


</span>
</div>
<div id="algx1.l4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l4.1.1.1" class="ltx_text" style="font-size:80%;">4:</span></span><span id="algx1.l4.2" class="ltx_text" style="font-size:80%;">    </span><span id="algx1.l4.3" class="ltx_text ltx_font_bold" style="font-size:80%;">if</span><span id="algx1.l4.4" class="ltx_text" style="font-size:80%;"> j!=1 </span><span id="algx1.l4.5" class="ltx_text ltx_font_bold" style="font-size:80%;">then</span><span id="algx1.l4.6" class="ltx_text" style="font-size:80%;">


</span>
</div>
<div id="algx1.l5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l5.1.1.1" class="ltx_text" style="font-size:80%;">5:</span></span><span id="algx1.l5.2" class="ltx_text" style="font-size:80%;">      Get </span><math id="algx1.l5.m1.2" class="ltx_Math" alttext="K_{j},V_{j}" display="inline"><semantics id="algx1.l5.m1.2a"><mrow id="algx1.l5.m1.2.2.2" xref="algx1.l5.m1.2.2.3.cmml"><msub id="algx1.l5.m1.1.1.1.1" xref="algx1.l5.m1.1.1.1.1.cmml"><mi mathsize="80%" id="algx1.l5.m1.1.1.1.1.2" xref="algx1.l5.m1.1.1.1.1.2.cmml">K</mi><mi mathsize="80%" id="algx1.l5.m1.1.1.1.1.3" xref="algx1.l5.m1.1.1.1.1.3.cmml">j</mi></msub><mo mathsize="80%" id="algx1.l5.m1.2.2.2.3" xref="algx1.l5.m1.2.2.3.cmml">,</mo><msub id="algx1.l5.m1.2.2.2.2" xref="algx1.l5.m1.2.2.2.2.cmml"><mi mathsize="80%" id="algx1.l5.m1.2.2.2.2.2" xref="algx1.l5.m1.2.2.2.2.2.cmml">V</mi><mi mathsize="80%" id="algx1.l5.m1.2.2.2.2.3" xref="algx1.l5.m1.2.2.2.2.3.cmml">j</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="algx1.l5.m1.2b"><list id="algx1.l5.m1.2.2.3.cmml" xref="algx1.l5.m1.2.2.2"><apply id="algx1.l5.m1.1.1.1.1.cmml" xref="algx1.l5.m1.1.1.1.1"><csymbol cd="ambiguous" id="algx1.l5.m1.1.1.1.1.1.cmml" xref="algx1.l5.m1.1.1.1.1">subscript</csymbol><ci id="algx1.l5.m1.1.1.1.1.2.cmml" xref="algx1.l5.m1.1.1.1.1.2">𝐾</ci><ci id="algx1.l5.m1.1.1.1.1.3.cmml" xref="algx1.l5.m1.1.1.1.1.3">𝑗</ci></apply><apply id="algx1.l5.m1.2.2.2.2.cmml" xref="algx1.l5.m1.2.2.2.2"><csymbol cd="ambiguous" id="algx1.l5.m1.2.2.2.2.1.cmml" xref="algx1.l5.m1.2.2.2.2">subscript</csymbol><ci id="algx1.l5.m1.2.2.2.2.2.cmml" xref="algx1.l5.m1.2.2.2.2.2">𝑉</ci><ci id="algx1.l5.m1.2.2.2.2.3.cmml" xref="algx1.l5.m1.2.2.2.2.3">𝑗</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="algx1.l5.m1.2c">K_{j},V_{j}</annotation></semantics></math><span id="algx1.l5.3" class="ltx_text" style="font-size:80%;"> from </span><math id="algx1.l5.m2.2" class="ltx_Math" alttext="K_{buf},V_{buf}" display="inline"><semantics id="algx1.l5.m2.2a"><mrow id="algx1.l5.m2.2.2.2" xref="algx1.l5.m2.2.2.3.cmml"><msub id="algx1.l5.m2.1.1.1.1" xref="algx1.l5.m2.1.1.1.1.cmml"><mi mathsize="80%" id="algx1.l5.m2.1.1.1.1.2" xref="algx1.l5.m2.1.1.1.1.2.cmml">K</mi><mrow id="algx1.l5.m2.1.1.1.1.3" xref="algx1.l5.m2.1.1.1.1.3.cmml"><mi mathsize="80%" id="algx1.l5.m2.1.1.1.1.3.2" xref="algx1.l5.m2.1.1.1.1.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="algx1.l5.m2.1.1.1.1.3.1" xref="algx1.l5.m2.1.1.1.1.3.1.cmml">​</mo><mi mathsize="80%" id="algx1.l5.m2.1.1.1.1.3.3" xref="algx1.l5.m2.1.1.1.1.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="algx1.l5.m2.1.1.1.1.3.1a" xref="algx1.l5.m2.1.1.1.1.3.1.cmml">​</mo><mi mathsize="80%" id="algx1.l5.m2.1.1.1.1.3.4" xref="algx1.l5.m2.1.1.1.1.3.4.cmml">f</mi></mrow></msub><mo mathsize="80%" id="algx1.l5.m2.2.2.2.3" xref="algx1.l5.m2.2.2.3.cmml">,</mo><msub id="algx1.l5.m2.2.2.2.2" xref="algx1.l5.m2.2.2.2.2.cmml"><mi mathsize="80%" id="algx1.l5.m2.2.2.2.2.2" xref="algx1.l5.m2.2.2.2.2.2.cmml">V</mi><mrow id="algx1.l5.m2.2.2.2.2.3" xref="algx1.l5.m2.2.2.2.2.3.cmml"><mi mathsize="80%" id="algx1.l5.m2.2.2.2.2.3.2" xref="algx1.l5.m2.2.2.2.2.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="algx1.l5.m2.2.2.2.2.3.1" xref="algx1.l5.m2.2.2.2.2.3.1.cmml">​</mo><mi mathsize="80%" id="algx1.l5.m2.2.2.2.2.3.3" xref="algx1.l5.m2.2.2.2.2.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="algx1.l5.m2.2.2.2.2.3.1a" xref="algx1.l5.m2.2.2.2.2.3.1.cmml">​</mo><mi mathsize="80%" id="algx1.l5.m2.2.2.2.2.3.4" xref="algx1.l5.m2.2.2.2.2.3.4.cmml">f</mi></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="algx1.l5.m2.2b"><list id="algx1.l5.m2.2.2.3.cmml" xref="algx1.l5.m2.2.2.2"><apply id="algx1.l5.m2.1.1.1.1.cmml" xref="algx1.l5.m2.1.1.1.1"><csymbol cd="ambiguous" id="algx1.l5.m2.1.1.1.1.1.cmml" xref="algx1.l5.m2.1.1.1.1">subscript</csymbol><ci id="algx1.l5.m2.1.1.1.1.2.cmml" xref="algx1.l5.m2.1.1.1.1.2">𝐾</ci><apply id="algx1.l5.m2.1.1.1.1.3.cmml" xref="algx1.l5.m2.1.1.1.1.3"><times id="algx1.l5.m2.1.1.1.1.3.1.cmml" xref="algx1.l5.m2.1.1.1.1.3.1"></times><ci id="algx1.l5.m2.1.1.1.1.3.2.cmml" xref="algx1.l5.m2.1.1.1.1.3.2">𝑏</ci><ci id="algx1.l5.m2.1.1.1.1.3.3.cmml" xref="algx1.l5.m2.1.1.1.1.3.3">𝑢</ci><ci id="algx1.l5.m2.1.1.1.1.3.4.cmml" xref="algx1.l5.m2.1.1.1.1.3.4">𝑓</ci></apply></apply><apply id="algx1.l5.m2.2.2.2.2.cmml" xref="algx1.l5.m2.2.2.2.2"><csymbol cd="ambiguous" id="algx1.l5.m2.2.2.2.2.1.cmml" xref="algx1.l5.m2.2.2.2.2">subscript</csymbol><ci id="algx1.l5.m2.2.2.2.2.2.cmml" xref="algx1.l5.m2.2.2.2.2.2">𝑉</ci><apply id="algx1.l5.m2.2.2.2.2.3.cmml" xref="algx1.l5.m2.2.2.2.2.3"><times id="algx1.l5.m2.2.2.2.2.3.1.cmml" xref="algx1.l5.m2.2.2.2.2.3.1"></times><ci id="algx1.l5.m2.2.2.2.2.3.2.cmml" xref="algx1.l5.m2.2.2.2.2.3.2">𝑏</ci><ci id="algx1.l5.m2.2.2.2.2.3.3.cmml" xref="algx1.l5.m2.2.2.2.2.3.3">𝑢</ci><ci id="algx1.l5.m2.2.2.2.2.3.4.cmml" xref="algx1.l5.m2.2.2.2.2.3.4">𝑓</ci></apply></apply></list></annotation-xml><annotation encoding="application/x-tex" id="algx1.l5.m2.2c">K_{buf},V_{buf}</annotation></semantics></math><span id="algx1.l5.4" class="ltx_text" style="font-size:80%;">; {Wait communication thread’s job finished}
</span>
</div>
<div id="algx1.l6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l6.1.1.1" class="ltx_text" style="font-size:80%;">6:</span></span><span id="algx1.l6.2" class="ltx_text" style="font-size:80%;">    </span><span id="algx1.l6.3" class="ltx_text ltx_font_bold" style="font-size:80%;">end</span><span id="algx1.l6.4" class="ltx_text" style="font-size:80%;"> </span><span id="algx1.l6.5" class="ltx_text ltx_font_bold" style="font-size:80%;">if</span>
</div>
<div id="algx1.l7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l7.1.1.1" class="ltx_text" style="font-size:80%;">7:</span></span><span id="algx1.l7.2" class="ltx_text" style="font-size:80%;">    </span><span id="algx1.l7.3" class="ltx_text ltx_font_bold" style="font-size:80%;">AsyncCommunicationCall</span><span id="algx1.l7.4" class="ltx_text" style="font-size:80%;">:
</span>
</div>
<div id="algx1.l8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l8.1.1.1" class="ltx_text" style="font-size:80%;">8:</span></span><span id="algx1.l8.2" class="ltx_text" style="font-size:80%;">     Initiate asynchronous communication thread
</span>
</div>
<div id="algx1.l9" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l9.1.1.1" class="ltx_text" style="font-size:80%;">9:</span></span><span id="algx1.l9.2" class="ltx_text" style="font-size:80%;">     Let </span><math id="algx1.l9.m1.2" class="ltx_Math" alttext="\text{Buf}=(K_{buf},V_{buf})" display="inline"><semantics id="algx1.l9.m1.2a"><mrow id="algx1.l9.m1.2.2" xref="algx1.l9.m1.2.2.cmml"><mtext mathsize="80%" id="algx1.l9.m1.2.2.4" xref="algx1.l9.m1.2.2.4a.cmml">Buf</mtext><mo mathsize="80%" id="algx1.l9.m1.2.2.3" xref="algx1.l9.m1.2.2.3.cmml">=</mo><mrow id="algx1.l9.m1.2.2.2.2" xref="algx1.l9.m1.2.2.2.3.cmml"><mo maxsize="80%" minsize="80%" id="algx1.l9.m1.2.2.2.2.3" xref="algx1.l9.m1.2.2.2.3.cmml">(</mo><msub id="algx1.l9.m1.1.1.1.1.1" xref="algx1.l9.m1.1.1.1.1.1.cmml"><mi mathsize="80%" id="algx1.l9.m1.1.1.1.1.1.2" xref="algx1.l9.m1.1.1.1.1.1.2.cmml">K</mi><mrow id="algx1.l9.m1.1.1.1.1.1.3" xref="algx1.l9.m1.1.1.1.1.1.3.cmml"><mi mathsize="80%" id="algx1.l9.m1.1.1.1.1.1.3.2" xref="algx1.l9.m1.1.1.1.1.1.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="algx1.l9.m1.1.1.1.1.1.3.1" xref="algx1.l9.m1.1.1.1.1.1.3.1.cmml">​</mo><mi mathsize="80%" id="algx1.l9.m1.1.1.1.1.1.3.3" xref="algx1.l9.m1.1.1.1.1.1.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="algx1.l9.m1.1.1.1.1.1.3.1a" xref="algx1.l9.m1.1.1.1.1.1.3.1.cmml">​</mo><mi mathsize="80%" id="algx1.l9.m1.1.1.1.1.1.3.4" xref="algx1.l9.m1.1.1.1.1.1.3.4.cmml">f</mi></mrow></msub><mo mathsize="80%" id="algx1.l9.m1.2.2.2.2.4" xref="algx1.l9.m1.2.2.2.3.cmml">,</mo><msub id="algx1.l9.m1.2.2.2.2.2" xref="algx1.l9.m1.2.2.2.2.2.cmml"><mi mathsize="80%" id="algx1.l9.m1.2.2.2.2.2.2" xref="algx1.l9.m1.2.2.2.2.2.2.cmml">V</mi><mrow id="algx1.l9.m1.2.2.2.2.2.3" xref="algx1.l9.m1.2.2.2.2.2.3.cmml"><mi mathsize="80%" id="algx1.l9.m1.2.2.2.2.2.3.2" xref="algx1.l9.m1.2.2.2.2.2.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="algx1.l9.m1.2.2.2.2.2.3.1" xref="algx1.l9.m1.2.2.2.2.2.3.1.cmml">​</mo><mi mathsize="80%" id="algx1.l9.m1.2.2.2.2.2.3.3" xref="algx1.l9.m1.2.2.2.2.2.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="algx1.l9.m1.2.2.2.2.2.3.1a" xref="algx1.l9.m1.2.2.2.2.2.3.1.cmml">​</mo><mi mathsize="80%" id="algx1.l9.m1.2.2.2.2.2.3.4" xref="algx1.l9.m1.2.2.2.2.2.3.4.cmml">f</mi></mrow></msub><mo maxsize="80%" minsize="80%" id="algx1.l9.m1.2.2.2.2.5" xref="algx1.l9.m1.2.2.2.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="algx1.l9.m1.2b"><apply id="algx1.l9.m1.2.2.cmml" xref="algx1.l9.m1.2.2"><eq id="algx1.l9.m1.2.2.3.cmml" xref="algx1.l9.m1.2.2.3"></eq><ci id="algx1.l9.m1.2.2.4a.cmml" xref="algx1.l9.m1.2.2.4"><mtext mathsize="80%" id="algx1.l9.m1.2.2.4.cmml" xref="algx1.l9.m1.2.2.4">Buf</mtext></ci><interval closure="open" id="algx1.l9.m1.2.2.2.3.cmml" xref="algx1.l9.m1.2.2.2.2"><apply id="algx1.l9.m1.1.1.1.1.1.cmml" xref="algx1.l9.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="algx1.l9.m1.1.1.1.1.1.1.cmml" xref="algx1.l9.m1.1.1.1.1.1">subscript</csymbol><ci id="algx1.l9.m1.1.1.1.1.1.2.cmml" xref="algx1.l9.m1.1.1.1.1.1.2">𝐾</ci><apply id="algx1.l9.m1.1.1.1.1.1.3.cmml" xref="algx1.l9.m1.1.1.1.1.1.3"><times id="algx1.l9.m1.1.1.1.1.1.3.1.cmml" xref="algx1.l9.m1.1.1.1.1.1.3.1"></times><ci id="algx1.l9.m1.1.1.1.1.1.3.2.cmml" xref="algx1.l9.m1.1.1.1.1.1.3.2">𝑏</ci><ci id="algx1.l9.m1.1.1.1.1.1.3.3.cmml" xref="algx1.l9.m1.1.1.1.1.1.3.3">𝑢</ci><ci id="algx1.l9.m1.1.1.1.1.1.3.4.cmml" xref="algx1.l9.m1.1.1.1.1.1.3.4">𝑓</ci></apply></apply><apply id="algx1.l9.m1.2.2.2.2.2.cmml" xref="algx1.l9.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="algx1.l9.m1.2.2.2.2.2.1.cmml" xref="algx1.l9.m1.2.2.2.2.2">subscript</csymbol><ci id="algx1.l9.m1.2.2.2.2.2.2.cmml" xref="algx1.l9.m1.2.2.2.2.2.2">𝑉</ci><apply id="algx1.l9.m1.2.2.2.2.2.3.cmml" xref="algx1.l9.m1.2.2.2.2.2.3"><times id="algx1.l9.m1.2.2.2.2.2.3.1.cmml" xref="algx1.l9.m1.2.2.2.2.2.3.1"></times><ci id="algx1.l9.m1.2.2.2.2.2.3.2.cmml" xref="algx1.l9.m1.2.2.2.2.2.3.2">𝑏</ci><ci id="algx1.l9.m1.2.2.2.2.2.3.3.cmml" xref="algx1.l9.m1.2.2.2.2.2.3.3">𝑢</ci><ci id="algx1.l9.m1.2.2.2.2.2.3.4.cmml" xref="algx1.l9.m1.2.2.2.2.2.3.4">𝑓</ci></apply></apply></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="algx1.l9.m1.2c">\text{Buf}=(K_{buf},V_{buf})</annotation></semantics></math><span id="algx1.l9.3" class="ltx_text" style="font-size:80%;">
</span>
</div>
<div id="algx1.l10" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l10.1.1.1" class="ltx_text" style="font-size:80%;">10:</span></span><span id="algx1.l10.2" class="ltx_text" style="font-size:80%;">     Asynchronously Send the </span><span id="algx1.l10.3" class="ltx_text ltx_markedasmath" style="font-size:80%;">Buf</span><span id="algx1.l10.4" class="ltx_text" style="font-size:80%;"> to next device and recvive </span><span id="algx1.l10.5" class="ltx_text ltx_markedasmath" style="font-size:80%;">Buf</span><span id="algx1.l10.6" class="ltx_text" style="font-size:80%;"> from previous device
</span>
</div>
<div id="algx1.l11" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l11.1.1.1" class="ltx_text" style="font-size:80%;">11:</span></span><span id="algx1.l11.2" class="ltx_text" style="font-size:80%;">    </span><math id="algx1.l11.m1.2" class="ltx_Math" alttext="\mathbf{S}_{i,j}=\mathbf{Q}_{i}\mathbf{K}_{j}^{T}" display="inline"><semantics id="algx1.l11.m1.2a"><mrow id="algx1.l11.m1.2.3" xref="algx1.l11.m1.2.3.cmml"><msub id="algx1.l11.m1.2.3.2" xref="algx1.l11.m1.2.3.2.cmml"><mi mathsize="80%" id="algx1.l11.m1.2.3.2.2" xref="algx1.l11.m1.2.3.2.2.cmml">𝐒</mi><mrow id="algx1.l11.m1.2.2.2.4" xref="algx1.l11.m1.2.2.2.3.cmml"><mi mathsize="80%" id="algx1.l11.m1.1.1.1.1" xref="algx1.l11.m1.1.1.1.1.cmml">i</mi><mo mathsize="80%" id="algx1.l11.m1.2.2.2.4.1" xref="algx1.l11.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="algx1.l11.m1.2.2.2.2" xref="algx1.l11.m1.2.2.2.2.cmml">j</mi></mrow></msub><mo mathsize="80%" id="algx1.l11.m1.2.3.1" xref="algx1.l11.m1.2.3.1.cmml">=</mo><mrow id="algx1.l11.m1.2.3.3" xref="algx1.l11.m1.2.3.3.cmml"><msub id="algx1.l11.m1.2.3.3.2" xref="algx1.l11.m1.2.3.3.2.cmml"><mi mathsize="80%" id="algx1.l11.m1.2.3.3.2.2" xref="algx1.l11.m1.2.3.3.2.2.cmml">𝐐</mi><mi mathsize="80%" id="algx1.l11.m1.2.3.3.2.3" xref="algx1.l11.m1.2.3.3.2.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="algx1.l11.m1.2.3.3.1" xref="algx1.l11.m1.2.3.3.1.cmml">​</mo><msubsup id="algx1.l11.m1.2.3.3.3" xref="algx1.l11.m1.2.3.3.3.cmml"><mi mathsize="80%" id="algx1.l11.m1.2.3.3.3.2.2" xref="algx1.l11.m1.2.3.3.3.2.2.cmml">𝐊</mi><mi mathsize="80%" id="algx1.l11.m1.2.3.3.3.2.3" xref="algx1.l11.m1.2.3.3.3.2.3.cmml">j</mi><mi mathsize="80%" id="algx1.l11.m1.2.3.3.3.3" xref="algx1.l11.m1.2.3.3.3.3.cmml">T</mi></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="algx1.l11.m1.2b"><apply id="algx1.l11.m1.2.3.cmml" xref="algx1.l11.m1.2.3"><eq id="algx1.l11.m1.2.3.1.cmml" xref="algx1.l11.m1.2.3.1"></eq><apply id="algx1.l11.m1.2.3.2.cmml" xref="algx1.l11.m1.2.3.2"><csymbol cd="ambiguous" id="algx1.l11.m1.2.3.2.1.cmml" xref="algx1.l11.m1.2.3.2">subscript</csymbol><ci id="algx1.l11.m1.2.3.2.2.cmml" xref="algx1.l11.m1.2.3.2.2">𝐒</ci><list id="algx1.l11.m1.2.2.2.3.cmml" xref="algx1.l11.m1.2.2.2.4"><ci id="algx1.l11.m1.1.1.1.1.cmml" xref="algx1.l11.m1.1.1.1.1">𝑖</ci><ci id="algx1.l11.m1.2.2.2.2.cmml" xref="algx1.l11.m1.2.2.2.2">𝑗</ci></list></apply><apply id="algx1.l11.m1.2.3.3.cmml" xref="algx1.l11.m1.2.3.3"><times id="algx1.l11.m1.2.3.3.1.cmml" xref="algx1.l11.m1.2.3.3.1"></times><apply id="algx1.l11.m1.2.3.3.2.cmml" xref="algx1.l11.m1.2.3.3.2"><csymbol cd="ambiguous" id="algx1.l11.m1.2.3.3.2.1.cmml" xref="algx1.l11.m1.2.3.3.2">subscript</csymbol><ci id="algx1.l11.m1.2.3.3.2.2.cmml" xref="algx1.l11.m1.2.3.3.2.2">𝐐</ci><ci id="algx1.l11.m1.2.3.3.2.3.cmml" xref="algx1.l11.m1.2.3.3.2.3">𝑖</ci></apply><apply id="algx1.l11.m1.2.3.3.3.cmml" xref="algx1.l11.m1.2.3.3.3"><csymbol cd="ambiguous" id="algx1.l11.m1.2.3.3.3.1.cmml" xref="algx1.l11.m1.2.3.3.3">superscript</csymbol><apply id="algx1.l11.m1.2.3.3.3.2.cmml" xref="algx1.l11.m1.2.3.3.3"><csymbol cd="ambiguous" id="algx1.l11.m1.2.3.3.3.2.1.cmml" xref="algx1.l11.m1.2.3.3.3">subscript</csymbol><ci id="algx1.l11.m1.2.3.3.3.2.2.cmml" xref="algx1.l11.m1.2.3.3.3.2.2">𝐊</ci><ci id="algx1.l11.m1.2.3.3.3.2.3.cmml" xref="algx1.l11.m1.2.3.3.3.2.3">𝑗</ci></apply><ci id="algx1.l11.m1.2.3.3.3.3.cmml" xref="algx1.l11.m1.2.3.3.3.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="algx1.l11.m1.2c">\mathbf{S}_{i,j}=\mathbf{Q}_{i}\mathbf{K}_{j}^{T}</annotation></semantics></math><span id="algx1.l11.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="algx1.l12" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l12.1.1.1" class="ltx_text" style="font-size:80%;">12:</span></span><span id="algx1.l12.2" class="ltx_text" style="font-size:80%;">    </span><math id="algx1.l12.m1.5" class="ltx_Math" alttext="m_{i,j}=\text{rowmax}(\mathbf{S}_{i,j})" display="inline"><semantics id="algx1.l12.m1.5a"><mrow id="algx1.l12.m1.5.5" xref="algx1.l12.m1.5.5.cmml"><msub id="algx1.l12.m1.5.5.3" xref="algx1.l12.m1.5.5.3.cmml"><mi mathsize="80%" id="algx1.l12.m1.5.5.3.2" xref="algx1.l12.m1.5.5.3.2.cmml">m</mi><mrow id="algx1.l12.m1.2.2.2.4" xref="algx1.l12.m1.2.2.2.3.cmml"><mi mathsize="80%" id="algx1.l12.m1.1.1.1.1" xref="algx1.l12.m1.1.1.1.1.cmml">i</mi><mo mathsize="80%" id="algx1.l12.m1.2.2.2.4.1" xref="algx1.l12.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="algx1.l12.m1.2.2.2.2" xref="algx1.l12.m1.2.2.2.2.cmml">j</mi></mrow></msub><mo mathsize="80%" id="algx1.l12.m1.5.5.2" xref="algx1.l12.m1.5.5.2.cmml">=</mo><mrow id="algx1.l12.m1.5.5.1" xref="algx1.l12.m1.5.5.1.cmml"><mtext mathsize="80%" id="algx1.l12.m1.5.5.1.3" xref="algx1.l12.m1.5.5.1.3a.cmml">rowmax</mtext><mo lspace="0em" rspace="0em" id="algx1.l12.m1.5.5.1.2" xref="algx1.l12.m1.5.5.1.2.cmml">​</mo><mrow id="algx1.l12.m1.5.5.1.1.1" xref="algx1.l12.m1.5.5.1.1.1.1.cmml"><mo maxsize="80%" minsize="80%" id="algx1.l12.m1.5.5.1.1.1.2" xref="algx1.l12.m1.5.5.1.1.1.1.cmml">(</mo><msub id="algx1.l12.m1.5.5.1.1.1.1" xref="algx1.l12.m1.5.5.1.1.1.1.cmml"><mi mathsize="80%" id="algx1.l12.m1.5.5.1.1.1.1.2" xref="algx1.l12.m1.5.5.1.1.1.1.2.cmml">𝐒</mi><mrow id="algx1.l12.m1.4.4.2.4" xref="algx1.l12.m1.4.4.2.3.cmml"><mi mathsize="80%" id="algx1.l12.m1.3.3.1.1" xref="algx1.l12.m1.3.3.1.1.cmml">i</mi><mo mathsize="80%" id="algx1.l12.m1.4.4.2.4.1" xref="algx1.l12.m1.4.4.2.3.cmml">,</mo><mi mathsize="80%" id="algx1.l12.m1.4.4.2.2" xref="algx1.l12.m1.4.4.2.2.cmml">j</mi></mrow></msub><mo maxsize="80%" minsize="80%" id="algx1.l12.m1.5.5.1.1.1.3" xref="algx1.l12.m1.5.5.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="algx1.l12.m1.5b"><apply id="algx1.l12.m1.5.5.cmml" xref="algx1.l12.m1.5.5"><eq id="algx1.l12.m1.5.5.2.cmml" xref="algx1.l12.m1.5.5.2"></eq><apply id="algx1.l12.m1.5.5.3.cmml" xref="algx1.l12.m1.5.5.3"><csymbol cd="ambiguous" id="algx1.l12.m1.5.5.3.1.cmml" xref="algx1.l12.m1.5.5.3">subscript</csymbol><ci id="algx1.l12.m1.5.5.3.2.cmml" xref="algx1.l12.m1.5.5.3.2">𝑚</ci><list id="algx1.l12.m1.2.2.2.3.cmml" xref="algx1.l12.m1.2.2.2.4"><ci id="algx1.l12.m1.1.1.1.1.cmml" xref="algx1.l12.m1.1.1.1.1">𝑖</ci><ci id="algx1.l12.m1.2.2.2.2.cmml" xref="algx1.l12.m1.2.2.2.2">𝑗</ci></list></apply><apply id="algx1.l12.m1.5.5.1.cmml" xref="algx1.l12.m1.5.5.1"><times id="algx1.l12.m1.5.5.1.2.cmml" xref="algx1.l12.m1.5.5.1.2"></times><ci id="algx1.l12.m1.5.5.1.3a.cmml" xref="algx1.l12.m1.5.5.1.3"><mtext mathsize="80%" id="algx1.l12.m1.5.5.1.3.cmml" xref="algx1.l12.m1.5.5.1.3">rowmax</mtext></ci><apply id="algx1.l12.m1.5.5.1.1.1.1.cmml" xref="algx1.l12.m1.5.5.1.1.1"><csymbol cd="ambiguous" id="algx1.l12.m1.5.5.1.1.1.1.1.cmml" xref="algx1.l12.m1.5.5.1.1.1">subscript</csymbol><ci id="algx1.l12.m1.5.5.1.1.1.1.2.cmml" xref="algx1.l12.m1.5.5.1.1.1.1.2">𝐒</ci><list id="algx1.l12.m1.4.4.2.3.cmml" xref="algx1.l12.m1.4.4.2.4"><ci id="algx1.l12.m1.3.3.1.1.cmml" xref="algx1.l12.m1.3.3.1.1">𝑖</ci><ci id="algx1.l12.m1.4.4.2.2.cmml" xref="algx1.l12.m1.4.4.2.2">𝑗</ci></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="algx1.l12.m1.5c">m_{i,j}=\text{rowmax}(\mathbf{S}_{i,j})</annotation></semantics></math><span id="algx1.l12.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="algx1.l13" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l13.1.1.1" class="ltx_text" style="font-size:80%;">13:</span></span><span id="algx1.l13.2" class="ltx_text" style="font-size:80%;">    </span><math id="algx1.l13.m1.7" class="ltx_Math" alttext="\mathbf{P}_{i,j}=\text{exp}(\mathbf{S}_{i,j}-m_{i,j})" display="inline"><semantics id="algx1.l13.m1.7a"><mrow id="algx1.l13.m1.7.7" xref="algx1.l13.m1.7.7.cmml"><msub id="algx1.l13.m1.7.7.3" xref="algx1.l13.m1.7.7.3.cmml"><mi mathsize="80%" id="algx1.l13.m1.7.7.3.2" xref="algx1.l13.m1.7.7.3.2.cmml">𝐏</mi><mrow id="algx1.l13.m1.2.2.2.4" xref="algx1.l13.m1.2.2.2.3.cmml"><mi mathsize="80%" id="algx1.l13.m1.1.1.1.1" xref="algx1.l13.m1.1.1.1.1.cmml">i</mi><mo mathsize="80%" id="algx1.l13.m1.2.2.2.4.1" xref="algx1.l13.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="algx1.l13.m1.2.2.2.2" xref="algx1.l13.m1.2.2.2.2.cmml">j</mi></mrow></msub><mo mathsize="80%" id="algx1.l13.m1.7.7.2" xref="algx1.l13.m1.7.7.2.cmml">=</mo><mrow id="algx1.l13.m1.7.7.1" xref="algx1.l13.m1.7.7.1.cmml"><mtext mathsize="80%" id="algx1.l13.m1.7.7.1.3" xref="algx1.l13.m1.7.7.1.3a.cmml">exp</mtext><mo lspace="0em" rspace="0em" id="algx1.l13.m1.7.7.1.2" xref="algx1.l13.m1.7.7.1.2.cmml">​</mo><mrow id="algx1.l13.m1.7.7.1.1.1" xref="algx1.l13.m1.7.7.1.1.1.1.cmml"><mo maxsize="80%" minsize="80%" id="algx1.l13.m1.7.7.1.1.1.2" xref="algx1.l13.m1.7.7.1.1.1.1.cmml">(</mo><mrow id="algx1.l13.m1.7.7.1.1.1.1" xref="algx1.l13.m1.7.7.1.1.1.1.cmml"><msub id="algx1.l13.m1.7.7.1.1.1.1.2" xref="algx1.l13.m1.7.7.1.1.1.1.2.cmml"><mi mathsize="80%" id="algx1.l13.m1.7.7.1.1.1.1.2.2" xref="algx1.l13.m1.7.7.1.1.1.1.2.2.cmml">𝐒</mi><mrow id="algx1.l13.m1.4.4.2.4" xref="algx1.l13.m1.4.4.2.3.cmml"><mi mathsize="80%" id="algx1.l13.m1.3.3.1.1" xref="algx1.l13.m1.3.3.1.1.cmml">i</mi><mo mathsize="80%" id="algx1.l13.m1.4.4.2.4.1" xref="algx1.l13.m1.4.4.2.3.cmml">,</mo><mi mathsize="80%" id="algx1.l13.m1.4.4.2.2" xref="algx1.l13.m1.4.4.2.2.cmml">j</mi></mrow></msub><mo mathsize="80%" id="algx1.l13.m1.7.7.1.1.1.1.1" xref="algx1.l13.m1.7.7.1.1.1.1.1.cmml">−</mo><msub id="algx1.l13.m1.7.7.1.1.1.1.3" xref="algx1.l13.m1.7.7.1.1.1.1.3.cmml"><mi mathsize="80%" id="algx1.l13.m1.7.7.1.1.1.1.3.2" xref="algx1.l13.m1.7.7.1.1.1.1.3.2.cmml">m</mi><mrow id="algx1.l13.m1.6.6.2.4" xref="algx1.l13.m1.6.6.2.3.cmml"><mi mathsize="80%" id="algx1.l13.m1.5.5.1.1" xref="algx1.l13.m1.5.5.1.1.cmml">i</mi><mo mathsize="80%" id="algx1.l13.m1.6.6.2.4.1" xref="algx1.l13.m1.6.6.2.3.cmml">,</mo><mi mathsize="80%" id="algx1.l13.m1.6.6.2.2" xref="algx1.l13.m1.6.6.2.2.cmml">j</mi></mrow></msub></mrow><mo maxsize="80%" minsize="80%" id="algx1.l13.m1.7.7.1.1.1.3" xref="algx1.l13.m1.7.7.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="algx1.l13.m1.7b"><apply id="algx1.l13.m1.7.7.cmml" xref="algx1.l13.m1.7.7"><eq id="algx1.l13.m1.7.7.2.cmml" xref="algx1.l13.m1.7.7.2"></eq><apply id="algx1.l13.m1.7.7.3.cmml" xref="algx1.l13.m1.7.7.3"><csymbol cd="ambiguous" id="algx1.l13.m1.7.7.3.1.cmml" xref="algx1.l13.m1.7.7.3">subscript</csymbol><ci id="algx1.l13.m1.7.7.3.2.cmml" xref="algx1.l13.m1.7.7.3.2">𝐏</ci><list id="algx1.l13.m1.2.2.2.3.cmml" xref="algx1.l13.m1.2.2.2.4"><ci id="algx1.l13.m1.1.1.1.1.cmml" xref="algx1.l13.m1.1.1.1.1">𝑖</ci><ci id="algx1.l13.m1.2.2.2.2.cmml" xref="algx1.l13.m1.2.2.2.2">𝑗</ci></list></apply><apply id="algx1.l13.m1.7.7.1.cmml" xref="algx1.l13.m1.7.7.1"><times id="algx1.l13.m1.7.7.1.2.cmml" xref="algx1.l13.m1.7.7.1.2"></times><ci id="algx1.l13.m1.7.7.1.3a.cmml" xref="algx1.l13.m1.7.7.1.3"><mtext mathsize="80%" id="algx1.l13.m1.7.7.1.3.cmml" xref="algx1.l13.m1.7.7.1.3">exp</mtext></ci><apply id="algx1.l13.m1.7.7.1.1.1.1.cmml" xref="algx1.l13.m1.7.7.1.1.1"><minus id="algx1.l13.m1.7.7.1.1.1.1.1.cmml" xref="algx1.l13.m1.7.7.1.1.1.1.1"></minus><apply id="algx1.l13.m1.7.7.1.1.1.1.2.cmml" xref="algx1.l13.m1.7.7.1.1.1.1.2"><csymbol cd="ambiguous" id="algx1.l13.m1.7.7.1.1.1.1.2.1.cmml" xref="algx1.l13.m1.7.7.1.1.1.1.2">subscript</csymbol><ci id="algx1.l13.m1.7.7.1.1.1.1.2.2.cmml" xref="algx1.l13.m1.7.7.1.1.1.1.2.2">𝐒</ci><list id="algx1.l13.m1.4.4.2.3.cmml" xref="algx1.l13.m1.4.4.2.4"><ci id="algx1.l13.m1.3.3.1.1.cmml" xref="algx1.l13.m1.3.3.1.1">𝑖</ci><ci id="algx1.l13.m1.4.4.2.2.cmml" xref="algx1.l13.m1.4.4.2.2">𝑗</ci></list></apply><apply id="algx1.l13.m1.7.7.1.1.1.1.3.cmml" xref="algx1.l13.m1.7.7.1.1.1.1.3"><csymbol cd="ambiguous" id="algx1.l13.m1.7.7.1.1.1.1.3.1.cmml" xref="algx1.l13.m1.7.7.1.1.1.1.3">subscript</csymbol><ci id="algx1.l13.m1.7.7.1.1.1.1.3.2.cmml" xref="algx1.l13.m1.7.7.1.1.1.1.3.2">𝑚</ci><list id="algx1.l13.m1.6.6.2.3.cmml" xref="algx1.l13.m1.6.6.2.4"><ci id="algx1.l13.m1.5.5.1.1.cmml" xref="algx1.l13.m1.5.5.1.1">𝑖</ci><ci id="algx1.l13.m1.6.6.2.2.cmml" xref="algx1.l13.m1.6.6.2.2">𝑗</ci></list></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="algx1.l13.m1.7c">\mathbf{P}_{i,j}=\text{exp}(\mathbf{S}_{i,j}-m_{i,j})</annotation></semantics></math><span id="algx1.l13.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="algx1.l14" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l14.1.1.1" class="ltx_text" style="font-size:80%;">14:</span></span><span id="algx1.l14.2" class="ltx_text" style="font-size:80%;">    </span><math id="algx1.l14.m1.5" class="ltx_Math" alttext="l_{i,j}=\text{rowsum}(\mathbf{P}_{i,j})" display="inline"><semantics id="algx1.l14.m1.5a"><mrow id="algx1.l14.m1.5.5" xref="algx1.l14.m1.5.5.cmml"><msub id="algx1.l14.m1.5.5.3" xref="algx1.l14.m1.5.5.3.cmml"><mi mathsize="80%" id="algx1.l14.m1.5.5.3.2" xref="algx1.l14.m1.5.5.3.2.cmml">l</mi><mrow id="algx1.l14.m1.2.2.2.4" xref="algx1.l14.m1.2.2.2.3.cmml"><mi mathsize="80%" id="algx1.l14.m1.1.1.1.1" xref="algx1.l14.m1.1.1.1.1.cmml">i</mi><mo mathsize="80%" id="algx1.l14.m1.2.2.2.4.1" xref="algx1.l14.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="algx1.l14.m1.2.2.2.2" xref="algx1.l14.m1.2.2.2.2.cmml">j</mi></mrow></msub><mo mathsize="80%" id="algx1.l14.m1.5.5.2" xref="algx1.l14.m1.5.5.2.cmml">=</mo><mrow id="algx1.l14.m1.5.5.1" xref="algx1.l14.m1.5.5.1.cmml"><mtext mathsize="80%" id="algx1.l14.m1.5.5.1.3" xref="algx1.l14.m1.5.5.1.3a.cmml">rowsum</mtext><mo lspace="0em" rspace="0em" id="algx1.l14.m1.5.5.1.2" xref="algx1.l14.m1.5.5.1.2.cmml">​</mo><mrow id="algx1.l14.m1.5.5.1.1.1" xref="algx1.l14.m1.5.5.1.1.1.1.cmml"><mo maxsize="80%" minsize="80%" id="algx1.l14.m1.5.5.1.1.1.2" xref="algx1.l14.m1.5.5.1.1.1.1.cmml">(</mo><msub id="algx1.l14.m1.5.5.1.1.1.1" xref="algx1.l14.m1.5.5.1.1.1.1.cmml"><mi mathsize="80%" id="algx1.l14.m1.5.5.1.1.1.1.2" xref="algx1.l14.m1.5.5.1.1.1.1.2.cmml">𝐏</mi><mrow id="algx1.l14.m1.4.4.2.4" xref="algx1.l14.m1.4.4.2.3.cmml"><mi mathsize="80%" id="algx1.l14.m1.3.3.1.1" xref="algx1.l14.m1.3.3.1.1.cmml">i</mi><mo mathsize="80%" id="algx1.l14.m1.4.4.2.4.1" xref="algx1.l14.m1.4.4.2.3.cmml">,</mo><mi mathsize="80%" id="algx1.l14.m1.4.4.2.2" xref="algx1.l14.m1.4.4.2.2.cmml">j</mi></mrow></msub><mo maxsize="80%" minsize="80%" id="algx1.l14.m1.5.5.1.1.1.3" xref="algx1.l14.m1.5.5.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="algx1.l14.m1.5b"><apply id="algx1.l14.m1.5.5.cmml" xref="algx1.l14.m1.5.5"><eq id="algx1.l14.m1.5.5.2.cmml" xref="algx1.l14.m1.5.5.2"></eq><apply id="algx1.l14.m1.5.5.3.cmml" xref="algx1.l14.m1.5.5.3"><csymbol cd="ambiguous" id="algx1.l14.m1.5.5.3.1.cmml" xref="algx1.l14.m1.5.5.3">subscript</csymbol><ci id="algx1.l14.m1.5.5.3.2.cmml" xref="algx1.l14.m1.5.5.3.2">𝑙</ci><list id="algx1.l14.m1.2.2.2.3.cmml" xref="algx1.l14.m1.2.2.2.4"><ci id="algx1.l14.m1.1.1.1.1.cmml" xref="algx1.l14.m1.1.1.1.1">𝑖</ci><ci id="algx1.l14.m1.2.2.2.2.cmml" xref="algx1.l14.m1.2.2.2.2">𝑗</ci></list></apply><apply id="algx1.l14.m1.5.5.1.cmml" xref="algx1.l14.m1.5.5.1"><times id="algx1.l14.m1.5.5.1.2.cmml" xref="algx1.l14.m1.5.5.1.2"></times><ci id="algx1.l14.m1.5.5.1.3a.cmml" xref="algx1.l14.m1.5.5.1.3"><mtext mathsize="80%" id="algx1.l14.m1.5.5.1.3.cmml" xref="algx1.l14.m1.5.5.1.3">rowsum</mtext></ci><apply id="algx1.l14.m1.5.5.1.1.1.1.cmml" xref="algx1.l14.m1.5.5.1.1.1"><csymbol cd="ambiguous" id="algx1.l14.m1.5.5.1.1.1.1.1.cmml" xref="algx1.l14.m1.5.5.1.1.1">subscript</csymbol><ci id="algx1.l14.m1.5.5.1.1.1.1.2.cmml" xref="algx1.l14.m1.5.5.1.1.1.1.2">𝐏</ci><list id="algx1.l14.m1.4.4.2.3.cmml" xref="algx1.l14.m1.4.4.2.4"><ci id="algx1.l14.m1.3.3.1.1.cmml" xref="algx1.l14.m1.3.3.1.1">𝑖</ci><ci id="algx1.l14.m1.4.4.2.2.cmml" xref="algx1.l14.m1.4.4.2.2">𝑗</ci></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="algx1.l14.m1.5c">l_{i,j}=\text{rowsum}(\mathbf{P}_{i,j})</annotation></semantics></math><span id="algx1.l14.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="algx1.l15" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l15.1.1.1" class="ltx_text" style="font-size:80%;">15:</span></span><span id="algx1.l15.2" class="ltx_text" style="font-size:80%;">    </span><math id="algx1.l15.m1.4" class="ltx_Math" alttext="\mathbf{O}_{i,j}=\mathbf{P}_{i,j}\mathbf{V}_{j}" display="inline"><semantics id="algx1.l15.m1.4a"><mrow id="algx1.l15.m1.4.5" xref="algx1.l15.m1.4.5.cmml"><msub id="algx1.l15.m1.4.5.2" xref="algx1.l15.m1.4.5.2.cmml"><mi mathsize="80%" id="algx1.l15.m1.4.5.2.2" xref="algx1.l15.m1.4.5.2.2.cmml">𝐎</mi><mrow id="algx1.l15.m1.2.2.2.4" xref="algx1.l15.m1.2.2.2.3.cmml"><mi mathsize="80%" id="algx1.l15.m1.1.1.1.1" xref="algx1.l15.m1.1.1.1.1.cmml">i</mi><mo mathsize="80%" id="algx1.l15.m1.2.2.2.4.1" xref="algx1.l15.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="algx1.l15.m1.2.2.2.2" xref="algx1.l15.m1.2.2.2.2.cmml">j</mi></mrow></msub><mo mathsize="80%" id="algx1.l15.m1.4.5.1" xref="algx1.l15.m1.4.5.1.cmml">=</mo><mrow id="algx1.l15.m1.4.5.3" xref="algx1.l15.m1.4.5.3.cmml"><msub id="algx1.l15.m1.4.5.3.2" xref="algx1.l15.m1.4.5.3.2.cmml"><mi mathsize="80%" id="algx1.l15.m1.4.5.3.2.2" xref="algx1.l15.m1.4.5.3.2.2.cmml">𝐏</mi><mrow id="algx1.l15.m1.4.4.2.4" xref="algx1.l15.m1.4.4.2.3.cmml"><mi mathsize="80%" id="algx1.l15.m1.3.3.1.1" xref="algx1.l15.m1.3.3.1.1.cmml">i</mi><mo mathsize="80%" id="algx1.l15.m1.4.4.2.4.1" xref="algx1.l15.m1.4.4.2.3.cmml">,</mo><mi mathsize="80%" id="algx1.l15.m1.4.4.2.2" xref="algx1.l15.m1.4.4.2.2.cmml">j</mi></mrow></msub><mo lspace="0em" rspace="0em" id="algx1.l15.m1.4.5.3.1" xref="algx1.l15.m1.4.5.3.1.cmml">​</mo><msub id="algx1.l15.m1.4.5.3.3" xref="algx1.l15.m1.4.5.3.3.cmml"><mi mathsize="80%" id="algx1.l15.m1.4.5.3.3.2" xref="algx1.l15.m1.4.5.3.3.2.cmml">𝐕</mi><mi mathsize="80%" id="algx1.l15.m1.4.5.3.3.3" xref="algx1.l15.m1.4.5.3.3.3.cmml">j</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="algx1.l15.m1.4b"><apply id="algx1.l15.m1.4.5.cmml" xref="algx1.l15.m1.4.5"><eq id="algx1.l15.m1.4.5.1.cmml" xref="algx1.l15.m1.4.5.1"></eq><apply id="algx1.l15.m1.4.5.2.cmml" xref="algx1.l15.m1.4.5.2"><csymbol cd="ambiguous" id="algx1.l15.m1.4.5.2.1.cmml" xref="algx1.l15.m1.4.5.2">subscript</csymbol><ci id="algx1.l15.m1.4.5.2.2.cmml" xref="algx1.l15.m1.4.5.2.2">𝐎</ci><list id="algx1.l15.m1.2.2.2.3.cmml" xref="algx1.l15.m1.2.2.2.4"><ci id="algx1.l15.m1.1.1.1.1.cmml" xref="algx1.l15.m1.1.1.1.1">𝑖</ci><ci id="algx1.l15.m1.2.2.2.2.cmml" xref="algx1.l15.m1.2.2.2.2">𝑗</ci></list></apply><apply id="algx1.l15.m1.4.5.3.cmml" xref="algx1.l15.m1.4.5.3"><times id="algx1.l15.m1.4.5.3.1.cmml" xref="algx1.l15.m1.4.5.3.1"></times><apply id="algx1.l15.m1.4.5.3.2.cmml" xref="algx1.l15.m1.4.5.3.2"><csymbol cd="ambiguous" id="algx1.l15.m1.4.5.3.2.1.cmml" xref="algx1.l15.m1.4.5.3.2">subscript</csymbol><ci id="algx1.l15.m1.4.5.3.2.2.cmml" xref="algx1.l15.m1.4.5.3.2.2">𝐏</ci><list id="algx1.l15.m1.4.4.2.3.cmml" xref="algx1.l15.m1.4.4.2.4"><ci id="algx1.l15.m1.3.3.1.1.cmml" xref="algx1.l15.m1.3.3.1.1">𝑖</ci><ci id="algx1.l15.m1.4.4.2.2.cmml" xref="algx1.l15.m1.4.4.2.2">𝑗</ci></list></apply><apply id="algx1.l15.m1.4.5.3.3.cmml" xref="algx1.l15.m1.4.5.3.3"><csymbol cd="ambiguous" id="algx1.l15.m1.4.5.3.3.1.cmml" xref="algx1.l15.m1.4.5.3.3">subscript</csymbol><ci id="algx1.l15.m1.4.5.3.3.2.cmml" xref="algx1.l15.m1.4.5.3.3.2">𝐕</ci><ci id="algx1.l15.m1.4.5.3.3.3.cmml" xref="algx1.l15.m1.4.5.3.3.3">𝑗</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="algx1.l15.m1.4c">\mathbf{O}_{i,j}=\mathbf{P}_{i,j}\mathbf{V}_{j}</annotation></semantics></math><span id="algx1.l15.3" class="ltx_text" style="font-size:80%;">; {The end of the forward pass of local attention operations.}
</span>
</div>
<div id="algx1.l16" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l16.1.1.1" class="ltx_text" style="font-size:80%;">16:</span></span><span id="algx1.l16.2" class="ltx_text" style="font-size:80%;">    </span><math id="algx1.l16.m1.5" class="ltx_Math" alttext="m_{\text{new}}\leftarrow\max{\{m_{i},m_{i,j}\}}" display="inline"><semantics id="algx1.l16.m1.5a"><mrow id="algx1.l16.m1.5.5" xref="algx1.l16.m1.5.5.cmml"><msub id="algx1.l16.m1.5.5.4" xref="algx1.l16.m1.5.5.4.cmml"><mi mathsize="80%" id="algx1.l16.m1.5.5.4.2" xref="algx1.l16.m1.5.5.4.2.cmml">m</mi><mtext mathsize="80%" id="algx1.l16.m1.5.5.4.3" xref="algx1.l16.m1.5.5.4.3a.cmml">new</mtext></msub><mo mathsize="80%" stretchy="false" id="algx1.l16.m1.5.5.3" xref="algx1.l16.m1.5.5.3.cmml">←</mo><mrow id="algx1.l16.m1.5.5.2.2" xref="algx1.l16.m1.5.5.2.3.cmml"><mi mathsize="80%" id="algx1.l16.m1.3.3" xref="algx1.l16.m1.3.3.cmml">max</mi><mo id="algx1.l16.m1.5.5.2.2a" xref="algx1.l16.m1.5.5.2.3.cmml">⁡</mo><mrow id="algx1.l16.m1.5.5.2.2.2" xref="algx1.l16.m1.5.5.2.3.cmml"><mo maxsize="80%" minsize="80%" id="algx1.l16.m1.5.5.2.2.2.3" xref="algx1.l16.m1.5.5.2.3.cmml">{</mo><msub id="algx1.l16.m1.4.4.1.1.1.1" xref="algx1.l16.m1.4.4.1.1.1.1.cmml"><mi mathsize="80%" id="algx1.l16.m1.4.4.1.1.1.1.2" xref="algx1.l16.m1.4.4.1.1.1.1.2.cmml">m</mi><mi mathsize="80%" id="algx1.l16.m1.4.4.1.1.1.1.3" xref="algx1.l16.m1.4.4.1.1.1.1.3.cmml">i</mi></msub><mo mathsize="80%" id="algx1.l16.m1.5.5.2.2.2.4" xref="algx1.l16.m1.5.5.2.3.cmml">,</mo><msub id="algx1.l16.m1.5.5.2.2.2.2" xref="algx1.l16.m1.5.5.2.2.2.2.cmml"><mi mathsize="80%" id="algx1.l16.m1.5.5.2.2.2.2.2" xref="algx1.l16.m1.5.5.2.2.2.2.2.cmml">m</mi><mrow id="algx1.l16.m1.2.2.2.4" xref="algx1.l16.m1.2.2.2.3.cmml"><mi mathsize="80%" id="algx1.l16.m1.1.1.1.1" xref="algx1.l16.m1.1.1.1.1.cmml">i</mi><mo mathsize="80%" id="algx1.l16.m1.2.2.2.4.1" xref="algx1.l16.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="algx1.l16.m1.2.2.2.2" xref="algx1.l16.m1.2.2.2.2.cmml">j</mi></mrow></msub><mo maxsize="80%" minsize="80%" id="algx1.l16.m1.5.5.2.2.2.5" xref="algx1.l16.m1.5.5.2.3.cmml">}</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="algx1.l16.m1.5b"><apply id="algx1.l16.m1.5.5.cmml" xref="algx1.l16.m1.5.5"><ci id="algx1.l16.m1.5.5.3.cmml" xref="algx1.l16.m1.5.5.3">←</ci><apply id="algx1.l16.m1.5.5.4.cmml" xref="algx1.l16.m1.5.5.4"><csymbol cd="ambiguous" id="algx1.l16.m1.5.5.4.1.cmml" xref="algx1.l16.m1.5.5.4">subscript</csymbol><ci id="algx1.l16.m1.5.5.4.2.cmml" xref="algx1.l16.m1.5.5.4.2">𝑚</ci><ci id="algx1.l16.m1.5.5.4.3a.cmml" xref="algx1.l16.m1.5.5.4.3"><mtext mathsize="56%" id="algx1.l16.m1.5.5.4.3.cmml" xref="algx1.l16.m1.5.5.4.3">new</mtext></ci></apply><apply id="algx1.l16.m1.5.5.2.3.cmml" xref="algx1.l16.m1.5.5.2.2"><max id="algx1.l16.m1.3.3.cmml" xref="algx1.l16.m1.3.3"></max><apply id="algx1.l16.m1.4.4.1.1.1.1.cmml" xref="algx1.l16.m1.4.4.1.1.1.1"><csymbol cd="ambiguous" id="algx1.l16.m1.4.4.1.1.1.1.1.cmml" xref="algx1.l16.m1.4.4.1.1.1.1">subscript</csymbol><ci id="algx1.l16.m1.4.4.1.1.1.1.2.cmml" xref="algx1.l16.m1.4.4.1.1.1.1.2">𝑚</ci><ci id="algx1.l16.m1.4.4.1.1.1.1.3.cmml" xref="algx1.l16.m1.4.4.1.1.1.1.3">𝑖</ci></apply><apply id="algx1.l16.m1.5.5.2.2.2.2.cmml" xref="algx1.l16.m1.5.5.2.2.2.2"><csymbol cd="ambiguous" id="algx1.l16.m1.5.5.2.2.2.2.1.cmml" xref="algx1.l16.m1.5.5.2.2.2.2">subscript</csymbol><ci id="algx1.l16.m1.5.5.2.2.2.2.2.cmml" xref="algx1.l16.m1.5.5.2.2.2.2.2">𝑚</ci><list id="algx1.l16.m1.2.2.2.3.cmml" xref="algx1.l16.m1.2.2.2.4"><ci id="algx1.l16.m1.1.1.1.1.cmml" xref="algx1.l16.m1.1.1.1.1">𝑖</ci><ci id="algx1.l16.m1.2.2.2.2.cmml" xref="algx1.l16.m1.2.2.2.2">𝑗</ci></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="algx1.l16.m1.5c">m_{\text{new}}\leftarrow\max{\{m_{i},m_{i,j}\}}</annotation></semantics></math><span id="algx1.l16.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="algx1.l17" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l17.1.1.1" class="ltx_text" style="font-size:80%;">17:</span></span><span id="algx1.l17.2" class="ltx_text" style="font-size:80%;">    </span><math id="algx1.l17.m1.4" class="ltx_Math" alttext="\mathbf{l}_{i}=e^{m_{i}-m_{\text{new}}}l_{i}+e^{m_{i,j}-m_{\text{new}}}l_{i,j}" display="inline"><semantics id="algx1.l17.m1.4a"><mrow id="algx1.l17.m1.4.5" xref="algx1.l17.m1.4.5.cmml"><msub id="algx1.l17.m1.4.5.2" xref="algx1.l17.m1.4.5.2.cmml"><mi mathsize="80%" id="algx1.l17.m1.4.5.2.2" xref="algx1.l17.m1.4.5.2.2.cmml">𝐥</mi><mi mathsize="80%" id="algx1.l17.m1.4.5.2.3" xref="algx1.l17.m1.4.5.2.3.cmml">i</mi></msub><mo mathsize="80%" id="algx1.l17.m1.4.5.1" xref="algx1.l17.m1.4.5.1.cmml">=</mo><mrow id="algx1.l17.m1.4.5.3" xref="algx1.l17.m1.4.5.3.cmml"><mrow id="algx1.l17.m1.4.5.3.2" xref="algx1.l17.m1.4.5.3.2.cmml"><msup id="algx1.l17.m1.4.5.3.2.2" xref="algx1.l17.m1.4.5.3.2.2.cmml"><mi mathsize="80%" id="algx1.l17.m1.4.5.3.2.2.2" xref="algx1.l17.m1.4.5.3.2.2.2.cmml">e</mi><mrow id="algx1.l17.m1.4.5.3.2.2.3" xref="algx1.l17.m1.4.5.3.2.2.3.cmml"><msub id="algx1.l17.m1.4.5.3.2.2.3.2" xref="algx1.l17.m1.4.5.3.2.2.3.2.cmml"><mi mathsize="80%" id="algx1.l17.m1.4.5.3.2.2.3.2.2" xref="algx1.l17.m1.4.5.3.2.2.3.2.2.cmml">m</mi><mi mathsize="80%" id="algx1.l17.m1.4.5.3.2.2.3.2.3" xref="algx1.l17.m1.4.5.3.2.2.3.2.3.cmml">i</mi></msub><mo mathsize="80%" id="algx1.l17.m1.4.5.3.2.2.3.1" xref="algx1.l17.m1.4.5.3.2.2.3.1.cmml">−</mo><msub id="algx1.l17.m1.4.5.3.2.2.3.3" xref="algx1.l17.m1.4.5.3.2.2.3.3.cmml"><mi mathsize="80%" id="algx1.l17.m1.4.5.3.2.2.3.3.2" xref="algx1.l17.m1.4.5.3.2.2.3.3.2.cmml">m</mi><mtext mathsize="80%" id="algx1.l17.m1.4.5.3.2.2.3.3.3" xref="algx1.l17.m1.4.5.3.2.2.3.3.3a.cmml">new</mtext></msub></mrow></msup><mo lspace="0em" rspace="0em" id="algx1.l17.m1.4.5.3.2.1" xref="algx1.l17.m1.4.5.3.2.1.cmml">​</mo><msub id="algx1.l17.m1.4.5.3.2.3" xref="algx1.l17.m1.4.5.3.2.3.cmml"><mi mathsize="80%" id="algx1.l17.m1.4.5.3.2.3.2" xref="algx1.l17.m1.4.5.3.2.3.2.cmml">l</mi><mi mathsize="80%" id="algx1.l17.m1.4.5.3.2.3.3" xref="algx1.l17.m1.4.5.3.2.3.3.cmml">i</mi></msub></mrow><mo mathsize="80%" id="algx1.l17.m1.4.5.3.1" xref="algx1.l17.m1.4.5.3.1.cmml">+</mo><mrow id="algx1.l17.m1.4.5.3.3" xref="algx1.l17.m1.4.5.3.3.cmml"><msup id="algx1.l17.m1.4.5.3.3.2" xref="algx1.l17.m1.4.5.3.3.2.cmml"><mi mathsize="80%" id="algx1.l17.m1.4.5.3.3.2.2" xref="algx1.l17.m1.4.5.3.3.2.2.cmml">e</mi><mrow id="algx1.l17.m1.2.2.2" xref="algx1.l17.m1.2.2.2.cmml"><msub id="algx1.l17.m1.2.2.2.4" xref="algx1.l17.m1.2.2.2.4.cmml"><mi mathsize="80%" id="algx1.l17.m1.2.2.2.4.2" xref="algx1.l17.m1.2.2.2.4.2.cmml">m</mi><mrow id="algx1.l17.m1.2.2.2.2.2.4" xref="algx1.l17.m1.2.2.2.2.2.3.cmml"><mi mathsize="80%" id="algx1.l17.m1.1.1.1.1.1.1" xref="algx1.l17.m1.1.1.1.1.1.1.cmml">i</mi><mo mathsize="80%" id="algx1.l17.m1.2.2.2.2.2.4.1" xref="algx1.l17.m1.2.2.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="algx1.l17.m1.2.2.2.2.2.2" xref="algx1.l17.m1.2.2.2.2.2.2.cmml">j</mi></mrow></msub><mo mathsize="80%" id="algx1.l17.m1.2.2.2.3" xref="algx1.l17.m1.2.2.2.3.cmml">−</mo><msub id="algx1.l17.m1.2.2.2.5" xref="algx1.l17.m1.2.2.2.5.cmml"><mi mathsize="80%" id="algx1.l17.m1.2.2.2.5.2" xref="algx1.l17.m1.2.2.2.5.2.cmml">m</mi><mtext mathsize="80%" id="algx1.l17.m1.2.2.2.5.3" xref="algx1.l17.m1.2.2.2.5.3a.cmml">new</mtext></msub></mrow></msup><mo lspace="0em" rspace="0em" id="algx1.l17.m1.4.5.3.3.1" xref="algx1.l17.m1.4.5.3.3.1.cmml">​</mo><msub id="algx1.l17.m1.4.5.3.3.3" xref="algx1.l17.m1.4.5.3.3.3.cmml"><mi mathsize="80%" id="algx1.l17.m1.4.5.3.3.3.2" xref="algx1.l17.m1.4.5.3.3.3.2.cmml">l</mi><mrow id="algx1.l17.m1.4.4.2.4" xref="algx1.l17.m1.4.4.2.3.cmml"><mi mathsize="80%" id="algx1.l17.m1.3.3.1.1" xref="algx1.l17.m1.3.3.1.1.cmml">i</mi><mo mathsize="80%" id="algx1.l17.m1.4.4.2.4.1" xref="algx1.l17.m1.4.4.2.3.cmml">,</mo><mi mathsize="80%" id="algx1.l17.m1.4.4.2.2" xref="algx1.l17.m1.4.4.2.2.cmml">j</mi></mrow></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="algx1.l17.m1.4b"><apply id="algx1.l17.m1.4.5.cmml" xref="algx1.l17.m1.4.5"><eq id="algx1.l17.m1.4.5.1.cmml" xref="algx1.l17.m1.4.5.1"></eq><apply id="algx1.l17.m1.4.5.2.cmml" xref="algx1.l17.m1.4.5.2"><csymbol cd="ambiguous" id="algx1.l17.m1.4.5.2.1.cmml" xref="algx1.l17.m1.4.5.2">subscript</csymbol><ci id="algx1.l17.m1.4.5.2.2.cmml" xref="algx1.l17.m1.4.5.2.2">𝐥</ci><ci id="algx1.l17.m1.4.5.2.3.cmml" xref="algx1.l17.m1.4.5.2.3">𝑖</ci></apply><apply id="algx1.l17.m1.4.5.3.cmml" xref="algx1.l17.m1.4.5.3"><plus id="algx1.l17.m1.4.5.3.1.cmml" xref="algx1.l17.m1.4.5.3.1"></plus><apply id="algx1.l17.m1.4.5.3.2.cmml" xref="algx1.l17.m1.4.5.3.2"><times id="algx1.l17.m1.4.5.3.2.1.cmml" xref="algx1.l17.m1.4.5.3.2.1"></times><apply id="algx1.l17.m1.4.5.3.2.2.cmml" xref="algx1.l17.m1.4.5.3.2.2"><csymbol cd="ambiguous" id="algx1.l17.m1.4.5.3.2.2.1.cmml" xref="algx1.l17.m1.4.5.3.2.2">superscript</csymbol><ci id="algx1.l17.m1.4.5.3.2.2.2.cmml" xref="algx1.l17.m1.4.5.3.2.2.2">𝑒</ci><apply id="algx1.l17.m1.4.5.3.2.2.3.cmml" xref="algx1.l17.m1.4.5.3.2.2.3"><minus id="algx1.l17.m1.4.5.3.2.2.3.1.cmml" xref="algx1.l17.m1.4.5.3.2.2.3.1"></minus><apply id="algx1.l17.m1.4.5.3.2.2.3.2.cmml" xref="algx1.l17.m1.4.5.3.2.2.3.2"><csymbol cd="ambiguous" id="algx1.l17.m1.4.5.3.2.2.3.2.1.cmml" xref="algx1.l17.m1.4.5.3.2.2.3.2">subscript</csymbol><ci id="algx1.l17.m1.4.5.3.2.2.3.2.2.cmml" xref="algx1.l17.m1.4.5.3.2.2.3.2.2">𝑚</ci><ci id="algx1.l17.m1.4.5.3.2.2.3.2.3.cmml" xref="algx1.l17.m1.4.5.3.2.2.3.2.3">𝑖</ci></apply><apply id="algx1.l17.m1.4.5.3.2.2.3.3.cmml" xref="algx1.l17.m1.4.5.3.2.2.3.3"><csymbol cd="ambiguous" id="algx1.l17.m1.4.5.3.2.2.3.3.1.cmml" xref="algx1.l17.m1.4.5.3.2.2.3.3">subscript</csymbol><ci id="algx1.l17.m1.4.5.3.2.2.3.3.2.cmml" xref="algx1.l17.m1.4.5.3.2.2.3.3.2">𝑚</ci><ci id="algx1.l17.m1.4.5.3.2.2.3.3.3a.cmml" xref="algx1.l17.m1.4.5.3.2.2.3.3.3"><mtext mathsize="40%" id="algx1.l17.m1.4.5.3.2.2.3.3.3.cmml" xref="algx1.l17.m1.4.5.3.2.2.3.3.3">new</mtext></ci></apply></apply></apply><apply id="algx1.l17.m1.4.5.3.2.3.cmml" xref="algx1.l17.m1.4.5.3.2.3"><csymbol cd="ambiguous" id="algx1.l17.m1.4.5.3.2.3.1.cmml" xref="algx1.l17.m1.4.5.3.2.3">subscript</csymbol><ci id="algx1.l17.m1.4.5.3.2.3.2.cmml" xref="algx1.l17.m1.4.5.3.2.3.2">𝑙</ci><ci id="algx1.l17.m1.4.5.3.2.3.3.cmml" xref="algx1.l17.m1.4.5.3.2.3.3">𝑖</ci></apply></apply><apply id="algx1.l17.m1.4.5.3.3.cmml" xref="algx1.l17.m1.4.5.3.3"><times id="algx1.l17.m1.4.5.3.3.1.cmml" xref="algx1.l17.m1.4.5.3.3.1"></times><apply id="algx1.l17.m1.4.5.3.3.2.cmml" xref="algx1.l17.m1.4.5.3.3.2"><csymbol cd="ambiguous" id="algx1.l17.m1.4.5.3.3.2.1.cmml" xref="algx1.l17.m1.4.5.3.3.2">superscript</csymbol><ci id="algx1.l17.m1.4.5.3.3.2.2.cmml" xref="algx1.l17.m1.4.5.3.3.2.2">𝑒</ci><apply id="algx1.l17.m1.2.2.2.cmml" xref="algx1.l17.m1.2.2.2"><minus id="algx1.l17.m1.2.2.2.3.cmml" xref="algx1.l17.m1.2.2.2.3"></minus><apply id="algx1.l17.m1.2.2.2.4.cmml" xref="algx1.l17.m1.2.2.2.4"><csymbol cd="ambiguous" id="algx1.l17.m1.2.2.2.4.1.cmml" xref="algx1.l17.m1.2.2.2.4">subscript</csymbol><ci id="algx1.l17.m1.2.2.2.4.2.cmml" xref="algx1.l17.m1.2.2.2.4.2">𝑚</ci><list id="algx1.l17.m1.2.2.2.2.2.3.cmml" xref="algx1.l17.m1.2.2.2.2.2.4"><ci id="algx1.l17.m1.1.1.1.1.1.1.cmml" xref="algx1.l17.m1.1.1.1.1.1.1">𝑖</ci><ci id="algx1.l17.m1.2.2.2.2.2.2.cmml" xref="algx1.l17.m1.2.2.2.2.2.2">𝑗</ci></list></apply><apply id="algx1.l17.m1.2.2.2.5.cmml" xref="algx1.l17.m1.2.2.2.5"><csymbol cd="ambiguous" id="algx1.l17.m1.2.2.2.5.1.cmml" xref="algx1.l17.m1.2.2.2.5">subscript</csymbol><ci id="algx1.l17.m1.2.2.2.5.2.cmml" xref="algx1.l17.m1.2.2.2.5.2">𝑚</ci><ci id="algx1.l17.m1.2.2.2.5.3a.cmml" xref="algx1.l17.m1.2.2.2.5.3"><mtext mathsize="40%" id="algx1.l17.m1.2.2.2.5.3.cmml" xref="algx1.l17.m1.2.2.2.5.3">new</mtext></ci></apply></apply></apply><apply id="algx1.l17.m1.4.5.3.3.3.cmml" xref="algx1.l17.m1.4.5.3.3.3"><csymbol cd="ambiguous" id="algx1.l17.m1.4.5.3.3.3.1.cmml" xref="algx1.l17.m1.4.5.3.3.3">subscript</csymbol><ci id="algx1.l17.m1.4.5.3.3.3.2.cmml" xref="algx1.l17.m1.4.5.3.3.3.2">𝑙</ci><list id="algx1.l17.m1.4.4.2.3.cmml" xref="algx1.l17.m1.4.4.2.4"><ci id="algx1.l17.m1.3.3.1.1.cmml" xref="algx1.l17.m1.3.3.1.1">𝑖</ci><ci id="algx1.l17.m1.4.4.2.2.cmml" xref="algx1.l17.m1.4.4.2.2">𝑗</ci></list></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="algx1.l17.m1.4c">\mathbf{l}_{i}=e^{m_{i}-m_{\text{new}}}l_{i}+e^{m_{i,j}-m_{\text{new}}}l_{i,j}</annotation></semantics></math><span id="algx1.l17.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="algx1.l18" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l18.1.1.1" class="ltx_text" style="font-size:80%;">18:</span></span><span id="algx1.l18.2" class="ltx_text" style="font-size:80%;">    </span><math id="algx1.l18.m1.4" class="ltx_Math" alttext="\mathbf{O}_{i}=e^{m_{i}-m_{\text{new}}}\mathbf{O}_{i}+e^{m_{i,j}-m_{\text{new}}}\mathbf{O}_{i,j}" display="inline"><semantics id="algx1.l18.m1.4a"><mrow id="algx1.l18.m1.4.5" xref="algx1.l18.m1.4.5.cmml"><msub id="algx1.l18.m1.4.5.2" xref="algx1.l18.m1.4.5.2.cmml"><mi mathsize="80%" id="algx1.l18.m1.4.5.2.2" xref="algx1.l18.m1.4.5.2.2.cmml">𝐎</mi><mi mathsize="80%" id="algx1.l18.m1.4.5.2.3" xref="algx1.l18.m1.4.5.2.3.cmml">i</mi></msub><mo mathsize="80%" id="algx1.l18.m1.4.5.1" xref="algx1.l18.m1.4.5.1.cmml">=</mo><mrow id="algx1.l18.m1.4.5.3" xref="algx1.l18.m1.4.5.3.cmml"><mrow id="algx1.l18.m1.4.5.3.2" xref="algx1.l18.m1.4.5.3.2.cmml"><msup id="algx1.l18.m1.4.5.3.2.2" xref="algx1.l18.m1.4.5.3.2.2.cmml"><mi mathsize="80%" id="algx1.l18.m1.4.5.3.2.2.2" xref="algx1.l18.m1.4.5.3.2.2.2.cmml">e</mi><mrow id="algx1.l18.m1.4.5.3.2.2.3" xref="algx1.l18.m1.4.5.3.2.2.3.cmml"><msub id="algx1.l18.m1.4.5.3.2.2.3.2" xref="algx1.l18.m1.4.5.3.2.2.3.2.cmml"><mi mathsize="80%" id="algx1.l18.m1.4.5.3.2.2.3.2.2" xref="algx1.l18.m1.4.5.3.2.2.3.2.2.cmml">m</mi><mi mathsize="80%" id="algx1.l18.m1.4.5.3.2.2.3.2.3" xref="algx1.l18.m1.4.5.3.2.2.3.2.3.cmml">i</mi></msub><mo mathsize="80%" id="algx1.l18.m1.4.5.3.2.2.3.1" xref="algx1.l18.m1.4.5.3.2.2.3.1.cmml">−</mo><msub id="algx1.l18.m1.4.5.3.2.2.3.3" xref="algx1.l18.m1.4.5.3.2.2.3.3.cmml"><mi mathsize="80%" id="algx1.l18.m1.4.5.3.2.2.3.3.2" xref="algx1.l18.m1.4.5.3.2.2.3.3.2.cmml">m</mi><mtext mathsize="80%" id="algx1.l18.m1.4.5.3.2.2.3.3.3" xref="algx1.l18.m1.4.5.3.2.2.3.3.3a.cmml">new</mtext></msub></mrow></msup><mo lspace="0em" rspace="0em" id="algx1.l18.m1.4.5.3.2.1" xref="algx1.l18.m1.4.5.3.2.1.cmml">​</mo><msub id="algx1.l18.m1.4.5.3.2.3" xref="algx1.l18.m1.4.5.3.2.3.cmml"><mi mathsize="80%" id="algx1.l18.m1.4.5.3.2.3.2" xref="algx1.l18.m1.4.5.3.2.3.2.cmml">𝐎</mi><mi mathsize="80%" id="algx1.l18.m1.4.5.3.2.3.3" xref="algx1.l18.m1.4.5.3.2.3.3.cmml">i</mi></msub></mrow><mo mathsize="80%" id="algx1.l18.m1.4.5.3.1" xref="algx1.l18.m1.4.5.3.1.cmml">+</mo><mrow id="algx1.l18.m1.4.5.3.3" xref="algx1.l18.m1.4.5.3.3.cmml"><msup id="algx1.l18.m1.4.5.3.3.2" xref="algx1.l18.m1.4.5.3.3.2.cmml"><mi mathsize="80%" id="algx1.l18.m1.4.5.3.3.2.2" xref="algx1.l18.m1.4.5.3.3.2.2.cmml">e</mi><mrow id="algx1.l18.m1.2.2.2" xref="algx1.l18.m1.2.2.2.cmml"><msub id="algx1.l18.m1.2.2.2.4" xref="algx1.l18.m1.2.2.2.4.cmml"><mi mathsize="80%" id="algx1.l18.m1.2.2.2.4.2" xref="algx1.l18.m1.2.2.2.4.2.cmml">m</mi><mrow id="algx1.l18.m1.2.2.2.2.2.4" xref="algx1.l18.m1.2.2.2.2.2.3.cmml"><mi mathsize="80%" id="algx1.l18.m1.1.1.1.1.1.1" xref="algx1.l18.m1.1.1.1.1.1.1.cmml">i</mi><mo mathsize="80%" id="algx1.l18.m1.2.2.2.2.2.4.1" xref="algx1.l18.m1.2.2.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="algx1.l18.m1.2.2.2.2.2.2" xref="algx1.l18.m1.2.2.2.2.2.2.cmml">j</mi></mrow></msub><mo mathsize="80%" id="algx1.l18.m1.2.2.2.3" xref="algx1.l18.m1.2.2.2.3.cmml">−</mo><msub id="algx1.l18.m1.2.2.2.5" xref="algx1.l18.m1.2.2.2.5.cmml"><mi mathsize="80%" id="algx1.l18.m1.2.2.2.5.2" xref="algx1.l18.m1.2.2.2.5.2.cmml">m</mi><mtext mathsize="80%" id="algx1.l18.m1.2.2.2.5.3" xref="algx1.l18.m1.2.2.2.5.3a.cmml">new</mtext></msub></mrow></msup><mo lspace="0em" rspace="0em" id="algx1.l18.m1.4.5.3.3.1" xref="algx1.l18.m1.4.5.3.3.1.cmml">​</mo><msub id="algx1.l18.m1.4.5.3.3.3" xref="algx1.l18.m1.4.5.3.3.3.cmml"><mi mathsize="80%" id="algx1.l18.m1.4.5.3.3.3.2" xref="algx1.l18.m1.4.5.3.3.3.2.cmml">𝐎</mi><mrow id="algx1.l18.m1.4.4.2.4" xref="algx1.l18.m1.4.4.2.3.cmml"><mi mathsize="80%" id="algx1.l18.m1.3.3.1.1" xref="algx1.l18.m1.3.3.1.1.cmml">i</mi><mo mathsize="80%" id="algx1.l18.m1.4.4.2.4.1" xref="algx1.l18.m1.4.4.2.3.cmml">,</mo><mi mathsize="80%" id="algx1.l18.m1.4.4.2.2" xref="algx1.l18.m1.4.4.2.2.cmml">j</mi></mrow></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="algx1.l18.m1.4b"><apply id="algx1.l18.m1.4.5.cmml" xref="algx1.l18.m1.4.5"><eq id="algx1.l18.m1.4.5.1.cmml" xref="algx1.l18.m1.4.5.1"></eq><apply id="algx1.l18.m1.4.5.2.cmml" xref="algx1.l18.m1.4.5.2"><csymbol cd="ambiguous" id="algx1.l18.m1.4.5.2.1.cmml" xref="algx1.l18.m1.4.5.2">subscript</csymbol><ci id="algx1.l18.m1.4.5.2.2.cmml" xref="algx1.l18.m1.4.5.2.2">𝐎</ci><ci id="algx1.l18.m1.4.5.2.3.cmml" xref="algx1.l18.m1.4.5.2.3">𝑖</ci></apply><apply id="algx1.l18.m1.4.5.3.cmml" xref="algx1.l18.m1.4.5.3"><plus id="algx1.l18.m1.4.5.3.1.cmml" xref="algx1.l18.m1.4.5.3.1"></plus><apply id="algx1.l18.m1.4.5.3.2.cmml" xref="algx1.l18.m1.4.5.3.2"><times id="algx1.l18.m1.4.5.3.2.1.cmml" xref="algx1.l18.m1.4.5.3.2.1"></times><apply id="algx1.l18.m1.4.5.3.2.2.cmml" xref="algx1.l18.m1.4.5.3.2.2"><csymbol cd="ambiguous" id="algx1.l18.m1.4.5.3.2.2.1.cmml" xref="algx1.l18.m1.4.5.3.2.2">superscript</csymbol><ci id="algx1.l18.m1.4.5.3.2.2.2.cmml" xref="algx1.l18.m1.4.5.3.2.2.2">𝑒</ci><apply id="algx1.l18.m1.4.5.3.2.2.3.cmml" xref="algx1.l18.m1.4.5.3.2.2.3"><minus id="algx1.l18.m1.4.5.3.2.2.3.1.cmml" xref="algx1.l18.m1.4.5.3.2.2.3.1"></minus><apply id="algx1.l18.m1.4.5.3.2.2.3.2.cmml" xref="algx1.l18.m1.4.5.3.2.2.3.2"><csymbol cd="ambiguous" id="algx1.l18.m1.4.5.3.2.2.3.2.1.cmml" xref="algx1.l18.m1.4.5.3.2.2.3.2">subscript</csymbol><ci id="algx1.l18.m1.4.5.3.2.2.3.2.2.cmml" xref="algx1.l18.m1.4.5.3.2.2.3.2.2">𝑚</ci><ci id="algx1.l18.m1.4.5.3.2.2.3.2.3.cmml" xref="algx1.l18.m1.4.5.3.2.2.3.2.3">𝑖</ci></apply><apply id="algx1.l18.m1.4.5.3.2.2.3.3.cmml" xref="algx1.l18.m1.4.5.3.2.2.3.3"><csymbol cd="ambiguous" id="algx1.l18.m1.4.5.3.2.2.3.3.1.cmml" xref="algx1.l18.m1.4.5.3.2.2.3.3">subscript</csymbol><ci id="algx1.l18.m1.4.5.3.2.2.3.3.2.cmml" xref="algx1.l18.m1.4.5.3.2.2.3.3.2">𝑚</ci><ci id="algx1.l18.m1.4.5.3.2.2.3.3.3a.cmml" xref="algx1.l18.m1.4.5.3.2.2.3.3.3"><mtext mathsize="40%" id="algx1.l18.m1.4.5.3.2.2.3.3.3.cmml" xref="algx1.l18.m1.4.5.3.2.2.3.3.3">new</mtext></ci></apply></apply></apply><apply id="algx1.l18.m1.4.5.3.2.3.cmml" xref="algx1.l18.m1.4.5.3.2.3"><csymbol cd="ambiguous" id="algx1.l18.m1.4.5.3.2.3.1.cmml" xref="algx1.l18.m1.4.5.3.2.3">subscript</csymbol><ci id="algx1.l18.m1.4.5.3.2.3.2.cmml" xref="algx1.l18.m1.4.5.3.2.3.2">𝐎</ci><ci id="algx1.l18.m1.4.5.3.2.3.3.cmml" xref="algx1.l18.m1.4.5.3.2.3.3">𝑖</ci></apply></apply><apply id="algx1.l18.m1.4.5.3.3.cmml" xref="algx1.l18.m1.4.5.3.3"><times id="algx1.l18.m1.4.5.3.3.1.cmml" xref="algx1.l18.m1.4.5.3.3.1"></times><apply id="algx1.l18.m1.4.5.3.3.2.cmml" xref="algx1.l18.m1.4.5.3.3.2"><csymbol cd="ambiguous" id="algx1.l18.m1.4.5.3.3.2.1.cmml" xref="algx1.l18.m1.4.5.3.3.2">superscript</csymbol><ci id="algx1.l18.m1.4.5.3.3.2.2.cmml" xref="algx1.l18.m1.4.5.3.3.2.2">𝑒</ci><apply id="algx1.l18.m1.2.2.2.cmml" xref="algx1.l18.m1.2.2.2"><minus id="algx1.l18.m1.2.2.2.3.cmml" xref="algx1.l18.m1.2.2.2.3"></minus><apply id="algx1.l18.m1.2.2.2.4.cmml" xref="algx1.l18.m1.2.2.2.4"><csymbol cd="ambiguous" id="algx1.l18.m1.2.2.2.4.1.cmml" xref="algx1.l18.m1.2.2.2.4">subscript</csymbol><ci id="algx1.l18.m1.2.2.2.4.2.cmml" xref="algx1.l18.m1.2.2.2.4.2">𝑚</ci><list id="algx1.l18.m1.2.2.2.2.2.3.cmml" xref="algx1.l18.m1.2.2.2.2.2.4"><ci id="algx1.l18.m1.1.1.1.1.1.1.cmml" xref="algx1.l18.m1.1.1.1.1.1.1">𝑖</ci><ci id="algx1.l18.m1.2.2.2.2.2.2.cmml" xref="algx1.l18.m1.2.2.2.2.2.2">𝑗</ci></list></apply><apply id="algx1.l18.m1.2.2.2.5.cmml" xref="algx1.l18.m1.2.2.2.5"><csymbol cd="ambiguous" id="algx1.l18.m1.2.2.2.5.1.cmml" xref="algx1.l18.m1.2.2.2.5">subscript</csymbol><ci id="algx1.l18.m1.2.2.2.5.2.cmml" xref="algx1.l18.m1.2.2.2.5.2">𝑚</ci><ci id="algx1.l18.m1.2.2.2.5.3a.cmml" xref="algx1.l18.m1.2.2.2.5.3"><mtext mathsize="40%" id="algx1.l18.m1.2.2.2.5.3.cmml" xref="algx1.l18.m1.2.2.2.5.3">new</mtext></ci></apply></apply></apply><apply id="algx1.l18.m1.4.5.3.3.3.cmml" xref="algx1.l18.m1.4.5.3.3.3"><csymbol cd="ambiguous" id="algx1.l18.m1.4.5.3.3.3.1.cmml" xref="algx1.l18.m1.4.5.3.3.3">subscript</csymbol><ci id="algx1.l18.m1.4.5.3.3.3.2.cmml" xref="algx1.l18.m1.4.5.3.3.3.2">𝐎</ci><list id="algx1.l18.m1.4.4.2.3.cmml" xref="algx1.l18.m1.4.4.2.4"><ci id="algx1.l18.m1.3.3.1.1.cmml" xref="algx1.l18.m1.3.3.1.1">𝑖</ci><ci id="algx1.l18.m1.4.4.2.2.cmml" xref="algx1.l18.m1.4.4.2.2">𝑗</ci></list></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="algx1.l18.m1.4c">\mathbf{O}_{i}=e^{m_{i}-m_{\text{new}}}\mathbf{O}_{i}+e^{m_{i,j}-m_{\text{new}}}\mathbf{O}_{i,j}</annotation></semantics></math><span id="algx1.l18.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="algx1.l19" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l19.1.1.1" class="ltx_text" style="font-size:80%;">19:</span></span><span id="algx1.l19.2" class="ltx_text" style="font-size:80%;">    </span><math id="algx1.l19.m1.1" class="ltx_Math" alttext="m_{i}=m_{\text{new}}" display="inline"><semantics id="algx1.l19.m1.1a"><mrow id="algx1.l19.m1.1.1" xref="algx1.l19.m1.1.1.cmml"><msub id="algx1.l19.m1.1.1.2" xref="algx1.l19.m1.1.1.2.cmml"><mi mathsize="80%" id="algx1.l19.m1.1.1.2.2" xref="algx1.l19.m1.1.1.2.2.cmml">m</mi><mi mathsize="80%" id="algx1.l19.m1.1.1.2.3" xref="algx1.l19.m1.1.1.2.3.cmml">i</mi></msub><mo mathsize="80%" id="algx1.l19.m1.1.1.1" xref="algx1.l19.m1.1.1.1.cmml">=</mo><msub id="algx1.l19.m1.1.1.3" xref="algx1.l19.m1.1.1.3.cmml"><mi mathsize="80%" id="algx1.l19.m1.1.1.3.2" xref="algx1.l19.m1.1.1.3.2.cmml">m</mi><mtext mathsize="80%" id="algx1.l19.m1.1.1.3.3" xref="algx1.l19.m1.1.1.3.3a.cmml">new</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="algx1.l19.m1.1b"><apply id="algx1.l19.m1.1.1.cmml" xref="algx1.l19.m1.1.1"><eq id="algx1.l19.m1.1.1.1.cmml" xref="algx1.l19.m1.1.1.1"></eq><apply id="algx1.l19.m1.1.1.2.cmml" xref="algx1.l19.m1.1.1.2"><csymbol cd="ambiguous" id="algx1.l19.m1.1.1.2.1.cmml" xref="algx1.l19.m1.1.1.2">subscript</csymbol><ci id="algx1.l19.m1.1.1.2.2.cmml" xref="algx1.l19.m1.1.1.2.2">𝑚</ci><ci id="algx1.l19.m1.1.1.2.3.cmml" xref="algx1.l19.m1.1.1.2.3">𝑖</ci></apply><apply id="algx1.l19.m1.1.1.3.cmml" xref="algx1.l19.m1.1.1.3"><csymbol cd="ambiguous" id="algx1.l19.m1.1.1.3.1.cmml" xref="algx1.l19.m1.1.1.3">subscript</csymbol><ci id="algx1.l19.m1.1.1.3.2.cmml" xref="algx1.l19.m1.1.1.3.2">𝑚</ci><ci id="algx1.l19.m1.1.1.3.3a.cmml" xref="algx1.l19.m1.1.1.3.3"><mtext mathsize="56%" id="algx1.l19.m1.1.1.3.3.cmml" xref="algx1.l19.m1.1.1.3.3">new</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="algx1.l19.m1.1c">m_{i}=m_{\text{new}}</annotation></semantics></math><span id="algx1.l19.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="algx1.l20" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l20.1.1.1" class="ltx_text" style="font-size:80%;">20:</span></span><span id="algx1.l20.2" class="ltx_text" style="font-size:80%;">  </span><span id="algx1.l20.3" class="ltx_text ltx_font_bold" style="font-size:80%;">end</span><span id="algx1.l20.4" class="ltx_text" style="font-size:80%;"> </span><span id="algx1.l20.5" class="ltx_text ltx_font_bold" style="font-size:80%;">for</span>
</div>
<div id="algx1.l21" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l21.1.1.1" class="ltx_text" style="font-size:80%;">21:</span></span><span id="algx1.l21.2" class="ltx_text" style="font-size:80%;">  </span><math id="algx1.l21.m1.1" class="ltx_Math" alttext="\mathbf{O}_{i}=\text{diag}(l_{i})^{-1}\mathbf{O}_{i}" display="inline"><semantics id="algx1.l21.m1.1a"><mrow id="algx1.l21.m1.1.1" xref="algx1.l21.m1.1.1.cmml"><msub id="algx1.l21.m1.1.1.3" xref="algx1.l21.m1.1.1.3.cmml"><mi mathsize="80%" id="algx1.l21.m1.1.1.3.2" xref="algx1.l21.m1.1.1.3.2.cmml">𝐎</mi><mi mathsize="80%" id="algx1.l21.m1.1.1.3.3" xref="algx1.l21.m1.1.1.3.3.cmml">i</mi></msub><mo mathsize="80%" id="algx1.l21.m1.1.1.2" xref="algx1.l21.m1.1.1.2.cmml">=</mo><mrow id="algx1.l21.m1.1.1.1" xref="algx1.l21.m1.1.1.1.cmml"><mtext mathsize="80%" id="algx1.l21.m1.1.1.1.3" xref="algx1.l21.m1.1.1.1.3a.cmml">diag</mtext><mo lspace="0em" rspace="0em" id="algx1.l21.m1.1.1.1.2" xref="algx1.l21.m1.1.1.1.2.cmml">​</mo><msup id="algx1.l21.m1.1.1.1.1" xref="algx1.l21.m1.1.1.1.1.cmml"><mrow id="algx1.l21.m1.1.1.1.1.1.1" xref="algx1.l21.m1.1.1.1.1.1.1.1.cmml"><mo maxsize="80%" minsize="80%" id="algx1.l21.m1.1.1.1.1.1.1.2" xref="algx1.l21.m1.1.1.1.1.1.1.1.cmml">(</mo><msub id="algx1.l21.m1.1.1.1.1.1.1.1" xref="algx1.l21.m1.1.1.1.1.1.1.1.cmml"><mi mathsize="80%" id="algx1.l21.m1.1.1.1.1.1.1.1.2" xref="algx1.l21.m1.1.1.1.1.1.1.1.2.cmml">l</mi><mi mathsize="80%" id="algx1.l21.m1.1.1.1.1.1.1.1.3" xref="algx1.l21.m1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo maxsize="80%" minsize="80%" id="algx1.l21.m1.1.1.1.1.1.1.3" xref="algx1.l21.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mrow id="algx1.l21.m1.1.1.1.1.3" xref="algx1.l21.m1.1.1.1.1.3.cmml"><mo mathsize="80%" id="algx1.l21.m1.1.1.1.1.3a" xref="algx1.l21.m1.1.1.1.1.3.cmml">−</mo><mn mathsize="80%" id="algx1.l21.m1.1.1.1.1.3.2" xref="algx1.l21.m1.1.1.1.1.3.2.cmml">1</mn></mrow></msup><mo lspace="0em" rspace="0em" id="algx1.l21.m1.1.1.1.2a" xref="algx1.l21.m1.1.1.1.2.cmml">​</mo><msub id="algx1.l21.m1.1.1.1.4" xref="algx1.l21.m1.1.1.1.4.cmml"><mi mathsize="80%" id="algx1.l21.m1.1.1.1.4.2" xref="algx1.l21.m1.1.1.1.4.2.cmml">𝐎</mi><mi mathsize="80%" id="algx1.l21.m1.1.1.1.4.3" xref="algx1.l21.m1.1.1.1.4.3.cmml">i</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="algx1.l21.m1.1b"><apply id="algx1.l21.m1.1.1.cmml" xref="algx1.l21.m1.1.1"><eq id="algx1.l21.m1.1.1.2.cmml" xref="algx1.l21.m1.1.1.2"></eq><apply id="algx1.l21.m1.1.1.3.cmml" xref="algx1.l21.m1.1.1.3"><csymbol cd="ambiguous" id="algx1.l21.m1.1.1.3.1.cmml" xref="algx1.l21.m1.1.1.3">subscript</csymbol><ci id="algx1.l21.m1.1.1.3.2.cmml" xref="algx1.l21.m1.1.1.3.2">𝐎</ci><ci id="algx1.l21.m1.1.1.3.3.cmml" xref="algx1.l21.m1.1.1.3.3">𝑖</ci></apply><apply id="algx1.l21.m1.1.1.1.cmml" xref="algx1.l21.m1.1.1.1"><times id="algx1.l21.m1.1.1.1.2.cmml" xref="algx1.l21.m1.1.1.1.2"></times><ci id="algx1.l21.m1.1.1.1.3a.cmml" xref="algx1.l21.m1.1.1.1.3"><mtext mathsize="80%" id="algx1.l21.m1.1.1.1.3.cmml" xref="algx1.l21.m1.1.1.1.3">diag</mtext></ci><apply id="algx1.l21.m1.1.1.1.1.cmml" xref="algx1.l21.m1.1.1.1.1"><csymbol cd="ambiguous" id="algx1.l21.m1.1.1.1.1.2.cmml" xref="algx1.l21.m1.1.1.1.1">superscript</csymbol><apply id="algx1.l21.m1.1.1.1.1.1.1.1.cmml" xref="algx1.l21.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="algx1.l21.m1.1.1.1.1.1.1.1.1.cmml" xref="algx1.l21.m1.1.1.1.1.1.1">subscript</csymbol><ci id="algx1.l21.m1.1.1.1.1.1.1.1.2.cmml" xref="algx1.l21.m1.1.1.1.1.1.1.1.2">𝑙</ci><ci id="algx1.l21.m1.1.1.1.1.1.1.1.3.cmml" xref="algx1.l21.m1.1.1.1.1.1.1.1.3">𝑖</ci></apply><apply id="algx1.l21.m1.1.1.1.1.3.cmml" xref="algx1.l21.m1.1.1.1.1.3"><minus id="algx1.l21.m1.1.1.1.1.3.1.cmml" xref="algx1.l21.m1.1.1.1.1.3"></minus><cn type="integer" id="algx1.l21.m1.1.1.1.1.3.2.cmml" xref="algx1.l21.m1.1.1.1.1.3.2">1</cn></apply></apply><apply id="algx1.l21.m1.1.1.1.4.cmml" xref="algx1.l21.m1.1.1.1.4"><csymbol cd="ambiguous" id="algx1.l21.m1.1.1.1.4.1.cmml" xref="algx1.l21.m1.1.1.1.4">subscript</csymbol><ci id="algx1.l21.m1.1.1.1.4.2.cmml" xref="algx1.l21.m1.1.1.1.4.2">𝐎</ci><ci id="algx1.l21.m1.1.1.1.4.3.cmml" xref="algx1.l21.m1.1.1.1.4.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="algx1.l21.m1.1c">\mathbf{O}_{i}=\text{diag}(l_{i})^{-1}\mathbf{O}_{i}</annotation></semantics></math><span id="algx1.l21.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="algx1.l22" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l22.1.1.1" class="ltx_text" style="font-size:80%;">22:</span></span><span id="algx1.l22.2" class="ltx_text" style="font-size:80%;">  </span><math id="algx1.l22.m1.1" class="ltx_Math" alttext="lse_{i}=m_{i}+\log{l_{i}}" display="inline"><semantics id="algx1.l22.m1.1a"><mrow id="algx1.l22.m1.1.1" xref="algx1.l22.m1.1.1.cmml"><mrow id="algx1.l22.m1.1.1.2" xref="algx1.l22.m1.1.1.2.cmml"><mi mathsize="80%" id="algx1.l22.m1.1.1.2.2" xref="algx1.l22.m1.1.1.2.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="algx1.l22.m1.1.1.2.1" xref="algx1.l22.m1.1.1.2.1.cmml">​</mo><mi mathsize="80%" id="algx1.l22.m1.1.1.2.3" xref="algx1.l22.m1.1.1.2.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="algx1.l22.m1.1.1.2.1a" xref="algx1.l22.m1.1.1.2.1.cmml">​</mo><msub id="algx1.l22.m1.1.1.2.4" xref="algx1.l22.m1.1.1.2.4.cmml"><mi mathsize="80%" id="algx1.l22.m1.1.1.2.4.2" xref="algx1.l22.m1.1.1.2.4.2.cmml">e</mi><mi mathsize="80%" id="algx1.l22.m1.1.1.2.4.3" xref="algx1.l22.m1.1.1.2.4.3.cmml">i</mi></msub></mrow><mo mathsize="80%" id="algx1.l22.m1.1.1.1" xref="algx1.l22.m1.1.1.1.cmml">=</mo><mrow id="algx1.l22.m1.1.1.3" xref="algx1.l22.m1.1.1.3.cmml"><msub id="algx1.l22.m1.1.1.3.2" xref="algx1.l22.m1.1.1.3.2.cmml"><mi mathsize="80%" id="algx1.l22.m1.1.1.3.2.2" xref="algx1.l22.m1.1.1.3.2.2.cmml">m</mi><mi mathsize="80%" id="algx1.l22.m1.1.1.3.2.3" xref="algx1.l22.m1.1.1.3.2.3.cmml">i</mi></msub><mo mathsize="80%" id="algx1.l22.m1.1.1.3.1" xref="algx1.l22.m1.1.1.3.1.cmml">+</mo><mrow id="algx1.l22.m1.1.1.3.3" xref="algx1.l22.m1.1.1.3.3.cmml"><mi mathsize="80%" id="algx1.l22.m1.1.1.3.3.1" xref="algx1.l22.m1.1.1.3.3.1.cmml">log</mi><mo lspace="0.167em" id="algx1.l22.m1.1.1.3.3a" xref="algx1.l22.m1.1.1.3.3.cmml">⁡</mo><msub id="algx1.l22.m1.1.1.3.3.2" xref="algx1.l22.m1.1.1.3.3.2.cmml"><mi mathsize="80%" id="algx1.l22.m1.1.1.3.3.2.2" xref="algx1.l22.m1.1.1.3.3.2.2.cmml">l</mi><mi mathsize="80%" id="algx1.l22.m1.1.1.3.3.2.3" xref="algx1.l22.m1.1.1.3.3.2.3.cmml">i</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="algx1.l22.m1.1b"><apply id="algx1.l22.m1.1.1.cmml" xref="algx1.l22.m1.1.1"><eq id="algx1.l22.m1.1.1.1.cmml" xref="algx1.l22.m1.1.1.1"></eq><apply id="algx1.l22.m1.1.1.2.cmml" xref="algx1.l22.m1.1.1.2"><times id="algx1.l22.m1.1.1.2.1.cmml" xref="algx1.l22.m1.1.1.2.1"></times><ci id="algx1.l22.m1.1.1.2.2.cmml" xref="algx1.l22.m1.1.1.2.2">𝑙</ci><ci id="algx1.l22.m1.1.1.2.3.cmml" xref="algx1.l22.m1.1.1.2.3">𝑠</ci><apply id="algx1.l22.m1.1.1.2.4.cmml" xref="algx1.l22.m1.1.1.2.4"><csymbol cd="ambiguous" id="algx1.l22.m1.1.1.2.4.1.cmml" xref="algx1.l22.m1.1.1.2.4">subscript</csymbol><ci id="algx1.l22.m1.1.1.2.4.2.cmml" xref="algx1.l22.m1.1.1.2.4.2">𝑒</ci><ci id="algx1.l22.m1.1.1.2.4.3.cmml" xref="algx1.l22.m1.1.1.2.4.3">𝑖</ci></apply></apply><apply id="algx1.l22.m1.1.1.3.cmml" xref="algx1.l22.m1.1.1.3"><plus id="algx1.l22.m1.1.1.3.1.cmml" xref="algx1.l22.m1.1.1.3.1"></plus><apply id="algx1.l22.m1.1.1.3.2.cmml" xref="algx1.l22.m1.1.1.3.2"><csymbol cd="ambiguous" id="algx1.l22.m1.1.1.3.2.1.cmml" xref="algx1.l22.m1.1.1.3.2">subscript</csymbol><ci id="algx1.l22.m1.1.1.3.2.2.cmml" xref="algx1.l22.m1.1.1.3.2.2">𝑚</ci><ci id="algx1.l22.m1.1.1.3.2.3.cmml" xref="algx1.l22.m1.1.1.3.2.3">𝑖</ci></apply><apply id="algx1.l22.m1.1.1.3.3.cmml" xref="algx1.l22.m1.1.1.3.3"><log id="algx1.l22.m1.1.1.3.3.1.cmml" xref="algx1.l22.m1.1.1.3.3.1"></log><apply id="algx1.l22.m1.1.1.3.3.2.cmml" xref="algx1.l22.m1.1.1.3.3.2"><csymbol cd="ambiguous" id="algx1.l22.m1.1.1.3.3.2.1.cmml" xref="algx1.l22.m1.1.1.3.3.2">subscript</csymbol><ci id="algx1.l22.m1.1.1.3.3.2.2.cmml" xref="algx1.l22.m1.1.1.3.3.2.2">𝑙</ci><ci id="algx1.l22.m1.1.1.3.3.2.3.cmml" xref="algx1.l22.m1.1.1.3.3.2.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="algx1.l22.m1.1c">lse_{i}=m_{i}+\log{l_{i}}</annotation></semantics></math><span id="algx1.l22.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="algx1.l22a" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="algx1.l22a.1.1.1" class="ltx_text" style="font-size:80%;">22:</span></span><span id="algx1.l22a.2" class="ltx_text" style="font-size:80%;">  </span><math id="algx1.l22a.m1.2" class="ltx_Math" alttext="\mathbf{O}_{i},lse_{i}" display="inline"><semantics id="algx1.l22a.m1.2a"><mrow id="algx1.l22a.m1.2.2.2" xref="algx1.l22a.m1.2.2.3.cmml"><msub id="algx1.l22a.m1.1.1.1.1" xref="algx1.l22a.m1.1.1.1.1.cmml"><mi mathsize="80%" id="algx1.l22a.m1.1.1.1.1.2" xref="algx1.l22a.m1.1.1.1.1.2.cmml">𝐎</mi><mi mathsize="80%" id="algx1.l22a.m1.1.1.1.1.3" xref="algx1.l22a.m1.1.1.1.1.3.cmml">i</mi></msub><mo mathsize="80%" id="algx1.l22a.m1.2.2.2.3" xref="algx1.l22a.m1.2.2.3.cmml">,</mo><mrow id="algx1.l22a.m1.2.2.2.2" xref="algx1.l22a.m1.2.2.2.2.cmml"><mi mathsize="80%" id="algx1.l22a.m1.2.2.2.2.2" xref="algx1.l22a.m1.2.2.2.2.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="algx1.l22a.m1.2.2.2.2.1" xref="algx1.l22a.m1.2.2.2.2.1.cmml">​</mo><mi mathsize="80%" id="algx1.l22a.m1.2.2.2.2.3" xref="algx1.l22a.m1.2.2.2.2.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="algx1.l22a.m1.2.2.2.2.1a" xref="algx1.l22a.m1.2.2.2.2.1.cmml">​</mo><msub id="algx1.l22a.m1.2.2.2.2.4" xref="algx1.l22a.m1.2.2.2.2.4.cmml"><mi mathsize="80%" id="algx1.l22a.m1.2.2.2.2.4.2" xref="algx1.l22a.m1.2.2.2.2.4.2.cmml">e</mi><mi mathsize="80%" id="algx1.l22a.m1.2.2.2.2.4.3" xref="algx1.l22a.m1.2.2.2.2.4.3.cmml">i</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="algx1.l22a.m1.2b"><list id="algx1.l22a.m1.2.2.3.cmml" xref="algx1.l22a.m1.2.2.2"><apply id="algx1.l22a.m1.1.1.1.1.cmml" xref="algx1.l22a.m1.1.1.1.1"><csymbol cd="ambiguous" id="algx1.l22a.m1.1.1.1.1.1.cmml" xref="algx1.l22a.m1.1.1.1.1">subscript</csymbol><ci id="algx1.l22a.m1.1.1.1.1.2.cmml" xref="algx1.l22a.m1.1.1.1.1.2">𝐎</ci><ci id="algx1.l22a.m1.1.1.1.1.3.cmml" xref="algx1.l22a.m1.1.1.1.1.3">𝑖</ci></apply><apply id="algx1.l22a.m1.2.2.2.2.cmml" xref="algx1.l22a.m1.2.2.2.2"><times id="algx1.l22a.m1.2.2.2.2.1.cmml" xref="algx1.l22a.m1.2.2.2.2.1"></times><ci id="algx1.l22a.m1.2.2.2.2.2.cmml" xref="algx1.l22a.m1.2.2.2.2.2">𝑙</ci><ci id="algx1.l22a.m1.2.2.2.2.3.cmml" xref="algx1.l22a.m1.2.2.2.2.3">𝑠</ci><apply id="algx1.l22a.m1.2.2.2.2.4.cmml" xref="algx1.l22a.m1.2.2.2.2.4"><csymbol cd="ambiguous" id="algx1.l22a.m1.2.2.2.2.4.1.cmml" xref="algx1.l22a.m1.2.2.2.2.4">subscript</csymbol><ci id="algx1.l22a.m1.2.2.2.2.4.2.cmml" xref="algx1.l22a.m1.2.2.2.2.4.2">𝑒</ci><ci id="algx1.l22a.m1.2.2.2.2.4.3.cmml" xref="algx1.l22a.m1.2.2.2.2.4.3">𝑖</ci></apply></apply></list></annotation-xml><annotation encoding="application/x-tex" id="algx1.l22a.m1.2c">\mathbf{O}_{i},lse_{i}</annotation></semantics></math><span id="algx1.l22a.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
</div>
</div>
</div>
</figure>
<figure id="alg4" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<div id="alg4.2" class="ltx_listing ltx_listing">
<div id="alg4.l0" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l0.1.1.1" class="ltx_text" style="font-size:80%;">0:</span></span><span id="alg4.l0.2" class="ltx_text" style="font-size:80%;">  Matrices </span><math id="alg4.l0.m1.5" class="ltx_Math" alttext="\mathbf{Q}_{i},\mathbf{K}_{i},\mathbf{V}_{i},\mathbf{O}_{i},\mathbf{dO}_{i}\in\mathbb{R}^{{\frac{N}{G}}\times d}" display="inline"><semantics id="alg4.l0.m1.5a"><mrow id="alg4.l0.m1.5.5" xref="alg4.l0.m1.5.5.cmml"><mrow id="alg4.l0.m1.5.5.5.5" xref="alg4.l0.m1.5.5.5.6.cmml"><msub id="alg4.l0.m1.1.1.1.1.1" xref="alg4.l0.m1.1.1.1.1.1.cmml"><mi mathsize="80%" id="alg4.l0.m1.1.1.1.1.1.2" xref="alg4.l0.m1.1.1.1.1.1.2.cmml">𝐐</mi><mi mathsize="80%" id="alg4.l0.m1.1.1.1.1.1.3" xref="alg4.l0.m1.1.1.1.1.1.3.cmml">i</mi></msub><mo mathsize="80%" id="alg4.l0.m1.5.5.5.5.6" xref="alg4.l0.m1.5.5.5.6.cmml">,</mo><msub id="alg4.l0.m1.2.2.2.2.2" xref="alg4.l0.m1.2.2.2.2.2.cmml"><mi mathsize="80%" id="alg4.l0.m1.2.2.2.2.2.2" xref="alg4.l0.m1.2.2.2.2.2.2.cmml">𝐊</mi><mi mathsize="80%" id="alg4.l0.m1.2.2.2.2.2.3" xref="alg4.l0.m1.2.2.2.2.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg4.l0.m1.5.5.5.5.7" xref="alg4.l0.m1.5.5.5.6.cmml">,</mo><msub id="alg4.l0.m1.3.3.3.3.3" xref="alg4.l0.m1.3.3.3.3.3.cmml"><mi mathsize="80%" id="alg4.l0.m1.3.3.3.3.3.2" xref="alg4.l0.m1.3.3.3.3.3.2.cmml">𝐕</mi><mi mathsize="80%" id="alg4.l0.m1.3.3.3.3.3.3" xref="alg4.l0.m1.3.3.3.3.3.3.cmml">i</mi></msub><mo mathsize="80%" id="alg4.l0.m1.5.5.5.5.8" xref="alg4.l0.m1.5.5.5.6.cmml">,</mo><msub id="alg4.l0.m1.4.4.4.4.4" xref="alg4.l0.m1.4.4.4.4.4.cmml"><mi mathsize="80%" id="alg4.l0.m1.4.4.4.4.4.2" xref="alg4.l0.m1.4.4.4.4.4.2.cmml">𝐎</mi><mi mathsize="80%" id="alg4.l0.m1.4.4.4.4.4.3" xref="alg4.l0.m1.4.4.4.4.4.3.cmml">i</mi></msub><mo mathsize="80%" id="alg4.l0.m1.5.5.5.5.9" xref="alg4.l0.m1.5.5.5.6.cmml">,</mo><msub id="alg4.l0.m1.5.5.5.5.5" xref="alg4.l0.m1.5.5.5.5.5.cmml"><mi mathsize="80%" id="alg4.l0.m1.5.5.5.5.5.2" xref="alg4.l0.m1.5.5.5.5.5.2.cmml">𝐝𝐎</mi><mi mathsize="80%" id="alg4.l0.m1.5.5.5.5.5.3" xref="alg4.l0.m1.5.5.5.5.5.3.cmml">i</mi></msub></mrow><mo mathsize="80%" id="alg4.l0.m1.5.5.6" xref="alg4.l0.m1.5.5.6.cmml">∈</mo><msup id="alg4.l0.m1.5.5.7" xref="alg4.l0.m1.5.5.7.cmml"><mi mathsize="80%" id="alg4.l0.m1.5.5.7.2" xref="alg4.l0.m1.5.5.7.2.cmml">ℝ</mi><mrow id="alg4.l0.m1.5.5.7.3" xref="alg4.l0.m1.5.5.7.3.cmml"><mfrac id="alg4.l0.m1.5.5.7.3.2" xref="alg4.l0.m1.5.5.7.3.2.cmml"><mi mathsize="80%" id="alg4.l0.m1.5.5.7.3.2.2" xref="alg4.l0.m1.5.5.7.3.2.2.cmml">N</mi><mi mathsize="80%" id="alg4.l0.m1.5.5.7.3.2.3" xref="alg4.l0.m1.5.5.7.3.2.3.cmml">G</mi></mfrac><mo lspace="0.222em" mathsize="80%" rspace="0.222em" id="alg4.l0.m1.5.5.7.3.1" xref="alg4.l0.m1.5.5.7.3.1.cmml">×</mo><mi mathsize="80%" id="alg4.l0.m1.5.5.7.3.3" xref="alg4.l0.m1.5.5.7.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="alg4.l0.m1.5b"><apply id="alg4.l0.m1.5.5.cmml" xref="alg4.l0.m1.5.5"><in id="alg4.l0.m1.5.5.6.cmml" xref="alg4.l0.m1.5.5.6"></in><list id="alg4.l0.m1.5.5.5.6.cmml" xref="alg4.l0.m1.5.5.5.5"><apply id="alg4.l0.m1.1.1.1.1.1.cmml" xref="alg4.l0.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg4.l0.m1.1.1.1.1.1.1.cmml" xref="alg4.l0.m1.1.1.1.1.1">subscript</csymbol><ci id="alg4.l0.m1.1.1.1.1.1.2.cmml" xref="alg4.l0.m1.1.1.1.1.1.2">𝐐</ci><ci id="alg4.l0.m1.1.1.1.1.1.3.cmml" xref="alg4.l0.m1.1.1.1.1.1.3">𝑖</ci></apply><apply id="alg4.l0.m1.2.2.2.2.2.cmml" xref="alg4.l0.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="alg4.l0.m1.2.2.2.2.2.1.cmml" xref="alg4.l0.m1.2.2.2.2.2">subscript</csymbol><ci id="alg4.l0.m1.2.2.2.2.2.2.cmml" xref="alg4.l0.m1.2.2.2.2.2.2">𝐊</ci><ci id="alg4.l0.m1.2.2.2.2.2.3.cmml" xref="alg4.l0.m1.2.2.2.2.2.3">𝑖</ci></apply><apply id="alg4.l0.m1.3.3.3.3.3.cmml" xref="alg4.l0.m1.3.3.3.3.3"><csymbol cd="ambiguous" id="alg4.l0.m1.3.3.3.3.3.1.cmml" xref="alg4.l0.m1.3.3.3.3.3">subscript</csymbol><ci id="alg4.l0.m1.3.3.3.3.3.2.cmml" xref="alg4.l0.m1.3.3.3.3.3.2">𝐕</ci><ci id="alg4.l0.m1.3.3.3.3.3.3.cmml" xref="alg4.l0.m1.3.3.3.3.3.3">𝑖</ci></apply><apply id="alg4.l0.m1.4.4.4.4.4.cmml" xref="alg4.l0.m1.4.4.4.4.4"><csymbol cd="ambiguous" id="alg4.l0.m1.4.4.4.4.4.1.cmml" xref="alg4.l0.m1.4.4.4.4.4">subscript</csymbol><ci id="alg4.l0.m1.4.4.4.4.4.2.cmml" xref="alg4.l0.m1.4.4.4.4.4.2">𝐎</ci><ci id="alg4.l0.m1.4.4.4.4.4.3.cmml" xref="alg4.l0.m1.4.4.4.4.4.3">𝑖</ci></apply><apply id="alg4.l0.m1.5.5.5.5.5.cmml" xref="alg4.l0.m1.5.5.5.5.5"><csymbol cd="ambiguous" id="alg4.l0.m1.5.5.5.5.5.1.cmml" xref="alg4.l0.m1.5.5.5.5.5">subscript</csymbol><ci id="alg4.l0.m1.5.5.5.5.5.2.cmml" xref="alg4.l0.m1.5.5.5.5.5.2">𝐝𝐎</ci><ci id="alg4.l0.m1.5.5.5.5.5.3.cmml" xref="alg4.l0.m1.5.5.5.5.5.3">𝑖</ci></apply></list><apply id="alg4.l0.m1.5.5.7.cmml" xref="alg4.l0.m1.5.5.7"><csymbol cd="ambiguous" id="alg4.l0.m1.5.5.7.1.cmml" xref="alg4.l0.m1.5.5.7">superscript</csymbol><ci id="alg4.l0.m1.5.5.7.2.cmml" xref="alg4.l0.m1.5.5.7.2">ℝ</ci><apply id="alg4.l0.m1.5.5.7.3.cmml" xref="alg4.l0.m1.5.5.7.3"><times id="alg4.l0.m1.5.5.7.3.1.cmml" xref="alg4.l0.m1.5.5.7.3.1"></times><apply id="alg4.l0.m1.5.5.7.3.2.cmml" xref="alg4.l0.m1.5.5.7.3.2"><divide id="alg4.l0.m1.5.5.7.3.2.1.cmml" xref="alg4.l0.m1.5.5.7.3.2"></divide><ci id="alg4.l0.m1.5.5.7.3.2.2.cmml" xref="alg4.l0.m1.5.5.7.3.2.2">𝑁</ci><ci id="alg4.l0.m1.5.5.7.3.2.3.cmml" xref="alg4.l0.m1.5.5.7.3.2.3">𝐺</ci></apply><ci id="alg4.l0.m1.5.5.7.3.3.cmml" xref="alg4.l0.m1.5.5.7.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l0.m1.5c">\mathbf{Q}_{i},\mathbf{K}_{i},\mathbf{V}_{i},\mathbf{O}_{i},\mathbf{dO}_{i}\in\mathbb{R}^{{\frac{N}{G}}\times d}</annotation></semantics></math><span id="alg4.l0.3" class="ltx_text" style="font-size:80%;">, </span><math id="alg4.l0.m2.1" class="ltx_Math" alttext="lse_{i}\in\mathbb{R}^{N}" display="inline"><semantics id="alg4.l0.m2.1a"><mrow id="alg4.l0.m2.1.1" xref="alg4.l0.m2.1.1.cmml"><mrow id="alg4.l0.m2.1.1.2" xref="alg4.l0.m2.1.1.2.cmml"><mi mathsize="80%" id="alg4.l0.m2.1.1.2.2" xref="alg4.l0.m2.1.1.2.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg4.l0.m2.1.1.2.1" xref="alg4.l0.m2.1.1.2.1.cmml">​</mo><mi mathsize="80%" id="alg4.l0.m2.1.1.2.3" xref="alg4.l0.m2.1.1.2.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg4.l0.m2.1.1.2.1a" xref="alg4.l0.m2.1.1.2.1.cmml">​</mo><msub id="alg4.l0.m2.1.1.2.4" xref="alg4.l0.m2.1.1.2.4.cmml"><mi mathsize="80%" id="alg4.l0.m2.1.1.2.4.2" xref="alg4.l0.m2.1.1.2.4.2.cmml">e</mi><mi mathsize="80%" id="alg4.l0.m2.1.1.2.4.3" xref="alg4.l0.m2.1.1.2.4.3.cmml">i</mi></msub></mrow><mo mathsize="80%" id="alg4.l0.m2.1.1.1" xref="alg4.l0.m2.1.1.1.cmml">∈</mo><msup id="alg4.l0.m2.1.1.3" xref="alg4.l0.m2.1.1.3.cmml"><mi mathsize="80%" id="alg4.l0.m2.1.1.3.2" xref="alg4.l0.m2.1.1.3.2.cmml">ℝ</mi><mi mathsize="80%" id="alg4.l0.m2.1.1.3.3" xref="alg4.l0.m2.1.1.3.3.cmml">N</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="alg4.l0.m2.1b"><apply id="alg4.l0.m2.1.1.cmml" xref="alg4.l0.m2.1.1"><in id="alg4.l0.m2.1.1.1.cmml" xref="alg4.l0.m2.1.1.1"></in><apply id="alg4.l0.m2.1.1.2.cmml" xref="alg4.l0.m2.1.1.2"><times id="alg4.l0.m2.1.1.2.1.cmml" xref="alg4.l0.m2.1.1.2.1"></times><ci id="alg4.l0.m2.1.1.2.2.cmml" xref="alg4.l0.m2.1.1.2.2">𝑙</ci><ci id="alg4.l0.m2.1.1.2.3.cmml" xref="alg4.l0.m2.1.1.2.3">𝑠</ci><apply id="alg4.l0.m2.1.1.2.4.cmml" xref="alg4.l0.m2.1.1.2.4"><csymbol cd="ambiguous" id="alg4.l0.m2.1.1.2.4.1.cmml" xref="alg4.l0.m2.1.1.2.4">subscript</csymbol><ci id="alg4.l0.m2.1.1.2.4.2.cmml" xref="alg4.l0.m2.1.1.2.4.2">𝑒</ci><ci id="alg4.l0.m2.1.1.2.4.3.cmml" xref="alg4.l0.m2.1.1.2.4.3">𝑖</ci></apply></apply><apply id="alg4.l0.m2.1.1.3.cmml" xref="alg4.l0.m2.1.1.3"><csymbol cd="ambiguous" id="alg4.l0.m2.1.1.3.1.cmml" xref="alg4.l0.m2.1.1.3">superscript</csymbol><ci id="alg4.l0.m2.1.1.3.2.cmml" xref="alg4.l0.m2.1.1.3.2">ℝ</ci><ci id="alg4.l0.m2.1.1.3.3.cmml" xref="alg4.l0.m2.1.1.3.3">𝑁</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l0.m2.1c">lse_{i}\in\mathbb{R}^{N}</annotation></semantics></math><span id="alg4.l0.4" class="ltx_text" style="font-size:80%;"> on the </span><math id="alg4.l0.m3.1" class="ltx_Math" alttext="i" display="inline"><semantics id="alg4.l0.m3.1a"><mi mathsize="80%" id="alg4.l0.m3.1.1" xref="alg4.l0.m3.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="alg4.l0.m3.1b"><ci id="alg4.l0.m3.1.1.cmml" xref="alg4.l0.m3.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="alg4.l0.m3.1c">i</annotation></semantics></math><span id="alg4.l0.5" class="ltx_text" style="font-size:80%;">-th device;
</span>
</div>
<div id="alg4.l1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l1.1.1.1" class="ltx_text" style="font-size:80%;">1:</span></span><span id="alg4.l1.2" class="ltx_text" style="font-size:80%;">  Initialize </span><math id="alg4.l1.m1.4" class="ltx_Math" alttext="\mathbf{dQ}_{i},\mathbf{dK_{i}},\mathbf{dV_{i}}=(0)_{{\frac{N}{G}}\times d}\in\mathbb{R}^{{\frac{N}{G}}\times d}" display="inline"><semantics id="alg4.l1.m1.4a"><mrow id="alg4.l1.m1.4.4" xref="alg4.l1.m1.4.4.cmml"><mrow id="alg4.l1.m1.4.4.3.3" xref="alg4.l1.m1.4.4.3.4.cmml"><msub id="alg4.l1.m1.2.2.1.1.1" xref="alg4.l1.m1.2.2.1.1.1.cmml"><mi mathsize="80%" id="alg4.l1.m1.2.2.1.1.1.2" xref="alg4.l1.m1.2.2.1.1.1.2.cmml">𝐝𝐐</mi><mi mathsize="80%" id="alg4.l1.m1.2.2.1.1.1.3" xref="alg4.l1.m1.2.2.1.1.1.3.cmml">i</mi></msub><mo mathsize="80%" id="alg4.l1.m1.4.4.3.3.4" xref="alg4.l1.m1.4.4.3.4.cmml">,</mo><msub id="alg4.l1.m1.3.3.2.2.2" xref="alg4.l1.m1.3.3.2.2.2.cmml"><mi mathsize="80%" id="alg4.l1.m1.3.3.2.2.2.2" xref="alg4.l1.m1.3.3.2.2.2.2.cmml">𝐝𝐊</mi><mi mathsize="80%" id="alg4.l1.m1.3.3.2.2.2.3" xref="alg4.l1.m1.3.3.2.2.2.3.cmml">𝐢</mi></msub><mo mathsize="80%" id="alg4.l1.m1.4.4.3.3.5" xref="alg4.l1.m1.4.4.3.4.cmml">,</mo><msub id="alg4.l1.m1.4.4.3.3.3" xref="alg4.l1.m1.4.4.3.3.3.cmml"><mi mathsize="80%" id="alg4.l1.m1.4.4.3.3.3.2" xref="alg4.l1.m1.4.4.3.3.3.2.cmml">𝐝𝐕</mi><mi mathsize="80%" id="alg4.l1.m1.4.4.3.3.3.3" xref="alg4.l1.m1.4.4.3.3.3.3.cmml">𝐢</mi></msub></mrow><mo mathsize="80%" id="alg4.l1.m1.4.4.5" xref="alg4.l1.m1.4.4.5.cmml">=</mo><msub id="alg4.l1.m1.4.4.6" xref="alg4.l1.m1.4.4.6.cmml"><mrow id="alg4.l1.m1.4.4.6.2.2" xref="alg4.l1.m1.4.4.6.cmml"><mo maxsize="80%" minsize="80%" id="alg4.l1.m1.4.4.6.2.2.1" xref="alg4.l1.m1.4.4.6.cmml">(</mo><mn mathsize="80%" id="alg4.l1.m1.1.1" xref="alg4.l1.m1.1.1.cmml">0</mn><mo maxsize="80%" minsize="80%" id="alg4.l1.m1.4.4.6.2.2.2" xref="alg4.l1.m1.4.4.6.cmml">)</mo></mrow><mrow id="alg4.l1.m1.4.4.6.3" xref="alg4.l1.m1.4.4.6.3.cmml"><mfrac id="alg4.l1.m1.4.4.6.3.2" xref="alg4.l1.m1.4.4.6.3.2.cmml"><mi mathsize="80%" id="alg4.l1.m1.4.4.6.3.2.2" xref="alg4.l1.m1.4.4.6.3.2.2.cmml">N</mi><mi mathsize="80%" id="alg4.l1.m1.4.4.6.3.2.3" xref="alg4.l1.m1.4.4.6.3.2.3.cmml">G</mi></mfrac><mo lspace="0.222em" mathsize="80%" rspace="0.222em" id="alg4.l1.m1.4.4.6.3.1" xref="alg4.l1.m1.4.4.6.3.1.cmml">×</mo><mi mathsize="80%" id="alg4.l1.m1.4.4.6.3.3" xref="alg4.l1.m1.4.4.6.3.3.cmml">d</mi></mrow></msub><mo mathsize="80%" id="alg4.l1.m1.4.4.7" xref="alg4.l1.m1.4.4.7.cmml">∈</mo><msup id="alg4.l1.m1.4.4.8" xref="alg4.l1.m1.4.4.8.cmml"><mi mathsize="80%" id="alg4.l1.m1.4.4.8.2" xref="alg4.l1.m1.4.4.8.2.cmml">ℝ</mi><mrow id="alg4.l1.m1.4.4.8.3" xref="alg4.l1.m1.4.4.8.3.cmml"><mfrac id="alg4.l1.m1.4.4.8.3.2" xref="alg4.l1.m1.4.4.8.3.2.cmml"><mi mathsize="80%" id="alg4.l1.m1.4.4.8.3.2.2" xref="alg4.l1.m1.4.4.8.3.2.2.cmml">N</mi><mi mathsize="80%" id="alg4.l1.m1.4.4.8.3.2.3" xref="alg4.l1.m1.4.4.8.3.2.3.cmml">G</mi></mfrac><mo lspace="0.222em" mathsize="80%" rspace="0.222em" id="alg4.l1.m1.4.4.8.3.1" xref="alg4.l1.m1.4.4.8.3.1.cmml">×</mo><mi mathsize="80%" id="alg4.l1.m1.4.4.8.3.3" xref="alg4.l1.m1.4.4.8.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="alg4.l1.m1.4b"><apply id="alg4.l1.m1.4.4.cmml" xref="alg4.l1.m1.4.4"><and id="alg4.l1.m1.4.4a.cmml" xref="alg4.l1.m1.4.4"></and><apply id="alg4.l1.m1.4.4b.cmml" xref="alg4.l1.m1.4.4"><eq id="alg4.l1.m1.4.4.5.cmml" xref="alg4.l1.m1.4.4.5"></eq><list id="alg4.l1.m1.4.4.3.4.cmml" xref="alg4.l1.m1.4.4.3.3"><apply id="alg4.l1.m1.2.2.1.1.1.cmml" xref="alg4.l1.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="alg4.l1.m1.2.2.1.1.1.1.cmml" xref="alg4.l1.m1.2.2.1.1.1">subscript</csymbol><ci id="alg4.l1.m1.2.2.1.1.1.2.cmml" xref="alg4.l1.m1.2.2.1.1.1.2">𝐝𝐐</ci><ci id="alg4.l1.m1.2.2.1.1.1.3.cmml" xref="alg4.l1.m1.2.2.1.1.1.3">𝑖</ci></apply><apply id="alg4.l1.m1.3.3.2.2.2.cmml" xref="alg4.l1.m1.3.3.2.2.2"><csymbol cd="ambiguous" id="alg4.l1.m1.3.3.2.2.2.1.cmml" xref="alg4.l1.m1.3.3.2.2.2">subscript</csymbol><ci id="alg4.l1.m1.3.3.2.2.2.2.cmml" xref="alg4.l1.m1.3.3.2.2.2.2">𝐝𝐊</ci><ci id="alg4.l1.m1.3.3.2.2.2.3.cmml" xref="alg4.l1.m1.3.3.2.2.2.3">𝐢</ci></apply><apply id="alg4.l1.m1.4.4.3.3.3.cmml" xref="alg4.l1.m1.4.4.3.3.3"><csymbol cd="ambiguous" id="alg4.l1.m1.4.4.3.3.3.1.cmml" xref="alg4.l1.m1.4.4.3.3.3">subscript</csymbol><ci id="alg4.l1.m1.4.4.3.3.3.2.cmml" xref="alg4.l1.m1.4.4.3.3.3.2">𝐝𝐕</ci><ci id="alg4.l1.m1.4.4.3.3.3.3.cmml" xref="alg4.l1.m1.4.4.3.3.3.3">𝐢</ci></apply></list><apply id="alg4.l1.m1.4.4.6.cmml" xref="alg4.l1.m1.4.4.6"><csymbol cd="ambiguous" id="alg4.l1.m1.4.4.6.1.cmml" xref="alg4.l1.m1.4.4.6">subscript</csymbol><cn type="integer" id="alg4.l1.m1.1.1.cmml" xref="alg4.l1.m1.1.1">0</cn><apply id="alg4.l1.m1.4.4.6.3.cmml" xref="alg4.l1.m1.4.4.6.3"><times id="alg4.l1.m1.4.4.6.3.1.cmml" xref="alg4.l1.m1.4.4.6.3.1"></times><apply id="alg4.l1.m1.4.4.6.3.2.cmml" xref="alg4.l1.m1.4.4.6.3.2"><divide id="alg4.l1.m1.4.4.6.3.2.1.cmml" xref="alg4.l1.m1.4.4.6.3.2"></divide><ci id="alg4.l1.m1.4.4.6.3.2.2.cmml" xref="alg4.l1.m1.4.4.6.3.2.2">𝑁</ci><ci id="alg4.l1.m1.4.4.6.3.2.3.cmml" xref="alg4.l1.m1.4.4.6.3.2.3">𝐺</ci></apply><ci id="alg4.l1.m1.4.4.6.3.3.cmml" xref="alg4.l1.m1.4.4.6.3.3">𝑑</ci></apply></apply></apply><apply id="alg4.l1.m1.4.4c.cmml" xref="alg4.l1.m1.4.4"><in id="alg4.l1.m1.4.4.7.cmml" xref="alg4.l1.m1.4.4.7"></in><share href="#alg4.l1.m1.4.4.6.cmml" id="alg4.l1.m1.4.4d.cmml" xref="alg4.l1.m1.4.4"></share><apply id="alg4.l1.m1.4.4.8.cmml" xref="alg4.l1.m1.4.4.8"><csymbol cd="ambiguous" id="alg4.l1.m1.4.4.8.1.cmml" xref="alg4.l1.m1.4.4.8">superscript</csymbol><ci id="alg4.l1.m1.4.4.8.2.cmml" xref="alg4.l1.m1.4.4.8.2">ℝ</ci><apply id="alg4.l1.m1.4.4.8.3.cmml" xref="alg4.l1.m1.4.4.8.3"><times id="alg4.l1.m1.4.4.8.3.1.cmml" xref="alg4.l1.m1.4.4.8.3.1"></times><apply id="alg4.l1.m1.4.4.8.3.2.cmml" xref="alg4.l1.m1.4.4.8.3.2"><divide id="alg4.l1.m1.4.4.8.3.2.1.cmml" xref="alg4.l1.m1.4.4.8.3.2"></divide><ci id="alg4.l1.m1.4.4.8.3.2.2.cmml" xref="alg4.l1.m1.4.4.8.3.2.2">𝑁</ci><ci id="alg4.l1.m1.4.4.8.3.2.3.cmml" xref="alg4.l1.m1.4.4.8.3.2.3">𝐺</ci></apply><ci id="alg4.l1.m1.4.4.8.3.3.cmml" xref="alg4.l1.m1.4.4.8.3.3">𝑑</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l1.m1.4c">\mathbf{dQ}_{i},\mathbf{dK_{i}},\mathbf{dV_{i}}=(0)_{{\frac{N}{G}}\times d}\in\mathbb{R}^{{\frac{N}{G}}\times d}</annotation></semantics></math><span id="alg4.l1.3" class="ltx_text" style="font-size:80%;"> 
</span>
</div>
<div id="alg4.l2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l2.1.1.1" class="ltx_text" style="font-size:80%;">2:</span></span><span id="alg4.l2.2" class="ltx_text" style="font-size:80%;">  </span><math id="alg4.l2.m1.1" class="ltx_Math" alttext="D_{i}=\text{rowsum}(\mathbf{dO}_{i}\circ\mathbf{O}_{i})" display="inline"><semantics id="alg4.l2.m1.1a"><mrow id="alg4.l2.m1.1.1" xref="alg4.l2.m1.1.1.cmml"><msub id="alg4.l2.m1.1.1.3" xref="alg4.l2.m1.1.1.3.cmml"><mi mathsize="80%" id="alg4.l2.m1.1.1.3.2" xref="alg4.l2.m1.1.1.3.2.cmml">D</mi><mi mathsize="80%" id="alg4.l2.m1.1.1.3.3" xref="alg4.l2.m1.1.1.3.3.cmml">i</mi></msub><mo mathsize="80%" id="alg4.l2.m1.1.1.2" xref="alg4.l2.m1.1.1.2.cmml">=</mo><mrow id="alg4.l2.m1.1.1.1" xref="alg4.l2.m1.1.1.1.cmml"><mtext mathsize="80%" id="alg4.l2.m1.1.1.1.3" xref="alg4.l2.m1.1.1.1.3a.cmml">rowsum</mtext><mo lspace="0em" rspace="0em" id="alg4.l2.m1.1.1.1.2" xref="alg4.l2.m1.1.1.1.2.cmml">​</mo><mrow id="alg4.l2.m1.1.1.1.1.1" xref="alg4.l2.m1.1.1.1.1.1.1.cmml"><mo maxsize="80%" minsize="80%" id="alg4.l2.m1.1.1.1.1.1.2" xref="alg4.l2.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="alg4.l2.m1.1.1.1.1.1.1" xref="alg4.l2.m1.1.1.1.1.1.1.cmml"><msub id="alg4.l2.m1.1.1.1.1.1.1.2" xref="alg4.l2.m1.1.1.1.1.1.1.2.cmml"><mi mathsize="80%" id="alg4.l2.m1.1.1.1.1.1.1.2.2" xref="alg4.l2.m1.1.1.1.1.1.1.2.2.cmml">𝐝𝐎</mi><mi mathsize="80%" id="alg4.l2.m1.1.1.1.1.1.1.2.3" xref="alg4.l2.m1.1.1.1.1.1.1.2.3.cmml">i</mi></msub><mo lspace="0.222em" mathsize="80%" rspace="0.222em" id="alg4.l2.m1.1.1.1.1.1.1.1" xref="alg4.l2.m1.1.1.1.1.1.1.1.cmml">∘</mo><msub id="alg4.l2.m1.1.1.1.1.1.1.3" xref="alg4.l2.m1.1.1.1.1.1.1.3.cmml"><mi mathsize="80%" id="alg4.l2.m1.1.1.1.1.1.1.3.2" xref="alg4.l2.m1.1.1.1.1.1.1.3.2.cmml">𝐎</mi><mi mathsize="80%" id="alg4.l2.m1.1.1.1.1.1.1.3.3" xref="alg4.l2.m1.1.1.1.1.1.1.3.3.cmml">i</mi></msub></mrow><mo maxsize="80%" minsize="80%" id="alg4.l2.m1.1.1.1.1.1.3" xref="alg4.l2.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l2.m1.1b"><apply id="alg4.l2.m1.1.1.cmml" xref="alg4.l2.m1.1.1"><eq id="alg4.l2.m1.1.1.2.cmml" xref="alg4.l2.m1.1.1.2"></eq><apply id="alg4.l2.m1.1.1.3.cmml" xref="alg4.l2.m1.1.1.3"><csymbol cd="ambiguous" id="alg4.l2.m1.1.1.3.1.cmml" xref="alg4.l2.m1.1.1.3">subscript</csymbol><ci id="alg4.l2.m1.1.1.3.2.cmml" xref="alg4.l2.m1.1.1.3.2">𝐷</ci><ci id="alg4.l2.m1.1.1.3.3.cmml" xref="alg4.l2.m1.1.1.3.3">𝑖</ci></apply><apply id="alg4.l2.m1.1.1.1.cmml" xref="alg4.l2.m1.1.1.1"><times id="alg4.l2.m1.1.1.1.2.cmml" xref="alg4.l2.m1.1.1.1.2"></times><ci id="alg4.l2.m1.1.1.1.3a.cmml" xref="alg4.l2.m1.1.1.1.3"><mtext mathsize="80%" id="alg4.l2.m1.1.1.1.3.cmml" xref="alg4.l2.m1.1.1.1.3">rowsum</mtext></ci><apply id="alg4.l2.m1.1.1.1.1.1.1.cmml" xref="alg4.l2.m1.1.1.1.1.1"><compose id="alg4.l2.m1.1.1.1.1.1.1.1.cmml" xref="alg4.l2.m1.1.1.1.1.1.1.1"></compose><apply id="alg4.l2.m1.1.1.1.1.1.1.2.cmml" xref="alg4.l2.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="alg4.l2.m1.1.1.1.1.1.1.2.1.cmml" xref="alg4.l2.m1.1.1.1.1.1.1.2">subscript</csymbol><ci id="alg4.l2.m1.1.1.1.1.1.1.2.2.cmml" xref="alg4.l2.m1.1.1.1.1.1.1.2.2">𝐝𝐎</ci><ci id="alg4.l2.m1.1.1.1.1.1.1.2.3.cmml" xref="alg4.l2.m1.1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="alg4.l2.m1.1.1.1.1.1.1.3.cmml" xref="alg4.l2.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="alg4.l2.m1.1.1.1.1.1.1.3.1.cmml" xref="alg4.l2.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="alg4.l2.m1.1.1.1.1.1.1.3.2.cmml" xref="alg4.l2.m1.1.1.1.1.1.1.3.2">𝐎</ci><ci id="alg4.l2.m1.1.1.1.1.1.1.3.3.cmml" xref="alg4.l2.m1.1.1.1.1.1.1.3.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l2.m1.1c">D_{i}=\text{rowsum}(\mathbf{dO}_{i}\circ\mathbf{O}_{i})</annotation></semantics></math><span id="alg4.l2.3" class="ltx_text" style="font-size:80%;"> (pointwise multiply);
</span>
</div>
<div id="alg4.l3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l3.1.1.1" class="ltx_text" style="font-size:80%;">3:</span></span><span id="alg4.l3.2" class="ltx_text" style="font-size:80%;">  Initialize Buffer </span><math id="alg4.l3.m1.4" class="ltx_Math" alttext="\mathbf{Q}_{buf}\mathbf{dQ}_{buf},\mathbf{dO}_{buf},D_{buf},lse_{buf}" display="inline"><semantics id="alg4.l3.m1.4a"><mrow id="alg4.l3.m1.4.4.4" xref="alg4.l3.m1.4.4.5.cmml"><mrow id="alg4.l3.m1.1.1.1.1" xref="alg4.l3.m1.1.1.1.1.cmml"><msub id="alg4.l3.m1.1.1.1.1.2" xref="alg4.l3.m1.1.1.1.1.2.cmml"><mi mathsize="80%" id="alg4.l3.m1.1.1.1.1.2.2" xref="alg4.l3.m1.1.1.1.1.2.2.cmml">𝐐</mi><mrow id="alg4.l3.m1.1.1.1.1.2.3" xref="alg4.l3.m1.1.1.1.1.2.3.cmml"><mi mathsize="80%" id="alg4.l3.m1.1.1.1.1.2.3.2" xref="alg4.l3.m1.1.1.1.1.2.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg4.l3.m1.1.1.1.1.2.3.1" xref="alg4.l3.m1.1.1.1.1.2.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l3.m1.1.1.1.1.2.3.3" xref="alg4.l3.m1.1.1.1.1.2.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg4.l3.m1.1.1.1.1.2.3.1a" xref="alg4.l3.m1.1.1.1.1.2.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l3.m1.1.1.1.1.2.3.4" xref="alg4.l3.m1.1.1.1.1.2.3.4.cmml">f</mi></mrow></msub><mo lspace="0em" rspace="0em" id="alg4.l3.m1.1.1.1.1.1" xref="alg4.l3.m1.1.1.1.1.1.cmml">​</mo><msub id="alg4.l3.m1.1.1.1.1.3" xref="alg4.l3.m1.1.1.1.1.3.cmml"><mi mathsize="80%" id="alg4.l3.m1.1.1.1.1.3.2" xref="alg4.l3.m1.1.1.1.1.3.2.cmml">𝐝𝐐</mi><mrow id="alg4.l3.m1.1.1.1.1.3.3" xref="alg4.l3.m1.1.1.1.1.3.3.cmml"><mi mathsize="80%" id="alg4.l3.m1.1.1.1.1.3.3.2" xref="alg4.l3.m1.1.1.1.1.3.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg4.l3.m1.1.1.1.1.3.3.1" xref="alg4.l3.m1.1.1.1.1.3.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l3.m1.1.1.1.1.3.3.3" xref="alg4.l3.m1.1.1.1.1.3.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg4.l3.m1.1.1.1.1.3.3.1a" xref="alg4.l3.m1.1.1.1.1.3.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l3.m1.1.1.1.1.3.3.4" xref="alg4.l3.m1.1.1.1.1.3.3.4.cmml">f</mi></mrow></msub></mrow><mo mathsize="80%" id="alg4.l3.m1.4.4.4.5" xref="alg4.l3.m1.4.4.5.cmml">,</mo><msub id="alg4.l3.m1.2.2.2.2" xref="alg4.l3.m1.2.2.2.2.cmml"><mi mathsize="80%" id="alg4.l3.m1.2.2.2.2.2" xref="alg4.l3.m1.2.2.2.2.2.cmml">𝐝𝐎</mi><mrow id="alg4.l3.m1.2.2.2.2.3" xref="alg4.l3.m1.2.2.2.2.3.cmml"><mi mathsize="80%" id="alg4.l3.m1.2.2.2.2.3.2" xref="alg4.l3.m1.2.2.2.2.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg4.l3.m1.2.2.2.2.3.1" xref="alg4.l3.m1.2.2.2.2.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l3.m1.2.2.2.2.3.3" xref="alg4.l3.m1.2.2.2.2.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg4.l3.m1.2.2.2.2.3.1a" xref="alg4.l3.m1.2.2.2.2.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l3.m1.2.2.2.2.3.4" xref="alg4.l3.m1.2.2.2.2.3.4.cmml">f</mi></mrow></msub><mo mathsize="80%" id="alg4.l3.m1.4.4.4.6" xref="alg4.l3.m1.4.4.5.cmml">,</mo><msub id="alg4.l3.m1.3.3.3.3" xref="alg4.l3.m1.3.3.3.3.cmml"><mi mathsize="80%" id="alg4.l3.m1.3.3.3.3.2" xref="alg4.l3.m1.3.3.3.3.2.cmml">D</mi><mrow id="alg4.l3.m1.3.3.3.3.3" xref="alg4.l3.m1.3.3.3.3.3.cmml"><mi mathsize="80%" id="alg4.l3.m1.3.3.3.3.3.2" xref="alg4.l3.m1.3.3.3.3.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg4.l3.m1.3.3.3.3.3.1" xref="alg4.l3.m1.3.3.3.3.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l3.m1.3.3.3.3.3.3" xref="alg4.l3.m1.3.3.3.3.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg4.l3.m1.3.3.3.3.3.1a" xref="alg4.l3.m1.3.3.3.3.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l3.m1.3.3.3.3.3.4" xref="alg4.l3.m1.3.3.3.3.3.4.cmml">f</mi></mrow></msub><mo mathsize="80%" id="alg4.l3.m1.4.4.4.7" xref="alg4.l3.m1.4.4.5.cmml">,</mo><mrow id="alg4.l3.m1.4.4.4.4" xref="alg4.l3.m1.4.4.4.4.cmml"><mi mathsize="80%" id="alg4.l3.m1.4.4.4.4.2" xref="alg4.l3.m1.4.4.4.4.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg4.l3.m1.4.4.4.4.1" xref="alg4.l3.m1.4.4.4.4.1.cmml">​</mo><mi mathsize="80%" id="alg4.l3.m1.4.4.4.4.3" xref="alg4.l3.m1.4.4.4.4.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg4.l3.m1.4.4.4.4.1a" xref="alg4.l3.m1.4.4.4.4.1.cmml">​</mo><msub id="alg4.l3.m1.4.4.4.4.4" xref="alg4.l3.m1.4.4.4.4.4.cmml"><mi mathsize="80%" id="alg4.l3.m1.4.4.4.4.4.2" xref="alg4.l3.m1.4.4.4.4.4.2.cmml">e</mi><mrow id="alg4.l3.m1.4.4.4.4.4.3" xref="alg4.l3.m1.4.4.4.4.4.3.cmml"><mi mathsize="80%" id="alg4.l3.m1.4.4.4.4.4.3.2" xref="alg4.l3.m1.4.4.4.4.4.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg4.l3.m1.4.4.4.4.4.3.1" xref="alg4.l3.m1.4.4.4.4.4.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l3.m1.4.4.4.4.4.3.3" xref="alg4.l3.m1.4.4.4.4.4.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg4.l3.m1.4.4.4.4.4.3.1a" xref="alg4.l3.m1.4.4.4.4.4.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l3.m1.4.4.4.4.4.3.4" xref="alg4.l3.m1.4.4.4.4.4.3.4.cmml">f</mi></mrow></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l3.m1.4b"><list id="alg4.l3.m1.4.4.5.cmml" xref="alg4.l3.m1.4.4.4"><apply id="alg4.l3.m1.1.1.1.1.cmml" xref="alg4.l3.m1.1.1.1.1"><times id="alg4.l3.m1.1.1.1.1.1.cmml" xref="alg4.l3.m1.1.1.1.1.1"></times><apply id="alg4.l3.m1.1.1.1.1.2.cmml" xref="alg4.l3.m1.1.1.1.1.2"><csymbol cd="ambiguous" id="alg4.l3.m1.1.1.1.1.2.1.cmml" xref="alg4.l3.m1.1.1.1.1.2">subscript</csymbol><ci id="alg4.l3.m1.1.1.1.1.2.2.cmml" xref="alg4.l3.m1.1.1.1.1.2.2">𝐐</ci><apply id="alg4.l3.m1.1.1.1.1.2.3.cmml" xref="alg4.l3.m1.1.1.1.1.2.3"><times id="alg4.l3.m1.1.1.1.1.2.3.1.cmml" xref="alg4.l3.m1.1.1.1.1.2.3.1"></times><ci id="alg4.l3.m1.1.1.1.1.2.3.2.cmml" xref="alg4.l3.m1.1.1.1.1.2.3.2">𝑏</ci><ci id="alg4.l3.m1.1.1.1.1.2.3.3.cmml" xref="alg4.l3.m1.1.1.1.1.2.3.3">𝑢</ci><ci id="alg4.l3.m1.1.1.1.1.2.3.4.cmml" xref="alg4.l3.m1.1.1.1.1.2.3.4">𝑓</ci></apply></apply><apply id="alg4.l3.m1.1.1.1.1.3.cmml" xref="alg4.l3.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="alg4.l3.m1.1.1.1.1.3.1.cmml" xref="alg4.l3.m1.1.1.1.1.3">subscript</csymbol><ci id="alg4.l3.m1.1.1.1.1.3.2.cmml" xref="alg4.l3.m1.1.1.1.1.3.2">𝐝𝐐</ci><apply id="alg4.l3.m1.1.1.1.1.3.3.cmml" xref="alg4.l3.m1.1.1.1.1.3.3"><times id="alg4.l3.m1.1.1.1.1.3.3.1.cmml" xref="alg4.l3.m1.1.1.1.1.3.3.1"></times><ci id="alg4.l3.m1.1.1.1.1.3.3.2.cmml" xref="alg4.l3.m1.1.1.1.1.3.3.2">𝑏</ci><ci id="alg4.l3.m1.1.1.1.1.3.3.3.cmml" xref="alg4.l3.m1.1.1.1.1.3.3.3">𝑢</ci><ci id="alg4.l3.m1.1.1.1.1.3.3.4.cmml" xref="alg4.l3.m1.1.1.1.1.3.3.4">𝑓</ci></apply></apply></apply><apply id="alg4.l3.m1.2.2.2.2.cmml" xref="alg4.l3.m1.2.2.2.2"><csymbol cd="ambiguous" id="alg4.l3.m1.2.2.2.2.1.cmml" xref="alg4.l3.m1.2.2.2.2">subscript</csymbol><ci id="alg4.l3.m1.2.2.2.2.2.cmml" xref="alg4.l3.m1.2.2.2.2.2">𝐝𝐎</ci><apply id="alg4.l3.m1.2.2.2.2.3.cmml" xref="alg4.l3.m1.2.2.2.2.3"><times id="alg4.l3.m1.2.2.2.2.3.1.cmml" xref="alg4.l3.m1.2.2.2.2.3.1"></times><ci id="alg4.l3.m1.2.2.2.2.3.2.cmml" xref="alg4.l3.m1.2.2.2.2.3.2">𝑏</ci><ci id="alg4.l3.m1.2.2.2.2.3.3.cmml" xref="alg4.l3.m1.2.2.2.2.3.3">𝑢</ci><ci id="alg4.l3.m1.2.2.2.2.3.4.cmml" xref="alg4.l3.m1.2.2.2.2.3.4">𝑓</ci></apply></apply><apply id="alg4.l3.m1.3.3.3.3.cmml" xref="alg4.l3.m1.3.3.3.3"><csymbol cd="ambiguous" id="alg4.l3.m1.3.3.3.3.1.cmml" xref="alg4.l3.m1.3.3.3.3">subscript</csymbol><ci id="alg4.l3.m1.3.3.3.3.2.cmml" xref="alg4.l3.m1.3.3.3.3.2">𝐷</ci><apply id="alg4.l3.m1.3.3.3.3.3.cmml" xref="alg4.l3.m1.3.3.3.3.3"><times id="alg4.l3.m1.3.3.3.3.3.1.cmml" xref="alg4.l3.m1.3.3.3.3.3.1"></times><ci id="alg4.l3.m1.3.3.3.3.3.2.cmml" xref="alg4.l3.m1.3.3.3.3.3.2">𝑏</ci><ci id="alg4.l3.m1.3.3.3.3.3.3.cmml" xref="alg4.l3.m1.3.3.3.3.3.3">𝑢</ci><ci id="alg4.l3.m1.3.3.3.3.3.4.cmml" xref="alg4.l3.m1.3.3.3.3.3.4">𝑓</ci></apply></apply><apply id="alg4.l3.m1.4.4.4.4.cmml" xref="alg4.l3.m1.4.4.4.4"><times id="alg4.l3.m1.4.4.4.4.1.cmml" xref="alg4.l3.m1.4.4.4.4.1"></times><ci id="alg4.l3.m1.4.4.4.4.2.cmml" xref="alg4.l3.m1.4.4.4.4.2">𝑙</ci><ci id="alg4.l3.m1.4.4.4.4.3.cmml" xref="alg4.l3.m1.4.4.4.4.3">𝑠</ci><apply id="alg4.l3.m1.4.4.4.4.4.cmml" xref="alg4.l3.m1.4.4.4.4.4"><csymbol cd="ambiguous" id="alg4.l3.m1.4.4.4.4.4.1.cmml" xref="alg4.l3.m1.4.4.4.4.4">subscript</csymbol><ci id="alg4.l3.m1.4.4.4.4.4.2.cmml" xref="alg4.l3.m1.4.4.4.4.4.2">𝑒</ci><apply id="alg4.l3.m1.4.4.4.4.4.3.cmml" xref="alg4.l3.m1.4.4.4.4.4.3"><times id="alg4.l3.m1.4.4.4.4.4.3.1.cmml" xref="alg4.l3.m1.4.4.4.4.4.3.1"></times><ci id="alg4.l3.m1.4.4.4.4.4.3.2.cmml" xref="alg4.l3.m1.4.4.4.4.4.3.2">𝑏</ci><ci id="alg4.l3.m1.4.4.4.4.4.3.3.cmml" xref="alg4.l3.m1.4.4.4.4.4.3.3">𝑢</ci><ci id="alg4.l3.m1.4.4.4.4.4.3.4.cmml" xref="alg4.l3.m1.4.4.4.4.4.3.4">𝑓</ci></apply></apply></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg4.l3.m1.4c">\mathbf{Q}_{buf}\mathbf{dQ}_{buf},\mathbf{dO}_{buf},D_{buf},lse_{buf}</annotation></semantics></math><span id="alg4.l3.3" class="ltx_text" style="font-size:80%;"> from </span><math id="alg4.l3.m2.5" class="ltx_Math" alttext="\mathbf{Q}_{j},\mathbf{dQ}_{j},\mathbf{dO}_{j},D_{j},lse_{j}" display="inline"><semantics id="alg4.l3.m2.5a"><mrow id="alg4.l3.m2.5.5.5" xref="alg4.l3.m2.5.5.6.cmml"><msub id="alg4.l3.m2.1.1.1.1" xref="alg4.l3.m2.1.1.1.1.cmml"><mi mathsize="80%" id="alg4.l3.m2.1.1.1.1.2" xref="alg4.l3.m2.1.1.1.1.2.cmml">𝐐</mi><mi mathsize="80%" id="alg4.l3.m2.1.1.1.1.3" xref="alg4.l3.m2.1.1.1.1.3.cmml">j</mi></msub><mo mathsize="80%" id="alg4.l3.m2.5.5.5.6" xref="alg4.l3.m2.5.5.6.cmml">,</mo><msub id="alg4.l3.m2.2.2.2.2" xref="alg4.l3.m2.2.2.2.2.cmml"><mi mathsize="80%" id="alg4.l3.m2.2.2.2.2.2" xref="alg4.l3.m2.2.2.2.2.2.cmml">𝐝𝐐</mi><mi mathsize="80%" id="alg4.l3.m2.2.2.2.2.3" xref="alg4.l3.m2.2.2.2.2.3.cmml">j</mi></msub><mo mathsize="80%" id="alg4.l3.m2.5.5.5.7" xref="alg4.l3.m2.5.5.6.cmml">,</mo><msub id="alg4.l3.m2.3.3.3.3" xref="alg4.l3.m2.3.3.3.3.cmml"><mi mathsize="80%" id="alg4.l3.m2.3.3.3.3.2" xref="alg4.l3.m2.3.3.3.3.2.cmml">𝐝𝐎</mi><mi mathsize="80%" id="alg4.l3.m2.3.3.3.3.3" xref="alg4.l3.m2.3.3.3.3.3.cmml">j</mi></msub><mo mathsize="80%" id="alg4.l3.m2.5.5.5.8" xref="alg4.l3.m2.5.5.6.cmml">,</mo><msub id="alg4.l3.m2.4.4.4.4" xref="alg4.l3.m2.4.4.4.4.cmml"><mi mathsize="80%" id="alg4.l3.m2.4.4.4.4.2" xref="alg4.l3.m2.4.4.4.4.2.cmml">D</mi><mi mathsize="80%" id="alg4.l3.m2.4.4.4.4.3" xref="alg4.l3.m2.4.4.4.4.3.cmml">j</mi></msub><mo mathsize="80%" id="alg4.l3.m2.5.5.5.9" xref="alg4.l3.m2.5.5.6.cmml">,</mo><mrow id="alg4.l3.m2.5.5.5.5" xref="alg4.l3.m2.5.5.5.5.cmml"><mi mathsize="80%" id="alg4.l3.m2.5.5.5.5.2" xref="alg4.l3.m2.5.5.5.5.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg4.l3.m2.5.5.5.5.1" xref="alg4.l3.m2.5.5.5.5.1.cmml">​</mo><mi mathsize="80%" id="alg4.l3.m2.5.5.5.5.3" xref="alg4.l3.m2.5.5.5.5.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg4.l3.m2.5.5.5.5.1a" xref="alg4.l3.m2.5.5.5.5.1.cmml">​</mo><msub id="alg4.l3.m2.5.5.5.5.4" xref="alg4.l3.m2.5.5.5.5.4.cmml"><mi mathsize="80%" id="alg4.l3.m2.5.5.5.5.4.2" xref="alg4.l3.m2.5.5.5.5.4.2.cmml">e</mi><mi mathsize="80%" id="alg4.l3.m2.5.5.5.5.4.3" xref="alg4.l3.m2.5.5.5.5.4.3.cmml">j</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l3.m2.5b"><list id="alg4.l3.m2.5.5.6.cmml" xref="alg4.l3.m2.5.5.5"><apply id="alg4.l3.m2.1.1.1.1.cmml" xref="alg4.l3.m2.1.1.1.1"><csymbol cd="ambiguous" id="alg4.l3.m2.1.1.1.1.1.cmml" xref="alg4.l3.m2.1.1.1.1">subscript</csymbol><ci id="alg4.l3.m2.1.1.1.1.2.cmml" xref="alg4.l3.m2.1.1.1.1.2">𝐐</ci><ci id="alg4.l3.m2.1.1.1.1.3.cmml" xref="alg4.l3.m2.1.1.1.1.3">𝑗</ci></apply><apply id="alg4.l3.m2.2.2.2.2.cmml" xref="alg4.l3.m2.2.2.2.2"><csymbol cd="ambiguous" id="alg4.l3.m2.2.2.2.2.1.cmml" xref="alg4.l3.m2.2.2.2.2">subscript</csymbol><ci id="alg4.l3.m2.2.2.2.2.2.cmml" xref="alg4.l3.m2.2.2.2.2.2">𝐝𝐐</ci><ci id="alg4.l3.m2.2.2.2.2.3.cmml" xref="alg4.l3.m2.2.2.2.2.3">𝑗</ci></apply><apply id="alg4.l3.m2.3.3.3.3.cmml" xref="alg4.l3.m2.3.3.3.3"><csymbol cd="ambiguous" id="alg4.l3.m2.3.3.3.3.1.cmml" xref="alg4.l3.m2.3.3.3.3">subscript</csymbol><ci id="alg4.l3.m2.3.3.3.3.2.cmml" xref="alg4.l3.m2.3.3.3.3.2">𝐝𝐎</ci><ci id="alg4.l3.m2.3.3.3.3.3.cmml" xref="alg4.l3.m2.3.3.3.3.3">𝑗</ci></apply><apply id="alg4.l3.m2.4.4.4.4.cmml" xref="alg4.l3.m2.4.4.4.4"><csymbol cd="ambiguous" id="alg4.l3.m2.4.4.4.4.1.cmml" xref="alg4.l3.m2.4.4.4.4">subscript</csymbol><ci id="alg4.l3.m2.4.4.4.4.2.cmml" xref="alg4.l3.m2.4.4.4.4.2">𝐷</ci><ci id="alg4.l3.m2.4.4.4.4.3.cmml" xref="alg4.l3.m2.4.4.4.4.3">𝑗</ci></apply><apply id="alg4.l3.m2.5.5.5.5.cmml" xref="alg4.l3.m2.5.5.5.5"><times id="alg4.l3.m2.5.5.5.5.1.cmml" xref="alg4.l3.m2.5.5.5.5.1"></times><ci id="alg4.l3.m2.5.5.5.5.2.cmml" xref="alg4.l3.m2.5.5.5.5.2">𝑙</ci><ci id="alg4.l3.m2.5.5.5.5.3.cmml" xref="alg4.l3.m2.5.5.5.5.3">𝑠</ci><apply id="alg4.l3.m2.5.5.5.5.4.cmml" xref="alg4.l3.m2.5.5.5.5.4"><csymbol cd="ambiguous" id="alg4.l3.m2.5.5.5.5.4.1.cmml" xref="alg4.l3.m2.5.5.5.5.4">subscript</csymbol><ci id="alg4.l3.m2.5.5.5.5.4.2.cmml" xref="alg4.l3.m2.5.5.5.5.4.2">𝑒</ci><ci id="alg4.l3.m2.5.5.5.5.4.3.cmml" xref="alg4.l3.m2.5.5.5.5.4.3">𝑗</ci></apply></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg4.l3.m2.5c">\mathbf{Q}_{j},\mathbf{dQ}_{j},\mathbf{dO}_{j},D_{j},lse_{j}</annotation></semantics></math><span id="alg4.l3.4" class="ltx_text" style="font-size:80%;">

</span>
</div>
<div id="alg4.l4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l4.1.1.1" class="ltx_text" style="font-size:80%;">4:</span></span><span id="alg4.l4.2" class="ltx_text" style="font-size:80%;">  </span><span id="alg4.l4.3" class="ltx_text ltx_font_bold" style="font-size:80%;">for</span><span id="alg4.l4.4" class="ltx_text" style="font-size:80%;"> </span><math id="alg4.l4.m1.1" class="ltx_Math" alttext="j=1" display="inline"><semantics id="alg4.l4.m1.1a"><mrow id="alg4.l4.m1.1.1" xref="alg4.l4.m1.1.1.cmml"><mi mathsize="80%" id="alg4.l4.m1.1.1.2" xref="alg4.l4.m1.1.1.2.cmml">j</mi><mo mathsize="80%" id="alg4.l4.m1.1.1.1" xref="alg4.l4.m1.1.1.1.cmml">=</mo><mn mathsize="80%" id="alg4.l4.m1.1.1.3" xref="alg4.l4.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg4.l4.m1.1b"><apply id="alg4.l4.m1.1.1.cmml" xref="alg4.l4.m1.1.1"><eq id="alg4.l4.m1.1.1.1.cmml" xref="alg4.l4.m1.1.1.1"></eq><ci id="alg4.l4.m1.1.1.2.cmml" xref="alg4.l4.m1.1.1.2">𝑗</ci><cn type="integer" id="alg4.l4.m1.1.1.3.cmml" xref="alg4.l4.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l4.m1.1c">j=1</annotation></semantics></math><span id="alg4.l4.5" class="ltx_text" style="font-size:80%;"> to </span><math id="alg4.l4.m2.1" class="ltx_Math" alttext="G" display="inline"><semantics id="alg4.l4.m2.1a"><mi mathsize="80%" id="alg4.l4.m2.1.1" xref="alg4.l4.m2.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="alg4.l4.m2.1b"><ci id="alg4.l4.m2.1.1.cmml" xref="alg4.l4.m2.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="alg4.l4.m2.1c">G</annotation></semantics></math><span id="alg4.l4.6" class="ltx_text" style="font-size:80%;"> </span><span id="alg4.l4.7" class="ltx_text ltx_font_bold" style="font-size:80%;">do</span><span id="alg4.l4.8" class="ltx_text" style="font-size:80%;">


</span>
</div>
<div id="alg4.l5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l5.1.1.1" class="ltx_text" style="font-size:80%;">5:</span></span><span id="alg4.l5.2" class="ltx_text" style="font-size:80%;">    </span><span id="alg4.l5.3" class="ltx_text ltx_font_bold" style="font-size:80%;">if</span><span id="alg4.l5.4" class="ltx_text" style="font-size:80%;"> j!=1 </span><span id="alg4.l5.5" class="ltx_text ltx_font_bold" style="font-size:80%;">then</span><span id="alg4.l5.6" class="ltx_text" style="font-size:80%;">


</span>
</div>
<div id="alg4.l6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l6.1.1.1" class="ltx_text" style="font-size:80%;">6:</span></span><span id="alg4.l6.2" class="ltx_text" style="font-size:80%;">      Get </span><math id="alg4.l6.m1.3" class="ltx_Math" alttext="dQ_{j},dK_{j},dV_{j}" display="inline"><semantics id="alg4.l6.m1.3a"><mrow id="alg4.l6.m1.3.3.3" xref="alg4.l6.m1.3.3.4.cmml"><mrow id="alg4.l6.m1.1.1.1.1" xref="alg4.l6.m1.1.1.1.1.cmml"><mi mathsize="80%" id="alg4.l6.m1.1.1.1.1.2" xref="alg4.l6.m1.1.1.1.1.2.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg4.l6.m1.1.1.1.1.1" xref="alg4.l6.m1.1.1.1.1.1.cmml">​</mo><msub id="alg4.l6.m1.1.1.1.1.3" xref="alg4.l6.m1.1.1.1.1.3.cmml"><mi mathsize="80%" id="alg4.l6.m1.1.1.1.1.3.2" xref="alg4.l6.m1.1.1.1.1.3.2.cmml">Q</mi><mi mathsize="80%" id="alg4.l6.m1.1.1.1.1.3.3" xref="alg4.l6.m1.1.1.1.1.3.3.cmml">j</mi></msub></mrow><mo mathsize="80%" id="alg4.l6.m1.3.3.3.4" xref="alg4.l6.m1.3.3.4.cmml">,</mo><mrow id="alg4.l6.m1.2.2.2.2" xref="alg4.l6.m1.2.2.2.2.cmml"><mi mathsize="80%" id="alg4.l6.m1.2.2.2.2.2" xref="alg4.l6.m1.2.2.2.2.2.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg4.l6.m1.2.2.2.2.1" xref="alg4.l6.m1.2.2.2.2.1.cmml">​</mo><msub id="alg4.l6.m1.2.2.2.2.3" xref="alg4.l6.m1.2.2.2.2.3.cmml"><mi mathsize="80%" id="alg4.l6.m1.2.2.2.2.3.2" xref="alg4.l6.m1.2.2.2.2.3.2.cmml">K</mi><mi mathsize="80%" id="alg4.l6.m1.2.2.2.2.3.3" xref="alg4.l6.m1.2.2.2.2.3.3.cmml">j</mi></msub></mrow><mo mathsize="80%" id="alg4.l6.m1.3.3.3.5" xref="alg4.l6.m1.3.3.4.cmml">,</mo><mrow id="alg4.l6.m1.3.3.3.3" xref="alg4.l6.m1.3.3.3.3.cmml"><mi mathsize="80%" id="alg4.l6.m1.3.3.3.3.2" xref="alg4.l6.m1.3.3.3.3.2.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg4.l6.m1.3.3.3.3.1" xref="alg4.l6.m1.3.3.3.3.1.cmml">​</mo><msub id="alg4.l6.m1.3.3.3.3.3" xref="alg4.l6.m1.3.3.3.3.3.cmml"><mi mathsize="80%" id="alg4.l6.m1.3.3.3.3.3.2" xref="alg4.l6.m1.3.3.3.3.3.2.cmml">V</mi><mi mathsize="80%" id="alg4.l6.m1.3.3.3.3.3.3" xref="alg4.l6.m1.3.3.3.3.3.3.cmml">j</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l6.m1.3b"><list id="alg4.l6.m1.3.3.4.cmml" xref="alg4.l6.m1.3.3.3"><apply id="alg4.l6.m1.1.1.1.1.cmml" xref="alg4.l6.m1.1.1.1.1"><times id="alg4.l6.m1.1.1.1.1.1.cmml" xref="alg4.l6.m1.1.1.1.1.1"></times><ci id="alg4.l6.m1.1.1.1.1.2.cmml" xref="alg4.l6.m1.1.1.1.1.2">𝑑</ci><apply id="alg4.l6.m1.1.1.1.1.3.cmml" xref="alg4.l6.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="alg4.l6.m1.1.1.1.1.3.1.cmml" xref="alg4.l6.m1.1.1.1.1.3">subscript</csymbol><ci id="alg4.l6.m1.1.1.1.1.3.2.cmml" xref="alg4.l6.m1.1.1.1.1.3.2">𝑄</ci><ci id="alg4.l6.m1.1.1.1.1.3.3.cmml" xref="alg4.l6.m1.1.1.1.1.3.3">𝑗</ci></apply></apply><apply id="alg4.l6.m1.2.2.2.2.cmml" xref="alg4.l6.m1.2.2.2.2"><times id="alg4.l6.m1.2.2.2.2.1.cmml" xref="alg4.l6.m1.2.2.2.2.1"></times><ci id="alg4.l6.m1.2.2.2.2.2.cmml" xref="alg4.l6.m1.2.2.2.2.2">𝑑</ci><apply id="alg4.l6.m1.2.2.2.2.3.cmml" xref="alg4.l6.m1.2.2.2.2.3"><csymbol cd="ambiguous" id="alg4.l6.m1.2.2.2.2.3.1.cmml" xref="alg4.l6.m1.2.2.2.2.3">subscript</csymbol><ci id="alg4.l6.m1.2.2.2.2.3.2.cmml" xref="alg4.l6.m1.2.2.2.2.3.2">𝐾</ci><ci id="alg4.l6.m1.2.2.2.2.3.3.cmml" xref="alg4.l6.m1.2.2.2.2.3.3">𝑗</ci></apply></apply><apply id="alg4.l6.m1.3.3.3.3.cmml" xref="alg4.l6.m1.3.3.3.3"><times id="alg4.l6.m1.3.3.3.3.1.cmml" xref="alg4.l6.m1.3.3.3.3.1"></times><ci id="alg4.l6.m1.3.3.3.3.2.cmml" xref="alg4.l6.m1.3.3.3.3.2">𝑑</ci><apply id="alg4.l6.m1.3.3.3.3.3.cmml" xref="alg4.l6.m1.3.3.3.3.3"><csymbol cd="ambiguous" id="alg4.l6.m1.3.3.3.3.3.1.cmml" xref="alg4.l6.m1.3.3.3.3.3">subscript</csymbol><ci id="alg4.l6.m1.3.3.3.3.3.2.cmml" xref="alg4.l6.m1.3.3.3.3.3.2">𝑉</ci><ci id="alg4.l6.m1.3.3.3.3.3.3.cmml" xref="alg4.l6.m1.3.3.3.3.3.3">𝑗</ci></apply></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg4.l6.m1.3c">dQ_{j},dK_{j},dV_{j}</annotation></semantics></math><span id="alg4.l6.3" class="ltx_text" style="font-size:80%;"> from </span><math id="alg4.l6.m2.3" class="ltx_Math" alttext="dQ_{buf},dK_{buf},dV_{buf}" display="inline"><semantics id="alg4.l6.m2.3a"><mrow id="alg4.l6.m2.3.3.3" xref="alg4.l6.m2.3.3.4.cmml"><mrow id="alg4.l6.m2.1.1.1.1" xref="alg4.l6.m2.1.1.1.1.cmml"><mi mathsize="80%" id="alg4.l6.m2.1.1.1.1.2" xref="alg4.l6.m2.1.1.1.1.2.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg4.l6.m2.1.1.1.1.1" xref="alg4.l6.m2.1.1.1.1.1.cmml">​</mo><msub id="alg4.l6.m2.1.1.1.1.3" xref="alg4.l6.m2.1.1.1.1.3.cmml"><mi mathsize="80%" id="alg4.l6.m2.1.1.1.1.3.2" xref="alg4.l6.m2.1.1.1.1.3.2.cmml">Q</mi><mrow id="alg4.l6.m2.1.1.1.1.3.3" xref="alg4.l6.m2.1.1.1.1.3.3.cmml"><mi mathsize="80%" id="alg4.l6.m2.1.1.1.1.3.3.2" xref="alg4.l6.m2.1.1.1.1.3.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg4.l6.m2.1.1.1.1.3.3.1" xref="alg4.l6.m2.1.1.1.1.3.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l6.m2.1.1.1.1.3.3.3" xref="alg4.l6.m2.1.1.1.1.3.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg4.l6.m2.1.1.1.1.3.3.1a" xref="alg4.l6.m2.1.1.1.1.3.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l6.m2.1.1.1.1.3.3.4" xref="alg4.l6.m2.1.1.1.1.3.3.4.cmml">f</mi></mrow></msub></mrow><mo mathsize="80%" id="alg4.l6.m2.3.3.3.4" xref="alg4.l6.m2.3.3.4.cmml">,</mo><mrow id="alg4.l6.m2.2.2.2.2" xref="alg4.l6.m2.2.2.2.2.cmml"><mi mathsize="80%" id="alg4.l6.m2.2.2.2.2.2" xref="alg4.l6.m2.2.2.2.2.2.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg4.l6.m2.2.2.2.2.1" xref="alg4.l6.m2.2.2.2.2.1.cmml">​</mo><msub id="alg4.l6.m2.2.2.2.2.3" xref="alg4.l6.m2.2.2.2.2.3.cmml"><mi mathsize="80%" id="alg4.l6.m2.2.2.2.2.3.2" xref="alg4.l6.m2.2.2.2.2.3.2.cmml">K</mi><mrow id="alg4.l6.m2.2.2.2.2.3.3" xref="alg4.l6.m2.2.2.2.2.3.3.cmml"><mi mathsize="80%" id="alg4.l6.m2.2.2.2.2.3.3.2" xref="alg4.l6.m2.2.2.2.2.3.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg4.l6.m2.2.2.2.2.3.3.1" xref="alg4.l6.m2.2.2.2.2.3.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l6.m2.2.2.2.2.3.3.3" xref="alg4.l6.m2.2.2.2.2.3.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg4.l6.m2.2.2.2.2.3.3.1a" xref="alg4.l6.m2.2.2.2.2.3.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l6.m2.2.2.2.2.3.3.4" xref="alg4.l6.m2.2.2.2.2.3.3.4.cmml">f</mi></mrow></msub></mrow><mo mathsize="80%" id="alg4.l6.m2.3.3.3.5" xref="alg4.l6.m2.3.3.4.cmml">,</mo><mrow id="alg4.l6.m2.3.3.3.3" xref="alg4.l6.m2.3.3.3.3.cmml"><mi mathsize="80%" id="alg4.l6.m2.3.3.3.3.2" xref="alg4.l6.m2.3.3.3.3.2.cmml">d</mi><mo lspace="0em" rspace="0em" id="alg4.l6.m2.3.3.3.3.1" xref="alg4.l6.m2.3.3.3.3.1.cmml">​</mo><msub id="alg4.l6.m2.3.3.3.3.3" xref="alg4.l6.m2.3.3.3.3.3.cmml"><mi mathsize="80%" id="alg4.l6.m2.3.3.3.3.3.2" xref="alg4.l6.m2.3.3.3.3.3.2.cmml">V</mi><mrow id="alg4.l6.m2.3.3.3.3.3.3" xref="alg4.l6.m2.3.3.3.3.3.3.cmml"><mi mathsize="80%" id="alg4.l6.m2.3.3.3.3.3.3.2" xref="alg4.l6.m2.3.3.3.3.3.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg4.l6.m2.3.3.3.3.3.3.1" xref="alg4.l6.m2.3.3.3.3.3.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l6.m2.3.3.3.3.3.3.3" xref="alg4.l6.m2.3.3.3.3.3.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg4.l6.m2.3.3.3.3.3.3.1a" xref="alg4.l6.m2.3.3.3.3.3.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l6.m2.3.3.3.3.3.3.4" xref="alg4.l6.m2.3.3.3.3.3.3.4.cmml">f</mi></mrow></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l6.m2.3b"><list id="alg4.l6.m2.3.3.4.cmml" xref="alg4.l6.m2.3.3.3"><apply id="alg4.l6.m2.1.1.1.1.cmml" xref="alg4.l6.m2.1.1.1.1"><times id="alg4.l6.m2.1.1.1.1.1.cmml" xref="alg4.l6.m2.1.1.1.1.1"></times><ci id="alg4.l6.m2.1.1.1.1.2.cmml" xref="alg4.l6.m2.1.1.1.1.2">𝑑</ci><apply id="alg4.l6.m2.1.1.1.1.3.cmml" xref="alg4.l6.m2.1.1.1.1.3"><csymbol cd="ambiguous" id="alg4.l6.m2.1.1.1.1.3.1.cmml" xref="alg4.l6.m2.1.1.1.1.3">subscript</csymbol><ci id="alg4.l6.m2.1.1.1.1.3.2.cmml" xref="alg4.l6.m2.1.1.1.1.3.2">𝑄</ci><apply id="alg4.l6.m2.1.1.1.1.3.3.cmml" xref="alg4.l6.m2.1.1.1.1.3.3"><times id="alg4.l6.m2.1.1.1.1.3.3.1.cmml" xref="alg4.l6.m2.1.1.1.1.3.3.1"></times><ci id="alg4.l6.m2.1.1.1.1.3.3.2.cmml" xref="alg4.l6.m2.1.1.1.1.3.3.2">𝑏</ci><ci id="alg4.l6.m2.1.1.1.1.3.3.3.cmml" xref="alg4.l6.m2.1.1.1.1.3.3.3">𝑢</ci><ci id="alg4.l6.m2.1.1.1.1.3.3.4.cmml" xref="alg4.l6.m2.1.1.1.1.3.3.4">𝑓</ci></apply></apply></apply><apply id="alg4.l6.m2.2.2.2.2.cmml" xref="alg4.l6.m2.2.2.2.2"><times id="alg4.l6.m2.2.2.2.2.1.cmml" xref="alg4.l6.m2.2.2.2.2.1"></times><ci id="alg4.l6.m2.2.2.2.2.2.cmml" xref="alg4.l6.m2.2.2.2.2.2">𝑑</ci><apply id="alg4.l6.m2.2.2.2.2.3.cmml" xref="alg4.l6.m2.2.2.2.2.3"><csymbol cd="ambiguous" id="alg4.l6.m2.2.2.2.2.3.1.cmml" xref="alg4.l6.m2.2.2.2.2.3">subscript</csymbol><ci id="alg4.l6.m2.2.2.2.2.3.2.cmml" xref="alg4.l6.m2.2.2.2.2.3.2">𝐾</ci><apply id="alg4.l6.m2.2.2.2.2.3.3.cmml" xref="alg4.l6.m2.2.2.2.2.3.3"><times id="alg4.l6.m2.2.2.2.2.3.3.1.cmml" xref="alg4.l6.m2.2.2.2.2.3.3.1"></times><ci id="alg4.l6.m2.2.2.2.2.3.3.2.cmml" xref="alg4.l6.m2.2.2.2.2.3.3.2">𝑏</ci><ci id="alg4.l6.m2.2.2.2.2.3.3.3.cmml" xref="alg4.l6.m2.2.2.2.2.3.3.3">𝑢</ci><ci id="alg4.l6.m2.2.2.2.2.3.3.4.cmml" xref="alg4.l6.m2.2.2.2.2.3.3.4">𝑓</ci></apply></apply></apply><apply id="alg4.l6.m2.3.3.3.3.cmml" xref="alg4.l6.m2.3.3.3.3"><times id="alg4.l6.m2.3.3.3.3.1.cmml" xref="alg4.l6.m2.3.3.3.3.1"></times><ci id="alg4.l6.m2.3.3.3.3.2.cmml" xref="alg4.l6.m2.3.3.3.3.2">𝑑</ci><apply id="alg4.l6.m2.3.3.3.3.3.cmml" xref="alg4.l6.m2.3.3.3.3.3"><csymbol cd="ambiguous" id="alg4.l6.m2.3.3.3.3.3.1.cmml" xref="alg4.l6.m2.3.3.3.3.3">subscript</csymbol><ci id="alg4.l6.m2.3.3.3.3.3.2.cmml" xref="alg4.l6.m2.3.3.3.3.3.2">𝑉</ci><apply id="alg4.l6.m2.3.3.3.3.3.3.cmml" xref="alg4.l6.m2.3.3.3.3.3.3"><times id="alg4.l6.m2.3.3.3.3.3.3.1.cmml" xref="alg4.l6.m2.3.3.3.3.3.3.1"></times><ci id="alg4.l6.m2.3.3.3.3.3.3.2.cmml" xref="alg4.l6.m2.3.3.3.3.3.3.2">𝑏</ci><ci id="alg4.l6.m2.3.3.3.3.3.3.3.cmml" xref="alg4.l6.m2.3.3.3.3.3.3.3">𝑢</ci><ci id="alg4.l6.m2.3.3.3.3.3.3.4.cmml" xref="alg4.l6.m2.3.3.3.3.3.3.4">𝑓</ci></apply></apply></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg4.l6.m2.3c">dQ_{buf},dK_{buf},dV_{buf}</annotation></semantics></math><span id="alg4.l6.4" class="ltx_text" style="font-size:80%;">; {Wait communication thread’s job finished}
</span>
</div>
<div id="alg4.l7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l7.1.1.1" class="ltx_text" style="font-size:80%;">7:</span></span><span id="alg4.l7.2" class="ltx_text" style="font-size:80%;">    </span><span id="alg4.l7.3" class="ltx_text ltx_font_bold" style="font-size:80%;">end</span><span id="alg4.l7.4" class="ltx_text" style="font-size:80%;"> </span><span id="alg4.l7.5" class="ltx_text ltx_font_bold" style="font-size:80%;">if</span>
</div>
<div id="alg4.l8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l8.1.1.1" class="ltx_text" style="font-size:80%;">8:</span></span><span id="alg4.l8.2" class="ltx_text" style="font-size:80%;">    </span><span id="alg4.l8.3" class="ltx_text ltx_font_bold" style="font-size:80%;">AsyncCommunicationCall</span><span id="alg4.l8.4" class="ltx_text" style="font-size:80%;">:
</span>
</div>
<div id="alg4.l9" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l9.1.1.1" class="ltx_text" style="font-size:80%;">9:</span></span><span id="alg4.l9.2" class="ltx_text" style="font-size:80%;">    Initiate asynchronous communication thread
</span>
</div>
<div id="alg4.l10" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l10.1.1.1" class="ltx_text" style="font-size:80%;">10:</span></span><span id="alg4.l10.2" class="ltx_text" style="font-size:80%;">     Let </span><math id="alg4.l10.m1.4" class="ltx_Math" alttext="\text{Buf}=(\mathbf{Q}_{buf}\mathbf{dQ}_{buf},\mathbf{dO}_{buf},D_{buf},lse_{buf})" display="inline"><semantics id="alg4.l10.m1.4a"><mrow id="alg4.l10.m1.4.4" xref="alg4.l10.m1.4.4.cmml"><mtext mathsize="80%" id="alg4.l10.m1.4.4.6" xref="alg4.l10.m1.4.4.6a.cmml">Buf</mtext><mo mathsize="80%" id="alg4.l10.m1.4.4.5" xref="alg4.l10.m1.4.4.5.cmml">=</mo><mrow id="alg4.l10.m1.4.4.4.4" xref="alg4.l10.m1.4.4.4.5.cmml"><mo maxsize="80%" minsize="80%" id="alg4.l10.m1.4.4.4.4.5" xref="alg4.l10.m1.4.4.4.5.cmml">(</mo><mrow id="alg4.l10.m1.1.1.1.1.1" xref="alg4.l10.m1.1.1.1.1.1.cmml"><msub id="alg4.l10.m1.1.1.1.1.1.2" xref="alg4.l10.m1.1.1.1.1.1.2.cmml"><mi mathsize="80%" id="alg4.l10.m1.1.1.1.1.1.2.2" xref="alg4.l10.m1.1.1.1.1.1.2.2.cmml">𝐐</mi><mrow id="alg4.l10.m1.1.1.1.1.1.2.3" xref="alg4.l10.m1.1.1.1.1.1.2.3.cmml"><mi mathsize="80%" id="alg4.l10.m1.1.1.1.1.1.2.3.2" xref="alg4.l10.m1.1.1.1.1.1.2.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg4.l10.m1.1.1.1.1.1.2.3.1" xref="alg4.l10.m1.1.1.1.1.1.2.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l10.m1.1.1.1.1.1.2.3.3" xref="alg4.l10.m1.1.1.1.1.1.2.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg4.l10.m1.1.1.1.1.1.2.3.1a" xref="alg4.l10.m1.1.1.1.1.1.2.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l10.m1.1.1.1.1.1.2.3.4" xref="alg4.l10.m1.1.1.1.1.1.2.3.4.cmml">f</mi></mrow></msub><mo lspace="0em" rspace="0em" id="alg4.l10.m1.1.1.1.1.1.1" xref="alg4.l10.m1.1.1.1.1.1.1.cmml">​</mo><msub id="alg4.l10.m1.1.1.1.1.1.3" xref="alg4.l10.m1.1.1.1.1.1.3.cmml"><mi mathsize="80%" id="alg4.l10.m1.1.1.1.1.1.3.2" xref="alg4.l10.m1.1.1.1.1.1.3.2.cmml">𝐝𝐐</mi><mrow id="alg4.l10.m1.1.1.1.1.1.3.3" xref="alg4.l10.m1.1.1.1.1.1.3.3.cmml"><mi mathsize="80%" id="alg4.l10.m1.1.1.1.1.1.3.3.2" xref="alg4.l10.m1.1.1.1.1.1.3.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg4.l10.m1.1.1.1.1.1.3.3.1" xref="alg4.l10.m1.1.1.1.1.1.3.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l10.m1.1.1.1.1.1.3.3.3" xref="alg4.l10.m1.1.1.1.1.1.3.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg4.l10.m1.1.1.1.1.1.3.3.1a" xref="alg4.l10.m1.1.1.1.1.1.3.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l10.m1.1.1.1.1.1.3.3.4" xref="alg4.l10.m1.1.1.1.1.1.3.3.4.cmml">f</mi></mrow></msub></mrow><mo mathsize="80%" id="alg4.l10.m1.4.4.4.4.6" xref="alg4.l10.m1.4.4.4.5.cmml">,</mo><msub id="alg4.l10.m1.2.2.2.2.2" xref="alg4.l10.m1.2.2.2.2.2.cmml"><mi mathsize="80%" id="alg4.l10.m1.2.2.2.2.2.2" xref="alg4.l10.m1.2.2.2.2.2.2.cmml">𝐝𝐎</mi><mrow id="alg4.l10.m1.2.2.2.2.2.3" xref="alg4.l10.m1.2.2.2.2.2.3.cmml"><mi mathsize="80%" id="alg4.l10.m1.2.2.2.2.2.3.2" xref="alg4.l10.m1.2.2.2.2.2.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg4.l10.m1.2.2.2.2.2.3.1" xref="alg4.l10.m1.2.2.2.2.2.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l10.m1.2.2.2.2.2.3.3" xref="alg4.l10.m1.2.2.2.2.2.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg4.l10.m1.2.2.2.2.2.3.1a" xref="alg4.l10.m1.2.2.2.2.2.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l10.m1.2.2.2.2.2.3.4" xref="alg4.l10.m1.2.2.2.2.2.3.4.cmml">f</mi></mrow></msub><mo mathsize="80%" id="alg4.l10.m1.4.4.4.4.7" xref="alg4.l10.m1.4.4.4.5.cmml">,</mo><msub id="alg4.l10.m1.3.3.3.3.3" xref="alg4.l10.m1.3.3.3.3.3.cmml"><mi mathsize="80%" id="alg4.l10.m1.3.3.3.3.3.2" xref="alg4.l10.m1.3.3.3.3.3.2.cmml">D</mi><mrow id="alg4.l10.m1.3.3.3.3.3.3" xref="alg4.l10.m1.3.3.3.3.3.3.cmml"><mi mathsize="80%" id="alg4.l10.m1.3.3.3.3.3.3.2" xref="alg4.l10.m1.3.3.3.3.3.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg4.l10.m1.3.3.3.3.3.3.1" xref="alg4.l10.m1.3.3.3.3.3.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l10.m1.3.3.3.3.3.3.3" xref="alg4.l10.m1.3.3.3.3.3.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg4.l10.m1.3.3.3.3.3.3.1a" xref="alg4.l10.m1.3.3.3.3.3.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l10.m1.3.3.3.3.3.3.4" xref="alg4.l10.m1.3.3.3.3.3.3.4.cmml">f</mi></mrow></msub><mo mathsize="80%" id="alg4.l10.m1.4.4.4.4.8" xref="alg4.l10.m1.4.4.4.5.cmml">,</mo><mrow id="alg4.l10.m1.4.4.4.4.4" xref="alg4.l10.m1.4.4.4.4.4.cmml"><mi mathsize="80%" id="alg4.l10.m1.4.4.4.4.4.2" xref="alg4.l10.m1.4.4.4.4.4.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg4.l10.m1.4.4.4.4.4.1" xref="alg4.l10.m1.4.4.4.4.4.1.cmml">​</mo><mi mathsize="80%" id="alg4.l10.m1.4.4.4.4.4.3" xref="alg4.l10.m1.4.4.4.4.4.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg4.l10.m1.4.4.4.4.4.1a" xref="alg4.l10.m1.4.4.4.4.4.1.cmml">​</mo><msub id="alg4.l10.m1.4.4.4.4.4.4" xref="alg4.l10.m1.4.4.4.4.4.4.cmml"><mi mathsize="80%" id="alg4.l10.m1.4.4.4.4.4.4.2" xref="alg4.l10.m1.4.4.4.4.4.4.2.cmml">e</mi><mrow id="alg4.l10.m1.4.4.4.4.4.4.3" xref="alg4.l10.m1.4.4.4.4.4.4.3.cmml"><mi mathsize="80%" id="alg4.l10.m1.4.4.4.4.4.4.3.2" xref="alg4.l10.m1.4.4.4.4.4.4.3.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="alg4.l10.m1.4.4.4.4.4.4.3.1" xref="alg4.l10.m1.4.4.4.4.4.4.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l10.m1.4.4.4.4.4.4.3.3" xref="alg4.l10.m1.4.4.4.4.4.4.3.3.cmml">u</mi><mo lspace="0em" rspace="0em" id="alg4.l10.m1.4.4.4.4.4.4.3.1a" xref="alg4.l10.m1.4.4.4.4.4.4.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l10.m1.4.4.4.4.4.4.3.4" xref="alg4.l10.m1.4.4.4.4.4.4.3.4.cmml">f</mi></mrow></msub></mrow><mo maxsize="80%" minsize="80%" id="alg4.l10.m1.4.4.4.4.9" xref="alg4.l10.m1.4.4.4.5.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l10.m1.4b"><apply id="alg4.l10.m1.4.4.cmml" xref="alg4.l10.m1.4.4"><eq id="alg4.l10.m1.4.4.5.cmml" xref="alg4.l10.m1.4.4.5"></eq><ci id="alg4.l10.m1.4.4.6a.cmml" xref="alg4.l10.m1.4.4.6"><mtext mathsize="80%" id="alg4.l10.m1.4.4.6.cmml" xref="alg4.l10.m1.4.4.6">Buf</mtext></ci><vector id="alg4.l10.m1.4.4.4.5.cmml" xref="alg4.l10.m1.4.4.4.4"><apply id="alg4.l10.m1.1.1.1.1.1.cmml" xref="alg4.l10.m1.1.1.1.1.1"><times id="alg4.l10.m1.1.1.1.1.1.1.cmml" xref="alg4.l10.m1.1.1.1.1.1.1"></times><apply id="alg4.l10.m1.1.1.1.1.1.2.cmml" xref="alg4.l10.m1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="alg4.l10.m1.1.1.1.1.1.2.1.cmml" xref="alg4.l10.m1.1.1.1.1.1.2">subscript</csymbol><ci id="alg4.l10.m1.1.1.1.1.1.2.2.cmml" xref="alg4.l10.m1.1.1.1.1.1.2.2">𝐐</ci><apply id="alg4.l10.m1.1.1.1.1.1.2.3.cmml" xref="alg4.l10.m1.1.1.1.1.1.2.3"><times id="alg4.l10.m1.1.1.1.1.1.2.3.1.cmml" xref="alg4.l10.m1.1.1.1.1.1.2.3.1"></times><ci id="alg4.l10.m1.1.1.1.1.1.2.3.2.cmml" xref="alg4.l10.m1.1.1.1.1.1.2.3.2">𝑏</ci><ci id="alg4.l10.m1.1.1.1.1.1.2.3.3.cmml" xref="alg4.l10.m1.1.1.1.1.1.2.3.3">𝑢</ci><ci id="alg4.l10.m1.1.1.1.1.1.2.3.4.cmml" xref="alg4.l10.m1.1.1.1.1.1.2.3.4">𝑓</ci></apply></apply><apply id="alg4.l10.m1.1.1.1.1.1.3.cmml" xref="alg4.l10.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="alg4.l10.m1.1.1.1.1.1.3.1.cmml" xref="alg4.l10.m1.1.1.1.1.1.3">subscript</csymbol><ci id="alg4.l10.m1.1.1.1.1.1.3.2.cmml" xref="alg4.l10.m1.1.1.1.1.1.3.2">𝐝𝐐</ci><apply id="alg4.l10.m1.1.1.1.1.1.3.3.cmml" xref="alg4.l10.m1.1.1.1.1.1.3.3"><times id="alg4.l10.m1.1.1.1.1.1.3.3.1.cmml" xref="alg4.l10.m1.1.1.1.1.1.3.3.1"></times><ci id="alg4.l10.m1.1.1.1.1.1.3.3.2.cmml" xref="alg4.l10.m1.1.1.1.1.1.3.3.2">𝑏</ci><ci id="alg4.l10.m1.1.1.1.1.1.3.3.3.cmml" xref="alg4.l10.m1.1.1.1.1.1.3.3.3">𝑢</ci><ci id="alg4.l10.m1.1.1.1.1.1.3.3.4.cmml" xref="alg4.l10.m1.1.1.1.1.1.3.3.4">𝑓</ci></apply></apply></apply><apply id="alg4.l10.m1.2.2.2.2.2.cmml" xref="alg4.l10.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="alg4.l10.m1.2.2.2.2.2.1.cmml" xref="alg4.l10.m1.2.2.2.2.2">subscript</csymbol><ci id="alg4.l10.m1.2.2.2.2.2.2.cmml" xref="alg4.l10.m1.2.2.2.2.2.2">𝐝𝐎</ci><apply id="alg4.l10.m1.2.2.2.2.2.3.cmml" xref="alg4.l10.m1.2.2.2.2.2.3"><times id="alg4.l10.m1.2.2.2.2.2.3.1.cmml" xref="alg4.l10.m1.2.2.2.2.2.3.1"></times><ci id="alg4.l10.m1.2.2.2.2.2.3.2.cmml" xref="alg4.l10.m1.2.2.2.2.2.3.2">𝑏</ci><ci id="alg4.l10.m1.2.2.2.2.2.3.3.cmml" xref="alg4.l10.m1.2.2.2.2.2.3.3">𝑢</ci><ci id="alg4.l10.m1.2.2.2.2.2.3.4.cmml" xref="alg4.l10.m1.2.2.2.2.2.3.4">𝑓</ci></apply></apply><apply id="alg4.l10.m1.3.3.3.3.3.cmml" xref="alg4.l10.m1.3.3.3.3.3"><csymbol cd="ambiguous" id="alg4.l10.m1.3.3.3.3.3.1.cmml" xref="alg4.l10.m1.3.3.3.3.3">subscript</csymbol><ci id="alg4.l10.m1.3.3.3.3.3.2.cmml" xref="alg4.l10.m1.3.3.3.3.3.2">𝐷</ci><apply id="alg4.l10.m1.3.3.3.3.3.3.cmml" xref="alg4.l10.m1.3.3.3.3.3.3"><times id="alg4.l10.m1.3.3.3.3.3.3.1.cmml" xref="alg4.l10.m1.3.3.3.3.3.3.1"></times><ci id="alg4.l10.m1.3.3.3.3.3.3.2.cmml" xref="alg4.l10.m1.3.3.3.3.3.3.2">𝑏</ci><ci id="alg4.l10.m1.3.3.3.3.3.3.3.cmml" xref="alg4.l10.m1.3.3.3.3.3.3.3">𝑢</ci><ci id="alg4.l10.m1.3.3.3.3.3.3.4.cmml" xref="alg4.l10.m1.3.3.3.3.3.3.4">𝑓</ci></apply></apply><apply id="alg4.l10.m1.4.4.4.4.4.cmml" xref="alg4.l10.m1.4.4.4.4.4"><times id="alg4.l10.m1.4.4.4.4.4.1.cmml" xref="alg4.l10.m1.4.4.4.4.4.1"></times><ci id="alg4.l10.m1.4.4.4.4.4.2.cmml" xref="alg4.l10.m1.4.4.4.4.4.2">𝑙</ci><ci id="alg4.l10.m1.4.4.4.4.4.3.cmml" xref="alg4.l10.m1.4.4.4.4.4.3">𝑠</ci><apply id="alg4.l10.m1.4.4.4.4.4.4.cmml" xref="alg4.l10.m1.4.4.4.4.4.4"><csymbol cd="ambiguous" id="alg4.l10.m1.4.4.4.4.4.4.1.cmml" xref="alg4.l10.m1.4.4.4.4.4.4">subscript</csymbol><ci id="alg4.l10.m1.4.4.4.4.4.4.2.cmml" xref="alg4.l10.m1.4.4.4.4.4.4.2">𝑒</ci><apply id="alg4.l10.m1.4.4.4.4.4.4.3.cmml" xref="alg4.l10.m1.4.4.4.4.4.4.3"><times id="alg4.l10.m1.4.4.4.4.4.4.3.1.cmml" xref="alg4.l10.m1.4.4.4.4.4.4.3.1"></times><ci id="alg4.l10.m1.4.4.4.4.4.4.3.2.cmml" xref="alg4.l10.m1.4.4.4.4.4.4.3.2">𝑏</ci><ci id="alg4.l10.m1.4.4.4.4.4.4.3.3.cmml" xref="alg4.l10.m1.4.4.4.4.4.4.3.3">𝑢</ci><ci id="alg4.l10.m1.4.4.4.4.4.4.3.4.cmml" xref="alg4.l10.m1.4.4.4.4.4.4.3.4">𝑓</ci></apply></apply></apply></vector></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l10.m1.4c">\text{Buf}=(\mathbf{Q}_{buf}\mathbf{dQ}_{buf},\mathbf{dO}_{buf},D_{buf},lse_{buf})</annotation></semantics></math><span id="alg4.l10.3" class="ltx_text" style="font-size:80%;">
</span>
</div>
<div id="alg4.l11" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l11.1.1.1" class="ltx_text" style="font-size:80%;">11:</span></span><span id="alg4.l11.2" class="ltx_text" style="font-size:80%;">     Send the </span><span id="alg4.l11.3" class="ltx_text ltx_markedasmath" style="font-size:80%;">Buf</span><span id="alg4.l11.4" class="ltx_text" style="font-size:80%;"> to next device and recvive new </span><span id="alg4.l11.5" class="ltx_text ltx_markedasmath" style="font-size:80%;">Buf</span><span id="alg4.l11.6" class="ltx_text" style="font-size:80%;"> from previous device;
</span>
</div>
<div id="alg4.l12" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l12.1.1.1" class="ltx_text" style="font-size:80%;">12:</span></span><span id="alg4.l12.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg4.l12.m1.2" class="ltx_Math" alttext="\mathbf{S}_{j,i}=\mathbf{Q}_{j}\mathbf{K}_{i}^{T}" display="inline"><semantics id="alg4.l12.m1.2a"><mrow id="alg4.l12.m1.2.3" xref="alg4.l12.m1.2.3.cmml"><msub id="alg4.l12.m1.2.3.2" xref="alg4.l12.m1.2.3.2.cmml"><mi mathsize="80%" id="alg4.l12.m1.2.3.2.2" xref="alg4.l12.m1.2.3.2.2.cmml">𝐒</mi><mrow id="alg4.l12.m1.2.2.2.4" xref="alg4.l12.m1.2.2.2.3.cmml"><mi mathsize="80%" id="alg4.l12.m1.1.1.1.1" xref="alg4.l12.m1.1.1.1.1.cmml">j</mi><mo mathsize="80%" id="alg4.l12.m1.2.2.2.4.1" xref="alg4.l12.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg4.l12.m1.2.2.2.2" xref="alg4.l12.m1.2.2.2.2.cmml">i</mi></mrow></msub><mo mathsize="80%" id="alg4.l12.m1.2.3.1" xref="alg4.l12.m1.2.3.1.cmml">=</mo><mrow id="alg4.l12.m1.2.3.3" xref="alg4.l12.m1.2.3.3.cmml"><msub id="alg4.l12.m1.2.3.3.2" xref="alg4.l12.m1.2.3.3.2.cmml"><mi mathsize="80%" id="alg4.l12.m1.2.3.3.2.2" xref="alg4.l12.m1.2.3.3.2.2.cmml">𝐐</mi><mi mathsize="80%" id="alg4.l12.m1.2.3.3.2.3" xref="alg4.l12.m1.2.3.3.2.3.cmml">j</mi></msub><mo lspace="0em" rspace="0em" id="alg4.l12.m1.2.3.3.1" xref="alg4.l12.m1.2.3.3.1.cmml">​</mo><msubsup id="alg4.l12.m1.2.3.3.3" xref="alg4.l12.m1.2.3.3.3.cmml"><mi mathsize="80%" id="alg4.l12.m1.2.3.3.3.2.2" xref="alg4.l12.m1.2.3.3.3.2.2.cmml">𝐊</mi><mi mathsize="80%" id="alg4.l12.m1.2.3.3.3.2.3" xref="alg4.l12.m1.2.3.3.3.2.3.cmml">i</mi><mi mathsize="80%" id="alg4.l12.m1.2.3.3.3.3" xref="alg4.l12.m1.2.3.3.3.3.cmml">T</mi></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l12.m1.2b"><apply id="alg4.l12.m1.2.3.cmml" xref="alg4.l12.m1.2.3"><eq id="alg4.l12.m1.2.3.1.cmml" xref="alg4.l12.m1.2.3.1"></eq><apply id="alg4.l12.m1.2.3.2.cmml" xref="alg4.l12.m1.2.3.2"><csymbol cd="ambiguous" id="alg4.l12.m1.2.3.2.1.cmml" xref="alg4.l12.m1.2.3.2">subscript</csymbol><ci id="alg4.l12.m1.2.3.2.2.cmml" xref="alg4.l12.m1.2.3.2.2">𝐒</ci><list id="alg4.l12.m1.2.2.2.3.cmml" xref="alg4.l12.m1.2.2.2.4"><ci id="alg4.l12.m1.1.1.1.1.cmml" xref="alg4.l12.m1.1.1.1.1">𝑗</ci><ci id="alg4.l12.m1.2.2.2.2.cmml" xref="alg4.l12.m1.2.2.2.2">𝑖</ci></list></apply><apply id="alg4.l12.m1.2.3.3.cmml" xref="alg4.l12.m1.2.3.3"><times id="alg4.l12.m1.2.3.3.1.cmml" xref="alg4.l12.m1.2.3.3.1"></times><apply id="alg4.l12.m1.2.3.3.2.cmml" xref="alg4.l12.m1.2.3.3.2"><csymbol cd="ambiguous" id="alg4.l12.m1.2.3.3.2.1.cmml" xref="alg4.l12.m1.2.3.3.2">subscript</csymbol><ci id="alg4.l12.m1.2.3.3.2.2.cmml" xref="alg4.l12.m1.2.3.3.2.2">𝐐</ci><ci id="alg4.l12.m1.2.3.3.2.3.cmml" xref="alg4.l12.m1.2.3.3.2.3">𝑗</ci></apply><apply id="alg4.l12.m1.2.3.3.3.cmml" xref="alg4.l12.m1.2.3.3.3"><csymbol cd="ambiguous" id="alg4.l12.m1.2.3.3.3.1.cmml" xref="alg4.l12.m1.2.3.3.3">superscript</csymbol><apply id="alg4.l12.m1.2.3.3.3.2.cmml" xref="alg4.l12.m1.2.3.3.3"><csymbol cd="ambiguous" id="alg4.l12.m1.2.3.3.3.2.1.cmml" xref="alg4.l12.m1.2.3.3.3">subscript</csymbol><ci id="alg4.l12.m1.2.3.3.3.2.2.cmml" xref="alg4.l12.m1.2.3.3.3.2.2">𝐊</ci><ci id="alg4.l12.m1.2.3.3.3.2.3.cmml" xref="alg4.l12.m1.2.3.3.3.2.3">𝑖</ci></apply><ci id="alg4.l12.m1.2.3.3.3.3.cmml" xref="alg4.l12.m1.2.3.3.3.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l12.m1.2c">\mathbf{S}_{j,i}=\mathbf{Q}_{j}\mathbf{K}_{i}^{T}</annotation></semantics></math><span id="alg4.l12.3" class="ltx_text" style="font-size:80%;">;
{The backward pass of local attention operations (w/o LAO).}
</span>
</div>
<div id="alg4.l13" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l13.1.1.1" class="ltx_text" style="font-size:80%;">13:</span></span><span id="alg4.l13.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg4.l13.m1.5" class="ltx_Math" alttext="\mathbf{P}_{j,i}=\text{exp}(\mathbf{S}_{j,i}-lse_{j})" display="inline"><semantics id="alg4.l13.m1.5a"><mrow id="alg4.l13.m1.5.5" xref="alg4.l13.m1.5.5.cmml"><msub id="alg4.l13.m1.5.5.3" xref="alg4.l13.m1.5.5.3.cmml"><mi mathsize="80%" id="alg4.l13.m1.5.5.3.2" xref="alg4.l13.m1.5.5.3.2.cmml">𝐏</mi><mrow id="alg4.l13.m1.2.2.2.4" xref="alg4.l13.m1.2.2.2.3.cmml"><mi mathsize="80%" id="alg4.l13.m1.1.1.1.1" xref="alg4.l13.m1.1.1.1.1.cmml">j</mi><mo mathsize="80%" id="alg4.l13.m1.2.2.2.4.1" xref="alg4.l13.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg4.l13.m1.2.2.2.2" xref="alg4.l13.m1.2.2.2.2.cmml">i</mi></mrow></msub><mo mathsize="80%" id="alg4.l13.m1.5.5.2" xref="alg4.l13.m1.5.5.2.cmml">=</mo><mrow id="alg4.l13.m1.5.5.1" xref="alg4.l13.m1.5.5.1.cmml"><mtext mathsize="80%" id="alg4.l13.m1.5.5.1.3" xref="alg4.l13.m1.5.5.1.3a.cmml">exp</mtext><mo lspace="0em" rspace="0em" id="alg4.l13.m1.5.5.1.2" xref="alg4.l13.m1.5.5.1.2.cmml">​</mo><mrow id="alg4.l13.m1.5.5.1.1.1" xref="alg4.l13.m1.5.5.1.1.1.1.cmml"><mo maxsize="80%" minsize="80%" id="alg4.l13.m1.5.5.1.1.1.2" xref="alg4.l13.m1.5.5.1.1.1.1.cmml">(</mo><mrow id="alg4.l13.m1.5.5.1.1.1.1" xref="alg4.l13.m1.5.5.1.1.1.1.cmml"><msub id="alg4.l13.m1.5.5.1.1.1.1.2" xref="alg4.l13.m1.5.5.1.1.1.1.2.cmml"><mi mathsize="80%" id="alg4.l13.m1.5.5.1.1.1.1.2.2" xref="alg4.l13.m1.5.5.1.1.1.1.2.2.cmml">𝐒</mi><mrow id="alg4.l13.m1.4.4.2.4" xref="alg4.l13.m1.4.4.2.3.cmml"><mi mathsize="80%" id="alg4.l13.m1.3.3.1.1" xref="alg4.l13.m1.3.3.1.1.cmml">j</mi><mo mathsize="80%" id="alg4.l13.m1.4.4.2.4.1" xref="alg4.l13.m1.4.4.2.3.cmml">,</mo><mi mathsize="80%" id="alg4.l13.m1.4.4.2.2" xref="alg4.l13.m1.4.4.2.2.cmml">i</mi></mrow></msub><mo mathsize="80%" id="alg4.l13.m1.5.5.1.1.1.1.1" xref="alg4.l13.m1.5.5.1.1.1.1.1.cmml">−</mo><mrow id="alg4.l13.m1.5.5.1.1.1.1.3" xref="alg4.l13.m1.5.5.1.1.1.1.3.cmml"><mi mathsize="80%" id="alg4.l13.m1.5.5.1.1.1.1.3.2" xref="alg4.l13.m1.5.5.1.1.1.1.3.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg4.l13.m1.5.5.1.1.1.1.3.1" xref="alg4.l13.m1.5.5.1.1.1.1.3.1.cmml">​</mo><mi mathsize="80%" id="alg4.l13.m1.5.5.1.1.1.1.3.3" xref="alg4.l13.m1.5.5.1.1.1.1.3.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg4.l13.m1.5.5.1.1.1.1.3.1a" xref="alg4.l13.m1.5.5.1.1.1.1.3.1.cmml">​</mo><msub id="alg4.l13.m1.5.5.1.1.1.1.3.4" xref="alg4.l13.m1.5.5.1.1.1.1.3.4.cmml"><mi mathsize="80%" id="alg4.l13.m1.5.5.1.1.1.1.3.4.2" xref="alg4.l13.m1.5.5.1.1.1.1.3.4.2.cmml">e</mi><mi mathsize="80%" id="alg4.l13.m1.5.5.1.1.1.1.3.4.3" xref="alg4.l13.m1.5.5.1.1.1.1.3.4.3.cmml">j</mi></msub></mrow></mrow><mo maxsize="80%" minsize="80%" id="alg4.l13.m1.5.5.1.1.1.3" xref="alg4.l13.m1.5.5.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l13.m1.5b"><apply id="alg4.l13.m1.5.5.cmml" xref="alg4.l13.m1.5.5"><eq id="alg4.l13.m1.5.5.2.cmml" xref="alg4.l13.m1.5.5.2"></eq><apply id="alg4.l13.m1.5.5.3.cmml" xref="alg4.l13.m1.5.5.3"><csymbol cd="ambiguous" id="alg4.l13.m1.5.5.3.1.cmml" xref="alg4.l13.m1.5.5.3">subscript</csymbol><ci id="alg4.l13.m1.5.5.3.2.cmml" xref="alg4.l13.m1.5.5.3.2">𝐏</ci><list id="alg4.l13.m1.2.2.2.3.cmml" xref="alg4.l13.m1.2.2.2.4"><ci id="alg4.l13.m1.1.1.1.1.cmml" xref="alg4.l13.m1.1.1.1.1">𝑗</ci><ci id="alg4.l13.m1.2.2.2.2.cmml" xref="alg4.l13.m1.2.2.2.2">𝑖</ci></list></apply><apply id="alg4.l13.m1.5.5.1.cmml" xref="alg4.l13.m1.5.5.1"><times id="alg4.l13.m1.5.5.1.2.cmml" xref="alg4.l13.m1.5.5.1.2"></times><ci id="alg4.l13.m1.5.5.1.3a.cmml" xref="alg4.l13.m1.5.5.1.3"><mtext mathsize="80%" id="alg4.l13.m1.5.5.1.3.cmml" xref="alg4.l13.m1.5.5.1.3">exp</mtext></ci><apply id="alg4.l13.m1.5.5.1.1.1.1.cmml" xref="alg4.l13.m1.5.5.1.1.1"><minus id="alg4.l13.m1.5.5.1.1.1.1.1.cmml" xref="alg4.l13.m1.5.5.1.1.1.1.1"></minus><apply id="alg4.l13.m1.5.5.1.1.1.1.2.cmml" xref="alg4.l13.m1.5.5.1.1.1.1.2"><csymbol cd="ambiguous" id="alg4.l13.m1.5.5.1.1.1.1.2.1.cmml" xref="alg4.l13.m1.5.5.1.1.1.1.2">subscript</csymbol><ci id="alg4.l13.m1.5.5.1.1.1.1.2.2.cmml" xref="alg4.l13.m1.5.5.1.1.1.1.2.2">𝐒</ci><list id="alg4.l13.m1.4.4.2.3.cmml" xref="alg4.l13.m1.4.4.2.4"><ci id="alg4.l13.m1.3.3.1.1.cmml" xref="alg4.l13.m1.3.3.1.1">𝑗</ci><ci id="alg4.l13.m1.4.4.2.2.cmml" xref="alg4.l13.m1.4.4.2.2">𝑖</ci></list></apply><apply id="alg4.l13.m1.5.5.1.1.1.1.3.cmml" xref="alg4.l13.m1.5.5.1.1.1.1.3"><times id="alg4.l13.m1.5.5.1.1.1.1.3.1.cmml" xref="alg4.l13.m1.5.5.1.1.1.1.3.1"></times><ci id="alg4.l13.m1.5.5.1.1.1.1.3.2.cmml" xref="alg4.l13.m1.5.5.1.1.1.1.3.2">𝑙</ci><ci id="alg4.l13.m1.5.5.1.1.1.1.3.3.cmml" xref="alg4.l13.m1.5.5.1.1.1.1.3.3">𝑠</ci><apply id="alg4.l13.m1.5.5.1.1.1.1.3.4.cmml" xref="alg4.l13.m1.5.5.1.1.1.1.3.4"><csymbol cd="ambiguous" id="alg4.l13.m1.5.5.1.1.1.1.3.4.1.cmml" xref="alg4.l13.m1.5.5.1.1.1.1.3.4">subscript</csymbol><ci id="alg4.l13.m1.5.5.1.1.1.1.3.4.2.cmml" xref="alg4.l13.m1.5.5.1.1.1.1.3.4.2">𝑒</ci><ci id="alg4.l13.m1.5.5.1.1.1.1.3.4.3.cmml" xref="alg4.l13.m1.5.5.1.1.1.1.3.4.3">𝑗</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l13.m1.5c">\mathbf{P}_{j,i}=\text{exp}(\mathbf{S}_{j,i}-lse_{j})</annotation></semantics></math><span id="alg4.l13.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg4.l14" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l14.1.1.1" class="ltx_text" style="font-size:80%;">14:</span></span><span id="alg4.l14.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg4.l14.m1.2" class="ltx_Math" alttext="\mathbf{dV}_{i}=\mathbf{dV}_{i}+\mathbf{P}_{j,i}^{T}\mathbf{dO}_{j}" display="inline"><semantics id="alg4.l14.m1.2a"><mrow id="alg4.l14.m1.2.3" xref="alg4.l14.m1.2.3.cmml"><msub id="alg4.l14.m1.2.3.2" xref="alg4.l14.m1.2.3.2.cmml"><mi mathsize="80%" id="alg4.l14.m1.2.3.2.2" xref="alg4.l14.m1.2.3.2.2.cmml">𝐝𝐕</mi><mi mathsize="80%" id="alg4.l14.m1.2.3.2.3" xref="alg4.l14.m1.2.3.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg4.l14.m1.2.3.1" xref="alg4.l14.m1.2.3.1.cmml">=</mo><mrow id="alg4.l14.m1.2.3.3" xref="alg4.l14.m1.2.3.3.cmml"><msub id="alg4.l14.m1.2.3.3.2" xref="alg4.l14.m1.2.3.3.2.cmml"><mi mathsize="80%" id="alg4.l14.m1.2.3.3.2.2" xref="alg4.l14.m1.2.3.3.2.2.cmml">𝐝𝐕</mi><mi mathsize="80%" id="alg4.l14.m1.2.3.3.2.3" xref="alg4.l14.m1.2.3.3.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg4.l14.m1.2.3.3.1" xref="alg4.l14.m1.2.3.3.1.cmml">+</mo><mrow id="alg4.l14.m1.2.3.3.3" xref="alg4.l14.m1.2.3.3.3.cmml"><msubsup id="alg4.l14.m1.2.3.3.3.2" xref="alg4.l14.m1.2.3.3.3.2.cmml"><mi mathsize="80%" id="alg4.l14.m1.2.3.3.3.2.2.2" xref="alg4.l14.m1.2.3.3.3.2.2.2.cmml">𝐏</mi><mrow id="alg4.l14.m1.2.2.2.4" xref="alg4.l14.m1.2.2.2.3.cmml"><mi mathsize="80%" id="alg4.l14.m1.1.1.1.1" xref="alg4.l14.m1.1.1.1.1.cmml">j</mi><mo mathsize="80%" id="alg4.l14.m1.2.2.2.4.1" xref="alg4.l14.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg4.l14.m1.2.2.2.2" xref="alg4.l14.m1.2.2.2.2.cmml">i</mi></mrow><mi mathsize="80%" id="alg4.l14.m1.2.3.3.3.2.3" xref="alg4.l14.m1.2.3.3.3.2.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="alg4.l14.m1.2.3.3.3.1" xref="alg4.l14.m1.2.3.3.3.1.cmml">​</mo><msub id="alg4.l14.m1.2.3.3.3.3" xref="alg4.l14.m1.2.3.3.3.3.cmml"><mi mathsize="80%" id="alg4.l14.m1.2.3.3.3.3.2" xref="alg4.l14.m1.2.3.3.3.3.2.cmml">𝐝𝐎</mi><mi mathsize="80%" id="alg4.l14.m1.2.3.3.3.3.3" xref="alg4.l14.m1.2.3.3.3.3.3.cmml">j</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l14.m1.2b"><apply id="alg4.l14.m1.2.3.cmml" xref="alg4.l14.m1.2.3"><eq id="alg4.l14.m1.2.3.1.cmml" xref="alg4.l14.m1.2.3.1"></eq><apply id="alg4.l14.m1.2.3.2.cmml" xref="alg4.l14.m1.2.3.2"><csymbol cd="ambiguous" id="alg4.l14.m1.2.3.2.1.cmml" xref="alg4.l14.m1.2.3.2">subscript</csymbol><ci id="alg4.l14.m1.2.3.2.2.cmml" xref="alg4.l14.m1.2.3.2.2">𝐝𝐕</ci><ci id="alg4.l14.m1.2.3.2.3.cmml" xref="alg4.l14.m1.2.3.2.3">𝑖</ci></apply><apply id="alg4.l14.m1.2.3.3.cmml" xref="alg4.l14.m1.2.3.3"><plus id="alg4.l14.m1.2.3.3.1.cmml" xref="alg4.l14.m1.2.3.3.1"></plus><apply id="alg4.l14.m1.2.3.3.2.cmml" xref="alg4.l14.m1.2.3.3.2"><csymbol cd="ambiguous" id="alg4.l14.m1.2.3.3.2.1.cmml" xref="alg4.l14.m1.2.3.3.2">subscript</csymbol><ci id="alg4.l14.m1.2.3.3.2.2.cmml" xref="alg4.l14.m1.2.3.3.2.2">𝐝𝐕</ci><ci id="alg4.l14.m1.2.3.3.2.3.cmml" xref="alg4.l14.m1.2.3.3.2.3">𝑖</ci></apply><apply id="alg4.l14.m1.2.3.3.3.cmml" xref="alg4.l14.m1.2.3.3.3"><times id="alg4.l14.m1.2.3.3.3.1.cmml" xref="alg4.l14.m1.2.3.3.3.1"></times><apply id="alg4.l14.m1.2.3.3.3.2.cmml" xref="alg4.l14.m1.2.3.3.3.2"><csymbol cd="ambiguous" id="alg4.l14.m1.2.3.3.3.2.1.cmml" xref="alg4.l14.m1.2.3.3.3.2">superscript</csymbol><apply id="alg4.l14.m1.2.3.3.3.2.2.cmml" xref="alg4.l14.m1.2.3.3.3.2"><csymbol cd="ambiguous" id="alg4.l14.m1.2.3.3.3.2.2.1.cmml" xref="alg4.l14.m1.2.3.3.3.2">subscript</csymbol><ci id="alg4.l14.m1.2.3.3.3.2.2.2.cmml" xref="alg4.l14.m1.2.3.3.3.2.2.2">𝐏</ci><list id="alg4.l14.m1.2.2.2.3.cmml" xref="alg4.l14.m1.2.2.2.4"><ci id="alg4.l14.m1.1.1.1.1.cmml" xref="alg4.l14.m1.1.1.1.1">𝑗</ci><ci id="alg4.l14.m1.2.2.2.2.cmml" xref="alg4.l14.m1.2.2.2.2">𝑖</ci></list></apply><ci id="alg4.l14.m1.2.3.3.3.2.3.cmml" xref="alg4.l14.m1.2.3.3.3.2.3">𝑇</ci></apply><apply id="alg4.l14.m1.2.3.3.3.3.cmml" xref="alg4.l14.m1.2.3.3.3.3"><csymbol cd="ambiguous" id="alg4.l14.m1.2.3.3.3.3.1.cmml" xref="alg4.l14.m1.2.3.3.3.3">subscript</csymbol><ci id="alg4.l14.m1.2.3.3.3.3.2.cmml" xref="alg4.l14.m1.2.3.3.3.3.2">𝐝𝐎</ci><ci id="alg4.l14.m1.2.3.3.3.3.3.cmml" xref="alg4.l14.m1.2.3.3.3.3.3">𝑗</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l14.m1.2c">\mathbf{dV}_{i}=\mathbf{dV}_{i}+\mathbf{P}_{j,i}^{T}\mathbf{dO}_{j}</annotation></semantics></math><span id="alg4.l14.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg4.l15" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l15.1.1.1" class="ltx_text" style="font-size:80%;">15:</span></span><span id="alg4.l15.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg4.l15.m1.2" class="ltx_Math" alttext="\mathbf{dP}_{j,i}=\mathbf{dO}_{j}~{}\mathbf{V}_{i}^{T}" display="inline"><semantics id="alg4.l15.m1.2a"><mrow id="alg4.l15.m1.2.3" xref="alg4.l15.m1.2.3.cmml"><msub id="alg4.l15.m1.2.3.2" xref="alg4.l15.m1.2.3.2.cmml"><mi mathsize="80%" id="alg4.l15.m1.2.3.2.2" xref="alg4.l15.m1.2.3.2.2.cmml">𝐝𝐏</mi><mrow id="alg4.l15.m1.2.2.2.4" xref="alg4.l15.m1.2.2.2.3.cmml"><mi mathsize="80%" id="alg4.l15.m1.1.1.1.1" xref="alg4.l15.m1.1.1.1.1.cmml">j</mi><mo mathsize="80%" id="alg4.l15.m1.2.2.2.4.1" xref="alg4.l15.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg4.l15.m1.2.2.2.2" xref="alg4.l15.m1.2.2.2.2.cmml">i</mi></mrow></msub><mo mathsize="80%" id="alg4.l15.m1.2.3.1" xref="alg4.l15.m1.2.3.1.cmml">=</mo><mrow id="alg4.l15.m1.2.3.3" xref="alg4.l15.m1.2.3.3.cmml"><msub id="alg4.l15.m1.2.3.3.2" xref="alg4.l15.m1.2.3.3.2.cmml"><mi mathsize="80%" id="alg4.l15.m1.2.3.3.2.2" xref="alg4.l15.m1.2.3.3.2.2.cmml">𝐝𝐎</mi><mi mathsize="80%" id="alg4.l15.m1.2.3.3.2.3" xref="alg4.l15.m1.2.3.3.2.3.cmml">j</mi></msub><mo lspace="0em" rspace="0em" id="alg4.l15.m1.2.3.3.1" xref="alg4.l15.m1.2.3.3.1.cmml">​</mo><msubsup id="alg4.l15.m1.2.3.3.3" xref="alg4.l15.m1.2.3.3.3.cmml"><mi mathsize="80%" id="alg4.l15.m1.2.3.3.3.2.2" xref="alg4.l15.m1.2.3.3.3.2.2.cmml">𝐕</mi><mi mathsize="80%" id="alg4.l15.m1.2.3.3.3.2.3" xref="alg4.l15.m1.2.3.3.3.2.3.cmml">i</mi><mi mathsize="80%" id="alg4.l15.m1.2.3.3.3.3" xref="alg4.l15.m1.2.3.3.3.3.cmml">T</mi></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l15.m1.2b"><apply id="alg4.l15.m1.2.3.cmml" xref="alg4.l15.m1.2.3"><eq id="alg4.l15.m1.2.3.1.cmml" xref="alg4.l15.m1.2.3.1"></eq><apply id="alg4.l15.m1.2.3.2.cmml" xref="alg4.l15.m1.2.3.2"><csymbol cd="ambiguous" id="alg4.l15.m1.2.3.2.1.cmml" xref="alg4.l15.m1.2.3.2">subscript</csymbol><ci id="alg4.l15.m1.2.3.2.2.cmml" xref="alg4.l15.m1.2.3.2.2">𝐝𝐏</ci><list id="alg4.l15.m1.2.2.2.3.cmml" xref="alg4.l15.m1.2.2.2.4"><ci id="alg4.l15.m1.1.1.1.1.cmml" xref="alg4.l15.m1.1.1.1.1">𝑗</ci><ci id="alg4.l15.m1.2.2.2.2.cmml" xref="alg4.l15.m1.2.2.2.2">𝑖</ci></list></apply><apply id="alg4.l15.m1.2.3.3.cmml" xref="alg4.l15.m1.2.3.3"><times id="alg4.l15.m1.2.3.3.1.cmml" xref="alg4.l15.m1.2.3.3.1"></times><apply id="alg4.l15.m1.2.3.3.2.cmml" xref="alg4.l15.m1.2.3.3.2"><csymbol cd="ambiguous" id="alg4.l15.m1.2.3.3.2.1.cmml" xref="alg4.l15.m1.2.3.3.2">subscript</csymbol><ci id="alg4.l15.m1.2.3.3.2.2.cmml" xref="alg4.l15.m1.2.3.3.2.2">𝐝𝐎</ci><ci id="alg4.l15.m1.2.3.3.2.3.cmml" xref="alg4.l15.m1.2.3.3.2.3">𝑗</ci></apply><apply id="alg4.l15.m1.2.3.3.3.cmml" xref="alg4.l15.m1.2.3.3.3"><csymbol cd="ambiguous" id="alg4.l15.m1.2.3.3.3.1.cmml" xref="alg4.l15.m1.2.3.3.3">superscript</csymbol><apply id="alg4.l15.m1.2.3.3.3.2.cmml" xref="alg4.l15.m1.2.3.3.3"><csymbol cd="ambiguous" id="alg4.l15.m1.2.3.3.3.2.1.cmml" xref="alg4.l15.m1.2.3.3.3">subscript</csymbol><ci id="alg4.l15.m1.2.3.3.3.2.2.cmml" xref="alg4.l15.m1.2.3.3.3.2.2">𝐕</ci><ci id="alg4.l15.m1.2.3.3.3.2.3.cmml" xref="alg4.l15.m1.2.3.3.3.2.3">𝑖</ci></apply><ci id="alg4.l15.m1.2.3.3.3.3.cmml" xref="alg4.l15.m1.2.3.3.3.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l15.m1.2c">\mathbf{dP}_{j,i}=\mathbf{dO}_{j}~{}\mathbf{V}_{i}^{T}</annotation></semantics></math><span id="alg4.l15.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg4.l16" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l16.1.1.1" class="ltx_text" style="font-size:80%;">16:</span></span><span id="alg4.l16.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg4.l16.m1.7" class="ltx_Math" alttext="\mathbf{dS}_{j,i}=\mathbf{P}_{j,i}\circ(\mathbf{dP}_{j,i}-D_{j})" display="inline"><semantics id="alg4.l16.m1.7a"><mrow id="alg4.l16.m1.7.7" xref="alg4.l16.m1.7.7.cmml"><msub id="alg4.l16.m1.7.7.3" xref="alg4.l16.m1.7.7.3.cmml"><mi mathsize="80%" id="alg4.l16.m1.7.7.3.2" xref="alg4.l16.m1.7.7.3.2.cmml">𝐝𝐒</mi><mrow id="alg4.l16.m1.2.2.2.4" xref="alg4.l16.m1.2.2.2.3.cmml"><mi mathsize="80%" id="alg4.l16.m1.1.1.1.1" xref="alg4.l16.m1.1.1.1.1.cmml">j</mi><mo mathsize="80%" id="alg4.l16.m1.2.2.2.4.1" xref="alg4.l16.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg4.l16.m1.2.2.2.2" xref="alg4.l16.m1.2.2.2.2.cmml">i</mi></mrow></msub><mo mathsize="80%" id="alg4.l16.m1.7.7.2" xref="alg4.l16.m1.7.7.2.cmml">=</mo><mrow id="alg4.l16.m1.7.7.1" xref="alg4.l16.m1.7.7.1.cmml"><msub id="alg4.l16.m1.7.7.1.3" xref="alg4.l16.m1.7.7.1.3.cmml"><mi mathsize="80%" id="alg4.l16.m1.7.7.1.3.2" xref="alg4.l16.m1.7.7.1.3.2.cmml">𝐏</mi><mrow id="alg4.l16.m1.4.4.2.4" xref="alg4.l16.m1.4.4.2.3.cmml"><mi mathsize="80%" id="alg4.l16.m1.3.3.1.1" xref="alg4.l16.m1.3.3.1.1.cmml">j</mi><mo mathsize="80%" id="alg4.l16.m1.4.4.2.4.1" xref="alg4.l16.m1.4.4.2.3.cmml">,</mo><mi mathsize="80%" id="alg4.l16.m1.4.4.2.2" xref="alg4.l16.m1.4.4.2.2.cmml">i</mi></mrow></msub><mo lspace="0.222em" mathsize="80%" rspace="0.222em" id="alg4.l16.m1.7.7.1.2" xref="alg4.l16.m1.7.7.1.2.cmml">∘</mo><mrow id="alg4.l16.m1.7.7.1.1.1" xref="alg4.l16.m1.7.7.1.1.1.1.cmml"><mo maxsize="80%" minsize="80%" id="alg4.l16.m1.7.7.1.1.1.2" xref="alg4.l16.m1.7.7.1.1.1.1.cmml">(</mo><mrow id="alg4.l16.m1.7.7.1.1.1.1" xref="alg4.l16.m1.7.7.1.1.1.1.cmml"><msub id="alg4.l16.m1.7.7.1.1.1.1.2" xref="alg4.l16.m1.7.7.1.1.1.1.2.cmml"><mi mathsize="80%" id="alg4.l16.m1.7.7.1.1.1.1.2.2" xref="alg4.l16.m1.7.7.1.1.1.1.2.2.cmml">𝐝𝐏</mi><mrow id="alg4.l16.m1.6.6.2.4" xref="alg4.l16.m1.6.6.2.3.cmml"><mi mathsize="80%" id="alg4.l16.m1.5.5.1.1" xref="alg4.l16.m1.5.5.1.1.cmml">j</mi><mo mathsize="80%" id="alg4.l16.m1.6.6.2.4.1" xref="alg4.l16.m1.6.6.2.3.cmml">,</mo><mi mathsize="80%" id="alg4.l16.m1.6.6.2.2" xref="alg4.l16.m1.6.6.2.2.cmml">i</mi></mrow></msub><mo mathsize="80%" id="alg4.l16.m1.7.7.1.1.1.1.1" xref="alg4.l16.m1.7.7.1.1.1.1.1.cmml">−</mo><msub id="alg4.l16.m1.7.7.1.1.1.1.3" xref="alg4.l16.m1.7.7.1.1.1.1.3.cmml"><mi mathsize="80%" id="alg4.l16.m1.7.7.1.1.1.1.3.2" xref="alg4.l16.m1.7.7.1.1.1.1.3.2.cmml">D</mi><mi mathsize="80%" id="alg4.l16.m1.7.7.1.1.1.1.3.3" xref="alg4.l16.m1.7.7.1.1.1.1.3.3.cmml">j</mi></msub></mrow><mo maxsize="80%" minsize="80%" id="alg4.l16.m1.7.7.1.1.1.3" xref="alg4.l16.m1.7.7.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l16.m1.7b"><apply id="alg4.l16.m1.7.7.cmml" xref="alg4.l16.m1.7.7"><eq id="alg4.l16.m1.7.7.2.cmml" xref="alg4.l16.m1.7.7.2"></eq><apply id="alg4.l16.m1.7.7.3.cmml" xref="alg4.l16.m1.7.7.3"><csymbol cd="ambiguous" id="alg4.l16.m1.7.7.3.1.cmml" xref="alg4.l16.m1.7.7.3">subscript</csymbol><ci id="alg4.l16.m1.7.7.3.2.cmml" xref="alg4.l16.m1.7.7.3.2">𝐝𝐒</ci><list id="alg4.l16.m1.2.2.2.3.cmml" xref="alg4.l16.m1.2.2.2.4"><ci id="alg4.l16.m1.1.1.1.1.cmml" xref="alg4.l16.m1.1.1.1.1">𝑗</ci><ci id="alg4.l16.m1.2.2.2.2.cmml" xref="alg4.l16.m1.2.2.2.2">𝑖</ci></list></apply><apply id="alg4.l16.m1.7.7.1.cmml" xref="alg4.l16.m1.7.7.1"><compose id="alg4.l16.m1.7.7.1.2.cmml" xref="alg4.l16.m1.7.7.1.2"></compose><apply id="alg4.l16.m1.7.7.1.3.cmml" xref="alg4.l16.m1.7.7.1.3"><csymbol cd="ambiguous" id="alg4.l16.m1.7.7.1.3.1.cmml" xref="alg4.l16.m1.7.7.1.3">subscript</csymbol><ci id="alg4.l16.m1.7.7.1.3.2.cmml" xref="alg4.l16.m1.7.7.1.3.2">𝐏</ci><list id="alg4.l16.m1.4.4.2.3.cmml" xref="alg4.l16.m1.4.4.2.4"><ci id="alg4.l16.m1.3.3.1.1.cmml" xref="alg4.l16.m1.3.3.1.1">𝑗</ci><ci id="alg4.l16.m1.4.4.2.2.cmml" xref="alg4.l16.m1.4.4.2.2">𝑖</ci></list></apply><apply id="alg4.l16.m1.7.7.1.1.1.1.cmml" xref="alg4.l16.m1.7.7.1.1.1"><minus id="alg4.l16.m1.7.7.1.1.1.1.1.cmml" xref="alg4.l16.m1.7.7.1.1.1.1.1"></minus><apply id="alg4.l16.m1.7.7.1.1.1.1.2.cmml" xref="alg4.l16.m1.7.7.1.1.1.1.2"><csymbol cd="ambiguous" id="alg4.l16.m1.7.7.1.1.1.1.2.1.cmml" xref="alg4.l16.m1.7.7.1.1.1.1.2">subscript</csymbol><ci id="alg4.l16.m1.7.7.1.1.1.1.2.2.cmml" xref="alg4.l16.m1.7.7.1.1.1.1.2.2">𝐝𝐏</ci><list id="alg4.l16.m1.6.6.2.3.cmml" xref="alg4.l16.m1.6.6.2.4"><ci id="alg4.l16.m1.5.5.1.1.cmml" xref="alg4.l16.m1.5.5.1.1">𝑗</ci><ci id="alg4.l16.m1.6.6.2.2.cmml" xref="alg4.l16.m1.6.6.2.2">𝑖</ci></list></apply><apply id="alg4.l16.m1.7.7.1.1.1.1.3.cmml" xref="alg4.l16.m1.7.7.1.1.1.1.3"><csymbol cd="ambiguous" id="alg4.l16.m1.7.7.1.1.1.1.3.1.cmml" xref="alg4.l16.m1.7.7.1.1.1.1.3">subscript</csymbol><ci id="alg4.l16.m1.7.7.1.1.1.1.3.2.cmml" xref="alg4.l16.m1.7.7.1.1.1.1.3.2">𝐷</ci><ci id="alg4.l16.m1.7.7.1.1.1.1.3.3.cmml" xref="alg4.l16.m1.7.7.1.1.1.1.3.3">𝑗</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l16.m1.7c">\mathbf{dS}_{j,i}=\mathbf{P}_{j,i}\circ(\mathbf{dP}_{j,i}-D_{j})</annotation></semantics></math><span id="alg4.l16.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg4.l17" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l17.1.1.1" class="ltx_text" style="font-size:80%;">17:</span></span><span id="alg4.l17.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg4.l17.m1.2" class="ltx_Math" alttext="\mathbf{dK}_{i}=\mathbf{dK}_{i}+\mathbf{dS}_{j,i}^{T}\mathbf{Q}_{j}" display="inline"><semantics id="alg4.l17.m1.2a"><mrow id="alg4.l17.m1.2.3" xref="alg4.l17.m1.2.3.cmml"><msub id="alg4.l17.m1.2.3.2" xref="alg4.l17.m1.2.3.2.cmml"><mi mathsize="80%" id="alg4.l17.m1.2.3.2.2" xref="alg4.l17.m1.2.3.2.2.cmml">𝐝𝐊</mi><mi mathsize="80%" id="alg4.l17.m1.2.3.2.3" xref="alg4.l17.m1.2.3.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg4.l17.m1.2.3.1" xref="alg4.l17.m1.2.3.1.cmml">=</mo><mrow id="alg4.l17.m1.2.3.3" xref="alg4.l17.m1.2.3.3.cmml"><msub id="alg4.l17.m1.2.3.3.2" xref="alg4.l17.m1.2.3.3.2.cmml"><mi mathsize="80%" id="alg4.l17.m1.2.3.3.2.2" xref="alg4.l17.m1.2.3.3.2.2.cmml">𝐝𝐊</mi><mi mathsize="80%" id="alg4.l17.m1.2.3.3.2.3" xref="alg4.l17.m1.2.3.3.2.3.cmml">i</mi></msub><mo mathsize="80%" id="alg4.l17.m1.2.3.3.1" xref="alg4.l17.m1.2.3.3.1.cmml">+</mo><mrow id="alg4.l17.m1.2.3.3.3" xref="alg4.l17.m1.2.3.3.3.cmml"><msubsup id="alg4.l17.m1.2.3.3.3.2" xref="alg4.l17.m1.2.3.3.3.2.cmml"><mi mathsize="80%" id="alg4.l17.m1.2.3.3.3.2.2.2" xref="alg4.l17.m1.2.3.3.3.2.2.2.cmml">𝐝𝐒</mi><mrow id="alg4.l17.m1.2.2.2.4" xref="alg4.l17.m1.2.2.2.3.cmml"><mi mathsize="80%" id="alg4.l17.m1.1.1.1.1" xref="alg4.l17.m1.1.1.1.1.cmml">j</mi><mo mathsize="80%" id="alg4.l17.m1.2.2.2.4.1" xref="alg4.l17.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg4.l17.m1.2.2.2.2" xref="alg4.l17.m1.2.2.2.2.cmml">i</mi></mrow><mi mathsize="80%" id="alg4.l17.m1.2.3.3.3.2.3" xref="alg4.l17.m1.2.3.3.3.2.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="alg4.l17.m1.2.3.3.3.1" xref="alg4.l17.m1.2.3.3.3.1.cmml">​</mo><msub id="alg4.l17.m1.2.3.3.3.3" xref="alg4.l17.m1.2.3.3.3.3.cmml"><mi mathsize="80%" id="alg4.l17.m1.2.3.3.3.3.2" xref="alg4.l17.m1.2.3.3.3.3.2.cmml">𝐐</mi><mi mathsize="80%" id="alg4.l17.m1.2.3.3.3.3.3" xref="alg4.l17.m1.2.3.3.3.3.3.cmml">j</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l17.m1.2b"><apply id="alg4.l17.m1.2.3.cmml" xref="alg4.l17.m1.2.3"><eq id="alg4.l17.m1.2.3.1.cmml" xref="alg4.l17.m1.2.3.1"></eq><apply id="alg4.l17.m1.2.3.2.cmml" xref="alg4.l17.m1.2.3.2"><csymbol cd="ambiguous" id="alg4.l17.m1.2.3.2.1.cmml" xref="alg4.l17.m1.2.3.2">subscript</csymbol><ci id="alg4.l17.m1.2.3.2.2.cmml" xref="alg4.l17.m1.2.3.2.2">𝐝𝐊</ci><ci id="alg4.l17.m1.2.3.2.3.cmml" xref="alg4.l17.m1.2.3.2.3">𝑖</ci></apply><apply id="alg4.l17.m1.2.3.3.cmml" xref="alg4.l17.m1.2.3.3"><plus id="alg4.l17.m1.2.3.3.1.cmml" xref="alg4.l17.m1.2.3.3.1"></plus><apply id="alg4.l17.m1.2.3.3.2.cmml" xref="alg4.l17.m1.2.3.3.2"><csymbol cd="ambiguous" id="alg4.l17.m1.2.3.3.2.1.cmml" xref="alg4.l17.m1.2.3.3.2">subscript</csymbol><ci id="alg4.l17.m1.2.3.3.2.2.cmml" xref="alg4.l17.m1.2.3.3.2.2">𝐝𝐊</ci><ci id="alg4.l17.m1.2.3.3.2.3.cmml" xref="alg4.l17.m1.2.3.3.2.3">𝑖</ci></apply><apply id="alg4.l17.m1.2.3.3.3.cmml" xref="alg4.l17.m1.2.3.3.3"><times id="alg4.l17.m1.2.3.3.3.1.cmml" xref="alg4.l17.m1.2.3.3.3.1"></times><apply id="alg4.l17.m1.2.3.3.3.2.cmml" xref="alg4.l17.m1.2.3.3.3.2"><csymbol cd="ambiguous" id="alg4.l17.m1.2.3.3.3.2.1.cmml" xref="alg4.l17.m1.2.3.3.3.2">superscript</csymbol><apply id="alg4.l17.m1.2.3.3.3.2.2.cmml" xref="alg4.l17.m1.2.3.3.3.2"><csymbol cd="ambiguous" id="alg4.l17.m1.2.3.3.3.2.2.1.cmml" xref="alg4.l17.m1.2.3.3.3.2">subscript</csymbol><ci id="alg4.l17.m1.2.3.3.3.2.2.2.cmml" xref="alg4.l17.m1.2.3.3.3.2.2.2">𝐝𝐒</ci><list id="alg4.l17.m1.2.2.2.3.cmml" xref="alg4.l17.m1.2.2.2.4"><ci id="alg4.l17.m1.1.1.1.1.cmml" xref="alg4.l17.m1.1.1.1.1">𝑗</ci><ci id="alg4.l17.m1.2.2.2.2.cmml" xref="alg4.l17.m1.2.2.2.2">𝑖</ci></list></apply><ci id="alg4.l17.m1.2.3.3.3.2.3.cmml" xref="alg4.l17.m1.2.3.3.3.2.3">𝑇</ci></apply><apply id="alg4.l17.m1.2.3.3.3.3.cmml" xref="alg4.l17.m1.2.3.3.3.3"><csymbol cd="ambiguous" id="alg4.l17.m1.2.3.3.3.3.1.cmml" xref="alg4.l17.m1.2.3.3.3.3">subscript</csymbol><ci id="alg4.l17.m1.2.3.3.3.3.2.cmml" xref="alg4.l17.m1.2.3.3.3.3.2">𝐐</ci><ci id="alg4.l17.m1.2.3.3.3.3.3.cmml" xref="alg4.l17.m1.2.3.3.3.3.3">𝑗</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l17.m1.2c">\mathbf{dK}_{i}=\mathbf{dK}_{i}+\mathbf{dS}_{j,i}^{T}\mathbf{Q}_{j}</annotation></semantics></math><span id="alg4.l17.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
<div id="alg4.l18" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l18.1.1.1" class="ltx_text" style="font-size:80%;">18:</span></span><span id="alg4.l18.2" class="ltx_text" style="font-size:80%;">    </span><math id="alg4.l18.m1.2" class="ltx_Math" alttext="\mathbf{dQ}_{j}=\mathbf{dQ}_{j}+\mathbf{dS}_{j,i}~{}\mathbf{K}_{i}" display="inline"><semantics id="alg4.l18.m1.2a"><mrow id="alg4.l18.m1.2.3" xref="alg4.l18.m1.2.3.cmml"><msub id="alg4.l18.m1.2.3.2" xref="alg4.l18.m1.2.3.2.cmml"><mi mathsize="80%" id="alg4.l18.m1.2.3.2.2" xref="alg4.l18.m1.2.3.2.2.cmml">𝐝𝐐</mi><mi mathsize="80%" id="alg4.l18.m1.2.3.2.3" xref="alg4.l18.m1.2.3.2.3.cmml">j</mi></msub><mo mathsize="80%" id="alg4.l18.m1.2.3.1" xref="alg4.l18.m1.2.3.1.cmml">=</mo><mrow id="alg4.l18.m1.2.3.3" xref="alg4.l18.m1.2.3.3.cmml"><msub id="alg4.l18.m1.2.3.3.2" xref="alg4.l18.m1.2.3.3.2.cmml"><mi mathsize="80%" id="alg4.l18.m1.2.3.3.2.2" xref="alg4.l18.m1.2.3.3.2.2.cmml">𝐝𝐐</mi><mi mathsize="80%" id="alg4.l18.m1.2.3.3.2.3" xref="alg4.l18.m1.2.3.3.2.3.cmml">j</mi></msub><mo mathsize="80%" id="alg4.l18.m1.2.3.3.1" xref="alg4.l18.m1.2.3.3.1.cmml">+</mo><mrow id="alg4.l18.m1.2.3.3.3" xref="alg4.l18.m1.2.3.3.3.cmml"><msub id="alg4.l18.m1.2.3.3.3.2" xref="alg4.l18.m1.2.3.3.3.2.cmml"><mi mathsize="80%" id="alg4.l18.m1.2.3.3.3.2.2" xref="alg4.l18.m1.2.3.3.3.2.2.cmml">𝐝𝐒</mi><mrow id="alg4.l18.m1.2.2.2.4" xref="alg4.l18.m1.2.2.2.3.cmml"><mi mathsize="80%" id="alg4.l18.m1.1.1.1.1" xref="alg4.l18.m1.1.1.1.1.cmml">j</mi><mo mathsize="80%" id="alg4.l18.m1.2.2.2.4.1" xref="alg4.l18.m1.2.2.2.3.cmml">,</mo><mi mathsize="80%" id="alg4.l18.m1.2.2.2.2" xref="alg4.l18.m1.2.2.2.2.cmml">i</mi></mrow></msub><mo lspace="0em" rspace="0em" id="alg4.l18.m1.2.3.3.3.1" xref="alg4.l18.m1.2.3.3.3.1.cmml">​</mo><msub id="alg4.l18.m1.2.3.3.3.3" xref="alg4.l18.m1.2.3.3.3.3.cmml"><mi mathsize="80%" id="alg4.l18.m1.2.3.3.3.3.2" xref="alg4.l18.m1.2.3.3.3.3.2.cmml">𝐊</mi><mi mathsize="80%" id="alg4.l18.m1.2.3.3.3.3.3" xref="alg4.l18.m1.2.3.3.3.3.3.cmml">i</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg4.l18.m1.2b"><apply id="alg4.l18.m1.2.3.cmml" xref="alg4.l18.m1.2.3"><eq id="alg4.l18.m1.2.3.1.cmml" xref="alg4.l18.m1.2.3.1"></eq><apply id="alg4.l18.m1.2.3.2.cmml" xref="alg4.l18.m1.2.3.2"><csymbol cd="ambiguous" id="alg4.l18.m1.2.3.2.1.cmml" xref="alg4.l18.m1.2.3.2">subscript</csymbol><ci id="alg4.l18.m1.2.3.2.2.cmml" xref="alg4.l18.m1.2.3.2.2">𝐝𝐐</ci><ci id="alg4.l18.m1.2.3.2.3.cmml" xref="alg4.l18.m1.2.3.2.3">𝑗</ci></apply><apply id="alg4.l18.m1.2.3.3.cmml" xref="alg4.l18.m1.2.3.3"><plus id="alg4.l18.m1.2.3.3.1.cmml" xref="alg4.l18.m1.2.3.3.1"></plus><apply id="alg4.l18.m1.2.3.3.2.cmml" xref="alg4.l18.m1.2.3.3.2"><csymbol cd="ambiguous" id="alg4.l18.m1.2.3.3.2.1.cmml" xref="alg4.l18.m1.2.3.3.2">subscript</csymbol><ci id="alg4.l18.m1.2.3.3.2.2.cmml" xref="alg4.l18.m1.2.3.3.2.2">𝐝𝐐</ci><ci id="alg4.l18.m1.2.3.3.2.3.cmml" xref="alg4.l18.m1.2.3.3.2.3">𝑗</ci></apply><apply id="alg4.l18.m1.2.3.3.3.cmml" xref="alg4.l18.m1.2.3.3.3"><times id="alg4.l18.m1.2.3.3.3.1.cmml" xref="alg4.l18.m1.2.3.3.3.1"></times><apply id="alg4.l18.m1.2.3.3.3.2.cmml" xref="alg4.l18.m1.2.3.3.3.2"><csymbol cd="ambiguous" id="alg4.l18.m1.2.3.3.3.2.1.cmml" xref="alg4.l18.m1.2.3.3.3.2">subscript</csymbol><ci id="alg4.l18.m1.2.3.3.3.2.2.cmml" xref="alg4.l18.m1.2.3.3.3.2.2">𝐝𝐒</ci><list id="alg4.l18.m1.2.2.2.3.cmml" xref="alg4.l18.m1.2.2.2.4"><ci id="alg4.l18.m1.1.1.1.1.cmml" xref="alg4.l18.m1.1.1.1.1">𝑗</ci><ci id="alg4.l18.m1.2.2.2.2.cmml" xref="alg4.l18.m1.2.2.2.2">𝑖</ci></list></apply><apply id="alg4.l18.m1.2.3.3.3.3.cmml" xref="alg4.l18.m1.2.3.3.3.3"><csymbol cd="ambiguous" id="alg4.l18.m1.2.3.3.3.3.1.cmml" xref="alg4.l18.m1.2.3.3.3.3">subscript</csymbol><ci id="alg4.l18.m1.2.3.3.3.3.2.cmml" xref="alg4.l18.m1.2.3.3.3.3.2">𝐊</ci><ci id="alg4.l18.m1.2.3.3.3.3.3.cmml" xref="alg4.l18.m1.2.3.3.3.3.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg4.l18.m1.2c">\mathbf{dQ}_{j}=\mathbf{dQ}_{j}+\mathbf{dS}_{j,i}~{}\mathbf{K}_{i}</annotation></semantics></math><span id="alg4.l18.3" class="ltx_text" style="font-size:80%;"> ;
{The end of the backward pass of local attention operations.}
</span>
</div>
<div id="alg4.l19" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l19.1.1.1" class="ltx_text" style="font-size:80%;">19:</span></span><span id="alg4.l19.2" class="ltx_text" style="font-size:80%;">  </span><span id="alg4.l19.3" class="ltx_text ltx_font_bold" style="font-size:80%;">end</span><span id="alg4.l19.4" class="ltx_text" style="font-size:80%;"> </span><span id="alg4.l19.5" class="ltx_text ltx_font_bold" style="font-size:80%;">for</span>
</div>
<div id="alg4.l19a" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg4.l19a.1.1.1" class="ltx_text" style="font-size:80%;">19:</span></span><span id="alg4.l19a.2" class="ltx_text" style="font-size:80%;">  </span><math id="alg4.l19a.m1.3" class="ltx_Math" alttext="\mathbf{dQ}_{G},\mathbf{dK}_{G},\mathbf{dV}_{G}" display="inline"><semantics id="alg4.l19a.m1.3a"><mrow id="alg4.l19a.m1.3.3.3" xref="alg4.l19a.m1.3.3.4.cmml"><msub id="alg4.l19a.m1.1.1.1.1" xref="alg4.l19a.m1.1.1.1.1.cmml"><mi mathsize="80%" id="alg4.l19a.m1.1.1.1.1.2" xref="alg4.l19a.m1.1.1.1.1.2.cmml">𝐝𝐐</mi><mi mathsize="80%" id="alg4.l19a.m1.1.1.1.1.3" xref="alg4.l19a.m1.1.1.1.1.3.cmml">G</mi></msub><mo mathsize="80%" id="alg4.l19a.m1.3.3.3.4" xref="alg4.l19a.m1.3.3.4.cmml">,</mo><msub id="alg4.l19a.m1.2.2.2.2" xref="alg4.l19a.m1.2.2.2.2.cmml"><mi mathsize="80%" id="alg4.l19a.m1.2.2.2.2.2" xref="alg4.l19a.m1.2.2.2.2.2.cmml">𝐝𝐊</mi><mi mathsize="80%" id="alg4.l19a.m1.2.2.2.2.3" xref="alg4.l19a.m1.2.2.2.2.3.cmml">G</mi></msub><mo mathsize="80%" id="alg4.l19a.m1.3.3.3.5" xref="alg4.l19a.m1.3.3.4.cmml">,</mo><msub id="alg4.l19a.m1.3.3.3.3" xref="alg4.l19a.m1.3.3.3.3.cmml"><mi mathsize="80%" id="alg4.l19a.m1.3.3.3.3.2" xref="alg4.l19a.m1.3.3.3.3.2.cmml">𝐝𝐕</mi><mi mathsize="80%" id="alg4.l19a.m1.3.3.3.3.3" xref="alg4.l19a.m1.3.3.3.3.3.cmml">G</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="alg4.l19a.m1.3b"><list id="alg4.l19a.m1.3.3.4.cmml" xref="alg4.l19a.m1.3.3.3"><apply id="alg4.l19a.m1.1.1.1.1.cmml" xref="alg4.l19a.m1.1.1.1.1"><csymbol cd="ambiguous" id="alg4.l19a.m1.1.1.1.1.1.cmml" xref="alg4.l19a.m1.1.1.1.1">subscript</csymbol><ci id="alg4.l19a.m1.1.1.1.1.2.cmml" xref="alg4.l19a.m1.1.1.1.1.2">𝐝𝐐</ci><ci id="alg4.l19a.m1.1.1.1.1.3.cmml" xref="alg4.l19a.m1.1.1.1.1.3">𝐺</ci></apply><apply id="alg4.l19a.m1.2.2.2.2.cmml" xref="alg4.l19a.m1.2.2.2.2"><csymbol cd="ambiguous" id="alg4.l19a.m1.2.2.2.2.1.cmml" xref="alg4.l19a.m1.2.2.2.2">subscript</csymbol><ci id="alg4.l19a.m1.2.2.2.2.2.cmml" xref="alg4.l19a.m1.2.2.2.2.2">𝐝𝐊</ci><ci id="alg4.l19a.m1.2.2.2.2.3.cmml" xref="alg4.l19a.m1.2.2.2.2.3">𝐺</ci></apply><apply id="alg4.l19a.m1.3.3.3.3.cmml" xref="alg4.l19a.m1.3.3.3.3"><csymbol cd="ambiguous" id="alg4.l19a.m1.3.3.3.3.1.cmml" xref="alg4.l19a.m1.3.3.3.3">subscript</csymbol><ci id="alg4.l19a.m1.3.3.3.3.2.cmml" xref="alg4.l19a.m1.3.3.3.3.2">𝐝𝐕</ci><ci id="alg4.l19a.m1.3.3.3.3.3.cmml" xref="alg4.l19a.m1.3.3.3.3.3">𝐺</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg4.l19a.m1.3c">\mathbf{dQ}_{G},\mathbf{dK}_{G},\mathbf{dV}_{G}</annotation></semantics></math><span id="alg4.l19a.3" class="ltx_text" style="font-size:80%;">;
</span>
</div>
</div>
<figcaption class="ltx_caption" style="font-size:80%;"><span class="ltx_tag ltx_tag_float"><span id="alg4.5.1.1" class="ltx_text ltx_font_bold">Algorithm 4</span> </span> The backward pass of GAO with overlapping</figcaption>
</figure>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Runtime Analysis of Tensor Parallelism in one Transformer Block</h2>

<div id="A2.Thmtheorem1" class="ltx_theorem ltx_theorem_theorem">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A2.Thmtheorem1.1.1.1" class="ltx_text ltx_font_bold">Theorem B.1</span></span><span id="A2.Thmtheorem1.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A2.Thmtheorem1.p1" class="ltx_para">
<p id="A2.Thmtheorem1.p1.1" class="ltx_p"><span id="A2.Thmtheorem1.p1.1.1" class="ltx_text ltx_font_italic">In a Transformer block employing Tensor Parallelism (TP) within the Megatron-V3 framework, the total runtime <math id="A2.Thmtheorem1.p1.1.1.m1.1" class="ltx_Math" alttext="T" display="inline"><semantics id="A2.Thmtheorem1.p1.1.1.m1.1a"><mi id="A2.Thmtheorem1.p1.1.1.m1.1.1" xref="A2.Thmtheorem1.p1.1.1.m1.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="A2.Thmtheorem1.p1.1.1.m1.1b"><ci id="A2.Thmtheorem1.p1.1.1.m1.1.1.cmml" xref="A2.Thmtheorem1.p1.1.1.m1.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.Thmtheorem1.p1.1.1.m1.1c">T</annotation></semantics></math> is determined by the sum of communication times for all-gather and reduce-scatter operations, and the computation times for the attention (attn) and feedforward (ffn) modules, distributed across the devices.</span></p>
</div>
</div>
<div id="A2.Thmtheorem2" class="ltx_theorem ltx_theorem_definition">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A2.Thmtheorem2.1.1.1" class="ltx_text ltx_font_bold">Definition B.2</span></span><span id="A2.Thmtheorem2.2.2" class="ltx_text ltx_font_bold"> </span>(Input Tensor and Cluster Configuration)<span id="A2.Thmtheorem2.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A2.Thmtheorem2.p1" class="ltx_para">
<p id="A2.Thmtheorem2.p1.8" class="ltx_p">Let the input tensor <math id="A2.Thmtheorem2.p1.1.m1.1" class="ltx_Math" alttext="x" display="inline"><semantics id="A2.Thmtheorem2.p1.1.m1.1a"><mi id="A2.Thmtheorem2.p1.1.m1.1.1" xref="A2.Thmtheorem2.p1.1.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="A2.Thmtheorem2.p1.1.m1.1b"><ci id="A2.Thmtheorem2.p1.1.m1.1.1.cmml" xref="A2.Thmtheorem2.p1.1.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.Thmtheorem2.p1.1.m1.1c">x</annotation></semantics></math> have dimensions <math id="A2.Thmtheorem2.p1.2.m2.4" class="ltx_Math" alttext="(B,N,Z^{\prime},d)" display="inline"><semantics id="A2.Thmtheorem2.p1.2.m2.4a"><mrow id="A2.Thmtheorem2.p1.2.m2.4.4.1" xref="A2.Thmtheorem2.p1.2.m2.4.4.2.cmml"><mo stretchy="false" id="A2.Thmtheorem2.p1.2.m2.4.4.1.2" xref="A2.Thmtheorem2.p1.2.m2.4.4.2.cmml">(</mo><mi id="A2.Thmtheorem2.p1.2.m2.1.1" xref="A2.Thmtheorem2.p1.2.m2.1.1.cmml">B</mi><mo id="A2.Thmtheorem2.p1.2.m2.4.4.1.3" xref="A2.Thmtheorem2.p1.2.m2.4.4.2.cmml">,</mo><mi id="A2.Thmtheorem2.p1.2.m2.2.2" xref="A2.Thmtheorem2.p1.2.m2.2.2.cmml">N</mi><mo id="A2.Thmtheorem2.p1.2.m2.4.4.1.4" xref="A2.Thmtheorem2.p1.2.m2.4.4.2.cmml">,</mo><msup id="A2.Thmtheorem2.p1.2.m2.4.4.1.1" xref="A2.Thmtheorem2.p1.2.m2.4.4.1.1.cmml"><mi id="A2.Thmtheorem2.p1.2.m2.4.4.1.1.2" xref="A2.Thmtheorem2.p1.2.m2.4.4.1.1.2.cmml">Z</mi><mo id="A2.Thmtheorem2.p1.2.m2.4.4.1.1.3" xref="A2.Thmtheorem2.p1.2.m2.4.4.1.1.3.cmml">′</mo></msup><mo id="A2.Thmtheorem2.p1.2.m2.4.4.1.5" xref="A2.Thmtheorem2.p1.2.m2.4.4.2.cmml">,</mo><mi id="A2.Thmtheorem2.p1.2.m2.3.3" xref="A2.Thmtheorem2.p1.2.m2.3.3.cmml">d</mi><mo stretchy="false" id="A2.Thmtheorem2.p1.2.m2.4.4.1.6" xref="A2.Thmtheorem2.p1.2.m2.4.4.2.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A2.Thmtheorem2.p1.2.m2.4b"><vector id="A2.Thmtheorem2.p1.2.m2.4.4.2.cmml" xref="A2.Thmtheorem2.p1.2.m2.4.4.1"><ci id="A2.Thmtheorem2.p1.2.m2.1.1.cmml" xref="A2.Thmtheorem2.p1.2.m2.1.1">𝐵</ci><ci id="A2.Thmtheorem2.p1.2.m2.2.2.cmml" xref="A2.Thmtheorem2.p1.2.m2.2.2">𝑁</ci><apply id="A2.Thmtheorem2.p1.2.m2.4.4.1.1.cmml" xref="A2.Thmtheorem2.p1.2.m2.4.4.1.1"><csymbol cd="ambiguous" id="A2.Thmtheorem2.p1.2.m2.4.4.1.1.1.cmml" xref="A2.Thmtheorem2.p1.2.m2.4.4.1.1">superscript</csymbol><ci id="A2.Thmtheorem2.p1.2.m2.4.4.1.1.2.cmml" xref="A2.Thmtheorem2.p1.2.m2.4.4.1.1.2">𝑍</ci><ci id="A2.Thmtheorem2.p1.2.m2.4.4.1.1.3.cmml" xref="A2.Thmtheorem2.p1.2.m2.4.4.1.1.3">′</ci></apply><ci id="A2.Thmtheorem2.p1.2.m2.3.3.cmml" xref="A2.Thmtheorem2.p1.2.m2.3.3">𝑑</ci></vector></annotation-xml><annotation encoding="application/x-tex" id="A2.Thmtheorem2.p1.2.m2.4c">(B,N,Z^{\prime},d)</annotation></semantics></math>, where <math id="A2.Thmtheorem2.p1.3.m3.1" class="ltx_Math" alttext="B" display="inline"><semantics id="A2.Thmtheorem2.p1.3.m3.1a"><mi id="A2.Thmtheorem2.p1.3.m3.1.1" xref="A2.Thmtheorem2.p1.3.m3.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="A2.Thmtheorem2.p1.3.m3.1b"><ci id="A2.Thmtheorem2.p1.3.m3.1.1.cmml" xref="A2.Thmtheorem2.p1.3.m3.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.Thmtheorem2.p1.3.m3.1c">B</annotation></semantics></math> is the batch size, <math id="A2.Thmtheorem2.p1.4.m4.1" class="ltx_Math" alttext="N" display="inline"><semantics id="A2.Thmtheorem2.p1.4.m4.1a"><mi id="A2.Thmtheorem2.p1.4.m4.1.1" xref="A2.Thmtheorem2.p1.4.m4.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="A2.Thmtheorem2.p1.4.m4.1b"><ci id="A2.Thmtheorem2.p1.4.m4.1.1.cmml" xref="A2.Thmtheorem2.p1.4.m4.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.Thmtheorem2.p1.4.m4.1c">N</annotation></semantics></math> is the sequence length, <math id="A2.Thmtheorem2.p1.5.m5.1" class="ltx_Math" alttext="Z^{\prime}" display="inline"><semantics id="A2.Thmtheorem2.p1.5.m5.1a"><msup id="A2.Thmtheorem2.p1.5.m5.1.1" xref="A2.Thmtheorem2.p1.5.m5.1.1.cmml"><mi id="A2.Thmtheorem2.p1.5.m5.1.1.2" xref="A2.Thmtheorem2.p1.5.m5.1.1.2.cmml">Z</mi><mo id="A2.Thmtheorem2.p1.5.m5.1.1.3" xref="A2.Thmtheorem2.p1.5.m5.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="A2.Thmtheorem2.p1.5.m5.1b"><apply id="A2.Thmtheorem2.p1.5.m5.1.1.cmml" xref="A2.Thmtheorem2.p1.5.m5.1.1"><csymbol cd="ambiguous" id="A2.Thmtheorem2.p1.5.m5.1.1.1.cmml" xref="A2.Thmtheorem2.p1.5.m5.1.1">superscript</csymbol><ci id="A2.Thmtheorem2.p1.5.m5.1.1.2.cmml" xref="A2.Thmtheorem2.p1.5.m5.1.1.2">𝑍</ci><ci id="A2.Thmtheorem2.p1.5.m5.1.1.3.cmml" xref="A2.Thmtheorem2.p1.5.m5.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.Thmtheorem2.p1.5.m5.1c">Z^{\prime}</annotation></semantics></math> is the number of partition heads per device, and <math id="A2.Thmtheorem2.p1.6.m6.1" class="ltx_Math" alttext="d" display="inline"><semantics id="A2.Thmtheorem2.p1.6.m6.1a"><mi id="A2.Thmtheorem2.p1.6.m6.1.1" xref="A2.Thmtheorem2.p1.6.m6.1.1.cmml">d</mi><annotation-xml encoding="MathML-Content" id="A2.Thmtheorem2.p1.6.m6.1b"><ci id="A2.Thmtheorem2.p1.6.m6.1.1.cmml" xref="A2.Thmtheorem2.p1.6.m6.1.1">𝑑</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.Thmtheorem2.p1.6.m6.1c">d</annotation></semantics></math> is the hidden dimension per attention head. The cluster bandwidth <math id="A2.Thmtheorem2.p1.7.m7.1" class="ltx_Math" alttext="b" display="inline"><semantics id="A2.Thmtheorem2.p1.7.m7.1a"><mi id="A2.Thmtheorem2.p1.7.m7.1.1" xref="A2.Thmtheorem2.p1.7.m7.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="A2.Thmtheorem2.p1.7.m7.1b"><ci id="A2.Thmtheorem2.p1.7.m7.1.1.cmml" xref="A2.Thmtheorem2.p1.7.m7.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.Thmtheorem2.p1.7.m7.1c">b</annotation></semantics></math> is assumed to be uniform across all <math id="A2.Thmtheorem2.p1.8.m8.1" class="ltx_Math" alttext="G" display="inline"><semantics id="A2.Thmtheorem2.p1.8.m8.1a"><mi id="A2.Thmtheorem2.p1.8.m8.1.1" xref="A2.Thmtheorem2.p1.8.m8.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="A2.Thmtheorem2.p1.8.m8.1b"><ci id="A2.Thmtheorem2.p1.8.m8.1.1.cmml" xref="A2.Thmtheorem2.p1.8.m8.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.Thmtheorem2.p1.8.m8.1c">G</annotation></semantics></math> devices.</p>
</div>
</div>
<div id="A2.Thmtheorem3" class="ltx_theorem ltx_theorem_lemma">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A2.Thmtheorem3.1.1.1" class="ltx_text ltx_font_bold">Lemma B.3</span></span><span id="A2.Thmtheorem3.2.2" class="ltx_text ltx_font_bold"> </span>(Communication Time)<span id="A2.Thmtheorem3.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A2.Thmtheorem3.p1" class="ltx_para">
<p id="A2.Thmtheorem3.p1.1" class="ltx_p"><span id="A2.Thmtheorem3.p1.1.1" class="ltx_text ltx_font_italic">The time <math id="A2.Thmtheorem3.p1.1.1.m1.1" class="ltx_Math" alttext="t_{\text{comm}}" display="inline"><semantics id="A2.Thmtheorem3.p1.1.1.m1.1a"><msub id="A2.Thmtheorem3.p1.1.1.m1.1.1" xref="A2.Thmtheorem3.p1.1.1.m1.1.1.cmml"><mi id="A2.Thmtheorem3.p1.1.1.m1.1.1.2" xref="A2.Thmtheorem3.p1.1.1.m1.1.1.2.cmml">t</mi><mtext class="ltx_mathvariant_italic" id="A2.Thmtheorem3.p1.1.1.m1.1.1.3" xref="A2.Thmtheorem3.p1.1.1.m1.1.1.3a.cmml">comm</mtext></msub><annotation-xml encoding="MathML-Content" id="A2.Thmtheorem3.p1.1.1.m1.1b"><apply id="A2.Thmtheorem3.p1.1.1.m1.1.1.cmml" xref="A2.Thmtheorem3.p1.1.1.m1.1.1"><csymbol cd="ambiguous" id="A2.Thmtheorem3.p1.1.1.m1.1.1.1.cmml" xref="A2.Thmtheorem3.p1.1.1.m1.1.1">subscript</csymbol><ci id="A2.Thmtheorem3.p1.1.1.m1.1.1.2.cmml" xref="A2.Thmtheorem3.p1.1.1.m1.1.1.2">𝑡</ci><ci id="A2.Thmtheorem3.p1.1.1.m1.1.1.3a.cmml" xref="A2.Thmtheorem3.p1.1.1.m1.1.1.3"><mtext class="ltx_mathvariant_italic" mathsize="70%" id="A2.Thmtheorem3.p1.1.1.m1.1.1.3.cmml" xref="A2.Thmtheorem3.p1.1.1.m1.1.1.3">comm</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.Thmtheorem3.p1.1.1.m1.1c">t_{\text{comm}}</annotation></semantics></math> for each all-gather or reduce-scatter operation in TP is given by</span></p>
<table id="A2.Ex1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A2.Ex1.m1.3" class="ltx_Math" alttext="t_{\text{comm}}=\frac{(B\times N\times Z^{\prime}\times d)\times M\times(G-1)}{b\times G}," display="block"><semantics id="A2.Ex1.m1.3a"><mrow id="A2.Ex1.m1.3.3.1" xref="A2.Ex1.m1.3.3.1.1.cmml"><mrow id="A2.Ex1.m1.3.3.1.1" xref="A2.Ex1.m1.3.3.1.1.cmml"><msub id="A2.Ex1.m1.3.3.1.1.2" xref="A2.Ex1.m1.3.3.1.1.2.cmml"><mi id="A2.Ex1.m1.3.3.1.1.2.2" xref="A2.Ex1.m1.3.3.1.1.2.2.cmml">t</mi><mtext class="ltx_mathvariant_italic" id="A2.Ex1.m1.3.3.1.1.2.3" xref="A2.Ex1.m1.3.3.1.1.2.3a.cmml">comm</mtext></msub><mo id="A2.Ex1.m1.3.3.1.1.1" xref="A2.Ex1.m1.3.3.1.1.1.cmml">=</mo><mfrac id="A2.Ex1.m1.2.2" xref="A2.Ex1.m1.2.2.cmml"><mrow id="A2.Ex1.m1.2.2.2" xref="A2.Ex1.m1.2.2.2.cmml"><mrow id="A2.Ex1.m1.1.1.1.1.1" xref="A2.Ex1.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A2.Ex1.m1.1.1.1.1.1.2" xref="A2.Ex1.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="A2.Ex1.m1.1.1.1.1.1.1" xref="A2.Ex1.m1.1.1.1.1.1.1.cmml"><mi id="A2.Ex1.m1.1.1.1.1.1.1.2" xref="A2.Ex1.m1.1.1.1.1.1.1.2.cmml">B</mi><mo lspace="0.222em" rspace="0.222em" id="A2.Ex1.m1.1.1.1.1.1.1.1" xref="A2.Ex1.m1.1.1.1.1.1.1.1.cmml">×</mo><mi id="A2.Ex1.m1.1.1.1.1.1.1.3" xref="A2.Ex1.m1.1.1.1.1.1.1.3.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="A2.Ex1.m1.1.1.1.1.1.1.1a" xref="A2.Ex1.m1.1.1.1.1.1.1.1.cmml">×</mo><msup id="A2.Ex1.m1.1.1.1.1.1.1.4" xref="A2.Ex1.m1.1.1.1.1.1.1.4.cmml"><mi id="A2.Ex1.m1.1.1.1.1.1.1.4.2" xref="A2.Ex1.m1.1.1.1.1.1.1.4.2.cmml">Z</mi><mo id="A2.Ex1.m1.1.1.1.1.1.1.4.3" xref="A2.Ex1.m1.1.1.1.1.1.1.4.3.cmml">′</mo></msup><mo lspace="0.222em" rspace="0.222em" id="A2.Ex1.m1.1.1.1.1.1.1.1b" xref="A2.Ex1.m1.1.1.1.1.1.1.1.cmml">×</mo><mi id="A2.Ex1.m1.1.1.1.1.1.1.5" xref="A2.Ex1.m1.1.1.1.1.1.1.5.cmml">d</mi></mrow><mo rspace="0.055em" stretchy="false" id="A2.Ex1.m1.1.1.1.1.1.3" xref="A2.Ex1.m1.1.1.1.1.1.1.cmml">)</mo></mrow><mo rspace="0.222em" id="A2.Ex1.m1.2.2.2.3" xref="A2.Ex1.m1.2.2.2.3.cmml">×</mo><mi id="A2.Ex1.m1.2.2.2.4" xref="A2.Ex1.m1.2.2.2.4.cmml">M</mi><mo lspace="0.222em" rspace="0.222em" id="A2.Ex1.m1.2.2.2.3a" xref="A2.Ex1.m1.2.2.2.3.cmml">×</mo><mrow id="A2.Ex1.m1.2.2.2.2.1" xref="A2.Ex1.m1.2.2.2.2.1.1.cmml"><mo stretchy="false" id="A2.Ex1.m1.2.2.2.2.1.2" xref="A2.Ex1.m1.2.2.2.2.1.1.cmml">(</mo><mrow id="A2.Ex1.m1.2.2.2.2.1.1" xref="A2.Ex1.m1.2.2.2.2.1.1.cmml"><mi id="A2.Ex1.m1.2.2.2.2.1.1.2" xref="A2.Ex1.m1.2.2.2.2.1.1.2.cmml">G</mi><mo id="A2.Ex1.m1.2.2.2.2.1.1.1" xref="A2.Ex1.m1.2.2.2.2.1.1.1.cmml">−</mo><mn id="A2.Ex1.m1.2.2.2.2.1.1.3" xref="A2.Ex1.m1.2.2.2.2.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A2.Ex1.m1.2.2.2.2.1.3" xref="A2.Ex1.m1.2.2.2.2.1.1.cmml">)</mo></mrow></mrow><mrow id="A2.Ex1.m1.2.2.4" xref="A2.Ex1.m1.2.2.4.cmml"><mi id="A2.Ex1.m1.2.2.4.2" xref="A2.Ex1.m1.2.2.4.2.cmml">b</mi><mo lspace="0.222em" rspace="0.222em" id="A2.Ex1.m1.2.2.4.1" xref="A2.Ex1.m1.2.2.4.1.cmml">×</mo><mi id="A2.Ex1.m1.2.2.4.3" xref="A2.Ex1.m1.2.2.4.3.cmml">G</mi></mrow></mfrac></mrow><mo id="A2.Ex1.m1.3.3.1.2" xref="A2.Ex1.m1.3.3.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="A2.Ex1.m1.3b"><apply id="A2.Ex1.m1.3.3.1.1.cmml" xref="A2.Ex1.m1.3.3.1"><eq id="A2.Ex1.m1.3.3.1.1.1.cmml" xref="A2.Ex1.m1.3.3.1.1.1"></eq><apply id="A2.Ex1.m1.3.3.1.1.2.cmml" xref="A2.Ex1.m1.3.3.1.1.2"><csymbol cd="ambiguous" id="A2.Ex1.m1.3.3.1.1.2.1.cmml" xref="A2.Ex1.m1.3.3.1.1.2">subscript</csymbol><ci id="A2.Ex1.m1.3.3.1.1.2.2.cmml" xref="A2.Ex1.m1.3.3.1.1.2.2">𝑡</ci><ci id="A2.Ex1.m1.3.3.1.1.2.3a.cmml" xref="A2.Ex1.m1.3.3.1.1.2.3"><mtext class="ltx_mathvariant_italic" mathsize="70%" id="A2.Ex1.m1.3.3.1.1.2.3.cmml" xref="A2.Ex1.m1.3.3.1.1.2.3">comm</mtext></ci></apply><apply id="A2.Ex1.m1.2.2.cmml" xref="A2.Ex1.m1.2.2"><divide id="A2.Ex1.m1.2.2.3.cmml" xref="A2.Ex1.m1.2.2"></divide><apply id="A2.Ex1.m1.2.2.2.cmml" xref="A2.Ex1.m1.2.2.2"><times id="A2.Ex1.m1.2.2.2.3.cmml" xref="A2.Ex1.m1.2.2.2.3"></times><apply id="A2.Ex1.m1.1.1.1.1.1.1.cmml" xref="A2.Ex1.m1.1.1.1.1.1"><times id="A2.Ex1.m1.1.1.1.1.1.1.1.cmml" xref="A2.Ex1.m1.1.1.1.1.1.1.1"></times><ci id="A2.Ex1.m1.1.1.1.1.1.1.2.cmml" xref="A2.Ex1.m1.1.1.1.1.1.1.2">𝐵</ci><ci id="A2.Ex1.m1.1.1.1.1.1.1.3.cmml" xref="A2.Ex1.m1.1.1.1.1.1.1.3">𝑁</ci><apply id="A2.Ex1.m1.1.1.1.1.1.1.4.cmml" xref="A2.Ex1.m1.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="A2.Ex1.m1.1.1.1.1.1.1.4.1.cmml" xref="A2.Ex1.m1.1.1.1.1.1.1.4">superscript</csymbol><ci id="A2.Ex1.m1.1.1.1.1.1.1.4.2.cmml" xref="A2.Ex1.m1.1.1.1.1.1.1.4.2">𝑍</ci><ci id="A2.Ex1.m1.1.1.1.1.1.1.4.3.cmml" xref="A2.Ex1.m1.1.1.1.1.1.1.4.3">′</ci></apply><ci id="A2.Ex1.m1.1.1.1.1.1.1.5.cmml" xref="A2.Ex1.m1.1.1.1.1.1.1.5">𝑑</ci></apply><ci id="A2.Ex1.m1.2.2.2.4.cmml" xref="A2.Ex1.m1.2.2.2.4">𝑀</ci><apply id="A2.Ex1.m1.2.2.2.2.1.1.cmml" xref="A2.Ex1.m1.2.2.2.2.1"><minus id="A2.Ex1.m1.2.2.2.2.1.1.1.cmml" xref="A2.Ex1.m1.2.2.2.2.1.1.1"></minus><ci id="A2.Ex1.m1.2.2.2.2.1.1.2.cmml" xref="A2.Ex1.m1.2.2.2.2.1.1.2">𝐺</ci><cn type="integer" id="A2.Ex1.m1.2.2.2.2.1.1.3.cmml" xref="A2.Ex1.m1.2.2.2.2.1.1.3">1</cn></apply></apply><apply id="A2.Ex1.m1.2.2.4.cmml" xref="A2.Ex1.m1.2.2.4"><times id="A2.Ex1.m1.2.2.4.1.cmml" xref="A2.Ex1.m1.2.2.4.1"></times><ci id="A2.Ex1.m1.2.2.4.2.cmml" xref="A2.Ex1.m1.2.2.4.2">𝑏</ci><ci id="A2.Ex1.m1.2.2.4.3.cmml" xref="A2.Ex1.m1.2.2.4.3">𝐺</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.Ex1.m1.3c">t_{\text{comm}}=\frac{(B\times N\times Z^{\prime}\times d)\times M\times(G-1)}{b\times G},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="A2.Thmtheorem3.p1.2" class="ltx_p"><span id="A2.Thmtheorem3.p1.2.1" class="ltx_text ltx_font_italic">where <math id="A2.Thmtheorem3.p1.2.1.m1.1" class="ltx_Math" alttext="M" display="inline"><semantics id="A2.Thmtheorem3.p1.2.1.m1.1a"><mi id="A2.Thmtheorem3.p1.2.1.m1.1.1" xref="A2.Thmtheorem3.p1.2.1.m1.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="A2.Thmtheorem3.p1.2.1.m1.1b"><ci id="A2.Thmtheorem3.p1.2.1.m1.1.1.cmml" xref="A2.Thmtheorem3.p1.2.1.m1.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.Thmtheorem3.p1.2.1.m1.1c">M</annotation></semantics></math> represents the number of bits required to store one tensor element.</span></p>
</div>
</div>
<div id="A2.Thmtheorem4" class="ltx_theorem ltx_theorem_proposition">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A2.Thmtheorem4.1.1.1" class="ltx_text ltx_font_bold">Proposition B.4</span></span><span id="A2.Thmtheorem4.2.2" class="ltx_text ltx_font_bold"> </span>(Runtime Calculation)<span id="A2.Thmtheorem4.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A2.Thmtheorem4.p1" class="ltx_para">
<p id="A2.Thmtheorem4.p1.1" class="ltx_p"><span id="A2.Thmtheorem4.p1.1.1" class="ltx_text ltx_font_italic">The total runtime <math id="A2.Thmtheorem4.p1.1.1.m1.1" class="ltx_Math" alttext="T" display="inline"><semantics id="A2.Thmtheorem4.p1.1.1.m1.1a"><mi id="A2.Thmtheorem4.p1.1.1.m1.1.1" xref="A2.Thmtheorem4.p1.1.1.m1.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="A2.Thmtheorem4.p1.1.1.m1.1b"><ci id="A2.Thmtheorem4.p1.1.1.m1.1.1.cmml" xref="A2.Thmtheorem4.p1.1.1.m1.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.Thmtheorem4.p1.1.1.m1.1c">T</annotation></semantics></math> for processing one Transformer block under TP is</span></p>
<table id="A2.Ex2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A2.Ex2.m1.1" class="ltx_Math" alttext="T=4\times t_{\text{comm}}+\frac{T_{\text{attn}}}{G}+\frac{T_{\text{ffn}}}{G}," display="block"><semantics id="A2.Ex2.m1.1a"><mrow id="A2.Ex2.m1.1.1.1" xref="A2.Ex2.m1.1.1.1.1.cmml"><mrow id="A2.Ex2.m1.1.1.1.1" xref="A2.Ex2.m1.1.1.1.1.cmml"><mi id="A2.Ex2.m1.1.1.1.1.2" xref="A2.Ex2.m1.1.1.1.1.2.cmml">T</mi><mo id="A2.Ex2.m1.1.1.1.1.1" xref="A2.Ex2.m1.1.1.1.1.1.cmml">=</mo><mrow id="A2.Ex2.m1.1.1.1.1.3" xref="A2.Ex2.m1.1.1.1.1.3.cmml"><mrow id="A2.Ex2.m1.1.1.1.1.3.2" xref="A2.Ex2.m1.1.1.1.1.3.2.cmml"><mn id="A2.Ex2.m1.1.1.1.1.3.2.2" xref="A2.Ex2.m1.1.1.1.1.3.2.2.cmml">4</mn><mo lspace="0.222em" rspace="0.222em" id="A2.Ex2.m1.1.1.1.1.3.2.1" xref="A2.Ex2.m1.1.1.1.1.3.2.1.cmml">×</mo><msub id="A2.Ex2.m1.1.1.1.1.3.2.3" xref="A2.Ex2.m1.1.1.1.1.3.2.3.cmml"><mi id="A2.Ex2.m1.1.1.1.1.3.2.3.2" xref="A2.Ex2.m1.1.1.1.1.3.2.3.2.cmml">t</mi><mtext class="ltx_mathvariant_italic" id="A2.Ex2.m1.1.1.1.1.3.2.3.3" xref="A2.Ex2.m1.1.1.1.1.3.2.3.3a.cmml">comm</mtext></msub></mrow><mo id="A2.Ex2.m1.1.1.1.1.3.1" xref="A2.Ex2.m1.1.1.1.1.3.1.cmml">+</mo><mfrac id="A2.Ex2.m1.1.1.1.1.3.3" xref="A2.Ex2.m1.1.1.1.1.3.3.cmml"><msub id="A2.Ex2.m1.1.1.1.1.3.3.2" xref="A2.Ex2.m1.1.1.1.1.3.3.2.cmml"><mi id="A2.Ex2.m1.1.1.1.1.3.3.2.2" xref="A2.Ex2.m1.1.1.1.1.3.3.2.2.cmml">T</mi><mtext class="ltx_mathvariant_italic" id="A2.Ex2.m1.1.1.1.1.3.3.2.3" xref="A2.Ex2.m1.1.1.1.1.3.3.2.3a.cmml">attn</mtext></msub><mi id="A2.Ex2.m1.1.1.1.1.3.3.3" xref="A2.Ex2.m1.1.1.1.1.3.3.3.cmml">G</mi></mfrac><mo id="A2.Ex2.m1.1.1.1.1.3.1a" xref="A2.Ex2.m1.1.1.1.1.3.1.cmml">+</mo><mfrac id="A2.Ex2.m1.1.1.1.1.3.4" xref="A2.Ex2.m1.1.1.1.1.3.4.cmml"><msub id="A2.Ex2.m1.1.1.1.1.3.4.2" xref="A2.Ex2.m1.1.1.1.1.3.4.2.cmml"><mi id="A2.Ex2.m1.1.1.1.1.3.4.2.2" xref="A2.Ex2.m1.1.1.1.1.3.4.2.2.cmml">T</mi><mtext class="ltx_mathvariant_italic" id="A2.Ex2.m1.1.1.1.1.3.4.2.3" xref="A2.Ex2.m1.1.1.1.1.3.4.2.3a.cmml">ffn</mtext></msub><mi id="A2.Ex2.m1.1.1.1.1.3.4.3" xref="A2.Ex2.m1.1.1.1.1.3.4.3.cmml">G</mi></mfrac></mrow></mrow><mo id="A2.Ex2.m1.1.1.1.2" xref="A2.Ex2.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="A2.Ex2.m1.1b"><apply id="A2.Ex2.m1.1.1.1.1.cmml" xref="A2.Ex2.m1.1.1.1"><eq id="A2.Ex2.m1.1.1.1.1.1.cmml" xref="A2.Ex2.m1.1.1.1.1.1"></eq><ci id="A2.Ex2.m1.1.1.1.1.2.cmml" xref="A2.Ex2.m1.1.1.1.1.2">𝑇</ci><apply id="A2.Ex2.m1.1.1.1.1.3.cmml" xref="A2.Ex2.m1.1.1.1.1.3"><plus id="A2.Ex2.m1.1.1.1.1.3.1.cmml" xref="A2.Ex2.m1.1.1.1.1.3.1"></plus><apply id="A2.Ex2.m1.1.1.1.1.3.2.cmml" xref="A2.Ex2.m1.1.1.1.1.3.2"><times id="A2.Ex2.m1.1.1.1.1.3.2.1.cmml" xref="A2.Ex2.m1.1.1.1.1.3.2.1"></times><cn type="integer" id="A2.Ex2.m1.1.1.1.1.3.2.2.cmml" xref="A2.Ex2.m1.1.1.1.1.3.2.2">4</cn><apply id="A2.Ex2.m1.1.1.1.1.3.2.3.cmml" xref="A2.Ex2.m1.1.1.1.1.3.2.3"><csymbol cd="ambiguous" id="A2.Ex2.m1.1.1.1.1.3.2.3.1.cmml" xref="A2.Ex2.m1.1.1.1.1.3.2.3">subscript</csymbol><ci id="A2.Ex2.m1.1.1.1.1.3.2.3.2.cmml" xref="A2.Ex2.m1.1.1.1.1.3.2.3.2">𝑡</ci><ci id="A2.Ex2.m1.1.1.1.1.3.2.3.3a.cmml" xref="A2.Ex2.m1.1.1.1.1.3.2.3.3"><mtext class="ltx_mathvariant_italic" mathsize="70%" id="A2.Ex2.m1.1.1.1.1.3.2.3.3.cmml" xref="A2.Ex2.m1.1.1.1.1.3.2.3.3">comm</mtext></ci></apply></apply><apply id="A2.Ex2.m1.1.1.1.1.3.3.cmml" xref="A2.Ex2.m1.1.1.1.1.3.3"><divide id="A2.Ex2.m1.1.1.1.1.3.3.1.cmml" xref="A2.Ex2.m1.1.1.1.1.3.3"></divide><apply id="A2.Ex2.m1.1.1.1.1.3.3.2.cmml" xref="A2.Ex2.m1.1.1.1.1.3.3.2"><csymbol cd="ambiguous" id="A2.Ex2.m1.1.1.1.1.3.3.2.1.cmml" xref="A2.Ex2.m1.1.1.1.1.3.3.2">subscript</csymbol><ci id="A2.Ex2.m1.1.1.1.1.3.3.2.2.cmml" xref="A2.Ex2.m1.1.1.1.1.3.3.2.2">𝑇</ci><ci id="A2.Ex2.m1.1.1.1.1.3.3.2.3a.cmml" xref="A2.Ex2.m1.1.1.1.1.3.3.2.3"><mtext class="ltx_mathvariant_italic" mathsize="70%" id="A2.Ex2.m1.1.1.1.1.3.3.2.3.cmml" xref="A2.Ex2.m1.1.1.1.1.3.3.2.3">attn</mtext></ci></apply><ci id="A2.Ex2.m1.1.1.1.1.3.3.3.cmml" xref="A2.Ex2.m1.1.1.1.1.3.3.3">𝐺</ci></apply><apply id="A2.Ex2.m1.1.1.1.1.3.4.cmml" xref="A2.Ex2.m1.1.1.1.1.3.4"><divide id="A2.Ex2.m1.1.1.1.1.3.4.1.cmml" xref="A2.Ex2.m1.1.1.1.1.3.4"></divide><apply id="A2.Ex2.m1.1.1.1.1.3.4.2.cmml" xref="A2.Ex2.m1.1.1.1.1.3.4.2"><csymbol cd="ambiguous" id="A2.Ex2.m1.1.1.1.1.3.4.2.1.cmml" xref="A2.Ex2.m1.1.1.1.1.3.4.2">subscript</csymbol><ci id="A2.Ex2.m1.1.1.1.1.3.4.2.2.cmml" xref="A2.Ex2.m1.1.1.1.1.3.4.2.2">𝑇</ci><ci id="A2.Ex2.m1.1.1.1.1.3.4.2.3a.cmml" xref="A2.Ex2.m1.1.1.1.1.3.4.2.3"><mtext class="ltx_mathvariant_italic" mathsize="70%" id="A2.Ex2.m1.1.1.1.1.3.4.2.3.cmml" xref="A2.Ex2.m1.1.1.1.1.3.4.2.3">ffn</mtext></ci></apply><ci id="A2.Ex2.m1.1.1.1.1.3.4.3.cmml" xref="A2.Ex2.m1.1.1.1.1.3.4.3">𝐺</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.Ex2.m1.1c">T=4\times t_{\text{comm}}+\frac{T_{\text{attn}}}{G}+\frac{T_{\text{ffn}}}{G},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="A2.Thmtheorem4.p1.2" class="ltx_p"><span id="A2.Thmtheorem4.p1.2.1" class="ltx_text ltx_font_italic">accounting for two all-gather and two reduce-scatter operations, and the parallelized computation times for the attention (attn) and feedforward (ffn) modules.</span></p>
</div>
</div>
<div id="A2.Thmtheorem5" class="ltx_theorem ltx_theorem_remark">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A2.Thmtheorem5.1.1.1" class="ltx_text ltx_font_italic">Remark B.5</span></span><span id="A2.Thmtheorem5.2.2" class="ltx_text ltx_font_italic">.</span>
</h6>
<div id="A2.Thmtheorem5.p1" class="ltx_para">
<p id="A2.Thmtheorem5.p1.1" class="ltx_p">This analysis assumes the use of full attention mechanisms in the Transformer block. It does not account for sparse, approximate, or causal attention methods that could alter computational and communication complexities.</p>
</div>
</div>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Runtime Analysis of BurstAttention in One Transformer Block</h2>

<div id="A3.Thmtheorem1" class="ltx_theorem ltx_theorem_theorem">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A3.Thmtheorem1.1.1.1" class="ltx_text ltx_font_bold">Theorem C.1</span></span><span id="A3.Thmtheorem1.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A3.Thmtheorem1.p1" class="ltx_para">
<p id="A3.Thmtheorem1.p1.1" class="ltx_p"><span id="A3.Thmtheorem1.p1.1.1" class="ltx_text ltx_font_italic">In the BurstAttention framework, the total runtime for a given Transformer block is influenced by the communication and computation times for both the attention and feedforward modules. The runtime accounts for asymmetric communication processes in both forward and backward passes.</span></p>
</div>
</div>
<div id="A3.Thmtheorem2" class="ltx_theorem ltx_theorem_definition">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A3.Thmtheorem2.1.1.1" class="ltx_text ltx_font_bold">Definition C.2</span></span><span id="A3.Thmtheorem2.2.2" class="ltx_text ltx_font_bold"> </span>(Input Tensor and Cluster Configuration)<span id="A3.Thmtheorem2.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A3.Thmtheorem2.p1" class="ltx_para">
<p id="A3.Thmtheorem2.p1.9" class="ltx_p">Let the input tensor <math id="A3.Thmtheorem2.p1.1.m1.1" class="ltx_Math" alttext="x" display="inline"><semantics id="A3.Thmtheorem2.p1.1.m1.1a"><mi id="A3.Thmtheorem2.p1.1.m1.1.1" xref="A3.Thmtheorem2.p1.1.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem2.p1.1.m1.1b"><ci id="A3.Thmtheorem2.p1.1.m1.1.1.cmml" xref="A3.Thmtheorem2.p1.1.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem2.p1.1.m1.1c">x</annotation></semantics></math> have dimensions <math id="A3.Thmtheorem2.p1.2.m2.4" class="ltx_Math" alttext="(B,N^{\prime},Z,d)" display="inline"><semantics id="A3.Thmtheorem2.p1.2.m2.4a"><mrow id="A3.Thmtheorem2.p1.2.m2.4.4.1" xref="A3.Thmtheorem2.p1.2.m2.4.4.2.cmml"><mo stretchy="false" id="A3.Thmtheorem2.p1.2.m2.4.4.1.2" xref="A3.Thmtheorem2.p1.2.m2.4.4.2.cmml">(</mo><mi id="A3.Thmtheorem2.p1.2.m2.1.1" xref="A3.Thmtheorem2.p1.2.m2.1.1.cmml">B</mi><mo id="A3.Thmtheorem2.p1.2.m2.4.4.1.3" xref="A3.Thmtheorem2.p1.2.m2.4.4.2.cmml">,</mo><msup id="A3.Thmtheorem2.p1.2.m2.4.4.1.1" xref="A3.Thmtheorem2.p1.2.m2.4.4.1.1.cmml"><mi id="A3.Thmtheorem2.p1.2.m2.4.4.1.1.2" xref="A3.Thmtheorem2.p1.2.m2.4.4.1.1.2.cmml">N</mi><mo id="A3.Thmtheorem2.p1.2.m2.4.4.1.1.3" xref="A3.Thmtheorem2.p1.2.m2.4.4.1.1.3.cmml">′</mo></msup><mo id="A3.Thmtheorem2.p1.2.m2.4.4.1.4" xref="A3.Thmtheorem2.p1.2.m2.4.4.2.cmml">,</mo><mi id="A3.Thmtheorem2.p1.2.m2.2.2" xref="A3.Thmtheorem2.p1.2.m2.2.2.cmml">Z</mi><mo id="A3.Thmtheorem2.p1.2.m2.4.4.1.5" xref="A3.Thmtheorem2.p1.2.m2.4.4.2.cmml">,</mo><mi id="A3.Thmtheorem2.p1.2.m2.3.3" xref="A3.Thmtheorem2.p1.2.m2.3.3.cmml">d</mi><mo stretchy="false" id="A3.Thmtheorem2.p1.2.m2.4.4.1.6" xref="A3.Thmtheorem2.p1.2.m2.4.4.2.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem2.p1.2.m2.4b"><vector id="A3.Thmtheorem2.p1.2.m2.4.4.2.cmml" xref="A3.Thmtheorem2.p1.2.m2.4.4.1"><ci id="A3.Thmtheorem2.p1.2.m2.1.1.cmml" xref="A3.Thmtheorem2.p1.2.m2.1.1">𝐵</ci><apply id="A3.Thmtheorem2.p1.2.m2.4.4.1.1.cmml" xref="A3.Thmtheorem2.p1.2.m2.4.4.1.1"><csymbol cd="ambiguous" id="A3.Thmtheorem2.p1.2.m2.4.4.1.1.1.cmml" xref="A3.Thmtheorem2.p1.2.m2.4.4.1.1">superscript</csymbol><ci id="A3.Thmtheorem2.p1.2.m2.4.4.1.1.2.cmml" xref="A3.Thmtheorem2.p1.2.m2.4.4.1.1.2">𝑁</ci><ci id="A3.Thmtheorem2.p1.2.m2.4.4.1.1.3.cmml" xref="A3.Thmtheorem2.p1.2.m2.4.4.1.1.3">′</ci></apply><ci id="A3.Thmtheorem2.p1.2.m2.2.2.cmml" xref="A3.Thmtheorem2.p1.2.m2.2.2">𝑍</ci><ci id="A3.Thmtheorem2.p1.2.m2.3.3.cmml" xref="A3.Thmtheorem2.p1.2.m2.3.3">𝑑</ci></vector></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem2.p1.2.m2.4c">(B,N^{\prime},Z,d)</annotation></semantics></math>, where <math id="A3.Thmtheorem2.p1.3.m3.1" class="ltx_Math" alttext="B" display="inline"><semantics id="A3.Thmtheorem2.p1.3.m3.1a"><mi id="A3.Thmtheorem2.p1.3.m3.1.1" xref="A3.Thmtheorem2.p1.3.m3.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem2.p1.3.m3.1b"><ci id="A3.Thmtheorem2.p1.3.m3.1.1.cmml" xref="A3.Thmtheorem2.p1.3.m3.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem2.p1.3.m3.1c">B</annotation></semantics></math> is the batch size, <math id="A3.Thmtheorem2.p1.4.m4.1" class="ltx_Math" alttext="N^{\prime}" display="inline"><semantics id="A3.Thmtheorem2.p1.4.m4.1a"><msup id="A3.Thmtheorem2.p1.4.m4.1.1" xref="A3.Thmtheorem2.p1.4.m4.1.1.cmml"><mi id="A3.Thmtheorem2.p1.4.m4.1.1.2" xref="A3.Thmtheorem2.p1.4.m4.1.1.2.cmml">N</mi><mo id="A3.Thmtheorem2.p1.4.m4.1.1.3" xref="A3.Thmtheorem2.p1.4.m4.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem2.p1.4.m4.1b"><apply id="A3.Thmtheorem2.p1.4.m4.1.1.cmml" xref="A3.Thmtheorem2.p1.4.m4.1.1"><csymbol cd="ambiguous" id="A3.Thmtheorem2.p1.4.m4.1.1.1.cmml" xref="A3.Thmtheorem2.p1.4.m4.1.1">superscript</csymbol><ci id="A3.Thmtheorem2.p1.4.m4.1.1.2.cmml" xref="A3.Thmtheorem2.p1.4.m4.1.1.2">𝑁</ci><ci id="A3.Thmtheorem2.p1.4.m4.1.1.3.cmml" xref="A3.Thmtheorem2.p1.4.m4.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem2.p1.4.m4.1c">N^{\prime}</annotation></semantics></math> is the partitioned sequence length per device, <math id="A3.Thmtheorem2.p1.5.m5.1" class="ltx_Math" alttext="Z" display="inline"><semantics id="A3.Thmtheorem2.p1.5.m5.1a"><mi id="A3.Thmtheorem2.p1.5.m5.1.1" xref="A3.Thmtheorem2.p1.5.m5.1.1.cmml">Z</mi><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem2.p1.5.m5.1b"><ci id="A3.Thmtheorem2.p1.5.m5.1.1.cmml" xref="A3.Thmtheorem2.p1.5.m5.1.1">𝑍</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem2.p1.5.m5.1c">Z</annotation></semantics></math> is the number of attention heads per device, and <math id="A3.Thmtheorem2.p1.6.m6.1" class="ltx_Math" alttext="d" display="inline"><semantics id="A3.Thmtheorem2.p1.6.m6.1a"><mi id="A3.Thmtheorem2.p1.6.m6.1.1" xref="A3.Thmtheorem2.p1.6.m6.1.1.cmml">d</mi><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem2.p1.6.m6.1b"><ci id="A3.Thmtheorem2.p1.6.m6.1.1.cmml" xref="A3.Thmtheorem2.p1.6.m6.1.1">𝑑</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem2.p1.6.m6.1c">d</annotation></semantics></math> is the hidden dimension per attention head. The cluster’s uniform bandwidth is <math id="A3.Thmtheorem2.p1.7.m7.1" class="ltx_Math" alttext="b" display="inline"><semantics id="A3.Thmtheorem2.p1.7.m7.1a"><mi id="A3.Thmtheorem2.p1.7.m7.1.1" xref="A3.Thmtheorem2.p1.7.m7.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem2.p1.7.m7.1b"><ci id="A3.Thmtheorem2.p1.7.m7.1.1.cmml" xref="A3.Thmtheorem2.p1.7.m7.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem2.p1.7.m7.1c">b</annotation></semantics></math> across all <math id="A3.Thmtheorem2.p1.8.m8.1" class="ltx_Math" alttext="G" display="inline"><semantics id="A3.Thmtheorem2.p1.8.m8.1a"><mi id="A3.Thmtheorem2.p1.8.m8.1.1" xref="A3.Thmtheorem2.p1.8.m8.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem2.p1.8.m8.1b"><ci id="A3.Thmtheorem2.p1.8.m8.1.1.cmml" xref="A3.Thmtheorem2.p1.8.m8.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem2.p1.8.m8.1c">G</annotation></semantics></math> devices, and <math id="A3.Thmtheorem2.p1.9.m9.1" class="ltx_Math" alttext="d_{ffn}" display="inline"><semantics id="A3.Thmtheorem2.p1.9.m9.1a"><msub id="A3.Thmtheorem2.p1.9.m9.1.1" xref="A3.Thmtheorem2.p1.9.m9.1.1.cmml"><mi id="A3.Thmtheorem2.p1.9.m9.1.1.2" xref="A3.Thmtheorem2.p1.9.m9.1.1.2.cmml">d</mi><mrow id="A3.Thmtheorem2.p1.9.m9.1.1.3" xref="A3.Thmtheorem2.p1.9.m9.1.1.3.cmml"><mi id="A3.Thmtheorem2.p1.9.m9.1.1.3.2" xref="A3.Thmtheorem2.p1.9.m9.1.1.3.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="A3.Thmtheorem2.p1.9.m9.1.1.3.1" xref="A3.Thmtheorem2.p1.9.m9.1.1.3.1.cmml">​</mo><mi id="A3.Thmtheorem2.p1.9.m9.1.1.3.3" xref="A3.Thmtheorem2.p1.9.m9.1.1.3.3.cmml">f</mi><mo lspace="0em" rspace="0em" id="A3.Thmtheorem2.p1.9.m9.1.1.3.1a" xref="A3.Thmtheorem2.p1.9.m9.1.1.3.1.cmml">​</mo><mi id="A3.Thmtheorem2.p1.9.m9.1.1.3.4" xref="A3.Thmtheorem2.p1.9.m9.1.1.3.4.cmml">n</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem2.p1.9.m9.1b"><apply id="A3.Thmtheorem2.p1.9.m9.1.1.cmml" xref="A3.Thmtheorem2.p1.9.m9.1.1"><csymbol cd="ambiguous" id="A3.Thmtheorem2.p1.9.m9.1.1.1.cmml" xref="A3.Thmtheorem2.p1.9.m9.1.1">subscript</csymbol><ci id="A3.Thmtheorem2.p1.9.m9.1.1.2.cmml" xref="A3.Thmtheorem2.p1.9.m9.1.1.2">𝑑</ci><apply id="A3.Thmtheorem2.p1.9.m9.1.1.3.cmml" xref="A3.Thmtheorem2.p1.9.m9.1.1.3"><times id="A3.Thmtheorem2.p1.9.m9.1.1.3.1.cmml" xref="A3.Thmtheorem2.p1.9.m9.1.1.3.1"></times><ci id="A3.Thmtheorem2.p1.9.m9.1.1.3.2.cmml" xref="A3.Thmtheorem2.p1.9.m9.1.1.3.2">𝑓</ci><ci id="A3.Thmtheorem2.p1.9.m9.1.1.3.3.cmml" xref="A3.Thmtheorem2.p1.9.m9.1.1.3.3">𝑓</ci><ci id="A3.Thmtheorem2.p1.9.m9.1.1.3.4.cmml" xref="A3.Thmtheorem2.p1.9.m9.1.1.3.4">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem2.p1.9.m9.1c">d_{ffn}</annotation></semantics></math> denotes the intermediate dimension of the feedforward layer.</p>
</div>
</div>
<div id="A3.Thmtheorem3" class="ltx_theorem ltx_theorem_lemma">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A3.Thmtheorem3.1.1.1" class="ltx_text ltx_font_bold">Lemma C.3</span></span><span id="A3.Thmtheorem3.2.2" class="ltx_text ltx_font_bold"> </span>(Activation Communication Time in BurstAttention)<span id="A3.Thmtheorem3.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A3.Thmtheorem3.p1" class="ltx_para">
<p id="A3.Thmtheorem3.p1.8" class="ltx_p"><span id="A3.Thmtheorem3.p1.8.8" class="ltx_text ltx_font_italic">In BurstAttention, there are two ring-style communications for key (<math id="A3.Thmtheorem3.p1.1.1.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="A3.Thmtheorem3.p1.1.1.m1.1a"><mi id="A3.Thmtheorem3.p1.1.1.m1.1.1" xref="A3.Thmtheorem3.p1.1.1.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem3.p1.1.1.m1.1b"><ci id="A3.Thmtheorem3.p1.1.1.m1.1.1.cmml" xref="A3.Thmtheorem3.p1.1.1.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem3.p1.1.1.m1.1c">K</annotation></semantics></math>) and value (<math id="A3.Thmtheorem3.p1.2.2.m2.1" class="ltx_Math" alttext="V" display="inline"><semantics id="A3.Thmtheorem3.p1.2.2.m2.1a"><mi id="A3.Thmtheorem3.p1.2.2.m2.1.1" xref="A3.Thmtheorem3.p1.2.2.m2.1.1.cmml">V</mi><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem3.p1.2.2.m2.1b"><ci id="A3.Thmtheorem3.p1.2.2.m2.1.1.cmml" xref="A3.Thmtheorem3.p1.2.2.m2.1.1">𝑉</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem3.p1.2.2.m2.1c">V</annotation></semantics></math>) activations and five for query (<math id="A3.Thmtheorem3.p1.3.3.m3.1" class="ltx_Math" alttext="Q" display="inline"><semantics id="A3.Thmtheorem3.p1.3.3.m3.1a"><mi id="A3.Thmtheorem3.p1.3.3.m3.1.1" xref="A3.Thmtheorem3.p1.3.3.m3.1.1.cmml">Q</mi><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem3.p1.3.3.m3.1b"><ci id="A3.Thmtheorem3.p1.3.3.m3.1.1.cmml" xref="A3.Thmtheorem3.p1.3.3.m3.1.1">𝑄</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem3.p1.3.3.m3.1c">Q</annotation></semantics></math>), gradient with respect to <math id="A3.Thmtheorem3.p1.4.4.m4.1" class="ltx_Math" alttext="Q" display="inline"><semantics id="A3.Thmtheorem3.p1.4.4.m4.1a"><mi id="A3.Thmtheorem3.p1.4.4.m4.1.1" xref="A3.Thmtheorem3.p1.4.4.m4.1.1.cmml">Q</mi><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem3.p1.4.4.m4.1b"><ci id="A3.Thmtheorem3.p1.4.4.m4.1.1.cmml" xref="A3.Thmtheorem3.p1.4.4.m4.1.1">𝑄</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem3.p1.4.4.m4.1c">Q</annotation></semantics></math> (<math id="A3.Thmtheorem3.p1.5.5.m5.1" class="ltx_Math" alttext="d_{Q}" display="inline"><semantics id="A3.Thmtheorem3.p1.5.5.m5.1a"><msub id="A3.Thmtheorem3.p1.5.5.m5.1.1" xref="A3.Thmtheorem3.p1.5.5.m5.1.1.cmml"><mi id="A3.Thmtheorem3.p1.5.5.m5.1.1.2" xref="A3.Thmtheorem3.p1.5.5.m5.1.1.2.cmml">d</mi><mi id="A3.Thmtheorem3.p1.5.5.m5.1.1.3" xref="A3.Thmtheorem3.p1.5.5.m5.1.1.3.cmml">Q</mi></msub><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem3.p1.5.5.m5.1b"><apply id="A3.Thmtheorem3.p1.5.5.m5.1.1.cmml" xref="A3.Thmtheorem3.p1.5.5.m5.1.1"><csymbol cd="ambiguous" id="A3.Thmtheorem3.p1.5.5.m5.1.1.1.cmml" xref="A3.Thmtheorem3.p1.5.5.m5.1.1">subscript</csymbol><ci id="A3.Thmtheorem3.p1.5.5.m5.1.1.2.cmml" xref="A3.Thmtheorem3.p1.5.5.m5.1.1.2">𝑑</ci><ci id="A3.Thmtheorem3.p1.5.5.m5.1.1.3.cmml" xref="A3.Thmtheorem3.p1.5.5.m5.1.1.3">𝑄</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem3.p1.5.5.m5.1c">d_{Q}</annotation></semantics></math>), gradient with respect to the attention output (<math id="A3.Thmtheorem3.p1.6.6.m6.1" class="ltx_Math" alttext="dO" display="inline"><semantics id="A3.Thmtheorem3.p1.6.6.m6.1a"><mrow id="A3.Thmtheorem3.p1.6.6.m6.1.1" xref="A3.Thmtheorem3.p1.6.6.m6.1.1.cmml"><mi id="A3.Thmtheorem3.p1.6.6.m6.1.1.2" xref="A3.Thmtheorem3.p1.6.6.m6.1.1.2.cmml">d</mi><mo lspace="0em" rspace="0em" id="A3.Thmtheorem3.p1.6.6.m6.1.1.1" xref="A3.Thmtheorem3.p1.6.6.m6.1.1.1.cmml">​</mo><mi id="A3.Thmtheorem3.p1.6.6.m6.1.1.3" xref="A3.Thmtheorem3.p1.6.6.m6.1.1.3.cmml">O</mi></mrow><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem3.p1.6.6.m6.1b"><apply id="A3.Thmtheorem3.p1.6.6.m6.1.1.cmml" xref="A3.Thmtheorem3.p1.6.6.m6.1.1"><times id="A3.Thmtheorem3.p1.6.6.m6.1.1.1.cmml" xref="A3.Thmtheorem3.p1.6.6.m6.1.1.1"></times><ci id="A3.Thmtheorem3.p1.6.6.m6.1.1.2.cmml" xref="A3.Thmtheorem3.p1.6.6.m6.1.1.2">𝑑</ci><ci id="A3.Thmtheorem3.p1.6.6.m6.1.1.3.cmml" xref="A3.Thmtheorem3.p1.6.6.m6.1.1.3">𝑂</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem3.p1.6.6.m6.1c">dO</annotation></semantics></math>), and reduction variables (<math id="A3.Thmtheorem3.p1.7.7.m7.1" class="ltx_Math" alttext="D" display="inline"><semantics id="A3.Thmtheorem3.p1.7.7.m7.1a"><mi id="A3.Thmtheorem3.p1.7.7.m7.1.1" xref="A3.Thmtheorem3.p1.7.7.m7.1.1.cmml">D</mi><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem3.p1.7.7.m7.1b"><ci id="A3.Thmtheorem3.p1.7.7.m7.1.1.cmml" xref="A3.Thmtheorem3.p1.7.7.m7.1.1">𝐷</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem3.p1.7.7.m7.1c">D</annotation></semantics></math>, <math id="A3.Thmtheorem3.p1.8.8.m8.1" class="ltx_Math" alttext="lse" display="inline"><semantics id="A3.Thmtheorem3.p1.8.8.m8.1a"><mrow id="A3.Thmtheorem3.p1.8.8.m8.1.1" xref="A3.Thmtheorem3.p1.8.8.m8.1.1.cmml"><mi id="A3.Thmtheorem3.p1.8.8.m8.1.1.2" xref="A3.Thmtheorem3.p1.8.8.m8.1.1.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="A3.Thmtheorem3.p1.8.8.m8.1.1.1" xref="A3.Thmtheorem3.p1.8.8.m8.1.1.1.cmml">​</mo><mi id="A3.Thmtheorem3.p1.8.8.m8.1.1.3" xref="A3.Thmtheorem3.p1.8.8.m8.1.1.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="A3.Thmtheorem3.p1.8.8.m8.1.1.1a" xref="A3.Thmtheorem3.p1.8.8.m8.1.1.1.cmml">​</mo><mi id="A3.Thmtheorem3.p1.8.8.m8.1.1.4" xref="A3.Thmtheorem3.p1.8.8.m8.1.1.4.cmml">e</mi></mrow><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem3.p1.8.8.m8.1b"><apply id="A3.Thmtheorem3.p1.8.8.m8.1.1.cmml" xref="A3.Thmtheorem3.p1.8.8.m8.1.1"><times id="A3.Thmtheorem3.p1.8.8.m8.1.1.1.cmml" xref="A3.Thmtheorem3.p1.8.8.m8.1.1.1"></times><ci id="A3.Thmtheorem3.p1.8.8.m8.1.1.2.cmml" xref="A3.Thmtheorem3.p1.8.8.m8.1.1.2">𝑙</ci><ci id="A3.Thmtheorem3.p1.8.8.m8.1.1.3.cmml" xref="A3.Thmtheorem3.p1.8.8.m8.1.1.3">𝑠</ci><ci id="A3.Thmtheorem3.p1.8.8.m8.1.1.4.cmml" xref="A3.Thmtheorem3.p1.8.8.m8.1.1.4">𝑒</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem3.p1.8.8.m8.1c">lse</annotation></semantics></math>) during the backward pass. The communication times for these activations in the forward and backward processes are:</span></p>
<table id="A4.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A3.Ex3"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A3.Ex3.m1.1" class="ltx_Math" alttext="\displaystyle t_{\text{comm\_attn\_f}}" display="inline"><semantics id="A3.Ex3.m1.1a"><msub id="A3.Ex3.m1.1.1" xref="A3.Ex3.m1.1.1.cmml"><mi id="A3.Ex3.m1.1.1.2" xref="A3.Ex3.m1.1.1.2.cmml">t</mi><mtext class="ltx_mathvariant_italic" id="A3.Ex3.m1.1.1.3" xref="A3.Ex3.m1.1.1.3a.cmml">comm_attn_f</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.Ex3.m1.1b"><apply id="A3.Ex3.m1.1.1.cmml" xref="A3.Ex3.m1.1.1"><csymbol cd="ambiguous" id="A3.Ex3.m1.1.1.1.cmml" xref="A3.Ex3.m1.1.1">subscript</csymbol><ci id="A3.Ex3.m1.1.1.2.cmml" xref="A3.Ex3.m1.1.1.2">𝑡</ci><ci id="A3.Ex3.m1.1.1.3a.cmml" xref="A3.Ex3.m1.1.1.3"><mtext class="ltx_mathvariant_italic" mathsize="70%" id="A3.Ex3.m1.1.1.3.cmml" xref="A3.Ex3.m1.1.1.3">comm_attn_f</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex3.m1.1c">\displaystyle t_{\text{comm\_attn\_f}}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A3.Ex3.m2.2" class="ltx_Math" alttext="\displaystyle=\frac{2\times B\times N^{\prime}\times Z\times d\times M\times(G-1)}{b\times G}," display="inline"><semantics id="A3.Ex3.m2.2a"><mrow id="A3.Ex3.m2.2.2.1" xref="A3.Ex3.m2.2.2.1.1.cmml"><mrow id="A3.Ex3.m2.2.2.1.1" xref="A3.Ex3.m2.2.2.1.1.cmml"><mi id="A3.Ex3.m2.2.2.1.1.2" xref="A3.Ex3.m2.2.2.1.1.2.cmml"></mi><mo id="A3.Ex3.m2.2.2.1.1.1" xref="A3.Ex3.m2.2.2.1.1.1.cmml">=</mo><mstyle displaystyle="true" id="A3.Ex3.m2.1.1" xref="A3.Ex3.m2.1.1.cmml"><mfrac id="A3.Ex3.m2.1.1a" xref="A3.Ex3.m2.1.1.cmml"><mrow id="A3.Ex3.m2.1.1.1" xref="A3.Ex3.m2.1.1.1.cmml"><mn id="A3.Ex3.m2.1.1.1.3" xref="A3.Ex3.m2.1.1.1.3.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="A3.Ex3.m2.1.1.1.2" xref="A3.Ex3.m2.1.1.1.2.cmml">×</mo><mi id="A3.Ex3.m2.1.1.1.4" xref="A3.Ex3.m2.1.1.1.4.cmml">B</mi><mo lspace="0.222em" rspace="0.222em" id="A3.Ex3.m2.1.1.1.2a" xref="A3.Ex3.m2.1.1.1.2.cmml">×</mo><msup id="A3.Ex3.m2.1.1.1.5" xref="A3.Ex3.m2.1.1.1.5.cmml"><mi id="A3.Ex3.m2.1.1.1.5.2" xref="A3.Ex3.m2.1.1.1.5.2.cmml">N</mi><mo id="A3.Ex3.m2.1.1.1.5.3" xref="A3.Ex3.m2.1.1.1.5.3.cmml">′</mo></msup><mo lspace="0.222em" rspace="0.222em" id="A3.Ex3.m2.1.1.1.2b" xref="A3.Ex3.m2.1.1.1.2.cmml">×</mo><mi id="A3.Ex3.m2.1.1.1.6" xref="A3.Ex3.m2.1.1.1.6.cmml">Z</mi><mo lspace="0.222em" rspace="0.222em" id="A3.Ex3.m2.1.1.1.2c" xref="A3.Ex3.m2.1.1.1.2.cmml">×</mo><mi id="A3.Ex3.m2.1.1.1.7" xref="A3.Ex3.m2.1.1.1.7.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="A3.Ex3.m2.1.1.1.2d" xref="A3.Ex3.m2.1.1.1.2.cmml">×</mo><mi id="A3.Ex3.m2.1.1.1.8" xref="A3.Ex3.m2.1.1.1.8.cmml">M</mi><mo lspace="0.222em" rspace="0.222em" id="A3.Ex3.m2.1.1.1.2e" xref="A3.Ex3.m2.1.1.1.2.cmml">×</mo><mrow id="A3.Ex3.m2.1.1.1.1.1" xref="A3.Ex3.m2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A3.Ex3.m2.1.1.1.1.1.2" xref="A3.Ex3.m2.1.1.1.1.1.1.cmml">(</mo><mrow id="A3.Ex3.m2.1.1.1.1.1.1" xref="A3.Ex3.m2.1.1.1.1.1.1.cmml"><mi id="A3.Ex3.m2.1.1.1.1.1.1.2" xref="A3.Ex3.m2.1.1.1.1.1.1.2.cmml">G</mi><mo id="A3.Ex3.m2.1.1.1.1.1.1.1" xref="A3.Ex3.m2.1.1.1.1.1.1.1.cmml">−</mo><mn id="A3.Ex3.m2.1.1.1.1.1.1.3" xref="A3.Ex3.m2.1.1.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A3.Ex3.m2.1.1.1.1.1.3" xref="A3.Ex3.m2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mrow id="A3.Ex3.m2.1.1.3" xref="A3.Ex3.m2.1.1.3.cmml"><mi id="A3.Ex3.m2.1.1.3.2" xref="A3.Ex3.m2.1.1.3.2.cmml">b</mi><mo lspace="0.222em" rspace="0.222em" id="A3.Ex3.m2.1.1.3.1" xref="A3.Ex3.m2.1.1.3.1.cmml">×</mo><mi id="A3.Ex3.m2.1.1.3.3" xref="A3.Ex3.m2.1.1.3.3.cmml">G</mi></mrow></mfrac></mstyle></mrow><mo id="A3.Ex3.m2.2.2.1.2" xref="A3.Ex3.m2.2.2.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="A3.Ex3.m2.2b"><apply id="A3.Ex3.m2.2.2.1.1.cmml" xref="A3.Ex3.m2.2.2.1"><eq id="A3.Ex3.m2.2.2.1.1.1.cmml" xref="A3.Ex3.m2.2.2.1.1.1"></eq><csymbol cd="latexml" id="A3.Ex3.m2.2.2.1.1.2.cmml" xref="A3.Ex3.m2.2.2.1.1.2">absent</csymbol><apply id="A3.Ex3.m2.1.1.cmml" xref="A3.Ex3.m2.1.1"><divide id="A3.Ex3.m2.1.1.2.cmml" xref="A3.Ex3.m2.1.1"></divide><apply id="A3.Ex3.m2.1.1.1.cmml" xref="A3.Ex3.m2.1.1.1"><times id="A3.Ex3.m2.1.1.1.2.cmml" xref="A3.Ex3.m2.1.1.1.2"></times><cn type="integer" id="A3.Ex3.m2.1.1.1.3.cmml" xref="A3.Ex3.m2.1.1.1.3">2</cn><ci id="A3.Ex3.m2.1.1.1.4.cmml" xref="A3.Ex3.m2.1.1.1.4">𝐵</ci><apply id="A3.Ex3.m2.1.1.1.5.cmml" xref="A3.Ex3.m2.1.1.1.5"><csymbol cd="ambiguous" id="A3.Ex3.m2.1.1.1.5.1.cmml" xref="A3.Ex3.m2.1.1.1.5">superscript</csymbol><ci id="A3.Ex3.m2.1.1.1.5.2.cmml" xref="A3.Ex3.m2.1.1.1.5.2">𝑁</ci><ci id="A3.Ex3.m2.1.1.1.5.3.cmml" xref="A3.Ex3.m2.1.1.1.5.3">′</ci></apply><ci id="A3.Ex3.m2.1.1.1.6.cmml" xref="A3.Ex3.m2.1.1.1.6">𝑍</ci><ci id="A3.Ex3.m2.1.1.1.7.cmml" xref="A3.Ex3.m2.1.1.1.7">𝑑</ci><ci id="A3.Ex3.m2.1.1.1.8.cmml" xref="A3.Ex3.m2.1.1.1.8">𝑀</ci><apply id="A3.Ex3.m2.1.1.1.1.1.1.cmml" xref="A3.Ex3.m2.1.1.1.1.1"><minus id="A3.Ex3.m2.1.1.1.1.1.1.1.cmml" xref="A3.Ex3.m2.1.1.1.1.1.1.1"></minus><ci id="A3.Ex3.m2.1.1.1.1.1.1.2.cmml" xref="A3.Ex3.m2.1.1.1.1.1.1.2">𝐺</ci><cn type="integer" id="A3.Ex3.m2.1.1.1.1.1.1.3.cmml" xref="A3.Ex3.m2.1.1.1.1.1.1.3">1</cn></apply></apply><apply id="A3.Ex3.m2.1.1.3.cmml" xref="A3.Ex3.m2.1.1.3"><times id="A3.Ex3.m2.1.1.3.1.cmml" xref="A3.Ex3.m2.1.1.3.1"></times><ci id="A3.Ex3.m2.1.1.3.2.cmml" xref="A3.Ex3.m2.1.1.3.2">𝑏</ci><ci id="A3.Ex3.m2.1.1.3.3.cmml" xref="A3.Ex3.m2.1.1.3.3">𝐺</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex3.m2.2c">\displaystyle=\frac{2\times B\times N^{\prime}\times Z\times d\times M\times(G-1)}{b\times G},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="A3.Ex4"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A3.Ex4.m1.1" class="ltx_Math" alttext="\displaystyle t_{\text{comm\_attn\_b}}" display="inline"><semantics id="A3.Ex4.m1.1a"><msub id="A3.Ex4.m1.1.1" xref="A3.Ex4.m1.1.1.cmml"><mi id="A3.Ex4.m1.1.1.2" xref="A3.Ex4.m1.1.1.2.cmml">t</mi><mtext class="ltx_mathvariant_italic" id="A3.Ex4.m1.1.1.3" xref="A3.Ex4.m1.1.1.3a.cmml">comm_attn_b</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.Ex4.m1.1b"><apply id="A3.Ex4.m1.1.1.cmml" xref="A3.Ex4.m1.1.1"><csymbol cd="ambiguous" id="A3.Ex4.m1.1.1.1.cmml" xref="A3.Ex4.m1.1.1">subscript</csymbol><ci id="A3.Ex4.m1.1.1.2.cmml" xref="A3.Ex4.m1.1.1.2">𝑡</ci><ci id="A3.Ex4.m1.1.1.3a.cmml" xref="A3.Ex4.m1.1.1.3"><mtext class="ltx_mathvariant_italic" mathsize="70%" id="A3.Ex4.m1.1.1.3.cmml" xref="A3.Ex4.m1.1.1.3">comm_attn_b</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex4.m1.1c">\displaystyle t_{\text{comm\_attn\_b}}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A3.Ex4.m2.3" class="ltx_Math" alttext="\displaystyle=\frac{(3\times B\times N^{\prime}\times Z\times d+2\times B\times N^{\prime}\times Z)\times M\times(G-1)}{b\times G}." display="inline"><semantics id="A3.Ex4.m2.3a"><mrow id="A3.Ex4.m2.3.3.1" xref="A3.Ex4.m2.3.3.1.1.cmml"><mrow id="A3.Ex4.m2.3.3.1.1" xref="A3.Ex4.m2.3.3.1.1.cmml"><mi id="A3.Ex4.m2.3.3.1.1.2" xref="A3.Ex4.m2.3.3.1.1.2.cmml"></mi><mo id="A3.Ex4.m2.3.3.1.1.1" xref="A3.Ex4.m2.3.3.1.1.1.cmml">=</mo><mstyle displaystyle="true" id="A3.Ex4.m2.2.2" xref="A3.Ex4.m2.2.2.cmml"><mfrac id="A3.Ex4.m2.2.2a" xref="A3.Ex4.m2.2.2.cmml"><mrow id="A3.Ex4.m2.2.2.2" xref="A3.Ex4.m2.2.2.2.cmml"><mrow id="A3.Ex4.m2.1.1.1.1.1" xref="A3.Ex4.m2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A3.Ex4.m2.1.1.1.1.1.2" xref="A3.Ex4.m2.1.1.1.1.1.1.cmml">(</mo><mrow id="A3.Ex4.m2.1.1.1.1.1.1" xref="A3.Ex4.m2.1.1.1.1.1.1.cmml"><mrow id="A3.Ex4.m2.1.1.1.1.1.1.2" xref="A3.Ex4.m2.1.1.1.1.1.1.2.cmml"><mn id="A3.Ex4.m2.1.1.1.1.1.1.2.2" xref="A3.Ex4.m2.1.1.1.1.1.1.2.2.cmml">3</mn><mo lspace="0.222em" rspace="0.222em" id="A3.Ex4.m2.1.1.1.1.1.1.2.1" xref="A3.Ex4.m2.1.1.1.1.1.1.2.1.cmml">×</mo><mi id="A3.Ex4.m2.1.1.1.1.1.1.2.3" xref="A3.Ex4.m2.1.1.1.1.1.1.2.3.cmml">B</mi><mo lspace="0.222em" rspace="0.222em" id="A3.Ex4.m2.1.1.1.1.1.1.2.1a" xref="A3.Ex4.m2.1.1.1.1.1.1.2.1.cmml">×</mo><msup id="A3.Ex4.m2.1.1.1.1.1.1.2.4" xref="A3.Ex4.m2.1.1.1.1.1.1.2.4.cmml"><mi id="A3.Ex4.m2.1.1.1.1.1.1.2.4.2" xref="A3.Ex4.m2.1.1.1.1.1.1.2.4.2.cmml">N</mi><mo id="A3.Ex4.m2.1.1.1.1.1.1.2.4.3" xref="A3.Ex4.m2.1.1.1.1.1.1.2.4.3.cmml">′</mo></msup><mo lspace="0.222em" rspace="0.222em" id="A3.Ex4.m2.1.1.1.1.1.1.2.1b" xref="A3.Ex4.m2.1.1.1.1.1.1.2.1.cmml">×</mo><mi id="A3.Ex4.m2.1.1.1.1.1.1.2.5" xref="A3.Ex4.m2.1.1.1.1.1.1.2.5.cmml">Z</mi><mo lspace="0.222em" rspace="0.222em" id="A3.Ex4.m2.1.1.1.1.1.1.2.1c" xref="A3.Ex4.m2.1.1.1.1.1.1.2.1.cmml">×</mo><mi id="A3.Ex4.m2.1.1.1.1.1.1.2.6" xref="A3.Ex4.m2.1.1.1.1.1.1.2.6.cmml">d</mi></mrow><mo id="A3.Ex4.m2.1.1.1.1.1.1.1" xref="A3.Ex4.m2.1.1.1.1.1.1.1.cmml">+</mo><mrow id="A3.Ex4.m2.1.1.1.1.1.1.3" xref="A3.Ex4.m2.1.1.1.1.1.1.3.cmml"><mn id="A3.Ex4.m2.1.1.1.1.1.1.3.2" xref="A3.Ex4.m2.1.1.1.1.1.1.3.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="A3.Ex4.m2.1.1.1.1.1.1.3.1" xref="A3.Ex4.m2.1.1.1.1.1.1.3.1.cmml">×</mo><mi id="A3.Ex4.m2.1.1.1.1.1.1.3.3" xref="A3.Ex4.m2.1.1.1.1.1.1.3.3.cmml">B</mi><mo lspace="0.222em" rspace="0.222em" id="A3.Ex4.m2.1.1.1.1.1.1.3.1a" xref="A3.Ex4.m2.1.1.1.1.1.1.3.1.cmml">×</mo><msup id="A3.Ex4.m2.1.1.1.1.1.1.3.4" xref="A3.Ex4.m2.1.1.1.1.1.1.3.4.cmml"><mi id="A3.Ex4.m2.1.1.1.1.1.1.3.4.2" xref="A3.Ex4.m2.1.1.1.1.1.1.3.4.2.cmml">N</mi><mo id="A3.Ex4.m2.1.1.1.1.1.1.3.4.3" xref="A3.Ex4.m2.1.1.1.1.1.1.3.4.3.cmml">′</mo></msup><mo lspace="0.222em" rspace="0.222em" id="A3.Ex4.m2.1.1.1.1.1.1.3.1b" xref="A3.Ex4.m2.1.1.1.1.1.1.3.1.cmml">×</mo><mi id="A3.Ex4.m2.1.1.1.1.1.1.3.5" xref="A3.Ex4.m2.1.1.1.1.1.1.3.5.cmml">Z</mi></mrow></mrow><mo rspace="0.055em" stretchy="false" id="A3.Ex4.m2.1.1.1.1.1.3" xref="A3.Ex4.m2.1.1.1.1.1.1.cmml">)</mo></mrow><mo rspace="0.222em" id="A3.Ex4.m2.2.2.2.3" xref="A3.Ex4.m2.2.2.2.3.cmml">×</mo><mi id="A3.Ex4.m2.2.2.2.4" xref="A3.Ex4.m2.2.2.2.4.cmml">M</mi><mo lspace="0.222em" rspace="0.222em" id="A3.Ex4.m2.2.2.2.3a" xref="A3.Ex4.m2.2.2.2.3.cmml">×</mo><mrow id="A3.Ex4.m2.2.2.2.2.1" xref="A3.Ex4.m2.2.2.2.2.1.1.cmml"><mo stretchy="false" id="A3.Ex4.m2.2.2.2.2.1.2" xref="A3.Ex4.m2.2.2.2.2.1.1.cmml">(</mo><mrow id="A3.Ex4.m2.2.2.2.2.1.1" xref="A3.Ex4.m2.2.2.2.2.1.1.cmml"><mi id="A3.Ex4.m2.2.2.2.2.1.1.2" xref="A3.Ex4.m2.2.2.2.2.1.1.2.cmml">G</mi><mo id="A3.Ex4.m2.2.2.2.2.1.1.1" xref="A3.Ex4.m2.2.2.2.2.1.1.1.cmml">−</mo><mn id="A3.Ex4.m2.2.2.2.2.1.1.3" xref="A3.Ex4.m2.2.2.2.2.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A3.Ex4.m2.2.2.2.2.1.3" xref="A3.Ex4.m2.2.2.2.2.1.1.cmml">)</mo></mrow></mrow><mrow id="A3.Ex4.m2.2.2.4" xref="A3.Ex4.m2.2.2.4.cmml"><mi id="A3.Ex4.m2.2.2.4.2" xref="A3.Ex4.m2.2.2.4.2.cmml">b</mi><mo lspace="0.222em" rspace="0.222em" id="A3.Ex4.m2.2.2.4.1" xref="A3.Ex4.m2.2.2.4.1.cmml">×</mo><mi id="A3.Ex4.m2.2.2.4.3" xref="A3.Ex4.m2.2.2.4.3.cmml">G</mi></mrow></mfrac></mstyle></mrow><mo lspace="0em" id="A3.Ex4.m2.3.3.1.2" xref="A3.Ex4.m2.3.3.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A3.Ex4.m2.3b"><apply id="A3.Ex4.m2.3.3.1.1.cmml" xref="A3.Ex4.m2.3.3.1"><eq id="A3.Ex4.m2.3.3.1.1.1.cmml" xref="A3.Ex4.m2.3.3.1.1.1"></eq><csymbol cd="latexml" id="A3.Ex4.m2.3.3.1.1.2.cmml" xref="A3.Ex4.m2.3.3.1.1.2">absent</csymbol><apply id="A3.Ex4.m2.2.2.cmml" xref="A3.Ex4.m2.2.2"><divide id="A3.Ex4.m2.2.2.3.cmml" xref="A3.Ex4.m2.2.2"></divide><apply id="A3.Ex4.m2.2.2.2.cmml" xref="A3.Ex4.m2.2.2.2"><times id="A3.Ex4.m2.2.2.2.3.cmml" xref="A3.Ex4.m2.2.2.2.3"></times><apply id="A3.Ex4.m2.1.1.1.1.1.1.cmml" xref="A3.Ex4.m2.1.1.1.1.1"><plus id="A3.Ex4.m2.1.1.1.1.1.1.1.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.1"></plus><apply id="A3.Ex4.m2.1.1.1.1.1.1.2.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.2"><times id="A3.Ex4.m2.1.1.1.1.1.1.2.1.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.2.1"></times><cn type="integer" id="A3.Ex4.m2.1.1.1.1.1.1.2.2.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.2.2">3</cn><ci id="A3.Ex4.m2.1.1.1.1.1.1.2.3.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.2.3">𝐵</ci><apply id="A3.Ex4.m2.1.1.1.1.1.1.2.4.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.2.4"><csymbol cd="ambiguous" id="A3.Ex4.m2.1.1.1.1.1.1.2.4.1.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.2.4">superscript</csymbol><ci id="A3.Ex4.m2.1.1.1.1.1.1.2.4.2.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.2.4.2">𝑁</ci><ci id="A3.Ex4.m2.1.1.1.1.1.1.2.4.3.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.2.4.3">′</ci></apply><ci id="A3.Ex4.m2.1.1.1.1.1.1.2.5.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.2.5">𝑍</ci><ci id="A3.Ex4.m2.1.1.1.1.1.1.2.6.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.2.6">𝑑</ci></apply><apply id="A3.Ex4.m2.1.1.1.1.1.1.3.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.3"><times id="A3.Ex4.m2.1.1.1.1.1.1.3.1.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.3.1"></times><cn type="integer" id="A3.Ex4.m2.1.1.1.1.1.1.3.2.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.3.2">2</cn><ci id="A3.Ex4.m2.1.1.1.1.1.1.3.3.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.3.3">𝐵</ci><apply id="A3.Ex4.m2.1.1.1.1.1.1.3.4.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.3.4"><csymbol cd="ambiguous" id="A3.Ex4.m2.1.1.1.1.1.1.3.4.1.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.3.4">superscript</csymbol><ci id="A3.Ex4.m2.1.1.1.1.1.1.3.4.2.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.3.4.2">𝑁</ci><ci id="A3.Ex4.m2.1.1.1.1.1.1.3.4.3.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.3.4.3">′</ci></apply><ci id="A3.Ex4.m2.1.1.1.1.1.1.3.5.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.3.5">𝑍</ci></apply></apply><ci id="A3.Ex4.m2.2.2.2.4.cmml" xref="A3.Ex4.m2.2.2.2.4">𝑀</ci><apply id="A3.Ex4.m2.2.2.2.2.1.1.cmml" xref="A3.Ex4.m2.2.2.2.2.1"><minus id="A3.Ex4.m2.2.2.2.2.1.1.1.cmml" xref="A3.Ex4.m2.2.2.2.2.1.1.1"></minus><ci id="A3.Ex4.m2.2.2.2.2.1.1.2.cmml" xref="A3.Ex4.m2.2.2.2.2.1.1.2">𝐺</ci><cn type="integer" id="A3.Ex4.m2.2.2.2.2.1.1.3.cmml" xref="A3.Ex4.m2.2.2.2.2.1.1.3">1</cn></apply></apply><apply id="A3.Ex4.m2.2.2.4.cmml" xref="A3.Ex4.m2.2.2.4"><times id="A3.Ex4.m2.2.2.4.1.cmml" xref="A3.Ex4.m2.2.2.4.1"></times><ci id="A3.Ex4.m2.2.2.4.2.cmml" xref="A3.Ex4.m2.2.2.4.2">𝑏</ci><ci id="A3.Ex4.m2.2.2.4.3.cmml" xref="A3.Ex4.m2.2.2.4.3">𝐺</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex4.m2.3c">\displaystyle=\frac{(3\times B\times N^{\prime}\times Z\times d+2\times B\times N^{\prime}\times Z)\times M\times(G-1)}{b\times G}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
</div>
<div id="A3.Thmtheorem4" class="ltx_theorem ltx_theorem_lemma">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A3.Thmtheorem4.1.1.1" class="ltx_text ltx_font_bold">Lemma C.4</span></span><span id="A3.Thmtheorem4.2.2" class="ltx_text ltx_font_bold"> </span>(Weight Communication Time in BurstAttention)<span id="A3.Thmtheorem4.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A3.Thmtheorem4.p1" class="ltx_para">
<p id="A3.Thmtheorem4.p1.3" class="ltx_p"><span id="A3.Thmtheorem4.p1.3.3" class="ltx_text ltx_font_italic">In BurstAttention’s attention module, there are four linear layers with weights of dimension <math id="A3.Thmtheorem4.p1.1.1.m1.1" class="ltx_Math" alttext="H\times Z\times d" display="inline"><semantics id="A3.Thmtheorem4.p1.1.1.m1.1a"><mrow id="A3.Thmtheorem4.p1.1.1.m1.1.1" xref="A3.Thmtheorem4.p1.1.1.m1.1.1.cmml"><mi id="A3.Thmtheorem4.p1.1.1.m1.1.1.2" xref="A3.Thmtheorem4.p1.1.1.m1.1.1.2.cmml">H</mi><mo lspace="0.222em" rspace="0.222em" id="A3.Thmtheorem4.p1.1.1.m1.1.1.1" xref="A3.Thmtheorem4.p1.1.1.m1.1.1.1.cmml">×</mo><mi id="A3.Thmtheorem4.p1.1.1.m1.1.1.3" xref="A3.Thmtheorem4.p1.1.1.m1.1.1.3.cmml">Z</mi><mo lspace="0.222em" rspace="0.222em" id="A3.Thmtheorem4.p1.1.1.m1.1.1.1a" xref="A3.Thmtheorem4.p1.1.1.m1.1.1.1.cmml">×</mo><mi id="A3.Thmtheorem4.p1.1.1.m1.1.1.4" xref="A3.Thmtheorem4.p1.1.1.m1.1.1.4.cmml">d</mi></mrow><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem4.p1.1.1.m1.1b"><apply id="A3.Thmtheorem4.p1.1.1.m1.1.1.cmml" xref="A3.Thmtheorem4.p1.1.1.m1.1.1"><times id="A3.Thmtheorem4.p1.1.1.m1.1.1.1.cmml" xref="A3.Thmtheorem4.p1.1.1.m1.1.1.1"></times><ci id="A3.Thmtheorem4.p1.1.1.m1.1.1.2.cmml" xref="A3.Thmtheorem4.p1.1.1.m1.1.1.2">𝐻</ci><ci id="A3.Thmtheorem4.p1.1.1.m1.1.1.3.cmml" xref="A3.Thmtheorem4.p1.1.1.m1.1.1.3">𝑍</ci><ci id="A3.Thmtheorem4.p1.1.1.m1.1.1.4.cmml" xref="A3.Thmtheorem4.p1.1.1.m1.1.1.4">𝑑</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem4.p1.1.1.m1.1c">H\times Z\times d</annotation></semantics></math>. The feedforward module has two linear layers with dimensions <math id="A3.Thmtheorem4.p1.2.2.m2.1" class="ltx_Math" alttext="H\times d_{ffn}" display="inline"><semantics id="A3.Thmtheorem4.p1.2.2.m2.1a"><mrow id="A3.Thmtheorem4.p1.2.2.m2.1.1" xref="A3.Thmtheorem4.p1.2.2.m2.1.1.cmml"><mi id="A3.Thmtheorem4.p1.2.2.m2.1.1.2" xref="A3.Thmtheorem4.p1.2.2.m2.1.1.2.cmml">H</mi><mo lspace="0.222em" rspace="0.222em" id="A3.Thmtheorem4.p1.2.2.m2.1.1.1" xref="A3.Thmtheorem4.p1.2.2.m2.1.1.1.cmml">×</mo><msub id="A3.Thmtheorem4.p1.2.2.m2.1.1.3" xref="A3.Thmtheorem4.p1.2.2.m2.1.1.3.cmml"><mi id="A3.Thmtheorem4.p1.2.2.m2.1.1.3.2" xref="A3.Thmtheorem4.p1.2.2.m2.1.1.3.2.cmml">d</mi><mrow id="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3" xref="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3.cmml"><mi id="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3.2" xref="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3.1" xref="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3.1.cmml">​</mo><mi id="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3.3" xref="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3.3.cmml">f</mi><mo lspace="0em" rspace="0em" id="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3.1a" xref="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3.1.cmml">​</mo><mi id="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3.4" xref="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3.4.cmml">n</mi></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem4.p1.2.2.m2.1b"><apply id="A3.Thmtheorem4.p1.2.2.m2.1.1.cmml" xref="A3.Thmtheorem4.p1.2.2.m2.1.1"><times id="A3.Thmtheorem4.p1.2.2.m2.1.1.1.cmml" xref="A3.Thmtheorem4.p1.2.2.m2.1.1.1"></times><ci id="A3.Thmtheorem4.p1.2.2.m2.1.1.2.cmml" xref="A3.Thmtheorem4.p1.2.2.m2.1.1.2">𝐻</ci><apply id="A3.Thmtheorem4.p1.2.2.m2.1.1.3.cmml" xref="A3.Thmtheorem4.p1.2.2.m2.1.1.3"><csymbol cd="ambiguous" id="A3.Thmtheorem4.p1.2.2.m2.1.1.3.1.cmml" xref="A3.Thmtheorem4.p1.2.2.m2.1.1.3">subscript</csymbol><ci id="A3.Thmtheorem4.p1.2.2.m2.1.1.3.2.cmml" xref="A3.Thmtheorem4.p1.2.2.m2.1.1.3.2">𝑑</ci><apply id="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3.cmml" xref="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3"><times id="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3.1.cmml" xref="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3.1"></times><ci id="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3.2.cmml" xref="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3.2">𝑓</ci><ci id="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3.3.cmml" xref="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3.3">𝑓</ci><ci id="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3.4.cmml" xref="A3.Thmtheorem4.p1.2.2.m2.1.1.3.3.4">𝑛</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem4.p1.2.2.m2.1c">H\times d_{ffn}</annotation></semantics></math> and <math id="A3.Thmtheorem4.p1.3.3.m3.1" class="ltx_Math" alttext="d_{ffn}\times H" display="inline"><semantics id="A3.Thmtheorem4.p1.3.3.m3.1a"><mrow id="A3.Thmtheorem4.p1.3.3.m3.1.1" xref="A3.Thmtheorem4.p1.3.3.m3.1.1.cmml"><msub id="A3.Thmtheorem4.p1.3.3.m3.1.1.2" xref="A3.Thmtheorem4.p1.3.3.m3.1.1.2.cmml"><mi id="A3.Thmtheorem4.p1.3.3.m3.1.1.2.2" xref="A3.Thmtheorem4.p1.3.3.m3.1.1.2.2.cmml">d</mi><mrow id="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3" xref="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3.cmml"><mi id="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3.2" xref="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3.1" xref="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3.1.cmml">​</mo><mi id="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3.3" xref="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3.3.cmml">f</mi><mo lspace="0em" rspace="0em" id="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3.1a" xref="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3.1.cmml">​</mo><mi id="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3.4" xref="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3.4.cmml">n</mi></mrow></msub><mo lspace="0.222em" rspace="0.222em" id="A3.Thmtheorem4.p1.3.3.m3.1.1.1" xref="A3.Thmtheorem4.p1.3.3.m3.1.1.1.cmml">×</mo><mi id="A3.Thmtheorem4.p1.3.3.m3.1.1.3" xref="A3.Thmtheorem4.p1.3.3.m3.1.1.3.cmml">H</mi></mrow><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem4.p1.3.3.m3.1b"><apply id="A3.Thmtheorem4.p1.3.3.m3.1.1.cmml" xref="A3.Thmtheorem4.p1.3.3.m3.1.1"><times id="A3.Thmtheorem4.p1.3.3.m3.1.1.1.cmml" xref="A3.Thmtheorem4.p1.3.3.m3.1.1.1"></times><apply id="A3.Thmtheorem4.p1.3.3.m3.1.1.2.cmml" xref="A3.Thmtheorem4.p1.3.3.m3.1.1.2"><csymbol cd="ambiguous" id="A3.Thmtheorem4.p1.3.3.m3.1.1.2.1.cmml" xref="A3.Thmtheorem4.p1.3.3.m3.1.1.2">subscript</csymbol><ci id="A3.Thmtheorem4.p1.3.3.m3.1.1.2.2.cmml" xref="A3.Thmtheorem4.p1.3.3.m3.1.1.2.2">𝑑</ci><apply id="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3.cmml" xref="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3"><times id="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3.1.cmml" xref="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3.1"></times><ci id="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3.2.cmml" xref="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3.2">𝑓</ci><ci id="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3.3.cmml" xref="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3.3">𝑓</ci><ci id="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3.4.cmml" xref="A3.Thmtheorem4.p1.3.3.m3.1.1.2.3.4">𝑛</ci></apply></apply><ci id="A3.Thmtheorem4.p1.3.3.m3.1.1.3.cmml" xref="A3.Thmtheorem4.p1.3.3.m3.1.1.3">𝐻</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem4.p1.3.3.m3.1c">d_{ffn}\times H</annotation></semantics></math>. The communication time for the weights of these layers is calculated as:</span></p>
<table id="A3.Ex5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A3.Ex5.m1.3" class="ltx_Math" alttext="t_{\text{comm\_weights}}=\frac{(4\times H\times Z\times d+2\times H\times d_{ffn})\times M\times(G-1)}{b\times G}," display="block"><semantics id="A3.Ex5.m1.3a"><mrow id="A3.Ex5.m1.3.3.1" xref="A3.Ex5.m1.3.3.1.1.cmml"><mrow id="A3.Ex5.m1.3.3.1.1" xref="A3.Ex5.m1.3.3.1.1.cmml"><msub id="A3.Ex5.m1.3.3.1.1.2" xref="A3.Ex5.m1.3.3.1.1.2.cmml"><mi id="A3.Ex5.m1.3.3.1.1.2.2" xref="A3.Ex5.m1.3.3.1.1.2.2.cmml">t</mi><mtext class="ltx_mathvariant_italic" id="A3.Ex5.m1.3.3.1.1.2.3" xref="A3.Ex5.m1.3.3.1.1.2.3a.cmml">comm_weights</mtext></msub><mo id="A3.Ex5.m1.3.3.1.1.1" xref="A3.Ex5.m1.3.3.1.1.1.cmml">=</mo><mfrac id="A3.Ex5.m1.2.2" xref="A3.Ex5.m1.2.2.cmml"><mrow id="A3.Ex5.m1.2.2.2" xref="A3.Ex5.m1.2.2.2.cmml"><mrow id="A3.Ex5.m1.1.1.1.1.1" xref="A3.Ex5.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A3.Ex5.m1.1.1.1.1.1.2" xref="A3.Ex5.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="A3.Ex5.m1.1.1.1.1.1.1" xref="A3.Ex5.m1.1.1.1.1.1.1.cmml"><mrow id="A3.Ex5.m1.1.1.1.1.1.1.2" xref="A3.Ex5.m1.1.1.1.1.1.1.2.cmml"><mn id="A3.Ex5.m1.1.1.1.1.1.1.2.2" xref="A3.Ex5.m1.1.1.1.1.1.1.2.2.cmml">4</mn><mo lspace="0.222em" rspace="0.222em" id="A3.Ex5.m1.1.1.1.1.1.1.2.1" xref="A3.Ex5.m1.1.1.1.1.1.1.2.1.cmml">×</mo><mi id="A3.Ex5.m1.1.1.1.1.1.1.2.3" xref="A3.Ex5.m1.1.1.1.1.1.1.2.3.cmml">H</mi><mo lspace="0.222em" rspace="0.222em" id="A3.Ex5.m1.1.1.1.1.1.1.2.1a" xref="A3.Ex5.m1.1.1.1.1.1.1.2.1.cmml">×</mo><mi id="A3.Ex5.m1.1.1.1.1.1.1.2.4" xref="A3.Ex5.m1.1.1.1.1.1.1.2.4.cmml">Z</mi><mo lspace="0.222em" rspace="0.222em" id="A3.Ex5.m1.1.1.1.1.1.1.2.1b" xref="A3.Ex5.m1.1.1.1.1.1.1.2.1.cmml">×</mo><mi id="A3.Ex5.m1.1.1.1.1.1.1.2.5" xref="A3.Ex5.m1.1.1.1.1.1.1.2.5.cmml">d</mi></mrow><mo id="A3.Ex5.m1.1.1.1.1.1.1.1" xref="A3.Ex5.m1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="A3.Ex5.m1.1.1.1.1.1.1.3" xref="A3.Ex5.m1.1.1.1.1.1.1.3.cmml"><mn id="A3.Ex5.m1.1.1.1.1.1.1.3.2" xref="A3.Ex5.m1.1.1.1.1.1.1.3.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="A3.Ex5.m1.1.1.1.1.1.1.3.1" xref="A3.Ex5.m1.1.1.1.1.1.1.3.1.cmml">×</mo><mi id="A3.Ex5.m1.1.1.1.1.1.1.3.3" xref="A3.Ex5.m1.1.1.1.1.1.1.3.3.cmml">H</mi><mo lspace="0.222em" rspace="0.222em" id="A3.Ex5.m1.1.1.1.1.1.1.3.1a" xref="A3.Ex5.m1.1.1.1.1.1.1.3.1.cmml">×</mo><msub id="A3.Ex5.m1.1.1.1.1.1.1.3.4" xref="A3.Ex5.m1.1.1.1.1.1.1.3.4.cmml"><mi id="A3.Ex5.m1.1.1.1.1.1.1.3.4.2" xref="A3.Ex5.m1.1.1.1.1.1.1.3.4.2.cmml">d</mi><mrow id="A3.Ex5.m1.1.1.1.1.1.1.3.4.3" xref="A3.Ex5.m1.1.1.1.1.1.1.3.4.3.cmml"><mi id="A3.Ex5.m1.1.1.1.1.1.1.3.4.3.2" xref="A3.Ex5.m1.1.1.1.1.1.1.3.4.3.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="A3.Ex5.m1.1.1.1.1.1.1.3.4.3.1" xref="A3.Ex5.m1.1.1.1.1.1.1.3.4.3.1.cmml">​</mo><mi id="A3.Ex5.m1.1.1.1.1.1.1.3.4.3.3" xref="A3.Ex5.m1.1.1.1.1.1.1.3.4.3.3.cmml">f</mi><mo lspace="0em" rspace="0em" id="A3.Ex5.m1.1.1.1.1.1.1.3.4.3.1a" xref="A3.Ex5.m1.1.1.1.1.1.1.3.4.3.1.cmml">​</mo><mi id="A3.Ex5.m1.1.1.1.1.1.1.3.4.3.4" xref="A3.Ex5.m1.1.1.1.1.1.1.3.4.3.4.cmml">n</mi></mrow></msub></mrow></mrow><mo rspace="0.055em" stretchy="false" id="A3.Ex5.m1.1.1.1.1.1.3" xref="A3.Ex5.m1.1.1.1.1.1.1.cmml">)</mo></mrow><mo rspace="0.222em" id="A3.Ex5.m1.2.2.2.3" xref="A3.Ex5.m1.2.2.2.3.cmml">×</mo><mi id="A3.Ex5.m1.2.2.2.4" xref="A3.Ex5.m1.2.2.2.4.cmml">M</mi><mo lspace="0.222em" rspace="0.222em" id="A3.Ex5.m1.2.2.2.3a" xref="A3.Ex5.m1.2.2.2.3.cmml">×</mo><mrow id="A3.Ex5.m1.2.2.2.2.1" xref="A3.Ex5.m1.2.2.2.2.1.1.cmml"><mo stretchy="false" id="A3.Ex5.m1.2.2.2.2.1.2" xref="A3.Ex5.m1.2.2.2.2.1.1.cmml">(</mo><mrow id="A3.Ex5.m1.2.2.2.2.1.1" xref="A3.Ex5.m1.2.2.2.2.1.1.cmml"><mi id="A3.Ex5.m1.2.2.2.2.1.1.2" xref="A3.Ex5.m1.2.2.2.2.1.1.2.cmml">G</mi><mo id="A3.Ex5.m1.2.2.2.2.1.1.1" xref="A3.Ex5.m1.2.2.2.2.1.1.1.cmml">−</mo><mn id="A3.Ex5.m1.2.2.2.2.1.1.3" xref="A3.Ex5.m1.2.2.2.2.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A3.Ex5.m1.2.2.2.2.1.3" xref="A3.Ex5.m1.2.2.2.2.1.1.cmml">)</mo></mrow></mrow><mrow id="A3.Ex5.m1.2.2.4" xref="A3.Ex5.m1.2.2.4.cmml"><mi id="A3.Ex5.m1.2.2.4.2" xref="A3.Ex5.m1.2.2.4.2.cmml">b</mi><mo lspace="0.222em" rspace="0.222em" id="A3.Ex5.m1.2.2.4.1" xref="A3.Ex5.m1.2.2.4.1.cmml">×</mo><mi id="A3.Ex5.m1.2.2.4.3" xref="A3.Ex5.m1.2.2.4.3.cmml">G</mi></mrow></mfrac></mrow><mo id="A3.Ex5.m1.3.3.1.2" xref="A3.Ex5.m1.3.3.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="A3.Ex5.m1.3b"><apply id="A3.Ex5.m1.3.3.1.1.cmml" xref="A3.Ex5.m1.3.3.1"><eq id="A3.Ex5.m1.3.3.1.1.1.cmml" xref="A3.Ex5.m1.3.3.1.1.1"></eq><apply id="A3.Ex5.m1.3.3.1.1.2.cmml" xref="A3.Ex5.m1.3.3.1.1.2"><csymbol cd="ambiguous" id="A3.Ex5.m1.3.3.1.1.2.1.cmml" xref="A3.Ex5.m1.3.3.1.1.2">subscript</csymbol><ci id="A3.Ex5.m1.3.3.1.1.2.2.cmml" xref="A3.Ex5.m1.3.3.1.1.2.2">𝑡</ci><ci id="A3.Ex5.m1.3.3.1.1.2.3a.cmml" xref="A3.Ex5.m1.3.3.1.1.2.3"><mtext class="ltx_mathvariant_italic" mathsize="70%" id="A3.Ex5.m1.3.3.1.1.2.3.cmml" xref="A3.Ex5.m1.3.3.1.1.2.3">comm_weights</mtext></ci></apply><apply id="A3.Ex5.m1.2.2.cmml" xref="A3.Ex5.m1.2.2"><divide id="A3.Ex5.m1.2.2.3.cmml" xref="A3.Ex5.m1.2.2"></divide><apply id="A3.Ex5.m1.2.2.2.cmml" xref="A3.Ex5.m1.2.2.2"><times id="A3.Ex5.m1.2.2.2.3.cmml" xref="A3.Ex5.m1.2.2.2.3"></times><apply id="A3.Ex5.m1.1.1.1.1.1.1.cmml" xref="A3.Ex5.m1.1.1.1.1.1"><plus id="A3.Ex5.m1.1.1.1.1.1.1.1.cmml" xref="A3.Ex5.m1.1.1.1.1.1.1.1"></plus><apply id="A3.Ex5.m1.1.1.1.1.1.1.2.cmml" xref="A3.Ex5.m1.1.1.1.1.1.1.2"><times id="A3.Ex5.m1.1.1.1.1.1.1.2.1.cmml" xref="A3.Ex5.m1.1.1.1.1.1.1.2.1"></times><cn type="integer" id="A3.Ex5.m1.1.1.1.1.1.1.2.2.cmml" xref="A3.Ex5.m1.1.1.1.1.1.1.2.2">4</cn><ci id="A3.Ex5.m1.1.1.1.1.1.1.2.3.cmml" xref="A3.Ex5.m1.1.1.1.1.1.1.2.3">𝐻</ci><ci id="A3.Ex5.m1.1.1.1.1.1.1.2.4.cmml" xref="A3.Ex5.m1.1.1.1.1.1.1.2.4">𝑍</ci><ci id="A3.Ex5.m1.1.1.1.1.1.1.2.5.cmml" xref="A3.Ex5.m1.1.1.1.1.1.1.2.5">𝑑</ci></apply><apply id="A3.Ex5.m1.1.1.1.1.1.1.3.cmml" xref="A3.Ex5.m1.1.1.1.1.1.1.3"><times id="A3.Ex5.m1.1.1.1.1.1.1.3.1.cmml" xref="A3.Ex5.m1.1.1.1.1.1.1.3.1"></times><cn type="integer" id="A3.Ex5.m1.1.1.1.1.1.1.3.2.cmml" xref="A3.Ex5.m1.1.1.1.1.1.1.3.2">2</cn><ci id="A3.Ex5.m1.1.1.1.1.1.1.3.3.cmml" xref="A3.Ex5.m1.1.1.1.1.1.1.3.3">𝐻</ci><apply id="A3.Ex5.m1.1.1.1.1.1.1.3.4.cmml" xref="A3.Ex5.m1.1.1.1.1.1.1.3.4"><csymbol cd="ambiguous" id="A3.Ex5.m1.1.1.1.1.1.1.3.4.1.cmml" xref="A3.Ex5.m1.1.1.1.1.1.1.3.4">subscript</csymbol><ci id="A3.Ex5.m1.1.1.1.1.1.1.3.4.2.cmml" xref="A3.Ex5.m1.1.1.1.1.1.1.3.4.2">𝑑</ci><apply id="A3.Ex5.m1.1.1.1.1.1.1.3.4.3.cmml" xref="A3.Ex5.m1.1.1.1.1.1.1.3.4.3"><times id="A3.Ex5.m1.1.1.1.1.1.1.3.4.3.1.cmml" xref="A3.Ex5.m1.1.1.1.1.1.1.3.4.3.1"></times><ci id="A3.Ex5.m1.1.1.1.1.1.1.3.4.3.2.cmml" xref="A3.Ex5.m1.1.1.1.1.1.1.3.4.3.2">𝑓</ci><ci id="A3.Ex5.m1.1.1.1.1.1.1.3.4.3.3.cmml" xref="A3.Ex5.m1.1.1.1.1.1.1.3.4.3.3">𝑓</ci><ci id="A3.Ex5.m1.1.1.1.1.1.1.3.4.3.4.cmml" xref="A3.Ex5.m1.1.1.1.1.1.1.3.4.3.4">𝑛</ci></apply></apply></apply></apply><ci id="A3.Ex5.m1.2.2.2.4.cmml" xref="A3.Ex5.m1.2.2.2.4">𝑀</ci><apply id="A3.Ex5.m1.2.2.2.2.1.1.cmml" xref="A3.Ex5.m1.2.2.2.2.1"><minus id="A3.Ex5.m1.2.2.2.2.1.1.1.cmml" xref="A3.Ex5.m1.2.2.2.2.1.1.1"></minus><ci id="A3.Ex5.m1.2.2.2.2.1.1.2.cmml" xref="A3.Ex5.m1.2.2.2.2.1.1.2">𝐺</ci><cn type="integer" id="A3.Ex5.m1.2.2.2.2.1.1.3.cmml" xref="A3.Ex5.m1.2.2.2.2.1.1.3">1</cn></apply></apply><apply id="A3.Ex5.m1.2.2.4.cmml" xref="A3.Ex5.m1.2.2.4"><times id="A3.Ex5.m1.2.2.4.1.cmml" xref="A3.Ex5.m1.2.2.4.1"></times><ci id="A3.Ex5.m1.2.2.4.2.cmml" xref="A3.Ex5.m1.2.2.4.2">𝑏</ci><ci id="A3.Ex5.m1.2.2.4.3.cmml" xref="A3.Ex5.m1.2.2.4.3">𝐺</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex5.m1.3c">t_{\text{comm\_weights}}=\frac{(4\times H\times Z\times d+2\times H\times d_{ffn})\times M\times(G-1)}{b\times G},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
</div>
<div id="A3.Thmtheorem5" class="ltx_theorem ltx_theorem_proposition">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A3.Thmtheorem5.1.1.1" class="ltx_text ltx_font_bold">Proposition C.5</span></span><span id="A3.Thmtheorem5.2.2" class="ltx_text ltx_font_bold"> </span>(Runtime Calculation in BurstAttention)<span id="A3.Thmtheorem5.3.3" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="A3.Thmtheorem5.p1" class="ltx_para">
<p id="A3.Thmtheorem5.p1.4" class="ltx_p"><span id="A3.Thmtheorem5.p1.4.1" class="ltx_text ltx_font_italic">The total runtime for the BurstAttention framework is calculated as:</span></p>
<table id="A3.Ex6" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A3.Ex6.m1.3" class="ltx_Math" alttext="T_{\text{total}}=\max(T_{\text{attn\_f}},t_{\text{comm\_attn\_f}})+\max(T_{\text{attn\_b}},t_{\text{comm\_attn\_b}})+T_{\text{ffn}}+t_{\text{comm\_weights}}," display="block"><semantics id="A3.Ex6.m1.3a"><mrow id="A3.Ex6.m1.3.3.1" xref="A3.Ex6.m1.3.3.1.1.cmml"><mrow id="A3.Ex6.m1.3.3.1.1" xref="A3.Ex6.m1.3.3.1.1.cmml"><msub id="A3.Ex6.m1.3.3.1.1.6" xref="A3.Ex6.m1.3.3.1.1.6.cmml"><mi id="A3.Ex6.m1.3.3.1.1.6.2" xref="A3.Ex6.m1.3.3.1.1.6.2.cmml">T</mi><mtext class="ltx_mathvariant_italic" id="A3.Ex6.m1.3.3.1.1.6.3" xref="A3.Ex6.m1.3.3.1.1.6.3a.cmml">total</mtext></msub><mo id="A3.Ex6.m1.3.3.1.1.5" xref="A3.Ex6.m1.3.3.1.1.5.cmml">=</mo><mrow id="A3.Ex6.m1.3.3.1.1.4" xref="A3.Ex6.m1.3.3.1.1.4.cmml"><mrow id="A3.Ex6.m1.3.3.1.1.2.2.2" xref="A3.Ex6.m1.3.3.1.1.2.2.3.cmml"><mi id="A3.Ex6.m1.1.1" xref="A3.Ex6.m1.1.1.cmml">max</mi><mo id="A3.Ex6.m1.3.3.1.1.2.2.2a" xref="A3.Ex6.m1.3.3.1.1.2.2.3.cmml">⁡</mo><mrow id="A3.Ex6.m1.3.3.1.1.2.2.2.2" xref="A3.Ex6.m1.3.3.1.1.2.2.3.cmml"><mo stretchy="false" id="A3.Ex6.m1.3.3.1.1.2.2.2.2.3" xref="A3.Ex6.m1.3.3.1.1.2.2.3.cmml">(</mo><msub id="A3.Ex6.m1.3.3.1.1.1.1.1.1.1" xref="A3.Ex6.m1.3.3.1.1.1.1.1.1.1.cmml"><mi id="A3.Ex6.m1.3.3.1.1.1.1.1.1.1.2" xref="A3.Ex6.m1.3.3.1.1.1.1.1.1.1.2.cmml">T</mi><mtext class="ltx_mathvariant_italic" id="A3.Ex6.m1.3.3.1.1.1.1.1.1.1.3" xref="A3.Ex6.m1.3.3.1.1.1.1.1.1.1.3a.cmml">attn_f</mtext></msub><mo id="A3.Ex6.m1.3.3.1.1.2.2.2.2.4" xref="A3.Ex6.m1.3.3.1.1.2.2.3.cmml">,</mo><msub id="A3.Ex6.m1.3.3.1.1.2.2.2.2.2" xref="A3.Ex6.m1.3.3.1.1.2.2.2.2.2.cmml"><mi id="A3.Ex6.m1.3.3.1.1.2.2.2.2.2.2" xref="A3.Ex6.m1.3.3.1.1.2.2.2.2.2.2.cmml">t</mi><mtext class="ltx_mathvariant_italic" id="A3.Ex6.m1.3.3.1.1.2.2.2.2.2.3" xref="A3.Ex6.m1.3.3.1.1.2.2.2.2.2.3a.cmml">comm_attn_f</mtext></msub><mo stretchy="false" id="A3.Ex6.m1.3.3.1.1.2.2.2.2.5" xref="A3.Ex6.m1.3.3.1.1.2.2.3.cmml">)</mo></mrow></mrow><mo id="A3.Ex6.m1.3.3.1.1.4.5" xref="A3.Ex6.m1.3.3.1.1.4.5.cmml">+</mo><mrow id="A3.Ex6.m1.3.3.1.1.4.4.2" xref="A3.Ex6.m1.3.3.1.1.4.4.3.cmml"><mi id="A3.Ex6.m1.2.2" xref="A3.Ex6.m1.2.2.cmml">max</mi><mo id="A3.Ex6.m1.3.3.1.1.4.4.2a" xref="A3.Ex6.m1.3.3.1.1.4.4.3.cmml">⁡</mo><mrow id="A3.Ex6.m1.3.3.1.1.4.4.2.2" xref="A3.Ex6.m1.3.3.1.1.4.4.3.cmml"><mo stretchy="false" id="A3.Ex6.m1.3.3.1.1.4.4.2.2.3" xref="A3.Ex6.m1.3.3.1.1.4.4.3.cmml">(</mo><msub id="A3.Ex6.m1.3.3.1.1.3.3.1.1.1" xref="A3.Ex6.m1.3.3.1.1.3.3.1.1.1.cmml"><mi id="A3.Ex6.m1.3.3.1.1.3.3.1.1.1.2" xref="A3.Ex6.m1.3.3.1.1.3.3.1.1.1.2.cmml">T</mi><mtext class="ltx_mathvariant_italic" id="A3.Ex6.m1.3.3.1.1.3.3.1.1.1.3" xref="A3.Ex6.m1.3.3.1.1.3.3.1.1.1.3a.cmml">attn_b</mtext></msub><mo id="A3.Ex6.m1.3.3.1.1.4.4.2.2.4" xref="A3.Ex6.m1.3.3.1.1.4.4.3.cmml">,</mo><msub id="A3.Ex6.m1.3.3.1.1.4.4.2.2.2" xref="A3.Ex6.m1.3.3.1.1.4.4.2.2.2.cmml"><mi id="A3.Ex6.m1.3.3.1.1.4.4.2.2.2.2" xref="A3.Ex6.m1.3.3.1.1.4.4.2.2.2.2.cmml">t</mi><mtext class="ltx_mathvariant_italic" id="A3.Ex6.m1.3.3.1.1.4.4.2.2.2.3" xref="A3.Ex6.m1.3.3.1.1.4.4.2.2.2.3a.cmml">comm_attn_b</mtext></msub><mo stretchy="false" id="A3.Ex6.m1.3.3.1.1.4.4.2.2.5" xref="A3.Ex6.m1.3.3.1.1.4.4.3.cmml">)</mo></mrow></mrow><mo id="A3.Ex6.m1.3.3.1.1.4.5a" xref="A3.Ex6.m1.3.3.1.1.4.5.cmml">+</mo><msub id="A3.Ex6.m1.3.3.1.1.4.6" xref="A3.Ex6.m1.3.3.1.1.4.6.cmml"><mi id="A3.Ex6.m1.3.3.1.1.4.6.2" xref="A3.Ex6.m1.3.3.1.1.4.6.2.cmml">T</mi><mtext class="ltx_mathvariant_italic" id="A3.Ex6.m1.3.3.1.1.4.6.3" xref="A3.Ex6.m1.3.3.1.1.4.6.3a.cmml">ffn</mtext></msub><mo id="A3.Ex6.m1.3.3.1.1.4.5b" xref="A3.Ex6.m1.3.3.1.1.4.5.cmml">+</mo><msub id="A3.Ex6.m1.3.3.1.1.4.7" xref="A3.Ex6.m1.3.3.1.1.4.7.cmml"><mi id="A3.Ex6.m1.3.3.1.1.4.7.2" xref="A3.Ex6.m1.3.3.1.1.4.7.2.cmml">t</mi><mtext class="ltx_mathvariant_italic" id="A3.Ex6.m1.3.3.1.1.4.7.3" xref="A3.Ex6.m1.3.3.1.1.4.7.3a.cmml">comm_weights</mtext></msub></mrow></mrow><mo id="A3.Ex6.m1.3.3.1.2" xref="A3.Ex6.m1.3.3.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="A3.Ex6.m1.3b"><apply id="A3.Ex6.m1.3.3.1.1.cmml" xref="A3.Ex6.m1.3.3.1"><eq id="A3.Ex6.m1.3.3.1.1.5.cmml" xref="A3.Ex6.m1.3.3.1.1.5"></eq><apply id="A3.Ex6.m1.3.3.1.1.6.cmml" xref="A3.Ex6.m1.3.3.1.1.6"><csymbol cd="ambiguous" id="A3.Ex6.m1.3.3.1.1.6.1.cmml" xref="A3.Ex6.m1.3.3.1.1.6">subscript</csymbol><ci id="A3.Ex6.m1.3.3.1.1.6.2.cmml" xref="A3.Ex6.m1.3.3.1.1.6.2">𝑇</ci><ci id="A3.Ex6.m1.3.3.1.1.6.3a.cmml" xref="A3.Ex6.m1.3.3.1.1.6.3"><mtext class="ltx_mathvariant_italic" mathsize="70%" id="A3.Ex6.m1.3.3.1.1.6.3.cmml" xref="A3.Ex6.m1.3.3.1.1.6.3">total</mtext></ci></apply><apply id="A3.Ex6.m1.3.3.1.1.4.cmml" xref="A3.Ex6.m1.3.3.1.1.4"><plus id="A3.Ex6.m1.3.3.1.1.4.5.cmml" xref="A3.Ex6.m1.3.3.1.1.4.5"></plus><apply id="A3.Ex6.m1.3.3.1.1.2.2.3.cmml" xref="A3.Ex6.m1.3.3.1.1.2.2.2"><max id="A3.Ex6.m1.1.1.cmml" xref="A3.Ex6.m1.1.1"></max><apply id="A3.Ex6.m1.3.3.1.1.1.1.1.1.1.cmml" xref="A3.Ex6.m1.3.3.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A3.Ex6.m1.3.3.1.1.1.1.1.1.1.1.cmml" xref="A3.Ex6.m1.3.3.1.1.1.1.1.1.1">subscript</csymbol><ci id="A3.Ex6.m1.3.3.1.1.1.1.1.1.1.2.cmml" xref="A3.Ex6.m1.3.3.1.1.1.1.1.1.1.2">𝑇</ci><ci id="A3.Ex6.m1.3.3.1.1.1.1.1.1.1.3a.cmml" xref="A3.Ex6.m1.3.3.1.1.1.1.1.1.1.3"><mtext class="ltx_mathvariant_italic" mathsize="70%" id="A3.Ex6.m1.3.3.1.1.1.1.1.1.1.3.cmml" xref="A3.Ex6.m1.3.3.1.1.1.1.1.1.1.3">attn_f</mtext></ci></apply><apply id="A3.Ex6.m1.3.3.1.1.2.2.2.2.2.cmml" xref="A3.Ex6.m1.3.3.1.1.2.2.2.2.2"><csymbol cd="ambiguous" id="A3.Ex6.m1.3.3.1.1.2.2.2.2.2.1.cmml" xref="A3.Ex6.m1.3.3.1.1.2.2.2.2.2">subscript</csymbol><ci id="A3.Ex6.m1.3.3.1.1.2.2.2.2.2.2.cmml" xref="A3.Ex6.m1.3.3.1.1.2.2.2.2.2.2">𝑡</ci><ci id="A3.Ex6.m1.3.3.1.1.2.2.2.2.2.3a.cmml" xref="A3.Ex6.m1.3.3.1.1.2.2.2.2.2.3"><mtext class="ltx_mathvariant_italic" mathsize="70%" id="A3.Ex6.m1.3.3.1.1.2.2.2.2.2.3.cmml" xref="A3.Ex6.m1.3.3.1.1.2.2.2.2.2.3">comm_attn_f</mtext></ci></apply></apply><apply id="A3.Ex6.m1.3.3.1.1.4.4.3.cmml" xref="A3.Ex6.m1.3.3.1.1.4.4.2"><max id="A3.Ex6.m1.2.2.cmml" xref="A3.Ex6.m1.2.2"></max><apply id="A3.Ex6.m1.3.3.1.1.3.3.1.1.1.cmml" xref="A3.Ex6.m1.3.3.1.1.3.3.1.1.1"><csymbol cd="ambiguous" id="A3.Ex6.m1.3.3.1.1.3.3.1.1.1.1.cmml" xref="A3.Ex6.m1.3.3.1.1.3.3.1.1.1">subscript</csymbol><ci id="A3.Ex6.m1.3.3.1.1.3.3.1.1.1.2.cmml" xref="A3.Ex6.m1.3.3.1.1.3.3.1.1.1.2">𝑇</ci><ci id="A3.Ex6.m1.3.3.1.1.3.3.1.1.1.3a.cmml" xref="A3.Ex6.m1.3.3.1.1.3.3.1.1.1.3"><mtext class="ltx_mathvariant_italic" mathsize="70%" id="A3.Ex6.m1.3.3.1.1.3.3.1.1.1.3.cmml" xref="A3.Ex6.m1.3.3.1.1.3.3.1.1.1.3">attn_b</mtext></ci></apply><apply id="A3.Ex6.m1.3.3.1.1.4.4.2.2.2.cmml" xref="A3.Ex6.m1.3.3.1.1.4.4.2.2.2"><csymbol cd="ambiguous" id="A3.Ex6.m1.3.3.1.1.4.4.2.2.2.1.cmml" xref="A3.Ex6.m1.3.3.1.1.4.4.2.2.2">subscript</csymbol><ci id="A3.Ex6.m1.3.3.1.1.4.4.2.2.2.2.cmml" xref="A3.Ex6.m1.3.3.1.1.4.4.2.2.2.2">𝑡</ci><ci id="A3.Ex6.m1.3.3.1.1.4.4.2.2.2.3a.cmml" xref="A3.Ex6.m1.3.3.1.1.4.4.2.2.2.3"><mtext class="ltx_mathvariant_italic" mathsize="70%" id="A3.Ex6.m1.3.3.1.1.4.4.2.2.2.3.cmml" xref="A3.Ex6.m1.3.3.1.1.4.4.2.2.2.3">comm_attn_b</mtext></ci></apply></apply><apply id="A3.Ex6.m1.3.3.1.1.4.6.cmml" xref="A3.Ex6.m1.3.3.1.1.4.6"><csymbol cd="ambiguous" id="A3.Ex6.m1.3.3.1.1.4.6.1.cmml" xref="A3.Ex6.m1.3.3.1.1.4.6">subscript</csymbol><ci id="A3.Ex6.m1.3.3.1.1.4.6.2.cmml" xref="A3.Ex6.m1.3.3.1.1.4.6.2">𝑇</ci><ci id="A3.Ex6.m1.3.3.1.1.4.6.3a.cmml" xref="A3.Ex6.m1.3.3.1.1.4.6.3"><mtext class="ltx_mathvariant_italic" mathsize="70%" id="A3.Ex6.m1.3.3.1.1.4.6.3.cmml" xref="A3.Ex6.m1.3.3.1.1.4.6.3">ffn</mtext></ci></apply><apply id="A3.Ex6.m1.3.3.1.1.4.7.cmml" xref="A3.Ex6.m1.3.3.1.1.4.7"><csymbol cd="ambiguous" id="A3.Ex6.m1.3.3.1.1.4.7.1.cmml" xref="A3.Ex6.m1.3.3.1.1.4.7">subscript</csymbol><ci id="A3.Ex6.m1.3.3.1.1.4.7.2.cmml" xref="A3.Ex6.m1.3.3.1.1.4.7.2">𝑡</ci><ci id="A3.Ex6.m1.3.3.1.1.4.7.3a.cmml" xref="A3.Ex6.m1.3.3.1.1.4.7.3"><mtext class="ltx_mathvariant_italic" mathsize="70%" id="A3.Ex6.m1.3.3.1.1.4.7.3.cmml" xref="A3.Ex6.m1.3.3.1.1.4.7.3">comm_weights</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex6.m1.3c">T_{\text{total}}=\max(T_{\text{attn\_f}},t_{\text{comm\_attn\_f}})+\max(T_{\text{attn\_b}},t_{\text{comm\_attn\_b}})+T_{\text{ffn}}+t_{\text{comm\_weights}},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="A3.Thmtheorem5.p1.3" class="ltx_p"><span id="A3.Thmtheorem5.p1.3.3" class="ltx_text ltx_font_italic">where <math id="A3.Thmtheorem5.p1.1.1.m1.1" class="ltx_Math" alttext="T_{\text{attn\_f}}" display="inline"><semantics id="A3.Thmtheorem5.p1.1.1.m1.1a"><msub id="A3.Thmtheorem5.p1.1.1.m1.1.1" xref="A3.Thmtheorem5.p1.1.1.m1.1.1.cmml"><mi id="A3.Thmtheorem5.p1.1.1.m1.1.1.2" xref="A3.Thmtheorem5.p1.1.1.m1.1.1.2.cmml">T</mi><mtext class="ltx_mathvariant_italic" id="A3.Thmtheorem5.p1.1.1.m1.1.1.3" xref="A3.Thmtheorem5.p1.1.1.m1.1.1.3a.cmml">attn_f</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem5.p1.1.1.m1.1b"><apply id="A3.Thmtheorem5.p1.1.1.m1.1.1.cmml" xref="A3.Thmtheorem5.p1.1.1.m1.1.1"><csymbol cd="ambiguous" id="A3.Thmtheorem5.p1.1.1.m1.1.1.1.cmml" xref="A3.Thmtheorem5.p1.1.1.m1.1.1">subscript</csymbol><ci id="A3.Thmtheorem5.p1.1.1.m1.1.1.2.cmml" xref="A3.Thmtheorem5.p1.1.1.m1.1.1.2">𝑇</ci><ci id="A3.Thmtheorem5.p1.1.1.m1.1.1.3a.cmml" xref="A3.Thmtheorem5.p1.1.1.m1.1.1.3"><mtext class="ltx_mathvariant_italic" mathsize="70%" id="A3.Thmtheorem5.p1.1.1.m1.1.1.3.cmml" xref="A3.Thmtheorem5.p1.1.1.m1.1.1.3">attn_f</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem5.p1.1.1.m1.1c">T_{\text{attn\_f}}</annotation></semantics></math> and <math id="A3.Thmtheorem5.p1.2.2.m2.1" class="ltx_Math" alttext="T_{\text{attn\_b}}" display="inline"><semantics id="A3.Thmtheorem5.p1.2.2.m2.1a"><msub id="A3.Thmtheorem5.p1.2.2.m2.1.1" xref="A3.Thmtheorem5.p1.2.2.m2.1.1.cmml"><mi id="A3.Thmtheorem5.p1.2.2.m2.1.1.2" xref="A3.Thmtheorem5.p1.2.2.m2.1.1.2.cmml">T</mi><mtext class="ltx_mathvariant_italic" id="A3.Thmtheorem5.p1.2.2.m2.1.1.3" xref="A3.Thmtheorem5.p1.2.2.m2.1.1.3a.cmml">attn_b</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem5.p1.2.2.m2.1b"><apply id="A3.Thmtheorem5.p1.2.2.m2.1.1.cmml" xref="A3.Thmtheorem5.p1.2.2.m2.1.1"><csymbol cd="ambiguous" id="A3.Thmtheorem5.p1.2.2.m2.1.1.1.cmml" xref="A3.Thmtheorem5.p1.2.2.m2.1.1">subscript</csymbol><ci id="A3.Thmtheorem5.p1.2.2.m2.1.1.2.cmml" xref="A3.Thmtheorem5.p1.2.2.m2.1.1.2">𝑇</ci><ci id="A3.Thmtheorem5.p1.2.2.m2.1.1.3a.cmml" xref="A3.Thmtheorem5.p1.2.2.m2.1.1.3"><mtext class="ltx_mathvariant_italic" mathsize="70%" id="A3.Thmtheorem5.p1.2.2.m2.1.1.3.cmml" xref="A3.Thmtheorem5.p1.2.2.m2.1.1.3">attn_b</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem5.p1.2.2.m2.1c">T_{\text{attn\_b}}</annotation></semantics></math> represent the computation times for the forward and backward processes of the attention module, respectively, and <math id="A3.Thmtheorem5.p1.3.3.m3.1" class="ltx_Math" alttext="T_{\text{ffn}}" display="inline"><semantics id="A3.Thmtheorem5.p1.3.3.m3.1a"><msub id="A3.Thmtheorem5.p1.3.3.m3.1.1" xref="A3.Thmtheorem5.p1.3.3.m3.1.1.cmml"><mi id="A3.Thmtheorem5.p1.3.3.m3.1.1.2" xref="A3.Thmtheorem5.p1.3.3.m3.1.1.2.cmml">T</mi><mtext class="ltx_mathvariant_italic" id="A3.Thmtheorem5.p1.3.3.m3.1.1.3" xref="A3.Thmtheorem5.p1.3.3.m3.1.1.3a.cmml">ffn</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.Thmtheorem5.p1.3.3.m3.1b"><apply id="A3.Thmtheorem5.p1.3.3.m3.1.1.cmml" xref="A3.Thmtheorem5.p1.3.3.m3.1.1"><csymbol cd="ambiguous" id="A3.Thmtheorem5.p1.3.3.m3.1.1.1.cmml" xref="A3.Thmtheorem5.p1.3.3.m3.1.1">subscript</csymbol><ci id="A3.Thmtheorem5.p1.3.3.m3.1.1.2.cmml" xref="A3.Thmtheorem5.p1.3.3.m3.1.1.2">𝑇</ci><ci id="A3.Thmtheorem5.p1.3.3.m3.1.1.3a.cmml" xref="A3.Thmtheorem5.p1.3.3.m3.1.1.3"><mtext class="ltx_mathvariant_italic" mathsize="70%" id="A3.Thmtheorem5.p1.3.3.m3.1.1.3.cmml" xref="A3.Thmtheorem5.p1.3.3.m3.1.1.3">ffn</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Thmtheorem5.p1.3.3.m3.1c">T_{\text{ffn}}</annotation></semantics></math> is the runtime of the feedforward module.</span></p>
</div>
</div>
<div id="A3.Thmtheorem6" class="ltx_theorem ltx_theorem_remark">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="A3.Thmtheorem6.1.1.1" class="ltx_text ltx_font_italic">Remark C.6</span></span><span id="A3.Thmtheorem6.2.2" class="ltx_text ltx_font_italic">.</span>
</h6>
<div id="A3.Thmtheorem6.p1" class="ltx_para">
<p id="A3.Thmtheorem6.p1.1" class="ltx_p">Same with analysis of TP, this analysis does not account for sparse, approximate, or causal attention methods that could alter computational and communication complexities.</p>
</div>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Perplexity</h2>

<div id="A4.p1" class="ltx_para">
<p id="A4.p1.1" class="ltx_p">We sample 100 examples from C4 <cite class="ltx_cite ltx_citemacro_citep">(Raffel et al., <a href="#bib.bib21" title="" class="ltx_ref">2020</a>)</cite> and evaluate the perplexity (PPL) of LLaMA-7b implemented based on different distributed attention solutions. By evaluating PPL scores, we can evaluate the correctness of these implementation. From Table <a href="#A4.T4" title="Table 4 ‣ Appendix D Perplexity ‣ BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, we can find BurstAttention would not bring performance penalty, as compared to other distributed attention solutions.</p>
</div>
<figure id="A4.T4" class="ltx_table ltx_figure_panel ltx_align_center">
<div id="A4.T4.1" class="ltx_inline-block ltx_figure_panel ltx_transformed_outer" style="width:135.7pt;height:97.2pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-7.5pt,5.4pt) scale(0.9,0.9) ;">
<table id="A4.T4.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A4.T4.1.1.1.1" class="ltx_tr">
<th id="A4.T4.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_r ltx_border_tt"><span id="A4.T4.1.1.1.1.1.1" class="ltx_text" style="font-size:90%;">Method</span></th>
<th id="A4.T4.1.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="A4.T4.1.1.1.1.2.1" class="ltx_text" style="font-size:90%;">PPL</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A4.T4.1.1.2.1" class="ltx_tr">
<td id="A4.T4.1.1.2.1.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t"><span id="A4.T4.1.1.2.1.1.1" class="ltx_text" style="font-size:90%;">TP</span></td>
<td id="A4.T4.1.1.2.1.2" class="ltx_td ltx_align_left ltx_border_t"><span id="A4.T4.1.1.2.1.2.1" class="ltx_text" style="font-size:90%;">9.901</span></td>
</tr>
<tr id="A4.T4.1.1.3.2" class="ltx_tr">
<td id="A4.T4.1.1.3.2.1" class="ltx_td ltx_align_left ltx_border_r"><span id="A4.T4.1.1.3.2.1.1" class="ltx_text" style="font-size:90%;">TP w/ FlashAttention</span></td>
<td id="A4.T4.1.1.3.2.2" class="ltx_td ltx_align_left"><span id="A4.T4.1.1.3.2.2.1" class="ltx_text" style="font-size:90%;">9.902</span></td>
</tr>
<tr id="A4.T4.1.1.4.3" class="ltx_tr">
<td id="A4.T4.1.1.4.3.1" class="ltx_td ltx_align_left ltx_border_r"><span id="A4.T4.1.1.4.3.1.1" class="ltx_text" style="font-size:90%;">RingAttention</span></td>
<td id="A4.T4.1.1.4.3.2" class="ltx_td ltx_align_left"><span id="A4.T4.1.1.4.3.2.1" class="ltx_text" style="font-size:90%;">9.904</span></td>
</tr>
<tr id="A4.T4.1.1.5.4" class="ltx_tr">
<td id="A4.T4.1.1.5.4.1" class="ltx_td ltx_align_left ltx_border_r"><span id="A4.T4.1.1.5.4.1.1" class="ltx_text" style="font-size:90%;">BurstAttention w/o LAO</span></td>
<td id="A4.T4.1.1.5.4.2" class="ltx_td ltx_align_left"><span id="A4.T4.1.1.5.4.2.1" class="ltx_text" style="font-size:90%;">9.901</span></td>
</tr>
<tr id="A4.T4.1.1.6.5" class="ltx_tr">
<td id="A4.T4.1.1.6.5.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r"><span id="A4.T4.1.1.6.5.1.1" class="ltx_text" style="font-size:90%;">BurstAttention</span></td>
<td id="A4.T4.1.1.6.5.2" class="ltx_td ltx_align_left ltx_border_bb"><span id="A4.T4.1.1.6.5.2.1" class="ltx_text" style="font-size:90%;">9.901</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption" style="font-size:90%;"><span class="ltx_tag ltx_tag_table">Table 4: </span>LLaMA-7b PPL on C4.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2403.09346" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2403.09347" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2403.09347">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2403.09347" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2403.09348" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Fri Apr  5 14:14:59 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2403.09347
Effective attention modules have played a crucial role in the success of Transformer-based large language models
(LLMs) , but the quadratic time and memory complexities of these attention modules also pose a challenge when processing long sequences.
One potential solution for the long sequence problem is to utilize distributed clusters to parallelize the computation of attention modules across multiple devices
(e.g., GPUs) .
However, adopting a distributed approach inevitably introduces extra memory overheads to store local attention results and incurs additional communication costs to aggregate local results into global ones.
In this paper, we propose a distributed attention framework named “BurstAttention” to optimize memory access and communication operations at both the global cluster and local device levels by partitioning attention along the sequence dimension across device.
Through our experiments, we compare BurstAttention with other competitive long-sequence distributed attention solutions.
The experimental results under different lengths demonstrate that BurstAttention offers significant advantages for processing long sequences compared with these competitive baselines, especially tensor parallelism (Megatron-V3)  with FlashAttention, reducing 40% communication overheads and achieving 2speedup during training 128K sequence length on 8A100.
abstract
BurstAttention: An Efficient Distributed Attention Framework forExtremely Long Sequences
Abstract
2403.09347
1IntroductionTransformershave emerged as the dominant architectures for large language models (LLMs) due to their remarkable capacities to understand complex text and generate controllable responses. Empirically, the power of Transformers lies largely in their multi-head attention modules, which enable Transformers to capture rich semantic information from textual contexts effectively. For every plus, there is a minus. Despite the success of Transformers’ attention modules, these modules exhibit quadratic time and memory complexity concerning sequence length, posing challenges in terms of both computing time and memory overheads as sequence length increases.Various efforts have been devoted to making attention modules more efficient and enabling LLMs to process longer sequences.
One direction is taking full advantage of a single device’s compute and storage units (e.g., a GPU)  to process long sequences, such as FlashAttention. FlashAttention can significantly accelerate the computation of attention modules by using more efficient static random access memory (SRAM)  instead of high-bandwidth memory (HBM)  in devices to store intermediate attention states.
Another direction is using distributed clusters containing multiple devices (e.g., multiple GPUs)  to process long sequences, such as RingAttention. RingAttention divides sequences into multiple subsequences and processes subsequences separately on different devices.Figure 1:BurstAttention undertakes a two-step partitioning: dividing the sequence across multiple devices (inter-device) , and then splitting the subsequences within each single device (intra-device) . First, BurstAttention partitions the query, key, and value across devices and pass each sliced subsequence through all devices in a ring-like communication. This allows each device to process only a local attention at a time, and avoids the burden on memory caused by processing extremely long sequence at once.
By transmittingand aggregating local attention results using online softmax, BurstAttention avoids storing the intermediate result, which has quadratic memory complexity, and instead recomputes it during the backward pass, which we call global attention optimization (GAO) .
BurstAttention further partitions the subsequences into smaller tiles, aiming to perform block-wise computations within local attention. This can utilize the high bandwidth of SRAM while minimizing access to the lower bandwidth HBM, which we call local attention optimization (LAO) . Also, by using double-buffer, the communication can be overlapped with computation in BurstAttention.All the above improvements orienting to speedup attention operation have achieved promising results, each targeting different bottleneck. However an intuitive problem is raised — whether we can combine these improvements to achieve a more efficient attention solution.
The concept is straightforward, yet in a distributed setting, simple combination of two methods may not benefit from their strength. Moreover , the RingAttention approach cannot directly incorporate with online softmax, and the FlashAttention implementation focuses exclusively on optimizing the computation of attention on a single device. To address these challenges,this paper introduces an efficient distributed attention framework to handle extremely long sequences named “BurstAttention”. BurstAttention can take full advantage of the power of both distributed clusters and single devices within clusters.
Specifically, given an extremely long sequence, BurstAttention first divides the sequence into partitions according to the number of devices in distributed clusters, and each partition is assigned to one of these devices. Then, each device projects the partitioned sequence into query, value, and key embedding partitions. The query partitions are pinned, and all key-value partitions are passed through all devices to compute their local attention scores with each pinned query partition. Based on the local attention scores, a global attention operation is adopted to aggregate the local results into the final global results.By fine-grained scheduling the computation and communication operations of devices during computing attention modules, as well as introducing online softmax, BurstAttention proposes global attention optimization (GAO)  and local attention optimization (LAO) , which can fully optimize the input-output (I/O)  and communication procedures in distributed clusters. These two strategies offer substantial benefits for computing local attention scores in each device and aggregating local results into global ones in the whole cluster, including improved memory consumption, reduced communication overhead, and enhanced cache utilization.
Owing to just splitting sequences, BurstAttention is orthogonal to other distributed methods and can be integrated with them for training and inferring Transformer-based LLMs, such as data parallelism, tensor parallelism, pipeline parallelism, and zero redundancy optimizer.We evaluate BurstAttention and current competitive distributed attention solutionsunder various sequence length settings. Comparing to tensor parallelism (Megatron-V3)  with FlashAttention methods, our method reducing 40% communication overheads and achieving 2speedup during training 128K sequence length on 8A100. The experimental results show that BurstAttention is a memory-efficient solution for attention modules to process long sequences and achieve good data throughputs. Moreover, since BurstAttention greatly optimizes the communication operations during the computation process of attention modules, BurstAttention makes it more difficult for device’s communication to become a bottleneck as the devices in distributed clusters increase, and thus can better utilize distributed clusters than other distributed solutions.
introduction
BurstAttention: An Efficient Distributed Attention Framework forExtremely Long Sequences
1Introduction
2403.09347
2Methodology2.1PreliminaryAs the key module in Transformers, an attention module can be formalized as(1) whereindicates the embeddings of the query sequence,is the length of the query sequence, andis the embedding dimension.andindicate the embeddings of the key sequence and the value sequence, respectively.andindicate the attention scores and the attention probabilities, respectively.is the final attention result, which is the average of the value sequence embeddings weighted by the similarities between the query and key sequences. In this paper, we mainly use self-attention modules to illustrate BurstAttention, but BurstAttention can be easily extended to cross-attention modules.
For more details of various attention modules in the Transformer architecture, we recommend referring to the original paper of Transformers, and we will not go into details.Algorithm 1The forward pass of GAO0:Matriceson the-th device1:Initialize2:Putinto communication ring3:fortodo4:Conduct one step of ring communication;5:Getfrom communication ring;6:{The forward pass of local attention (w/o LAO) .}7:;8:;9:;10:;11:;12:{The end of the forward pass of local attention.}13:;14:;15:;16:;17:Putinto communication ring;18:endfor19:;20:;20:;0:Matrices,on the-th device1:Initialize2:(elementwise multiplication) 3:Putinto communication ring4:fortodo5:Conduct one step of ring communication;6:Getfrom communication ring;7:{The backward pass of local attention (w/o LAO) .}8:;9:;10:;11:;12:;13:;14:;15:{The end of the backward pass of local attention.}16:Putinto communication ring;17:endfor17:;Algorithm 2The backward pass of GAO2.2The Whole Framework of BurstAttentionIn BurstAttention,,andare divided into multiple partitions along the sequence dimension according to the number of devices (e.g., GPUs)  in a distributed cluster. Each device in the cluster will be assigned a query partition, a key partition, and a value partition.
Formally, given the device number, the-th device will be assigned. As shown in Figure1, at each step,
the-th device receives a key partitionand a value partitionfrom its previous neighbor and performs local attention operations. After that, the-th device sends its received key and value partitionsandto its next neighbor for the use of the next step, which forms a ring-style communication process. This ring-style communication process continues until allandpartitions have made a full circle around the ring, completing local attention operations on all devices.
The local attention operations can be formalized as(2) whereindicates the local attention results between the device-assigned query partitionand the device-received partitionsand.andindicate the local attention scores and the local attention probabilities, respectively.Obviously, Eq. (1)  and Eq. (2)  are not equivalent, we thus introduce global attention operations to aggregate all local attention resultsinto the final partitioned attention results, andis the final global attention results. To make both the global and local attention operations more efficient, we introduce Global Attention Optimization (GAO)  and Local Attention Optimization (LAO) , respectively. Next, we will introduce these attention optimization strategies in detail.2.3Global Attention Optimization (GAO) Global attention operations are to aggregatein Eq. (2)  into.
For some conventional methods such as RingAttention, for the-th query partition, they store the intermediate resultsandfor every. This introduces a non-negligible memory overhead.
To get rid of this memory overhead, we introduce GAO.As shown in Figure1, GAO consists of two main steps. First, devices are organized in a ring for communication. Each round,partitions are shifted along the ring to the next adjacent device. Second, after each round oftransmission, each deviceperforms a local attention operation using the partitionsand its received partition, and, as described in Eq. (2) . The local attention resultare then dynamically accumulated into global attention resultby employing online softmax, which eliminates the need to store intermediate resultsand.As depicted in Algorithm1, in the forward pass, we dynamically maintain the row-wise maximum valueofas in Line13and the row-wise sumofas in Line14to avoid storingand, and useandfor scaling during the aggregation ofas in Line15. Note that, the functionsandcan be formalized asi(3) whereis the-th element of the vector,is the element in the-th row and-th column of the matrix.
To make the subsequent backward pass more efficient, we storebesides the global resultsafter the forward pass. During the backward pass, as depicted in Algorithm2,
we employ the same strategy for the forward pass to obtain gradients based only on recomputedand.2.4Local Attention Optimization (LAO) Given,, and, the local attention operations that involve these partitions are performed only on a single device (e.g., a GPU) .
When computingin Eq. (2) ,andare computed and stored on the HBM of the device.
To avoid frequent I/O operations ofandon the HBM, the local attention operations of BurstAttention further divide,, andinto tiles along the sequence dimension, with each tilesequence length, whererepresents the SRAM size of the device,represents the attention head dimension.As shown in Figure1, during computing, each thread block reads the tiles offrom the HBM to SRAM, the tiles ofandare computed and then written on the SRAM instead of the HBM,are dynamically accumulated based on online softmax operations and written back to the HBM. Since the SRAM has a much higher I/O bandwidth than the HBM, the above optimization can make local attention operations more efficient.
Although the memory of the SRAM is tiny, further dividing,, andinto many fine-grained tiles ensure the intermediate resultsandcan be entirely stored into the SRAM.Intuitively, when BurstAttention is running on a single device rather than a distributed cluster, there is no need to use GAO, and LAO will play the same role as FlashAttention, i.e., FlashAttention can be viewed as a specialization of BurstAttention using a single device.MethodFlashATT/LAOMemory OverheadsCommunication OverheadsParameterActivationForwardBackwardRingAttentionw/oRingAttention†Tensor Parallelismw/oTensor Parallelismw/ FlashATTBurstAttentionw/oBurstAttentionw/ LAOTable 1:The overheads of different distributed attention solutions.is the device number,denotes the batch size,represents the sequence length,signifies the number of attention heads,corresponds to the hidden dimension per head,represents the model dimension of Transformers, andrepresents the device SRAM size.†means from an implementation perspective, RingAttention’s separatingandinto two independent rounds of communication cannot be combined with FlashAttention to improve efficiency.2.5Overlapping Communication and ComputationAlthough splitting sequences can efficiently utilize distributed clusters to handle the long-sequence attention, this also inevitably introduces additional time costs to transmit partitions between devices. To this end, BurstAttention leverages the potential of devices (e.g., GPUs)  for overlapping communication and computation.
This contrasts with some other typical distributed methods like tensor parallelism, where such overlapping is not feasible due to the dependency of subsequent layers’ computations on preceding layers’ outputs.To address this, BurstAttention adopts a double-buffer technique, enabling concurrent execution of communication and computation.
The technique designs two buffers for each device, one is used as input to local attention operations, and the other is used to receive data from other devices. As depicted in Figure1, each element (query, key, or value)  involved in the ring-style communication process is allocated a dedicated buffer. Concurrent with the initiation of each local attention round, the double-buffer technique triggers the transmission of the corresponding buffer tensor. This preemptive action ensures that, by the commencement of the subsequent local attention round, the required data is already available on each device, having been carried over by the buffer. The process is then repeated until all local attention operations are completed, with each round of local attention operations initiating the transmission of data required for the next round of local attention operations. More details can be found in our appendix3.2.6Integrating Sparse Attention MethodsVarious sparse attention methods, including low-rank methods, kernel-based methodsand downsampling methodsare also widely explored. These methods reduce the time and memory costs of attention modules by computing a limited selection of similarity scores from a sequence rather than all possible pairs, resulting in sparse attention softmax logits rather than dense ones. Recently,have explored sparse attention based on distributed clusters and achieved promising results.The sequence parallelism mechanism makes BurstAttention easy to cooperate with sparse attention methods. During the computation process of BurstAttention, given,,, if there is no need to compute the similarities between these partitions, then the local attention operations on these partitions can be skipped directly. If just some tokens in,andare required to compute their similarities for final attention results, we can similarly skip unnecessary operations in local attention operations. Note that these sparse attention methods inevitably lead to significant performance degradation, along with reducing the time and memory overheads. Although BurstAttention is well compatible with sparse attention methods, in the actual processing of long sequences, the use of these lossy methods needs to be cautious.
body
BurstAttention: An Efficient Distributed Attention Framework forExtremely Long Sequences
2Methodology
2403.09347
3Overhead AnalysisIn this section, we will analyze the memory, I/O, and communication overheads of BurstAttention as compared to existing competitive distributed attention solutions.
As data parallelism and pipeline parallelism are often used as the most basic distributed strategies and cannot reduce the cost of long sequence processing, we focus here on comparing BurstAttention, tensor parallelism, and the typical sequence parallelism method RingAttention.3.1Memory and I/O OverheadsWhen we split the input along the sequence dimension across devices for global operations and further split them in each device for local operations, the memory overheads caused bywill be reduced toof the original ones. Table1shows the memory overheads of various distributed attention solutions.
The table shows that BurstAttention has lower activation memory while tensor parallelism has lower parameter memory.
This means that the longer the sequence, the more pronounced the advantage of BurstAttention. Moreover, by combining BurstAttention with some parallelism strategies like zero redundancy optimizerto partition parameters, BurstAttention can easily obtain the same parameter memory overheads as tensor parallelism.
In terms of I/O overheads, RingAttention requiresmemory accesses on every single device of the whole cluster; tensor parallelism and BurstAttention only requirememory accesses.
This indicates that BurstAttention can significantly reduce I/O time costs compared to other distributed attention baselines.3.2Communication OverheadsIn the forward pass, BurstAttention involves one round of ring-style peer-to-peer communications on the, with a total cost of.
In the backward pass, BurstAttention requires one round of ring-style communication on tensorsand, with a total cost of.
Table1shows the communication overheads of various distributed attention solutions. The forward communication of RingAttention is the same as BurstAttention, which is, but without GAO and LAO, RingAttention requires a total cost ofin the backward pass, which is about twice that of BurstAttention. Therefore, BurstAttention has great advantage of communication overheads during training than RingAttention.
The forward communication of tensor parallelism isand the total communication is, thus BurstAttention also has higher communication efficiency during both inferring and training than tensor parallelism.Table 2:The first token latency of the LLaMA-7b inference (s) .Sequence Length4,0968,19216,38432,76865,536131,072262,144RingAttention0.42±0.010.87±0.012.00±0.015.13±0.05OOMOOMOOMTP(Megatron V1)  w/ Flash0.67±0.011.29±0.012.58±0.015.27±0.0111.63±0.0227.54±0.0171.52±0.06TP(Megatron V3)  w/ Flash0.73±0.021.36±0.012.68±0.015.67±0.0112.25±0.0128.73±0.0375.52±0.05BurstAttention w/o LAO0.46±0.010.88±0.011.79±0.013.88±0.0110.78±0.01OOMOOMBurstAttention0.44±0.010.84±0.011.68±0.013.27±0.016.49±0.0116.01±0.0149.32±0.11Table 3:The first token latency of the LLaMA-13b inference (s) .Sequence Length4,0968,19216,38432,76865,536131,072262,144RingAttention0.66±0.011.36±0.013.08±0.017.98±0.02OOMOOMOOMTP(Megatron V1)  w/ Flash1.05±0.012.01±0.014.03±0.018.41±0.0118.56±0.0244.39±0.04OOMTP(Megatron V3)  w/ Flash1.07±0.012.09±0.014.20±0.018.76±0.0119.06±0.0645.46±0.03119.03±0.04BurstAttention w/o LAO0.72±0.011.39±0.012.77±0.055.99±0.0116.95±0.01OOMOOMBurstAttention0.69±0.011.40±0.052.57±0.035.08±0.029.92±0.0125.91±0.0178.80±0.07(a) Training time(b) Training memoryFigure 2:The training time and memory of LLaMA-7b on 8A100.
body
BurstAttention: An Efficient Distributed Attention Framework forExtremely Long Sequences
3Overhead Analysis
2403.09347
4Experiments4.1Experimental SettingsWe perform experiments in two configurations: one involves a single node equipped with 8 A100 GPUs linked via PCI-E, and the other is a distributed setup comprising four identical nodes, each with the same 8 A100 GPU configuration, interconnected by a 600 Gb/s RoCE network. We adopts two LLMs’ settings in our experiments, LLaMA-2 with 7 billion parameters (7b)  and LLaMA-2 with 13 billion parameters (13b) .
Our experiments consist of the following methods:(1) TP, which refers to tensor parallelism, a commonly used distributed strategy in the stages of both training and inference. Note that here we futher classify TP intoTP (Megatron V1) andTP (Megatron V3) based on the detail communication operations (Megatron V1 uses the all-reduce operation while Megatron V3 uses the combination of the all-gather and reduce-scatter operations) .
(2) TP w/ FlashAttention, which combines FlashAttentionwith tensor parallelism as a strong baseline.Note that this is a commonly used strategy in current LLM pre-training and inference.(3) RingAttention, a typical sequence parallelism baseline.
(4) BurstAttention, our distributed attention method includes both GAO and LAO strategies.
(5) BurstAttention w/o LAO, where we remove the LAO strategy for ablation studies.
(6) BurstAttention+ZeRO, where we futher optimize the memory overhead of BurstAttention by adopting the ZeROtechnique to shard model parameters across devices.As we mentioned before, data parallelism and pipeline parallelism cannot effectively reduce the cost of long sequence processing, and we do not use them as baselines. In fact, we conduct some experiments to adapt data parallelism and pipeline parallelism for long-sequence attention, but unfortunately, these two parallelism methods cannot process extremely long sequences.From our pilot experiments, directly adopting data parallelism or pipeline parallelism can only handle sequences shorter than 8192, much shorter than RingAttention and TP.Our experiments does not specifically focus on any particular attention masking mechanism. However, for the methods we compared against, such as Tensor Parallelism (Megatron V3)  with FlashAttention, we adopt its causal implementation in these experiments. This means that our baselines can bypass half of the attention computations owing to the causal attention structure. We observe that this approach yields only a marginal improvement, as communication remains the bottleneck in our experimental environment. Notably, in our implementation of BurstAttention, the computation is overlapped by the communication, which is a key factor in the observed performance gains. This distinction is crucial to understand the context and the specific conditions under which our method demonstrates its advantages.4.2Inference LatencyIn this section, we focus on the latency needed for generating the first token (i.e., the first token latency)  in the inference process. We concentrate on the time of the first token generation because the long-sequence attention computation mainly exists in the inference encoding process. Since the first token latency is much higher than the latency of generating subsequent tokens, the first token latency thus becomes one of the most critical targets existing works seek to optimize.In real-time AI services such as ChatGPT, the system’s responsiveness significantly impacts the user experience, and these applications usually output results in a streaming manner to improve responsiveness. Since the first token latency is the longest, the first token latency directly influences the perceived responsiveness and efficiency of the model in these streaming scenarios.As shown in Table2and Table3, we can see that, compared with tensor parallelism, sequence parallelism methods are more suitable to infer long sequences. Compared with the RingAttention method, by using GAO, BurstAttention can support longer sequences. By further using LAO, BurstAttention can achieve more latency improvements and support much longer sequences. Note that, although TP (Megatron V3)  is more memory efficient than TP (Megatron V1) , the all-reduce operation used by TP (Megatron V1)  is better optimized than the reduce-scatter and all-gather operations used by TP(Megatron V3) . In the actual inference, TP(Megatron V1)  is slightly faster than TP (Megatron V3) . Since TP (Megatron V3)  has a similar time to TP (Megatron V1)  but better memory efficiency, we mainly compare our method with TP (Megatron V3)  in subsequent experiments.(a) Training time(b) Training memoryFigure 3:The training time and memory of LLaMA-7b on 32A100.(a) LLaMA-13b latency - GPU number(b) LLaMA-7b throughput - batch sizeFigure 4:Scaling abilities on different GPU numbers and batch sizes.4.3Training PerformanceFor training LLMs, a batch is required to have 2 to 4 million tokens, otherwise, the model performance may be degraded, i.e., the longer the sequence length is, the smaller the batch size is. Due to this, several GPUs may need to process one example together.
For example, using 2048 GPUs to train 128-layer GPT-3, the sequence length is 4096, the batch size is 1024, data parallelism is 16, pipeline parallelism is 32, and tensor parallelism is 4. In this scenario, the optimal setup is to divide a batch into 64 micro-batches with a micro-batch size of 1.
In this case, four GPUs under the same tensor parallelism group are inevitably required to process one piece of data together. In view of this, we fix the batch size to 1 for experimental convenience and vary the input sequence length from 1K to 32K.As can be seen from Figure2(a) , although tensor parallelism adopts FlashAttention to improve its processing of long sequences, both RingAttention and BurstAttention have better training time than tensor parallelism when processing long sequences. This is also why existing works using tensor parallelism to train LLMs usually set the training length between 2048 and 4096. Compared with BurstAttention, RingAttention is limited by the sequence length since it stores too many intermediate states, but BurstAttention can support the longest input length. On the other hand, BurstAttention without LAO has a similar trend of training time as RingAttention and tensor parallelism.From Figure3, BurstAttention achieves nearly 2.0speedup when the sequence is longer than 128K. Also combining BurstAttention with ZeRO optimization brings significant improvements in memory efficiency. Although BurstAttention+ZeRO brings little additional communication overheads, BurstAttention+ZeRO still achieves memory efficiency comparable to Megatron V3 and demonstrates superior speed in both multi-node and single-node setups than Megatron V3. This suggests that BurstAttention, with its current optimizations, offers a more efficient solution in terms of speed, even when faced with a memory-efficient competitor like Megatron V3.4.4Scaling AbilityIn this section, we further verify the scaling ability of BurstAttention.
In Figure4(a) , we set batch size to 1 and sequence length to 65,536, and then evaluate the latency changes with increasing GPU numbers. As shown in the figure, in the single-GPU scenario, BurstAttention with LAO is equivalent to FlashAttention, and its inference latency is on par with the baseline using FlashAttention. Tensor parallelism cannot further decrease the latency when the number of GPUs increases from 4 to 8 due to the communication overhead with increased batch-size, while BurstAttention can achieve better scaling trends. Note that RingAttention requires storingmemory for each layer, which is extremely large and cannot fit into GPUs even sharded on 8 GPUs. In Figure4(b) , we fix the sequence length to 4096 and the number of GPUs to 8 to evaluate the training throughput changes with increasing batch sizes. The experimental results show that BurstAttention can support a larger batch size, and the throughput grows with the increase of batch sizes in training scenario.
body
BurstAttention: An Efficient Distributed Attention Framework forExtremely Long Sequences
4Experiments
2403.09347
5Related WorkTransformer-based LLMs such as GPT, LLaMA, and PaLMhave achieved great success. Despite the success of these LLMs, they still face efficiency challenges: one is that as these models continue to grow in size, the time and memory costs associated with training and inference have become bottlenecks. Another is that the quadratic attention computational complexity of the Transformer architecture makes these LLMs difficult to handle long sequences. Up to now, various parallelism strategiesand memory optimization strategies,
have well solved the bottleneck caused by the model size growth, but it is still challenging to solve the efficiency issue caused by the sequence growth.To enable LLMs to process longer sequences more efficiently, several attention solutions have been proposed.adopt selective activation recomputation to avoid storing attention softmax logits during the forward pass, and then recompute these logits during the backward pass to build a computation graph for backpropagation, significantly reducing memory overheads of attention modules to process long sequences.formalize the computation of attention modules at the block level and make each thread block in devices handle the attention computation of a subsequence, further reducing temporary memory consumptions and achieving a logarithmic memory complexity relative to the sequence length. Based on these works,introduce FlashAttention, a CUDA implementation of attention modules that leverages the fast I/O capabilities of the SRAM in devices for further speedup. FlashAttention optimizes the attention algorithm by introducing I/O complexity analysis and minimizing the I/O costs on the HBM in devices, offering a new perspective on attention optimization.While the above solutions focus on optimizing the long-sequence attention problem using a single device, they still struggle to handle extremely long sequences due to the limitations of a single device’s performance. Some efforts have therefore aimed to address this long-sequence challenge using distributed clusters.
Adopting general parallelism strategies is most straightforward method, such as data parallelism, tensor parallelism, pipeline parallelism, and zero redundancy optimizer. To better process long sequences using distributed clusters,propose the sequence parallelism method RingAttention, which splits the computation and memory overheads of attention modules across multiple devices following the sequence dimension.
body
BurstAttention: An Efficient Distributed Attention Framework forExtremely Long Sequences
5Related Work
2403.09347
6ConclusionWe present an efficient distributed attention framework named BurstAttention, which can enhance performance in terms of memory consumption and running speed when processing extremely long sequences. When running on a single device, BurstAttention can achieve comparable efficiency to FlashAttention. When running on a distributed cluster, BurstAttention can outperform existing competitive distributed attention solutions, including RingAttention and tensor parallelism. Moreover, the experimental results show that BurstAttention also has greater scaling abilities than existing solutions as increasing devices and batch sizes.
body
BurstAttention: An Efficient Distributed Attention Framework forExtremely Long Sequences
6Conclusion
2403.09347
AcknowledgementsThanks for the hardware support of OpenBMB and technique support of Huawei.
conclusion
BurstAttention: An Efficient Distributed Attention Framework forExtremely Long Sequences
Acknowledgements