<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2407.04620] Learning to (Learn at Test Time): RNNs with Expressive Hidden States</title><meta property="og:description" content="Self-attention performs well in long context but has quadratic complexity.
Existing RNN layers have linear complexity, but their performance in long context is limited by the expressive power of their hidden state.
We …">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Learning to (Learn at Test Time): RNNs with Expressive Hidden States">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Learning to (Learn at Test Time): RNNs with Expressive Hidden States">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2407.04620">

<!--Generated on Mon Aug  5 18:07:45 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document" style="font-size:144%;">Learning to (Learn at Test Time):
<br class="ltx_break">RNNs with Expressive Hidden States

</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">
Yu Sun<math id="id6.6.m1.1" class="ltx_Math" alttext="~\,{}^{1}" display="inline"><semantics id="id6.6.m1.1a"><msup id="id6.6.m1.1.1" xref="id6.6.m1.1.1.cmml"><mi id="id6.6.m1.1.1a" xref="id6.6.m1.1.1.cmml"></mi><mn id="id6.6.m1.1.1.1" xref="id6.6.m1.1.1.1.cmml">1</mn></msup><annotation-xml encoding="MathML-Content" id="id6.6.m1.1b"><apply id="id6.6.m1.1.1.cmml" xref="id6.6.m1.1.1"><cn type="integer" id="id6.6.m1.1.1.1.cmml" xref="id6.6.m1.1.1.1">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="id6.6.m1.1c">~\,{}^{1}</annotation></semantics></math>,
Xinhao Li<sup id="id25.25.id1" class="ltx_sup">∗</sup><sup id="id26.26.id2" class="ltx_sup">2</sup>,
Karan Dalal<sup id="id27.27.id3" class="ltx_sup">∗</sup><sup id="id28.28.id4" class="ltx_sup">3</sup>, 
<br class="ltx_break">Jiarui Xu<sup id="id29.29.id5" class="ltx_sup">2</sup>,
Arjun Vikram<sup id="id30.30.id6" class="ltx_sup">1</sup>,
Genghan Zhang<sup id="id31.31.id7" class="ltx_sup">1</sup>,
Yann Dubois<sup id="id32.32.id8" class="ltx_sup">1</sup>, 
<br class="ltx_break">Xinlei Chen<sup id="id33.33.id9" class="ltx_sup"><span id="id33.33.id9.1" class="ltx_text ltx_font_italic">†</span></sup><sup id="id34.34.id10" class="ltx_sup">4</sup>,
Xiaolong Wang<sup id="id35.35.id11" class="ltx_sup"><span id="id35.35.id11.1" class="ltx_text ltx_font_italic">†</span></sup><sup id="id36.36.id12" class="ltx_sup">2</sup>,
Sanmi Koyejo<sup id="id37.37.id13" class="ltx_sup"><span id="id37.37.id13.1" class="ltx_text ltx_font_italic">†</span></sup><sup id="id38.38.id14" class="ltx_sup">1</sup>,
Tatsunori Hashimoto<sup id="id39.39.id15" class="ltx_sup"><span id="id39.39.id15.1" class="ltx_text ltx_font_italic">†</span></sup><sup id="id40.40.id16" class="ltx_sup">1</sup>,
Carlos Guestrin<sup id="id41.41.id17" class="ltx_sup"><span id="id41.41.id17.1" class="ltx_text ltx_font_italic">†</span></sup><sup id="id42.42.id18" class="ltx_sup">1</sup>
<br class="ltx_break">
</span><span class="ltx_author_notes"> Core contributors.
 <sup id="id43.43.id1" class="ltx_sup"><span id="id43.43.id1.1" class="ltx_text ltx_font_italic">†</span></sup> Joint advising.   See author contributions at the end of the paper.

<br class="ltx_break"><span id="id44.44.id2" class="ltx_text">     </span><sup id="id45.45.id3" class="ltx_sup">1</sup> Stanford University.
 <sup id="id46.46.id4" class="ltx_sup">2</sup> UC San Diego.
 <sup id="id47.47.id5" class="ltx_sup">3</sup> UC Berkeley.
 <sup id="id48.48.id6" class="ltx_sup">4</sup> Meta AI.

<br class="ltx_break"><span id="id49.49.id7" class="ltx_text">     </span>
Correspondence to:
<span id="id50.50.id8" class="ltx_text ltx_font_typewriter">yusun@cs.stanford.edu</span>,
<span id="id51.51.id9" class="ltx_text ltx_font_typewriter">xil202@ucsd.edu</span>,
<span id="id52.52.id10" class="ltx_text ltx_font_typewriter">kdalal@berkeley.edu</span>.

<br class="ltx_break"><span id="id53.53.id11" class="ltx_text">     </span>
Code available in <a target="_blank" href="https://github.com/test-time-training/ttt-lm-jax" title="" class="ltx_ref ltx_href">JAX</a> and <a target="_blank" href="https://github.com/test-time-training/ttt-lm-pytorch" title="" class="ltx_ref ltx_href">PyTorch</a>.
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id54.id1" class="ltx_p">Self-attention performs well in long context but has quadratic complexity.
Existing RNN layers have linear complexity, but their performance in long context is limited by the expressive power of their hidden state.
We propose a new class of sequence modeling layers with linear complexity and an expressive hidden state.
The key idea is to make the hidden state a machine learning model itself, and the update rule a step of self-supervised learning.
Since the hidden state is updated by training even on test sequences, our layers are called <em id="id54.id1.1" class="ltx_emph ltx_font_italic">Test-Time Training (TTT) layers</em>.
We consider two instantiations: TTT-Linear and TTT-MLP, whose hidden state is a linear model and a two-layer MLP respectively.
We evaluate our instantiations at the scale of 125M to 1.3B parameters, comparing with a strong Transformer and Mamba, a modern RNN.
Both TTT-Linear and TTT-MLP match or exceed the baselines.
Similar to Transformer, they can
keep reducing perplexity by conditioning on more tokens, while Mamba cannot after 16k context.
With preliminary systems optimization, TTT-Linear is already faster than Transformer at 8k context and matches Mamba in wall-clock time.
TTT-MLP still faces challenges in memory I/O, but shows larger potential in long context, pointing to a promising direction for future research.</p>
</div>
<figure id="S0.F1" class="ltx_figure"><img src="/html/2407.04620/assets/x1.png" id="S0.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="438" height="92" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>All sequence modeling layers can be expressed as a hidden state that transitions according to an update rule.
Our key idea is to make the hidden state itself a model <math id="S0.F1.5.m1.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S0.F1.5.m1.1b"><mi id="S0.F1.5.m1.1.1" xref="S0.F1.5.m1.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S0.F1.5.m1.1c"><ci id="S0.F1.5.m1.1.1.cmml" xref="S0.F1.5.m1.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S0.F1.5.m1.1d">f</annotation></semantics></math> with weights <math id="S0.F1.6.m2.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S0.F1.6.m2.1b"><mi id="S0.F1.6.m2.1.1" xref="S0.F1.6.m2.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S0.F1.6.m2.1c"><ci id="S0.F1.6.m2.1.1.cmml" xref="S0.F1.6.m2.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S0.F1.6.m2.1d">W</annotation></semantics></math>, and the update rule a gradient step on the self-supervised loss <math id="S0.F1.7.m3.1" class="ltx_Math" alttext="\ell" display="inline"><semantics id="S0.F1.7.m3.1b"><mi mathvariant="normal" id="S0.F1.7.m3.1.1" xref="S0.F1.7.m3.1.1.cmml">ℓ</mi><annotation-xml encoding="MathML-Content" id="S0.F1.7.m3.1c"><ci id="S0.F1.7.m3.1.1.cmml" xref="S0.F1.7.m3.1.1">ℓ</ci></annotation-xml><annotation encoding="application/x-tex" id="S0.F1.7.m3.1d">\ell</annotation></semantics></math>.
Therefore, updating the hidden state on a test sequence is equivalent to training the model <math id="S0.F1.8.m4.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S0.F1.8.m4.1b"><mi id="S0.F1.8.m4.1.1" xref="S0.F1.8.m4.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S0.F1.8.m4.1c"><ci id="S0.F1.8.m4.1.1.cmml" xref="S0.F1.8.m4.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S0.F1.8.m4.1d">f</annotation></semantics></math> at test time.
This process, known as test-time training (TTT), is programmed into our TTT layers.
</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
<figure id="S0.F2" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2407.04620/assets/x2.png" id="S0.F2.g1" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="226" height="150" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2407.04620/assets/x3.png" id="S0.F2.g2" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="226" height="150" alt="Refer to caption"></div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>
<span id="S0.F2.2.1" class="ltx_text ltx_font_bold">Comparing to Mamba, TTT-Linear has better perplexity and fewer FLOPs (left), and better use of long context (right).</span>
Evaluations follow Kaplan et al. <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib40" title="" class="ltx_ref">40</a>]</cite>.
Left:
Scaling trends on Books, zoomed in between 350M and 1.3B parameters.
At 760M and 1.3B, TTT-Linear outperforms Mamba in perplexity using fewer FLOPs, and outperforms Transformer under linear interpolation.
Right:
Transformer and TTT-Linear can keep reducing perplexity as it conditions on more tokens, while Mamba cannot after 16k context.
All methods have matched training FLOPs as Mamba 1.4B.
Details in Subsection <a href="#S3.SS2" title="3.2 Long context: Books ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>.
</figcaption>
</figure>
<figure id="S0.F3.fig1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:238.5pt;">
<div id="S0.F3.1" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:186.5pt;">
<img src="/html/2407.04620/assets/x4.png" id="S0.F3.1.g1" class="ltx_graphics ltx_img_landscape" width="438" height="292" alt="Refer to caption">
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Forward time per token (latency) for batch size 16 as context length varies.
All models are 1.3B (1.4B for Mamba).
Forward time per token grows linearly for Transformer as context length increases, but stays roughly constant for the other two methods.
TTT-Linear is faster than Transformer at 8k context and matches Mamba.
Details in Subsection <a href="#S3.SS3" title="3.3 Wall-clock time ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>.
</figcaption>
</figure>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">In 2020, the OpenAI scaling law paper (Kaplan et. al <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib40" title="" class="ltx_ref">40</a>]</cite>) showed that LSTMs (a type of RNN) could not scale similarly to Transformers or effectively use long context.
Now, with modern RNNs and best practices, we re-evaluate these findings in Figure <a href="#S0.F2" title="Figure 2 ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">On the left, we observe that Mamba <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib26" title="" class="ltx_ref">26</a>]</cite>
– one of the most popular RNNs today – scales similarly to a strong Transformer, showing great progress since the LSTMs in 2020.
However, on the right, we observe the same issue with Mamba as Kaplan et al. did with LSTMs.
Tokens later in a sequence should be easier to predict on average, since they condition on more information.
This is indeed the case for Transformer, whose average perplexity at each token index decreases throughout its 32k context.
In contrast, the same metric plateaus for Mamba after 16k.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">This result represents an awkward reality for existing RNNs.
On one hand, the main advantage of RNNs (vs. Transformers) is their linear (vs. quadratic) complexity. This asymptotic advantage is only realized in practice for long context, which according to Figure <a href="#S0.F3.fig1" title="Figure 3 ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> is after 8k.
On the other hand, once context is long enough, existing RNNs such as Mamba struggle to actually take advantage of the extra information being conditioned on.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">The difficulty with long context is inherent to the very nature of RNN layers:
Unlike self-attention, RNN layers have to compress context into a hidden state of fixed size.
As a compression heuristic, the update rule needs to discover the underlying structures and relationships among thousands or potentially millions of tokens.
In this paper, we begin with the observation that self-supervised learning can compress a massive training set into the weights of a model such as an LLM, which often exhibits deep understanding about the semantic connections among its training data – exactly what we need from a compression heuristic.</p>
</div>
<section id="S1.SS0.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">TTT layers.</h5>

<div id="S1.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S1.SS0.SSS0.Px1.p1.1" class="ltx_p">Motivated by this observation, we design a new class of sequence modeling layers where the hidden state is a model, and the update rule is a a step of self-supervised learning.
Because the process of updating the hidden state on a test sequence is equivalent to training a model at test time, this new class of layers is called <em id="S1.SS0.SSS0.Px1.p1.1.1" class="ltx_emph ltx_font_italic">Test-Time Training (TTT) layers</em>.
We introduce two simple instantiations within this class: TTT-Linear and TTT-MLP, where the hidden state is a linear model and a two-layer MLP, respectively.
TTT layers can be integrated into any network architecture and optimized end-to-end, similar to RNNs layers and self-attention.</p>
</div>
</section>
<section id="S1.SS0.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Wall-clock time.</h5>

<div id="S1.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S1.SS0.SSS0.Px2.p1.1" class="ltx_p">While the TTT layer is already efficient in FLOPs, we propose two practical innovations to make it efficient in wall-clock time. First, similar to the standard practice of taking gradient steps on mini-batches of sequences during regular training for better parallelism, we use mini-batches of tokens during TTT.
Second, we develop a dual form for operations inside each TTT mini-batch, to better take advantage of modern GPUs and TPUs.
The dual form is equivalent in output to the naive implementation, but trains more than <math id="S1.SS0.SSS0.Px2.p1.1.m1.1" class="ltx_math_unparsed" alttext="5\times" display="inline"><semantics id="S1.SS0.SSS0.Px2.p1.1.m1.1a"><mrow id="S1.SS0.SSS0.Px2.p1.1.m1.1b"><mn id="S1.SS0.SSS0.Px2.p1.1.m1.1.1">5</mn><mo lspace="0.222em" id="S1.SS0.SSS0.Px2.p1.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S1.SS0.SSS0.Px2.p1.1.m1.1c">5\times</annotation></semantics></math> faster. As shown in Figure <a href="#S0.F3.fig1" title="Figure 3 ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, TTT-Linear is faster than Transformer at 8k context and matches Mamba.</p>
</div>
</section>
<section id="S1.SS0.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Evaluations and open problems.</h5>

<div id="S1.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="S1.SS0.SSS0.Px3.p1.1" class="ltx_p">While we have highlighted some results for TTT-Linear at the beginning of the paper, Section <a href="#S3" title="3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> presents more comprehensive evaluations for both TTT-Linear and TTT-MLP, and open problems exposed by our evaluations.
For example, our evaluations following the Chinchilla recipe <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib34" title="" class="ltx_ref">34</a>]</cite> do not cleanly fit a linear scaling trend even for the Transformer baseline.
Constrained by our academic resources, we encourage the community to join us in exploring solutions to these problems.</p>
</div>
</section>
<section id="S1.SS0.SSS0.Px4" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Summary of contributions.</h5>

<div id="S1.SS0.SSS0.Px4.p1" class="ltx_para">
<ol id="S1.I1" class="ltx_enumerate">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p">We propose TTT layers, a new class of sequence modeling layers where the hidden state is a model, and the update rule is self-supervised learning.
Our perspective that the forward pass of a layer contains a training loop itself opens up a new direction for future research.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p">TTT-Linear, one simple instantiation of TTT layers, outperforms Transformers and Mamba in our evaluations ranging from 125M to 1.3B parameters.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.1" class="ltx_p">We improve the hardware efficiency of TTT layers through mini-batch TTT and the dual form, making TTT-Linear already a practical building block for LLMs.</p>
</div>
</li>
</ol>
</div>
<figure id="S1.F4" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="S1.F4.1" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:433.6pt;">
<img src="/html/2407.04620/assets/x5.png" id="S1.F4.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="93" alt="Refer to caption">
</div>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<table id="S1.F4.13" class="ltx_tabular ltx_centering ltx_figure_panel ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S1.F4.13.13.1" class="ltx_tr">
<th id="S1.F4.13.13.1.1" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_l ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;"></th>
<th id="S1.F4.13.13.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;"><span id="S1.F4.13.13.1.2.1" class="ltx_text ltx_font_bold">Initial state</span></th>
<th id="S1.F4.13.13.1.3" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;"><span id="S1.F4.13.13.1.3.1" class="ltx_text ltx_font_bold">Update rule</span></th>
<th id="S1.F4.13.13.1.4" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;"><span id="S1.F4.13.13.1.4.1" class="ltx_text ltx_font_bold">Output rule</span></th>
<th id="S1.F4.13.13.1.5" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;"><span id="S1.F4.13.13.1.5.1" class="ltx_text ltx_font_bold">Cost</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S1.F4.5.4" class="ltx_tr">
<th id="S1.F4.5.4.5" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;"><span id="S1.F4.5.4.5.1" class="ltx_text ltx_font_bold">Naive RNN</span></th>
<td id="S1.F4.2.1.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;"><math id="S1.F4.2.1.1.m1.1" class="ltx_Math" alttext="s_{0}=\texttt{vector()}" display="inline"><semantics id="S1.F4.2.1.1.m1.1a"><mrow id="S1.F4.2.1.1.m1.1.1" xref="S1.F4.2.1.1.m1.1.1.cmml"><msub id="S1.F4.2.1.1.m1.1.1.2" xref="S1.F4.2.1.1.m1.1.1.2.cmml"><mi id="S1.F4.2.1.1.m1.1.1.2.2" xref="S1.F4.2.1.1.m1.1.1.2.2.cmml">s</mi><mn id="S1.F4.2.1.1.m1.1.1.2.3" xref="S1.F4.2.1.1.m1.1.1.2.3.cmml">0</mn></msub><mo id="S1.F4.2.1.1.m1.1.1.1" xref="S1.F4.2.1.1.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" id="S1.F4.2.1.1.m1.1.1.3" xref="S1.F4.2.1.1.m1.1.1.3a.cmml">vector()</mtext></mrow><annotation-xml encoding="MathML-Content" id="S1.F4.2.1.1.m1.1b"><apply id="S1.F4.2.1.1.m1.1.1.cmml" xref="S1.F4.2.1.1.m1.1.1"><eq id="S1.F4.2.1.1.m1.1.1.1.cmml" xref="S1.F4.2.1.1.m1.1.1.1"></eq><apply id="S1.F4.2.1.1.m1.1.1.2.cmml" xref="S1.F4.2.1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S1.F4.2.1.1.m1.1.1.2.1.cmml" xref="S1.F4.2.1.1.m1.1.1.2">subscript</csymbol><ci id="S1.F4.2.1.1.m1.1.1.2.2.cmml" xref="S1.F4.2.1.1.m1.1.1.2.2">𝑠</ci><cn type="integer" id="S1.F4.2.1.1.m1.1.1.2.3.cmml" xref="S1.F4.2.1.1.m1.1.1.2.3">0</cn></apply><ci id="S1.F4.2.1.1.m1.1.1.3a.cmml" xref="S1.F4.2.1.1.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" id="S1.F4.2.1.1.m1.1.1.3.cmml" xref="S1.F4.2.1.1.m1.1.1.3">vector()</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F4.2.1.1.m1.1c">s_{0}=\texttt{vector()}</annotation></semantics></math></td>
<td id="S1.F4.3.2.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;"><math id="S1.F4.3.2.2.m1.1" class="ltx_Math" alttext="s_{t}=\sigma\left(\theta_{ss}s_{t-1}+\theta_{sx}x_{t}\right)" display="inline"><semantics id="S1.F4.3.2.2.m1.1a"><mrow id="S1.F4.3.2.2.m1.1.1" xref="S1.F4.3.2.2.m1.1.1.cmml"><msub id="S1.F4.3.2.2.m1.1.1.3" xref="S1.F4.3.2.2.m1.1.1.3.cmml"><mi id="S1.F4.3.2.2.m1.1.1.3.2" xref="S1.F4.3.2.2.m1.1.1.3.2.cmml">s</mi><mi id="S1.F4.3.2.2.m1.1.1.3.3" xref="S1.F4.3.2.2.m1.1.1.3.3.cmml">t</mi></msub><mo id="S1.F4.3.2.2.m1.1.1.2" xref="S1.F4.3.2.2.m1.1.1.2.cmml">=</mo><mrow id="S1.F4.3.2.2.m1.1.1.1" xref="S1.F4.3.2.2.m1.1.1.1.cmml"><mi id="S1.F4.3.2.2.m1.1.1.1.3" xref="S1.F4.3.2.2.m1.1.1.1.3.cmml">σ</mi><mo lspace="0em" rspace="0em" id="S1.F4.3.2.2.m1.1.1.1.2" xref="S1.F4.3.2.2.m1.1.1.1.2.cmml">​</mo><mrow id="S1.F4.3.2.2.m1.1.1.1.1.1" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.cmml"><mo id="S1.F4.3.2.2.m1.1.1.1.1.1.2" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S1.F4.3.2.2.m1.1.1.1.1.1.1" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.cmml"><mrow id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.cmml"><msub id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.cmml"><mi id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.2" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.2.cmml">θ</mi><mrow id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.3" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.3.cmml"><mi id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.3.2" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.3.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.3.1" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.3.1.cmml">​</mo><mi id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.3.3" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.3.3.cmml">s</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.1" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.1.cmml">​</mo><msub id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.cmml"><mi id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.2" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.2.cmml">s</mi><mrow id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.3" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.3.cmml"><mi id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.3.2" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.3.2.cmml">t</mi><mo id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.3.1" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.3.1.cmml">−</mo><mn id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.3.3" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.3.3.cmml">1</mn></mrow></msub></mrow><mo id="S1.F4.3.2.2.m1.1.1.1.1.1.1.1" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.cmml"><msub id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.cmml"><mi id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.2" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.2.cmml">θ</mi><mrow id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.3" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.3.cmml"><mi id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.3.2" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.3.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.3.1" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.3.1.cmml">​</mo><mi id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.3.3" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.3.3.cmml">x</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.1" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.3" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.3.cmml"><mi id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.3.2" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.3.2.cmml">x</mi><mi id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.3.3" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.3.3.cmml">t</mi></msub></mrow></mrow><mo id="S1.F4.3.2.2.m1.1.1.1.1.1.3" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.F4.3.2.2.m1.1b"><apply id="S1.F4.3.2.2.m1.1.1.cmml" xref="S1.F4.3.2.2.m1.1.1"><eq id="S1.F4.3.2.2.m1.1.1.2.cmml" xref="S1.F4.3.2.2.m1.1.1.2"></eq><apply id="S1.F4.3.2.2.m1.1.1.3.cmml" xref="S1.F4.3.2.2.m1.1.1.3"><csymbol cd="ambiguous" id="S1.F4.3.2.2.m1.1.1.3.1.cmml" xref="S1.F4.3.2.2.m1.1.1.3">subscript</csymbol><ci id="S1.F4.3.2.2.m1.1.1.3.2.cmml" xref="S1.F4.3.2.2.m1.1.1.3.2">𝑠</ci><ci id="S1.F4.3.2.2.m1.1.1.3.3.cmml" xref="S1.F4.3.2.2.m1.1.1.3.3">𝑡</ci></apply><apply id="S1.F4.3.2.2.m1.1.1.1.cmml" xref="S1.F4.3.2.2.m1.1.1.1"><times id="S1.F4.3.2.2.m1.1.1.1.2.cmml" xref="S1.F4.3.2.2.m1.1.1.1.2"></times><ci id="S1.F4.3.2.2.m1.1.1.1.3.cmml" xref="S1.F4.3.2.2.m1.1.1.1.3">𝜎</ci><apply id="S1.F4.3.2.2.m1.1.1.1.1.1.1.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1"><plus id="S1.F4.3.2.2.m1.1.1.1.1.1.1.1.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.1"></plus><apply id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2"><times id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.1.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.1"></times><apply id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.1.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2">subscript</csymbol><ci id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.2.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.2">𝜃</ci><apply id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.3.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.3"><times id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.3.1.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.3.1"></times><ci id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.3.2.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.3.2">𝑠</ci><ci id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.3.3.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.2.3.3">𝑠</ci></apply></apply><apply id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.1.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3">subscript</csymbol><ci id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.2.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.2">𝑠</ci><apply id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.3.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.3"><minus id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.3.1.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.3.1"></minus><ci id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.3.2.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.3.2">𝑡</ci><cn type="integer" id="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.3.3.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.2.3.3.3">1</cn></apply></apply></apply><apply id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3"><times id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.1.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.1"></times><apply id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.1.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2">subscript</csymbol><ci id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.2.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.2">𝜃</ci><apply id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.3.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.3"><times id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.3.1.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.3.1"></times><ci id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.3.2.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.3.2">𝑠</ci><ci id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.3.3.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.2.3.3">𝑥</ci></apply></apply><apply id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.3.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.3.1.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.3">subscript</csymbol><ci id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.3.2.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.3.2">𝑥</ci><ci id="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.3.3.cmml" xref="S1.F4.3.2.2.m1.1.1.1.1.1.1.3.3.3">𝑡</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F4.3.2.2.m1.1c">s_{t}=\sigma\left(\theta_{ss}s_{t-1}+\theta_{sx}x_{t}\right)</annotation></semantics></math></td>
<td id="S1.F4.4.3.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;"><math id="S1.F4.4.3.3.m1.1" class="ltx_Math" alttext="z_{t}=\theta_{zs}s_{t}+\theta_{zx}x_{t}" display="inline"><semantics id="S1.F4.4.3.3.m1.1a"><mrow id="S1.F4.4.3.3.m1.1.1" xref="S1.F4.4.3.3.m1.1.1.cmml"><msub id="S1.F4.4.3.3.m1.1.1.2" xref="S1.F4.4.3.3.m1.1.1.2.cmml"><mi id="S1.F4.4.3.3.m1.1.1.2.2" xref="S1.F4.4.3.3.m1.1.1.2.2.cmml">z</mi><mi id="S1.F4.4.3.3.m1.1.1.2.3" xref="S1.F4.4.3.3.m1.1.1.2.3.cmml">t</mi></msub><mo id="S1.F4.4.3.3.m1.1.1.1" xref="S1.F4.4.3.3.m1.1.1.1.cmml">=</mo><mrow id="S1.F4.4.3.3.m1.1.1.3" xref="S1.F4.4.3.3.m1.1.1.3.cmml"><mrow id="S1.F4.4.3.3.m1.1.1.3.2" xref="S1.F4.4.3.3.m1.1.1.3.2.cmml"><msub id="S1.F4.4.3.3.m1.1.1.3.2.2" xref="S1.F4.4.3.3.m1.1.1.3.2.2.cmml"><mi id="S1.F4.4.3.3.m1.1.1.3.2.2.2" xref="S1.F4.4.3.3.m1.1.1.3.2.2.2.cmml">θ</mi><mrow id="S1.F4.4.3.3.m1.1.1.3.2.2.3" xref="S1.F4.4.3.3.m1.1.1.3.2.2.3.cmml"><mi id="S1.F4.4.3.3.m1.1.1.3.2.2.3.2" xref="S1.F4.4.3.3.m1.1.1.3.2.2.3.2.cmml">z</mi><mo lspace="0em" rspace="0em" id="S1.F4.4.3.3.m1.1.1.3.2.2.3.1" xref="S1.F4.4.3.3.m1.1.1.3.2.2.3.1.cmml">​</mo><mi id="S1.F4.4.3.3.m1.1.1.3.2.2.3.3" xref="S1.F4.4.3.3.m1.1.1.3.2.2.3.3.cmml">s</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S1.F4.4.3.3.m1.1.1.3.2.1" xref="S1.F4.4.3.3.m1.1.1.3.2.1.cmml">​</mo><msub id="S1.F4.4.3.3.m1.1.1.3.2.3" xref="S1.F4.4.3.3.m1.1.1.3.2.3.cmml"><mi id="S1.F4.4.3.3.m1.1.1.3.2.3.2" xref="S1.F4.4.3.3.m1.1.1.3.2.3.2.cmml">s</mi><mi id="S1.F4.4.3.3.m1.1.1.3.2.3.3" xref="S1.F4.4.3.3.m1.1.1.3.2.3.3.cmml">t</mi></msub></mrow><mo id="S1.F4.4.3.3.m1.1.1.3.1" xref="S1.F4.4.3.3.m1.1.1.3.1.cmml">+</mo><mrow id="S1.F4.4.3.3.m1.1.1.3.3" xref="S1.F4.4.3.3.m1.1.1.3.3.cmml"><msub id="S1.F4.4.3.3.m1.1.1.3.3.2" xref="S1.F4.4.3.3.m1.1.1.3.3.2.cmml"><mi id="S1.F4.4.3.3.m1.1.1.3.3.2.2" xref="S1.F4.4.3.3.m1.1.1.3.3.2.2.cmml">θ</mi><mrow id="S1.F4.4.3.3.m1.1.1.3.3.2.3" xref="S1.F4.4.3.3.m1.1.1.3.3.2.3.cmml"><mi id="S1.F4.4.3.3.m1.1.1.3.3.2.3.2" xref="S1.F4.4.3.3.m1.1.1.3.3.2.3.2.cmml">z</mi><mo lspace="0em" rspace="0em" id="S1.F4.4.3.3.m1.1.1.3.3.2.3.1" xref="S1.F4.4.3.3.m1.1.1.3.3.2.3.1.cmml">​</mo><mi id="S1.F4.4.3.3.m1.1.1.3.3.2.3.3" xref="S1.F4.4.3.3.m1.1.1.3.3.2.3.3.cmml">x</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S1.F4.4.3.3.m1.1.1.3.3.1" xref="S1.F4.4.3.3.m1.1.1.3.3.1.cmml">​</mo><msub id="S1.F4.4.3.3.m1.1.1.3.3.3" xref="S1.F4.4.3.3.m1.1.1.3.3.3.cmml"><mi id="S1.F4.4.3.3.m1.1.1.3.3.3.2" xref="S1.F4.4.3.3.m1.1.1.3.3.3.2.cmml">x</mi><mi id="S1.F4.4.3.3.m1.1.1.3.3.3.3" xref="S1.F4.4.3.3.m1.1.1.3.3.3.3.cmml">t</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.F4.4.3.3.m1.1b"><apply id="S1.F4.4.3.3.m1.1.1.cmml" xref="S1.F4.4.3.3.m1.1.1"><eq id="S1.F4.4.3.3.m1.1.1.1.cmml" xref="S1.F4.4.3.3.m1.1.1.1"></eq><apply id="S1.F4.4.3.3.m1.1.1.2.cmml" xref="S1.F4.4.3.3.m1.1.1.2"><csymbol cd="ambiguous" id="S1.F4.4.3.3.m1.1.1.2.1.cmml" xref="S1.F4.4.3.3.m1.1.1.2">subscript</csymbol><ci id="S1.F4.4.3.3.m1.1.1.2.2.cmml" xref="S1.F4.4.3.3.m1.1.1.2.2">𝑧</ci><ci id="S1.F4.4.3.3.m1.1.1.2.3.cmml" xref="S1.F4.4.3.3.m1.1.1.2.3">𝑡</ci></apply><apply id="S1.F4.4.3.3.m1.1.1.3.cmml" xref="S1.F4.4.3.3.m1.1.1.3"><plus id="S1.F4.4.3.3.m1.1.1.3.1.cmml" xref="S1.F4.4.3.3.m1.1.1.3.1"></plus><apply id="S1.F4.4.3.3.m1.1.1.3.2.cmml" xref="S1.F4.4.3.3.m1.1.1.3.2"><times id="S1.F4.4.3.3.m1.1.1.3.2.1.cmml" xref="S1.F4.4.3.3.m1.1.1.3.2.1"></times><apply id="S1.F4.4.3.3.m1.1.1.3.2.2.cmml" xref="S1.F4.4.3.3.m1.1.1.3.2.2"><csymbol cd="ambiguous" id="S1.F4.4.3.3.m1.1.1.3.2.2.1.cmml" xref="S1.F4.4.3.3.m1.1.1.3.2.2">subscript</csymbol><ci id="S1.F4.4.3.3.m1.1.1.3.2.2.2.cmml" xref="S1.F4.4.3.3.m1.1.1.3.2.2.2">𝜃</ci><apply id="S1.F4.4.3.3.m1.1.1.3.2.2.3.cmml" xref="S1.F4.4.3.3.m1.1.1.3.2.2.3"><times id="S1.F4.4.3.3.m1.1.1.3.2.2.3.1.cmml" xref="S1.F4.4.3.3.m1.1.1.3.2.2.3.1"></times><ci id="S1.F4.4.3.3.m1.1.1.3.2.2.3.2.cmml" xref="S1.F4.4.3.3.m1.1.1.3.2.2.3.2">𝑧</ci><ci id="S1.F4.4.3.3.m1.1.1.3.2.2.3.3.cmml" xref="S1.F4.4.3.3.m1.1.1.3.2.2.3.3">𝑠</ci></apply></apply><apply id="S1.F4.4.3.3.m1.1.1.3.2.3.cmml" xref="S1.F4.4.3.3.m1.1.1.3.2.3"><csymbol cd="ambiguous" id="S1.F4.4.3.3.m1.1.1.3.2.3.1.cmml" xref="S1.F4.4.3.3.m1.1.1.3.2.3">subscript</csymbol><ci id="S1.F4.4.3.3.m1.1.1.3.2.3.2.cmml" xref="S1.F4.4.3.3.m1.1.1.3.2.3.2">𝑠</ci><ci id="S1.F4.4.3.3.m1.1.1.3.2.3.3.cmml" xref="S1.F4.4.3.3.m1.1.1.3.2.3.3">𝑡</ci></apply></apply><apply id="S1.F4.4.3.3.m1.1.1.3.3.cmml" xref="S1.F4.4.3.3.m1.1.1.3.3"><times id="S1.F4.4.3.3.m1.1.1.3.3.1.cmml" xref="S1.F4.4.3.3.m1.1.1.3.3.1"></times><apply id="S1.F4.4.3.3.m1.1.1.3.3.2.cmml" xref="S1.F4.4.3.3.m1.1.1.3.3.2"><csymbol cd="ambiguous" id="S1.F4.4.3.3.m1.1.1.3.3.2.1.cmml" xref="S1.F4.4.3.3.m1.1.1.3.3.2">subscript</csymbol><ci id="S1.F4.4.3.3.m1.1.1.3.3.2.2.cmml" xref="S1.F4.4.3.3.m1.1.1.3.3.2.2">𝜃</ci><apply id="S1.F4.4.3.3.m1.1.1.3.3.2.3.cmml" xref="S1.F4.4.3.3.m1.1.1.3.3.2.3"><times id="S1.F4.4.3.3.m1.1.1.3.3.2.3.1.cmml" xref="S1.F4.4.3.3.m1.1.1.3.3.2.3.1"></times><ci id="S1.F4.4.3.3.m1.1.1.3.3.2.3.2.cmml" xref="S1.F4.4.3.3.m1.1.1.3.3.2.3.2">𝑧</ci><ci id="S1.F4.4.3.3.m1.1.1.3.3.2.3.3.cmml" xref="S1.F4.4.3.3.m1.1.1.3.3.2.3.3">𝑥</ci></apply></apply><apply id="S1.F4.4.3.3.m1.1.1.3.3.3.cmml" xref="S1.F4.4.3.3.m1.1.1.3.3.3"><csymbol cd="ambiguous" id="S1.F4.4.3.3.m1.1.1.3.3.3.1.cmml" xref="S1.F4.4.3.3.m1.1.1.3.3.3">subscript</csymbol><ci id="S1.F4.4.3.3.m1.1.1.3.3.3.2.cmml" xref="S1.F4.4.3.3.m1.1.1.3.3.3.2">𝑥</ci><ci id="S1.F4.4.3.3.m1.1.1.3.3.3.3.cmml" xref="S1.F4.4.3.3.m1.1.1.3.3.3.3">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F4.4.3.3.m1.1c">z_{t}=\theta_{zs}s_{t}+\theta_{zx}x_{t}</annotation></semantics></math></td>
<td id="S1.F4.5.4.4" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;"><math id="S1.F4.5.4.4.m1.1" class="ltx_Math" alttext="O(1)" display="inline"><semantics id="S1.F4.5.4.4.m1.1a"><mrow id="S1.F4.5.4.4.m1.1.2" xref="S1.F4.5.4.4.m1.1.2.cmml"><mi id="S1.F4.5.4.4.m1.1.2.2" xref="S1.F4.5.4.4.m1.1.2.2.cmml">O</mi><mo lspace="0em" rspace="0em" id="S1.F4.5.4.4.m1.1.2.1" xref="S1.F4.5.4.4.m1.1.2.1.cmml">​</mo><mrow id="S1.F4.5.4.4.m1.1.2.3.2" xref="S1.F4.5.4.4.m1.1.2.cmml"><mo stretchy="false" id="S1.F4.5.4.4.m1.1.2.3.2.1" xref="S1.F4.5.4.4.m1.1.2.cmml">(</mo><mn id="S1.F4.5.4.4.m1.1.1" xref="S1.F4.5.4.4.m1.1.1.cmml">1</mn><mo stretchy="false" id="S1.F4.5.4.4.m1.1.2.3.2.2" xref="S1.F4.5.4.4.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.F4.5.4.4.m1.1b"><apply id="S1.F4.5.4.4.m1.1.2.cmml" xref="S1.F4.5.4.4.m1.1.2"><times id="S1.F4.5.4.4.m1.1.2.1.cmml" xref="S1.F4.5.4.4.m1.1.2.1"></times><ci id="S1.F4.5.4.4.m1.1.2.2.cmml" xref="S1.F4.5.4.4.m1.1.2.2">𝑂</ci><cn type="integer" id="S1.F4.5.4.4.m1.1.1.cmml" xref="S1.F4.5.4.4.m1.1.1">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F4.5.4.4.m1.1c">O(1)</annotation></semantics></math></td>
</tr>
<tr id="S1.F4.9.8" class="ltx_tr">
<th id="S1.F4.9.8.5" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r" style="padding-top:4pt;padding-bottom:4pt;"><span id="S1.F4.9.8.5.1" class="ltx_text ltx_font_bold">Self-attention</span></th>
<td id="S1.F4.6.5.1" class="ltx_td ltx_align_left ltx_border_r" style="padding-top:4pt;padding-bottom:4pt;"><math id="S1.F4.6.5.1.m1.1" class="ltx_Math" alttext="s_{0}=\texttt{list()}" display="inline"><semantics id="S1.F4.6.5.1.m1.1a"><mrow id="S1.F4.6.5.1.m1.1.1" xref="S1.F4.6.5.1.m1.1.1.cmml"><msub id="S1.F4.6.5.1.m1.1.1.2" xref="S1.F4.6.5.1.m1.1.1.2.cmml"><mi id="S1.F4.6.5.1.m1.1.1.2.2" xref="S1.F4.6.5.1.m1.1.1.2.2.cmml">s</mi><mn id="S1.F4.6.5.1.m1.1.1.2.3" xref="S1.F4.6.5.1.m1.1.1.2.3.cmml">0</mn></msub><mo id="S1.F4.6.5.1.m1.1.1.1" xref="S1.F4.6.5.1.m1.1.1.1.cmml">=</mo><mtext class="ltx_mathvariant_monospace" id="S1.F4.6.5.1.m1.1.1.3" xref="S1.F4.6.5.1.m1.1.1.3a.cmml">list()</mtext></mrow><annotation-xml encoding="MathML-Content" id="S1.F4.6.5.1.m1.1b"><apply id="S1.F4.6.5.1.m1.1.1.cmml" xref="S1.F4.6.5.1.m1.1.1"><eq id="S1.F4.6.5.1.m1.1.1.1.cmml" xref="S1.F4.6.5.1.m1.1.1.1"></eq><apply id="S1.F4.6.5.1.m1.1.1.2.cmml" xref="S1.F4.6.5.1.m1.1.1.2"><csymbol cd="ambiguous" id="S1.F4.6.5.1.m1.1.1.2.1.cmml" xref="S1.F4.6.5.1.m1.1.1.2">subscript</csymbol><ci id="S1.F4.6.5.1.m1.1.1.2.2.cmml" xref="S1.F4.6.5.1.m1.1.1.2.2">𝑠</ci><cn type="integer" id="S1.F4.6.5.1.m1.1.1.2.3.cmml" xref="S1.F4.6.5.1.m1.1.1.2.3">0</cn></apply><ci id="S1.F4.6.5.1.m1.1.1.3a.cmml" xref="S1.F4.6.5.1.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" id="S1.F4.6.5.1.m1.1.1.3.cmml" xref="S1.F4.6.5.1.m1.1.1.3">list()</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F4.6.5.1.m1.1c">s_{0}=\texttt{list()}</annotation></semantics></math></td>
<td id="S1.F4.7.6.2" class="ltx_td ltx_align_left ltx_border_r" style="padding-top:4pt;padding-bottom:4pt;"><math id="S1.F4.7.6.2.m1.2" class="ltx_Math" alttext="s_{t}=s_{t-1}.\texttt{append}(k_{t},v_{t})" display="inline"><semantics id="S1.F4.7.6.2.m1.2a"><mrow id="S1.F4.7.6.2.m1.2.2.2" xref="S1.F4.7.6.2.m1.2.2.3.cmml"><mrow id="S1.F4.7.6.2.m1.1.1.1.1" xref="S1.F4.7.6.2.m1.1.1.1.1.cmml"><msub id="S1.F4.7.6.2.m1.1.1.1.1.2" xref="S1.F4.7.6.2.m1.1.1.1.1.2.cmml"><mi id="S1.F4.7.6.2.m1.1.1.1.1.2.2" xref="S1.F4.7.6.2.m1.1.1.1.1.2.2.cmml">s</mi><mi id="S1.F4.7.6.2.m1.1.1.1.1.2.3" xref="S1.F4.7.6.2.m1.1.1.1.1.2.3.cmml">t</mi></msub><mo id="S1.F4.7.6.2.m1.1.1.1.1.1" xref="S1.F4.7.6.2.m1.1.1.1.1.1.cmml">=</mo><msub id="S1.F4.7.6.2.m1.1.1.1.1.3" xref="S1.F4.7.6.2.m1.1.1.1.1.3.cmml"><mi id="S1.F4.7.6.2.m1.1.1.1.1.3.2" xref="S1.F4.7.6.2.m1.1.1.1.1.3.2.cmml">s</mi><mrow id="S1.F4.7.6.2.m1.1.1.1.1.3.3" xref="S1.F4.7.6.2.m1.1.1.1.1.3.3.cmml"><mi id="S1.F4.7.6.2.m1.1.1.1.1.3.3.2" xref="S1.F4.7.6.2.m1.1.1.1.1.3.3.2.cmml">t</mi><mo id="S1.F4.7.6.2.m1.1.1.1.1.3.3.1" xref="S1.F4.7.6.2.m1.1.1.1.1.3.3.1.cmml">−</mo><mn id="S1.F4.7.6.2.m1.1.1.1.1.3.3.3" xref="S1.F4.7.6.2.m1.1.1.1.1.3.3.3.cmml">1</mn></mrow></msub></mrow><mo lspace="0em" rspace="0.167em" id="S1.F4.7.6.2.m1.2.2.2.3" xref="S1.F4.7.6.2.m1.2.2.3a.cmml">.</mo><mrow id="S1.F4.7.6.2.m1.2.2.2.2" xref="S1.F4.7.6.2.m1.2.2.2.2.cmml"><mtext class="ltx_mathvariant_monospace" id="S1.F4.7.6.2.m1.2.2.2.2.4" xref="S1.F4.7.6.2.m1.2.2.2.2.4a.cmml">append</mtext><mo lspace="0em" rspace="0em" id="S1.F4.7.6.2.m1.2.2.2.2.3" xref="S1.F4.7.6.2.m1.2.2.2.2.3.cmml">​</mo><mrow id="S1.F4.7.6.2.m1.2.2.2.2.2.2" xref="S1.F4.7.6.2.m1.2.2.2.2.2.3.cmml"><mo stretchy="false" id="S1.F4.7.6.2.m1.2.2.2.2.2.2.3" xref="S1.F4.7.6.2.m1.2.2.2.2.2.3.cmml">(</mo><msub id="S1.F4.7.6.2.m1.2.2.2.2.1.1.1" xref="S1.F4.7.6.2.m1.2.2.2.2.1.1.1.cmml"><mi id="S1.F4.7.6.2.m1.2.2.2.2.1.1.1.2" xref="S1.F4.7.6.2.m1.2.2.2.2.1.1.1.2.cmml">k</mi><mi id="S1.F4.7.6.2.m1.2.2.2.2.1.1.1.3" xref="S1.F4.7.6.2.m1.2.2.2.2.1.1.1.3.cmml">t</mi></msub><mo id="S1.F4.7.6.2.m1.2.2.2.2.2.2.4" xref="S1.F4.7.6.2.m1.2.2.2.2.2.3.cmml">,</mo><msub id="S1.F4.7.6.2.m1.2.2.2.2.2.2.2" xref="S1.F4.7.6.2.m1.2.2.2.2.2.2.2.cmml"><mi id="S1.F4.7.6.2.m1.2.2.2.2.2.2.2.2" xref="S1.F4.7.6.2.m1.2.2.2.2.2.2.2.2.cmml">v</mi><mi id="S1.F4.7.6.2.m1.2.2.2.2.2.2.2.3" xref="S1.F4.7.6.2.m1.2.2.2.2.2.2.2.3.cmml">t</mi></msub><mo stretchy="false" id="S1.F4.7.6.2.m1.2.2.2.2.2.2.5" xref="S1.F4.7.6.2.m1.2.2.2.2.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.F4.7.6.2.m1.2b"><apply id="S1.F4.7.6.2.m1.2.2.3.cmml" xref="S1.F4.7.6.2.m1.2.2.2"><csymbol cd="ambiguous" id="S1.F4.7.6.2.m1.2.2.3a.cmml" xref="S1.F4.7.6.2.m1.2.2.2.3">formulae-sequence</csymbol><apply id="S1.F4.7.6.2.m1.1.1.1.1.cmml" xref="S1.F4.7.6.2.m1.1.1.1.1"><eq id="S1.F4.7.6.2.m1.1.1.1.1.1.cmml" xref="S1.F4.7.6.2.m1.1.1.1.1.1"></eq><apply id="S1.F4.7.6.2.m1.1.1.1.1.2.cmml" xref="S1.F4.7.6.2.m1.1.1.1.1.2"><csymbol cd="ambiguous" id="S1.F4.7.6.2.m1.1.1.1.1.2.1.cmml" xref="S1.F4.7.6.2.m1.1.1.1.1.2">subscript</csymbol><ci id="S1.F4.7.6.2.m1.1.1.1.1.2.2.cmml" xref="S1.F4.7.6.2.m1.1.1.1.1.2.2">𝑠</ci><ci id="S1.F4.7.6.2.m1.1.1.1.1.2.3.cmml" xref="S1.F4.7.6.2.m1.1.1.1.1.2.3">𝑡</ci></apply><apply id="S1.F4.7.6.2.m1.1.1.1.1.3.cmml" xref="S1.F4.7.6.2.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="S1.F4.7.6.2.m1.1.1.1.1.3.1.cmml" xref="S1.F4.7.6.2.m1.1.1.1.1.3">subscript</csymbol><ci id="S1.F4.7.6.2.m1.1.1.1.1.3.2.cmml" xref="S1.F4.7.6.2.m1.1.1.1.1.3.2">𝑠</ci><apply id="S1.F4.7.6.2.m1.1.1.1.1.3.3.cmml" xref="S1.F4.7.6.2.m1.1.1.1.1.3.3"><minus id="S1.F4.7.6.2.m1.1.1.1.1.3.3.1.cmml" xref="S1.F4.7.6.2.m1.1.1.1.1.3.3.1"></minus><ci id="S1.F4.7.6.2.m1.1.1.1.1.3.3.2.cmml" xref="S1.F4.7.6.2.m1.1.1.1.1.3.3.2">𝑡</ci><cn type="integer" id="S1.F4.7.6.2.m1.1.1.1.1.3.3.3.cmml" xref="S1.F4.7.6.2.m1.1.1.1.1.3.3.3">1</cn></apply></apply></apply><apply id="S1.F4.7.6.2.m1.2.2.2.2.cmml" xref="S1.F4.7.6.2.m1.2.2.2.2"><times id="S1.F4.7.6.2.m1.2.2.2.2.3.cmml" xref="S1.F4.7.6.2.m1.2.2.2.2.3"></times><ci id="S1.F4.7.6.2.m1.2.2.2.2.4a.cmml" xref="S1.F4.7.6.2.m1.2.2.2.2.4"><mtext class="ltx_mathvariant_monospace" id="S1.F4.7.6.2.m1.2.2.2.2.4.cmml" xref="S1.F4.7.6.2.m1.2.2.2.2.4">append</mtext></ci><interval closure="open" id="S1.F4.7.6.2.m1.2.2.2.2.2.3.cmml" xref="S1.F4.7.6.2.m1.2.2.2.2.2.2"><apply id="S1.F4.7.6.2.m1.2.2.2.2.1.1.1.cmml" xref="S1.F4.7.6.2.m1.2.2.2.2.1.1.1"><csymbol cd="ambiguous" id="S1.F4.7.6.2.m1.2.2.2.2.1.1.1.1.cmml" xref="S1.F4.7.6.2.m1.2.2.2.2.1.1.1">subscript</csymbol><ci id="S1.F4.7.6.2.m1.2.2.2.2.1.1.1.2.cmml" xref="S1.F4.7.6.2.m1.2.2.2.2.1.1.1.2">𝑘</ci><ci id="S1.F4.7.6.2.m1.2.2.2.2.1.1.1.3.cmml" xref="S1.F4.7.6.2.m1.2.2.2.2.1.1.1.3">𝑡</ci></apply><apply id="S1.F4.7.6.2.m1.2.2.2.2.2.2.2.cmml" xref="S1.F4.7.6.2.m1.2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S1.F4.7.6.2.m1.2.2.2.2.2.2.2.1.cmml" xref="S1.F4.7.6.2.m1.2.2.2.2.2.2.2">subscript</csymbol><ci id="S1.F4.7.6.2.m1.2.2.2.2.2.2.2.2.cmml" xref="S1.F4.7.6.2.m1.2.2.2.2.2.2.2.2">𝑣</ci><ci id="S1.F4.7.6.2.m1.2.2.2.2.2.2.2.3.cmml" xref="S1.F4.7.6.2.m1.2.2.2.2.2.2.2.3">𝑡</ci></apply></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F4.7.6.2.m1.2c">s_{t}=s_{t-1}.\texttt{append}(k_{t},v_{t})</annotation></semantics></math></td>
<td id="S1.F4.8.7.3" class="ltx_td ltx_align_left ltx_border_r" style="padding-top:4pt;padding-bottom:4pt;"><math id="S1.F4.8.7.3.m1.1" class="ltx_Math" alttext="z_{t}=V_{t}\texttt{softmax}\left(K^{T}_{t}q_{t}\right)" display="inline"><semantics id="S1.F4.8.7.3.m1.1a"><mrow id="S1.F4.8.7.3.m1.1.1" xref="S1.F4.8.7.3.m1.1.1.cmml"><msub id="S1.F4.8.7.3.m1.1.1.3" xref="S1.F4.8.7.3.m1.1.1.3.cmml"><mi id="S1.F4.8.7.3.m1.1.1.3.2" xref="S1.F4.8.7.3.m1.1.1.3.2.cmml">z</mi><mi id="S1.F4.8.7.3.m1.1.1.3.3" xref="S1.F4.8.7.3.m1.1.1.3.3.cmml">t</mi></msub><mo id="S1.F4.8.7.3.m1.1.1.2" xref="S1.F4.8.7.3.m1.1.1.2.cmml">=</mo><mrow id="S1.F4.8.7.3.m1.1.1.1" xref="S1.F4.8.7.3.m1.1.1.1.cmml"><msub id="S1.F4.8.7.3.m1.1.1.1.3" xref="S1.F4.8.7.3.m1.1.1.1.3.cmml"><mi id="S1.F4.8.7.3.m1.1.1.1.3.2" xref="S1.F4.8.7.3.m1.1.1.1.3.2.cmml">V</mi><mi id="S1.F4.8.7.3.m1.1.1.1.3.3" xref="S1.F4.8.7.3.m1.1.1.1.3.3.cmml">t</mi></msub><mo lspace="0em" rspace="0em" id="S1.F4.8.7.3.m1.1.1.1.2" xref="S1.F4.8.7.3.m1.1.1.1.2.cmml">​</mo><mtext class="ltx_mathvariant_monospace" id="S1.F4.8.7.3.m1.1.1.1.4" xref="S1.F4.8.7.3.m1.1.1.1.4a.cmml">softmax</mtext><mo lspace="0em" rspace="0em" id="S1.F4.8.7.3.m1.1.1.1.2a" xref="S1.F4.8.7.3.m1.1.1.1.2.cmml">​</mo><mrow id="S1.F4.8.7.3.m1.1.1.1.1.1" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.cmml"><mo id="S1.F4.8.7.3.m1.1.1.1.1.1.2" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S1.F4.8.7.3.m1.1.1.1.1.1.1" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.cmml"><msubsup id="S1.F4.8.7.3.m1.1.1.1.1.1.1.2" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.2.cmml"><mi id="S1.F4.8.7.3.m1.1.1.1.1.1.1.2.2.2" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.2.2.2.cmml">K</mi><mi id="S1.F4.8.7.3.m1.1.1.1.1.1.1.2.3" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.2.3.cmml">t</mi><mi id="S1.F4.8.7.3.m1.1.1.1.1.1.1.2.2.3" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.2.2.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="S1.F4.8.7.3.m1.1.1.1.1.1.1.1" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.1.cmml">​</mo><msub id="S1.F4.8.7.3.m1.1.1.1.1.1.1.3" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.3.cmml"><mi id="S1.F4.8.7.3.m1.1.1.1.1.1.1.3.2" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.3.2.cmml">q</mi><mi id="S1.F4.8.7.3.m1.1.1.1.1.1.1.3.3" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.3.3.cmml">t</mi></msub></mrow><mo id="S1.F4.8.7.3.m1.1.1.1.1.1.3" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.F4.8.7.3.m1.1b"><apply id="S1.F4.8.7.3.m1.1.1.cmml" xref="S1.F4.8.7.3.m1.1.1"><eq id="S1.F4.8.7.3.m1.1.1.2.cmml" xref="S1.F4.8.7.3.m1.1.1.2"></eq><apply id="S1.F4.8.7.3.m1.1.1.3.cmml" xref="S1.F4.8.7.3.m1.1.1.3"><csymbol cd="ambiguous" id="S1.F4.8.7.3.m1.1.1.3.1.cmml" xref="S1.F4.8.7.3.m1.1.1.3">subscript</csymbol><ci id="S1.F4.8.7.3.m1.1.1.3.2.cmml" xref="S1.F4.8.7.3.m1.1.1.3.2">𝑧</ci><ci id="S1.F4.8.7.3.m1.1.1.3.3.cmml" xref="S1.F4.8.7.3.m1.1.1.3.3">𝑡</ci></apply><apply id="S1.F4.8.7.3.m1.1.1.1.cmml" xref="S1.F4.8.7.3.m1.1.1.1"><times id="S1.F4.8.7.3.m1.1.1.1.2.cmml" xref="S1.F4.8.7.3.m1.1.1.1.2"></times><apply id="S1.F4.8.7.3.m1.1.1.1.3.cmml" xref="S1.F4.8.7.3.m1.1.1.1.3"><csymbol cd="ambiguous" id="S1.F4.8.7.3.m1.1.1.1.3.1.cmml" xref="S1.F4.8.7.3.m1.1.1.1.3">subscript</csymbol><ci id="S1.F4.8.7.3.m1.1.1.1.3.2.cmml" xref="S1.F4.8.7.3.m1.1.1.1.3.2">𝑉</ci><ci id="S1.F4.8.7.3.m1.1.1.1.3.3.cmml" xref="S1.F4.8.7.3.m1.1.1.1.3.3">𝑡</ci></apply><ci id="S1.F4.8.7.3.m1.1.1.1.4a.cmml" xref="S1.F4.8.7.3.m1.1.1.1.4"><mtext class="ltx_mathvariant_monospace" id="S1.F4.8.7.3.m1.1.1.1.4.cmml" xref="S1.F4.8.7.3.m1.1.1.1.4">softmax</mtext></ci><apply id="S1.F4.8.7.3.m1.1.1.1.1.1.1.cmml" xref="S1.F4.8.7.3.m1.1.1.1.1.1"><times id="S1.F4.8.7.3.m1.1.1.1.1.1.1.1.cmml" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.1"></times><apply id="S1.F4.8.7.3.m1.1.1.1.1.1.1.2.cmml" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S1.F4.8.7.3.m1.1.1.1.1.1.1.2.1.cmml" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.2">subscript</csymbol><apply id="S1.F4.8.7.3.m1.1.1.1.1.1.1.2.2.cmml" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S1.F4.8.7.3.m1.1.1.1.1.1.1.2.2.1.cmml" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.2">superscript</csymbol><ci id="S1.F4.8.7.3.m1.1.1.1.1.1.1.2.2.2.cmml" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.2.2.2">𝐾</ci><ci id="S1.F4.8.7.3.m1.1.1.1.1.1.1.2.2.3.cmml" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.2.2.3">𝑇</ci></apply><ci id="S1.F4.8.7.3.m1.1.1.1.1.1.1.2.3.cmml" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.2.3">𝑡</ci></apply><apply id="S1.F4.8.7.3.m1.1.1.1.1.1.1.3.cmml" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S1.F4.8.7.3.m1.1.1.1.1.1.1.3.1.cmml" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S1.F4.8.7.3.m1.1.1.1.1.1.1.3.2.cmml" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.3.2">𝑞</ci><ci id="S1.F4.8.7.3.m1.1.1.1.1.1.1.3.3.cmml" xref="S1.F4.8.7.3.m1.1.1.1.1.1.1.3.3">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F4.8.7.3.m1.1c">z_{t}=V_{t}\texttt{softmax}\left(K^{T}_{t}q_{t}\right)</annotation></semantics></math></td>
<td id="S1.F4.9.8.4" class="ltx_td ltx_align_left ltx_border_r" style="padding-top:4pt;padding-bottom:4pt;"><math id="S1.F4.9.8.4.m1.1" class="ltx_Math" alttext="O(t)" display="inline"><semantics id="S1.F4.9.8.4.m1.1a"><mrow id="S1.F4.9.8.4.m1.1.2" xref="S1.F4.9.8.4.m1.1.2.cmml"><mi id="S1.F4.9.8.4.m1.1.2.2" xref="S1.F4.9.8.4.m1.1.2.2.cmml">O</mi><mo lspace="0em" rspace="0em" id="S1.F4.9.8.4.m1.1.2.1" xref="S1.F4.9.8.4.m1.1.2.1.cmml">​</mo><mrow id="S1.F4.9.8.4.m1.1.2.3.2" xref="S1.F4.9.8.4.m1.1.2.cmml"><mo stretchy="false" id="S1.F4.9.8.4.m1.1.2.3.2.1" xref="S1.F4.9.8.4.m1.1.2.cmml">(</mo><mi id="S1.F4.9.8.4.m1.1.1" xref="S1.F4.9.8.4.m1.1.1.cmml">t</mi><mo stretchy="false" id="S1.F4.9.8.4.m1.1.2.3.2.2" xref="S1.F4.9.8.4.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.F4.9.8.4.m1.1b"><apply id="S1.F4.9.8.4.m1.1.2.cmml" xref="S1.F4.9.8.4.m1.1.2"><times id="S1.F4.9.8.4.m1.1.2.1.cmml" xref="S1.F4.9.8.4.m1.1.2.1"></times><ci id="S1.F4.9.8.4.m1.1.2.2.cmml" xref="S1.F4.9.8.4.m1.1.2.2">𝑂</ci><ci id="S1.F4.9.8.4.m1.1.1.cmml" xref="S1.F4.9.8.4.m1.1.1">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F4.9.8.4.m1.1c">O(t)</annotation></semantics></math></td>
</tr>
<tr id="S1.F4.13.12" class="ltx_tr">
<th id="S1.F4.13.12.5" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_b ltx_border_l ltx_border_r" style="padding-top:4pt;padding-bottom:4pt;"><span id="S1.F4.13.12.5.1" class="ltx_text ltx_font_bold">Naive TTT</span></th>
<td id="S1.F4.10.9.1" class="ltx_td ltx_align_left ltx_border_b ltx_border_r" style="padding-top:4pt;padding-bottom:4pt;"><math id="S1.F4.10.9.1.m1.2" class="ltx_Math" alttext="W_{0}=f.\texttt{params()}" display="inline"><semantics id="S1.F4.10.9.1.m1.2a"><mrow id="S1.F4.10.9.1.m1.2.2.1" xref="S1.F4.10.9.1.m1.2.2.2.cmml"><mrow id="S1.F4.10.9.1.m1.2.2.1.1" xref="S1.F4.10.9.1.m1.2.2.1.1.cmml"><msub id="S1.F4.10.9.1.m1.2.2.1.1.2" xref="S1.F4.10.9.1.m1.2.2.1.1.2.cmml"><mi id="S1.F4.10.9.1.m1.2.2.1.1.2.2" xref="S1.F4.10.9.1.m1.2.2.1.1.2.2.cmml">W</mi><mn id="S1.F4.10.9.1.m1.2.2.1.1.2.3" xref="S1.F4.10.9.1.m1.2.2.1.1.2.3.cmml">0</mn></msub><mo id="S1.F4.10.9.1.m1.2.2.1.1.1" xref="S1.F4.10.9.1.m1.2.2.1.1.1.cmml">=</mo><mi id="S1.F4.10.9.1.m1.2.2.1.1.3" xref="S1.F4.10.9.1.m1.2.2.1.1.3.cmml">f</mi></mrow><mo lspace="0em" rspace="0.167em" id="S1.F4.10.9.1.m1.2.2.1.2" xref="S1.F4.10.9.1.m1.2.2.2a.cmml">.</mo><mtext class="ltx_mathvariant_monospace" id="S1.F4.10.9.1.m1.1.1" xref="S1.F4.10.9.1.m1.1.1a.cmml">params()</mtext></mrow><annotation-xml encoding="MathML-Content" id="S1.F4.10.9.1.m1.2b"><apply id="S1.F4.10.9.1.m1.2.2.2.cmml" xref="S1.F4.10.9.1.m1.2.2.1"><csymbol cd="ambiguous" id="S1.F4.10.9.1.m1.2.2.2a.cmml" xref="S1.F4.10.9.1.m1.2.2.1.2">formulae-sequence</csymbol><apply id="S1.F4.10.9.1.m1.2.2.1.1.cmml" xref="S1.F4.10.9.1.m1.2.2.1.1"><eq id="S1.F4.10.9.1.m1.2.2.1.1.1.cmml" xref="S1.F4.10.9.1.m1.2.2.1.1.1"></eq><apply id="S1.F4.10.9.1.m1.2.2.1.1.2.cmml" xref="S1.F4.10.9.1.m1.2.2.1.1.2"><csymbol cd="ambiguous" id="S1.F4.10.9.1.m1.2.2.1.1.2.1.cmml" xref="S1.F4.10.9.1.m1.2.2.1.1.2">subscript</csymbol><ci id="S1.F4.10.9.1.m1.2.2.1.1.2.2.cmml" xref="S1.F4.10.9.1.m1.2.2.1.1.2.2">𝑊</ci><cn type="integer" id="S1.F4.10.9.1.m1.2.2.1.1.2.3.cmml" xref="S1.F4.10.9.1.m1.2.2.1.1.2.3">0</cn></apply><ci id="S1.F4.10.9.1.m1.2.2.1.1.3.cmml" xref="S1.F4.10.9.1.m1.2.2.1.1.3">𝑓</ci></apply><ci id="S1.F4.10.9.1.m1.1.1a.cmml" xref="S1.F4.10.9.1.m1.1.1"><mtext class="ltx_mathvariant_monospace" id="S1.F4.10.9.1.m1.1.1.cmml" xref="S1.F4.10.9.1.m1.1.1">params()</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F4.10.9.1.m1.2c">W_{0}=f.\texttt{params()}</annotation></semantics></math></td>
<td id="S1.F4.11.10.2" class="ltx_td ltx_align_left ltx_border_b ltx_border_r" style="padding-top:4pt;padding-bottom:4pt;"><math id="S1.F4.11.10.2.m1.2" class="ltx_Math" alttext="W_{t}=W_{t-1}-\eta\nabla\ell(W_{t-1};x_{t})" display="inline"><semantics id="S1.F4.11.10.2.m1.2a"><mrow id="S1.F4.11.10.2.m1.2.2" xref="S1.F4.11.10.2.m1.2.2.cmml"><msub id="S1.F4.11.10.2.m1.2.2.4" xref="S1.F4.11.10.2.m1.2.2.4.cmml"><mi id="S1.F4.11.10.2.m1.2.2.4.2" xref="S1.F4.11.10.2.m1.2.2.4.2.cmml">W</mi><mi id="S1.F4.11.10.2.m1.2.2.4.3" xref="S1.F4.11.10.2.m1.2.2.4.3.cmml">t</mi></msub><mo id="S1.F4.11.10.2.m1.2.2.3" xref="S1.F4.11.10.2.m1.2.2.3.cmml">=</mo><mrow id="S1.F4.11.10.2.m1.2.2.2" xref="S1.F4.11.10.2.m1.2.2.2.cmml"><msub id="S1.F4.11.10.2.m1.2.2.2.4" xref="S1.F4.11.10.2.m1.2.2.2.4.cmml"><mi id="S1.F4.11.10.2.m1.2.2.2.4.2" xref="S1.F4.11.10.2.m1.2.2.2.4.2.cmml">W</mi><mrow id="S1.F4.11.10.2.m1.2.2.2.4.3" xref="S1.F4.11.10.2.m1.2.2.2.4.3.cmml"><mi id="S1.F4.11.10.2.m1.2.2.2.4.3.2" xref="S1.F4.11.10.2.m1.2.2.2.4.3.2.cmml">t</mi><mo id="S1.F4.11.10.2.m1.2.2.2.4.3.1" xref="S1.F4.11.10.2.m1.2.2.2.4.3.1.cmml">−</mo><mn id="S1.F4.11.10.2.m1.2.2.2.4.3.3" xref="S1.F4.11.10.2.m1.2.2.2.4.3.3.cmml">1</mn></mrow></msub><mo id="S1.F4.11.10.2.m1.2.2.2.3" xref="S1.F4.11.10.2.m1.2.2.2.3.cmml">−</mo><mrow id="S1.F4.11.10.2.m1.2.2.2.2" xref="S1.F4.11.10.2.m1.2.2.2.2.cmml"><mi id="S1.F4.11.10.2.m1.2.2.2.2.4" xref="S1.F4.11.10.2.m1.2.2.2.2.4.cmml">η</mi><mo lspace="0.167em" rspace="0em" id="S1.F4.11.10.2.m1.2.2.2.2.3" xref="S1.F4.11.10.2.m1.2.2.2.2.3.cmml">​</mo><mrow id="S1.F4.11.10.2.m1.2.2.2.2.5" xref="S1.F4.11.10.2.m1.2.2.2.2.5.cmml"><mo rspace="0.167em" id="S1.F4.11.10.2.m1.2.2.2.2.5.1" xref="S1.F4.11.10.2.m1.2.2.2.2.5.1.cmml">∇</mo><mi mathvariant="normal" id="S1.F4.11.10.2.m1.2.2.2.2.5.2" xref="S1.F4.11.10.2.m1.2.2.2.2.5.2.cmml">ℓ</mi></mrow><mo lspace="0em" rspace="0em" id="S1.F4.11.10.2.m1.2.2.2.2.3a" xref="S1.F4.11.10.2.m1.2.2.2.2.3.cmml">​</mo><mrow id="S1.F4.11.10.2.m1.2.2.2.2.2.2" xref="S1.F4.11.10.2.m1.2.2.2.2.2.3.cmml"><mo stretchy="false" id="S1.F4.11.10.2.m1.2.2.2.2.2.2.3" xref="S1.F4.11.10.2.m1.2.2.2.2.2.3.cmml">(</mo><msub id="S1.F4.11.10.2.m1.1.1.1.1.1.1.1" xref="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.cmml"><mi id="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.2" xref="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.2.cmml">W</mi><mrow id="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.3" xref="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.3.cmml"><mi id="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.3.2" xref="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.3.2.cmml">t</mi><mo id="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.3.1" xref="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.3.1.cmml">−</mo><mn id="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.3.3" xref="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.3.3.cmml">1</mn></mrow></msub><mo id="S1.F4.11.10.2.m1.2.2.2.2.2.2.4" xref="S1.F4.11.10.2.m1.2.2.2.2.2.3.cmml">;</mo><msub id="S1.F4.11.10.2.m1.2.2.2.2.2.2.2" xref="S1.F4.11.10.2.m1.2.2.2.2.2.2.2.cmml"><mi id="S1.F4.11.10.2.m1.2.2.2.2.2.2.2.2" xref="S1.F4.11.10.2.m1.2.2.2.2.2.2.2.2.cmml">x</mi><mi id="S1.F4.11.10.2.m1.2.2.2.2.2.2.2.3" xref="S1.F4.11.10.2.m1.2.2.2.2.2.2.2.3.cmml">t</mi></msub><mo stretchy="false" id="S1.F4.11.10.2.m1.2.2.2.2.2.2.5" xref="S1.F4.11.10.2.m1.2.2.2.2.2.3.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.F4.11.10.2.m1.2b"><apply id="S1.F4.11.10.2.m1.2.2.cmml" xref="S1.F4.11.10.2.m1.2.2"><eq id="S1.F4.11.10.2.m1.2.2.3.cmml" xref="S1.F4.11.10.2.m1.2.2.3"></eq><apply id="S1.F4.11.10.2.m1.2.2.4.cmml" xref="S1.F4.11.10.2.m1.2.2.4"><csymbol cd="ambiguous" id="S1.F4.11.10.2.m1.2.2.4.1.cmml" xref="S1.F4.11.10.2.m1.2.2.4">subscript</csymbol><ci id="S1.F4.11.10.2.m1.2.2.4.2.cmml" xref="S1.F4.11.10.2.m1.2.2.4.2">𝑊</ci><ci id="S1.F4.11.10.2.m1.2.2.4.3.cmml" xref="S1.F4.11.10.2.m1.2.2.4.3">𝑡</ci></apply><apply id="S1.F4.11.10.2.m1.2.2.2.cmml" xref="S1.F4.11.10.2.m1.2.2.2"><minus id="S1.F4.11.10.2.m1.2.2.2.3.cmml" xref="S1.F4.11.10.2.m1.2.2.2.3"></minus><apply id="S1.F4.11.10.2.m1.2.2.2.4.cmml" xref="S1.F4.11.10.2.m1.2.2.2.4"><csymbol cd="ambiguous" id="S1.F4.11.10.2.m1.2.2.2.4.1.cmml" xref="S1.F4.11.10.2.m1.2.2.2.4">subscript</csymbol><ci id="S1.F4.11.10.2.m1.2.2.2.4.2.cmml" xref="S1.F4.11.10.2.m1.2.2.2.4.2">𝑊</ci><apply id="S1.F4.11.10.2.m1.2.2.2.4.3.cmml" xref="S1.F4.11.10.2.m1.2.2.2.4.3"><minus id="S1.F4.11.10.2.m1.2.2.2.4.3.1.cmml" xref="S1.F4.11.10.2.m1.2.2.2.4.3.1"></minus><ci id="S1.F4.11.10.2.m1.2.2.2.4.3.2.cmml" xref="S1.F4.11.10.2.m1.2.2.2.4.3.2">𝑡</ci><cn type="integer" id="S1.F4.11.10.2.m1.2.2.2.4.3.3.cmml" xref="S1.F4.11.10.2.m1.2.2.2.4.3.3">1</cn></apply></apply><apply id="S1.F4.11.10.2.m1.2.2.2.2.cmml" xref="S1.F4.11.10.2.m1.2.2.2.2"><times id="S1.F4.11.10.2.m1.2.2.2.2.3.cmml" xref="S1.F4.11.10.2.m1.2.2.2.2.3"></times><ci id="S1.F4.11.10.2.m1.2.2.2.2.4.cmml" xref="S1.F4.11.10.2.m1.2.2.2.2.4">𝜂</ci><apply id="S1.F4.11.10.2.m1.2.2.2.2.5.cmml" xref="S1.F4.11.10.2.m1.2.2.2.2.5"><ci id="S1.F4.11.10.2.m1.2.2.2.2.5.1.cmml" xref="S1.F4.11.10.2.m1.2.2.2.2.5.1">∇</ci><ci id="S1.F4.11.10.2.m1.2.2.2.2.5.2.cmml" xref="S1.F4.11.10.2.m1.2.2.2.2.5.2">ℓ</ci></apply><list id="S1.F4.11.10.2.m1.2.2.2.2.2.3.cmml" xref="S1.F4.11.10.2.m1.2.2.2.2.2.2"><apply id="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.cmml" xref="S1.F4.11.10.2.m1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.1.cmml" xref="S1.F4.11.10.2.m1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.2.cmml" xref="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.2">𝑊</ci><apply id="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.3.cmml" xref="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.3"><minus id="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.3.1.cmml" xref="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.3.1"></minus><ci id="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.3.2.cmml" xref="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.3.2">𝑡</ci><cn type="integer" id="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.3.3.cmml" xref="S1.F4.11.10.2.m1.1.1.1.1.1.1.1.3.3">1</cn></apply></apply><apply id="S1.F4.11.10.2.m1.2.2.2.2.2.2.2.cmml" xref="S1.F4.11.10.2.m1.2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S1.F4.11.10.2.m1.2.2.2.2.2.2.2.1.cmml" xref="S1.F4.11.10.2.m1.2.2.2.2.2.2.2">subscript</csymbol><ci id="S1.F4.11.10.2.m1.2.2.2.2.2.2.2.2.cmml" xref="S1.F4.11.10.2.m1.2.2.2.2.2.2.2.2">𝑥</ci><ci id="S1.F4.11.10.2.m1.2.2.2.2.2.2.2.3.cmml" xref="S1.F4.11.10.2.m1.2.2.2.2.2.2.2.3">𝑡</ci></apply></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F4.11.10.2.m1.2c">W_{t}=W_{t-1}-\eta\nabla\ell(W_{t-1};x_{t})</annotation></semantics></math></td>
<td id="S1.F4.12.11.3" class="ltx_td ltx_align_left ltx_border_b ltx_border_r" style="padding-top:4pt;padding-bottom:4pt;"><math id="S1.F4.12.11.3.m1.2" class="ltx_Math" alttext="z_{t}=f(x_{t};W_{t})" display="inline"><semantics id="S1.F4.12.11.3.m1.2a"><mrow id="S1.F4.12.11.3.m1.2.2" xref="S1.F4.12.11.3.m1.2.2.cmml"><msub id="S1.F4.12.11.3.m1.2.2.4" xref="S1.F4.12.11.3.m1.2.2.4.cmml"><mi id="S1.F4.12.11.3.m1.2.2.4.2" xref="S1.F4.12.11.3.m1.2.2.4.2.cmml">z</mi><mi id="S1.F4.12.11.3.m1.2.2.4.3" xref="S1.F4.12.11.3.m1.2.2.4.3.cmml">t</mi></msub><mo id="S1.F4.12.11.3.m1.2.2.3" xref="S1.F4.12.11.3.m1.2.2.3.cmml">=</mo><mrow id="S1.F4.12.11.3.m1.2.2.2" xref="S1.F4.12.11.3.m1.2.2.2.cmml"><mi id="S1.F4.12.11.3.m1.2.2.2.4" xref="S1.F4.12.11.3.m1.2.2.2.4.cmml">f</mi><mo lspace="0em" rspace="0em" id="S1.F4.12.11.3.m1.2.2.2.3" xref="S1.F4.12.11.3.m1.2.2.2.3.cmml">​</mo><mrow id="S1.F4.12.11.3.m1.2.2.2.2.2" xref="S1.F4.12.11.3.m1.2.2.2.2.3.cmml"><mo stretchy="false" id="S1.F4.12.11.3.m1.2.2.2.2.2.3" xref="S1.F4.12.11.3.m1.2.2.2.2.3.cmml">(</mo><msub id="S1.F4.12.11.3.m1.1.1.1.1.1.1" xref="S1.F4.12.11.3.m1.1.1.1.1.1.1.cmml"><mi id="S1.F4.12.11.3.m1.1.1.1.1.1.1.2" xref="S1.F4.12.11.3.m1.1.1.1.1.1.1.2.cmml">x</mi><mi id="S1.F4.12.11.3.m1.1.1.1.1.1.1.3" xref="S1.F4.12.11.3.m1.1.1.1.1.1.1.3.cmml">t</mi></msub><mo id="S1.F4.12.11.3.m1.2.2.2.2.2.4" xref="S1.F4.12.11.3.m1.2.2.2.2.3.cmml">;</mo><msub id="S1.F4.12.11.3.m1.2.2.2.2.2.2" xref="S1.F4.12.11.3.m1.2.2.2.2.2.2.cmml"><mi id="S1.F4.12.11.3.m1.2.2.2.2.2.2.2" xref="S1.F4.12.11.3.m1.2.2.2.2.2.2.2.cmml">W</mi><mi id="S1.F4.12.11.3.m1.2.2.2.2.2.2.3" xref="S1.F4.12.11.3.m1.2.2.2.2.2.2.3.cmml">t</mi></msub><mo stretchy="false" id="S1.F4.12.11.3.m1.2.2.2.2.2.5" xref="S1.F4.12.11.3.m1.2.2.2.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.F4.12.11.3.m1.2b"><apply id="S1.F4.12.11.3.m1.2.2.cmml" xref="S1.F4.12.11.3.m1.2.2"><eq id="S1.F4.12.11.3.m1.2.2.3.cmml" xref="S1.F4.12.11.3.m1.2.2.3"></eq><apply id="S1.F4.12.11.3.m1.2.2.4.cmml" xref="S1.F4.12.11.3.m1.2.2.4"><csymbol cd="ambiguous" id="S1.F4.12.11.3.m1.2.2.4.1.cmml" xref="S1.F4.12.11.3.m1.2.2.4">subscript</csymbol><ci id="S1.F4.12.11.3.m1.2.2.4.2.cmml" xref="S1.F4.12.11.3.m1.2.2.4.2">𝑧</ci><ci id="S1.F4.12.11.3.m1.2.2.4.3.cmml" xref="S1.F4.12.11.3.m1.2.2.4.3">𝑡</ci></apply><apply id="S1.F4.12.11.3.m1.2.2.2.cmml" xref="S1.F4.12.11.3.m1.2.2.2"><times id="S1.F4.12.11.3.m1.2.2.2.3.cmml" xref="S1.F4.12.11.3.m1.2.2.2.3"></times><ci id="S1.F4.12.11.3.m1.2.2.2.4.cmml" xref="S1.F4.12.11.3.m1.2.2.2.4">𝑓</ci><list id="S1.F4.12.11.3.m1.2.2.2.2.3.cmml" xref="S1.F4.12.11.3.m1.2.2.2.2.2"><apply id="S1.F4.12.11.3.m1.1.1.1.1.1.1.cmml" xref="S1.F4.12.11.3.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S1.F4.12.11.3.m1.1.1.1.1.1.1.1.cmml" xref="S1.F4.12.11.3.m1.1.1.1.1.1.1">subscript</csymbol><ci id="S1.F4.12.11.3.m1.1.1.1.1.1.1.2.cmml" xref="S1.F4.12.11.3.m1.1.1.1.1.1.1.2">𝑥</ci><ci id="S1.F4.12.11.3.m1.1.1.1.1.1.1.3.cmml" xref="S1.F4.12.11.3.m1.1.1.1.1.1.1.3">𝑡</ci></apply><apply id="S1.F4.12.11.3.m1.2.2.2.2.2.2.cmml" xref="S1.F4.12.11.3.m1.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S1.F4.12.11.3.m1.2.2.2.2.2.2.1.cmml" xref="S1.F4.12.11.3.m1.2.2.2.2.2.2">subscript</csymbol><ci id="S1.F4.12.11.3.m1.2.2.2.2.2.2.2.cmml" xref="S1.F4.12.11.3.m1.2.2.2.2.2.2.2">𝑊</ci><ci id="S1.F4.12.11.3.m1.2.2.2.2.2.2.3.cmml" xref="S1.F4.12.11.3.m1.2.2.2.2.2.2.3">𝑡</ci></apply></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F4.12.11.3.m1.2c">z_{t}=f(x_{t};W_{t})</annotation></semantics></math></td>
<td id="S1.F4.13.12.4" class="ltx_td ltx_align_left ltx_border_b ltx_border_r" style="padding-top:4pt;padding-bottom:4pt;"><math id="S1.F4.13.12.4.m1.1" class="ltx_Math" alttext="O(1)" display="inline"><semantics id="S1.F4.13.12.4.m1.1a"><mrow id="S1.F4.13.12.4.m1.1.2" xref="S1.F4.13.12.4.m1.1.2.cmml"><mi id="S1.F4.13.12.4.m1.1.2.2" xref="S1.F4.13.12.4.m1.1.2.2.cmml">O</mi><mo lspace="0em" rspace="0em" id="S1.F4.13.12.4.m1.1.2.1" xref="S1.F4.13.12.4.m1.1.2.1.cmml">​</mo><mrow id="S1.F4.13.12.4.m1.1.2.3.2" xref="S1.F4.13.12.4.m1.1.2.cmml"><mo stretchy="false" id="S1.F4.13.12.4.m1.1.2.3.2.1" xref="S1.F4.13.12.4.m1.1.2.cmml">(</mo><mn id="S1.F4.13.12.4.m1.1.1" xref="S1.F4.13.12.4.m1.1.1.cmml">1</mn><mo stretchy="false" id="S1.F4.13.12.4.m1.1.2.3.2.2" xref="S1.F4.13.12.4.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.F4.13.12.4.m1.1b"><apply id="S1.F4.13.12.4.m1.1.2.cmml" xref="S1.F4.13.12.4.m1.1.2"><times id="S1.F4.13.12.4.m1.1.2.1.cmml" xref="S1.F4.13.12.4.m1.1.2.1"></times><ci id="S1.F4.13.12.4.m1.1.2.2.cmml" xref="S1.F4.13.12.4.m1.1.2.2">𝑂</ci><cn type="integer" id="S1.F4.13.12.4.m1.1.1.cmml" xref="S1.F4.13.12.4.m1.1.1">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F4.13.12.4.m1.1c">O(1)</annotation></semantics></math></td>
</tr>
</tbody>
</table>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span><span id="S1.F4.16.1" class="ltx_text ltx_font_bold">Top</span>: A generic sequence modeling layer expressed as a hidden state that transitions according to an update rule.
All sequence modeling layers can be viewed as different instantiations of three components in this figure: the initial state, update rule and output rule.
<span id="S1.F4.17.2" class="ltx_text ltx_font_bold">Bottom</span>: Examples of sequence modeling layers and their instantiations of the three components. The naive TTT layer was shown in Figure <a href="#S0.F1" title="Figure 1 ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.
Self-attention has a hidden state growing with context, therefore growing cost per token.
Both the naive RNN and TTT layer compress the growing context into a hidden state of fixed size, therefore their cost per token stays constant.
</figcaption>
</figure>
</section>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Method</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.3" class="ltx_p">All sequence modeling layers can be viewed from the perspective of storing historic context into a hidden state, as shown in Figure <a href="#S1.F4" title="Figure 4 ‣ Summary of contributions. ‣ 1 Introduction ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>.<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>

We define a sequence modeling layer as an autoregressive mapping from one sequence to another.
</span></span></span>
For example, RNN layers – such as LSTM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib33" title="" class="ltx_ref">33</a>]</cite>, RWKV <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib56" title="" class="ltx_ref">56</a>]</cite> and Mamba <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib26" title="" class="ltx_ref">26</a>]</cite> layers – compress context into a state of fixed size across time.
This compression has two consequences.
On one hand, mapping an input token <math id="S2.p1.1.m1.1" class="ltx_Math" alttext="x_{t}" display="inline"><semantics id="S2.p1.1.m1.1a"><msub id="S2.p1.1.m1.1.1" xref="S2.p1.1.m1.1.1.cmml"><mi id="S2.p1.1.m1.1.1.2" xref="S2.p1.1.m1.1.1.2.cmml">x</mi><mi id="S2.p1.1.m1.1.1.3" xref="S2.p1.1.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.p1.1.m1.1b"><apply id="S2.p1.1.m1.1.1.cmml" xref="S2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.p1.1.m1.1.1.1.cmml" xref="S2.p1.1.m1.1.1">subscript</csymbol><ci id="S2.p1.1.m1.1.1.2.cmml" xref="S2.p1.1.m1.1.1.2">𝑥</ci><ci id="S2.p1.1.m1.1.1.3.cmml" xref="S2.p1.1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.1.m1.1c">x_{t}</annotation></semantics></math> to output token <math id="S2.p1.2.m2.1" class="ltx_Math" alttext="z_{t}" display="inline"><semantics id="S2.p1.2.m2.1a"><msub id="S2.p1.2.m2.1.1" xref="S2.p1.2.m2.1.1.cmml"><mi id="S2.p1.2.m2.1.1.2" xref="S2.p1.2.m2.1.1.2.cmml">z</mi><mi id="S2.p1.2.m2.1.1.3" xref="S2.p1.2.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.p1.2.m2.1b"><apply id="S2.p1.2.m2.1.1.cmml" xref="S2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S2.p1.2.m2.1.1.1.cmml" xref="S2.p1.2.m2.1.1">subscript</csymbol><ci id="S2.p1.2.m2.1.1.2.cmml" xref="S2.p1.2.m2.1.1.2">𝑧</ci><ci id="S2.p1.2.m2.1.1.3.cmml" xref="S2.p1.2.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.2.m2.1c">z_{t}</annotation></semantics></math> is efficient, because both the update rule and output rule take constant time per token.
On the other hand, the performance of RNN layers in long context is limited by the expressive power of its hidden state <math id="S2.p1.3.m3.1" class="ltx_Math" alttext="s_{t}" display="inline"><semantics id="S2.p1.3.m3.1a"><msub id="S2.p1.3.m3.1.1" xref="S2.p1.3.m3.1.1.cmml"><mi id="S2.p1.3.m3.1.1.2" xref="S2.p1.3.m3.1.1.2.cmml">s</mi><mi id="S2.p1.3.m3.1.1.3" xref="S2.p1.3.m3.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.p1.3.m3.1b"><apply id="S2.p1.3.m3.1.1.cmml" xref="S2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S2.p1.3.m3.1.1.1.cmml" xref="S2.p1.3.m3.1.1">subscript</csymbol><ci id="S2.p1.3.m3.1.1.2.cmml" xref="S2.p1.3.m3.1.1.2">𝑠</ci><ci id="S2.p1.3.m3.1.1.3.cmml" xref="S2.p1.3.m3.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.p1.3.m3.1c">s_{t}</annotation></semantics></math>.</p>
</div>
<div id="S2.p2" class="ltx_para">
<p id="S2.p2.2" class="ltx_p">Self-attention can also be viewed from the perspective above, except that its hidden state, commonly known as the Key-Value (KV) cache, is a list that grows linearly with <math id="S2.p2.1.m1.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S2.p2.1.m1.1a"><mi id="S2.p2.1.m1.1.1" xref="S2.p2.1.m1.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S2.p2.1.m1.1b"><ci id="S2.p2.1.m1.1.1.cmml" xref="S2.p2.1.m1.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p2.1.m1.1c">t</annotation></semantics></math>.
Its update rule simply appends the current KV tuple to this list, and the output rule scans over all tuples up to <math id="S2.p2.2.m2.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S2.p2.2.m2.1a"><mi id="S2.p2.2.m2.1.1" xref="S2.p2.2.m2.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S2.p2.2.m2.1b"><ci id="S2.p2.2.m2.1.1.cmml" xref="S2.p2.2.m2.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.p2.2.m2.1c">t</annotation></semantics></math> to form the attention matrix.
The hidden state explicitly stores all historic context without compression, making self-attention more expressive than RNN layers for long context.
However, scanning this linearly growing hidden state also takes linearly growing time per token.</p>
</div>
<div id="S2.p3" class="ltx_para">
<p id="S2.p3.1" class="ltx_p">To remain both efficient and expressive in long context, we need a better compression heuristic.
Specifically, we need to compress thousands or potentially millions of tokens into a hidden state that can effectively capture their underlying structures and relationships.
This might sound like a tall order, but all of us are actually already familiar with such a heuristic.</p>
</div>
<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>TTT as updating a hidden state</h3>

<div id="S2.SS1.p1" class="ltx_para">
<p id="S2.SS1.p1.1" class="ltx_p">The process of parametric learning can be viewed as compressing a massive training set into the weights of a model.
Specifically, we know that models trained with self-supervision can capture the underlying structures and relationships behind their training data <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib48" title="" class="ltx_ref">48</a>]</cite> – exactly what we need from a compression heuristic.</p>
</div>
<div id="S2.SS1.p2" class="ltx_para">
<p id="S2.SS1.p2.1" class="ltx_p">LLMs themselves are great examples. Trained with the self-supervised task of next-token prediction, their weights can be viewed as a compressed form of storage for existing knowledge on the internet.
By querying LLMs, we can extract knowledge from their weights.
More importantly, LLMs often exhibit a deep understanding of the semantic connections among existing knowledge to express new pieces of reasoning <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib1" title="" class="ltx_ref">1</a>]</cite>.</p>
</div>
<div id="S2.SS1.p3" class="ltx_para">
<p id="S2.SS1.p3.5" class="ltx_p">Our key idea is to use self-supervised learning to compress the historic context <math id="S2.SS1.p3.1.m1.3" class="ltx_Math" alttext="x_{1},\dots,x_{t}" display="inline"><semantics id="S2.SS1.p3.1.m1.3a"><mrow id="S2.SS1.p3.1.m1.3.3.2" xref="S2.SS1.p3.1.m1.3.3.3.cmml"><msub id="S2.SS1.p3.1.m1.2.2.1.1" xref="S2.SS1.p3.1.m1.2.2.1.1.cmml"><mi id="S2.SS1.p3.1.m1.2.2.1.1.2" xref="S2.SS1.p3.1.m1.2.2.1.1.2.cmml">x</mi><mn id="S2.SS1.p3.1.m1.2.2.1.1.3" xref="S2.SS1.p3.1.m1.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.SS1.p3.1.m1.3.3.2.3" xref="S2.SS1.p3.1.m1.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS1.p3.1.m1.1.1" xref="S2.SS1.p3.1.m1.1.1.cmml">…</mi><mo id="S2.SS1.p3.1.m1.3.3.2.4" xref="S2.SS1.p3.1.m1.3.3.3.cmml">,</mo><msub id="S2.SS1.p3.1.m1.3.3.2.2" xref="S2.SS1.p3.1.m1.3.3.2.2.cmml"><mi id="S2.SS1.p3.1.m1.3.3.2.2.2" xref="S2.SS1.p3.1.m1.3.3.2.2.2.cmml">x</mi><mi id="S2.SS1.p3.1.m1.3.3.2.2.3" xref="S2.SS1.p3.1.m1.3.3.2.2.3.cmml">t</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.1.m1.3b"><list id="S2.SS1.p3.1.m1.3.3.3.cmml" xref="S2.SS1.p3.1.m1.3.3.2"><apply id="S2.SS1.p3.1.m1.2.2.1.1.cmml" xref="S2.SS1.p3.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p3.1.m1.2.2.1.1.1.cmml" xref="S2.SS1.p3.1.m1.2.2.1.1">subscript</csymbol><ci id="S2.SS1.p3.1.m1.2.2.1.1.2.cmml" xref="S2.SS1.p3.1.m1.2.2.1.1.2">𝑥</ci><cn type="integer" id="S2.SS1.p3.1.m1.2.2.1.1.3.cmml" xref="S2.SS1.p3.1.m1.2.2.1.1.3">1</cn></apply><ci id="S2.SS1.p3.1.m1.1.1.cmml" xref="S2.SS1.p3.1.m1.1.1">…</ci><apply id="S2.SS1.p3.1.m1.3.3.2.2.cmml" xref="S2.SS1.p3.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S2.SS1.p3.1.m1.3.3.2.2.1.cmml" xref="S2.SS1.p3.1.m1.3.3.2.2">subscript</csymbol><ci id="S2.SS1.p3.1.m1.3.3.2.2.2.cmml" xref="S2.SS1.p3.1.m1.3.3.2.2.2">𝑥</ci><ci id="S2.SS1.p3.1.m1.3.3.2.2.3.cmml" xref="S2.SS1.p3.1.m1.3.3.2.2.3">𝑡</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.1.m1.3c">x_{1},\dots,x_{t}</annotation></semantics></math> into a hidden state <math id="S2.SS1.p3.2.m2.1" class="ltx_Math" alttext="s_{t}" display="inline"><semantics id="S2.SS1.p3.2.m2.1a"><msub id="S2.SS1.p3.2.m2.1.1" xref="S2.SS1.p3.2.m2.1.1.cmml"><mi id="S2.SS1.p3.2.m2.1.1.2" xref="S2.SS1.p3.2.m2.1.1.2.cmml">s</mi><mi id="S2.SS1.p3.2.m2.1.1.3" xref="S2.SS1.p3.2.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.2.m2.1b"><apply id="S2.SS1.p3.2.m2.1.1.cmml" xref="S2.SS1.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p3.2.m2.1.1.1.cmml" xref="S2.SS1.p3.2.m2.1.1">subscript</csymbol><ci id="S2.SS1.p3.2.m2.1.1.2.cmml" xref="S2.SS1.p3.2.m2.1.1.2">𝑠</ci><ci id="S2.SS1.p3.2.m2.1.1.3.cmml" xref="S2.SS1.p3.2.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.2.m2.1c">s_{t}</annotation></semantics></math>, by making the context an unlabeled dataset and the state a model.
Concretely, the hidden state <math id="S2.SS1.p3.3.m3.1" class="ltx_Math" alttext="s_{t}" display="inline"><semantics id="S2.SS1.p3.3.m3.1a"><msub id="S2.SS1.p3.3.m3.1.1" xref="S2.SS1.p3.3.m3.1.1.cmml"><mi id="S2.SS1.p3.3.m3.1.1.2" xref="S2.SS1.p3.3.m3.1.1.2.cmml">s</mi><mi id="S2.SS1.p3.3.m3.1.1.3" xref="S2.SS1.p3.3.m3.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.3.m3.1b"><apply id="S2.SS1.p3.3.m3.1.1.cmml" xref="S2.SS1.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS1.p3.3.m3.1.1.1.cmml" xref="S2.SS1.p3.3.m3.1.1">subscript</csymbol><ci id="S2.SS1.p3.3.m3.1.1.2.cmml" xref="S2.SS1.p3.3.m3.1.1.2">𝑠</ci><ci id="S2.SS1.p3.3.m3.1.1.3.cmml" xref="S2.SS1.p3.3.m3.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.3.m3.1c">s_{t}</annotation></semantics></math> is now equivalent to <math id="S2.SS1.p3.4.m4.1" class="ltx_Math" alttext="W_{t}" display="inline"><semantics id="S2.SS1.p3.4.m4.1a"><msub id="S2.SS1.p3.4.m4.1.1" xref="S2.SS1.p3.4.m4.1.1.cmml"><mi id="S2.SS1.p3.4.m4.1.1.2" xref="S2.SS1.p3.4.m4.1.1.2.cmml">W</mi><mi id="S2.SS1.p3.4.m4.1.1.3" xref="S2.SS1.p3.4.m4.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.4.m4.1b"><apply id="S2.SS1.p3.4.m4.1.1.cmml" xref="S2.SS1.p3.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS1.p3.4.m4.1.1.1.cmml" xref="S2.SS1.p3.4.m4.1.1">subscript</csymbol><ci id="S2.SS1.p3.4.m4.1.1.2.cmml" xref="S2.SS1.p3.4.m4.1.1.2">𝑊</ci><ci id="S2.SS1.p3.4.m4.1.1.3.cmml" xref="S2.SS1.p3.4.m4.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.4.m4.1c">W_{t}</annotation></semantics></math>, the weights of a model <math id="S2.SS1.p3.5.m5.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S2.SS1.p3.5.m5.1a"><mi id="S2.SS1.p3.5.m5.1.1" xref="S2.SS1.p3.5.m5.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.5.m5.1b"><ci id="S2.SS1.p3.5.m5.1.1.cmml" xref="S2.SS1.p3.5.m5.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.5.m5.1c">f</annotation></semantics></math>, which can be a linear model, a small neural network, or anything else.
The output rule is simply:</p>
<table id="S2.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E1.m1.1" class="ltx_Math" alttext="z_{t}=f(x_{t};W_{t})." display="block"><semantics id="S2.E1.m1.1a"><mrow id="S2.E1.m1.1.1.1" xref="S2.E1.m1.1.1.1.1.cmml"><mrow id="S2.E1.m1.1.1.1.1" xref="S2.E1.m1.1.1.1.1.cmml"><msub id="S2.E1.m1.1.1.1.1.4" xref="S2.E1.m1.1.1.1.1.4.cmml"><mi id="S2.E1.m1.1.1.1.1.4.2" xref="S2.E1.m1.1.1.1.1.4.2.cmml">z</mi><mi id="S2.E1.m1.1.1.1.1.4.3" xref="S2.E1.m1.1.1.1.1.4.3.cmml">t</mi></msub><mo id="S2.E1.m1.1.1.1.1.3" xref="S2.E1.m1.1.1.1.1.3.cmml">=</mo><mrow id="S2.E1.m1.1.1.1.1.2" xref="S2.E1.m1.1.1.1.1.2.cmml"><mi id="S2.E1.m1.1.1.1.1.2.4" xref="S2.E1.m1.1.1.1.1.2.4.cmml">f</mi><mo lspace="0em" rspace="0em" id="S2.E1.m1.1.1.1.1.2.3" xref="S2.E1.m1.1.1.1.1.2.3.cmml">​</mo><mrow id="S2.E1.m1.1.1.1.1.2.2.2" xref="S2.E1.m1.1.1.1.1.2.2.3.cmml"><mo stretchy="false" id="S2.E1.m1.1.1.1.1.2.2.2.3" xref="S2.E1.m1.1.1.1.1.2.2.3.cmml">(</mo><msub id="S2.E1.m1.1.1.1.1.1.1.1.1" xref="S2.E1.m1.1.1.1.1.1.1.1.1.cmml"><mi id="S2.E1.m1.1.1.1.1.1.1.1.1.2" xref="S2.E1.m1.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="S2.E1.m1.1.1.1.1.1.1.1.1.3" xref="S2.E1.m1.1.1.1.1.1.1.1.1.3.cmml">t</mi></msub><mo id="S2.E1.m1.1.1.1.1.2.2.2.4" xref="S2.E1.m1.1.1.1.1.2.2.3.cmml">;</mo><msub id="S2.E1.m1.1.1.1.1.2.2.2.2" xref="S2.E1.m1.1.1.1.1.2.2.2.2.cmml"><mi id="S2.E1.m1.1.1.1.1.2.2.2.2.2" xref="S2.E1.m1.1.1.1.1.2.2.2.2.2.cmml">W</mi><mi id="S2.E1.m1.1.1.1.1.2.2.2.2.3" xref="S2.E1.m1.1.1.1.1.2.2.2.2.3.cmml">t</mi></msub><mo stretchy="false" id="S2.E1.m1.1.1.1.1.2.2.2.5" xref="S2.E1.m1.1.1.1.1.2.2.3.cmml">)</mo></mrow></mrow></mrow><mo lspace="0em" id="S2.E1.m1.1.1.1.2" xref="S2.E1.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E1.m1.1b"><apply id="S2.E1.m1.1.1.1.1.cmml" xref="S2.E1.m1.1.1.1"><eq id="S2.E1.m1.1.1.1.1.3.cmml" xref="S2.E1.m1.1.1.1.1.3"></eq><apply id="S2.E1.m1.1.1.1.1.4.cmml" xref="S2.E1.m1.1.1.1.1.4"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.4.1.cmml" xref="S2.E1.m1.1.1.1.1.4">subscript</csymbol><ci id="S2.E1.m1.1.1.1.1.4.2.cmml" xref="S2.E1.m1.1.1.1.1.4.2">𝑧</ci><ci id="S2.E1.m1.1.1.1.1.4.3.cmml" xref="S2.E1.m1.1.1.1.1.4.3">𝑡</ci></apply><apply id="S2.E1.m1.1.1.1.1.2.cmml" xref="S2.E1.m1.1.1.1.1.2"><times id="S2.E1.m1.1.1.1.1.2.3.cmml" xref="S2.E1.m1.1.1.1.1.2.3"></times><ci id="S2.E1.m1.1.1.1.1.2.4.cmml" xref="S2.E1.m1.1.1.1.1.2.4">𝑓</ci><list id="S2.E1.m1.1.1.1.1.2.2.3.cmml" xref="S2.E1.m1.1.1.1.1.2.2.2"><apply id="S2.E1.m1.1.1.1.1.1.1.1.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E1.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="S2.E1.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.E1.m1.1.1.1.1.1.1.1.1.3">𝑡</ci></apply><apply id="S2.E1.m1.1.1.1.1.2.2.2.2.cmml" xref="S2.E1.m1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S2.E1.m1.1.1.1.1.2.2.2.2.1.cmml" xref="S2.E1.m1.1.1.1.1.2.2.2.2">subscript</csymbol><ci id="S2.E1.m1.1.1.1.1.2.2.2.2.2.cmml" xref="S2.E1.m1.1.1.1.1.2.2.2.2.2">𝑊</ci><ci id="S2.E1.m1.1.1.1.1.2.2.2.2.3.cmml" xref="S2.E1.m1.1.1.1.1.2.2.2.2.3">𝑡</ci></apply></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E1.m1.1c">z_{t}=f(x_{t};W_{t}).</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<p id="S2.SS1.p3.9" class="ltx_p">Intuitively, the output token is just the prediction on <math id="S2.SS1.p3.6.m1.1" class="ltx_Math" alttext="x_{t}" display="inline"><semantics id="S2.SS1.p3.6.m1.1a"><msub id="S2.SS1.p3.6.m1.1.1" xref="S2.SS1.p3.6.m1.1.1.cmml"><mi id="S2.SS1.p3.6.m1.1.1.2" xref="S2.SS1.p3.6.m1.1.1.2.cmml">x</mi><mi id="S2.SS1.p3.6.m1.1.1.3" xref="S2.SS1.p3.6.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.6.m1.1b"><apply id="S2.SS1.p3.6.m1.1.1.cmml" xref="S2.SS1.p3.6.m1.1.1"><csymbol cd="ambiguous" id="S2.SS1.p3.6.m1.1.1.1.cmml" xref="S2.SS1.p3.6.m1.1.1">subscript</csymbol><ci id="S2.SS1.p3.6.m1.1.1.2.cmml" xref="S2.SS1.p3.6.m1.1.1.2">𝑥</ci><ci id="S2.SS1.p3.6.m1.1.1.3.cmml" xref="S2.SS1.p3.6.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.6.m1.1c">x_{t}</annotation></semantics></math>, made by <math id="S2.SS1.p3.7.m2.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S2.SS1.p3.7.m2.1a"><mi id="S2.SS1.p3.7.m2.1.1" xref="S2.SS1.p3.7.m2.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.7.m2.1b"><ci id="S2.SS1.p3.7.m2.1.1.cmml" xref="S2.SS1.p3.7.m2.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.7.m2.1c">f</annotation></semantics></math> with the updated weights <math id="S2.SS1.p3.8.m3.1" class="ltx_Math" alttext="W_{t}" display="inline"><semantics id="S2.SS1.p3.8.m3.1a"><msub id="S2.SS1.p3.8.m3.1.1" xref="S2.SS1.p3.8.m3.1.1.cmml"><mi id="S2.SS1.p3.8.m3.1.1.2" xref="S2.SS1.p3.8.m3.1.1.2.cmml">W</mi><mi id="S2.SS1.p3.8.m3.1.1.3" xref="S2.SS1.p3.8.m3.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.8.m3.1b"><apply id="S2.SS1.p3.8.m3.1.1.cmml" xref="S2.SS1.p3.8.m3.1.1"><csymbol cd="ambiguous" id="S2.SS1.p3.8.m3.1.1.1.cmml" xref="S2.SS1.p3.8.m3.1.1">subscript</csymbol><ci id="S2.SS1.p3.8.m3.1.1.2.cmml" xref="S2.SS1.p3.8.m3.1.1.2">𝑊</ci><ci id="S2.SS1.p3.8.m3.1.1.3.cmml" xref="S2.SS1.p3.8.m3.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.8.m3.1c">W_{t}</annotation></semantics></math>.
The update rule is a step of gradient descent on some self-supervised loss <math id="S2.SS1.p3.9.m4.1" class="ltx_Math" alttext="\ell" display="inline"><semantics id="S2.SS1.p3.9.m4.1a"><mi mathvariant="normal" id="S2.SS1.p3.9.m4.1.1" xref="S2.SS1.p3.9.m4.1.1.cmml">ℓ</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.9.m4.1b"><ci id="S2.SS1.p3.9.m4.1.1.cmml" xref="S2.SS1.p3.9.m4.1.1">ℓ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.9.m4.1c">\ell</annotation></semantics></math>:</p>
<table id="S2.E2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E2.m1.1" class="ltx_Math" alttext="W_{t}=W_{t-1}-\eta\,\nabla\ell(W_{t-1};x_{t})," display="block"><semantics id="S2.E2.m1.1a"><mrow id="S2.E2.m1.1.1.1" xref="S2.E2.m1.1.1.1.1.cmml"><mrow id="S2.E2.m1.1.1.1.1" xref="S2.E2.m1.1.1.1.1.cmml"><msub id="S2.E2.m1.1.1.1.1.4" xref="S2.E2.m1.1.1.1.1.4.cmml"><mi id="S2.E2.m1.1.1.1.1.4.2" xref="S2.E2.m1.1.1.1.1.4.2.cmml">W</mi><mi id="S2.E2.m1.1.1.1.1.4.3" xref="S2.E2.m1.1.1.1.1.4.3.cmml">t</mi></msub><mo id="S2.E2.m1.1.1.1.1.3" xref="S2.E2.m1.1.1.1.1.3.cmml">=</mo><mrow id="S2.E2.m1.1.1.1.1.2" xref="S2.E2.m1.1.1.1.1.2.cmml"><msub id="S2.E2.m1.1.1.1.1.2.4" xref="S2.E2.m1.1.1.1.1.2.4.cmml"><mi id="S2.E2.m1.1.1.1.1.2.4.2" xref="S2.E2.m1.1.1.1.1.2.4.2.cmml">W</mi><mrow id="S2.E2.m1.1.1.1.1.2.4.3" xref="S2.E2.m1.1.1.1.1.2.4.3.cmml"><mi id="S2.E2.m1.1.1.1.1.2.4.3.2" xref="S2.E2.m1.1.1.1.1.2.4.3.2.cmml">t</mi><mo id="S2.E2.m1.1.1.1.1.2.4.3.1" xref="S2.E2.m1.1.1.1.1.2.4.3.1.cmml">−</mo><mn id="S2.E2.m1.1.1.1.1.2.4.3.3" xref="S2.E2.m1.1.1.1.1.2.4.3.3.cmml">1</mn></mrow></msub><mo id="S2.E2.m1.1.1.1.1.2.3" xref="S2.E2.m1.1.1.1.1.2.3.cmml">−</mo><mrow id="S2.E2.m1.1.1.1.1.2.2" xref="S2.E2.m1.1.1.1.1.2.2.cmml"><mi id="S2.E2.m1.1.1.1.1.2.2.4" xref="S2.E2.m1.1.1.1.1.2.2.4.cmml">η</mi><mo lspace="0.337em" rspace="0em" id="S2.E2.m1.1.1.1.1.2.2.3" xref="S2.E2.m1.1.1.1.1.2.2.3.cmml">​</mo><mrow id="S2.E2.m1.1.1.1.1.2.2.5" xref="S2.E2.m1.1.1.1.1.2.2.5.cmml"><mo rspace="0.167em" id="S2.E2.m1.1.1.1.1.2.2.5.1" xref="S2.E2.m1.1.1.1.1.2.2.5.1.cmml">∇</mo><mi mathvariant="normal" id="S2.E2.m1.1.1.1.1.2.2.5.2" xref="S2.E2.m1.1.1.1.1.2.2.5.2.cmml">ℓ</mi></mrow><mo lspace="0em" rspace="0em" id="S2.E2.m1.1.1.1.1.2.2.3a" xref="S2.E2.m1.1.1.1.1.2.2.3.cmml">​</mo><mrow id="S2.E2.m1.1.1.1.1.2.2.2.2" xref="S2.E2.m1.1.1.1.1.2.2.2.3.cmml"><mo stretchy="false" id="S2.E2.m1.1.1.1.1.2.2.2.2.3" xref="S2.E2.m1.1.1.1.1.2.2.2.3.cmml">(</mo><msub id="S2.E2.m1.1.1.1.1.1.1.1.1.1" xref="S2.E2.m1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S2.E2.m1.1.1.1.1.1.1.1.1.1.2" xref="S2.E2.m1.1.1.1.1.1.1.1.1.1.2.cmml">W</mi><mrow id="S2.E2.m1.1.1.1.1.1.1.1.1.1.3" xref="S2.E2.m1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S2.E2.m1.1.1.1.1.1.1.1.1.1.3.2" xref="S2.E2.m1.1.1.1.1.1.1.1.1.1.3.2.cmml">t</mi><mo id="S2.E2.m1.1.1.1.1.1.1.1.1.1.3.1" xref="S2.E2.m1.1.1.1.1.1.1.1.1.1.3.1.cmml">−</mo><mn id="S2.E2.m1.1.1.1.1.1.1.1.1.1.3.3" xref="S2.E2.m1.1.1.1.1.1.1.1.1.1.3.3.cmml">1</mn></mrow></msub><mo id="S2.E2.m1.1.1.1.1.2.2.2.2.4" xref="S2.E2.m1.1.1.1.1.2.2.2.3.cmml">;</mo><msub id="S2.E2.m1.1.1.1.1.2.2.2.2.2" xref="S2.E2.m1.1.1.1.1.2.2.2.2.2.cmml"><mi id="S2.E2.m1.1.1.1.1.2.2.2.2.2.2" xref="S2.E2.m1.1.1.1.1.2.2.2.2.2.2.cmml">x</mi><mi id="S2.E2.m1.1.1.1.1.2.2.2.2.2.3" xref="S2.E2.m1.1.1.1.1.2.2.2.2.2.3.cmml">t</mi></msub><mo stretchy="false" id="S2.E2.m1.1.1.1.1.2.2.2.2.5" xref="S2.E2.m1.1.1.1.1.2.2.2.3.cmml">)</mo></mrow></mrow></mrow></mrow><mo id="S2.E2.m1.1.1.1.2" xref="S2.E2.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E2.m1.1b"><apply id="S2.E2.m1.1.1.1.1.cmml" xref="S2.E2.m1.1.1.1"><eq id="S2.E2.m1.1.1.1.1.3.cmml" xref="S2.E2.m1.1.1.1.1.3"></eq><apply id="S2.E2.m1.1.1.1.1.4.cmml" xref="S2.E2.m1.1.1.1.1.4"><csymbol cd="ambiguous" id="S2.E2.m1.1.1.1.1.4.1.cmml" xref="S2.E2.m1.1.1.1.1.4">subscript</csymbol><ci id="S2.E2.m1.1.1.1.1.4.2.cmml" xref="S2.E2.m1.1.1.1.1.4.2">𝑊</ci><ci id="S2.E2.m1.1.1.1.1.4.3.cmml" xref="S2.E2.m1.1.1.1.1.4.3">𝑡</ci></apply><apply id="S2.E2.m1.1.1.1.1.2.cmml" xref="S2.E2.m1.1.1.1.1.2"><minus id="S2.E2.m1.1.1.1.1.2.3.cmml" xref="S2.E2.m1.1.1.1.1.2.3"></minus><apply id="S2.E2.m1.1.1.1.1.2.4.cmml" xref="S2.E2.m1.1.1.1.1.2.4"><csymbol cd="ambiguous" id="S2.E2.m1.1.1.1.1.2.4.1.cmml" xref="S2.E2.m1.1.1.1.1.2.4">subscript</csymbol><ci id="S2.E2.m1.1.1.1.1.2.4.2.cmml" xref="S2.E2.m1.1.1.1.1.2.4.2">𝑊</ci><apply id="S2.E2.m1.1.1.1.1.2.4.3.cmml" xref="S2.E2.m1.1.1.1.1.2.4.3"><minus id="S2.E2.m1.1.1.1.1.2.4.3.1.cmml" xref="S2.E2.m1.1.1.1.1.2.4.3.1"></minus><ci id="S2.E2.m1.1.1.1.1.2.4.3.2.cmml" xref="S2.E2.m1.1.1.1.1.2.4.3.2">𝑡</ci><cn type="integer" id="S2.E2.m1.1.1.1.1.2.4.3.3.cmml" xref="S2.E2.m1.1.1.1.1.2.4.3.3">1</cn></apply></apply><apply id="S2.E2.m1.1.1.1.1.2.2.cmml" xref="S2.E2.m1.1.1.1.1.2.2"><times id="S2.E2.m1.1.1.1.1.2.2.3.cmml" xref="S2.E2.m1.1.1.1.1.2.2.3"></times><ci id="S2.E2.m1.1.1.1.1.2.2.4.cmml" xref="S2.E2.m1.1.1.1.1.2.2.4">𝜂</ci><apply id="S2.E2.m1.1.1.1.1.2.2.5.cmml" xref="S2.E2.m1.1.1.1.1.2.2.5"><ci id="S2.E2.m1.1.1.1.1.2.2.5.1.cmml" xref="S2.E2.m1.1.1.1.1.2.2.5.1">∇</ci><ci id="S2.E2.m1.1.1.1.1.2.2.5.2.cmml" xref="S2.E2.m1.1.1.1.1.2.2.5.2">ℓ</ci></apply><list id="S2.E2.m1.1.1.1.1.2.2.2.3.cmml" xref="S2.E2.m1.1.1.1.1.2.2.2.2"><apply id="S2.E2.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E2.m1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E2.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E2.m1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E2.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E2.m1.1.1.1.1.1.1.1.1.1.2">𝑊</ci><apply id="S2.E2.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.E2.m1.1.1.1.1.1.1.1.1.1.3"><minus id="S2.E2.m1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S2.E2.m1.1.1.1.1.1.1.1.1.1.3.1"></minus><ci id="S2.E2.m1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S2.E2.m1.1.1.1.1.1.1.1.1.1.3.2">𝑡</ci><cn type="integer" id="S2.E2.m1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S2.E2.m1.1.1.1.1.1.1.1.1.1.3.3">1</cn></apply></apply><apply id="S2.E2.m1.1.1.1.1.2.2.2.2.2.cmml" xref="S2.E2.m1.1.1.1.1.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.E2.m1.1.1.1.1.2.2.2.2.2.1.cmml" xref="S2.E2.m1.1.1.1.1.2.2.2.2.2">subscript</csymbol><ci id="S2.E2.m1.1.1.1.1.2.2.2.2.2.2.cmml" xref="S2.E2.m1.1.1.1.1.2.2.2.2.2.2">𝑥</ci><ci id="S2.E2.m1.1.1.1.1.2.2.2.2.2.3.cmml" xref="S2.E2.m1.1.1.1.1.2.2.2.2.2.3">𝑡</ci></apply></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E2.m1.1c">W_{t}=W_{t-1}-\eta\,\nabla\ell(W_{t-1};x_{t}),</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
<p id="S2.SS1.p3.12" class="ltx_p">with learning rate <math id="S2.SS1.p3.10.m1.1" class="ltx_Math" alttext="\eta" display="inline"><semantics id="S2.SS1.p3.10.m1.1a"><mi id="S2.SS1.p3.10.m1.1.1" xref="S2.SS1.p3.10.m1.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.10.m1.1b"><ci id="S2.SS1.p3.10.m1.1.1.cmml" xref="S2.SS1.p3.10.m1.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.10.m1.1c">\eta</annotation></semantics></math>.<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>
For now, consider <math id="footnote2.m1.1" class="ltx_Math" alttext="W_{0}=0" display="inline"><semantics id="footnote2.m1.1b"><mrow id="footnote2.m1.1.1" xref="footnote2.m1.1.1.cmml"><msub id="footnote2.m1.1.1.2" xref="footnote2.m1.1.1.2.cmml"><mi id="footnote2.m1.1.1.2.2" xref="footnote2.m1.1.1.2.2.cmml">W</mi><mn id="footnote2.m1.1.1.2.3" xref="footnote2.m1.1.1.2.3.cmml">0</mn></msub><mo id="footnote2.m1.1.1.1" xref="footnote2.m1.1.1.1.cmml">=</mo><mn id="footnote2.m1.1.1.3" xref="footnote2.m1.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="footnote2.m1.1c"><apply id="footnote2.m1.1.1.cmml" xref="footnote2.m1.1.1"><eq id="footnote2.m1.1.1.1.cmml" xref="footnote2.m1.1.1.1"></eq><apply id="footnote2.m1.1.1.2.cmml" xref="footnote2.m1.1.1.2"><csymbol cd="ambiguous" id="footnote2.m1.1.1.2.1.cmml" xref="footnote2.m1.1.1.2">subscript</csymbol><ci id="footnote2.m1.1.1.2.2.cmml" xref="footnote2.m1.1.1.2.2">𝑊</ci><cn type="integer" id="footnote2.m1.1.1.2.3.cmml" xref="footnote2.m1.1.1.2.3">0</cn></apply><cn type="integer" id="footnote2.m1.1.1.3.cmml" xref="footnote2.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote2.m1.1d">W_{0}=0</annotation></semantics></math>. We will discuss more sophisticated techniques for initializing <math id="footnote2.m2.1" class="ltx_Math" alttext="W" display="inline"><semantics id="footnote2.m2.1b"><mi id="footnote2.m2.1.1" xref="footnote2.m2.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="footnote2.m2.1c"><ci id="footnote2.m2.1.1.cmml" xref="footnote2.m2.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote2.m2.1d">W</annotation></semantics></math> in Subsection <a href="#S2.SS7" title="2.7 Implementation details ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.7</span></a>.</span></span></span>
From the compression point of view, every heuristic needs to decide which input to remember or forget. Our <math id="S2.SS1.p3.11.m2.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.SS1.p3.11.m2.1a"><mi id="S2.SS1.p3.11.m2.1.1" xref="S2.SS1.p3.11.m2.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.11.m2.1b"><ci id="S2.SS1.p3.11.m2.1.1.cmml" xref="S2.SS1.p3.11.m2.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.11.m2.1c">W</annotation></semantics></math> remembers inputs that produce large gradients – intuitively, inputs that make <math id="S2.SS1.p3.12.m3.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.SS1.p3.12.m3.1a"><mi id="S2.SS1.p3.12.m3.1.1" xref="S2.SS1.p3.12.m3.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p3.12.m3.1b"><ci id="S2.SS1.p3.12.m3.1.1.cmml" xref="S2.SS1.p3.12.m3.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p3.12.m3.1c">W</annotation></semantics></math> learn a lot.</p>
</div>
<div id="S2.SS1.p4" class="ltx_para">
<p id="S2.SS1.p4.4" class="ltx_p">One choice of <math id="S2.SS1.p4.1.m1.1" class="ltx_Math" alttext="\ell" display="inline"><semantics id="S2.SS1.p4.1.m1.1a"><mi mathvariant="normal" id="S2.SS1.p4.1.m1.1.1" xref="S2.SS1.p4.1.m1.1.1.cmml">ℓ</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.1.m1.1b"><ci id="S2.SS1.p4.1.m1.1.1.cmml" xref="S2.SS1.p4.1.m1.1.1">ℓ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.1.m1.1c">\ell</annotation></semantics></math> is reconstructing <math id="S2.SS1.p4.2.m2.1" class="ltx_Math" alttext="x_{t}" display="inline"><semantics id="S2.SS1.p4.2.m2.1a"><msub id="S2.SS1.p4.2.m2.1.1" xref="S2.SS1.p4.2.m2.1.1.cmml"><mi id="S2.SS1.p4.2.m2.1.1.2" xref="S2.SS1.p4.2.m2.1.1.2.cmml">x</mi><mi id="S2.SS1.p4.2.m2.1.1.3" xref="S2.SS1.p4.2.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.2.m2.1b"><apply id="S2.SS1.p4.2.m2.1.1.cmml" xref="S2.SS1.p4.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p4.2.m2.1.1.1.cmml" xref="S2.SS1.p4.2.m2.1.1">subscript</csymbol><ci id="S2.SS1.p4.2.m2.1.1.2.cmml" xref="S2.SS1.p4.2.m2.1.1.2">𝑥</ci><ci id="S2.SS1.p4.2.m2.1.1.3.cmml" xref="S2.SS1.p4.2.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.2.m2.1c">x_{t}</annotation></semantics></math> itself.
To make the learning problem nontrivial, we first process <math id="S2.SS1.p4.3.m3.1" class="ltx_Math" alttext="x_{t}" display="inline"><semantics id="S2.SS1.p4.3.m3.1a"><msub id="S2.SS1.p4.3.m3.1.1" xref="S2.SS1.p4.3.m3.1.1.cmml"><mi id="S2.SS1.p4.3.m3.1.1.2" xref="S2.SS1.p4.3.m3.1.1.2.cmml">x</mi><mi id="S2.SS1.p4.3.m3.1.1.3" xref="S2.SS1.p4.3.m3.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.3.m3.1b"><apply id="S2.SS1.p4.3.m3.1.1.cmml" xref="S2.SS1.p4.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS1.p4.3.m3.1.1.1.cmml" xref="S2.SS1.p4.3.m3.1.1">subscript</csymbol><ci id="S2.SS1.p4.3.m3.1.1.2.cmml" xref="S2.SS1.p4.3.m3.1.1.2">𝑥</ci><ci id="S2.SS1.p4.3.m3.1.1.3.cmml" xref="S2.SS1.p4.3.m3.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.3.m3.1c">x_{t}</annotation></semantics></math> into a corrupted input <math id="S2.SS1.p4.4.m4.1" class="ltx_Math" alttext="\tilde{x}_{t}" display="inline"><semantics id="S2.SS1.p4.4.m4.1a"><msub id="S2.SS1.p4.4.m4.1.1" xref="S2.SS1.p4.4.m4.1.1.cmml"><mover accent="true" id="S2.SS1.p4.4.m4.1.1.2" xref="S2.SS1.p4.4.m4.1.1.2.cmml"><mi id="S2.SS1.p4.4.m4.1.1.2.2" xref="S2.SS1.p4.4.m4.1.1.2.2.cmml">x</mi><mo id="S2.SS1.p4.4.m4.1.1.2.1" xref="S2.SS1.p4.4.m4.1.1.2.1.cmml">~</mo></mover><mi id="S2.SS1.p4.4.m4.1.1.3" xref="S2.SS1.p4.4.m4.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.4.m4.1b"><apply id="S2.SS1.p4.4.m4.1.1.cmml" xref="S2.SS1.p4.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS1.p4.4.m4.1.1.1.cmml" xref="S2.SS1.p4.4.m4.1.1">subscript</csymbol><apply id="S2.SS1.p4.4.m4.1.1.2.cmml" xref="S2.SS1.p4.4.m4.1.1.2"><ci id="S2.SS1.p4.4.m4.1.1.2.1.cmml" xref="S2.SS1.p4.4.m4.1.1.2.1">~</ci><ci id="S2.SS1.p4.4.m4.1.1.2.2.cmml" xref="S2.SS1.p4.4.m4.1.1.2.2">𝑥</ci></apply><ci id="S2.SS1.p4.4.m4.1.1.3.cmml" xref="S2.SS1.p4.4.m4.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.4.m4.1c">\tilde{x}_{t}</annotation></semantics></math> (details in Subsection <a href="#S2.SS3" title="2.3 Learning a self-supervised task for TTT ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.3</span></a>), then optimize:</p>
<table id="S2.E3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E3.m1.3" class="ltx_Math" alttext="\ell(W;x_{t})=\|f(\tilde{x}_{t};W)-x_{t}\|^{2}." display="block"><semantics id="S2.E3.m1.3a"><mrow id="S2.E3.m1.3.3.1" xref="S2.E3.m1.3.3.1.1.cmml"><mrow id="S2.E3.m1.3.3.1.1" xref="S2.E3.m1.3.3.1.1.cmml"><mrow id="S2.E3.m1.3.3.1.1.1" xref="S2.E3.m1.3.3.1.1.1.cmml"><mi mathvariant="normal" id="S2.E3.m1.3.3.1.1.1.3" xref="S2.E3.m1.3.3.1.1.1.3.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="S2.E3.m1.3.3.1.1.1.2" xref="S2.E3.m1.3.3.1.1.1.2.cmml">​</mo><mrow id="S2.E3.m1.3.3.1.1.1.1.1" xref="S2.E3.m1.3.3.1.1.1.1.2.cmml"><mo stretchy="false" id="S2.E3.m1.3.3.1.1.1.1.1.2" xref="S2.E3.m1.3.3.1.1.1.1.2.cmml">(</mo><mi id="S2.E3.m1.1.1" xref="S2.E3.m1.1.1.cmml">W</mi><mo id="S2.E3.m1.3.3.1.1.1.1.1.3" xref="S2.E3.m1.3.3.1.1.1.1.2.cmml">;</mo><msub id="S2.E3.m1.3.3.1.1.1.1.1.1" xref="S2.E3.m1.3.3.1.1.1.1.1.1.cmml"><mi id="S2.E3.m1.3.3.1.1.1.1.1.1.2" xref="S2.E3.m1.3.3.1.1.1.1.1.1.2.cmml">x</mi><mi id="S2.E3.m1.3.3.1.1.1.1.1.1.3" xref="S2.E3.m1.3.3.1.1.1.1.1.1.3.cmml">t</mi></msub><mo stretchy="false" id="S2.E3.m1.3.3.1.1.1.1.1.4" xref="S2.E3.m1.3.3.1.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="S2.E3.m1.3.3.1.1.3" xref="S2.E3.m1.3.3.1.1.3.cmml">=</mo><msup id="S2.E3.m1.3.3.1.1.2" xref="S2.E3.m1.3.3.1.1.2.cmml"><mrow id="S2.E3.m1.3.3.1.1.2.1.1" xref="S2.E3.m1.3.3.1.1.2.1.2.cmml"><mo stretchy="false" id="S2.E3.m1.3.3.1.1.2.1.1.2" xref="S2.E3.m1.3.3.1.1.2.1.2.1.cmml">‖</mo><mrow id="S2.E3.m1.3.3.1.1.2.1.1.1" xref="S2.E3.m1.3.3.1.1.2.1.1.1.cmml"><mrow id="S2.E3.m1.3.3.1.1.2.1.1.1.1" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1.cmml"><mi id="S2.E3.m1.3.3.1.1.2.1.1.1.1.3" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1.3.cmml">f</mi><mo lspace="0em" rspace="0em" id="S2.E3.m1.3.3.1.1.2.1.1.1.1.2" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1.2.cmml">​</mo><mrow id="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.2" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.2.cmml">(</mo><msub id="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1.cmml"><mover accent="true" id="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1.2" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1.2.cmml"><mi id="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1.2.2" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1.2.2.cmml">x</mi><mo id="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1.2.1" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1.2.1.cmml">~</mo></mover><mi id="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1.3" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1.3.cmml">t</mi></msub><mo id="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.3" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.2.cmml">;</mo><mi id="S2.E3.m1.2.2" xref="S2.E3.m1.2.2.cmml">W</mi><mo stretchy="false" id="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.4" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="S2.E3.m1.3.3.1.1.2.1.1.1.2" xref="S2.E3.m1.3.3.1.1.2.1.1.1.2.cmml">−</mo><msub id="S2.E3.m1.3.3.1.1.2.1.1.1.3" xref="S2.E3.m1.3.3.1.1.2.1.1.1.3.cmml"><mi id="S2.E3.m1.3.3.1.1.2.1.1.1.3.2" xref="S2.E3.m1.3.3.1.1.2.1.1.1.3.2.cmml">x</mi><mi id="S2.E3.m1.3.3.1.1.2.1.1.1.3.3" xref="S2.E3.m1.3.3.1.1.2.1.1.1.3.3.cmml">t</mi></msub></mrow><mo stretchy="false" id="S2.E3.m1.3.3.1.1.2.1.1.3" xref="S2.E3.m1.3.3.1.1.2.1.2.1.cmml">‖</mo></mrow><mn id="S2.E3.m1.3.3.1.1.2.3" xref="S2.E3.m1.3.3.1.1.2.3.cmml">2</mn></msup></mrow><mo lspace="0em" id="S2.E3.m1.3.3.1.2" xref="S2.E3.m1.3.3.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E3.m1.3b"><apply id="S2.E3.m1.3.3.1.1.cmml" xref="S2.E3.m1.3.3.1"><eq id="S2.E3.m1.3.3.1.1.3.cmml" xref="S2.E3.m1.3.3.1.1.3"></eq><apply id="S2.E3.m1.3.3.1.1.1.cmml" xref="S2.E3.m1.3.3.1.1.1"><times id="S2.E3.m1.3.3.1.1.1.2.cmml" xref="S2.E3.m1.3.3.1.1.1.2"></times><ci id="S2.E3.m1.3.3.1.1.1.3.cmml" xref="S2.E3.m1.3.3.1.1.1.3">ℓ</ci><list id="S2.E3.m1.3.3.1.1.1.1.2.cmml" xref="S2.E3.m1.3.3.1.1.1.1.1"><ci id="S2.E3.m1.1.1.cmml" xref="S2.E3.m1.1.1">𝑊</ci><apply id="S2.E3.m1.3.3.1.1.1.1.1.1.cmml" xref="S2.E3.m1.3.3.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E3.m1.3.3.1.1.1.1.1.1.1.cmml" xref="S2.E3.m1.3.3.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E3.m1.3.3.1.1.1.1.1.1.2.cmml" xref="S2.E3.m1.3.3.1.1.1.1.1.1.2">𝑥</ci><ci id="S2.E3.m1.3.3.1.1.1.1.1.1.3.cmml" xref="S2.E3.m1.3.3.1.1.1.1.1.1.3">𝑡</ci></apply></list></apply><apply id="S2.E3.m1.3.3.1.1.2.cmml" xref="S2.E3.m1.3.3.1.1.2"><csymbol cd="ambiguous" id="S2.E3.m1.3.3.1.1.2.2.cmml" xref="S2.E3.m1.3.3.1.1.2">superscript</csymbol><apply id="S2.E3.m1.3.3.1.1.2.1.2.cmml" xref="S2.E3.m1.3.3.1.1.2.1.1"><csymbol cd="latexml" id="S2.E3.m1.3.3.1.1.2.1.2.1.cmml" xref="S2.E3.m1.3.3.1.1.2.1.1.2">norm</csymbol><apply id="S2.E3.m1.3.3.1.1.2.1.1.1.cmml" xref="S2.E3.m1.3.3.1.1.2.1.1.1"><minus id="S2.E3.m1.3.3.1.1.2.1.1.1.2.cmml" xref="S2.E3.m1.3.3.1.1.2.1.1.1.2"></minus><apply id="S2.E3.m1.3.3.1.1.2.1.1.1.1.cmml" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1"><times id="S2.E3.m1.3.3.1.1.2.1.1.1.1.2.cmml" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1.2"></times><ci id="S2.E3.m1.3.3.1.1.2.1.1.1.1.3.cmml" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1.3">𝑓</ci><list id="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.2.cmml" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1"><apply id="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1.cmml" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1.1.cmml" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1">subscript</csymbol><apply id="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1.2.cmml" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1.2"><ci id="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1.2.1.cmml" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1.2.1">~</ci><ci id="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1.2.2.cmml" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1.2.2">𝑥</ci></apply><ci id="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1.3.cmml" xref="S2.E3.m1.3.3.1.1.2.1.1.1.1.1.1.1.3">𝑡</ci></apply><ci id="S2.E3.m1.2.2.cmml" xref="S2.E3.m1.2.2">𝑊</ci></list></apply><apply id="S2.E3.m1.3.3.1.1.2.1.1.1.3.cmml" xref="S2.E3.m1.3.3.1.1.2.1.1.1.3"><csymbol cd="ambiguous" id="S2.E3.m1.3.3.1.1.2.1.1.1.3.1.cmml" xref="S2.E3.m1.3.3.1.1.2.1.1.1.3">subscript</csymbol><ci id="S2.E3.m1.3.3.1.1.2.1.1.1.3.2.cmml" xref="S2.E3.m1.3.3.1.1.2.1.1.1.3.2">𝑥</ci><ci id="S2.E3.m1.3.3.1.1.2.1.1.1.3.3.cmml" xref="S2.E3.m1.3.3.1.1.2.1.1.1.3.3">𝑡</ci></apply></apply></apply><cn type="integer" id="S2.E3.m1.3.3.1.1.2.3.cmml" xref="S2.E3.m1.3.3.1.1.2.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E3.m1.3c">\ell(W;x_{t})=\|f(\tilde{x}_{t};W)-x_{t}\|^{2}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
<p id="S2.SS1.p4.8" class="ltx_p">Similar to denoising autoencoders <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib75" title="" class="ltx_ref">75</a>]</cite>, <math id="S2.SS1.p4.5.m1.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S2.SS1.p4.5.m1.1a"><mi id="S2.SS1.p4.5.m1.1.1" xref="S2.SS1.p4.5.m1.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.5.m1.1b"><ci id="S2.SS1.p4.5.m1.1.1.cmml" xref="S2.SS1.p4.5.m1.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.5.m1.1c">f</annotation></semantics></math> needs to discover the correlations between dimensions of <math id="S2.SS1.p4.6.m2.1" class="ltx_Math" alttext="x_{t}" display="inline"><semantics id="S2.SS1.p4.6.m2.1a"><msub id="S2.SS1.p4.6.m2.1.1" xref="S2.SS1.p4.6.m2.1.1.cmml"><mi id="S2.SS1.p4.6.m2.1.1.2" xref="S2.SS1.p4.6.m2.1.1.2.cmml">x</mi><mi id="S2.SS1.p4.6.m2.1.1.3" xref="S2.SS1.p4.6.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.6.m2.1b"><apply id="S2.SS1.p4.6.m2.1.1.cmml" xref="S2.SS1.p4.6.m2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p4.6.m2.1.1.1.cmml" xref="S2.SS1.p4.6.m2.1.1">subscript</csymbol><ci id="S2.SS1.p4.6.m2.1.1.2.cmml" xref="S2.SS1.p4.6.m2.1.1.2">𝑥</ci><ci id="S2.SS1.p4.6.m2.1.1.3.cmml" xref="S2.SS1.p4.6.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.6.m2.1c">x_{t}</annotation></semantics></math> in order to reconstruct it from partial information <math id="S2.SS1.p4.7.m3.1" class="ltx_Math" alttext="\tilde{x}_{t}" display="inline"><semantics id="S2.SS1.p4.7.m3.1a"><msub id="S2.SS1.p4.7.m3.1.1" xref="S2.SS1.p4.7.m3.1.1.cmml"><mover accent="true" id="S2.SS1.p4.7.m3.1.1.2" xref="S2.SS1.p4.7.m3.1.1.2.cmml"><mi id="S2.SS1.p4.7.m3.1.1.2.2" xref="S2.SS1.p4.7.m3.1.1.2.2.cmml">x</mi><mo id="S2.SS1.p4.7.m3.1.1.2.1" xref="S2.SS1.p4.7.m3.1.1.2.1.cmml">~</mo></mover><mi id="S2.SS1.p4.7.m3.1.1.3" xref="S2.SS1.p4.7.m3.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.7.m3.1b"><apply id="S2.SS1.p4.7.m3.1.1.cmml" xref="S2.SS1.p4.7.m3.1.1"><csymbol cd="ambiguous" id="S2.SS1.p4.7.m3.1.1.1.cmml" xref="S2.SS1.p4.7.m3.1.1">subscript</csymbol><apply id="S2.SS1.p4.7.m3.1.1.2.cmml" xref="S2.SS1.p4.7.m3.1.1.2"><ci id="S2.SS1.p4.7.m3.1.1.2.1.cmml" xref="S2.SS1.p4.7.m3.1.1.2.1">~</ci><ci id="S2.SS1.p4.7.m3.1.1.2.2.cmml" xref="S2.SS1.p4.7.m3.1.1.2.2">𝑥</ci></apply><ci id="S2.SS1.p4.7.m3.1.1.3.cmml" xref="S2.SS1.p4.7.m3.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.7.m3.1c">\tilde{x}_{t}</annotation></semantics></math>.<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span>
In past experiments, we have also tried adding another model <math id="footnote3.m1.1" class="ltx_Math" alttext="g" display="inline"><semantics id="footnote3.m1.1b"><mi id="footnote3.m1.1.1" xref="footnote3.m1.1.1.cmml">g</mi><annotation-xml encoding="MathML-Content" id="footnote3.m1.1c"><ci id="footnote3.m1.1.1.cmml" xref="footnote3.m1.1.1">𝑔</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote3.m1.1d">g</annotation></semantics></math> (decoder) after <math id="footnote3.m2.1" class="ltx_Math" alttext="f" display="inline"><semantics id="footnote3.m2.1b"><mi id="footnote3.m2.1.1" xref="footnote3.m2.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="footnote3.m2.1c"><ci id="footnote3.m2.1.1.cmml" xref="footnote3.m2.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote3.m2.1d">f</annotation></semantics></math> (encoder), such that the reconstruction is produced by <math id="footnote3.m3.1" class="ltx_Math" alttext="g\circ f" display="inline"><semantics id="footnote3.m3.1b"><mrow id="footnote3.m3.1.1" xref="footnote3.m3.1.1.cmml"><mi id="footnote3.m3.1.1.2" xref="footnote3.m3.1.1.2.cmml">g</mi><mo lspace="0.222em" rspace="0.222em" id="footnote3.m3.1.1.1" xref="footnote3.m3.1.1.1.cmml">∘</mo><mi id="footnote3.m3.1.1.3" xref="footnote3.m3.1.1.3.cmml">f</mi></mrow><annotation-xml encoding="MathML-Content" id="footnote3.m3.1c"><apply id="footnote3.m3.1.1.cmml" xref="footnote3.m3.1.1"><compose id="footnote3.m3.1.1.1.cmml" xref="footnote3.m3.1.1.1"></compose><ci id="footnote3.m3.1.1.2.cmml" xref="footnote3.m3.1.1.2">𝑔</ci><ci id="footnote3.m3.1.1.3.cmml" xref="footnote3.m3.1.1.3">𝑓</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote3.m3.1d">g\circ f</annotation></semantics></math> instead of only <math id="footnote3.m4.1" class="ltx_Math" alttext="f" display="inline"><semantics id="footnote3.m4.1b"><mi id="footnote3.m4.1.1" xref="footnote3.m4.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="footnote3.m4.1c"><ci id="footnote3.m4.1.1.cmml" xref="footnote3.m4.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote3.m4.1d">f</annotation></semantics></math> itself.
While this heftier design did slightly improve results, it made overall training less stable and added significant computational cost.
Therefore we focus on the encoder-only design.
</span></span></span>
As shown in Figure <a href="#S2.F5" title="Figure 5 ‣ 2.1 TTT as updating a hidden state ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, gradient descent is able to reduce <math id="S2.SS1.p4.8.m4.1" class="ltx_Math" alttext="\ell" display="inline"><semantics id="S2.SS1.p4.8.m4.1a"><mi mathvariant="normal" id="S2.SS1.p4.8.m4.1.1" xref="S2.SS1.p4.8.m4.1.1.cmml">ℓ</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p4.8.m4.1b"><ci id="S2.SS1.p4.8.m4.1.1.cmml" xref="S2.SS1.p4.8.m4.1.1">ℓ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p4.8.m4.1c">\ell</annotation></semantics></math>, but cannot reduce it to zero.
We discuss more sophisticated formulations of the self-supervised task in Subsection <a href="#S2.SS3" title="2.3 Learning a self-supervised task for TTT ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.3</span></a>.</p>
</div>
<div id="S2.SS1.p5" class="ltx_para">
<p id="S2.SS1.p5.3" class="ltx_p">As with other RNN layers and self-attention, our algorithm that maps an input sequence <math id="S2.SS1.p5.1.m1.3" class="ltx_Math" alttext="x_{1},\dots,x_{T}" display="inline"><semantics id="S2.SS1.p5.1.m1.3a"><mrow id="S2.SS1.p5.1.m1.3.3.2" xref="S2.SS1.p5.1.m1.3.3.3.cmml"><msub id="S2.SS1.p5.1.m1.2.2.1.1" xref="S2.SS1.p5.1.m1.2.2.1.1.cmml"><mi id="S2.SS1.p5.1.m1.2.2.1.1.2" xref="S2.SS1.p5.1.m1.2.2.1.1.2.cmml">x</mi><mn id="S2.SS1.p5.1.m1.2.2.1.1.3" xref="S2.SS1.p5.1.m1.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.SS1.p5.1.m1.3.3.2.3" xref="S2.SS1.p5.1.m1.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS1.p5.1.m1.1.1" xref="S2.SS1.p5.1.m1.1.1.cmml">…</mi><mo id="S2.SS1.p5.1.m1.3.3.2.4" xref="S2.SS1.p5.1.m1.3.3.3.cmml">,</mo><msub id="S2.SS1.p5.1.m1.3.3.2.2" xref="S2.SS1.p5.1.m1.3.3.2.2.cmml"><mi id="S2.SS1.p5.1.m1.3.3.2.2.2" xref="S2.SS1.p5.1.m1.3.3.2.2.2.cmml">x</mi><mi id="S2.SS1.p5.1.m1.3.3.2.2.3" xref="S2.SS1.p5.1.m1.3.3.2.2.3.cmml">T</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p5.1.m1.3b"><list id="S2.SS1.p5.1.m1.3.3.3.cmml" xref="S2.SS1.p5.1.m1.3.3.2"><apply id="S2.SS1.p5.1.m1.2.2.1.1.cmml" xref="S2.SS1.p5.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p5.1.m1.2.2.1.1.1.cmml" xref="S2.SS1.p5.1.m1.2.2.1.1">subscript</csymbol><ci id="S2.SS1.p5.1.m1.2.2.1.1.2.cmml" xref="S2.SS1.p5.1.m1.2.2.1.1.2">𝑥</ci><cn type="integer" id="S2.SS1.p5.1.m1.2.2.1.1.3.cmml" xref="S2.SS1.p5.1.m1.2.2.1.1.3">1</cn></apply><ci id="S2.SS1.p5.1.m1.1.1.cmml" xref="S2.SS1.p5.1.m1.1.1">…</ci><apply id="S2.SS1.p5.1.m1.3.3.2.2.cmml" xref="S2.SS1.p5.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S2.SS1.p5.1.m1.3.3.2.2.1.cmml" xref="S2.SS1.p5.1.m1.3.3.2.2">subscript</csymbol><ci id="S2.SS1.p5.1.m1.3.3.2.2.2.cmml" xref="S2.SS1.p5.1.m1.3.3.2.2.2">𝑥</ci><ci id="S2.SS1.p5.1.m1.3.3.2.2.3.cmml" xref="S2.SS1.p5.1.m1.3.3.2.2.3">𝑇</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p5.1.m1.3c">x_{1},\dots,x_{T}</annotation></semantics></math> to output sequence <math id="S2.SS1.p5.2.m2.3" class="ltx_Math" alttext="z_{1},\dots,z_{T}" display="inline"><semantics id="S2.SS1.p5.2.m2.3a"><mrow id="S2.SS1.p5.2.m2.3.3.2" xref="S2.SS1.p5.2.m2.3.3.3.cmml"><msub id="S2.SS1.p5.2.m2.2.2.1.1" xref="S2.SS1.p5.2.m2.2.2.1.1.cmml"><mi id="S2.SS1.p5.2.m2.2.2.1.1.2" xref="S2.SS1.p5.2.m2.2.2.1.1.2.cmml">z</mi><mn id="S2.SS1.p5.2.m2.2.2.1.1.3" xref="S2.SS1.p5.2.m2.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.SS1.p5.2.m2.3.3.2.3" xref="S2.SS1.p5.2.m2.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS1.p5.2.m2.1.1" xref="S2.SS1.p5.2.m2.1.1.cmml">…</mi><mo id="S2.SS1.p5.2.m2.3.3.2.4" xref="S2.SS1.p5.2.m2.3.3.3.cmml">,</mo><msub id="S2.SS1.p5.2.m2.3.3.2.2" xref="S2.SS1.p5.2.m2.3.3.2.2.cmml"><mi id="S2.SS1.p5.2.m2.3.3.2.2.2" xref="S2.SS1.p5.2.m2.3.3.2.2.2.cmml">z</mi><mi id="S2.SS1.p5.2.m2.3.3.2.2.3" xref="S2.SS1.p5.2.m2.3.3.2.2.3.cmml">T</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p5.2.m2.3b"><list id="S2.SS1.p5.2.m2.3.3.3.cmml" xref="S2.SS1.p5.2.m2.3.3.2"><apply id="S2.SS1.p5.2.m2.2.2.1.1.cmml" xref="S2.SS1.p5.2.m2.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p5.2.m2.2.2.1.1.1.cmml" xref="S2.SS1.p5.2.m2.2.2.1.1">subscript</csymbol><ci id="S2.SS1.p5.2.m2.2.2.1.1.2.cmml" xref="S2.SS1.p5.2.m2.2.2.1.1.2">𝑧</ci><cn type="integer" id="S2.SS1.p5.2.m2.2.2.1.1.3.cmml" xref="S2.SS1.p5.2.m2.2.2.1.1.3">1</cn></apply><ci id="S2.SS1.p5.2.m2.1.1.cmml" xref="S2.SS1.p5.2.m2.1.1">…</ci><apply id="S2.SS1.p5.2.m2.3.3.2.2.cmml" xref="S2.SS1.p5.2.m2.3.3.2.2"><csymbol cd="ambiguous" id="S2.SS1.p5.2.m2.3.3.2.2.1.cmml" xref="S2.SS1.p5.2.m2.3.3.2.2">subscript</csymbol><ci id="S2.SS1.p5.2.m2.3.3.2.2.2.cmml" xref="S2.SS1.p5.2.m2.3.3.2.2.2">𝑧</ci><ci id="S2.SS1.p5.2.m2.3.3.2.2.3.cmml" xref="S2.SS1.p5.2.m2.3.3.2.2.3">𝑇</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p5.2.m2.3c">z_{1},\dots,z_{T}</annotation></semantics></math> can be programmed into the forward pass of a sequence modeling layer, using the hidden state, update rule, and output rule above.
Even at test time, our new layer still trains a different sequence of weights <math id="S2.SS1.p5.3.m3.3" class="ltx_Math" alttext="W_{1},\dots,W_{T}" display="inline"><semantics id="S2.SS1.p5.3.m3.3a"><mrow id="S2.SS1.p5.3.m3.3.3.2" xref="S2.SS1.p5.3.m3.3.3.3.cmml"><msub id="S2.SS1.p5.3.m3.2.2.1.1" xref="S2.SS1.p5.3.m3.2.2.1.1.cmml"><mi id="S2.SS1.p5.3.m3.2.2.1.1.2" xref="S2.SS1.p5.3.m3.2.2.1.1.2.cmml">W</mi><mn id="S2.SS1.p5.3.m3.2.2.1.1.3" xref="S2.SS1.p5.3.m3.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.SS1.p5.3.m3.3.3.2.3" xref="S2.SS1.p5.3.m3.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS1.p5.3.m3.1.1" xref="S2.SS1.p5.3.m3.1.1.cmml">…</mi><mo id="S2.SS1.p5.3.m3.3.3.2.4" xref="S2.SS1.p5.3.m3.3.3.3.cmml">,</mo><msub id="S2.SS1.p5.3.m3.3.3.2.2" xref="S2.SS1.p5.3.m3.3.3.2.2.cmml"><mi id="S2.SS1.p5.3.m3.3.3.2.2.2" xref="S2.SS1.p5.3.m3.3.3.2.2.2.cmml">W</mi><mi id="S2.SS1.p5.3.m3.3.3.2.2.3" xref="S2.SS1.p5.3.m3.3.3.2.2.3.cmml">T</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p5.3.m3.3b"><list id="S2.SS1.p5.3.m3.3.3.3.cmml" xref="S2.SS1.p5.3.m3.3.3.2"><apply id="S2.SS1.p5.3.m3.2.2.1.1.cmml" xref="S2.SS1.p5.3.m3.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p5.3.m3.2.2.1.1.1.cmml" xref="S2.SS1.p5.3.m3.2.2.1.1">subscript</csymbol><ci id="S2.SS1.p5.3.m3.2.2.1.1.2.cmml" xref="S2.SS1.p5.3.m3.2.2.1.1.2">𝑊</ci><cn type="integer" id="S2.SS1.p5.3.m3.2.2.1.1.3.cmml" xref="S2.SS1.p5.3.m3.2.2.1.1.3">1</cn></apply><ci id="S2.SS1.p5.3.m3.1.1.cmml" xref="S2.SS1.p5.3.m3.1.1">…</ci><apply id="S2.SS1.p5.3.m3.3.3.2.2.cmml" xref="S2.SS1.p5.3.m3.3.3.2.2"><csymbol cd="ambiguous" id="S2.SS1.p5.3.m3.3.3.2.2.1.cmml" xref="S2.SS1.p5.3.m3.3.3.2.2">subscript</csymbol><ci id="S2.SS1.p5.3.m3.3.3.2.2.2.cmml" xref="S2.SS1.p5.3.m3.3.3.2.2.2">𝑊</ci><ci id="S2.SS1.p5.3.m3.3.3.2.2.3.cmml" xref="S2.SS1.p5.3.m3.3.3.2.2.3">𝑇</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p5.3.m3.3c">W_{1},\dots,W_{T}</annotation></semantics></math> for every input sequence. Therefore, we call it the <em id="S2.SS1.p5.3.1" class="ltx_emph ltx_font_italic">Test-Time Training (TTT) layer</em>.</p>
</div>
<figure id="S2.F5" class="ltx_figure"><img src="/html/2407.04620/assets/x6.png" id="S2.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="115" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>The self-supervised TTT loss <math id="S2.F5.9.m1.1" class="ltx_Math" alttext="\ell" display="inline"><semantics id="S2.F5.9.m1.1b"><mi mathvariant="normal" id="S2.F5.9.m1.1.1" xref="S2.F5.9.m1.1.1.cmml">ℓ</mi><annotation-xml encoding="MathML-Content" id="S2.F5.9.m1.1c"><ci id="S2.F5.9.m1.1.1.cmml" xref="S2.F5.9.m1.1.1">ℓ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.F5.9.m1.1d">\ell</annotation></semantics></math> averaged over all test sequences of the form <math id="S2.F5.10.m2.3" class="ltx_Math" alttext="x_{1},\dots,x_{T}" display="inline"><semantics id="S2.F5.10.m2.3b"><mrow id="S2.F5.10.m2.3.3.2" xref="S2.F5.10.m2.3.3.3.cmml"><msub id="S2.F5.10.m2.2.2.1.1" xref="S2.F5.10.m2.2.2.1.1.cmml"><mi id="S2.F5.10.m2.2.2.1.1.2" xref="S2.F5.10.m2.2.2.1.1.2.cmml">x</mi><mn id="S2.F5.10.m2.2.2.1.1.3" xref="S2.F5.10.m2.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.F5.10.m2.3.3.2.3" xref="S2.F5.10.m2.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.F5.10.m2.1.1" xref="S2.F5.10.m2.1.1.cmml">…</mi><mo id="S2.F5.10.m2.3.3.2.4" xref="S2.F5.10.m2.3.3.3.cmml">,</mo><msub id="S2.F5.10.m2.3.3.2.2" xref="S2.F5.10.m2.3.3.2.2.cmml"><mi id="S2.F5.10.m2.3.3.2.2.2" xref="S2.F5.10.m2.3.3.2.2.2.cmml">x</mi><mi id="S2.F5.10.m2.3.3.2.2.3" xref="S2.F5.10.m2.3.3.2.2.3.cmml">T</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.F5.10.m2.3c"><list id="S2.F5.10.m2.3.3.3.cmml" xref="S2.F5.10.m2.3.3.2"><apply id="S2.F5.10.m2.2.2.1.1.cmml" xref="S2.F5.10.m2.2.2.1.1"><csymbol cd="ambiguous" id="S2.F5.10.m2.2.2.1.1.1.cmml" xref="S2.F5.10.m2.2.2.1.1">subscript</csymbol><ci id="S2.F5.10.m2.2.2.1.1.2.cmml" xref="S2.F5.10.m2.2.2.1.1.2">𝑥</ci><cn type="integer" id="S2.F5.10.m2.2.2.1.1.3.cmml" xref="S2.F5.10.m2.2.2.1.1.3">1</cn></apply><ci id="S2.F5.10.m2.1.1.cmml" xref="S2.F5.10.m2.1.1">…</ci><apply id="S2.F5.10.m2.3.3.2.2.cmml" xref="S2.F5.10.m2.3.3.2.2"><csymbol cd="ambiguous" id="S2.F5.10.m2.3.3.2.2.1.cmml" xref="S2.F5.10.m2.3.3.2.2">subscript</csymbol><ci id="S2.F5.10.m2.3.3.2.2.2.cmml" xref="S2.F5.10.m2.3.3.2.2.2">𝑥</ci><ci id="S2.F5.10.m2.3.3.2.2.3.cmml" xref="S2.F5.10.m2.3.3.2.2.3">𝑇</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.F5.10.m2.3d">x_{1},\dots,x_{T}</annotation></semantics></math> where <math id="S2.F5.11.m3.1" class="ltx_Math" alttext="T=2048" display="inline"><semantics id="S2.F5.11.m3.1b"><mrow id="S2.F5.11.m3.1.1" xref="S2.F5.11.m3.1.1.cmml"><mi id="S2.F5.11.m3.1.1.2" xref="S2.F5.11.m3.1.1.2.cmml">T</mi><mo id="S2.F5.11.m3.1.1.1" xref="S2.F5.11.m3.1.1.1.cmml">=</mo><mn id="S2.F5.11.m3.1.1.3" xref="S2.F5.11.m3.1.1.3.cmml">2048</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.F5.11.m3.1c"><apply id="S2.F5.11.m3.1.1.cmml" xref="S2.F5.11.m3.1.1"><eq id="S2.F5.11.m3.1.1.1.cmml" xref="S2.F5.11.m3.1.1.1"></eq><ci id="S2.F5.11.m3.1.1.2.cmml" xref="S2.F5.11.m3.1.1.2">𝑇</ci><cn type="integer" id="S2.F5.11.m3.1.1.3.cmml" xref="S2.F5.11.m3.1.1.3">2048</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F5.11.m3.1d">T=2048</annotation></semantics></math>, for the first three TTT layers in a network with 125M parameters.
One step of gradient descent is able to reduce TTT loss from <math id="S2.F5.12.m4.2" class="ltx_Math" alttext="\ell(W_{t-1};x_{t})" display="inline"><semantics id="S2.F5.12.m4.2b"><mrow id="S2.F5.12.m4.2.2" xref="S2.F5.12.m4.2.2.cmml"><mi mathvariant="normal" id="S2.F5.12.m4.2.2.4" xref="S2.F5.12.m4.2.2.4.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="S2.F5.12.m4.2.2.3" xref="S2.F5.12.m4.2.2.3.cmml">​</mo><mrow id="S2.F5.12.m4.2.2.2.2" xref="S2.F5.12.m4.2.2.2.3.cmml"><mo stretchy="false" id="S2.F5.12.m4.2.2.2.2.3" xref="S2.F5.12.m4.2.2.2.3.cmml">(</mo><msub id="S2.F5.12.m4.1.1.1.1.1" xref="S2.F5.12.m4.1.1.1.1.1.cmml"><mi id="S2.F5.12.m4.1.1.1.1.1.2" xref="S2.F5.12.m4.1.1.1.1.1.2.cmml">W</mi><mrow id="S2.F5.12.m4.1.1.1.1.1.3" xref="S2.F5.12.m4.1.1.1.1.1.3.cmml"><mi id="S2.F5.12.m4.1.1.1.1.1.3.2" xref="S2.F5.12.m4.1.1.1.1.1.3.2.cmml">t</mi><mo id="S2.F5.12.m4.1.1.1.1.1.3.1" xref="S2.F5.12.m4.1.1.1.1.1.3.1.cmml">−</mo><mn id="S2.F5.12.m4.1.1.1.1.1.3.3" xref="S2.F5.12.m4.1.1.1.1.1.3.3.cmml">1</mn></mrow></msub><mo id="S2.F5.12.m4.2.2.2.2.4" xref="S2.F5.12.m4.2.2.2.3.cmml">;</mo><msub id="S2.F5.12.m4.2.2.2.2.2" xref="S2.F5.12.m4.2.2.2.2.2.cmml"><mi id="S2.F5.12.m4.2.2.2.2.2.2" xref="S2.F5.12.m4.2.2.2.2.2.2.cmml">x</mi><mi id="S2.F5.12.m4.2.2.2.2.2.3" xref="S2.F5.12.m4.2.2.2.2.2.3.cmml">t</mi></msub><mo stretchy="false" id="S2.F5.12.m4.2.2.2.2.5" xref="S2.F5.12.m4.2.2.2.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.F5.12.m4.2c"><apply id="S2.F5.12.m4.2.2.cmml" xref="S2.F5.12.m4.2.2"><times id="S2.F5.12.m4.2.2.3.cmml" xref="S2.F5.12.m4.2.2.3"></times><ci id="S2.F5.12.m4.2.2.4.cmml" xref="S2.F5.12.m4.2.2.4">ℓ</ci><list id="S2.F5.12.m4.2.2.2.3.cmml" xref="S2.F5.12.m4.2.2.2.2"><apply id="S2.F5.12.m4.1.1.1.1.1.cmml" xref="S2.F5.12.m4.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.F5.12.m4.1.1.1.1.1.1.cmml" xref="S2.F5.12.m4.1.1.1.1.1">subscript</csymbol><ci id="S2.F5.12.m4.1.1.1.1.1.2.cmml" xref="S2.F5.12.m4.1.1.1.1.1.2">𝑊</ci><apply id="S2.F5.12.m4.1.1.1.1.1.3.cmml" xref="S2.F5.12.m4.1.1.1.1.1.3"><minus id="S2.F5.12.m4.1.1.1.1.1.3.1.cmml" xref="S2.F5.12.m4.1.1.1.1.1.3.1"></minus><ci id="S2.F5.12.m4.1.1.1.1.1.3.2.cmml" xref="S2.F5.12.m4.1.1.1.1.1.3.2">𝑡</ci><cn type="integer" id="S2.F5.12.m4.1.1.1.1.1.3.3.cmml" xref="S2.F5.12.m4.1.1.1.1.1.3.3">1</cn></apply></apply><apply id="S2.F5.12.m4.2.2.2.2.2.cmml" xref="S2.F5.12.m4.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.F5.12.m4.2.2.2.2.2.1.cmml" xref="S2.F5.12.m4.2.2.2.2.2">subscript</csymbol><ci id="S2.F5.12.m4.2.2.2.2.2.2.cmml" xref="S2.F5.12.m4.2.2.2.2.2.2">𝑥</ci><ci id="S2.F5.12.m4.2.2.2.2.2.3.cmml" xref="S2.F5.12.m4.2.2.2.2.2.3">𝑡</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F5.12.m4.2d">\ell(W_{t-1};x_{t})</annotation></semantics></math> to <math id="S2.F5.13.m5.2" class="ltx_Math" alttext="\ell(W_{t};x_{t})" display="inline"><semantics id="S2.F5.13.m5.2b"><mrow id="S2.F5.13.m5.2.2" xref="S2.F5.13.m5.2.2.cmml"><mi mathvariant="normal" id="S2.F5.13.m5.2.2.4" xref="S2.F5.13.m5.2.2.4.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="S2.F5.13.m5.2.2.3" xref="S2.F5.13.m5.2.2.3.cmml">​</mo><mrow id="S2.F5.13.m5.2.2.2.2" xref="S2.F5.13.m5.2.2.2.3.cmml"><mo stretchy="false" id="S2.F5.13.m5.2.2.2.2.3" xref="S2.F5.13.m5.2.2.2.3.cmml">(</mo><msub id="S2.F5.13.m5.1.1.1.1.1" xref="S2.F5.13.m5.1.1.1.1.1.cmml"><mi id="S2.F5.13.m5.1.1.1.1.1.2" xref="S2.F5.13.m5.1.1.1.1.1.2.cmml">W</mi><mi id="S2.F5.13.m5.1.1.1.1.1.3" xref="S2.F5.13.m5.1.1.1.1.1.3.cmml">t</mi></msub><mo id="S2.F5.13.m5.2.2.2.2.4" xref="S2.F5.13.m5.2.2.2.3.cmml">;</mo><msub id="S2.F5.13.m5.2.2.2.2.2" xref="S2.F5.13.m5.2.2.2.2.2.cmml"><mi id="S2.F5.13.m5.2.2.2.2.2.2" xref="S2.F5.13.m5.2.2.2.2.2.2.cmml">x</mi><mi id="S2.F5.13.m5.2.2.2.2.2.3" xref="S2.F5.13.m5.2.2.2.2.2.3.cmml">t</mi></msub><mo stretchy="false" id="S2.F5.13.m5.2.2.2.2.5" xref="S2.F5.13.m5.2.2.2.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.F5.13.m5.2c"><apply id="S2.F5.13.m5.2.2.cmml" xref="S2.F5.13.m5.2.2"><times id="S2.F5.13.m5.2.2.3.cmml" xref="S2.F5.13.m5.2.2.3"></times><ci id="S2.F5.13.m5.2.2.4.cmml" xref="S2.F5.13.m5.2.2.4">ℓ</ci><list id="S2.F5.13.m5.2.2.2.3.cmml" xref="S2.F5.13.m5.2.2.2.2"><apply id="S2.F5.13.m5.1.1.1.1.1.cmml" xref="S2.F5.13.m5.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.F5.13.m5.1.1.1.1.1.1.cmml" xref="S2.F5.13.m5.1.1.1.1.1">subscript</csymbol><ci id="S2.F5.13.m5.1.1.1.1.1.2.cmml" xref="S2.F5.13.m5.1.1.1.1.1.2">𝑊</ci><ci id="S2.F5.13.m5.1.1.1.1.1.3.cmml" xref="S2.F5.13.m5.1.1.1.1.1.3">𝑡</ci></apply><apply id="S2.F5.13.m5.2.2.2.2.2.cmml" xref="S2.F5.13.m5.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.F5.13.m5.2.2.2.2.2.1.cmml" xref="S2.F5.13.m5.2.2.2.2.2">subscript</csymbol><ci id="S2.F5.13.m5.2.2.2.2.2.2.cmml" xref="S2.F5.13.m5.2.2.2.2.2.2">𝑥</ci><ci id="S2.F5.13.m5.2.2.2.2.2.3.cmml" xref="S2.F5.13.m5.2.2.2.2.2.3">𝑡</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F5.13.m5.2d">\ell(W_{t};x_{t})</annotation></semantics></math>.
As <math id="S2.F5.14.m6.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S2.F5.14.m6.1b"><mi id="S2.F5.14.m6.1.1" xref="S2.F5.14.m6.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S2.F5.14.m6.1c"><ci id="S2.F5.14.m6.1.1.cmml" xref="S2.F5.14.m6.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.F5.14.m6.1d">t</annotation></semantics></math> moves further along the test sequence, <math id="S2.F5.15.m7.2" class="ltx_Math" alttext="\ell(W_{t};x_{t})" display="inline"><semantics id="S2.F5.15.m7.2b"><mrow id="S2.F5.15.m7.2.2" xref="S2.F5.15.m7.2.2.cmml"><mi mathvariant="normal" id="S2.F5.15.m7.2.2.4" xref="S2.F5.15.m7.2.2.4.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="S2.F5.15.m7.2.2.3" xref="S2.F5.15.m7.2.2.3.cmml">​</mo><mrow id="S2.F5.15.m7.2.2.2.2" xref="S2.F5.15.m7.2.2.2.3.cmml"><mo stretchy="false" id="S2.F5.15.m7.2.2.2.2.3" xref="S2.F5.15.m7.2.2.2.3.cmml">(</mo><msub id="S2.F5.15.m7.1.1.1.1.1" xref="S2.F5.15.m7.1.1.1.1.1.cmml"><mi id="S2.F5.15.m7.1.1.1.1.1.2" xref="S2.F5.15.m7.1.1.1.1.1.2.cmml">W</mi><mi id="S2.F5.15.m7.1.1.1.1.1.3" xref="S2.F5.15.m7.1.1.1.1.1.3.cmml">t</mi></msub><mo id="S2.F5.15.m7.2.2.2.2.4" xref="S2.F5.15.m7.2.2.2.3.cmml">;</mo><msub id="S2.F5.15.m7.2.2.2.2.2" xref="S2.F5.15.m7.2.2.2.2.2.cmml"><mi id="S2.F5.15.m7.2.2.2.2.2.2" xref="S2.F5.15.m7.2.2.2.2.2.2.cmml">x</mi><mi id="S2.F5.15.m7.2.2.2.2.2.3" xref="S2.F5.15.m7.2.2.2.2.2.3.cmml">t</mi></msub><mo stretchy="false" id="S2.F5.15.m7.2.2.2.2.5" xref="S2.F5.15.m7.2.2.2.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.F5.15.m7.2c"><apply id="S2.F5.15.m7.2.2.cmml" xref="S2.F5.15.m7.2.2"><times id="S2.F5.15.m7.2.2.3.cmml" xref="S2.F5.15.m7.2.2.3"></times><ci id="S2.F5.15.m7.2.2.4.cmml" xref="S2.F5.15.m7.2.2.4">ℓ</ci><list id="S2.F5.15.m7.2.2.2.3.cmml" xref="S2.F5.15.m7.2.2.2.2"><apply id="S2.F5.15.m7.1.1.1.1.1.cmml" xref="S2.F5.15.m7.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.F5.15.m7.1.1.1.1.1.1.cmml" xref="S2.F5.15.m7.1.1.1.1.1">subscript</csymbol><ci id="S2.F5.15.m7.1.1.1.1.1.2.cmml" xref="S2.F5.15.m7.1.1.1.1.1.2">𝑊</ci><ci id="S2.F5.15.m7.1.1.1.1.1.3.cmml" xref="S2.F5.15.m7.1.1.1.1.1.3">𝑡</ci></apply><apply id="S2.F5.15.m7.2.2.2.2.2.cmml" xref="S2.F5.15.m7.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.F5.15.m7.2.2.2.2.2.1.cmml" xref="S2.F5.15.m7.2.2.2.2.2">subscript</csymbol><ci id="S2.F5.15.m7.2.2.2.2.2.2.cmml" xref="S2.F5.15.m7.2.2.2.2.2.2">𝑥</ci><ci id="S2.F5.15.m7.2.2.2.2.2.3.cmml" xref="S2.F5.15.m7.2.2.2.2.2.3">𝑡</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F5.15.m7.2d">\ell(W_{t};x_{t})</annotation></semantics></math> also improves further from <math id="S2.F5.16.m8.2" class="ltx_Math" alttext="\ell(W_{0};x_{t})" display="inline"><semantics id="S2.F5.16.m8.2b"><mrow id="S2.F5.16.m8.2.2" xref="S2.F5.16.m8.2.2.cmml"><mi mathvariant="normal" id="S2.F5.16.m8.2.2.4" xref="S2.F5.16.m8.2.2.4.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="S2.F5.16.m8.2.2.3" xref="S2.F5.16.m8.2.2.3.cmml">​</mo><mrow id="S2.F5.16.m8.2.2.2.2" xref="S2.F5.16.m8.2.2.2.3.cmml"><mo stretchy="false" id="S2.F5.16.m8.2.2.2.2.3" xref="S2.F5.16.m8.2.2.2.3.cmml">(</mo><msub id="S2.F5.16.m8.1.1.1.1.1" xref="S2.F5.16.m8.1.1.1.1.1.cmml"><mi id="S2.F5.16.m8.1.1.1.1.1.2" xref="S2.F5.16.m8.1.1.1.1.1.2.cmml">W</mi><mn id="S2.F5.16.m8.1.1.1.1.1.3" xref="S2.F5.16.m8.1.1.1.1.1.3.cmml">0</mn></msub><mo id="S2.F5.16.m8.2.2.2.2.4" xref="S2.F5.16.m8.2.2.2.3.cmml">;</mo><msub id="S2.F5.16.m8.2.2.2.2.2" xref="S2.F5.16.m8.2.2.2.2.2.cmml"><mi id="S2.F5.16.m8.2.2.2.2.2.2" xref="S2.F5.16.m8.2.2.2.2.2.2.cmml">x</mi><mi id="S2.F5.16.m8.2.2.2.2.2.3" xref="S2.F5.16.m8.2.2.2.2.2.3.cmml">t</mi></msub><mo stretchy="false" id="S2.F5.16.m8.2.2.2.2.5" xref="S2.F5.16.m8.2.2.2.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.F5.16.m8.2c"><apply id="S2.F5.16.m8.2.2.cmml" xref="S2.F5.16.m8.2.2"><times id="S2.F5.16.m8.2.2.3.cmml" xref="S2.F5.16.m8.2.2.3"></times><ci id="S2.F5.16.m8.2.2.4.cmml" xref="S2.F5.16.m8.2.2.4">ℓ</ci><list id="S2.F5.16.m8.2.2.2.3.cmml" xref="S2.F5.16.m8.2.2.2.2"><apply id="S2.F5.16.m8.1.1.1.1.1.cmml" xref="S2.F5.16.m8.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.F5.16.m8.1.1.1.1.1.1.cmml" xref="S2.F5.16.m8.1.1.1.1.1">subscript</csymbol><ci id="S2.F5.16.m8.1.1.1.1.1.2.cmml" xref="S2.F5.16.m8.1.1.1.1.1.2">𝑊</ci><cn type="integer" id="S2.F5.16.m8.1.1.1.1.1.3.cmml" xref="S2.F5.16.m8.1.1.1.1.1.3">0</cn></apply><apply id="S2.F5.16.m8.2.2.2.2.2.cmml" xref="S2.F5.16.m8.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.F5.16.m8.2.2.2.2.2.1.cmml" xref="S2.F5.16.m8.2.2.2.2.2">subscript</csymbol><ci id="S2.F5.16.m8.2.2.2.2.2.2.cmml" xref="S2.F5.16.m8.2.2.2.2.2.2">𝑥</ci><ci id="S2.F5.16.m8.2.2.2.2.2.3.cmml" xref="S2.F5.16.m8.2.2.2.2.2.3">𝑡</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F5.16.m8.2d">\ell(W_{0};x_{t})</annotation></semantics></math>.
For visual clarity, loss values have been averaged over a sliding window of 10 timesteps.
See Figure <a href="#A3.F17" title="Figure 17 ‣ Gradient checkpointing through time. ‣ Appendix C Experiment details ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">17</span></a> (in Appendix) for complete results on all 12 layers.
</figcaption>
</figure>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Training a network with TTT layers</h3>

<div id="S2.SS2.p1" class="ltx_para">
<p id="S2.SS2.p1.6" class="ltx_p">The forward pass of a TTT layer also has a corresponding backward pass.
<span id="S2.SS2.p1.6.1" class="ltx_text">Our forward pass only</span> consists of standard differentiable operators except the gradient operator <math id="S2.SS2.p1.1.m1.1" class="ltx_Math" alttext="\nabla" display="inline"><semantics id="S2.SS2.p1.1.m1.1a"><mo id="S2.SS2.p1.1.m1.1.1" xref="S2.SS2.p1.1.m1.1.1.cmml">∇</mo><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.1.m1.1b"><ci id="S2.SS2.p1.1.m1.1.1.cmml" xref="S2.SS2.p1.1.m1.1.1">∇</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.1.m1.1c">\nabla</annotation></semantics></math>.
However, <math id="S2.SS2.p1.2.m2.1" class="ltx_Math" alttext="\nabla" display="inline"><semantics id="S2.SS2.p1.2.m2.1a"><mo id="S2.SS2.p1.2.m2.1.1" xref="S2.SS2.p1.2.m2.1.1.cmml">∇</mo><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.2.m2.1b"><ci id="S2.SS2.p1.2.m2.1.1.cmml" xref="S2.SS2.p1.2.m2.1.1">∇</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.2.m2.1c">\nabla</annotation></semantics></math> just maps one function to another, in this case <math id="S2.SS2.p1.3.m3.1" class="ltx_Math" alttext="\ell" display="inline"><semantics id="S2.SS2.p1.3.m3.1a"><mi mathvariant="normal" id="S2.SS2.p1.3.m3.1.1" xref="S2.SS2.p1.3.m3.1.1.cmml">ℓ</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.3.m3.1b"><ci id="S2.SS2.p1.3.m3.1.1.cmml" xref="S2.SS2.p1.3.m3.1.1">ℓ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.3.m3.1c">\ell</annotation></semantics></math> to <math id="S2.SS2.p1.4.m4.1" class="ltx_Math" alttext="\nabla\ell" display="inline"><semantics id="S2.SS2.p1.4.m4.1a"><mrow id="S2.SS2.p1.4.m4.1.1" xref="S2.SS2.p1.4.m4.1.1.cmml"><mo rspace="0.167em" id="S2.SS2.p1.4.m4.1.1.1" xref="S2.SS2.p1.4.m4.1.1.1.cmml">∇</mo><mi mathvariant="normal" id="S2.SS2.p1.4.m4.1.1.2" xref="S2.SS2.p1.4.m4.1.1.2.cmml">ℓ</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.4.m4.1b"><apply id="S2.SS2.p1.4.m4.1.1.cmml" xref="S2.SS2.p1.4.m4.1.1"><ci id="S2.SS2.p1.4.m4.1.1.1.cmml" xref="S2.SS2.p1.4.m4.1.1.1">∇</ci><ci id="S2.SS2.p1.4.m4.1.1.2.cmml" xref="S2.SS2.p1.4.m4.1.1.2">ℓ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.4.m4.1c">\nabla\ell</annotation></semantics></math>, and <math id="S2.SS2.p1.5.m5.1" class="ltx_Math" alttext="\nabla\ell" display="inline"><semantics id="S2.SS2.p1.5.m5.1a"><mrow id="S2.SS2.p1.5.m5.1.1" xref="S2.SS2.p1.5.m5.1.1.cmml"><mo rspace="0.167em" id="S2.SS2.p1.5.m5.1.1.1" xref="S2.SS2.p1.5.m5.1.1.1.cmml">∇</mo><mi mathvariant="normal" id="S2.SS2.p1.5.m5.1.1.2" xref="S2.SS2.p1.5.m5.1.1.2.cmml">ℓ</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.5.m5.1b"><apply id="S2.SS2.p1.5.m5.1.1.cmml" xref="S2.SS2.p1.5.m5.1.1"><ci id="S2.SS2.p1.5.m5.1.1.1.cmml" xref="S2.SS2.p1.5.m5.1.1.1">∇</ci><ci id="S2.SS2.p1.5.m5.1.1.2.cmml" xref="S2.SS2.p1.5.m5.1.1.2">ℓ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.5.m5.1c">\nabla\ell</annotation></semantics></math> is also composed of differentiable operators.
Conceptually, calling backward on <math id="S2.SS2.p1.6.m6.1" class="ltx_Math" alttext="\nabla\ell" display="inline"><semantics id="S2.SS2.p1.6.m6.1a"><mrow id="S2.SS2.p1.6.m6.1.1" xref="S2.SS2.p1.6.m6.1.1.cmml"><mo rspace="0.167em" id="S2.SS2.p1.6.m6.1.1.1" xref="S2.SS2.p1.6.m6.1.1.1.cmml">∇</mo><mi mathvariant="normal" id="S2.SS2.p1.6.m6.1.1.2" xref="S2.SS2.p1.6.m6.1.1.2.cmml">ℓ</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.6.m6.1b"><apply id="S2.SS2.p1.6.m6.1.1.cmml" xref="S2.SS2.p1.6.m6.1.1"><ci id="S2.SS2.p1.6.m6.1.1.1.cmml" xref="S2.SS2.p1.6.m6.1.1.1">∇</ci><ci id="S2.SS2.p1.6.m6.1.1.2.cmml" xref="S2.SS2.p1.6.m6.1.1.2">ℓ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.6.m6.1c">\nabla\ell</annotation></semantics></math> means taking gradients of gradients – a well explored technique in meta-learning <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib51" title="" class="ltx_ref">51</a>]</cite>.</p>
</div>
<div id="S2.SS2.p2" class="ltx_para">
<p id="S2.SS2.p2.1" class="ltx_p">TTT layers have the same interface as RNN layers and self-attention, therefore can be replaced in any larger network architecture, which usually contains many of these sequence modeling layers.
Training a network with TTT layers also works the same way as training any other language model, such as a Transformer.
The same data, recipe, and objective such as next-token prediction can be used to optimize parameters of the rest of the network.</p>
</div>
<div id="S2.SS2.p3" class="ltx_para">
<p id="S2.SS2.p3.6" class="ltx_p">We refer to training the larger network as the <em id="S2.SS2.p3.6.1" class="ltx_emph ltx_font_italic">outer loop</em>, and training <math id="S2.SS2.p3.1.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.SS2.p3.1.m1.1a"><mi id="S2.SS2.p3.1.m1.1.1" xref="S2.SS2.p3.1.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.1.m1.1b"><ci id="S2.SS2.p3.1.m1.1.1.cmml" xref="S2.SS2.p3.1.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.1.m1.1c">W</annotation></semantics></math> within each TTT layer as the <em id="S2.SS2.p3.6.2" class="ltx_emph ltx_font_italic">inner loop</em>.
An important difference between the two nested learning problems is that the inner-loop gradient <math id="S2.SS2.p3.2.m2.1" class="ltx_Math" alttext="\nabla\ell" display="inline"><semantics id="S2.SS2.p3.2.m2.1a"><mrow id="S2.SS2.p3.2.m2.1.1" xref="S2.SS2.p3.2.m2.1.1.cmml"><mo rspace="0.167em" id="S2.SS2.p3.2.m2.1.1.1" xref="S2.SS2.p3.2.m2.1.1.1.cmml">∇</mo><mi mathvariant="normal" id="S2.SS2.p3.2.m2.1.1.2" xref="S2.SS2.p3.2.m2.1.1.2.cmml">ℓ</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.2.m2.1b"><apply id="S2.SS2.p3.2.m2.1.1.cmml" xref="S2.SS2.p3.2.m2.1.1"><ci id="S2.SS2.p3.2.m2.1.1.1.cmml" xref="S2.SS2.p3.2.m2.1.1.1">∇</ci><ci id="S2.SS2.p3.2.m2.1.1.2.cmml" xref="S2.SS2.p3.2.m2.1.1.2">ℓ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.2.m2.1c">\nabla\ell</annotation></semantics></math> is taken w.r.t. <math id="S2.SS2.p3.3.m3.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.SS2.p3.3.m3.1a"><mi id="S2.SS2.p3.3.m3.1.1" xref="S2.SS2.p3.3.m3.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.3.m3.1b"><ci id="S2.SS2.p3.3.m3.1.1.cmml" xref="S2.SS2.p3.3.m3.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.3.m3.1c">W</annotation></semantics></math>, the parameters of <math id="S2.SS2.p3.4.m4.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S2.SS2.p3.4.m4.1a"><mi id="S2.SS2.p3.4.m4.1.1" xref="S2.SS2.p3.4.m4.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.4.m4.1b"><ci id="S2.SS2.p3.4.m4.1.1.cmml" xref="S2.SS2.p3.4.m4.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.4.m4.1c">f</annotation></semantics></math>, while the outer-loop gradient is taken w.r.t the parameters of the rest of the network, which we will denote by <math id="S2.SS2.p3.5.m5.1" class="ltx_Math" alttext="\theta_{\text{rest}}" display="inline"><semantics id="S2.SS2.p3.5.m5.1a"><msub id="S2.SS2.p3.5.m5.1.1" xref="S2.SS2.p3.5.m5.1.1.cmml"><mi id="S2.SS2.p3.5.m5.1.1.2" xref="S2.SS2.p3.5.m5.1.1.2.cmml">θ</mi><mtext id="S2.SS2.p3.5.m5.1.1.3" xref="S2.SS2.p3.5.m5.1.1.3a.cmml">rest</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.5.m5.1b"><apply id="S2.SS2.p3.5.m5.1.1.cmml" xref="S2.SS2.p3.5.m5.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.5.m5.1.1.1.cmml" xref="S2.SS2.p3.5.m5.1.1">subscript</csymbol><ci id="S2.SS2.p3.5.m5.1.1.2.cmml" xref="S2.SS2.p3.5.m5.1.1.2">𝜃</ci><ci id="S2.SS2.p3.5.m5.1.1.3a.cmml" xref="S2.SS2.p3.5.m5.1.1.3"><mtext mathsize="70%" id="S2.SS2.p3.5.m5.1.1.3.cmml" xref="S2.SS2.p3.5.m5.1.1.3">rest</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.5.m5.1c">\theta_{\text{rest}}</annotation></semantics></math>.
Throughout this paper, outer-loop parameters are always denoted by <math id="S2.SS2.p3.6.m6.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="S2.SS2.p3.6.m6.1a"><mi id="S2.SS2.p3.6.m6.1.1" xref="S2.SS2.p3.6.m6.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.6.m6.1b"><ci id="S2.SS2.p3.6.m6.1.1.cmml" xref="S2.SS2.p3.6.m6.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.6.m6.1c">\theta</annotation></semantics></math> with various subscripts.</p>
</div>
<div id="S2.SS2.p4" class="ltx_para">
<p id="S2.SS2.p4.1" class="ltx_p">So far, the TTT layer has no outer-loop parameters, in contrast to other RNN layers and self-attention.
In Subsection <a href="#S2.SS3" title="2.3 Learning a self-supervised task for TTT ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.3</span></a>, we add outer-loop parameters to the TTT layer to improve its self-supervised task.
Then in Subsection <a href="#S2.SS4" title="2.4 Parallelization with mini-batch TTT ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.4</span></a> and <a href="#S2.SS5" title="2.5 Dual form ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.5</span></a>, we discuss two ways to improve the wall-clock time of TTT layers.</p>
</div>
<figure id="S2.F6" class="ltx_figure">
<div id="S2.F6.1" class="ltx_listing ltx_lst_language_Python ltx_lstlisting ltx_listing" style="background-color:#FCF8ED;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,Y2xhc3MgVFRUX0xheWVyKG5uLk1vZHVsZSk6CiAgZGVmIF9faW5pdF9fKHNlbGYpOgogICAgc2VsZi50YXNrID0gVGFzaygpCgogIGRlZiBmb3J3YXJkKHNlbGYsIGluX3NlcSk6CiAgICBzdGF0ZSA9IExlYXJuZXIoc2VsZi50YXNrKQogICAgb3V0X3NlcSA9IFtdCiAgICBmb3IgdG9rIGluIGluX3NlcToKICAgICAgc3RhdGUudHJhaW4odG9rKQogICAgICBvdXRfc2VxLmFwcGVuZChzdGF0ZS5wcmVkaWN0KHRvaykpCiAgICByZXR1cm4gb3V0X3NlcQoKY2xhc3MgVGFzayhubi5Nb2R1bGUpOgogIGRlZiBfX2luaXRfXyhzZWxmKToKICAgIHNlbGYudGhldGFfSyA9IG5uLlBhcmFtKChkMSwgZDIpKQogICAgc2VsZi50aGV0YV9WID0gbm4uUGFyYW0oKGQxLCBkMikpCiAgICBzZWxmLnRoZXRhX1EgPSBubi5QYXJhbSgoZDEsIGQyKSkKCiAgZGVmIGxvc3Moc2VsZiwgZiwgeCk6CiAgICB0cmFpbl92aWV3ID0gc2VsZi50aGV0YV9LIEAgeAogICAgbGFiZWxfdmlldyA9IHNlbGYudGhldGFfViBAIHgKICAgIHJldHVybiBNU0UoZih0cmFpbl92aWV3KSwgbGFiZWxfdmlldykKY2xhc3MgTGVhcm5lcigpOgogIGRlZiBfX2luaXRfXyhzZWxmLCB0YXNrKToKICAgIHNlbGYudGFzayA9IHRhc2sKICAgICMgTGluZWFyIGhlcmUsIGJ1dCBjYW4gYmUgYW55IG1vZGVsCiAgICBzZWxmLm1vZGVsID0gTGluZWFyKCkKICAgICMgb25saW5lIEdEIGhlcmUgZm9yIHNpbXBsaWNpdHkKICAgIHNlbGYub3B0aW0gPSBPR0QoKQoKICBkZWYgdHJhaW4oc2VsZiwgeCk6CiAgICAjIGdyYWQgZnVuY3Rpb24gd3J0IGZpcnN0IGFyZwogICAgIyBvZiBsb3NzLCB3aGljaCBpcyBzZWxmLm1vZGVsCiAgICBncmFkX2ZuID0gZ3JhZChzZWxmLnRhc2subG9zcykKICAgICMgY2FsY3VsYXRlIGlubmVyLWxvb3AgZ3JhZAogICAgZ3JhZF9pbiA9IGdyYWRfZm4oc2VsZi5tb2RlbCwgeCkKCiAgICAjIHN0YXJ0aW5nIGZyb20gY3VycmVudCBwYXJhbXMsCiAgICAjIHN0ZXAgaW4gZGlyZWN0aW9uIG9mIGdyYWRfaW4sCiAgICBzZWxmLm9wdGltLnN0ZXAoc2VsZi5tb2RlbCwgZ3JhZF9pbikKCiAgZGVmIHByZWRpY3Qoc2VsZiwgeCk6CiAgICB0ZXN0X3ZpZXcgPSBzZWxmLnRhc2sudGhldGFfUSBAIHgKICAgIHJldHVybiBzZWxmLm1vZGVsKHRlc3Rfdmlldyk=" download="">⬇</a></div>
<div id="lstnumberx1" class="ltx_listingline">
<span id="lstnumberx1.1" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="font-size:90%;color:#0047AB;">class</span><span id="lstnumberx1.2" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx1.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">TTT_Layer</span><span id="lstnumberx1.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(</span><span id="lstnumberx1.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">nn</span><span id="lstnumberx1.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx1.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Module</span><span id="lstnumberx1.8" class="ltx_text ltx_font_typewriter" style="font-size:90%;">):</span>
</div>
<div id="lstnumberx2" class="ltx_listingline">
<span id="lstnumberx2.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">  </span><span id="lstnumberx2.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="font-size:90%;color:#0047AB;">def</span><span id="lstnumberx2.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx2.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">__init__</span><span id="lstnumberx2.5" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(</span><span id="lstnumberx2.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx2.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">):</span>
</div>
<div id="lstnumberx3" class="ltx_listingline">
<span id="lstnumberx3.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx3.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx3.3" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx3.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">task</span><span id="lstnumberx3.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx3.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx3.7" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx3.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Task</span><span id="lstnumberx3.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">()</span>
</div>
<div id="lstnumberx4" class="ltx_listingline">
</div>
<div id="lstnumberx5" class="ltx_listingline">
<span id="lstnumberx5.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">  </span><span id="lstnumberx5.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="font-size:90%;color:#0047AB;">def</span><span id="lstnumberx5.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx5.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">forward</span><span id="lstnumberx5.5" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(</span><span id="lstnumberx5.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx5.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">,</span><span id="lstnumberx5.8" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx5.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">in_seq</span><span id="lstnumberx5.10" class="ltx_text ltx_font_typewriter" style="font-size:90%;">):</span>
</div>
<div id="lstnumberx6" class="ltx_listingline">
<span id="lstnumberx6.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx6.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">state</span><span id="lstnumberx6.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx6.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx6.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx6.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Learner</span><span id="lstnumberx6.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(</span><span id="lstnumberx6.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx6.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx6.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">task</span><span id="lstnumberx6.11" class="ltx_text ltx_font_typewriter" style="font-size:90%;">)</span>
</div>
<div id="lstnumberx7" class="ltx_listingline">
<span id="lstnumberx7.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx7.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">out_seq</span><span id="lstnumberx7.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx7.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx7.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx7.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">[]</span>
</div>
<div id="lstnumberx8" class="ltx_listingline">
<span id="lstnumberx8.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx8.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="font-size:90%;color:#0047AB;">for</span><span id="lstnumberx8.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx8.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">tok</span><span id="lstnumberx8.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx8.6" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="font-size:90%;color:#0047AB;">in</span><span id="lstnumberx8.7" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx8.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">in_seq</span><span id="lstnumberx8.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">:</span>
</div>
<div id="lstnumberx9" class="ltx_listingline">
<span id="lstnumberx9.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">      </span><span id="lstnumberx9.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">state</span><span id="lstnumberx9.3" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx9.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">train</span><span id="lstnumberx9.5" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(</span><span id="lstnumberx9.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">tok</span><span id="lstnumberx9.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">)</span>
</div>
<div id="lstnumberx10" class="ltx_listingline">
<span id="lstnumberx10.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">      </span><span id="lstnumberx10.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">out_seq</span><span id="lstnumberx10.3" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx10.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">append</span><span id="lstnumberx10.5" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(</span><span id="lstnumberx10.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">state</span><span id="lstnumberx10.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx10.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">predict</span><span id="lstnumberx10.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(</span><span id="lstnumberx10.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">tok</span><span id="lstnumberx10.11" class="ltx_text ltx_font_typewriter" style="font-size:90%;">))</span>
</div>
<div id="lstnumberx11" class="ltx_listingline">
<span id="lstnumberx11.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx11.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="font-size:90%;color:#0047AB;">return</span><span id="lstnumberx11.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">out_seq</span>
</div>
<div id="lstnumberx12" class="ltx_listingline">
</div>
<div id="lstnumberx13" class="ltx_listingline">
<span id="lstnumberx13.1" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="font-size:90%;color:#0047AB;">class</span><span id="lstnumberx13.2" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Task</span><span id="lstnumberx13.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(</span><span id="lstnumberx13.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">nn</span><span id="lstnumberx13.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx13.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Module</span><span id="lstnumberx13.8" class="ltx_text ltx_font_typewriter" style="font-size:90%;">):</span>
</div>
<div id="lstnumberx14" class="ltx_listingline">
<span id="lstnumberx14.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">  </span><span id="lstnumberx14.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="font-size:90%;color:#0047AB;">def</span><span id="lstnumberx14.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx14.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">__init__</span><span id="lstnumberx14.5" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(</span><span id="lstnumberx14.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx14.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">):</span>
</div>
<div id="lstnumberx15" class="ltx_listingline">
<span id="lstnumberx15.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx15.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx15.3" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx15.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">theta_K</span><span id="lstnumberx15.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx15.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx15.7" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx15.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">nn</span><span id="lstnumberx15.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx15.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Param</span><span id="lstnumberx15.11" class="ltx_text ltx_font_typewriter" style="font-size:90%;">((</span><span id="lstnumberx15.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">d1</span><span id="lstnumberx15.13" class="ltx_text ltx_font_typewriter" style="font-size:90%;">,</span><span id="lstnumberx15.14" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx15.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">d2</span><span id="lstnumberx15.16" class="ltx_text ltx_font_typewriter" style="font-size:90%;">))</span>
</div>
<div id="lstnumberx16" class="ltx_listingline">
<span id="lstnumberx16.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx16.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx16.3" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx16.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">theta_V</span><span id="lstnumberx16.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx16.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx16.7" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx16.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">nn</span><span id="lstnumberx16.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx16.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Param</span><span id="lstnumberx16.11" class="ltx_text ltx_font_typewriter" style="font-size:90%;">((</span><span id="lstnumberx16.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">d1</span><span id="lstnumberx16.13" class="ltx_text ltx_font_typewriter" style="font-size:90%;">,</span><span id="lstnumberx16.14" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx16.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">d2</span><span id="lstnumberx16.16" class="ltx_text ltx_font_typewriter" style="font-size:90%;">))</span>
</div>
<div id="lstnumberx17" class="ltx_listingline">
<span id="lstnumberx17.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx17.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx17.3" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx17.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">theta_Q</span><span id="lstnumberx17.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx17.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx17.7" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx17.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">nn</span><span id="lstnumberx17.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx17.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Param</span><span id="lstnumberx17.11" class="ltx_text ltx_font_typewriter" style="font-size:90%;">((</span><span id="lstnumberx17.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">d1</span><span id="lstnumberx17.13" class="ltx_text ltx_font_typewriter" style="font-size:90%;">,</span><span id="lstnumberx17.14" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx17.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">d2</span><span id="lstnumberx17.16" class="ltx_text ltx_font_typewriter" style="font-size:90%;">))</span>
</div>
<div id="lstnumberx18" class="ltx_listingline">
</div>
<div id="lstnumberx19" class="ltx_listingline">
<span id="lstnumberx19.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">  </span><span id="lstnumberx19.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="font-size:90%;color:#0047AB;">def</span><span id="lstnumberx19.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx19.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">loss</span><span id="lstnumberx19.5" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(</span><span id="lstnumberx19.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx19.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">,</span><span id="lstnumberx19.8" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx19.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">f</span><span id="lstnumberx19.10" class="ltx_text ltx_font_typewriter" style="font-size:90%;">,</span><span id="lstnumberx19.11" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx19.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">x</span><span id="lstnumberx19.13" class="ltx_text ltx_font_typewriter" style="font-size:90%;">):</span>
</div>
<div id="lstnumberx20" class="ltx_listingline">
<span id="lstnumberx20.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx20.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">train_view</span><span id="lstnumberx20.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx20.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx20.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx20.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">theta_K</span><span id="lstnumberx20.9" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">@</span><span id="lstnumberx20.11" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">x</span>
</div>
<div id="lstnumberx21" class="ltx_listingline">
<span id="lstnumberx21.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx21.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">label_view</span><span id="lstnumberx21.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx21.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx21.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx21.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx21.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx21.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">theta_V</span><span id="lstnumberx21.9" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx21.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">@</span><span id="lstnumberx21.11" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx21.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">x</span>
</div>
<div id="lstnumberx22" class="ltx_listingline">
<span id="lstnumberx22.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx22.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="font-size:90%;color:#0047AB;">return</span><span id="lstnumberx22.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">MSE</span><span id="lstnumberx22.5" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(</span><span id="lstnumberx22.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">f</span><span id="lstnumberx22.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(</span><span id="lstnumberx22.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">train_view</span><span id="lstnumberx22.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">),</span><span id="lstnumberx22.10" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">label_view</span><span id="lstnumberx22.12" class="ltx_text ltx_font_typewriter" style="font-size:90%;">)</span>
</div>
<div id="lstnumberx23" class="ltx_listingline">
<span id="lstnumberx23.1" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="font-size:90%;color:#0047AB;">class</span><span id="lstnumberx23.2" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx23.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Learner</span><span id="lstnumberx23.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">():</span>
</div>
<div id="lstnumberx24" class="ltx_listingline">
<span id="lstnumberx24.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">  </span><span id="lstnumberx24.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="font-size:90%;color:#0047AB;">def</span><span id="lstnumberx24.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx24.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">__init__</span><span id="lstnumberx24.5" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(</span><span id="lstnumberx24.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx24.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">,</span><span id="lstnumberx24.8" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx24.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">task</span><span id="lstnumberx24.10" class="ltx_text ltx_font_typewriter" style="font-size:90%;">):</span>
</div>
<div id="lstnumberx25" class="ltx_listingline">
<span id="lstnumberx25.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx25.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx25.3" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx25.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">task</span><span id="lstnumberx25.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx25.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx25.7" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx25.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">task</span>
</div>
<div id="lstnumberx26" class="ltx_listingline">
<span id="lstnumberx26.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx26.2" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="font-size:90%;color:#5C7B92;">#<span id="lstnumberx26.2.1" class="ltx_text ltx_lst_space"> </span>Linear<span id="lstnumberx26.2.2" class="ltx_text ltx_lst_space"> </span>here,<span id="lstnumberx26.2.3" class="ltx_text ltx_lst_space"> </span>but<span id="lstnumberx26.2.4" class="ltx_text ltx_lst_space"> </span>can<span id="lstnumberx26.2.5" class="ltx_text ltx_lst_space"> </span>be<span id="lstnumberx26.2.6" class="ltx_text ltx_lst_space"> </span>any<span id="lstnumberx26.2.7" class="ltx_text ltx_lst_space"> </span>model</span>
</div>
<div id="lstnumberx27" class="ltx_listingline">
<span id="lstnumberx27.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx27.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx27.3" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx27.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">model</span><span id="lstnumberx27.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx27.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx27.7" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx27.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Linear</span><span id="lstnumberx27.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">()</span>
</div>
<div id="lstnumberx28" class="ltx_listingline">
<span id="lstnumberx28.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx28.2" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="font-size:90%;color:#5C7B92;">#<span id="lstnumberx28.2.1" class="ltx_text ltx_lst_space"> </span>online<span id="lstnumberx28.2.2" class="ltx_text ltx_lst_space"> </span>GD<span id="lstnumberx28.2.3" class="ltx_text ltx_lst_space"> </span>here<span id="lstnumberx28.2.4" class="ltx_text ltx_lst_space"> </span>for<span id="lstnumberx28.2.5" class="ltx_text ltx_lst_space"> </span>simplicity</span>
</div>
<div id="lstnumberx29" class="ltx_listingline">
<span id="lstnumberx29.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx29.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx29.3" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx29.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">optim</span><span id="lstnumberx29.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx29.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx29.7" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx29.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">OGD</span><span id="lstnumberx29.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">()</span>
</div>
<div id="lstnumberx30" class="ltx_listingline">
</div>
<div id="lstnumberx31" class="ltx_listingline">
<span id="lstnumberx31.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">  </span><span id="lstnumberx31.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="font-size:90%;color:#0047AB;">def</span><span id="lstnumberx31.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx31.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">train</span><span id="lstnumberx31.5" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(</span><span id="lstnumberx31.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx31.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">,</span><span id="lstnumberx31.8" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx31.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">x</span><span id="lstnumberx31.10" class="ltx_text ltx_font_typewriter" style="font-size:90%;">):</span>
</div>
<div id="lstnumberx32" class="ltx_listingline">
<span id="lstnumberx32.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx32.2" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="font-size:90%;color:#5C7B92;">#<span id="lstnumberx32.2.1" class="ltx_text ltx_lst_space"> </span>grad<span id="lstnumberx32.2.2" class="ltx_text ltx_lst_space"> </span>function<span id="lstnumberx32.2.3" class="ltx_text ltx_lst_space"> </span>wrt<span id="lstnumberx32.2.4" class="ltx_text ltx_lst_space"> </span>first<span id="lstnumberx32.2.5" class="ltx_text ltx_lst_space"> </span>arg</span>
</div>
<div id="lstnumberx33" class="ltx_listingline">
<span id="lstnumberx33.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx33.2" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="font-size:90%;color:#5C7B92;">#<span id="lstnumberx33.2.1" class="ltx_text ltx_lst_space"> </span>of<span id="lstnumberx33.2.2" class="ltx_text ltx_lst_space"> </span>loss,<span id="lstnumberx33.2.3" class="ltx_text ltx_lst_space"> </span>which<span id="lstnumberx33.2.4" class="ltx_text ltx_lst_space"> </span>is<span id="lstnumberx33.2.5" class="ltx_text ltx_lst_space"> </span>self.model</span>
</div>
<div id="lstnumberx34" class="ltx_listingline">
<span id="lstnumberx34.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx34.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">grad_fn</span><span id="lstnumberx34.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx34.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx34.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx34.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">grad</span><span id="lstnumberx34.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(</span><span id="lstnumberx34.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx34.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx34.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">task</span><span id="lstnumberx34.11" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx34.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">loss</span><span id="lstnumberx34.13" class="ltx_text ltx_font_typewriter" style="font-size:90%;">)</span>
</div>
<div id="lstnumberx35" class="ltx_listingline">
<span id="lstnumberx35.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx35.2" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="font-size:90%;color:#5C7B92;">#<span id="lstnumberx35.2.1" class="ltx_text ltx_lst_space"> </span>calculate<span id="lstnumberx35.2.2" class="ltx_text ltx_lst_space"> </span>inner-loop<span id="lstnumberx35.2.3" class="ltx_text ltx_lst_space"> </span>grad</span>
</div>
<div id="lstnumberx36" class="ltx_listingline">
<span id="lstnumberx36.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx36.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">grad_in</span><span id="lstnumberx36.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx36.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx36.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx36.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">grad_fn</span><span id="lstnumberx36.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(</span><span id="lstnumberx36.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx36.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx36.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">model</span><span id="lstnumberx36.11" class="ltx_text ltx_font_typewriter" style="font-size:90%;">,</span><span id="lstnumberx36.12" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx36.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">x</span><span id="lstnumberx36.14" class="ltx_text ltx_font_typewriter" style="font-size:90%;">)</span>
</div>
<div id="lstnumberx37" class="ltx_listingline">
</div>
<div id="lstnumberx38" class="ltx_listingline">
<span id="lstnumberx38.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx38.2" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="font-size:90%;color:#5C7B92;">#<span id="lstnumberx38.2.1" class="ltx_text ltx_lst_space"> </span>starting<span id="lstnumberx38.2.2" class="ltx_text ltx_lst_space"> </span>from<span id="lstnumberx38.2.3" class="ltx_text ltx_lst_space"> </span>current<span id="lstnumberx38.2.4" class="ltx_text ltx_lst_space"> </span>params,</span>
</div>
<div id="lstnumberx39" class="ltx_listingline">
<span id="lstnumberx39.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx39.2" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="font-size:90%;color:#5C7B92;">#<span id="lstnumberx39.2.1" class="ltx_text ltx_lst_space"> </span>step<span id="lstnumberx39.2.2" class="ltx_text ltx_lst_space"> </span>in<span id="lstnumberx39.2.3" class="ltx_text ltx_lst_space"> </span>direction<span id="lstnumberx39.2.4" class="ltx_text ltx_lst_space"> </span>of<span id="lstnumberx39.2.5" class="ltx_text ltx_lst_space"> </span>grad_in,</span>
</div>
<div id="lstnumberx40" class="ltx_listingline">
<span id="lstnumberx40.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx40.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx40.3" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx40.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">optim</span><span id="lstnumberx40.5" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx40.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">step</span><span id="lstnumberx40.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(</span><span id="lstnumberx40.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx40.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx40.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">model</span><span id="lstnumberx40.11" class="ltx_text ltx_font_typewriter" style="font-size:90%;">,</span><span id="lstnumberx40.12" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx40.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">grad_in</span><span id="lstnumberx40.14" class="ltx_text ltx_font_typewriter" style="font-size:90%;">)</span>
</div>
<div id="lstnumberx41" class="ltx_listingline">
</div>
<div id="lstnumberx42" class="ltx_listingline">
<span id="lstnumberx42.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">  </span><span id="lstnumberx42.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="font-size:90%;color:#0047AB;">def</span><span id="lstnumberx42.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">predict</span><span id="lstnumberx42.5" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(</span><span id="lstnumberx42.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx42.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">,</span><span id="lstnumberx42.8" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">x</span><span id="lstnumberx42.10" class="ltx_text ltx_font_typewriter" style="font-size:90%;">):</span>
</div>
<div id="lstnumberx43" class="ltx_listingline">
<span id="lstnumberx43.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx43.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">test_view</span><span id="lstnumberx43.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx43.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx43.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx43.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx43.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx43.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">task</span><span id="lstnumberx43.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx43.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">theta_Q</span><span id="lstnumberx43.11" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx43.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">@</span><span id="lstnumberx43.13" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx43.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">x</span>
</div>
<div id="lstnumberx44" class="ltx_listingline">
<span id="lstnumberx44.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx44.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="font-size:90%;color:#0047AB;">return</span><span id="lstnumberx44.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">self</span><span id="lstnumberx44.5" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx44.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">model</span><span id="lstnumberx44.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(</span><span id="lstnumberx44.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">test_view</span><span id="lstnumberx44.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">)</span>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>
Naive implementation of a TTT layer with a linear model and online GD in the style of PyTorch.
<span id="S2.F6.12.1" class="ltx_text ltx_font_typewriter">TTT_Layer</span> can be dropped into a larger network like other sequence modeling layers.
Training the network will optimize the parameters of <span id="S2.F6.13.2" class="ltx_text ltx_font_typewriter">Task</span> in <span id="S2.F6.14.3" class="ltx_text ltx_font_typewriter">TTT_Layer</span>, because both are subclasses of <span id="S2.F6.15.4" class="ltx_text ltx_font_typewriter">nn.Module</span>.
Since <span id="S2.F6.16.5" class="ltx_text ltx_font_typewriter">Learner</span> is not a subclass of <span id="S2.F6.17.6" class="ltx_text ltx_font_typewriter">nn.Module</span>, <span id="S2.F6.18.7" class="ltx_text ltx_font_typewriter">state.model</span> is updated manually in the inner loop for each call of <span id="S2.F6.19.8" class="ltx_text ltx_font_typewriter">state.train</span>.
For simplicity, we sometimes overload <span id="S2.F6.20.9" class="ltx_text ltx_font_typewriter">model</span> as <span id="S2.F6.21.10" class="ltx_text ltx_font_typewriter">model.parameters</span>.
</figcaption>
</figure>
</section>
<section id="S2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.3 </span>Learning a self-supervised task for TTT</h3>

<div id="S2.SS3.p1" class="ltx_para">
<p id="S2.SS3.p1.2" class="ltx_p">Arguably the most important part of TTT is the self-supervised task, because it determines the kind of features that <math id="S2.SS3.p1.1.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.SS3.p1.1.m1.1a"><mi id="S2.SS3.p1.1.m1.1.1" xref="S2.SS3.p1.1.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p1.1.m1.1b"><ci id="S2.SS3.p1.1.m1.1.1.cmml" xref="S2.SS3.p1.1.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p1.1.m1.1c">W</annotation></semantics></math> will learn from the test sequence.
So how should we design this task?
The final goal of TTT is for <math id="S2.SS3.p1.2.m2.2" class="ltx_Math" alttext="z_{t}=f(x_{t};W_{t})" display="inline"><semantics id="S2.SS3.p1.2.m2.2a"><mrow id="S2.SS3.p1.2.m2.2.2" xref="S2.SS3.p1.2.m2.2.2.cmml"><msub id="S2.SS3.p1.2.m2.2.2.4" xref="S2.SS3.p1.2.m2.2.2.4.cmml"><mi id="S2.SS3.p1.2.m2.2.2.4.2" xref="S2.SS3.p1.2.m2.2.2.4.2.cmml">z</mi><mi id="S2.SS3.p1.2.m2.2.2.4.3" xref="S2.SS3.p1.2.m2.2.2.4.3.cmml">t</mi></msub><mo id="S2.SS3.p1.2.m2.2.2.3" xref="S2.SS3.p1.2.m2.2.2.3.cmml">=</mo><mrow id="S2.SS3.p1.2.m2.2.2.2" xref="S2.SS3.p1.2.m2.2.2.2.cmml"><mi id="S2.SS3.p1.2.m2.2.2.2.4" xref="S2.SS3.p1.2.m2.2.2.2.4.cmml">f</mi><mo lspace="0em" rspace="0em" id="S2.SS3.p1.2.m2.2.2.2.3" xref="S2.SS3.p1.2.m2.2.2.2.3.cmml">​</mo><mrow id="S2.SS3.p1.2.m2.2.2.2.2.2" xref="S2.SS3.p1.2.m2.2.2.2.2.3.cmml"><mo stretchy="false" id="S2.SS3.p1.2.m2.2.2.2.2.2.3" xref="S2.SS3.p1.2.m2.2.2.2.2.3.cmml">(</mo><msub id="S2.SS3.p1.2.m2.1.1.1.1.1.1" xref="S2.SS3.p1.2.m2.1.1.1.1.1.1.cmml"><mi id="S2.SS3.p1.2.m2.1.1.1.1.1.1.2" xref="S2.SS3.p1.2.m2.1.1.1.1.1.1.2.cmml">x</mi><mi id="S2.SS3.p1.2.m2.1.1.1.1.1.1.3" xref="S2.SS3.p1.2.m2.1.1.1.1.1.1.3.cmml">t</mi></msub><mo id="S2.SS3.p1.2.m2.2.2.2.2.2.4" xref="S2.SS3.p1.2.m2.2.2.2.2.3.cmml">;</mo><msub id="S2.SS3.p1.2.m2.2.2.2.2.2.2" xref="S2.SS3.p1.2.m2.2.2.2.2.2.2.cmml"><mi id="S2.SS3.p1.2.m2.2.2.2.2.2.2.2" xref="S2.SS3.p1.2.m2.2.2.2.2.2.2.2.cmml">W</mi><mi id="S2.SS3.p1.2.m2.2.2.2.2.2.2.3" xref="S2.SS3.p1.2.m2.2.2.2.2.2.2.3.cmml">t</mi></msub><mo stretchy="false" id="S2.SS3.p1.2.m2.2.2.2.2.2.5" xref="S2.SS3.p1.2.m2.2.2.2.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p1.2.m2.2b"><apply id="S2.SS3.p1.2.m2.2.2.cmml" xref="S2.SS3.p1.2.m2.2.2"><eq id="S2.SS3.p1.2.m2.2.2.3.cmml" xref="S2.SS3.p1.2.m2.2.2.3"></eq><apply id="S2.SS3.p1.2.m2.2.2.4.cmml" xref="S2.SS3.p1.2.m2.2.2.4"><csymbol cd="ambiguous" id="S2.SS3.p1.2.m2.2.2.4.1.cmml" xref="S2.SS3.p1.2.m2.2.2.4">subscript</csymbol><ci id="S2.SS3.p1.2.m2.2.2.4.2.cmml" xref="S2.SS3.p1.2.m2.2.2.4.2">𝑧</ci><ci id="S2.SS3.p1.2.m2.2.2.4.3.cmml" xref="S2.SS3.p1.2.m2.2.2.4.3">𝑡</ci></apply><apply id="S2.SS3.p1.2.m2.2.2.2.cmml" xref="S2.SS3.p1.2.m2.2.2.2"><times id="S2.SS3.p1.2.m2.2.2.2.3.cmml" xref="S2.SS3.p1.2.m2.2.2.2.3"></times><ci id="S2.SS3.p1.2.m2.2.2.2.4.cmml" xref="S2.SS3.p1.2.m2.2.2.2.4">𝑓</ci><list id="S2.SS3.p1.2.m2.2.2.2.2.3.cmml" xref="S2.SS3.p1.2.m2.2.2.2.2.2"><apply id="S2.SS3.p1.2.m2.1.1.1.1.1.1.cmml" xref="S2.SS3.p1.2.m2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p1.2.m2.1.1.1.1.1.1.1.cmml" xref="S2.SS3.p1.2.m2.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS3.p1.2.m2.1.1.1.1.1.1.2.cmml" xref="S2.SS3.p1.2.m2.1.1.1.1.1.1.2">𝑥</ci><ci id="S2.SS3.p1.2.m2.1.1.1.1.1.1.3.cmml" xref="S2.SS3.p1.2.m2.1.1.1.1.1.1.3">𝑡</ci></apply><apply id="S2.SS3.p1.2.m2.2.2.2.2.2.2.cmml" xref="S2.SS3.p1.2.m2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS3.p1.2.m2.2.2.2.2.2.2.1.cmml" xref="S2.SS3.p1.2.m2.2.2.2.2.2.2">subscript</csymbol><ci id="S2.SS3.p1.2.m2.2.2.2.2.2.2.2.cmml" xref="S2.SS3.p1.2.m2.2.2.2.2.2.2.2">𝑊</ci><ci id="S2.SS3.p1.2.m2.2.2.2.2.2.2.3.cmml" xref="S2.SS3.p1.2.m2.2.2.2.2.2.2.3">𝑡</ci></apply></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p1.2.m2.2c">z_{t}=f(x_{t};W_{t})</annotation></semantics></math> to perform well on language modeling.
Instead of handcrafting a self-supervised task from human priors, we take a more end-to-end approach – directly optimizing the self-supervised task for the final goal of next-token prediction.</p>
</div>
<div id="S2.SS3.p2" class="ltx_para">
<p id="S2.SS3.p2.5" class="ltx_p">Concretely, we learn the self-supervised task as part of the outer loop.
Starting from the naive reconstruction task in Equation <a href="#S2.E3" title="In 2.1 TTT as updating a hidden state ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, we add some outer-loop parameters to make this task learnable.
In Subsection <a href="#S2.SS1" title="2.1 TTT as updating a hidden state ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.1</span></a>, we did not specify the corruption that produces <math id="S2.SS3.p2.1.m1.1" class="ltx_Math" alttext="\tilde{x}_{t}" display="inline"><semantics id="S2.SS3.p2.1.m1.1a"><msub id="S2.SS3.p2.1.m1.1.1" xref="S2.SS3.p2.1.m1.1.1.cmml"><mover accent="true" id="S2.SS3.p2.1.m1.1.1.2" xref="S2.SS3.p2.1.m1.1.1.2.cmml"><mi id="S2.SS3.p2.1.m1.1.1.2.2" xref="S2.SS3.p2.1.m1.1.1.2.2.cmml">x</mi><mo id="S2.SS3.p2.1.m1.1.1.2.1" xref="S2.SS3.p2.1.m1.1.1.2.1.cmml">~</mo></mover><mi id="S2.SS3.p2.1.m1.1.1.3" xref="S2.SS3.p2.1.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.1.m1.1b"><apply id="S2.SS3.p2.1.m1.1.1.cmml" xref="S2.SS3.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.1.m1.1.1.1.cmml" xref="S2.SS3.p2.1.m1.1.1">subscript</csymbol><apply id="S2.SS3.p2.1.m1.1.1.2.cmml" xref="S2.SS3.p2.1.m1.1.1.2"><ci id="S2.SS3.p2.1.m1.1.1.2.1.cmml" xref="S2.SS3.p2.1.m1.1.1.2.1">~</ci><ci id="S2.SS3.p2.1.m1.1.1.2.2.cmml" xref="S2.SS3.p2.1.m1.1.1.2.2">𝑥</ci></apply><ci id="S2.SS3.p2.1.m1.1.1.3.cmml" xref="S2.SS3.p2.1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.1.m1.1c">\tilde{x}_{t}</annotation></semantics></math> from <math id="S2.SS3.p2.2.m2.1" class="ltx_Math" alttext="x_{t}" display="inline"><semantics id="S2.SS3.p2.2.m2.1a"><msub id="S2.SS3.p2.2.m2.1.1" xref="S2.SS3.p2.2.m2.1.1.cmml"><mi id="S2.SS3.p2.2.m2.1.1.2" xref="S2.SS3.p2.2.m2.1.1.2.cmml">x</mi><mi id="S2.SS3.p2.2.m2.1.1.3" xref="S2.SS3.p2.2.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.2.m2.1b"><apply id="S2.SS3.p2.2.m2.1.1.cmml" xref="S2.SS3.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.2.m2.1.1.1.cmml" xref="S2.SS3.p2.2.m2.1.1">subscript</csymbol><ci id="S2.SS3.p2.2.m2.1.1.2.cmml" xref="S2.SS3.p2.2.m2.1.1.2">𝑥</ci><ci id="S2.SS3.p2.2.m2.1.1.3.cmml" xref="S2.SS3.p2.2.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.2.m2.1c">x_{t}</annotation></semantics></math>. One design is to make it a low-rank projection <math id="S2.SS3.p2.3.m3.1" class="ltx_Math" alttext="\tilde{x}_{t}=\theta_{K}x_{t}" display="inline"><semantics id="S2.SS3.p2.3.m3.1a"><mrow id="S2.SS3.p2.3.m3.1.1" xref="S2.SS3.p2.3.m3.1.1.cmml"><msub id="S2.SS3.p2.3.m3.1.1.2" xref="S2.SS3.p2.3.m3.1.1.2.cmml"><mover accent="true" id="S2.SS3.p2.3.m3.1.1.2.2" xref="S2.SS3.p2.3.m3.1.1.2.2.cmml"><mi id="S2.SS3.p2.3.m3.1.1.2.2.2" xref="S2.SS3.p2.3.m3.1.1.2.2.2.cmml">x</mi><mo id="S2.SS3.p2.3.m3.1.1.2.2.1" xref="S2.SS3.p2.3.m3.1.1.2.2.1.cmml">~</mo></mover><mi id="S2.SS3.p2.3.m3.1.1.2.3" xref="S2.SS3.p2.3.m3.1.1.2.3.cmml">t</mi></msub><mo id="S2.SS3.p2.3.m3.1.1.1" xref="S2.SS3.p2.3.m3.1.1.1.cmml">=</mo><mrow id="S2.SS3.p2.3.m3.1.1.3" xref="S2.SS3.p2.3.m3.1.1.3.cmml"><msub id="S2.SS3.p2.3.m3.1.1.3.2" xref="S2.SS3.p2.3.m3.1.1.3.2.cmml"><mi id="S2.SS3.p2.3.m3.1.1.3.2.2" xref="S2.SS3.p2.3.m3.1.1.3.2.2.cmml">θ</mi><mi id="S2.SS3.p2.3.m3.1.1.3.2.3" xref="S2.SS3.p2.3.m3.1.1.3.2.3.cmml">K</mi></msub><mo lspace="0em" rspace="0em" id="S2.SS3.p2.3.m3.1.1.3.1" xref="S2.SS3.p2.3.m3.1.1.3.1.cmml">​</mo><msub id="S2.SS3.p2.3.m3.1.1.3.3" xref="S2.SS3.p2.3.m3.1.1.3.3.cmml"><mi id="S2.SS3.p2.3.m3.1.1.3.3.2" xref="S2.SS3.p2.3.m3.1.1.3.3.2.cmml">x</mi><mi id="S2.SS3.p2.3.m3.1.1.3.3.3" xref="S2.SS3.p2.3.m3.1.1.3.3.3.cmml">t</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.3.m3.1b"><apply id="S2.SS3.p2.3.m3.1.1.cmml" xref="S2.SS3.p2.3.m3.1.1"><eq id="S2.SS3.p2.3.m3.1.1.1.cmml" xref="S2.SS3.p2.3.m3.1.1.1"></eq><apply id="S2.SS3.p2.3.m3.1.1.2.cmml" xref="S2.SS3.p2.3.m3.1.1.2"><csymbol cd="ambiguous" id="S2.SS3.p2.3.m3.1.1.2.1.cmml" xref="S2.SS3.p2.3.m3.1.1.2">subscript</csymbol><apply id="S2.SS3.p2.3.m3.1.1.2.2.cmml" xref="S2.SS3.p2.3.m3.1.1.2.2"><ci id="S2.SS3.p2.3.m3.1.1.2.2.1.cmml" xref="S2.SS3.p2.3.m3.1.1.2.2.1">~</ci><ci id="S2.SS3.p2.3.m3.1.1.2.2.2.cmml" xref="S2.SS3.p2.3.m3.1.1.2.2.2">𝑥</ci></apply><ci id="S2.SS3.p2.3.m3.1.1.2.3.cmml" xref="S2.SS3.p2.3.m3.1.1.2.3">𝑡</ci></apply><apply id="S2.SS3.p2.3.m3.1.1.3.cmml" xref="S2.SS3.p2.3.m3.1.1.3"><times id="S2.SS3.p2.3.m3.1.1.3.1.cmml" xref="S2.SS3.p2.3.m3.1.1.3.1"></times><apply id="S2.SS3.p2.3.m3.1.1.3.2.cmml" xref="S2.SS3.p2.3.m3.1.1.3.2"><csymbol cd="ambiguous" id="S2.SS3.p2.3.m3.1.1.3.2.1.cmml" xref="S2.SS3.p2.3.m3.1.1.3.2">subscript</csymbol><ci id="S2.SS3.p2.3.m3.1.1.3.2.2.cmml" xref="S2.SS3.p2.3.m3.1.1.3.2.2">𝜃</ci><ci id="S2.SS3.p2.3.m3.1.1.3.2.3.cmml" xref="S2.SS3.p2.3.m3.1.1.3.2.3">𝐾</ci></apply><apply id="S2.SS3.p2.3.m3.1.1.3.3.cmml" xref="S2.SS3.p2.3.m3.1.1.3.3"><csymbol cd="ambiguous" id="S2.SS3.p2.3.m3.1.1.3.3.1.cmml" xref="S2.SS3.p2.3.m3.1.1.3.3">subscript</csymbol><ci id="S2.SS3.p2.3.m3.1.1.3.3.2.cmml" xref="S2.SS3.p2.3.m3.1.1.3.3.2">𝑥</ci><ci id="S2.SS3.p2.3.m3.1.1.3.3.3.cmml" xref="S2.SS3.p2.3.m3.1.1.3.3.3">𝑡</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.3.m3.1c">\tilde{x}_{t}=\theta_{K}x_{t}</annotation></semantics></math>, where <math id="S2.SS3.p2.4.m4.1" class="ltx_Math" alttext="\theta_{K}" display="inline"><semantics id="S2.SS3.p2.4.m4.1a"><msub id="S2.SS3.p2.4.m4.1.1" xref="S2.SS3.p2.4.m4.1.1.cmml"><mi id="S2.SS3.p2.4.m4.1.1.2" xref="S2.SS3.p2.4.m4.1.1.2.cmml">θ</mi><mi id="S2.SS3.p2.4.m4.1.1.3" xref="S2.SS3.p2.4.m4.1.1.3.cmml">K</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.4.m4.1b"><apply id="S2.SS3.p2.4.m4.1.1.cmml" xref="S2.SS3.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS3.p2.4.m4.1.1.1.cmml" xref="S2.SS3.p2.4.m4.1.1">subscript</csymbol><ci id="S2.SS3.p2.4.m4.1.1.2.cmml" xref="S2.SS3.p2.4.m4.1.1.2">𝜃</ci><ci id="S2.SS3.p2.4.m4.1.1.3.cmml" xref="S2.SS3.p2.4.m4.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.4.m4.1c">\theta_{K}</annotation></semantics></math> is a learnable matrix.<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span>
The subscript <math id="footnote4.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="footnote4.m1.1b"><mi id="footnote4.m1.1.1" xref="footnote4.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="footnote4.m1.1c"><ci id="footnote4.m1.1.1.cmml" xref="footnote4.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote4.m1.1d">K</annotation></semantics></math> hints at a connection to self-attention, as we will establish in Subsection <a href="#S2.SS6" title="2.6 Theoretical equivalences ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.6</span></a>.
</span></span></span>
Following the terminology of multi-view reconstruction, <math id="S2.SS3.p2.5.m5.1" class="ltx_Math" alttext="\theta_{K}x_{t}" display="inline"><semantics id="S2.SS3.p2.5.m5.1a"><mrow id="S2.SS3.p2.5.m5.1.1" xref="S2.SS3.p2.5.m5.1.1.cmml"><msub id="S2.SS3.p2.5.m5.1.1.2" xref="S2.SS3.p2.5.m5.1.1.2.cmml"><mi id="S2.SS3.p2.5.m5.1.1.2.2" xref="S2.SS3.p2.5.m5.1.1.2.2.cmml">θ</mi><mi id="S2.SS3.p2.5.m5.1.1.2.3" xref="S2.SS3.p2.5.m5.1.1.2.3.cmml">K</mi></msub><mo lspace="0em" rspace="0em" id="S2.SS3.p2.5.m5.1.1.1" xref="S2.SS3.p2.5.m5.1.1.1.cmml">​</mo><msub id="S2.SS3.p2.5.m5.1.1.3" xref="S2.SS3.p2.5.m5.1.1.3.cmml"><mi id="S2.SS3.p2.5.m5.1.1.3.2" xref="S2.SS3.p2.5.m5.1.1.3.2.cmml">x</mi><mi id="S2.SS3.p2.5.m5.1.1.3.3" xref="S2.SS3.p2.5.m5.1.1.3.3.cmml">t</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p2.5.m5.1b"><apply id="S2.SS3.p2.5.m5.1.1.cmml" xref="S2.SS3.p2.5.m5.1.1"><times id="S2.SS3.p2.5.m5.1.1.1.cmml" xref="S2.SS3.p2.5.m5.1.1.1"></times><apply id="S2.SS3.p2.5.m5.1.1.2.cmml" xref="S2.SS3.p2.5.m5.1.1.2"><csymbol cd="ambiguous" id="S2.SS3.p2.5.m5.1.1.2.1.cmml" xref="S2.SS3.p2.5.m5.1.1.2">subscript</csymbol><ci id="S2.SS3.p2.5.m5.1.1.2.2.cmml" xref="S2.SS3.p2.5.m5.1.1.2.2">𝜃</ci><ci id="S2.SS3.p2.5.m5.1.1.2.3.cmml" xref="S2.SS3.p2.5.m5.1.1.2.3">𝐾</ci></apply><apply id="S2.SS3.p2.5.m5.1.1.3.cmml" xref="S2.SS3.p2.5.m5.1.1.3"><csymbol cd="ambiguous" id="S2.SS3.p2.5.m5.1.1.3.1.cmml" xref="S2.SS3.p2.5.m5.1.1.3">subscript</csymbol><ci id="S2.SS3.p2.5.m5.1.1.3.2.cmml" xref="S2.SS3.p2.5.m5.1.1.3.2">𝑥</ci><ci id="S2.SS3.p2.5.m5.1.1.3.3.cmml" xref="S2.SS3.p2.5.m5.1.1.3.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p2.5.m5.1c">\theta_{K}x_{t}</annotation></semantics></math> is called a <em id="S2.SS3.p2.5.1" class="ltx_emph ltx_font_italic">training view</em> <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib14" title="" class="ltx_ref">14</a>]</cite>.</p>
</div>
<div id="S2.SS3.p3" class="ltx_para">
<p id="S2.SS3.p3.5" class="ltx_p">Moreover, perhaps not all the information in <math id="S2.SS3.p3.1.m1.1" class="ltx_Math" alttext="x_{t}" display="inline"><semantics id="S2.SS3.p3.1.m1.1a"><msub id="S2.SS3.p3.1.m1.1.1" xref="S2.SS3.p3.1.m1.1.1.cmml"><mi id="S2.SS3.p3.1.m1.1.1.2" xref="S2.SS3.p3.1.m1.1.1.2.cmml">x</mi><mi id="S2.SS3.p3.1.m1.1.1.3" xref="S2.SS3.p3.1.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.1.m1.1b"><apply id="S2.SS3.p3.1.m1.1.1.cmml" xref="S2.SS3.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p3.1.m1.1.1.1.cmml" xref="S2.SS3.p3.1.m1.1.1">subscript</csymbol><ci id="S2.SS3.p3.1.m1.1.1.2.cmml" xref="S2.SS3.p3.1.m1.1.1.2">𝑥</ci><ci id="S2.SS3.p3.1.m1.1.1.3.cmml" xref="S2.SS3.p3.1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.1.m1.1c">x_{t}</annotation></semantics></math> is worth remembering, so the reconstruction label can be another low-rank projection <math id="S2.SS3.p3.2.m2.1" class="ltx_Math" alttext="\theta_{V}x_{t}" display="inline"><semantics id="S2.SS3.p3.2.m2.1a"><mrow id="S2.SS3.p3.2.m2.1.1" xref="S2.SS3.p3.2.m2.1.1.cmml"><msub id="S2.SS3.p3.2.m2.1.1.2" xref="S2.SS3.p3.2.m2.1.1.2.cmml"><mi id="S2.SS3.p3.2.m2.1.1.2.2" xref="S2.SS3.p3.2.m2.1.1.2.2.cmml">θ</mi><mi id="S2.SS3.p3.2.m2.1.1.2.3" xref="S2.SS3.p3.2.m2.1.1.2.3.cmml">V</mi></msub><mo lspace="0em" rspace="0em" id="S2.SS3.p3.2.m2.1.1.1" xref="S2.SS3.p3.2.m2.1.1.1.cmml">​</mo><msub id="S2.SS3.p3.2.m2.1.1.3" xref="S2.SS3.p3.2.m2.1.1.3.cmml"><mi id="S2.SS3.p3.2.m2.1.1.3.2" xref="S2.SS3.p3.2.m2.1.1.3.2.cmml">x</mi><mi id="S2.SS3.p3.2.m2.1.1.3.3" xref="S2.SS3.p3.2.m2.1.1.3.3.cmml">t</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.2.m2.1b"><apply id="S2.SS3.p3.2.m2.1.1.cmml" xref="S2.SS3.p3.2.m2.1.1"><times id="S2.SS3.p3.2.m2.1.1.1.cmml" xref="S2.SS3.p3.2.m2.1.1.1"></times><apply id="S2.SS3.p3.2.m2.1.1.2.cmml" xref="S2.SS3.p3.2.m2.1.1.2"><csymbol cd="ambiguous" id="S2.SS3.p3.2.m2.1.1.2.1.cmml" xref="S2.SS3.p3.2.m2.1.1.2">subscript</csymbol><ci id="S2.SS3.p3.2.m2.1.1.2.2.cmml" xref="S2.SS3.p3.2.m2.1.1.2.2">𝜃</ci><ci id="S2.SS3.p3.2.m2.1.1.2.3.cmml" xref="S2.SS3.p3.2.m2.1.1.2.3">𝑉</ci></apply><apply id="S2.SS3.p3.2.m2.1.1.3.cmml" xref="S2.SS3.p3.2.m2.1.1.3"><csymbol cd="ambiguous" id="S2.SS3.p3.2.m2.1.1.3.1.cmml" xref="S2.SS3.p3.2.m2.1.1.3">subscript</csymbol><ci id="S2.SS3.p3.2.m2.1.1.3.2.cmml" xref="S2.SS3.p3.2.m2.1.1.3.2">𝑥</ci><ci id="S2.SS3.p3.2.m2.1.1.3.3.cmml" xref="S2.SS3.p3.2.m2.1.1.3.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.2.m2.1c">\theta_{V}x_{t}</annotation></semantics></math> instead of <math id="S2.SS3.p3.3.m3.1" class="ltx_Math" alttext="x_{t}" display="inline"><semantics id="S2.SS3.p3.3.m3.1a"><msub id="S2.SS3.p3.3.m3.1.1" xref="S2.SS3.p3.3.m3.1.1.cmml"><mi id="S2.SS3.p3.3.m3.1.1.2" xref="S2.SS3.p3.3.m3.1.1.2.cmml">x</mi><mi id="S2.SS3.p3.3.m3.1.1.3" xref="S2.SS3.p3.3.m3.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.3.m3.1b"><apply id="S2.SS3.p3.3.m3.1.1.cmml" xref="S2.SS3.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS3.p3.3.m3.1.1.1.cmml" xref="S2.SS3.p3.3.m3.1.1">subscript</csymbol><ci id="S2.SS3.p3.3.m3.1.1.2.cmml" xref="S2.SS3.p3.3.m3.1.1.2">𝑥</ci><ci id="S2.SS3.p3.3.m3.1.1.3.cmml" xref="S2.SS3.p3.3.m3.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.3.m3.1c">x_{t}</annotation></semantics></math>.
Here <math id="S2.SS3.p3.4.m4.1" class="ltx_Math" alttext="\theta_{V}x_{t}" display="inline"><semantics id="S2.SS3.p3.4.m4.1a"><mrow id="S2.SS3.p3.4.m4.1.1" xref="S2.SS3.p3.4.m4.1.1.cmml"><msub id="S2.SS3.p3.4.m4.1.1.2" xref="S2.SS3.p3.4.m4.1.1.2.cmml"><mi id="S2.SS3.p3.4.m4.1.1.2.2" xref="S2.SS3.p3.4.m4.1.1.2.2.cmml">θ</mi><mi id="S2.SS3.p3.4.m4.1.1.2.3" xref="S2.SS3.p3.4.m4.1.1.2.3.cmml">V</mi></msub><mo lspace="0em" rspace="0em" id="S2.SS3.p3.4.m4.1.1.1" xref="S2.SS3.p3.4.m4.1.1.1.cmml">​</mo><msub id="S2.SS3.p3.4.m4.1.1.3" xref="S2.SS3.p3.4.m4.1.1.3.cmml"><mi id="S2.SS3.p3.4.m4.1.1.3.2" xref="S2.SS3.p3.4.m4.1.1.3.2.cmml">x</mi><mi id="S2.SS3.p3.4.m4.1.1.3.3" xref="S2.SS3.p3.4.m4.1.1.3.3.cmml">t</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.4.m4.1b"><apply id="S2.SS3.p3.4.m4.1.1.cmml" xref="S2.SS3.p3.4.m4.1.1"><times id="S2.SS3.p3.4.m4.1.1.1.cmml" xref="S2.SS3.p3.4.m4.1.1.1"></times><apply id="S2.SS3.p3.4.m4.1.1.2.cmml" xref="S2.SS3.p3.4.m4.1.1.2"><csymbol cd="ambiguous" id="S2.SS3.p3.4.m4.1.1.2.1.cmml" xref="S2.SS3.p3.4.m4.1.1.2">subscript</csymbol><ci id="S2.SS3.p3.4.m4.1.1.2.2.cmml" xref="S2.SS3.p3.4.m4.1.1.2.2">𝜃</ci><ci id="S2.SS3.p3.4.m4.1.1.2.3.cmml" xref="S2.SS3.p3.4.m4.1.1.2.3">𝑉</ci></apply><apply id="S2.SS3.p3.4.m4.1.1.3.cmml" xref="S2.SS3.p3.4.m4.1.1.3"><csymbol cd="ambiguous" id="S2.SS3.p3.4.m4.1.1.3.1.cmml" xref="S2.SS3.p3.4.m4.1.1.3">subscript</csymbol><ci id="S2.SS3.p3.4.m4.1.1.3.2.cmml" xref="S2.SS3.p3.4.m4.1.1.3.2">𝑥</ci><ci id="S2.SS3.p3.4.m4.1.1.3.3.cmml" xref="S2.SS3.p3.4.m4.1.1.3.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.4.m4.1c">\theta_{V}x_{t}</annotation></semantics></math> is called the <em id="S2.SS3.p3.5.1" class="ltx_emph ltx_font_italic">label view</em>, where <math id="S2.SS3.p3.5.m5.1" class="ltx_Math" alttext="\theta_{V}" display="inline"><semantics id="S2.SS3.p3.5.m5.1a"><msub id="S2.SS3.p3.5.m5.1.1" xref="S2.SS3.p3.5.m5.1.1.cmml"><mi id="S2.SS3.p3.5.m5.1.1.2" xref="S2.SS3.p3.5.m5.1.1.2.cmml">θ</mi><mi id="S2.SS3.p3.5.m5.1.1.3" xref="S2.SS3.p3.5.m5.1.1.3.cmml">V</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p3.5.m5.1b"><apply id="S2.SS3.p3.5.m5.1.1.cmml" xref="S2.SS3.p3.5.m5.1.1"><csymbol cd="ambiguous" id="S2.SS3.p3.5.m5.1.1.1.cmml" xref="S2.SS3.p3.5.m5.1.1">subscript</csymbol><ci id="S2.SS3.p3.5.m5.1.1.2.cmml" xref="S2.SS3.p3.5.m5.1.1.2">𝜃</ci><ci id="S2.SS3.p3.5.m5.1.1.3.cmml" xref="S2.SS3.p3.5.m5.1.1.3">𝑉</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p3.5.m5.1c">\theta_{V}</annotation></semantics></math> is also learnable.
In summary, our new self-supervised loss is:</p>
<table id="S2.E4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E4.m1.3" class="ltx_Math" alttext="\ell(W;x_{t})=\big{\|}f\left(\theta_{K}x_{t};W\right)-\theta_{V}x_{t}\big{\|}^{2}." display="block"><semantics id="S2.E4.m1.3a"><mrow id="S2.E4.m1.3.3.1" xref="S2.E4.m1.3.3.1.1.cmml"><mrow id="S2.E4.m1.3.3.1.1" xref="S2.E4.m1.3.3.1.1.cmml"><mrow id="S2.E4.m1.3.3.1.1.1" xref="S2.E4.m1.3.3.1.1.1.cmml"><mi mathvariant="normal" id="S2.E4.m1.3.3.1.1.1.3" xref="S2.E4.m1.3.3.1.1.1.3.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="S2.E4.m1.3.3.1.1.1.2" xref="S2.E4.m1.3.3.1.1.1.2.cmml">​</mo><mrow id="S2.E4.m1.3.3.1.1.1.1.1" xref="S2.E4.m1.3.3.1.1.1.1.2.cmml"><mo stretchy="false" id="S2.E4.m1.3.3.1.1.1.1.1.2" xref="S2.E4.m1.3.3.1.1.1.1.2.cmml">(</mo><mi id="S2.E4.m1.1.1" xref="S2.E4.m1.1.1.cmml">W</mi><mo id="S2.E4.m1.3.3.1.1.1.1.1.3" xref="S2.E4.m1.3.3.1.1.1.1.2.cmml">;</mo><msub id="S2.E4.m1.3.3.1.1.1.1.1.1" xref="S2.E4.m1.3.3.1.1.1.1.1.1.cmml"><mi id="S2.E4.m1.3.3.1.1.1.1.1.1.2" xref="S2.E4.m1.3.3.1.1.1.1.1.1.2.cmml">x</mi><mi id="S2.E4.m1.3.3.1.1.1.1.1.1.3" xref="S2.E4.m1.3.3.1.1.1.1.1.1.3.cmml">t</mi></msub><mo stretchy="false" id="S2.E4.m1.3.3.1.1.1.1.1.4" xref="S2.E4.m1.3.3.1.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="S2.E4.m1.3.3.1.1.3" xref="S2.E4.m1.3.3.1.1.3.cmml">=</mo><msup id="S2.E4.m1.3.3.1.1.2" xref="S2.E4.m1.3.3.1.1.2.cmml"><mrow id="S2.E4.m1.3.3.1.1.2.1.1" xref="S2.E4.m1.3.3.1.1.2.1.2.cmml"><mo maxsize="120%" minsize="120%" id="S2.E4.m1.3.3.1.1.2.1.1.2" xref="S2.E4.m1.3.3.1.1.2.1.2.1.cmml">‖</mo><mrow id="S2.E4.m1.3.3.1.1.2.1.1.1" xref="S2.E4.m1.3.3.1.1.2.1.1.1.cmml"><mrow id="S2.E4.m1.3.3.1.1.2.1.1.1.1" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.cmml"><mi id="S2.E4.m1.3.3.1.1.2.1.1.1.1.3" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.3.cmml">f</mi><mo lspace="0em" rspace="0em" id="S2.E4.m1.3.3.1.1.2.1.1.1.1.2" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.2.cmml">​</mo><mrow id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.2.cmml"><mo id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.2" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.2.cmml">(</mo><mrow id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.cmml"><msub id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.2" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.2.cmml"><mi id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.2.2" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.2.2.cmml">θ</mi><mi id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.2.3" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.2.3.cmml">K</mi></msub><mo lspace="0em" rspace="0em" id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.1" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.1.cmml">​</mo><msub id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.3" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.3.cmml"><mi id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.3.2" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.3.2.cmml">x</mi><mi id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.3.3" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.3.3.cmml">t</mi></msub></mrow><mo id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.3" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.2.cmml">;</mo><mi id="S2.E4.m1.2.2" xref="S2.E4.m1.2.2.cmml">W</mi><mo id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.4" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.2.cmml">)</mo></mrow></mrow><mo id="S2.E4.m1.3.3.1.1.2.1.1.1.2" xref="S2.E4.m1.3.3.1.1.2.1.1.1.2.cmml">−</mo><mrow id="S2.E4.m1.3.3.1.1.2.1.1.1.3" xref="S2.E4.m1.3.3.1.1.2.1.1.1.3.cmml"><msub id="S2.E4.m1.3.3.1.1.2.1.1.1.3.2" xref="S2.E4.m1.3.3.1.1.2.1.1.1.3.2.cmml"><mi id="S2.E4.m1.3.3.1.1.2.1.1.1.3.2.2" xref="S2.E4.m1.3.3.1.1.2.1.1.1.3.2.2.cmml">θ</mi><mi id="S2.E4.m1.3.3.1.1.2.1.1.1.3.2.3" xref="S2.E4.m1.3.3.1.1.2.1.1.1.3.2.3.cmml">V</mi></msub><mo lspace="0em" rspace="0em" id="S2.E4.m1.3.3.1.1.2.1.1.1.3.1" xref="S2.E4.m1.3.3.1.1.2.1.1.1.3.1.cmml">​</mo><msub id="S2.E4.m1.3.3.1.1.2.1.1.1.3.3" xref="S2.E4.m1.3.3.1.1.2.1.1.1.3.3.cmml"><mi id="S2.E4.m1.3.3.1.1.2.1.1.1.3.3.2" xref="S2.E4.m1.3.3.1.1.2.1.1.1.3.3.2.cmml">x</mi><mi id="S2.E4.m1.3.3.1.1.2.1.1.1.3.3.3" xref="S2.E4.m1.3.3.1.1.2.1.1.1.3.3.3.cmml">t</mi></msub></mrow></mrow><mo maxsize="120%" minsize="120%" id="S2.E4.m1.3.3.1.1.2.1.1.3" xref="S2.E4.m1.3.3.1.1.2.1.2.1.cmml">‖</mo></mrow><mn id="S2.E4.m1.3.3.1.1.2.3" xref="S2.E4.m1.3.3.1.1.2.3.cmml">2</mn></msup></mrow><mo lspace="0em" id="S2.E4.m1.3.3.1.2" xref="S2.E4.m1.3.3.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E4.m1.3b"><apply id="S2.E4.m1.3.3.1.1.cmml" xref="S2.E4.m1.3.3.1"><eq id="S2.E4.m1.3.3.1.1.3.cmml" xref="S2.E4.m1.3.3.1.1.3"></eq><apply id="S2.E4.m1.3.3.1.1.1.cmml" xref="S2.E4.m1.3.3.1.1.1"><times id="S2.E4.m1.3.3.1.1.1.2.cmml" xref="S2.E4.m1.3.3.1.1.1.2"></times><ci id="S2.E4.m1.3.3.1.1.1.3.cmml" xref="S2.E4.m1.3.3.1.1.1.3">ℓ</ci><list id="S2.E4.m1.3.3.1.1.1.1.2.cmml" xref="S2.E4.m1.3.3.1.1.1.1.1"><ci id="S2.E4.m1.1.1.cmml" xref="S2.E4.m1.1.1">𝑊</ci><apply id="S2.E4.m1.3.3.1.1.1.1.1.1.cmml" xref="S2.E4.m1.3.3.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E4.m1.3.3.1.1.1.1.1.1.1.cmml" xref="S2.E4.m1.3.3.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E4.m1.3.3.1.1.1.1.1.1.2.cmml" xref="S2.E4.m1.3.3.1.1.1.1.1.1.2">𝑥</ci><ci id="S2.E4.m1.3.3.1.1.1.1.1.1.3.cmml" xref="S2.E4.m1.3.3.1.1.1.1.1.1.3">𝑡</ci></apply></list></apply><apply id="S2.E4.m1.3.3.1.1.2.cmml" xref="S2.E4.m1.3.3.1.1.2"><csymbol cd="ambiguous" id="S2.E4.m1.3.3.1.1.2.2.cmml" xref="S2.E4.m1.3.3.1.1.2">superscript</csymbol><apply id="S2.E4.m1.3.3.1.1.2.1.2.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1"><csymbol cd="latexml" id="S2.E4.m1.3.3.1.1.2.1.2.1.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.2">norm</csymbol><apply id="S2.E4.m1.3.3.1.1.2.1.1.1.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1"><minus id="S2.E4.m1.3.3.1.1.2.1.1.1.2.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.2"></minus><apply id="S2.E4.m1.3.3.1.1.2.1.1.1.1.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1"><times id="S2.E4.m1.3.3.1.1.2.1.1.1.1.2.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.2"></times><ci id="S2.E4.m1.3.3.1.1.2.1.1.1.1.3.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.3">𝑓</ci><list id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.2.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1"><apply id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1"><times id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.1.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.1"></times><apply id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.2.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.2.1.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.2.2.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.2.2">𝜃</ci><ci id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.2.3.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.2.3">𝐾</ci></apply><apply id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.3.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.3.1.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.3.2.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.3.2">𝑥</ci><ci id="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.3.3.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.1.1.1.1.3.3">𝑡</ci></apply></apply><ci id="S2.E4.m1.2.2.cmml" xref="S2.E4.m1.2.2">𝑊</ci></list></apply><apply id="S2.E4.m1.3.3.1.1.2.1.1.1.3.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.3"><times id="S2.E4.m1.3.3.1.1.2.1.1.1.3.1.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.3.1"></times><apply id="S2.E4.m1.3.3.1.1.2.1.1.1.3.2.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.3.2"><csymbol cd="ambiguous" id="S2.E4.m1.3.3.1.1.2.1.1.1.3.2.1.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.3.2">subscript</csymbol><ci id="S2.E4.m1.3.3.1.1.2.1.1.1.3.2.2.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.3.2.2">𝜃</ci><ci id="S2.E4.m1.3.3.1.1.2.1.1.1.3.2.3.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.3.2.3">𝑉</ci></apply><apply id="S2.E4.m1.3.3.1.1.2.1.1.1.3.3.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.3.3"><csymbol cd="ambiguous" id="S2.E4.m1.3.3.1.1.2.1.1.1.3.3.1.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.3.3">subscript</csymbol><ci id="S2.E4.m1.3.3.1.1.2.1.1.1.3.3.2.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.3.3.2">𝑥</ci><ci id="S2.E4.m1.3.3.1.1.2.1.1.1.3.3.3.cmml" xref="S2.E4.m1.3.3.1.1.2.1.1.1.3.3.3">𝑡</ci></apply></apply></apply></apply><cn type="integer" id="S2.E4.m1.3.3.1.1.2.3.cmml" xref="S2.E4.m1.3.3.1.1.2.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E4.m1.3c">\ell(W;x_{t})=\big{\|}f\left(\theta_{K}x_{t};W\right)-\theta_{V}x_{t}\big{\|}^{2}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
</div>
<div id="S2.SS3.p4" class="ltx_para">
<p id="S2.SS3.p4.10" class="ltx_p">Since both <math id="S2.SS3.p4.1.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.SS3.p4.1.m1.1a"><mi id="S2.SS3.p4.1.m1.1.1" xref="S2.SS3.p4.1.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p4.1.m1.1b"><ci id="S2.SS3.p4.1.m1.1.1.cmml" xref="S2.SS3.p4.1.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p4.1.m1.1c">W</annotation></semantics></math> and various <math id="S2.SS3.p4.2.m2.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="S2.SS3.p4.2.m2.1a"><mi id="S2.SS3.p4.2.m2.1.1" xref="S2.SS3.p4.2.m2.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p4.2.m2.1b"><ci id="S2.SS3.p4.2.m2.1.1.cmml" xref="S2.SS3.p4.2.m2.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p4.2.m2.1c">\theta</annotation></semantics></math>s appear together in Equation <a href="#S2.E4" title="In 2.3 Learning a self-supervised task for TTT ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, we emphasize again their difference in nature.
In the inner loop, only <math id="S2.SS3.p4.3.m3.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.SS3.p4.3.m3.1a"><mi id="S2.SS3.p4.3.m3.1.1" xref="S2.SS3.p4.3.m3.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p4.3.m3.1b"><ci id="S2.SS3.p4.3.m3.1.1.cmml" xref="S2.SS3.p4.3.m3.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p4.3.m3.1c">W</annotation></semantics></math> is optimized, therefore written as an argument of <math id="S2.SS3.p4.4.m4.1" class="ltx_Math" alttext="\ell" display="inline"><semantics id="S2.SS3.p4.4.m4.1a"><mi mathvariant="normal" id="S2.SS3.p4.4.m4.1.1" xref="S2.SS3.p4.4.m4.1.1.cmml">ℓ</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p4.4.m4.1b"><ci id="S2.SS3.p4.4.m4.1.1.cmml" xref="S2.SS3.p4.4.m4.1.1">ℓ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p4.4.m4.1c">\ell</annotation></semantics></math>; the <math id="S2.SS3.p4.5.m5.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="S2.SS3.p4.5.m5.1a"><mi id="S2.SS3.p4.5.m5.1.1" xref="S2.SS3.p4.5.m5.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p4.5.m5.1b"><ci id="S2.SS3.p4.5.m5.1.1.cmml" xref="S2.SS3.p4.5.m5.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p4.5.m5.1c">\theta</annotation></semantics></math>s are "hyper-parameters" of this loss function.
In the outer loop, <math id="S2.SS3.p4.6.m6.3" class="ltx_Math" alttext="\theta_{K},\theta_{V},\theta_{Q}" display="inline"><semantics id="S2.SS3.p4.6.m6.3a"><mrow id="S2.SS3.p4.6.m6.3.3.3" xref="S2.SS3.p4.6.m6.3.3.4.cmml"><msub id="S2.SS3.p4.6.m6.1.1.1.1" xref="S2.SS3.p4.6.m6.1.1.1.1.cmml"><mi id="S2.SS3.p4.6.m6.1.1.1.1.2" xref="S2.SS3.p4.6.m6.1.1.1.1.2.cmml">θ</mi><mi id="S2.SS3.p4.6.m6.1.1.1.1.3" xref="S2.SS3.p4.6.m6.1.1.1.1.3.cmml">K</mi></msub><mo id="S2.SS3.p4.6.m6.3.3.3.4" xref="S2.SS3.p4.6.m6.3.3.4.cmml">,</mo><msub id="S2.SS3.p4.6.m6.2.2.2.2" xref="S2.SS3.p4.6.m6.2.2.2.2.cmml"><mi id="S2.SS3.p4.6.m6.2.2.2.2.2" xref="S2.SS3.p4.6.m6.2.2.2.2.2.cmml">θ</mi><mi id="S2.SS3.p4.6.m6.2.2.2.2.3" xref="S2.SS3.p4.6.m6.2.2.2.2.3.cmml">V</mi></msub><mo id="S2.SS3.p4.6.m6.3.3.3.5" xref="S2.SS3.p4.6.m6.3.3.4.cmml">,</mo><msub id="S2.SS3.p4.6.m6.3.3.3.3" xref="S2.SS3.p4.6.m6.3.3.3.3.cmml"><mi id="S2.SS3.p4.6.m6.3.3.3.3.2" xref="S2.SS3.p4.6.m6.3.3.3.3.2.cmml">θ</mi><mi id="S2.SS3.p4.6.m6.3.3.3.3.3" xref="S2.SS3.p4.6.m6.3.3.3.3.3.cmml">Q</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p4.6.m6.3b"><list id="S2.SS3.p4.6.m6.3.3.4.cmml" xref="S2.SS3.p4.6.m6.3.3.3"><apply id="S2.SS3.p4.6.m6.1.1.1.1.cmml" xref="S2.SS3.p4.6.m6.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p4.6.m6.1.1.1.1.1.cmml" xref="S2.SS3.p4.6.m6.1.1.1.1">subscript</csymbol><ci id="S2.SS3.p4.6.m6.1.1.1.1.2.cmml" xref="S2.SS3.p4.6.m6.1.1.1.1.2">𝜃</ci><ci id="S2.SS3.p4.6.m6.1.1.1.1.3.cmml" xref="S2.SS3.p4.6.m6.1.1.1.1.3">𝐾</ci></apply><apply id="S2.SS3.p4.6.m6.2.2.2.2.cmml" xref="S2.SS3.p4.6.m6.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS3.p4.6.m6.2.2.2.2.1.cmml" xref="S2.SS3.p4.6.m6.2.2.2.2">subscript</csymbol><ci id="S2.SS3.p4.6.m6.2.2.2.2.2.cmml" xref="S2.SS3.p4.6.m6.2.2.2.2.2">𝜃</ci><ci id="S2.SS3.p4.6.m6.2.2.2.2.3.cmml" xref="S2.SS3.p4.6.m6.2.2.2.2.3">𝑉</ci></apply><apply id="S2.SS3.p4.6.m6.3.3.3.3.cmml" xref="S2.SS3.p4.6.m6.3.3.3.3"><csymbol cd="ambiguous" id="S2.SS3.p4.6.m6.3.3.3.3.1.cmml" xref="S2.SS3.p4.6.m6.3.3.3.3">subscript</csymbol><ci id="S2.SS3.p4.6.m6.3.3.3.3.2.cmml" xref="S2.SS3.p4.6.m6.3.3.3.3.2">𝜃</ci><ci id="S2.SS3.p4.6.m6.3.3.3.3.3.cmml" xref="S2.SS3.p4.6.m6.3.3.3.3.3">𝑄</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p4.6.m6.3c">\theta_{K},\theta_{V},\theta_{Q}</annotation></semantics></math> are optimized alongside <math id="S2.SS3.p4.7.m7.1" class="ltx_Math" alttext="\theta_{\text{rest}}" display="inline"><semantics id="S2.SS3.p4.7.m7.1a"><msub id="S2.SS3.p4.7.m7.1.1" xref="S2.SS3.p4.7.m7.1.1.cmml"><mi id="S2.SS3.p4.7.m7.1.1.2" xref="S2.SS3.p4.7.m7.1.1.2.cmml">θ</mi><mtext id="S2.SS3.p4.7.m7.1.1.3" xref="S2.SS3.p4.7.m7.1.1.3a.cmml">rest</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p4.7.m7.1b"><apply id="S2.SS3.p4.7.m7.1.1.cmml" xref="S2.SS3.p4.7.m7.1.1"><csymbol cd="ambiguous" id="S2.SS3.p4.7.m7.1.1.1.cmml" xref="S2.SS3.p4.7.m7.1.1">subscript</csymbol><ci id="S2.SS3.p4.7.m7.1.1.2.cmml" xref="S2.SS3.p4.7.m7.1.1.2">𝜃</ci><ci id="S2.SS3.p4.7.m7.1.1.3a.cmml" xref="S2.SS3.p4.7.m7.1.1.3"><mtext mathsize="70%" id="S2.SS3.p4.7.m7.1.1.3.cmml" xref="S2.SS3.p4.7.m7.1.1.3">rest</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p4.7.m7.1c">\theta_{\text{rest}}</annotation></semantics></math>, and <math id="S2.SS3.p4.8.m8.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.SS3.p4.8.m8.1a"><mi id="S2.SS3.p4.8.m8.1.1" xref="S2.SS3.p4.8.m8.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.SS3.p4.8.m8.1b"><ci id="S2.SS3.p4.8.m8.1.1.cmml" xref="S2.SS3.p4.8.m8.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p4.8.m8.1c">W</annotation></semantics></math> is merely a hidden state, not a parameter.
Figure <a href="#S2.F6" title="Figure 6 ‣ 2.2 Training a network with TTT layers ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> illustrates this difference with code, where <math id="S2.SS3.p4.9.m9.1" class="ltx_Math" alttext="\theta_{K}" display="inline"><semantics id="S2.SS3.p4.9.m9.1a"><msub id="S2.SS3.p4.9.m9.1.1" xref="S2.SS3.p4.9.m9.1.1.cmml"><mi id="S2.SS3.p4.9.m9.1.1.2" xref="S2.SS3.p4.9.m9.1.1.2.cmml">θ</mi><mi id="S2.SS3.p4.9.m9.1.1.3" xref="S2.SS3.p4.9.m9.1.1.3.cmml">K</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p4.9.m9.1b"><apply id="S2.SS3.p4.9.m9.1.1.cmml" xref="S2.SS3.p4.9.m9.1.1"><csymbol cd="ambiguous" id="S2.SS3.p4.9.m9.1.1.1.cmml" xref="S2.SS3.p4.9.m9.1.1">subscript</csymbol><ci id="S2.SS3.p4.9.m9.1.1.2.cmml" xref="S2.SS3.p4.9.m9.1.1.2">𝜃</ci><ci id="S2.SS3.p4.9.m9.1.1.3.cmml" xref="S2.SS3.p4.9.m9.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p4.9.m9.1c">\theta_{K}</annotation></semantics></math> and <math id="S2.SS3.p4.10.m10.1" class="ltx_Math" alttext="\theta_{V}" display="inline"><semantics id="S2.SS3.p4.10.m10.1a"><msub id="S2.SS3.p4.10.m10.1.1" xref="S2.SS3.p4.10.m10.1.1.cmml"><mi id="S2.SS3.p4.10.m10.1.1.2" xref="S2.SS3.p4.10.m10.1.1.2.cmml">θ</mi><mi id="S2.SS3.p4.10.m10.1.1.3" xref="S2.SS3.p4.10.m10.1.1.3.cmml">V</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p4.10.m10.1b"><apply id="S2.SS3.p4.10.m10.1.1.cmml" xref="S2.SS3.p4.10.m10.1.1"><csymbol cd="ambiguous" id="S2.SS3.p4.10.m10.1.1.1.cmml" xref="S2.SS3.p4.10.m10.1.1">subscript</csymbol><ci id="S2.SS3.p4.10.m10.1.1.2.cmml" xref="S2.SS3.p4.10.m10.1.1.2">𝜃</ci><ci id="S2.SS3.p4.10.m10.1.1.3.cmml" xref="S2.SS3.p4.10.m10.1.1.3">𝑉</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p4.10.m10.1c">\theta_{V}</annotation></semantics></math> are implemented as parameters of the TTT layer,
analogous to the Key and Value parameters of self-attention.</p>
</div>
<div id="S2.SS3.p5" class="ltx_para">
<p id="S2.SS3.p5.3" class="ltx_p">Lastly, the training view <math id="S2.SS3.p5.1.m1.1" class="ltx_Math" alttext="\theta_{K}x_{t}" display="inline"><semantics id="S2.SS3.p5.1.m1.1a"><mrow id="S2.SS3.p5.1.m1.1.1" xref="S2.SS3.p5.1.m1.1.1.cmml"><msub id="S2.SS3.p5.1.m1.1.1.2" xref="S2.SS3.p5.1.m1.1.1.2.cmml"><mi id="S2.SS3.p5.1.m1.1.1.2.2" xref="S2.SS3.p5.1.m1.1.1.2.2.cmml">θ</mi><mi id="S2.SS3.p5.1.m1.1.1.2.3" xref="S2.SS3.p5.1.m1.1.1.2.3.cmml">K</mi></msub><mo lspace="0em" rspace="0em" id="S2.SS3.p5.1.m1.1.1.1" xref="S2.SS3.p5.1.m1.1.1.1.cmml">​</mo><msub id="S2.SS3.p5.1.m1.1.1.3" xref="S2.SS3.p5.1.m1.1.1.3.cmml"><mi id="S2.SS3.p5.1.m1.1.1.3.2" xref="S2.SS3.p5.1.m1.1.1.3.2.cmml">x</mi><mi id="S2.SS3.p5.1.m1.1.1.3.3" xref="S2.SS3.p5.1.m1.1.1.3.3.cmml">t</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p5.1.m1.1b"><apply id="S2.SS3.p5.1.m1.1.1.cmml" xref="S2.SS3.p5.1.m1.1.1"><times id="S2.SS3.p5.1.m1.1.1.1.cmml" xref="S2.SS3.p5.1.m1.1.1.1"></times><apply id="S2.SS3.p5.1.m1.1.1.2.cmml" xref="S2.SS3.p5.1.m1.1.1.2"><csymbol cd="ambiguous" id="S2.SS3.p5.1.m1.1.1.2.1.cmml" xref="S2.SS3.p5.1.m1.1.1.2">subscript</csymbol><ci id="S2.SS3.p5.1.m1.1.1.2.2.cmml" xref="S2.SS3.p5.1.m1.1.1.2.2">𝜃</ci><ci id="S2.SS3.p5.1.m1.1.1.2.3.cmml" xref="S2.SS3.p5.1.m1.1.1.2.3">𝐾</ci></apply><apply id="S2.SS3.p5.1.m1.1.1.3.cmml" xref="S2.SS3.p5.1.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS3.p5.1.m1.1.1.3.1.cmml" xref="S2.SS3.p5.1.m1.1.1.3">subscript</csymbol><ci id="S2.SS3.p5.1.m1.1.1.3.2.cmml" xref="S2.SS3.p5.1.m1.1.1.3.2">𝑥</ci><ci id="S2.SS3.p5.1.m1.1.1.3.3.cmml" xref="S2.SS3.p5.1.m1.1.1.3.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p5.1.m1.1c">\theta_{K}x_{t}</annotation></semantics></math> has fewer dimensions than <math id="S2.SS3.p5.2.m2.1" class="ltx_Math" alttext="x_{t}" display="inline"><semantics id="S2.SS3.p5.2.m2.1a"><msub id="S2.SS3.p5.2.m2.1.1" xref="S2.SS3.p5.2.m2.1.1.cmml"><mi id="S2.SS3.p5.2.m2.1.1.2" xref="S2.SS3.p5.2.m2.1.1.2.cmml">x</mi><mi id="S2.SS3.p5.2.m2.1.1.3" xref="S2.SS3.p5.2.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p5.2.m2.1b"><apply id="S2.SS3.p5.2.m2.1.1.cmml" xref="S2.SS3.p5.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS3.p5.2.m2.1.1.1.cmml" xref="S2.SS3.p5.2.m2.1.1">subscript</csymbol><ci id="S2.SS3.p5.2.m2.1.1.2.cmml" xref="S2.SS3.p5.2.m2.1.1.2">𝑥</ci><ci id="S2.SS3.p5.2.m2.1.1.3.cmml" xref="S2.SS3.p5.2.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p5.2.m2.1c">x_{t}</annotation></semantics></math>, so we can no longer use the output rule in Equation <a href="#S2.E1" title="In 2.1 TTT as updating a hidden state ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.
The simplest solution is to create a <em id="S2.SS3.p5.3.1" class="ltx_emph ltx_font_italic">test view</em> <math id="S2.SS3.p5.3.m3.1" class="ltx_Math" alttext="\theta_{Q}x_{t}" display="inline"><semantics id="S2.SS3.p5.3.m3.1a"><mrow id="S2.SS3.p5.3.m3.1.1" xref="S2.SS3.p5.3.m3.1.1.cmml"><msub id="S2.SS3.p5.3.m3.1.1.2" xref="S2.SS3.p5.3.m3.1.1.2.cmml"><mi id="S2.SS3.p5.3.m3.1.1.2.2" xref="S2.SS3.p5.3.m3.1.1.2.2.cmml">θ</mi><mi id="S2.SS3.p5.3.m3.1.1.2.3" xref="S2.SS3.p5.3.m3.1.1.2.3.cmml">Q</mi></msub><mo lspace="0em" rspace="0em" id="S2.SS3.p5.3.m3.1.1.1" xref="S2.SS3.p5.3.m3.1.1.1.cmml">​</mo><msub id="S2.SS3.p5.3.m3.1.1.3" xref="S2.SS3.p5.3.m3.1.1.3.cmml"><mi id="S2.SS3.p5.3.m3.1.1.3.2" xref="S2.SS3.p5.3.m3.1.1.3.2.cmml">x</mi><mi id="S2.SS3.p5.3.m3.1.1.3.3" xref="S2.SS3.p5.3.m3.1.1.3.3.cmml">t</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p5.3.m3.1b"><apply id="S2.SS3.p5.3.m3.1.1.cmml" xref="S2.SS3.p5.3.m3.1.1"><times id="S2.SS3.p5.3.m3.1.1.1.cmml" xref="S2.SS3.p5.3.m3.1.1.1"></times><apply id="S2.SS3.p5.3.m3.1.1.2.cmml" xref="S2.SS3.p5.3.m3.1.1.2"><csymbol cd="ambiguous" id="S2.SS3.p5.3.m3.1.1.2.1.cmml" xref="S2.SS3.p5.3.m3.1.1.2">subscript</csymbol><ci id="S2.SS3.p5.3.m3.1.1.2.2.cmml" xref="S2.SS3.p5.3.m3.1.1.2.2">𝜃</ci><ci id="S2.SS3.p5.3.m3.1.1.2.3.cmml" xref="S2.SS3.p5.3.m3.1.1.2.3">𝑄</ci></apply><apply id="S2.SS3.p5.3.m3.1.1.3.cmml" xref="S2.SS3.p5.3.m3.1.1.3"><csymbol cd="ambiguous" id="S2.SS3.p5.3.m3.1.1.3.1.cmml" xref="S2.SS3.p5.3.m3.1.1.3">subscript</csymbol><ci id="S2.SS3.p5.3.m3.1.1.3.2.cmml" xref="S2.SS3.p5.3.m3.1.1.3.2">𝑥</ci><ci id="S2.SS3.p5.3.m3.1.1.3.3.cmml" xref="S2.SS3.p5.3.m3.1.1.3.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p5.3.m3.1c">\theta_{Q}x_{t}</annotation></semantics></math>, and change our output rule to:</p>
<table id="S2.E5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E5.m1.1" class="ltx_Math" alttext="z_{t}=f\left(\theta_{Q}x_{t};W_{t}\right)." display="block"><semantics id="S2.E5.m1.1a"><mrow id="S2.E5.m1.1.1.1" xref="S2.E5.m1.1.1.1.1.cmml"><mrow id="S2.E5.m1.1.1.1.1" xref="S2.E5.m1.1.1.1.1.cmml"><msub id="S2.E5.m1.1.1.1.1.4" xref="S2.E5.m1.1.1.1.1.4.cmml"><mi id="S2.E5.m1.1.1.1.1.4.2" xref="S2.E5.m1.1.1.1.1.4.2.cmml">z</mi><mi id="S2.E5.m1.1.1.1.1.4.3" xref="S2.E5.m1.1.1.1.1.4.3.cmml">t</mi></msub><mo id="S2.E5.m1.1.1.1.1.3" xref="S2.E5.m1.1.1.1.1.3.cmml">=</mo><mrow id="S2.E5.m1.1.1.1.1.2" xref="S2.E5.m1.1.1.1.1.2.cmml"><mi id="S2.E5.m1.1.1.1.1.2.4" xref="S2.E5.m1.1.1.1.1.2.4.cmml">f</mi><mo lspace="0em" rspace="0em" id="S2.E5.m1.1.1.1.1.2.3" xref="S2.E5.m1.1.1.1.1.2.3.cmml">​</mo><mrow id="S2.E5.m1.1.1.1.1.2.2.2" xref="S2.E5.m1.1.1.1.1.2.2.3.cmml"><mo id="S2.E5.m1.1.1.1.1.2.2.2.3" xref="S2.E5.m1.1.1.1.1.2.2.3.cmml">(</mo><mrow id="S2.E5.m1.1.1.1.1.1.1.1.1" xref="S2.E5.m1.1.1.1.1.1.1.1.1.cmml"><msub id="S2.E5.m1.1.1.1.1.1.1.1.1.2" xref="S2.E5.m1.1.1.1.1.1.1.1.1.2.cmml"><mi id="S2.E5.m1.1.1.1.1.1.1.1.1.2.2" xref="S2.E5.m1.1.1.1.1.1.1.1.1.2.2.cmml">θ</mi><mi id="S2.E5.m1.1.1.1.1.1.1.1.1.2.3" xref="S2.E5.m1.1.1.1.1.1.1.1.1.2.3.cmml">Q</mi></msub><mo lspace="0em" rspace="0em" id="S2.E5.m1.1.1.1.1.1.1.1.1.1" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1.cmml">​</mo><msub id="S2.E5.m1.1.1.1.1.1.1.1.1.3" xref="S2.E5.m1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S2.E5.m1.1.1.1.1.1.1.1.1.3.2" xref="S2.E5.m1.1.1.1.1.1.1.1.1.3.2.cmml">x</mi><mi id="S2.E5.m1.1.1.1.1.1.1.1.1.3.3" xref="S2.E5.m1.1.1.1.1.1.1.1.1.3.3.cmml">t</mi></msub></mrow><mo id="S2.E5.m1.1.1.1.1.2.2.2.4" xref="S2.E5.m1.1.1.1.1.2.2.3.cmml">;</mo><msub id="S2.E5.m1.1.1.1.1.2.2.2.2" xref="S2.E5.m1.1.1.1.1.2.2.2.2.cmml"><mi id="S2.E5.m1.1.1.1.1.2.2.2.2.2" xref="S2.E5.m1.1.1.1.1.2.2.2.2.2.cmml">W</mi><mi id="S2.E5.m1.1.1.1.1.2.2.2.2.3" xref="S2.E5.m1.1.1.1.1.2.2.2.2.3.cmml">t</mi></msub><mo id="S2.E5.m1.1.1.1.1.2.2.2.5" xref="S2.E5.m1.1.1.1.1.2.2.3.cmml">)</mo></mrow></mrow></mrow><mo lspace="0em" id="S2.E5.m1.1.1.1.2" xref="S2.E5.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E5.m1.1b"><apply id="S2.E5.m1.1.1.1.1.cmml" xref="S2.E5.m1.1.1.1"><eq id="S2.E5.m1.1.1.1.1.3.cmml" xref="S2.E5.m1.1.1.1.1.3"></eq><apply id="S2.E5.m1.1.1.1.1.4.cmml" xref="S2.E5.m1.1.1.1.1.4"><csymbol cd="ambiguous" id="S2.E5.m1.1.1.1.1.4.1.cmml" xref="S2.E5.m1.1.1.1.1.4">subscript</csymbol><ci id="S2.E5.m1.1.1.1.1.4.2.cmml" xref="S2.E5.m1.1.1.1.1.4.2">𝑧</ci><ci id="S2.E5.m1.1.1.1.1.4.3.cmml" xref="S2.E5.m1.1.1.1.1.4.3">𝑡</ci></apply><apply id="S2.E5.m1.1.1.1.1.2.cmml" xref="S2.E5.m1.1.1.1.1.2"><times id="S2.E5.m1.1.1.1.1.2.3.cmml" xref="S2.E5.m1.1.1.1.1.2.3"></times><ci id="S2.E5.m1.1.1.1.1.2.4.cmml" xref="S2.E5.m1.1.1.1.1.2.4">𝑓</ci><list id="S2.E5.m1.1.1.1.1.2.2.3.cmml" xref="S2.E5.m1.1.1.1.1.2.2.2"><apply id="S2.E5.m1.1.1.1.1.1.1.1.1.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1"><times id="S2.E5.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.1"></times><apply id="S2.E5.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E5.m1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S2.E5.m1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.2.2">𝜃</ci><ci id="S2.E5.m1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.2.3">𝑄</ci></apply><apply id="S2.E5.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E5.m1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.E5.m1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.3.2">𝑥</ci><ci id="S2.E5.m1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S2.E5.m1.1.1.1.1.1.1.1.1.3.3">𝑡</ci></apply></apply><apply id="S2.E5.m1.1.1.1.1.2.2.2.2.cmml" xref="S2.E5.m1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S2.E5.m1.1.1.1.1.2.2.2.2.1.cmml" xref="S2.E5.m1.1.1.1.1.2.2.2.2">subscript</csymbol><ci id="S2.E5.m1.1.1.1.1.2.2.2.2.2.cmml" xref="S2.E5.m1.1.1.1.1.2.2.2.2.2">𝑊</ci><ci id="S2.E5.m1.1.1.1.1.2.2.2.2.3.cmml" xref="S2.E5.m1.1.1.1.1.2.2.2.2.3">𝑡</ci></apply></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E5.m1.1c">z_{t}=f\left(\theta_{Q}x_{t};W_{t}\right).</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(5)</span></td>
</tr></tbody>
</table>
<p id="S2.SS3.p5.6" class="ltx_p">This solution has an additional benefit.
The training and label views specify the information in <math id="S2.SS3.p5.4.m1.1" class="ltx_Math" alttext="x_{t}" display="inline"><semantics id="S2.SS3.p5.4.m1.1a"><msub id="S2.SS3.p5.4.m1.1.1" xref="S2.SS3.p5.4.m1.1.1.cmml"><mi id="S2.SS3.p5.4.m1.1.1.2" xref="S2.SS3.p5.4.m1.1.1.2.cmml">x</mi><mi id="S2.SS3.p5.4.m1.1.1.3" xref="S2.SS3.p5.4.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p5.4.m1.1b"><apply id="S2.SS3.p5.4.m1.1.1.cmml" xref="S2.SS3.p5.4.m1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p5.4.m1.1.1.1.cmml" xref="S2.SS3.p5.4.m1.1.1">subscript</csymbol><ci id="S2.SS3.p5.4.m1.1.1.2.cmml" xref="S2.SS3.p5.4.m1.1.1.2">𝑥</ci><ci id="S2.SS3.p5.4.m1.1.1.3.cmml" xref="S2.SS3.p5.4.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p5.4.m1.1c">x_{t}</annotation></semantics></math> that is compressed into <math id="S2.SS3.p5.5.m2.1" class="ltx_Math" alttext="W_{t}" display="inline"><semantics id="S2.SS3.p5.5.m2.1a"><msub id="S2.SS3.p5.5.m2.1.1" xref="S2.SS3.p5.5.m2.1.1.cmml"><mi id="S2.SS3.p5.5.m2.1.1.2" xref="S2.SS3.p5.5.m2.1.1.2.cmml">W</mi><mi id="S2.SS3.p5.5.m2.1.1.3" xref="S2.SS3.p5.5.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p5.5.m2.1b"><apply id="S2.SS3.p5.5.m2.1.1.cmml" xref="S2.SS3.p5.5.m2.1.1"><csymbol cd="ambiguous" id="S2.SS3.p5.5.m2.1.1.1.cmml" xref="S2.SS3.p5.5.m2.1.1">subscript</csymbol><ci id="S2.SS3.p5.5.m2.1.1.2.cmml" xref="S2.SS3.p5.5.m2.1.1.2">𝑊</ci><ci id="S2.SS3.p5.5.m2.1.1.3.cmml" xref="S2.SS3.p5.5.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p5.5.m2.1c">W_{t}</annotation></semantics></math> and propagated forward through time.
The test view specifies potentially different information that is mapped to the current output token <math id="S2.SS3.p5.6.m3.1" class="ltx_Math" alttext="z_{t}" display="inline"><semantics id="S2.SS3.p5.6.m3.1a"><msub id="S2.SS3.p5.6.m3.1.1" xref="S2.SS3.p5.6.m3.1.1.cmml"><mi id="S2.SS3.p5.6.m3.1.1.2" xref="S2.SS3.p5.6.m3.1.1.2.cmml">z</mi><mi id="S2.SS3.p5.6.m3.1.1.3" xref="S2.SS3.p5.6.m3.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS3.p5.6.m3.1b"><apply id="S2.SS3.p5.6.m3.1.1.cmml" xref="S2.SS3.p5.6.m3.1.1"><csymbol cd="ambiguous" id="S2.SS3.p5.6.m3.1.1.1.cmml" xref="S2.SS3.p5.6.m3.1.1">subscript</csymbol><ci id="S2.SS3.p5.6.m3.1.1.2.cmml" xref="S2.SS3.p5.6.m3.1.1.2">𝑧</ci><ci id="S2.SS3.p5.6.m3.1.1.3.cmml" xref="S2.SS3.p5.6.m3.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p5.6.m3.1c">z_{t}</annotation></semantics></math> and propagated forward through network layers, therefore adds more flexibility to the self-supervised task.</p>
</div>
<div id="S2.SS3.p6" class="ltx_para">
<p id="S2.SS3.p6.1" class="ltx_p">Altogether, the set of all possible choices for <math id="S2.SS3.p6.1.m1.3" class="ltx_Math" alttext="\theta_{K},\theta_{Q},\theta_{V}" display="inline"><semantics id="S2.SS3.p6.1.m1.3a"><mrow id="S2.SS3.p6.1.m1.3.3.3" xref="S2.SS3.p6.1.m1.3.3.4.cmml"><msub id="S2.SS3.p6.1.m1.1.1.1.1" xref="S2.SS3.p6.1.m1.1.1.1.1.cmml"><mi id="S2.SS3.p6.1.m1.1.1.1.1.2" xref="S2.SS3.p6.1.m1.1.1.1.1.2.cmml">θ</mi><mi id="S2.SS3.p6.1.m1.1.1.1.1.3" xref="S2.SS3.p6.1.m1.1.1.1.1.3.cmml">K</mi></msub><mo id="S2.SS3.p6.1.m1.3.3.3.4" xref="S2.SS3.p6.1.m1.3.3.4.cmml">,</mo><msub id="S2.SS3.p6.1.m1.2.2.2.2" xref="S2.SS3.p6.1.m1.2.2.2.2.cmml"><mi id="S2.SS3.p6.1.m1.2.2.2.2.2" xref="S2.SS3.p6.1.m1.2.2.2.2.2.cmml">θ</mi><mi id="S2.SS3.p6.1.m1.2.2.2.2.3" xref="S2.SS3.p6.1.m1.2.2.2.2.3.cmml">Q</mi></msub><mo id="S2.SS3.p6.1.m1.3.3.3.5" xref="S2.SS3.p6.1.m1.3.3.4.cmml">,</mo><msub id="S2.SS3.p6.1.m1.3.3.3.3" xref="S2.SS3.p6.1.m1.3.3.3.3.cmml"><mi id="S2.SS3.p6.1.m1.3.3.3.3.2" xref="S2.SS3.p6.1.m1.3.3.3.3.2.cmml">θ</mi><mi id="S2.SS3.p6.1.m1.3.3.3.3.3" xref="S2.SS3.p6.1.m1.3.3.3.3.3.cmml">V</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS3.p6.1.m1.3b"><list id="S2.SS3.p6.1.m1.3.3.4.cmml" xref="S2.SS3.p6.1.m1.3.3.3"><apply id="S2.SS3.p6.1.m1.1.1.1.1.cmml" xref="S2.SS3.p6.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS3.p6.1.m1.1.1.1.1.1.cmml" xref="S2.SS3.p6.1.m1.1.1.1.1">subscript</csymbol><ci id="S2.SS3.p6.1.m1.1.1.1.1.2.cmml" xref="S2.SS3.p6.1.m1.1.1.1.1.2">𝜃</ci><ci id="S2.SS3.p6.1.m1.1.1.1.1.3.cmml" xref="S2.SS3.p6.1.m1.1.1.1.1.3">𝐾</ci></apply><apply id="S2.SS3.p6.1.m1.2.2.2.2.cmml" xref="S2.SS3.p6.1.m1.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS3.p6.1.m1.2.2.2.2.1.cmml" xref="S2.SS3.p6.1.m1.2.2.2.2">subscript</csymbol><ci id="S2.SS3.p6.1.m1.2.2.2.2.2.cmml" xref="S2.SS3.p6.1.m1.2.2.2.2.2">𝜃</ci><ci id="S2.SS3.p6.1.m1.2.2.2.2.3.cmml" xref="S2.SS3.p6.1.m1.2.2.2.2.3">𝑄</ci></apply><apply id="S2.SS3.p6.1.m1.3.3.3.3.cmml" xref="S2.SS3.p6.1.m1.3.3.3.3"><csymbol cd="ambiguous" id="S2.SS3.p6.1.m1.3.3.3.3.1.cmml" xref="S2.SS3.p6.1.m1.3.3.3.3">subscript</csymbol><ci id="S2.SS3.p6.1.m1.3.3.3.3.2.cmml" xref="S2.SS3.p6.1.m1.3.3.3.3.2">𝜃</ci><ci id="S2.SS3.p6.1.m1.3.3.3.3.3.cmml" xref="S2.SS3.p6.1.m1.3.3.3.3.3">𝑉</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS3.p6.1.m1.3c">\theta_{K},\theta_{Q},\theta_{V}</annotation></semantics></math> induces a family of multi-view reconstruction tasks, and the outer loop can be interpreted as selecting a task from this family.
Here we have designed all views as linear projections for simplicity.
Future work might experiment with more flexible transformations, or bigger and different families of self-supervised tasks.</p>
</div>
</section>
<section id="S2.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.4 </span>Parallelization with mini-batch TTT</h3>

<div id="S2.SS4.p1" class="ltx_para">
<p id="S2.SS4.p1.5" class="ltx_p">The naive TTT layer developed so far is already efficient in the number of floating point operations (FLOPs).
However, its update rule <math id="S2.SS4.p1.1.m1.2" class="ltx_Math" alttext="W_{t}=W_{t-1}-\eta\nabla l(W_{t-1};x_{t})" display="inline"><semantics id="S2.SS4.p1.1.m1.2a"><mrow id="S2.SS4.p1.1.m1.2.2" xref="S2.SS4.p1.1.m1.2.2.cmml"><msub id="S2.SS4.p1.1.m1.2.2.4" xref="S2.SS4.p1.1.m1.2.2.4.cmml"><mi id="S2.SS4.p1.1.m1.2.2.4.2" xref="S2.SS4.p1.1.m1.2.2.4.2.cmml">W</mi><mi id="S2.SS4.p1.1.m1.2.2.4.3" xref="S2.SS4.p1.1.m1.2.2.4.3.cmml">t</mi></msub><mo id="S2.SS4.p1.1.m1.2.2.3" xref="S2.SS4.p1.1.m1.2.2.3.cmml">=</mo><mrow id="S2.SS4.p1.1.m1.2.2.2" xref="S2.SS4.p1.1.m1.2.2.2.cmml"><msub id="S2.SS4.p1.1.m1.2.2.2.4" xref="S2.SS4.p1.1.m1.2.2.2.4.cmml"><mi id="S2.SS4.p1.1.m1.2.2.2.4.2" xref="S2.SS4.p1.1.m1.2.2.2.4.2.cmml">W</mi><mrow id="S2.SS4.p1.1.m1.2.2.2.4.3" xref="S2.SS4.p1.1.m1.2.2.2.4.3.cmml"><mi id="S2.SS4.p1.1.m1.2.2.2.4.3.2" xref="S2.SS4.p1.1.m1.2.2.2.4.3.2.cmml">t</mi><mo id="S2.SS4.p1.1.m1.2.2.2.4.3.1" xref="S2.SS4.p1.1.m1.2.2.2.4.3.1.cmml">−</mo><mn id="S2.SS4.p1.1.m1.2.2.2.4.3.3" xref="S2.SS4.p1.1.m1.2.2.2.4.3.3.cmml">1</mn></mrow></msub><mo id="S2.SS4.p1.1.m1.2.2.2.3" xref="S2.SS4.p1.1.m1.2.2.2.3.cmml">−</mo><mrow id="S2.SS4.p1.1.m1.2.2.2.2" xref="S2.SS4.p1.1.m1.2.2.2.2.cmml"><mi id="S2.SS4.p1.1.m1.2.2.2.2.4" xref="S2.SS4.p1.1.m1.2.2.2.2.4.cmml">η</mi><mo lspace="0.167em" rspace="0em" id="S2.SS4.p1.1.m1.2.2.2.2.3" xref="S2.SS4.p1.1.m1.2.2.2.2.3.cmml">​</mo><mrow id="S2.SS4.p1.1.m1.2.2.2.2.5" xref="S2.SS4.p1.1.m1.2.2.2.2.5.cmml"><mo rspace="0.167em" id="S2.SS4.p1.1.m1.2.2.2.2.5.1" xref="S2.SS4.p1.1.m1.2.2.2.2.5.1.cmml">∇</mo><mi id="S2.SS4.p1.1.m1.2.2.2.2.5.2" xref="S2.SS4.p1.1.m1.2.2.2.2.5.2.cmml">l</mi></mrow><mo lspace="0em" rspace="0em" id="S2.SS4.p1.1.m1.2.2.2.2.3a" xref="S2.SS4.p1.1.m1.2.2.2.2.3.cmml">​</mo><mrow id="S2.SS4.p1.1.m1.2.2.2.2.2.2" xref="S2.SS4.p1.1.m1.2.2.2.2.2.3.cmml"><mo stretchy="false" id="S2.SS4.p1.1.m1.2.2.2.2.2.2.3" xref="S2.SS4.p1.1.m1.2.2.2.2.2.3.cmml">(</mo><msub id="S2.SS4.p1.1.m1.1.1.1.1.1.1.1" xref="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.cmml"><mi id="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.2" xref="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.2.cmml">W</mi><mrow id="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.3" xref="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.3.cmml"><mi id="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.3.2" xref="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.3.2.cmml">t</mi><mo id="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.3.1" xref="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.3.1.cmml">−</mo><mn id="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.3.3" xref="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.3.3.cmml">1</mn></mrow></msub><mo id="S2.SS4.p1.1.m1.2.2.2.2.2.2.4" xref="S2.SS4.p1.1.m1.2.2.2.2.2.3.cmml">;</mo><msub id="S2.SS4.p1.1.m1.2.2.2.2.2.2.2" xref="S2.SS4.p1.1.m1.2.2.2.2.2.2.2.cmml"><mi id="S2.SS4.p1.1.m1.2.2.2.2.2.2.2.2" xref="S2.SS4.p1.1.m1.2.2.2.2.2.2.2.2.cmml">x</mi><mi id="S2.SS4.p1.1.m1.2.2.2.2.2.2.2.3" xref="S2.SS4.p1.1.m1.2.2.2.2.2.2.2.3.cmml">t</mi></msub><mo stretchy="false" id="S2.SS4.p1.1.m1.2.2.2.2.2.2.5" xref="S2.SS4.p1.1.m1.2.2.2.2.2.3.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.1.m1.2b"><apply id="S2.SS4.p1.1.m1.2.2.cmml" xref="S2.SS4.p1.1.m1.2.2"><eq id="S2.SS4.p1.1.m1.2.2.3.cmml" xref="S2.SS4.p1.1.m1.2.2.3"></eq><apply id="S2.SS4.p1.1.m1.2.2.4.cmml" xref="S2.SS4.p1.1.m1.2.2.4"><csymbol cd="ambiguous" id="S2.SS4.p1.1.m1.2.2.4.1.cmml" xref="S2.SS4.p1.1.m1.2.2.4">subscript</csymbol><ci id="S2.SS4.p1.1.m1.2.2.4.2.cmml" xref="S2.SS4.p1.1.m1.2.2.4.2">𝑊</ci><ci id="S2.SS4.p1.1.m1.2.2.4.3.cmml" xref="S2.SS4.p1.1.m1.2.2.4.3">𝑡</ci></apply><apply id="S2.SS4.p1.1.m1.2.2.2.cmml" xref="S2.SS4.p1.1.m1.2.2.2"><minus id="S2.SS4.p1.1.m1.2.2.2.3.cmml" xref="S2.SS4.p1.1.m1.2.2.2.3"></minus><apply id="S2.SS4.p1.1.m1.2.2.2.4.cmml" xref="S2.SS4.p1.1.m1.2.2.2.4"><csymbol cd="ambiguous" id="S2.SS4.p1.1.m1.2.2.2.4.1.cmml" xref="S2.SS4.p1.1.m1.2.2.2.4">subscript</csymbol><ci id="S2.SS4.p1.1.m1.2.2.2.4.2.cmml" xref="S2.SS4.p1.1.m1.2.2.2.4.2">𝑊</ci><apply id="S2.SS4.p1.1.m1.2.2.2.4.3.cmml" xref="S2.SS4.p1.1.m1.2.2.2.4.3"><minus id="S2.SS4.p1.1.m1.2.2.2.4.3.1.cmml" xref="S2.SS4.p1.1.m1.2.2.2.4.3.1"></minus><ci id="S2.SS4.p1.1.m1.2.2.2.4.3.2.cmml" xref="S2.SS4.p1.1.m1.2.2.2.4.3.2">𝑡</ci><cn type="integer" id="S2.SS4.p1.1.m1.2.2.2.4.3.3.cmml" xref="S2.SS4.p1.1.m1.2.2.2.4.3.3">1</cn></apply></apply><apply id="S2.SS4.p1.1.m1.2.2.2.2.cmml" xref="S2.SS4.p1.1.m1.2.2.2.2"><times id="S2.SS4.p1.1.m1.2.2.2.2.3.cmml" xref="S2.SS4.p1.1.m1.2.2.2.2.3"></times><ci id="S2.SS4.p1.1.m1.2.2.2.2.4.cmml" xref="S2.SS4.p1.1.m1.2.2.2.2.4">𝜂</ci><apply id="S2.SS4.p1.1.m1.2.2.2.2.5.cmml" xref="S2.SS4.p1.1.m1.2.2.2.2.5"><ci id="S2.SS4.p1.1.m1.2.2.2.2.5.1.cmml" xref="S2.SS4.p1.1.m1.2.2.2.2.5.1">∇</ci><ci id="S2.SS4.p1.1.m1.2.2.2.2.5.2.cmml" xref="S2.SS4.p1.1.m1.2.2.2.2.5.2">𝑙</ci></apply><list id="S2.SS4.p1.1.m1.2.2.2.2.2.3.cmml" xref="S2.SS4.p1.1.m1.2.2.2.2.2.2"><apply id="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.cmml" xref="S2.SS4.p1.1.m1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.1.cmml" xref="S2.SS4.p1.1.m1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.2.cmml" xref="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.2">𝑊</ci><apply id="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.3.cmml" xref="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.3"><minus id="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.3.1.cmml" xref="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.3.1"></minus><ci id="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.3.2.cmml" xref="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.3.2">𝑡</ci><cn type="integer" id="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.3.3.cmml" xref="S2.SS4.p1.1.m1.1.1.1.1.1.1.1.3.3">1</cn></apply></apply><apply id="S2.SS4.p1.1.m1.2.2.2.2.2.2.2.cmml" xref="S2.SS4.p1.1.m1.2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS4.p1.1.m1.2.2.2.2.2.2.2.1.cmml" xref="S2.SS4.p1.1.m1.2.2.2.2.2.2.2">subscript</csymbol><ci id="S2.SS4.p1.1.m1.2.2.2.2.2.2.2.2.cmml" xref="S2.SS4.p1.1.m1.2.2.2.2.2.2.2.2">𝑥</ci><ci id="S2.SS4.p1.1.m1.2.2.2.2.2.2.2.3.cmml" xref="S2.SS4.p1.1.m1.2.2.2.2.2.2.2.3">𝑡</ci></apply></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.1.m1.2c">W_{t}=W_{t-1}-\eta\nabla l(W_{t-1};x_{t})</annotation></semantics></math> cannot be parallelized,
because <math id="S2.SS4.p1.2.m2.1" class="ltx_Math" alttext="W_{t}" display="inline"><semantics id="S2.SS4.p1.2.m2.1a"><msub id="S2.SS4.p1.2.m2.1.1" xref="S2.SS4.p1.2.m2.1.1.cmml"><mi id="S2.SS4.p1.2.m2.1.1.2" xref="S2.SS4.p1.2.m2.1.1.2.cmml">W</mi><mi id="S2.SS4.p1.2.m2.1.1.3" xref="S2.SS4.p1.2.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.2.m2.1b"><apply id="S2.SS4.p1.2.m2.1.1.cmml" xref="S2.SS4.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS4.p1.2.m2.1.1.1.cmml" xref="S2.SS4.p1.2.m2.1.1">subscript</csymbol><ci id="S2.SS4.p1.2.m2.1.1.2.cmml" xref="S2.SS4.p1.2.m2.1.1.2">𝑊</ci><ci id="S2.SS4.p1.2.m2.1.1.3.cmml" xref="S2.SS4.p1.2.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.2.m2.1c">W_{t}</annotation></semantics></math> depends on <math id="S2.SS4.p1.3.m3.1" class="ltx_Math" alttext="W_{t-1}" display="inline"><semantics id="S2.SS4.p1.3.m3.1a"><msub id="S2.SS4.p1.3.m3.1.1" xref="S2.SS4.p1.3.m3.1.1.cmml"><mi id="S2.SS4.p1.3.m3.1.1.2" xref="S2.SS4.p1.3.m3.1.1.2.cmml">W</mi><mrow id="S2.SS4.p1.3.m3.1.1.3" xref="S2.SS4.p1.3.m3.1.1.3.cmml"><mi id="S2.SS4.p1.3.m3.1.1.3.2" xref="S2.SS4.p1.3.m3.1.1.3.2.cmml">t</mi><mo id="S2.SS4.p1.3.m3.1.1.3.1" xref="S2.SS4.p1.3.m3.1.1.3.1.cmml">−</mo><mn id="S2.SS4.p1.3.m3.1.1.3.3" xref="S2.SS4.p1.3.m3.1.1.3.3.cmml">1</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.3.m3.1b"><apply id="S2.SS4.p1.3.m3.1.1.cmml" xref="S2.SS4.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS4.p1.3.m3.1.1.1.cmml" xref="S2.SS4.p1.3.m3.1.1">subscript</csymbol><ci id="S2.SS4.p1.3.m3.1.1.2.cmml" xref="S2.SS4.p1.3.m3.1.1.2">𝑊</ci><apply id="S2.SS4.p1.3.m3.1.1.3.cmml" xref="S2.SS4.p1.3.m3.1.1.3"><minus id="S2.SS4.p1.3.m3.1.1.3.1.cmml" xref="S2.SS4.p1.3.m3.1.1.3.1"></minus><ci id="S2.SS4.p1.3.m3.1.1.3.2.cmml" xref="S2.SS4.p1.3.m3.1.1.3.2">𝑡</ci><cn type="integer" id="S2.SS4.p1.3.m3.1.1.3.3.cmml" xref="S2.SS4.p1.3.m3.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.3.m3.1c">W_{t-1}</annotation></semantics></math> in two places: before the minus sign and inside <math id="S2.SS4.p1.4.m4.1" class="ltx_Math" alttext="\nabla l" display="inline"><semantics id="S2.SS4.p1.4.m4.1a"><mrow id="S2.SS4.p1.4.m4.1.1" xref="S2.SS4.p1.4.m4.1.1.cmml"><mo rspace="0.167em" id="S2.SS4.p1.4.m4.1.1.1" xref="S2.SS4.p1.4.m4.1.1.1.cmml">∇</mo><mi id="S2.SS4.p1.4.m4.1.1.2" xref="S2.SS4.p1.4.m4.1.1.2.cmml">l</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.4.m4.1b"><apply id="S2.SS4.p1.4.m4.1.1.cmml" xref="S2.SS4.p1.4.m4.1.1"><ci id="S2.SS4.p1.4.m4.1.1.1.cmml" xref="S2.SS4.p1.4.m4.1.1.1">∇</ci><ci id="S2.SS4.p1.4.m4.1.1.2.cmml" xref="S2.SS4.p1.4.m4.1.1.2">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.4.m4.1c">\nabla l</annotation></semantics></math>.
Since <math id="S2.SS4.p1.5.m5.1" class="ltx_Math" alttext="\nabla l" display="inline"><semantics id="S2.SS4.p1.5.m5.1a"><mrow id="S2.SS4.p1.5.m5.1.1" xref="S2.SS4.p1.5.m5.1.1.cmml"><mo rspace="0.167em" id="S2.SS4.p1.5.m5.1.1.1" xref="S2.SS4.p1.5.m5.1.1.1.cmml">∇</mo><mi id="S2.SS4.p1.5.m5.1.1.2" xref="S2.SS4.p1.5.m5.1.1.2.cmml">l</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p1.5.m5.1b"><apply id="S2.SS4.p1.5.m5.1.1.cmml" xref="S2.SS4.p1.5.m5.1.1"><ci id="S2.SS4.p1.5.m5.1.1.1.cmml" xref="S2.SS4.p1.5.m5.1.1.1">∇</ci><ci id="S2.SS4.p1.5.m5.1.1.2.cmml" xref="S2.SS4.p1.5.m5.1.1.2">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p1.5.m5.1c">\nabla l</annotation></semantics></math> contains the bulk of the computation, we focus on making this second part parallel.</p>
</div>
<div id="S2.SS4.p2" class="ltx_para">
<p id="S2.SS4.p2.6" class="ltx_p">We approach this systems challenge through concepts in the TTT framework. There are many variants of gradient descent (GD).
The general update rule of GD can be expressed as:</p>
<table id="S2.E6" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E6.m1.1" class="ltx_Math" alttext="W_{t}=W_{t-1}-\eta\,G_{t}=W_{0}-\eta\sum_{s=1}^{t}G_{s}," display="block"><semantics id="S2.E6.m1.1a"><mrow id="S2.E6.m1.1.1.1" xref="S2.E6.m1.1.1.1.1.cmml"><mrow id="S2.E6.m1.1.1.1.1" xref="S2.E6.m1.1.1.1.1.cmml"><msub id="S2.E6.m1.1.1.1.1.2" xref="S2.E6.m1.1.1.1.1.2.cmml"><mi id="S2.E6.m1.1.1.1.1.2.2" xref="S2.E6.m1.1.1.1.1.2.2.cmml">W</mi><mi id="S2.E6.m1.1.1.1.1.2.3" xref="S2.E6.m1.1.1.1.1.2.3.cmml">t</mi></msub><mo id="S2.E6.m1.1.1.1.1.3" xref="S2.E6.m1.1.1.1.1.3.cmml">=</mo><mrow id="S2.E6.m1.1.1.1.1.4" xref="S2.E6.m1.1.1.1.1.4.cmml"><msub id="S2.E6.m1.1.1.1.1.4.2" xref="S2.E6.m1.1.1.1.1.4.2.cmml"><mi id="S2.E6.m1.1.1.1.1.4.2.2" xref="S2.E6.m1.1.1.1.1.4.2.2.cmml">W</mi><mrow id="S2.E6.m1.1.1.1.1.4.2.3" xref="S2.E6.m1.1.1.1.1.4.2.3.cmml"><mi id="S2.E6.m1.1.1.1.1.4.2.3.2" xref="S2.E6.m1.1.1.1.1.4.2.3.2.cmml">t</mi><mo id="S2.E6.m1.1.1.1.1.4.2.3.1" xref="S2.E6.m1.1.1.1.1.4.2.3.1.cmml">−</mo><mn id="S2.E6.m1.1.1.1.1.4.2.3.3" xref="S2.E6.m1.1.1.1.1.4.2.3.3.cmml">1</mn></mrow></msub><mo id="S2.E6.m1.1.1.1.1.4.1" xref="S2.E6.m1.1.1.1.1.4.1.cmml">−</mo><mrow id="S2.E6.m1.1.1.1.1.4.3" xref="S2.E6.m1.1.1.1.1.4.3.cmml"><mi id="S2.E6.m1.1.1.1.1.4.3.2" xref="S2.E6.m1.1.1.1.1.4.3.2.cmml">η</mi><mo lspace="0.170em" rspace="0em" id="S2.E6.m1.1.1.1.1.4.3.1" xref="S2.E6.m1.1.1.1.1.4.3.1.cmml">​</mo><msub id="S2.E6.m1.1.1.1.1.4.3.3" xref="S2.E6.m1.1.1.1.1.4.3.3.cmml"><mi id="S2.E6.m1.1.1.1.1.4.3.3.2" xref="S2.E6.m1.1.1.1.1.4.3.3.2.cmml">G</mi><mi id="S2.E6.m1.1.1.1.1.4.3.3.3" xref="S2.E6.m1.1.1.1.1.4.3.3.3.cmml">t</mi></msub></mrow></mrow><mo id="S2.E6.m1.1.1.1.1.5" xref="S2.E6.m1.1.1.1.1.5.cmml">=</mo><mrow id="S2.E6.m1.1.1.1.1.6" xref="S2.E6.m1.1.1.1.1.6.cmml"><msub id="S2.E6.m1.1.1.1.1.6.2" xref="S2.E6.m1.1.1.1.1.6.2.cmml"><mi id="S2.E6.m1.1.1.1.1.6.2.2" xref="S2.E6.m1.1.1.1.1.6.2.2.cmml">W</mi><mn id="S2.E6.m1.1.1.1.1.6.2.3" xref="S2.E6.m1.1.1.1.1.6.2.3.cmml">0</mn></msub><mo id="S2.E6.m1.1.1.1.1.6.1" xref="S2.E6.m1.1.1.1.1.6.1.cmml">−</mo><mrow id="S2.E6.m1.1.1.1.1.6.3" xref="S2.E6.m1.1.1.1.1.6.3.cmml"><mi id="S2.E6.m1.1.1.1.1.6.3.2" xref="S2.E6.m1.1.1.1.1.6.3.2.cmml">η</mi><mo lspace="0em" rspace="0em" id="S2.E6.m1.1.1.1.1.6.3.1" xref="S2.E6.m1.1.1.1.1.6.3.1.cmml">​</mo><mrow id="S2.E6.m1.1.1.1.1.6.3.3" xref="S2.E6.m1.1.1.1.1.6.3.3.cmml"><munderover id="S2.E6.m1.1.1.1.1.6.3.3.1" xref="S2.E6.m1.1.1.1.1.6.3.3.1.cmml"><mo movablelimits="false" id="S2.E6.m1.1.1.1.1.6.3.3.1.2.2" xref="S2.E6.m1.1.1.1.1.6.3.3.1.2.2.cmml">∑</mo><mrow id="S2.E6.m1.1.1.1.1.6.3.3.1.2.3" xref="S2.E6.m1.1.1.1.1.6.3.3.1.2.3.cmml"><mi id="S2.E6.m1.1.1.1.1.6.3.3.1.2.3.2" xref="S2.E6.m1.1.1.1.1.6.3.3.1.2.3.2.cmml">s</mi><mo id="S2.E6.m1.1.1.1.1.6.3.3.1.2.3.1" xref="S2.E6.m1.1.1.1.1.6.3.3.1.2.3.1.cmml">=</mo><mn id="S2.E6.m1.1.1.1.1.6.3.3.1.2.3.3" xref="S2.E6.m1.1.1.1.1.6.3.3.1.2.3.3.cmml">1</mn></mrow><mi id="S2.E6.m1.1.1.1.1.6.3.3.1.3" xref="S2.E6.m1.1.1.1.1.6.3.3.1.3.cmml">t</mi></munderover><msub id="S2.E6.m1.1.1.1.1.6.3.3.2" xref="S2.E6.m1.1.1.1.1.6.3.3.2.cmml"><mi id="S2.E6.m1.1.1.1.1.6.3.3.2.2" xref="S2.E6.m1.1.1.1.1.6.3.3.2.2.cmml">G</mi><mi id="S2.E6.m1.1.1.1.1.6.3.3.2.3" xref="S2.E6.m1.1.1.1.1.6.3.3.2.3.cmml">s</mi></msub></mrow></mrow></mrow></mrow><mo id="S2.E6.m1.1.1.1.2" xref="S2.E6.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E6.m1.1b"><apply id="S2.E6.m1.1.1.1.1.cmml" xref="S2.E6.m1.1.1.1"><and id="S2.E6.m1.1.1.1.1a.cmml" xref="S2.E6.m1.1.1.1"></and><apply id="S2.E6.m1.1.1.1.1b.cmml" xref="S2.E6.m1.1.1.1"><eq id="S2.E6.m1.1.1.1.1.3.cmml" xref="S2.E6.m1.1.1.1.1.3"></eq><apply id="S2.E6.m1.1.1.1.1.2.cmml" xref="S2.E6.m1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E6.m1.1.1.1.1.2.1.cmml" xref="S2.E6.m1.1.1.1.1.2">subscript</csymbol><ci id="S2.E6.m1.1.1.1.1.2.2.cmml" xref="S2.E6.m1.1.1.1.1.2.2">𝑊</ci><ci id="S2.E6.m1.1.1.1.1.2.3.cmml" xref="S2.E6.m1.1.1.1.1.2.3">𝑡</ci></apply><apply id="S2.E6.m1.1.1.1.1.4.cmml" xref="S2.E6.m1.1.1.1.1.4"><minus id="S2.E6.m1.1.1.1.1.4.1.cmml" xref="S2.E6.m1.1.1.1.1.4.1"></minus><apply id="S2.E6.m1.1.1.1.1.4.2.cmml" xref="S2.E6.m1.1.1.1.1.4.2"><csymbol cd="ambiguous" id="S2.E6.m1.1.1.1.1.4.2.1.cmml" xref="S2.E6.m1.1.1.1.1.4.2">subscript</csymbol><ci id="S2.E6.m1.1.1.1.1.4.2.2.cmml" xref="S2.E6.m1.1.1.1.1.4.2.2">𝑊</ci><apply id="S2.E6.m1.1.1.1.1.4.2.3.cmml" xref="S2.E6.m1.1.1.1.1.4.2.3"><minus id="S2.E6.m1.1.1.1.1.4.2.3.1.cmml" xref="S2.E6.m1.1.1.1.1.4.2.3.1"></minus><ci id="S2.E6.m1.1.1.1.1.4.2.3.2.cmml" xref="S2.E6.m1.1.1.1.1.4.2.3.2">𝑡</ci><cn type="integer" id="S2.E6.m1.1.1.1.1.4.2.3.3.cmml" xref="S2.E6.m1.1.1.1.1.4.2.3.3">1</cn></apply></apply><apply id="S2.E6.m1.1.1.1.1.4.3.cmml" xref="S2.E6.m1.1.1.1.1.4.3"><times id="S2.E6.m1.1.1.1.1.4.3.1.cmml" xref="S2.E6.m1.1.1.1.1.4.3.1"></times><ci id="S2.E6.m1.1.1.1.1.4.3.2.cmml" xref="S2.E6.m1.1.1.1.1.4.3.2">𝜂</ci><apply id="S2.E6.m1.1.1.1.1.4.3.3.cmml" xref="S2.E6.m1.1.1.1.1.4.3.3"><csymbol cd="ambiguous" id="S2.E6.m1.1.1.1.1.4.3.3.1.cmml" xref="S2.E6.m1.1.1.1.1.4.3.3">subscript</csymbol><ci id="S2.E6.m1.1.1.1.1.4.3.3.2.cmml" xref="S2.E6.m1.1.1.1.1.4.3.3.2">𝐺</ci><ci id="S2.E6.m1.1.1.1.1.4.3.3.3.cmml" xref="S2.E6.m1.1.1.1.1.4.3.3.3">𝑡</ci></apply></apply></apply></apply><apply id="S2.E6.m1.1.1.1.1c.cmml" xref="S2.E6.m1.1.1.1"><eq id="S2.E6.m1.1.1.1.1.5.cmml" xref="S2.E6.m1.1.1.1.1.5"></eq><share href="#S2.E6.m1.1.1.1.1.4.cmml" id="S2.E6.m1.1.1.1.1d.cmml" xref="S2.E6.m1.1.1.1"></share><apply id="S2.E6.m1.1.1.1.1.6.cmml" xref="S2.E6.m1.1.1.1.1.6"><minus id="S2.E6.m1.1.1.1.1.6.1.cmml" xref="S2.E6.m1.1.1.1.1.6.1"></minus><apply id="S2.E6.m1.1.1.1.1.6.2.cmml" xref="S2.E6.m1.1.1.1.1.6.2"><csymbol cd="ambiguous" id="S2.E6.m1.1.1.1.1.6.2.1.cmml" xref="S2.E6.m1.1.1.1.1.6.2">subscript</csymbol><ci id="S2.E6.m1.1.1.1.1.6.2.2.cmml" xref="S2.E6.m1.1.1.1.1.6.2.2">𝑊</ci><cn type="integer" id="S2.E6.m1.1.1.1.1.6.2.3.cmml" xref="S2.E6.m1.1.1.1.1.6.2.3">0</cn></apply><apply id="S2.E6.m1.1.1.1.1.6.3.cmml" xref="S2.E6.m1.1.1.1.1.6.3"><times id="S2.E6.m1.1.1.1.1.6.3.1.cmml" xref="S2.E6.m1.1.1.1.1.6.3.1"></times><ci id="S2.E6.m1.1.1.1.1.6.3.2.cmml" xref="S2.E6.m1.1.1.1.1.6.3.2">𝜂</ci><apply id="S2.E6.m1.1.1.1.1.6.3.3.cmml" xref="S2.E6.m1.1.1.1.1.6.3.3"><apply id="S2.E6.m1.1.1.1.1.6.3.3.1.cmml" xref="S2.E6.m1.1.1.1.1.6.3.3.1"><csymbol cd="ambiguous" id="S2.E6.m1.1.1.1.1.6.3.3.1.1.cmml" xref="S2.E6.m1.1.1.1.1.6.3.3.1">superscript</csymbol><apply id="S2.E6.m1.1.1.1.1.6.3.3.1.2.cmml" xref="S2.E6.m1.1.1.1.1.6.3.3.1"><csymbol cd="ambiguous" id="S2.E6.m1.1.1.1.1.6.3.3.1.2.1.cmml" xref="S2.E6.m1.1.1.1.1.6.3.3.1">subscript</csymbol><sum id="S2.E6.m1.1.1.1.1.6.3.3.1.2.2.cmml" xref="S2.E6.m1.1.1.1.1.6.3.3.1.2.2"></sum><apply id="S2.E6.m1.1.1.1.1.6.3.3.1.2.3.cmml" xref="S2.E6.m1.1.1.1.1.6.3.3.1.2.3"><eq id="S2.E6.m1.1.1.1.1.6.3.3.1.2.3.1.cmml" xref="S2.E6.m1.1.1.1.1.6.3.3.1.2.3.1"></eq><ci id="S2.E6.m1.1.1.1.1.6.3.3.1.2.3.2.cmml" xref="S2.E6.m1.1.1.1.1.6.3.3.1.2.3.2">𝑠</ci><cn type="integer" id="S2.E6.m1.1.1.1.1.6.3.3.1.2.3.3.cmml" xref="S2.E6.m1.1.1.1.1.6.3.3.1.2.3.3">1</cn></apply></apply><ci id="S2.E6.m1.1.1.1.1.6.3.3.1.3.cmml" xref="S2.E6.m1.1.1.1.1.6.3.3.1.3">𝑡</ci></apply><apply id="S2.E6.m1.1.1.1.1.6.3.3.2.cmml" xref="S2.E6.m1.1.1.1.1.6.3.3.2"><csymbol cd="ambiguous" id="S2.E6.m1.1.1.1.1.6.3.3.2.1.cmml" xref="S2.E6.m1.1.1.1.1.6.3.3.2">subscript</csymbol><ci id="S2.E6.m1.1.1.1.1.6.3.3.2.2.cmml" xref="S2.E6.m1.1.1.1.1.6.3.3.2.2">𝐺</ci><ci id="S2.E6.m1.1.1.1.1.6.3.3.2.3.cmml" xref="S2.E6.m1.1.1.1.1.6.3.3.2.3">𝑠</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E6.m1.1c">W_{t}=W_{t-1}-\eta\,G_{t}=W_{0}-\eta\sum_{s=1}^{t}G_{s},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(6)</span></td>
</tr></tbody>
</table>
<p id="S2.SS4.p2.5" class="ltx_p">where <math id="S2.SS4.p2.1.m1.1" class="ltx_Math" alttext="G_{t}" display="inline"><semantics id="S2.SS4.p2.1.m1.1a"><msub id="S2.SS4.p2.1.m1.1.1" xref="S2.SS4.p2.1.m1.1.1.cmml"><mi id="S2.SS4.p2.1.m1.1.1.2" xref="S2.SS4.p2.1.m1.1.1.2.cmml">G</mi><mi id="S2.SS4.p2.1.m1.1.1.3" xref="S2.SS4.p2.1.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.1.m1.1b"><apply id="S2.SS4.p2.1.m1.1.1.cmml" xref="S2.SS4.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS4.p2.1.m1.1.1.1.cmml" xref="S2.SS4.p2.1.m1.1.1">subscript</csymbol><ci id="S2.SS4.p2.1.m1.1.1.2.cmml" xref="S2.SS4.p2.1.m1.1.1.2">𝐺</ci><ci id="S2.SS4.p2.1.m1.1.1.3.cmml" xref="S2.SS4.p2.1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.1.m1.1c">G_{t}</annotation></semantics></math> is the descent direction.
Note that once we have calculated <math id="S2.SS4.p2.2.m2.1" class="ltx_Math" alttext="G_{t}" display="inline"><semantics id="S2.SS4.p2.2.m2.1a"><msub id="S2.SS4.p2.2.m2.1.1" xref="S2.SS4.p2.2.m2.1.1.cmml"><mi id="S2.SS4.p2.2.m2.1.1.2" xref="S2.SS4.p2.2.m2.1.1.2.cmml">G</mi><mi id="S2.SS4.p2.2.m2.1.1.3" xref="S2.SS4.p2.2.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.2.m2.1b"><apply id="S2.SS4.p2.2.m2.1.1.cmml" xref="S2.SS4.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS4.p2.2.m2.1.1.1.cmml" xref="S2.SS4.p2.2.m2.1.1">subscript</csymbol><ci id="S2.SS4.p2.2.m2.1.1.2.cmml" xref="S2.SS4.p2.2.m2.1.1.2">𝐺</ci><ci id="S2.SS4.p2.2.m2.1.1.3.cmml" xref="S2.SS4.p2.2.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.2.m2.1c">G_{t}</annotation></semantics></math> for <math id="S2.SS4.p2.3.m3.3" class="ltx_Math" alttext="t=1,\dots,T" display="inline"><semantics id="S2.SS4.p2.3.m3.3a"><mrow id="S2.SS4.p2.3.m3.3.4" xref="S2.SS4.p2.3.m3.3.4.cmml"><mi id="S2.SS4.p2.3.m3.3.4.2" xref="S2.SS4.p2.3.m3.3.4.2.cmml">t</mi><mo id="S2.SS4.p2.3.m3.3.4.1" xref="S2.SS4.p2.3.m3.3.4.1.cmml">=</mo><mrow id="S2.SS4.p2.3.m3.3.4.3.2" xref="S2.SS4.p2.3.m3.3.4.3.1.cmml"><mn id="S2.SS4.p2.3.m3.1.1" xref="S2.SS4.p2.3.m3.1.1.cmml">1</mn><mo id="S2.SS4.p2.3.m3.3.4.3.2.1" xref="S2.SS4.p2.3.m3.3.4.3.1.cmml">,</mo><mi mathvariant="normal" id="S2.SS4.p2.3.m3.2.2" xref="S2.SS4.p2.3.m3.2.2.cmml">…</mi><mo id="S2.SS4.p2.3.m3.3.4.3.2.2" xref="S2.SS4.p2.3.m3.3.4.3.1.cmml">,</mo><mi id="S2.SS4.p2.3.m3.3.3" xref="S2.SS4.p2.3.m3.3.3.cmml">T</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.3.m3.3b"><apply id="S2.SS4.p2.3.m3.3.4.cmml" xref="S2.SS4.p2.3.m3.3.4"><eq id="S2.SS4.p2.3.m3.3.4.1.cmml" xref="S2.SS4.p2.3.m3.3.4.1"></eq><ci id="S2.SS4.p2.3.m3.3.4.2.cmml" xref="S2.SS4.p2.3.m3.3.4.2">𝑡</ci><list id="S2.SS4.p2.3.m3.3.4.3.1.cmml" xref="S2.SS4.p2.3.m3.3.4.3.2"><cn type="integer" id="S2.SS4.p2.3.m3.1.1.cmml" xref="S2.SS4.p2.3.m3.1.1">1</cn><ci id="S2.SS4.p2.3.m3.2.2.cmml" xref="S2.SS4.p2.3.m3.2.2">…</ci><ci id="S2.SS4.p2.3.m3.3.3.cmml" xref="S2.SS4.p2.3.m3.3.3">𝑇</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.3.m3.3c">t=1,\dots,T</annotation></semantics></math>, we can then obtain all the <math id="S2.SS4.p2.4.m4.1" class="ltx_Math" alttext="W_{t}" display="inline"><semantics id="S2.SS4.p2.4.m4.1a"><msub id="S2.SS4.p2.4.m4.1.1" xref="S2.SS4.p2.4.m4.1.1.cmml"><mi id="S2.SS4.p2.4.m4.1.1.2" xref="S2.SS4.p2.4.m4.1.1.2.cmml">W</mi><mi id="S2.SS4.p2.4.m4.1.1.3" xref="S2.SS4.p2.4.m4.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.4.m4.1b"><apply id="S2.SS4.p2.4.m4.1.1.cmml" xref="S2.SS4.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS4.p2.4.m4.1.1.1.cmml" xref="S2.SS4.p2.4.m4.1.1">subscript</csymbol><ci id="S2.SS4.p2.4.m4.1.1.2.cmml" xref="S2.SS4.p2.4.m4.1.1.2">𝑊</ci><ci id="S2.SS4.p2.4.m4.1.1.3.cmml" xref="S2.SS4.p2.4.m4.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.4.m4.1c">W_{t}</annotation></semantics></math>s through a <span id="S2.SS4.p2.5.2" class="ltx_text ltx_font_typewriter">cumsum</span> by the second half of Equation <a href="#S2.E6" title="In 2.4 Parallelization with mini-batch TTT ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.
Our naive update rule, known as <em id="S2.SS4.p2.5.3" class="ltx_emph ltx_font_italic">online gradient descent</em>,
uses <span id="S2.SS4.p2.5.1" class="ltx_text"><math id="S2.SS4.p2.5.1.m1.2" class="ltx_Math" alttext="G_{t}=\nabla l(W_{t-1};x_{t})" display="inline"><semantics id="S2.SS4.p2.5.1.m1.2a"><mrow id="S2.SS4.p2.5.1.m1.2.2" xref="S2.SS4.p2.5.1.m1.2.2.cmml"><msub id="S2.SS4.p2.5.1.m1.2.2.4" xref="S2.SS4.p2.5.1.m1.2.2.4.cmml"><mi id="S2.SS4.p2.5.1.m1.2.2.4.2" xref="S2.SS4.p2.5.1.m1.2.2.4.2.cmml">G</mi><mi id="S2.SS4.p2.5.1.m1.2.2.4.3" xref="S2.SS4.p2.5.1.m1.2.2.4.3.cmml">t</mi></msub><mo id="S2.SS4.p2.5.1.m1.2.2.3" xref="S2.SS4.p2.5.1.m1.2.2.3.cmml">=</mo><mrow id="S2.SS4.p2.5.1.m1.2.2.2" xref="S2.SS4.p2.5.1.m1.2.2.2.cmml"><mrow id="S2.SS4.p2.5.1.m1.2.2.2.4" xref="S2.SS4.p2.5.1.m1.2.2.2.4.cmml"><mo rspace="0.167em" id="S2.SS4.p2.5.1.m1.2.2.2.4.1" xref="S2.SS4.p2.5.1.m1.2.2.2.4.1.cmml">∇</mo><mi id="S2.SS4.p2.5.1.m1.2.2.2.4.2" xref="S2.SS4.p2.5.1.m1.2.2.2.4.2.cmml">l</mi></mrow><mo lspace="0em" rspace="0em" id="S2.SS4.p2.5.1.m1.2.2.2.3" xref="S2.SS4.p2.5.1.m1.2.2.2.3.cmml">​</mo><mrow id="S2.SS4.p2.5.1.m1.2.2.2.2.2" xref="S2.SS4.p2.5.1.m1.2.2.2.2.3.cmml"><mo stretchy="false" id="S2.SS4.p2.5.1.m1.2.2.2.2.2.3" xref="S2.SS4.p2.5.1.m1.2.2.2.2.3.cmml">(</mo><msub id="S2.SS4.p2.5.1.m1.1.1.1.1.1.1" xref="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.cmml"><mi id="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.2" xref="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.2.cmml">W</mi><mrow id="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.3" xref="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.3.cmml"><mi id="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.3.2" xref="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.3.2.cmml">t</mi><mo id="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.3.1" xref="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.3.1.cmml">−</mo><mn id="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.3.3" xref="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.3.3.cmml">1</mn></mrow></msub><mo id="S2.SS4.p2.5.1.m1.2.2.2.2.2.4" xref="S2.SS4.p2.5.1.m1.2.2.2.2.3.cmml">;</mo><msub id="S2.SS4.p2.5.1.m1.2.2.2.2.2.2" xref="S2.SS4.p2.5.1.m1.2.2.2.2.2.2.cmml"><mi id="S2.SS4.p2.5.1.m1.2.2.2.2.2.2.2" xref="S2.SS4.p2.5.1.m1.2.2.2.2.2.2.2.cmml">x</mi><mi id="S2.SS4.p2.5.1.m1.2.2.2.2.2.2.3" xref="S2.SS4.p2.5.1.m1.2.2.2.2.2.2.3.cmml">t</mi></msub><mo stretchy="false" id="S2.SS4.p2.5.1.m1.2.2.2.2.2.5" xref="S2.SS4.p2.5.1.m1.2.2.2.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p2.5.1.m1.2b"><apply id="S2.SS4.p2.5.1.m1.2.2.cmml" xref="S2.SS4.p2.5.1.m1.2.2"><eq id="S2.SS4.p2.5.1.m1.2.2.3.cmml" xref="S2.SS4.p2.5.1.m1.2.2.3"></eq><apply id="S2.SS4.p2.5.1.m1.2.2.4.cmml" xref="S2.SS4.p2.5.1.m1.2.2.4"><csymbol cd="ambiguous" id="S2.SS4.p2.5.1.m1.2.2.4.1.cmml" xref="S2.SS4.p2.5.1.m1.2.2.4">subscript</csymbol><ci id="S2.SS4.p2.5.1.m1.2.2.4.2.cmml" xref="S2.SS4.p2.5.1.m1.2.2.4.2">𝐺</ci><ci id="S2.SS4.p2.5.1.m1.2.2.4.3.cmml" xref="S2.SS4.p2.5.1.m1.2.2.4.3">𝑡</ci></apply><apply id="S2.SS4.p2.5.1.m1.2.2.2.cmml" xref="S2.SS4.p2.5.1.m1.2.2.2"><times id="S2.SS4.p2.5.1.m1.2.2.2.3.cmml" xref="S2.SS4.p2.5.1.m1.2.2.2.3"></times><apply id="S2.SS4.p2.5.1.m1.2.2.2.4.cmml" xref="S2.SS4.p2.5.1.m1.2.2.2.4"><ci id="S2.SS4.p2.5.1.m1.2.2.2.4.1.cmml" xref="S2.SS4.p2.5.1.m1.2.2.2.4.1">∇</ci><ci id="S2.SS4.p2.5.1.m1.2.2.2.4.2.cmml" xref="S2.SS4.p2.5.1.m1.2.2.2.4.2">𝑙</ci></apply><list id="S2.SS4.p2.5.1.m1.2.2.2.2.3.cmml" xref="S2.SS4.p2.5.1.m1.2.2.2.2.2"><apply id="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.cmml" xref="S2.SS4.p2.5.1.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.1.cmml" xref="S2.SS4.p2.5.1.m1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.2.cmml" xref="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.2">𝑊</ci><apply id="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.3.cmml" xref="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.3"><minus id="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.3.1.cmml" xref="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.3.1"></minus><ci id="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.3.2.cmml" xref="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.3.2">𝑡</ci><cn type="integer" id="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.3.3.cmml" xref="S2.SS4.p2.5.1.m1.1.1.1.1.1.1.3.3">1</cn></apply></apply><apply id="S2.SS4.p2.5.1.m1.2.2.2.2.2.2.cmml" xref="S2.SS4.p2.5.1.m1.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS4.p2.5.1.m1.2.2.2.2.2.2.1.cmml" xref="S2.SS4.p2.5.1.m1.2.2.2.2.2.2">subscript</csymbol><ci id="S2.SS4.p2.5.1.m1.2.2.2.2.2.2.2.cmml" xref="S2.SS4.p2.5.1.m1.2.2.2.2.2.2.2">𝑥</ci><ci id="S2.SS4.p2.5.1.m1.2.2.2.2.2.2.3.cmml" xref="S2.SS4.p2.5.1.m1.2.2.2.2.2.2.3">𝑡</ci></apply></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p2.5.1.m1.2c">G_{t}=\nabla l(W_{t-1};x_{t})</annotation></semantics></math></span>.</p>
</div>
<div id="S2.SS4.p3" class="ltx_para">
<p id="S2.SS4.p3.12" class="ltx_p">To parallelize <math id="S2.SS4.p3.1.m1.1" class="ltx_Math" alttext="G_{t}" display="inline"><semantics id="S2.SS4.p3.1.m1.1a"><msub id="S2.SS4.p3.1.m1.1.1" xref="S2.SS4.p3.1.m1.1.1.cmml"><mi id="S2.SS4.p3.1.m1.1.1.2" xref="S2.SS4.p3.1.m1.1.1.2.cmml">G</mi><mi id="S2.SS4.p3.1.m1.1.1.3" xref="S2.SS4.p3.1.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p3.1.m1.1b"><apply id="S2.SS4.p3.1.m1.1.1.cmml" xref="S2.SS4.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS4.p3.1.m1.1.1.1.cmml" xref="S2.SS4.p3.1.m1.1.1">subscript</csymbol><ci id="S2.SS4.p3.1.m1.1.1.2.cmml" xref="S2.SS4.p3.1.m1.1.1.2">𝐺</ci><ci id="S2.SS4.p3.1.m1.1.1.3.cmml" xref="S2.SS4.p3.1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p3.1.m1.1c">G_{t}</annotation></semantics></math> for <math id="S2.SS4.p3.2.m2.3" class="ltx_Math" alttext="t=1,\dots,T" display="inline"><semantics id="S2.SS4.p3.2.m2.3a"><mrow id="S2.SS4.p3.2.m2.3.4" xref="S2.SS4.p3.2.m2.3.4.cmml"><mi id="S2.SS4.p3.2.m2.3.4.2" xref="S2.SS4.p3.2.m2.3.4.2.cmml">t</mi><mo id="S2.SS4.p3.2.m2.3.4.1" xref="S2.SS4.p3.2.m2.3.4.1.cmml">=</mo><mrow id="S2.SS4.p3.2.m2.3.4.3.2" xref="S2.SS4.p3.2.m2.3.4.3.1.cmml"><mn id="S2.SS4.p3.2.m2.1.1" xref="S2.SS4.p3.2.m2.1.1.cmml">1</mn><mo id="S2.SS4.p3.2.m2.3.4.3.2.1" xref="S2.SS4.p3.2.m2.3.4.3.1.cmml">,</mo><mi mathvariant="normal" id="S2.SS4.p3.2.m2.2.2" xref="S2.SS4.p3.2.m2.2.2.cmml">…</mi><mo id="S2.SS4.p3.2.m2.3.4.3.2.2" xref="S2.SS4.p3.2.m2.3.4.3.1.cmml">,</mo><mi id="S2.SS4.p3.2.m2.3.3" xref="S2.SS4.p3.2.m2.3.3.cmml">T</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p3.2.m2.3b"><apply id="S2.SS4.p3.2.m2.3.4.cmml" xref="S2.SS4.p3.2.m2.3.4"><eq id="S2.SS4.p3.2.m2.3.4.1.cmml" xref="S2.SS4.p3.2.m2.3.4.1"></eq><ci id="S2.SS4.p3.2.m2.3.4.2.cmml" xref="S2.SS4.p3.2.m2.3.4.2">𝑡</ci><list id="S2.SS4.p3.2.m2.3.4.3.1.cmml" xref="S2.SS4.p3.2.m2.3.4.3.2"><cn type="integer" id="S2.SS4.p3.2.m2.1.1.cmml" xref="S2.SS4.p3.2.m2.1.1">1</cn><ci id="S2.SS4.p3.2.m2.2.2.cmml" xref="S2.SS4.p3.2.m2.2.2">…</ci><ci id="S2.SS4.p3.2.m2.3.3.cmml" xref="S2.SS4.p3.2.m2.3.3">𝑇</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p3.2.m2.3c">t=1,\dots,T</annotation></semantics></math>, we can take all of them w.r.t. <math id="S2.SS4.p3.3.m3.1" class="ltx_Math" alttext="W_{0}" display="inline"><semantics id="S2.SS4.p3.3.m3.1a"><msub id="S2.SS4.p3.3.m3.1.1" xref="S2.SS4.p3.3.m3.1.1.cmml"><mi id="S2.SS4.p3.3.m3.1.1.2" xref="S2.SS4.p3.3.m3.1.1.2.cmml">W</mi><mn id="S2.SS4.p3.3.m3.1.1.3" xref="S2.SS4.p3.3.m3.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p3.3.m3.1b"><apply id="S2.SS4.p3.3.m3.1.1.cmml" xref="S2.SS4.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS4.p3.3.m3.1.1.1.cmml" xref="S2.SS4.p3.3.m3.1.1">subscript</csymbol><ci id="S2.SS4.p3.3.m3.1.1.2.cmml" xref="S2.SS4.p3.3.m3.1.1.2">𝑊</ci><cn type="integer" id="S2.SS4.p3.3.m3.1.1.3.cmml" xref="S2.SS4.p3.3.m3.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p3.3.m3.1c">W_{0}</annotation></semantics></math>.
This variant with <math id="S2.SS4.p3.4.m4.2" class="ltx_Math" alttext="G_{t}=\nabla\ell\left(W_{0};x_{t}\right)" display="inline"><semantics id="S2.SS4.p3.4.m4.2a"><mrow id="S2.SS4.p3.4.m4.2.2" xref="S2.SS4.p3.4.m4.2.2.cmml"><msub id="S2.SS4.p3.4.m4.2.2.4" xref="S2.SS4.p3.4.m4.2.2.4.cmml"><mi id="S2.SS4.p3.4.m4.2.2.4.2" xref="S2.SS4.p3.4.m4.2.2.4.2.cmml">G</mi><mi id="S2.SS4.p3.4.m4.2.2.4.3" xref="S2.SS4.p3.4.m4.2.2.4.3.cmml">t</mi></msub><mo id="S2.SS4.p3.4.m4.2.2.3" xref="S2.SS4.p3.4.m4.2.2.3.cmml">=</mo><mrow id="S2.SS4.p3.4.m4.2.2.2" xref="S2.SS4.p3.4.m4.2.2.2.cmml"><mrow id="S2.SS4.p3.4.m4.2.2.2.4" xref="S2.SS4.p3.4.m4.2.2.2.4.cmml"><mo rspace="0.167em" id="S2.SS4.p3.4.m4.2.2.2.4.1" xref="S2.SS4.p3.4.m4.2.2.2.4.1.cmml">∇</mo><mi mathvariant="normal" id="S2.SS4.p3.4.m4.2.2.2.4.2" xref="S2.SS4.p3.4.m4.2.2.2.4.2.cmml">ℓ</mi></mrow><mo lspace="0em" rspace="0em" id="S2.SS4.p3.4.m4.2.2.2.3" xref="S2.SS4.p3.4.m4.2.2.2.3.cmml">​</mo><mrow id="S2.SS4.p3.4.m4.2.2.2.2.2" xref="S2.SS4.p3.4.m4.2.2.2.2.3.cmml"><mo id="S2.SS4.p3.4.m4.2.2.2.2.2.3" xref="S2.SS4.p3.4.m4.2.2.2.2.3.cmml">(</mo><msub id="S2.SS4.p3.4.m4.1.1.1.1.1.1" xref="S2.SS4.p3.4.m4.1.1.1.1.1.1.cmml"><mi id="S2.SS4.p3.4.m4.1.1.1.1.1.1.2" xref="S2.SS4.p3.4.m4.1.1.1.1.1.1.2.cmml">W</mi><mn id="S2.SS4.p3.4.m4.1.1.1.1.1.1.3" xref="S2.SS4.p3.4.m4.1.1.1.1.1.1.3.cmml">0</mn></msub><mo id="S2.SS4.p3.4.m4.2.2.2.2.2.4" xref="S2.SS4.p3.4.m4.2.2.2.2.3.cmml">;</mo><msub id="S2.SS4.p3.4.m4.2.2.2.2.2.2" xref="S2.SS4.p3.4.m4.2.2.2.2.2.2.cmml"><mi id="S2.SS4.p3.4.m4.2.2.2.2.2.2.2" xref="S2.SS4.p3.4.m4.2.2.2.2.2.2.2.cmml">x</mi><mi id="S2.SS4.p3.4.m4.2.2.2.2.2.2.3" xref="S2.SS4.p3.4.m4.2.2.2.2.2.2.3.cmml">t</mi></msub><mo id="S2.SS4.p3.4.m4.2.2.2.2.2.5" xref="S2.SS4.p3.4.m4.2.2.2.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p3.4.m4.2b"><apply id="S2.SS4.p3.4.m4.2.2.cmml" xref="S2.SS4.p3.4.m4.2.2"><eq id="S2.SS4.p3.4.m4.2.2.3.cmml" xref="S2.SS4.p3.4.m4.2.2.3"></eq><apply id="S2.SS4.p3.4.m4.2.2.4.cmml" xref="S2.SS4.p3.4.m4.2.2.4"><csymbol cd="ambiguous" id="S2.SS4.p3.4.m4.2.2.4.1.cmml" xref="S2.SS4.p3.4.m4.2.2.4">subscript</csymbol><ci id="S2.SS4.p3.4.m4.2.2.4.2.cmml" xref="S2.SS4.p3.4.m4.2.2.4.2">𝐺</ci><ci id="S2.SS4.p3.4.m4.2.2.4.3.cmml" xref="S2.SS4.p3.4.m4.2.2.4.3">𝑡</ci></apply><apply id="S2.SS4.p3.4.m4.2.2.2.cmml" xref="S2.SS4.p3.4.m4.2.2.2"><times id="S2.SS4.p3.4.m4.2.2.2.3.cmml" xref="S2.SS4.p3.4.m4.2.2.2.3"></times><apply id="S2.SS4.p3.4.m4.2.2.2.4.cmml" xref="S2.SS4.p3.4.m4.2.2.2.4"><ci id="S2.SS4.p3.4.m4.2.2.2.4.1.cmml" xref="S2.SS4.p3.4.m4.2.2.2.4.1">∇</ci><ci id="S2.SS4.p3.4.m4.2.2.2.4.2.cmml" xref="S2.SS4.p3.4.m4.2.2.2.4.2">ℓ</ci></apply><list id="S2.SS4.p3.4.m4.2.2.2.2.3.cmml" xref="S2.SS4.p3.4.m4.2.2.2.2.2"><apply id="S2.SS4.p3.4.m4.1.1.1.1.1.1.cmml" xref="S2.SS4.p3.4.m4.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS4.p3.4.m4.1.1.1.1.1.1.1.cmml" xref="S2.SS4.p3.4.m4.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS4.p3.4.m4.1.1.1.1.1.1.2.cmml" xref="S2.SS4.p3.4.m4.1.1.1.1.1.1.2">𝑊</ci><cn type="integer" id="S2.SS4.p3.4.m4.1.1.1.1.1.1.3.cmml" xref="S2.SS4.p3.4.m4.1.1.1.1.1.1.3">0</cn></apply><apply id="S2.SS4.p3.4.m4.2.2.2.2.2.2.cmml" xref="S2.SS4.p3.4.m4.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS4.p3.4.m4.2.2.2.2.2.2.1.cmml" xref="S2.SS4.p3.4.m4.2.2.2.2.2.2">subscript</csymbol><ci id="S2.SS4.p3.4.m4.2.2.2.2.2.2.2.cmml" xref="S2.SS4.p3.4.m4.2.2.2.2.2.2.2">𝑥</ci><ci id="S2.SS4.p3.4.m4.2.2.2.2.2.2.3.cmml" xref="S2.SS4.p3.4.m4.2.2.2.2.2.2.3">𝑡</ci></apply></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p3.4.m4.2c">G_{t}=\nabla\ell\left(W_{0};x_{t}\right)</annotation></semantics></math> is known as <em id="S2.SS4.p3.12.1" class="ltx_emph ltx_font_italic">batch gradient descent</em>, since
<math id="S2.SS4.p3.5.m5.2" class="ltx_Math" alttext="\sum_{s=1}^{t}\nabla\ell\left(W_{0};x_{t}\right)" display="inline"><semantics id="S2.SS4.p3.5.m5.2a"><mrow id="S2.SS4.p3.5.m5.2.2" xref="S2.SS4.p3.5.m5.2.2.cmml"><msubsup id="S2.SS4.p3.5.m5.2.2.3" xref="S2.SS4.p3.5.m5.2.2.3.cmml"><mo id="S2.SS4.p3.5.m5.2.2.3.2.2" xref="S2.SS4.p3.5.m5.2.2.3.2.2.cmml">∑</mo><mrow id="S2.SS4.p3.5.m5.2.2.3.2.3" xref="S2.SS4.p3.5.m5.2.2.3.2.3.cmml"><mi id="S2.SS4.p3.5.m5.2.2.3.2.3.2" xref="S2.SS4.p3.5.m5.2.2.3.2.3.2.cmml">s</mi><mo id="S2.SS4.p3.5.m5.2.2.3.2.3.1" xref="S2.SS4.p3.5.m5.2.2.3.2.3.1.cmml">=</mo><mn id="S2.SS4.p3.5.m5.2.2.3.2.3.3" xref="S2.SS4.p3.5.m5.2.2.3.2.3.3.cmml">1</mn></mrow><mi id="S2.SS4.p3.5.m5.2.2.3.3" xref="S2.SS4.p3.5.m5.2.2.3.3.cmml">t</mi></msubsup><mrow id="S2.SS4.p3.5.m5.2.2.2" xref="S2.SS4.p3.5.m5.2.2.2.cmml"><mrow id="S2.SS4.p3.5.m5.2.2.2.4" xref="S2.SS4.p3.5.m5.2.2.2.4.cmml"><mo rspace="0.167em" id="S2.SS4.p3.5.m5.2.2.2.4.1" xref="S2.SS4.p3.5.m5.2.2.2.4.1.cmml">∇</mo><mi mathvariant="normal" id="S2.SS4.p3.5.m5.2.2.2.4.2" xref="S2.SS4.p3.5.m5.2.2.2.4.2.cmml">ℓ</mi></mrow><mo lspace="0em" rspace="0em" id="S2.SS4.p3.5.m5.2.2.2.3" xref="S2.SS4.p3.5.m5.2.2.2.3.cmml">​</mo><mrow id="S2.SS4.p3.5.m5.2.2.2.2.2" xref="S2.SS4.p3.5.m5.2.2.2.2.3.cmml"><mo id="S2.SS4.p3.5.m5.2.2.2.2.2.3" xref="S2.SS4.p3.5.m5.2.2.2.2.3.cmml">(</mo><msub id="S2.SS4.p3.5.m5.1.1.1.1.1.1" xref="S2.SS4.p3.5.m5.1.1.1.1.1.1.cmml"><mi id="S2.SS4.p3.5.m5.1.1.1.1.1.1.2" xref="S2.SS4.p3.5.m5.1.1.1.1.1.1.2.cmml">W</mi><mn id="S2.SS4.p3.5.m5.1.1.1.1.1.1.3" xref="S2.SS4.p3.5.m5.1.1.1.1.1.1.3.cmml">0</mn></msub><mo id="S2.SS4.p3.5.m5.2.2.2.2.2.4" xref="S2.SS4.p3.5.m5.2.2.2.2.3.cmml">;</mo><msub id="S2.SS4.p3.5.m5.2.2.2.2.2.2" xref="S2.SS4.p3.5.m5.2.2.2.2.2.2.cmml"><mi id="S2.SS4.p3.5.m5.2.2.2.2.2.2.2" xref="S2.SS4.p3.5.m5.2.2.2.2.2.2.2.cmml">x</mi><mi id="S2.SS4.p3.5.m5.2.2.2.2.2.2.3" xref="S2.SS4.p3.5.m5.2.2.2.2.2.2.3.cmml">t</mi></msub><mo id="S2.SS4.p3.5.m5.2.2.2.2.2.5" xref="S2.SS4.p3.5.m5.2.2.2.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p3.5.m5.2b"><apply id="S2.SS4.p3.5.m5.2.2.cmml" xref="S2.SS4.p3.5.m5.2.2"><apply id="S2.SS4.p3.5.m5.2.2.3.cmml" xref="S2.SS4.p3.5.m5.2.2.3"><csymbol cd="ambiguous" id="S2.SS4.p3.5.m5.2.2.3.1.cmml" xref="S2.SS4.p3.5.m5.2.2.3">superscript</csymbol><apply id="S2.SS4.p3.5.m5.2.2.3.2.cmml" xref="S2.SS4.p3.5.m5.2.2.3"><csymbol cd="ambiguous" id="S2.SS4.p3.5.m5.2.2.3.2.1.cmml" xref="S2.SS4.p3.5.m5.2.2.3">subscript</csymbol><sum id="S2.SS4.p3.5.m5.2.2.3.2.2.cmml" xref="S2.SS4.p3.5.m5.2.2.3.2.2"></sum><apply id="S2.SS4.p3.5.m5.2.2.3.2.3.cmml" xref="S2.SS4.p3.5.m5.2.2.3.2.3"><eq id="S2.SS4.p3.5.m5.2.2.3.2.3.1.cmml" xref="S2.SS4.p3.5.m5.2.2.3.2.3.1"></eq><ci id="S2.SS4.p3.5.m5.2.2.3.2.3.2.cmml" xref="S2.SS4.p3.5.m5.2.2.3.2.3.2">𝑠</ci><cn type="integer" id="S2.SS4.p3.5.m5.2.2.3.2.3.3.cmml" xref="S2.SS4.p3.5.m5.2.2.3.2.3.3">1</cn></apply></apply><ci id="S2.SS4.p3.5.m5.2.2.3.3.cmml" xref="S2.SS4.p3.5.m5.2.2.3.3">𝑡</ci></apply><apply id="S2.SS4.p3.5.m5.2.2.2.cmml" xref="S2.SS4.p3.5.m5.2.2.2"><times id="S2.SS4.p3.5.m5.2.2.2.3.cmml" xref="S2.SS4.p3.5.m5.2.2.2.3"></times><apply id="S2.SS4.p3.5.m5.2.2.2.4.cmml" xref="S2.SS4.p3.5.m5.2.2.2.4"><ci id="S2.SS4.p3.5.m5.2.2.2.4.1.cmml" xref="S2.SS4.p3.5.m5.2.2.2.4.1">∇</ci><ci id="S2.SS4.p3.5.m5.2.2.2.4.2.cmml" xref="S2.SS4.p3.5.m5.2.2.2.4.2">ℓ</ci></apply><list id="S2.SS4.p3.5.m5.2.2.2.2.3.cmml" xref="S2.SS4.p3.5.m5.2.2.2.2.2"><apply id="S2.SS4.p3.5.m5.1.1.1.1.1.1.cmml" xref="S2.SS4.p3.5.m5.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS4.p3.5.m5.1.1.1.1.1.1.1.cmml" xref="S2.SS4.p3.5.m5.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS4.p3.5.m5.1.1.1.1.1.1.2.cmml" xref="S2.SS4.p3.5.m5.1.1.1.1.1.1.2">𝑊</ci><cn type="integer" id="S2.SS4.p3.5.m5.1.1.1.1.1.1.3.cmml" xref="S2.SS4.p3.5.m5.1.1.1.1.1.1.3">0</cn></apply><apply id="S2.SS4.p3.5.m5.2.2.2.2.2.2.cmml" xref="S2.SS4.p3.5.m5.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS4.p3.5.m5.2.2.2.2.2.2.1.cmml" xref="S2.SS4.p3.5.m5.2.2.2.2.2.2">subscript</csymbol><ci id="S2.SS4.p3.5.m5.2.2.2.2.2.2.2.cmml" xref="S2.SS4.p3.5.m5.2.2.2.2.2.2.2">𝑥</ci><ci id="S2.SS4.p3.5.m5.2.2.2.2.2.2.3.cmml" xref="S2.SS4.p3.5.m5.2.2.2.2.2.2.3">𝑡</ci></apply></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p3.5.m5.2c">\sum_{s=1}^{t}\nabla\ell\left(W_{0};x_{t}\right)</annotation></semantics></math> is the same as the gradient w.r.t. <math id="S2.SS4.p3.6.m6.1" class="ltx_Math" alttext="W_{0}" display="inline"><semantics id="S2.SS4.p3.6.m6.1a"><msub id="S2.SS4.p3.6.m6.1.1" xref="S2.SS4.p3.6.m6.1.1.cmml"><mi id="S2.SS4.p3.6.m6.1.1.2" xref="S2.SS4.p3.6.m6.1.1.2.cmml">W</mi><mn id="S2.SS4.p3.6.m6.1.1.3" xref="S2.SS4.p3.6.m6.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p3.6.m6.1b"><apply id="S2.SS4.p3.6.m6.1.1.cmml" xref="S2.SS4.p3.6.m6.1.1"><csymbol cd="ambiguous" id="S2.SS4.p3.6.m6.1.1.1.cmml" xref="S2.SS4.p3.6.m6.1.1">subscript</csymbol><ci id="S2.SS4.p3.6.m6.1.1.2.cmml" xref="S2.SS4.p3.6.m6.1.1.2">𝑊</ci><cn type="integer" id="S2.SS4.p3.6.m6.1.1.3.cmml" xref="S2.SS4.p3.6.m6.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p3.6.m6.1c">W_{0}</annotation></semantics></math> over <math id="S2.SS4.p3.7.m7.3" class="ltx_Math" alttext="x_{1},\dots,x_{t}" display="inline"><semantics id="S2.SS4.p3.7.m7.3a"><mrow id="S2.SS4.p3.7.m7.3.3.2" xref="S2.SS4.p3.7.m7.3.3.3.cmml"><msub id="S2.SS4.p3.7.m7.2.2.1.1" xref="S2.SS4.p3.7.m7.2.2.1.1.cmml"><mi id="S2.SS4.p3.7.m7.2.2.1.1.2" xref="S2.SS4.p3.7.m7.2.2.1.1.2.cmml">x</mi><mn id="S2.SS4.p3.7.m7.2.2.1.1.3" xref="S2.SS4.p3.7.m7.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.SS4.p3.7.m7.3.3.2.3" xref="S2.SS4.p3.7.m7.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS4.p3.7.m7.1.1" xref="S2.SS4.p3.7.m7.1.1.cmml">…</mi><mo id="S2.SS4.p3.7.m7.3.3.2.4" xref="S2.SS4.p3.7.m7.3.3.3.cmml">,</mo><msub id="S2.SS4.p3.7.m7.3.3.2.2" xref="S2.SS4.p3.7.m7.3.3.2.2.cmml"><mi id="S2.SS4.p3.7.m7.3.3.2.2.2" xref="S2.SS4.p3.7.m7.3.3.2.2.2.cmml">x</mi><mi id="S2.SS4.p3.7.m7.3.3.2.2.3" xref="S2.SS4.p3.7.m7.3.3.2.2.3.cmml">t</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p3.7.m7.3b"><list id="S2.SS4.p3.7.m7.3.3.3.cmml" xref="S2.SS4.p3.7.m7.3.3.2"><apply id="S2.SS4.p3.7.m7.2.2.1.1.cmml" xref="S2.SS4.p3.7.m7.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS4.p3.7.m7.2.2.1.1.1.cmml" xref="S2.SS4.p3.7.m7.2.2.1.1">subscript</csymbol><ci id="S2.SS4.p3.7.m7.2.2.1.1.2.cmml" xref="S2.SS4.p3.7.m7.2.2.1.1.2">𝑥</ci><cn type="integer" id="S2.SS4.p3.7.m7.2.2.1.1.3.cmml" xref="S2.SS4.p3.7.m7.2.2.1.1.3">1</cn></apply><ci id="S2.SS4.p3.7.m7.1.1.cmml" xref="S2.SS4.p3.7.m7.1.1">…</ci><apply id="S2.SS4.p3.7.m7.3.3.2.2.cmml" xref="S2.SS4.p3.7.m7.3.3.2.2"><csymbol cd="ambiguous" id="S2.SS4.p3.7.m7.3.3.2.2.1.cmml" xref="S2.SS4.p3.7.m7.3.3.2.2">subscript</csymbol><ci id="S2.SS4.p3.7.m7.3.3.2.2.2.cmml" xref="S2.SS4.p3.7.m7.3.3.2.2.2">𝑥</ci><ci id="S2.SS4.p3.7.m7.3.3.2.2.3.cmml" xref="S2.SS4.p3.7.m7.3.3.2.2.3">𝑡</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p3.7.m7.3c">x_{1},\dots,x_{t}</annotation></semantics></math> as a batch.
However, in batch GD, <math id="S2.SS4.p3.8.m8.1" class="ltx_Math" alttext="W_{t}" display="inline"><semantics id="S2.SS4.p3.8.m8.1a"><msub id="S2.SS4.p3.8.m8.1.1" xref="S2.SS4.p3.8.m8.1.1.cmml"><mi id="S2.SS4.p3.8.m8.1.1.2" xref="S2.SS4.p3.8.m8.1.1.2.cmml">W</mi><mi id="S2.SS4.p3.8.m8.1.1.3" xref="S2.SS4.p3.8.m8.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p3.8.m8.1b"><apply id="S2.SS4.p3.8.m8.1.1.cmml" xref="S2.SS4.p3.8.m8.1.1"><csymbol cd="ambiguous" id="S2.SS4.p3.8.m8.1.1.1.cmml" xref="S2.SS4.p3.8.m8.1.1">subscript</csymbol><ci id="S2.SS4.p3.8.m8.1.1.2.cmml" xref="S2.SS4.p3.8.m8.1.1.2">𝑊</ci><ci id="S2.SS4.p3.8.m8.1.1.3.cmml" xref="S2.SS4.p3.8.m8.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p3.8.m8.1c">W_{t}</annotation></semantics></math> is effectively only one gradient step away from <math id="S2.SS4.p3.9.m9.1" class="ltx_Math" alttext="W_{0}" display="inline"><semantics id="S2.SS4.p3.9.m9.1a"><msub id="S2.SS4.p3.9.m9.1.1" xref="S2.SS4.p3.9.m9.1.1.cmml"><mi id="S2.SS4.p3.9.m9.1.1.2" xref="S2.SS4.p3.9.m9.1.1.2.cmml">W</mi><mn id="S2.SS4.p3.9.m9.1.1.3" xref="S2.SS4.p3.9.m9.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p3.9.m9.1b"><apply id="S2.SS4.p3.9.m9.1.1.cmml" xref="S2.SS4.p3.9.m9.1.1"><csymbol cd="ambiguous" id="S2.SS4.p3.9.m9.1.1.1.cmml" xref="S2.SS4.p3.9.m9.1.1">subscript</csymbol><ci id="S2.SS4.p3.9.m9.1.1.2.cmml" xref="S2.SS4.p3.9.m9.1.1.2">𝑊</ci><cn type="integer" id="S2.SS4.p3.9.m9.1.1.3.cmml" xref="S2.SS4.p3.9.m9.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p3.9.m9.1c">W_{0}</annotation></semantics></math>, in contrast to online GD, where <math id="S2.SS4.p3.10.m10.1" class="ltx_Math" alttext="W_{t}" display="inline"><semantics id="S2.SS4.p3.10.m10.1a"><msub id="S2.SS4.p3.10.m10.1.1" xref="S2.SS4.p3.10.m10.1.1.cmml"><mi id="S2.SS4.p3.10.m10.1.1.2" xref="S2.SS4.p3.10.m10.1.1.2.cmml">W</mi><mi id="S2.SS4.p3.10.m10.1.1.3" xref="S2.SS4.p3.10.m10.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p3.10.m10.1b"><apply id="S2.SS4.p3.10.m10.1.1.cmml" xref="S2.SS4.p3.10.m10.1.1"><csymbol cd="ambiguous" id="S2.SS4.p3.10.m10.1.1.1.cmml" xref="S2.SS4.p3.10.m10.1.1">subscript</csymbol><ci id="S2.SS4.p3.10.m10.1.1.2.cmml" xref="S2.SS4.p3.10.m10.1.1.2">𝑊</ci><ci id="S2.SS4.p3.10.m10.1.1.3.cmml" xref="S2.SS4.p3.10.m10.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p3.10.m10.1c">W_{t}</annotation></semantics></math> is <math id="S2.SS4.p3.11.m11.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S2.SS4.p3.11.m11.1a"><mi id="S2.SS4.p3.11.m11.1.1" xref="S2.SS4.p3.11.m11.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S2.SS4.p3.11.m11.1b"><ci id="S2.SS4.p3.11.m11.1.1.cmml" xref="S2.SS4.p3.11.m11.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p3.11.m11.1c">t</annotation></semantics></math> steps away from <math id="S2.SS4.p3.12.m12.1" class="ltx_Math" alttext="W_{0}" display="inline"><semantics id="S2.SS4.p3.12.m12.1a"><msub id="S2.SS4.p3.12.m12.1.1" xref="S2.SS4.p3.12.m12.1.1.cmml"><mi id="S2.SS4.p3.12.m12.1.1.2" xref="S2.SS4.p3.12.m12.1.1.2.cmml">W</mi><mn id="S2.SS4.p3.12.m12.1.1.3" xref="S2.SS4.p3.12.m12.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p3.12.m12.1b"><apply id="S2.SS4.p3.12.m12.1.1.cmml" xref="S2.SS4.p3.12.m12.1.1"><csymbol cd="ambiguous" id="S2.SS4.p3.12.m12.1.1.1.cmml" xref="S2.SS4.p3.12.m12.1.1">subscript</csymbol><ci id="S2.SS4.p3.12.m12.1.1.2.cmml" xref="S2.SS4.p3.12.m12.1.1.2">𝑊</ci><cn type="integer" id="S2.SS4.p3.12.m12.1.1.3.cmml" xref="S2.SS4.p3.12.m12.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p3.12.m12.1c">W_{0}</annotation></semantics></math>.
Therefore, batch GD has a smaller effective search space, which ends up hurting performance for language modeling.</p>
</div>
<figure id="S2.F7" class="ltx_figure"><img src="/html/2407.04620/assets/x7.png" id="S2.F7.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="300" height="127" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 7: </span>High-level computation graph of the first TTT mini-batch, where nodes are variables and edges are computations.
The blue nodes are input variables, and yellow are output.
<span id="S2.F7.9.1" class="ltx_text ltx_font_bold">Subsection <a href="#S2.SS4" title="2.4 Parallelization with mini-batch TTT ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.4</span></a></span>: Since <math id="S2.F7.4.m1.3" class="ltx_Math" alttext="G_{1},\dots,G_{b}" display="inline"><semantics id="S2.F7.4.m1.3b"><mrow id="S2.F7.4.m1.3.3.2" xref="S2.F7.4.m1.3.3.3.cmml"><msub id="S2.F7.4.m1.2.2.1.1" xref="S2.F7.4.m1.2.2.1.1.cmml"><mi id="S2.F7.4.m1.2.2.1.1.2" xref="S2.F7.4.m1.2.2.1.1.2.cmml">G</mi><mn id="S2.F7.4.m1.2.2.1.1.3" xref="S2.F7.4.m1.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.F7.4.m1.3.3.2.3" xref="S2.F7.4.m1.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.F7.4.m1.1.1" xref="S2.F7.4.m1.1.1.cmml">…</mi><mo id="S2.F7.4.m1.3.3.2.4" xref="S2.F7.4.m1.3.3.3.cmml">,</mo><msub id="S2.F7.4.m1.3.3.2.2" xref="S2.F7.4.m1.3.3.2.2.cmml"><mi id="S2.F7.4.m1.3.3.2.2.2" xref="S2.F7.4.m1.3.3.2.2.2.cmml">G</mi><mi id="S2.F7.4.m1.3.3.2.2.3" xref="S2.F7.4.m1.3.3.2.2.3.cmml">b</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.F7.4.m1.3c"><list id="S2.F7.4.m1.3.3.3.cmml" xref="S2.F7.4.m1.3.3.2"><apply id="S2.F7.4.m1.2.2.1.1.cmml" xref="S2.F7.4.m1.2.2.1.1"><csymbol cd="ambiguous" id="S2.F7.4.m1.2.2.1.1.1.cmml" xref="S2.F7.4.m1.2.2.1.1">subscript</csymbol><ci id="S2.F7.4.m1.2.2.1.1.2.cmml" xref="S2.F7.4.m1.2.2.1.1.2">𝐺</ci><cn type="integer" id="S2.F7.4.m1.2.2.1.1.3.cmml" xref="S2.F7.4.m1.2.2.1.1.3">1</cn></apply><ci id="S2.F7.4.m1.1.1.cmml" xref="S2.F7.4.m1.1.1">…</ci><apply id="S2.F7.4.m1.3.3.2.2.cmml" xref="S2.F7.4.m1.3.3.2.2"><csymbol cd="ambiguous" id="S2.F7.4.m1.3.3.2.2.1.cmml" xref="S2.F7.4.m1.3.3.2.2">subscript</csymbol><ci id="S2.F7.4.m1.3.3.2.2.2.cmml" xref="S2.F7.4.m1.3.3.2.2.2">𝐺</ci><ci id="S2.F7.4.m1.3.3.2.2.3.cmml" xref="S2.F7.4.m1.3.3.2.2.3">𝑏</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.F7.4.m1.3d">G_{1},\dots,G_{b}</annotation></semantics></math> are not connected, they have no sequential dependency on each other, therefore can be computed in parallel.
<span id="S2.F7.10.2" class="ltx_text ltx_font_bold">Subsection <a href="#S2.SS5" title="2.5 Dual form ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.5</span></a></span>:
We do not actually materialize the white nodes – the intermediate <math id="S2.F7.5.m2.1" class="ltx_Math" alttext="G" display="inline"><semantics id="S2.F7.5.m2.1b"><mi id="S2.F7.5.m2.1.1" xref="S2.F7.5.m2.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="S2.F7.5.m2.1c"><ci id="S2.F7.5.m2.1.1.cmml" xref="S2.F7.5.m2.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.F7.5.m2.1d">G</annotation></semantics></math>s and <math id="S2.F7.6.m3.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.F7.6.m3.1b"><mi id="S2.F7.6.m3.1.1" xref="S2.F7.6.m3.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.F7.6.m3.1c"><ci id="S2.F7.6.m3.1.1.cmml" xref="S2.F7.6.m3.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.F7.6.m3.1d">W</annotation></semantics></math>s – to compute the output variables in the dual form.
</figcaption>
</figure>
<div id="S2.SS4.p4" class="ltx_para">
<p id="S2.SS4.p4.7" class="ltx_p">Our proposed solution – <em id="S2.SS4.p4.7.2" class="ltx_emph ltx_font_italic">mini-batch gradient descent</em> – is shown in Figure <a href="#S2.F7" title="Figure 7 ‣ 2.4 Parallelization with mini-batch TTT ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>.
Denote the TTT batch size by <math id="S2.SS4.p4.1.m1.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S2.SS4.p4.1.m1.1a"><mi id="S2.SS4.p4.1.m1.1.1" xref="S2.SS4.p4.1.m1.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S2.SS4.p4.1.m1.1b"><ci id="S2.SS4.p4.1.m1.1.1.cmml" xref="S2.SS4.p4.1.m1.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p4.1.m1.1c">b</annotation></semantics></math>. We use <math id="S2.SS4.p4.2.m2.2" class="ltx_Math" alttext="G_{t}=\nabla\ell\left(W_{t^{\prime}};x_{t}\right)" display="inline"><semantics id="S2.SS4.p4.2.m2.2a"><mrow id="S2.SS4.p4.2.m2.2.2" xref="S2.SS4.p4.2.m2.2.2.cmml"><msub id="S2.SS4.p4.2.m2.2.2.4" xref="S2.SS4.p4.2.m2.2.2.4.cmml"><mi id="S2.SS4.p4.2.m2.2.2.4.2" xref="S2.SS4.p4.2.m2.2.2.4.2.cmml">G</mi><mi id="S2.SS4.p4.2.m2.2.2.4.3" xref="S2.SS4.p4.2.m2.2.2.4.3.cmml">t</mi></msub><mo id="S2.SS4.p4.2.m2.2.2.3" xref="S2.SS4.p4.2.m2.2.2.3.cmml">=</mo><mrow id="S2.SS4.p4.2.m2.2.2.2" xref="S2.SS4.p4.2.m2.2.2.2.cmml"><mrow id="S2.SS4.p4.2.m2.2.2.2.4" xref="S2.SS4.p4.2.m2.2.2.2.4.cmml"><mo rspace="0.167em" id="S2.SS4.p4.2.m2.2.2.2.4.1" xref="S2.SS4.p4.2.m2.2.2.2.4.1.cmml">∇</mo><mi mathvariant="normal" id="S2.SS4.p4.2.m2.2.2.2.4.2" xref="S2.SS4.p4.2.m2.2.2.2.4.2.cmml">ℓ</mi></mrow><mo lspace="0em" rspace="0em" id="S2.SS4.p4.2.m2.2.2.2.3" xref="S2.SS4.p4.2.m2.2.2.2.3.cmml">​</mo><mrow id="S2.SS4.p4.2.m2.2.2.2.2.2" xref="S2.SS4.p4.2.m2.2.2.2.2.3.cmml"><mo id="S2.SS4.p4.2.m2.2.2.2.2.2.3" xref="S2.SS4.p4.2.m2.2.2.2.2.3.cmml">(</mo><msub id="S2.SS4.p4.2.m2.1.1.1.1.1.1" xref="S2.SS4.p4.2.m2.1.1.1.1.1.1.cmml"><mi id="S2.SS4.p4.2.m2.1.1.1.1.1.1.2" xref="S2.SS4.p4.2.m2.1.1.1.1.1.1.2.cmml">W</mi><msup id="S2.SS4.p4.2.m2.1.1.1.1.1.1.3" xref="S2.SS4.p4.2.m2.1.1.1.1.1.1.3.cmml"><mi id="S2.SS4.p4.2.m2.1.1.1.1.1.1.3.2" xref="S2.SS4.p4.2.m2.1.1.1.1.1.1.3.2.cmml">t</mi><mo id="S2.SS4.p4.2.m2.1.1.1.1.1.1.3.3" xref="S2.SS4.p4.2.m2.1.1.1.1.1.1.3.3.cmml">′</mo></msup></msub><mo id="S2.SS4.p4.2.m2.2.2.2.2.2.4" xref="S2.SS4.p4.2.m2.2.2.2.2.3.cmml">;</mo><msub id="S2.SS4.p4.2.m2.2.2.2.2.2.2" xref="S2.SS4.p4.2.m2.2.2.2.2.2.2.cmml"><mi id="S2.SS4.p4.2.m2.2.2.2.2.2.2.2" xref="S2.SS4.p4.2.m2.2.2.2.2.2.2.2.cmml">x</mi><mi id="S2.SS4.p4.2.m2.2.2.2.2.2.2.3" xref="S2.SS4.p4.2.m2.2.2.2.2.2.2.3.cmml">t</mi></msub><mo id="S2.SS4.p4.2.m2.2.2.2.2.2.5" xref="S2.SS4.p4.2.m2.2.2.2.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p4.2.m2.2b"><apply id="S2.SS4.p4.2.m2.2.2.cmml" xref="S2.SS4.p4.2.m2.2.2"><eq id="S2.SS4.p4.2.m2.2.2.3.cmml" xref="S2.SS4.p4.2.m2.2.2.3"></eq><apply id="S2.SS4.p4.2.m2.2.2.4.cmml" xref="S2.SS4.p4.2.m2.2.2.4"><csymbol cd="ambiguous" id="S2.SS4.p4.2.m2.2.2.4.1.cmml" xref="S2.SS4.p4.2.m2.2.2.4">subscript</csymbol><ci id="S2.SS4.p4.2.m2.2.2.4.2.cmml" xref="S2.SS4.p4.2.m2.2.2.4.2">𝐺</ci><ci id="S2.SS4.p4.2.m2.2.2.4.3.cmml" xref="S2.SS4.p4.2.m2.2.2.4.3">𝑡</ci></apply><apply id="S2.SS4.p4.2.m2.2.2.2.cmml" xref="S2.SS4.p4.2.m2.2.2.2"><times id="S2.SS4.p4.2.m2.2.2.2.3.cmml" xref="S2.SS4.p4.2.m2.2.2.2.3"></times><apply id="S2.SS4.p4.2.m2.2.2.2.4.cmml" xref="S2.SS4.p4.2.m2.2.2.2.4"><ci id="S2.SS4.p4.2.m2.2.2.2.4.1.cmml" xref="S2.SS4.p4.2.m2.2.2.2.4.1">∇</ci><ci id="S2.SS4.p4.2.m2.2.2.2.4.2.cmml" xref="S2.SS4.p4.2.m2.2.2.2.4.2">ℓ</ci></apply><list id="S2.SS4.p4.2.m2.2.2.2.2.3.cmml" xref="S2.SS4.p4.2.m2.2.2.2.2.2"><apply id="S2.SS4.p4.2.m2.1.1.1.1.1.1.cmml" xref="S2.SS4.p4.2.m2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS4.p4.2.m2.1.1.1.1.1.1.1.cmml" xref="S2.SS4.p4.2.m2.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS4.p4.2.m2.1.1.1.1.1.1.2.cmml" xref="S2.SS4.p4.2.m2.1.1.1.1.1.1.2">𝑊</ci><apply id="S2.SS4.p4.2.m2.1.1.1.1.1.1.3.cmml" xref="S2.SS4.p4.2.m2.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.SS4.p4.2.m2.1.1.1.1.1.1.3.1.cmml" xref="S2.SS4.p4.2.m2.1.1.1.1.1.1.3">superscript</csymbol><ci id="S2.SS4.p4.2.m2.1.1.1.1.1.1.3.2.cmml" xref="S2.SS4.p4.2.m2.1.1.1.1.1.1.3.2">𝑡</ci><ci id="S2.SS4.p4.2.m2.1.1.1.1.1.1.3.3.cmml" xref="S2.SS4.p4.2.m2.1.1.1.1.1.1.3.3">′</ci></apply></apply><apply id="S2.SS4.p4.2.m2.2.2.2.2.2.2.cmml" xref="S2.SS4.p4.2.m2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS4.p4.2.m2.2.2.2.2.2.2.1.cmml" xref="S2.SS4.p4.2.m2.2.2.2.2.2.2">subscript</csymbol><ci id="S2.SS4.p4.2.m2.2.2.2.2.2.2.2.cmml" xref="S2.SS4.p4.2.m2.2.2.2.2.2.2.2">𝑥</ci><ci id="S2.SS4.p4.2.m2.2.2.2.2.2.2.3.cmml" xref="S2.SS4.p4.2.m2.2.2.2.2.2.2.3">𝑡</ci></apply></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p4.2.m2.2c">G_{t}=\nabla\ell\left(W_{t^{\prime}};x_{t}\right)</annotation></semantics></math>, where <math id="S2.SS4.p4.3.m3.1" class="ltx_Math" alttext="t^{\prime}=t\,-\," display="inline"><semantics id="S2.SS4.p4.3.m3.1a"><mrow id="S2.SS4.p4.3.m3.1.1" xref="S2.SS4.p4.3.m3.1.1.cmml"><msup id="S2.SS4.p4.3.m3.1.1.2" xref="S2.SS4.p4.3.m3.1.1.2.cmml"><mi id="S2.SS4.p4.3.m3.1.1.2.2" xref="S2.SS4.p4.3.m3.1.1.2.2.cmml">t</mi><mo id="S2.SS4.p4.3.m3.1.1.2.3" xref="S2.SS4.p4.3.m3.1.1.2.3.cmml">′</mo></msup><mo id="S2.SS4.p4.3.m3.1.1.1" xref="S2.SS4.p4.3.m3.1.1.1.cmml">=</mo><mrow id="S2.SS4.p4.3.m3.1.1.3" xref="S2.SS4.p4.3.m3.1.1.3.cmml"><mi id="S2.SS4.p4.3.m3.1.1.3.2" xref="S2.SS4.p4.3.m3.1.1.3.2.cmml">t</mi><mo lspace="0.392em" id="S2.SS4.p4.3.m3.1.1.3.3" xref="S2.SS4.p4.3.m3.1.1.3.3.cmml">−</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p4.3.m3.1b"><apply id="S2.SS4.p4.3.m3.1.1.cmml" xref="S2.SS4.p4.3.m3.1.1"><eq id="S2.SS4.p4.3.m3.1.1.1.cmml" xref="S2.SS4.p4.3.m3.1.1.1"></eq><apply id="S2.SS4.p4.3.m3.1.1.2.cmml" xref="S2.SS4.p4.3.m3.1.1.2"><csymbol cd="ambiguous" id="S2.SS4.p4.3.m3.1.1.2.1.cmml" xref="S2.SS4.p4.3.m3.1.1.2">superscript</csymbol><ci id="S2.SS4.p4.3.m3.1.1.2.2.cmml" xref="S2.SS4.p4.3.m3.1.1.2.2">𝑡</ci><ci id="S2.SS4.p4.3.m3.1.1.2.3.cmml" xref="S2.SS4.p4.3.m3.1.1.2.3">′</ci></apply><apply id="S2.SS4.p4.3.m3.1.1.3.cmml" xref="S2.SS4.p4.3.m3.1.1.3"><csymbol cd="latexml" id="S2.SS4.p4.3.m3.1.1.3.1.cmml" xref="S2.SS4.p4.3.m3.1.1.3">limit-from</csymbol><ci id="S2.SS4.p4.3.m3.1.1.3.2.cmml" xref="S2.SS4.p4.3.m3.1.1.3.2">𝑡</ci><minus id="S2.SS4.p4.3.m3.1.1.3.3.cmml" xref="S2.SS4.p4.3.m3.1.1.3.3"></minus></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p4.3.m3.1c">t^{\prime}=t\,-\,</annotation></semantics></math><span id="S2.SS4.p4.4.1" class="ltx_text ltx_font_typewriter">mod<math id="S2.SS4.p4.4.1.m1.2" class="ltx_Math" alttext="(t,b)" display="inline"><semantics id="S2.SS4.p4.4.1.m1.2a"><mrow id="S2.SS4.p4.4.1.m1.2.3.2" xref="S2.SS4.p4.4.1.m1.2.3.1.cmml"><mo stretchy="false" id="S2.SS4.p4.4.1.m1.2.3.2.1" xref="S2.SS4.p4.4.1.m1.2.3.1.cmml">(</mo><mi id="S2.SS4.p4.4.1.m1.1.1" xref="S2.SS4.p4.4.1.m1.1.1.cmml">t</mi><mo id="S2.SS4.p4.4.1.m1.2.3.2.2" xref="S2.SS4.p4.4.1.m1.2.3.1.cmml">,</mo><mi id="S2.SS4.p4.4.1.m1.2.2" xref="S2.SS4.p4.4.1.m1.2.2.cmml">b</mi><mo stretchy="false" id="S2.SS4.p4.4.1.m1.2.3.2.3" xref="S2.SS4.p4.4.1.m1.2.3.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p4.4.1.m1.2b"><interval closure="open" id="S2.SS4.p4.4.1.m1.2.3.1.cmml" xref="S2.SS4.p4.4.1.m1.2.3.2"><ci id="S2.SS4.p4.4.1.m1.1.1.cmml" xref="S2.SS4.p4.4.1.m1.1.1">𝑡</ci><ci id="S2.SS4.p4.4.1.m1.2.2.cmml" xref="S2.SS4.p4.4.1.m1.2.2">𝑏</ci></interval></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p4.4.1.m1.2c">(t,b)</annotation></semantics></math></span> is the last timestep of the previous mini-batch (or 0 for the first mini-batch), so we can parallelize <math id="S2.SS4.p4.5.m4.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S2.SS4.p4.5.m4.1a"><mi id="S2.SS4.p4.5.m4.1.1" xref="S2.SS4.p4.5.m4.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S2.SS4.p4.5.m4.1b"><ci id="S2.SS4.p4.5.m4.1.1.cmml" xref="S2.SS4.p4.5.m4.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p4.5.m4.1c">b</annotation></semantics></math> gradient computations at a time.
Empirically, <math id="S2.SS4.p4.6.m5.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S2.SS4.p4.6.m5.1a"><mi id="S2.SS4.p4.6.m5.1.1" xref="S2.SS4.p4.6.m5.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S2.SS4.p4.6.m5.1b"><ci id="S2.SS4.p4.6.m5.1.1.cmml" xref="S2.SS4.p4.6.m5.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p4.6.m5.1c">b</annotation></semantics></math> controls a trade-off between speed and quality, as shown in Figure <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:fig:batch-size</span>.
We chose <math id="S2.SS4.p4.7.m6.1" class="ltx_Math" alttext="b=16" display="inline"><semantics id="S2.SS4.p4.7.m6.1a"><mrow id="S2.SS4.p4.7.m6.1.1" xref="S2.SS4.p4.7.m6.1.1.cmml"><mi id="S2.SS4.p4.7.m6.1.1.2" xref="S2.SS4.p4.7.m6.1.1.2.cmml">b</mi><mo id="S2.SS4.p4.7.m6.1.1.1" xref="S2.SS4.p4.7.m6.1.1.1.cmml">=</mo><mn id="S2.SS4.p4.7.m6.1.1.3" xref="S2.SS4.p4.7.m6.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p4.7.m6.1b"><apply id="S2.SS4.p4.7.m6.1.1.cmml" xref="S2.SS4.p4.7.m6.1.1"><eq id="S2.SS4.p4.7.m6.1.1.1.cmml" xref="S2.SS4.p4.7.m6.1.1.1"></eq><ci id="S2.SS4.p4.7.m6.1.1.2.cmml" xref="S2.SS4.p4.7.m6.1.1.2">𝑏</ci><cn type="integer" id="S2.SS4.p4.7.m6.1.1.3.cmml" xref="S2.SS4.p4.7.m6.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p4.7.m6.1c">b=16</annotation></semantics></math> for all experiments in this paper.</p>
</div>
<div id="S2.SS4.p5" class="ltx_para">
<p id="S2.SS4.p5.9" class="ltx_p">In summary, there are two potential channels to propagate information from <math id="S2.SS4.p5.1.m1.1" class="ltx_Math" alttext="W_{s}" display="inline"><semantics id="S2.SS4.p5.1.m1.1a"><msub id="S2.SS4.p5.1.m1.1.1" xref="S2.SS4.p5.1.m1.1.1.cmml"><mi id="S2.SS4.p5.1.m1.1.1.2" xref="S2.SS4.p5.1.m1.1.1.2.cmml">W</mi><mi id="S2.SS4.p5.1.m1.1.1.3" xref="S2.SS4.p5.1.m1.1.1.3.cmml">s</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p5.1.m1.1b"><apply id="S2.SS4.p5.1.m1.1.1.cmml" xref="S2.SS4.p5.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS4.p5.1.m1.1.1.1.cmml" xref="S2.SS4.p5.1.m1.1.1">subscript</csymbol><ci id="S2.SS4.p5.1.m1.1.1.2.cmml" xref="S2.SS4.p5.1.m1.1.1.2">𝑊</ci><ci id="S2.SS4.p5.1.m1.1.1.3.cmml" xref="S2.SS4.p5.1.m1.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p5.1.m1.1c">W_{s}</annotation></semantics></math> to <math id="S2.SS4.p5.2.m2.1" class="ltx_Math" alttext="W_{t}" display="inline"><semantics id="S2.SS4.p5.2.m2.1a"><msub id="S2.SS4.p5.2.m2.1.1" xref="S2.SS4.p5.2.m2.1.1.cmml"><mi id="S2.SS4.p5.2.m2.1.1.2" xref="S2.SS4.p5.2.m2.1.1.2.cmml">W</mi><mi id="S2.SS4.p5.2.m2.1.1.3" xref="S2.SS4.p5.2.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p5.2.m2.1b"><apply id="S2.SS4.p5.2.m2.1.1.cmml" xref="S2.SS4.p5.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS4.p5.2.m2.1.1.1.cmml" xref="S2.SS4.p5.2.m2.1.1">subscript</csymbol><ci id="S2.SS4.p5.2.m2.1.1.2.cmml" xref="S2.SS4.p5.2.m2.1.1.2">𝑊</ci><ci id="S2.SS4.p5.2.m2.1.1.3.cmml" xref="S2.SS4.p5.2.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p5.2.m2.1c">W_{t}</annotation></semantics></math> where <math id="S2.SS4.p5.3.m3.1" class="ltx_Math" alttext="s&lt;t" display="inline"><semantics id="S2.SS4.p5.3.m3.1a"><mrow id="S2.SS4.p5.3.m3.1.1" xref="S2.SS4.p5.3.m3.1.1.cmml"><mi id="S2.SS4.p5.3.m3.1.1.2" xref="S2.SS4.p5.3.m3.1.1.2.cmml">s</mi><mo id="S2.SS4.p5.3.m3.1.1.1" xref="S2.SS4.p5.3.m3.1.1.1.cmml">&lt;</mo><mi id="S2.SS4.p5.3.m3.1.1.3" xref="S2.SS4.p5.3.m3.1.1.3.cmml">t</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p5.3.m3.1b"><apply id="S2.SS4.p5.3.m3.1.1.cmml" xref="S2.SS4.p5.3.m3.1.1"><lt id="S2.SS4.p5.3.m3.1.1.1.cmml" xref="S2.SS4.p5.3.m3.1.1.1"></lt><ci id="S2.SS4.p5.3.m3.1.1.2.cmml" xref="S2.SS4.p5.3.m3.1.1.2">𝑠</ci><ci id="S2.SS4.p5.3.m3.1.1.3.cmml" xref="S2.SS4.p5.3.m3.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p5.3.m3.1c">s&lt;t</annotation></semantics></math>: <span id="S2.SS4.p5.9.1" class="ltx_text ltx_font_typewriter">cumsum</span> and the gradient operator. The <span id="S2.SS4.p5.9.2" class="ltx_text ltx_font_typewriter">cumsum</span> is always active, but the gradient channel is only active when <math id="S2.SS4.p5.4.m4.1" class="ltx_Math" alttext="W_{s}" display="inline"><semantics id="S2.SS4.p5.4.m4.1a"><msub id="S2.SS4.p5.4.m4.1.1" xref="S2.SS4.p5.4.m4.1.1.cmml"><mi id="S2.SS4.p5.4.m4.1.1.2" xref="S2.SS4.p5.4.m4.1.1.2.cmml">W</mi><mi id="S2.SS4.p5.4.m4.1.1.3" xref="S2.SS4.p5.4.m4.1.1.3.cmml">s</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p5.4.m4.1b"><apply id="S2.SS4.p5.4.m4.1.1.cmml" xref="S2.SS4.p5.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS4.p5.4.m4.1.1.1.cmml" xref="S2.SS4.p5.4.m4.1.1">subscript</csymbol><ci id="S2.SS4.p5.4.m4.1.1.2.cmml" xref="S2.SS4.p5.4.m4.1.1.2">𝑊</ci><ci id="S2.SS4.p5.4.m4.1.1.3.cmml" xref="S2.SS4.p5.4.m4.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p5.4.m4.1c">W_{s}</annotation></semantics></math> is from a previous mini-batch.
Different variants of gradient descent only affect the gradient channel, <em id="S2.SS4.p5.9.3" class="ltx_emph ltx_font_italic">i.e.</em>, the descent direction <math id="S2.SS4.p5.5.m5.1" class="ltx_Math" alttext="G_{t}" display="inline"><semantics id="S2.SS4.p5.5.m5.1a"><msub id="S2.SS4.p5.5.m5.1.1" xref="S2.SS4.p5.5.m5.1.1.cmml"><mi id="S2.SS4.p5.5.m5.1.1.2" xref="S2.SS4.p5.5.m5.1.1.2.cmml">G</mi><mi id="S2.SS4.p5.5.m5.1.1.3" xref="S2.SS4.p5.5.m5.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p5.5.m5.1b"><apply id="S2.SS4.p5.5.m5.1.1.cmml" xref="S2.SS4.p5.5.m5.1.1"><csymbol cd="ambiguous" id="S2.SS4.p5.5.m5.1.1.1.cmml" xref="S2.SS4.p5.5.m5.1.1">subscript</csymbol><ci id="S2.SS4.p5.5.m5.1.1.2.cmml" xref="S2.SS4.p5.5.m5.1.1.2">𝐺</ci><ci id="S2.SS4.p5.5.m5.1.1.3.cmml" xref="S2.SS4.p5.5.m5.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p5.5.m5.1c">G_{t}</annotation></semantics></math>, specifically w.r.t. which <math id="S2.SS4.p5.6.m6.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.SS4.p5.6.m6.1a"><mi id="S2.SS4.p5.6.m6.1.1" xref="S2.SS4.p5.6.m6.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.SS4.p5.6.m6.1b"><ci id="S2.SS4.p5.6.m6.1.1.cmml" xref="S2.SS4.p5.6.m6.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p5.6.m6.1c">W</annotation></semantics></math> the gradient is taken.
However, the descent step <math id="S2.SS4.p5.7.m7.1" class="ltx_Math" alttext="W_{t}=W_{t-1}-\eta\,G_{t}" display="inline"><semantics id="S2.SS4.p5.7.m7.1a"><mrow id="S2.SS4.p5.7.m7.1.1" xref="S2.SS4.p5.7.m7.1.1.cmml"><msub id="S2.SS4.p5.7.m7.1.1.2" xref="S2.SS4.p5.7.m7.1.1.2.cmml"><mi id="S2.SS4.p5.7.m7.1.1.2.2" xref="S2.SS4.p5.7.m7.1.1.2.2.cmml">W</mi><mi id="S2.SS4.p5.7.m7.1.1.2.3" xref="S2.SS4.p5.7.m7.1.1.2.3.cmml">t</mi></msub><mo id="S2.SS4.p5.7.m7.1.1.1" xref="S2.SS4.p5.7.m7.1.1.1.cmml">=</mo><mrow id="S2.SS4.p5.7.m7.1.1.3" xref="S2.SS4.p5.7.m7.1.1.3.cmml"><msub id="S2.SS4.p5.7.m7.1.1.3.2" xref="S2.SS4.p5.7.m7.1.1.3.2.cmml"><mi id="S2.SS4.p5.7.m7.1.1.3.2.2" xref="S2.SS4.p5.7.m7.1.1.3.2.2.cmml">W</mi><mrow id="S2.SS4.p5.7.m7.1.1.3.2.3" xref="S2.SS4.p5.7.m7.1.1.3.2.3.cmml"><mi id="S2.SS4.p5.7.m7.1.1.3.2.3.2" xref="S2.SS4.p5.7.m7.1.1.3.2.3.2.cmml">t</mi><mo id="S2.SS4.p5.7.m7.1.1.3.2.3.1" xref="S2.SS4.p5.7.m7.1.1.3.2.3.1.cmml">−</mo><mn id="S2.SS4.p5.7.m7.1.1.3.2.3.3" xref="S2.SS4.p5.7.m7.1.1.3.2.3.3.cmml">1</mn></mrow></msub><mo id="S2.SS4.p5.7.m7.1.1.3.1" xref="S2.SS4.p5.7.m7.1.1.3.1.cmml">−</mo><mrow id="S2.SS4.p5.7.m7.1.1.3.3" xref="S2.SS4.p5.7.m7.1.1.3.3.cmml"><mi id="S2.SS4.p5.7.m7.1.1.3.3.2" xref="S2.SS4.p5.7.m7.1.1.3.3.2.cmml">η</mi><mo lspace="0.170em" rspace="0em" id="S2.SS4.p5.7.m7.1.1.3.3.1" xref="S2.SS4.p5.7.m7.1.1.3.3.1.cmml">​</mo><msub id="S2.SS4.p5.7.m7.1.1.3.3.3" xref="S2.SS4.p5.7.m7.1.1.3.3.3.cmml"><mi id="S2.SS4.p5.7.m7.1.1.3.3.3.2" xref="S2.SS4.p5.7.m7.1.1.3.3.3.2.cmml">G</mi><mi id="S2.SS4.p5.7.m7.1.1.3.3.3.3" xref="S2.SS4.p5.7.m7.1.1.3.3.3.3.cmml">t</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS4.p5.7.m7.1b"><apply id="S2.SS4.p5.7.m7.1.1.cmml" xref="S2.SS4.p5.7.m7.1.1"><eq id="S2.SS4.p5.7.m7.1.1.1.cmml" xref="S2.SS4.p5.7.m7.1.1.1"></eq><apply id="S2.SS4.p5.7.m7.1.1.2.cmml" xref="S2.SS4.p5.7.m7.1.1.2"><csymbol cd="ambiguous" id="S2.SS4.p5.7.m7.1.1.2.1.cmml" xref="S2.SS4.p5.7.m7.1.1.2">subscript</csymbol><ci id="S2.SS4.p5.7.m7.1.1.2.2.cmml" xref="S2.SS4.p5.7.m7.1.1.2.2">𝑊</ci><ci id="S2.SS4.p5.7.m7.1.1.2.3.cmml" xref="S2.SS4.p5.7.m7.1.1.2.3">𝑡</ci></apply><apply id="S2.SS4.p5.7.m7.1.1.3.cmml" xref="S2.SS4.p5.7.m7.1.1.3"><minus id="S2.SS4.p5.7.m7.1.1.3.1.cmml" xref="S2.SS4.p5.7.m7.1.1.3.1"></minus><apply id="S2.SS4.p5.7.m7.1.1.3.2.cmml" xref="S2.SS4.p5.7.m7.1.1.3.2"><csymbol cd="ambiguous" id="S2.SS4.p5.7.m7.1.1.3.2.1.cmml" xref="S2.SS4.p5.7.m7.1.1.3.2">subscript</csymbol><ci id="S2.SS4.p5.7.m7.1.1.3.2.2.cmml" xref="S2.SS4.p5.7.m7.1.1.3.2.2">𝑊</ci><apply id="S2.SS4.p5.7.m7.1.1.3.2.3.cmml" xref="S2.SS4.p5.7.m7.1.1.3.2.3"><minus id="S2.SS4.p5.7.m7.1.1.3.2.3.1.cmml" xref="S2.SS4.p5.7.m7.1.1.3.2.3.1"></minus><ci id="S2.SS4.p5.7.m7.1.1.3.2.3.2.cmml" xref="S2.SS4.p5.7.m7.1.1.3.2.3.2">𝑡</ci><cn type="integer" id="S2.SS4.p5.7.m7.1.1.3.2.3.3.cmml" xref="S2.SS4.p5.7.m7.1.1.3.2.3.3">1</cn></apply></apply><apply id="S2.SS4.p5.7.m7.1.1.3.3.cmml" xref="S2.SS4.p5.7.m7.1.1.3.3"><times id="S2.SS4.p5.7.m7.1.1.3.3.1.cmml" xref="S2.SS4.p5.7.m7.1.1.3.3.1"></times><ci id="S2.SS4.p5.7.m7.1.1.3.3.2.cmml" xref="S2.SS4.p5.7.m7.1.1.3.3.2">𝜂</ci><apply id="S2.SS4.p5.7.m7.1.1.3.3.3.cmml" xref="S2.SS4.p5.7.m7.1.1.3.3.3"><csymbol cd="ambiguous" id="S2.SS4.p5.7.m7.1.1.3.3.3.1.cmml" xref="S2.SS4.p5.7.m7.1.1.3.3.3">subscript</csymbol><ci id="S2.SS4.p5.7.m7.1.1.3.3.3.2.cmml" xref="S2.SS4.p5.7.m7.1.1.3.3.3.2">𝐺</ci><ci id="S2.SS4.p5.7.m7.1.1.3.3.3.3.cmml" xref="S2.SS4.p5.7.m7.1.1.3.3.3.3">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p5.7.m7.1c">W_{t}=W_{t-1}-\eta\,G_{t}</annotation></semantics></math> always starts from <math id="S2.SS4.p5.8.m8.1" class="ltx_Math" alttext="W_{t-1}" display="inline"><semantics id="S2.SS4.p5.8.m8.1a"><msub id="S2.SS4.p5.8.m8.1.1" xref="S2.SS4.p5.8.m8.1.1.cmml"><mi id="S2.SS4.p5.8.m8.1.1.2" xref="S2.SS4.p5.8.m8.1.1.2.cmml">W</mi><mrow id="S2.SS4.p5.8.m8.1.1.3" xref="S2.SS4.p5.8.m8.1.1.3.cmml"><mi id="S2.SS4.p5.8.m8.1.1.3.2" xref="S2.SS4.p5.8.m8.1.1.3.2.cmml">t</mi><mo id="S2.SS4.p5.8.m8.1.1.3.1" xref="S2.SS4.p5.8.m8.1.1.3.1.cmml">−</mo><mn id="S2.SS4.p5.8.m8.1.1.3.3" xref="S2.SS4.p5.8.m8.1.1.3.3.cmml">1</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p5.8.m8.1b"><apply id="S2.SS4.p5.8.m8.1.1.cmml" xref="S2.SS4.p5.8.m8.1.1"><csymbol cd="ambiguous" id="S2.SS4.p5.8.m8.1.1.1.cmml" xref="S2.SS4.p5.8.m8.1.1">subscript</csymbol><ci id="S2.SS4.p5.8.m8.1.1.2.cmml" xref="S2.SS4.p5.8.m8.1.1.2">𝑊</ci><apply id="S2.SS4.p5.8.m8.1.1.3.cmml" xref="S2.SS4.p5.8.m8.1.1.3"><minus id="S2.SS4.p5.8.m8.1.1.3.1.cmml" xref="S2.SS4.p5.8.m8.1.1.3.1"></minus><ci id="S2.SS4.p5.8.m8.1.1.3.2.cmml" xref="S2.SS4.p5.8.m8.1.1.3.2">𝑡</ci><cn type="integer" id="S2.SS4.p5.8.m8.1.1.3.3.cmml" xref="S2.SS4.p5.8.m8.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p5.8.m8.1c">W_{t-1}</annotation></semantics></math>, due to the autoregressive nature of the update rule, which is orthogonal to the choice of <math id="S2.SS4.p5.9.m9.1" class="ltx_Math" alttext="G_{t}" display="inline"><semantics id="S2.SS4.p5.9.m9.1a"><msub id="S2.SS4.p5.9.m9.1.1" xref="S2.SS4.p5.9.m9.1.1.cmml"><mi id="S2.SS4.p5.9.m9.1.1.2" xref="S2.SS4.p5.9.m9.1.1.2.cmml">G</mi><mi id="S2.SS4.p5.9.m9.1.1.3" xref="S2.SS4.p5.9.m9.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS4.p5.9.m9.1b"><apply id="S2.SS4.p5.9.m9.1.1.cmml" xref="S2.SS4.p5.9.m9.1.1"><csymbol cd="ambiguous" id="S2.SS4.p5.9.m9.1.1.1.cmml" xref="S2.SS4.p5.9.m9.1.1">subscript</csymbol><ci id="S2.SS4.p5.9.m9.1.1.2.cmml" xref="S2.SS4.p5.9.m9.1.1.2">𝐺</ci><ci id="S2.SS4.p5.9.m9.1.1.3.cmml" xref="S2.SS4.p5.9.m9.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS4.p5.9.m9.1c">G_{t}</annotation></semantics></math>.</p>
</div>
<figure id="S2.F8" class="ltx_figure"><img src="/html/2407.04620/assets/x8.png" id="S2.F8.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="115" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 8: </span>
Ablations on TTT mini-batch size <math id="S2.F8.1.m1.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S2.F8.1.m1.1b"><mi id="S2.F8.1.m1.1.1" xref="S2.F8.1.m1.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S2.F8.1.m1.1c"><ci id="S2.F8.1.m1.1.1.cmml" xref="S2.F8.1.m1.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.F8.1.m1.1d">b</annotation></semantics></math>, where <math id="S2.F8.2.m2.1" class="ltx_Math" alttext="b=1" display="inline"><semantics id="S2.F8.2.m2.1b"><mrow id="S2.F8.2.m2.1.1" xref="S2.F8.2.m2.1.1.cmml"><mi id="S2.F8.2.m2.1.1.2" xref="S2.F8.2.m2.1.1.2.cmml">b</mi><mo id="S2.F8.2.m2.1.1.1" xref="S2.F8.2.m2.1.1.1.cmml">=</mo><mn id="S2.F8.2.m2.1.1.3" xref="S2.F8.2.m2.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.F8.2.m2.1c"><apply id="S2.F8.2.m2.1.1.cmml" xref="S2.F8.2.m2.1.1"><eq id="S2.F8.2.m2.1.1.1.cmml" xref="S2.F8.2.m2.1.1.1"></eq><ci id="S2.F8.2.m2.1.1.2.cmml" xref="S2.F8.2.m2.1.1.2">𝑏</ci><cn type="integer" id="S2.F8.2.m2.1.1.3.cmml" xref="S2.F8.2.m2.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F8.2.m2.1d">b=1</annotation></semantics></math> is online GD and <math id="S2.F8.3.m3.1" class="ltx_Math" alttext="b=T" display="inline"><semantics id="S2.F8.3.m3.1b"><mrow id="S2.F8.3.m3.1.1" xref="S2.F8.3.m3.1.1.cmml"><mi id="S2.F8.3.m3.1.1.2" xref="S2.F8.3.m3.1.1.2.cmml">b</mi><mo id="S2.F8.3.m3.1.1.1" xref="S2.F8.3.m3.1.1.1.cmml">=</mo><mi id="S2.F8.3.m3.1.1.3" xref="S2.F8.3.m3.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.F8.3.m3.1c"><apply id="S2.F8.3.m3.1.1.cmml" xref="S2.F8.3.m3.1.1"><eq id="S2.F8.3.m3.1.1.1.cmml" xref="S2.F8.3.m3.1.1.1"></eq><ci id="S2.F8.3.m3.1.1.2.cmml" xref="S2.F8.3.m3.1.1.2">𝑏</ci><ci id="S2.F8.3.m3.1.1.3.cmml" xref="S2.F8.3.m3.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F8.3.m3.1d">b=T</annotation></semantics></math> is batch GD.
We choose <math id="S2.F8.4.m4.1" class="ltx_Math" alttext="b=16" display="inline"><semantics id="S2.F8.4.m4.1b"><mrow id="S2.F8.4.m4.1.1" xref="S2.F8.4.m4.1.1.cmml"><mi id="S2.F8.4.m4.1.1.2" xref="S2.F8.4.m4.1.1.2.cmml">b</mi><mo id="S2.F8.4.m4.1.1.1" xref="S2.F8.4.m4.1.1.1.cmml">=</mo><mn id="S2.F8.4.m4.1.1.3" xref="S2.F8.4.m4.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.F8.4.m4.1c"><apply id="S2.F8.4.m4.1.1.cmml" xref="S2.F8.4.m4.1.1"><eq id="S2.F8.4.m4.1.1.1.cmml" xref="S2.F8.4.m4.1.1.1"></eq><ci id="S2.F8.4.m4.1.1.2.cmml" xref="S2.F8.4.m4.1.1.2">𝑏</ci><cn type="integer" id="S2.F8.4.m4.1.1.3.cmml" xref="S2.F8.4.m4.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F8.4.m4.1d">b=16</annotation></semantics></math> for all experiments in this paper.
<span id="S2.F8.18.1" class="ltx_text ltx_font_bold">Left</span>: Smaller <math id="S2.F8.5.m5.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S2.F8.5.m5.1b"><mi id="S2.F8.5.m5.1.1" xref="S2.F8.5.m5.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S2.F8.5.m5.1c"><ci id="S2.F8.5.m5.1.1.cmml" xref="S2.F8.5.m5.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.F8.5.m5.1d">b</annotation></semantics></math> improves perplexity since more GD steps are taken.<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span>
In theory, <math id="footnotex1.m1a.1" class="ltx_Math" alttext="b" display="inline"><semantics id="footnotex1.m1a.1c"><mi id="footnotex1.m1a.1.1" xref="footnotex1.m1a.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="footnotex1.m1a.1d"><ci id="footnotex1.m1a.1.1.cmml" xref="footnotex1.m1a.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="footnotex1.m1a.1e">b</annotation></semantics></math> can potentially be too small such that the variance between mini-batches is too high, hurting optimization.
However, we have not observed such an effect in practice.
</span></span></span> 
The perplexity of 11.09 at <math id="S2.F8.6.m6.1" class="ltx_Math" alttext="b=16" display="inline"><semantics id="S2.F8.6.m6.1b"><mrow id="S2.F8.6.m6.1.1" xref="S2.F8.6.m6.1.1.cmml"><mi id="S2.F8.6.m6.1.1.2" xref="S2.F8.6.m6.1.1.2.cmml">b</mi><mo id="S2.F8.6.m6.1.1.1" xref="S2.F8.6.m6.1.1.1.cmml">=</mo><mn id="S2.F8.6.m6.1.1.3" xref="S2.F8.6.m6.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.F8.6.m6.1c"><apply id="S2.F8.6.m6.1.1.cmml" xref="S2.F8.6.m6.1.1"><eq id="S2.F8.6.m6.1.1.1.cmml" xref="S2.F8.6.m6.1.1.1"></eq><ci id="S2.F8.6.m6.1.1.2.cmml" xref="S2.F8.6.m6.1.1.2">𝑏</ci><cn type="integer" id="S2.F8.6.m6.1.1.3.cmml" xref="S2.F8.6.m6.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F8.6.m6.1d">b=16</annotation></semantics></math> corresponds to the final result of TTT-Linear in Figure <a href="#S3.F11" title="Figure 11 ‣ 3.1 Short context: the Pile ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a>.
<span id="S2.F8.19.2" class="ltx_text ltx_font_bold">Right</span>: Forward time in dual form, with context length <math id="S2.F8.7.m7.1" class="ltx_Math" alttext="T=2048" display="inline"><semantics id="S2.F8.7.m7.1b"><mrow id="S2.F8.7.m7.1.1" xref="S2.F8.7.m7.1.1.cmml"><mi id="S2.F8.7.m7.1.1.2" xref="S2.F8.7.m7.1.1.2.cmml">T</mi><mo id="S2.F8.7.m7.1.1.1" xref="S2.F8.7.m7.1.1.1.cmml">=</mo><mn id="S2.F8.7.m7.1.1.3" xref="S2.F8.7.m7.1.1.3.cmml">2048</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.F8.7.m7.1c"><apply id="S2.F8.7.m7.1.1.cmml" xref="S2.F8.7.m7.1.1"><eq id="S2.F8.7.m7.1.1.1.cmml" xref="S2.F8.7.m7.1.1.1"></eq><ci id="S2.F8.7.m7.1.1.2.cmml" xref="S2.F8.7.m7.1.1.2">𝑇</ci><cn type="integer" id="S2.F8.7.m7.1.1.3.cmml" xref="S2.F8.7.m7.1.1.3">2048</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F8.7.m7.1d">T=2048</annotation></semantics></math>.
Total time (orange) can be decomposed into time for computing the <math id="S2.F8.8.m8.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.F8.8.m8.1b"><mi id="S2.F8.8.m8.1.1" xref="S2.F8.8.m8.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.F8.8.m8.1c"><ci id="S2.F8.8.m8.1.1.cmml" xref="S2.F8.8.m8.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.F8.8.m8.1d">W</annotation></semantics></math>s at the end of every mini-batch (blue) and time for <math id="S2.F8.9.m9.3" class="ltx_Math" alttext="z_{1},\dots,z_{T}" display="inline"><semantics id="S2.F8.9.m9.3b"><mrow id="S2.F8.9.m9.3.3.2" xref="S2.F8.9.m9.3.3.3.cmml"><msub id="S2.F8.9.m9.2.2.1.1" xref="S2.F8.9.m9.2.2.1.1.cmml"><mi id="S2.F8.9.m9.2.2.1.1.2" xref="S2.F8.9.m9.2.2.1.1.2.cmml">z</mi><mn id="S2.F8.9.m9.2.2.1.1.3" xref="S2.F8.9.m9.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.F8.9.m9.3.3.2.3" xref="S2.F8.9.m9.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.F8.9.m9.1.1" xref="S2.F8.9.m9.1.1.cmml">…</mi><mo id="S2.F8.9.m9.3.3.2.4" xref="S2.F8.9.m9.3.3.3.cmml">,</mo><msub id="S2.F8.9.m9.3.3.2.2" xref="S2.F8.9.m9.3.3.2.2.cmml"><mi id="S2.F8.9.m9.3.3.2.2.2" xref="S2.F8.9.m9.3.3.2.2.2.cmml">z</mi><mi id="S2.F8.9.m9.3.3.2.2.3" xref="S2.F8.9.m9.3.3.2.2.3.cmml">T</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.F8.9.m9.3c"><list id="S2.F8.9.m9.3.3.3.cmml" xref="S2.F8.9.m9.3.3.2"><apply id="S2.F8.9.m9.2.2.1.1.cmml" xref="S2.F8.9.m9.2.2.1.1"><csymbol cd="ambiguous" id="S2.F8.9.m9.2.2.1.1.1.cmml" xref="S2.F8.9.m9.2.2.1.1">subscript</csymbol><ci id="S2.F8.9.m9.2.2.1.1.2.cmml" xref="S2.F8.9.m9.2.2.1.1.2">𝑧</ci><cn type="integer" id="S2.F8.9.m9.2.2.1.1.3.cmml" xref="S2.F8.9.m9.2.2.1.1.3">1</cn></apply><ci id="S2.F8.9.m9.1.1.cmml" xref="S2.F8.9.m9.1.1">…</ci><apply id="S2.F8.9.m9.3.3.2.2.cmml" xref="S2.F8.9.m9.3.3.2.2"><csymbol cd="ambiguous" id="S2.F8.9.m9.3.3.2.2.1.cmml" xref="S2.F8.9.m9.3.3.2.2">subscript</csymbol><ci id="S2.F8.9.m9.3.3.2.2.2.cmml" xref="S2.F8.9.m9.3.3.2.2.2">𝑧</ci><ci id="S2.F8.9.m9.3.3.2.2.3.cmml" xref="S2.F8.9.m9.3.3.2.2.3">𝑇</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.F8.9.m9.3d">z_{1},\dots,z_{T}</annotation></semantics></math> (orange <math id="S2.F8.10.m10.1" class="ltx_Math" alttext="-" display="inline"><semantics id="S2.F8.10.m10.1b"><mo id="S2.F8.10.m10.1.1" xref="S2.F8.10.m10.1.1.cmml">−</mo><annotation-xml encoding="MathML-Content" id="S2.F8.10.m10.1c"><minus id="S2.F8.10.m10.1.1.cmml" xref="S2.F8.10.m10.1.1"></minus></annotation-xml><annotation encoding="application/x-tex" id="S2.F8.10.m10.1d">-</annotation></semantics></math> blue).<span id="footnote6" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">6</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">6</sup><span class="ltx_tag ltx_tag_note">6</span>
For Figure <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:fig:batch-size</span>, we use a single TTT layer in TTT-Linear 1.3B, implemented in pure PyTorch.
Our fused kernel significantly improves time efficiency, but makes it difficult to cleanly decompose the time for computing <math id="footnotex2.m1a.1" class="ltx_Math" alttext="W_{b}" display="inline"><semantics id="footnotex2.m1a.1c"><msub id="footnotex2.m1a.1.1" xref="footnotex2.m1a.1.1.cmml"><mi id="footnotex2.m1a.1.1.2" xref="footnotex2.m1a.1.1.2.cmml">W</mi><mi id="footnotex2.m1a.1.1.3" xref="footnotex2.m1a.1.1.3.cmml">b</mi></msub><annotation-xml encoding="MathML-Content" id="footnotex2.m1a.1d"><apply id="footnotex2.m1a.1.1.cmml" xref="footnotex2.m1a.1.1"><csymbol cd="ambiguous" id="footnotex2.m1a.1.1.1.cmml" xref="footnotex2.m1a.1.1">subscript</csymbol><ci id="footnotex2.m1a.1.1.2.cmml" xref="footnotex2.m1a.1.1.2">𝑊</ci><ci id="footnotex2.m1a.1.1.3.cmml" xref="footnotex2.m1a.1.1.3">𝑏</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="footnotex2.m1a.1e">W_{b}</annotation></semantics></math> vs. <math id="footnotex2.m2a.3" class="ltx_Math" alttext="z_{1},\dots,z_{b}" display="inline"><semantics id="footnotex2.m2a.3c"><mrow id="footnotex2.m2a.3.3.2" xref="footnotex2.m2a.3.3.3.cmml"><msub id="footnotex2.m2a.2.2.1.1" xref="footnotex2.m2a.2.2.1.1.cmml"><mi id="footnotex2.m2a.2.2.1.1.2" xref="footnotex2.m2a.2.2.1.1.2.cmml">z</mi><mn id="footnotex2.m2a.2.2.1.1.3" xref="footnotex2.m2a.2.2.1.1.3.cmml">1</mn></msub><mo id="footnotex2.m2a.3.3.2.3" xref="footnotex2.m2a.3.3.3.cmml">,</mo><mi mathvariant="normal" id="footnotex2.m2a.1.1" xref="footnotex2.m2a.1.1.cmml">…</mi><mo id="footnotex2.m2a.3.3.2.4" xref="footnotex2.m2a.3.3.3.cmml">,</mo><msub id="footnotex2.m2a.3.3.2.2" xref="footnotex2.m2a.3.3.2.2.cmml"><mi id="footnotex2.m2a.3.3.2.2.2" xref="footnotex2.m2a.3.3.2.2.2.cmml">z</mi><mi id="footnotex2.m2a.3.3.2.2.3" xref="footnotex2.m2a.3.3.2.2.3.cmml">b</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="footnotex2.m2a.3d"><list id="footnotex2.m2a.3.3.3.cmml" xref="footnotex2.m2a.3.3.2"><apply id="footnotex2.m2a.2.2.1.1.cmml" xref="footnotex2.m2a.2.2.1.1"><csymbol cd="ambiguous" id="footnotex2.m2a.2.2.1.1.1.cmml" xref="footnotex2.m2a.2.2.1.1">subscript</csymbol><ci id="footnotex2.m2a.2.2.1.1.2.cmml" xref="footnotex2.m2a.2.2.1.1.2">𝑧</ci><cn type="integer" id="footnotex2.m2a.2.2.1.1.3.cmml" xref="footnotex2.m2a.2.2.1.1.3">1</cn></apply><ci id="footnotex2.m2a.1.1.cmml" xref="footnotex2.m2a.1.1">…</ci><apply id="footnotex2.m2a.3.3.2.2.cmml" xref="footnotex2.m2a.3.3.2.2"><csymbol cd="ambiguous" id="footnotex2.m2a.3.3.2.2.1.cmml" xref="footnotex2.m2a.3.3.2.2">subscript</csymbol><ci id="footnotex2.m2a.3.3.2.2.2.cmml" xref="footnotex2.m2a.3.3.2.2.2">𝑧</ci><ci id="footnotex2.m2a.3.3.2.2.3.cmml" xref="footnotex2.m2a.3.3.2.2.3">𝑏</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="footnotex2.m2a.3e">z_{1},\dots,z_{b}</annotation></semantics></math>.
</span></span></span> 
Time complexity for the <math id="S2.F8.11.m11.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.F8.11.m11.1b"><mi id="S2.F8.11.m11.1.1" xref="S2.F8.11.m11.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.F8.11.m11.1c"><ci id="S2.F8.11.m11.1.1.cmml" xref="S2.F8.11.m11.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.F8.11.m11.1d">W</annotation></semantics></math>s is <math id="S2.F8.12.m12.1" class="ltx_Math" alttext="O(T\times d^{2})" display="inline"><semantics id="S2.F8.12.m12.1b"><mrow id="S2.F8.12.m12.1.1" xref="S2.F8.12.m12.1.1.cmml"><mi id="S2.F8.12.m12.1.1.3" xref="S2.F8.12.m12.1.1.3.cmml">O</mi><mo lspace="0em" rspace="0em" id="S2.F8.12.m12.1.1.2" xref="S2.F8.12.m12.1.1.2.cmml">​</mo><mrow id="S2.F8.12.m12.1.1.1.1" xref="S2.F8.12.m12.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.F8.12.m12.1.1.1.1.2" xref="S2.F8.12.m12.1.1.1.1.1.cmml">(</mo><mrow id="S2.F8.12.m12.1.1.1.1.1" xref="S2.F8.12.m12.1.1.1.1.1.cmml"><mi id="S2.F8.12.m12.1.1.1.1.1.2" xref="S2.F8.12.m12.1.1.1.1.1.2.cmml">T</mi><mo lspace="0.222em" rspace="0.222em" id="S2.F8.12.m12.1.1.1.1.1.1" xref="S2.F8.12.m12.1.1.1.1.1.1.cmml">×</mo><msup id="S2.F8.12.m12.1.1.1.1.1.3" xref="S2.F8.12.m12.1.1.1.1.1.3.cmml"><mi id="S2.F8.12.m12.1.1.1.1.1.3.2" xref="S2.F8.12.m12.1.1.1.1.1.3.2.cmml">d</mi><mn id="S2.F8.12.m12.1.1.1.1.1.3.3" xref="S2.F8.12.m12.1.1.1.1.1.3.3.cmml">2</mn></msup></mrow><mo stretchy="false" id="S2.F8.12.m12.1.1.1.1.3" xref="S2.F8.12.m12.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.F8.12.m12.1c"><apply id="S2.F8.12.m12.1.1.cmml" xref="S2.F8.12.m12.1.1"><times id="S2.F8.12.m12.1.1.2.cmml" xref="S2.F8.12.m12.1.1.2"></times><ci id="S2.F8.12.m12.1.1.3.cmml" xref="S2.F8.12.m12.1.1.3">𝑂</ci><apply id="S2.F8.12.m12.1.1.1.1.1.cmml" xref="S2.F8.12.m12.1.1.1.1"><times id="S2.F8.12.m12.1.1.1.1.1.1.cmml" xref="S2.F8.12.m12.1.1.1.1.1.1"></times><ci id="S2.F8.12.m12.1.1.1.1.1.2.cmml" xref="S2.F8.12.m12.1.1.1.1.1.2">𝑇</ci><apply id="S2.F8.12.m12.1.1.1.1.1.3.cmml" xref="S2.F8.12.m12.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.F8.12.m12.1.1.1.1.1.3.1.cmml" xref="S2.F8.12.m12.1.1.1.1.1.3">superscript</csymbol><ci id="S2.F8.12.m12.1.1.1.1.1.3.2.cmml" xref="S2.F8.12.m12.1.1.1.1.1.3.2">𝑑</ci><cn type="integer" id="S2.F8.12.m12.1.1.1.1.1.3.3.cmml" xref="S2.F8.12.m12.1.1.1.1.1.3.3">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F8.12.m12.1d">O(T\times d^{2})</annotation></semantics></math>, constant in <math id="S2.F8.13.m13.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S2.F8.13.m13.1b"><mi id="S2.F8.13.m13.1.1" xref="S2.F8.13.m13.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S2.F8.13.m13.1c"><ci id="S2.F8.13.m13.1.1.cmml" xref="S2.F8.13.m13.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.F8.13.m13.1d">b</annotation></semantics></math>, but the blue line decreases as larger <math id="S2.F8.14.m14.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S2.F8.14.m14.1b"><mi id="S2.F8.14.m14.1.1" xref="S2.F8.14.m14.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S2.F8.14.m14.1c"><ci id="S2.F8.14.m14.1.1.cmml" xref="S2.F8.14.m14.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.F8.14.m14.1d">b</annotation></semantics></math> allows more parallelization until hardware utilization saturates.
Time complexity for <math id="S2.F8.15.m15.3" class="ltx_Math" alttext="z_{1},\dots,z_{T}" display="inline"><semantics id="S2.F8.15.m15.3b"><mrow id="S2.F8.15.m15.3.3.2" xref="S2.F8.15.m15.3.3.3.cmml"><msub id="S2.F8.15.m15.2.2.1.1" xref="S2.F8.15.m15.2.2.1.1.cmml"><mi id="S2.F8.15.m15.2.2.1.1.2" xref="S2.F8.15.m15.2.2.1.1.2.cmml">z</mi><mn id="S2.F8.15.m15.2.2.1.1.3" xref="S2.F8.15.m15.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.F8.15.m15.3.3.2.3" xref="S2.F8.15.m15.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.F8.15.m15.1.1" xref="S2.F8.15.m15.1.1.cmml">…</mi><mo id="S2.F8.15.m15.3.3.2.4" xref="S2.F8.15.m15.3.3.3.cmml">,</mo><msub id="S2.F8.15.m15.3.3.2.2" xref="S2.F8.15.m15.3.3.2.2.cmml"><mi id="S2.F8.15.m15.3.3.2.2.2" xref="S2.F8.15.m15.3.3.2.2.2.cmml">z</mi><mi id="S2.F8.15.m15.3.3.2.2.3" xref="S2.F8.15.m15.3.3.2.2.3.cmml">T</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.F8.15.m15.3c"><list id="S2.F8.15.m15.3.3.3.cmml" xref="S2.F8.15.m15.3.3.2"><apply id="S2.F8.15.m15.2.2.1.1.cmml" xref="S2.F8.15.m15.2.2.1.1"><csymbol cd="ambiguous" id="S2.F8.15.m15.2.2.1.1.1.cmml" xref="S2.F8.15.m15.2.2.1.1">subscript</csymbol><ci id="S2.F8.15.m15.2.2.1.1.2.cmml" xref="S2.F8.15.m15.2.2.1.1.2">𝑧</ci><cn type="integer" id="S2.F8.15.m15.2.2.1.1.3.cmml" xref="S2.F8.15.m15.2.2.1.1.3">1</cn></apply><ci id="S2.F8.15.m15.1.1.cmml" xref="S2.F8.15.m15.1.1">…</ci><apply id="S2.F8.15.m15.3.3.2.2.cmml" xref="S2.F8.15.m15.3.3.2.2"><csymbol cd="ambiguous" id="S2.F8.15.m15.3.3.2.2.1.cmml" xref="S2.F8.15.m15.3.3.2.2">subscript</csymbol><ci id="S2.F8.15.m15.3.3.2.2.2.cmml" xref="S2.F8.15.m15.3.3.2.2.2">𝑧</ci><ci id="S2.F8.15.m15.3.3.2.2.3.cmml" xref="S2.F8.15.m15.3.3.2.2.3">𝑇</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.F8.15.m15.3d">z_{1},\dots,z_{T}</annotation></semantics></math> is <math id="S2.F8.16.m16.1" class="ltx_Math" alttext="O(T\times b\times d)" display="inline"><semantics id="S2.F8.16.m16.1b"><mrow id="S2.F8.16.m16.1.1" xref="S2.F8.16.m16.1.1.cmml"><mi id="S2.F8.16.m16.1.1.3" xref="S2.F8.16.m16.1.1.3.cmml">O</mi><mo lspace="0em" rspace="0em" id="S2.F8.16.m16.1.1.2" xref="S2.F8.16.m16.1.1.2.cmml">​</mo><mrow id="S2.F8.16.m16.1.1.1.1" xref="S2.F8.16.m16.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.F8.16.m16.1.1.1.1.2" xref="S2.F8.16.m16.1.1.1.1.1.cmml">(</mo><mrow id="S2.F8.16.m16.1.1.1.1.1" xref="S2.F8.16.m16.1.1.1.1.1.cmml"><mi id="S2.F8.16.m16.1.1.1.1.1.2" xref="S2.F8.16.m16.1.1.1.1.1.2.cmml">T</mi><mo lspace="0.222em" rspace="0.222em" id="S2.F8.16.m16.1.1.1.1.1.1" xref="S2.F8.16.m16.1.1.1.1.1.1.cmml">×</mo><mi id="S2.F8.16.m16.1.1.1.1.1.3" xref="S2.F8.16.m16.1.1.1.1.1.3.cmml">b</mi><mo lspace="0.222em" rspace="0.222em" id="S2.F8.16.m16.1.1.1.1.1.1b" xref="S2.F8.16.m16.1.1.1.1.1.1.cmml">×</mo><mi id="S2.F8.16.m16.1.1.1.1.1.4" xref="S2.F8.16.m16.1.1.1.1.1.4.cmml">d</mi></mrow><mo stretchy="false" id="S2.F8.16.m16.1.1.1.1.3" xref="S2.F8.16.m16.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.F8.16.m16.1c"><apply id="S2.F8.16.m16.1.1.cmml" xref="S2.F8.16.m16.1.1"><times id="S2.F8.16.m16.1.1.2.cmml" xref="S2.F8.16.m16.1.1.2"></times><ci id="S2.F8.16.m16.1.1.3.cmml" xref="S2.F8.16.m16.1.1.3">𝑂</ci><apply id="S2.F8.16.m16.1.1.1.1.1.cmml" xref="S2.F8.16.m16.1.1.1.1"><times id="S2.F8.16.m16.1.1.1.1.1.1.cmml" xref="S2.F8.16.m16.1.1.1.1.1.1"></times><ci id="S2.F8.16.m16.1.1.1.1.1.2.cmml" xref="S2.F8.16.m16.1.1.1.1.1.2">𝑇</ci><ci id="S2.F8.16.m16.1.1.1.1.1.3.cmml" xref="S2.F8.16.m16.1.1.1.1.1.3">𝑏</ci><ci id="S2.F8.16.m16.1.1.1.1.1.4.cmml" xref="S2.F8.16.m16.1.1.1.1.1.4">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F8.16.m16.1d">O(T\times b\times d)</annotation></semantics></math>, so the orange line first decreases with more parallelization, then increases as the extra computation for <math id="S2.F8.17.m17.3" class="ltx_Math" alttext="z_{1},\dots,z_{T}" display="inline"><semantics id="S2.F8.17.m17.3b"><mrow id="S2.F8.17.m17.3.3.2" xref="S2.F8.17.m17.3.3.3.cmml"><msub id="S2.F8.17.m17.2.2.1.1" xref="S2.F8.17.m17.2.2.1.1.cmml"><mi id="S2.F8.17.m17.2.2.1.1.2" xref="S2.F8.17.m17.2.2.1.1.2.cmml">z</mi><mn id="S2.F8.17.m17.2.2.1.1.3" xref="S2.F8.17.m17.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.F8.17.m17.3.3.2.3" xref="S2.F8.17.m17.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.F8.17.m17.1.1" xref="S2.F8.17.m17.1.1.cmml">…</mi><mo id="S2.F8.17.m17.3.3.2.4" xref="S2.F8.17.m17.3.3.3.cmml">,</mo><msub id="S2.F8.17.m17.3.3.2.2" xref="S2.F8.17.m17.3.3.2.2.cmml"><mi id="S2.F8.17.m17.3.3.2.2.2" xref="S2.F8.17.m17.3.3.2.2.2.cmml">z</mi><mi id="S2.F8.17.m17.3.3.2.2.3" xref="S2.F8.17.m17.3.3.2.2.3.cmml">T</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.F8.17.m17.3c"><list id="S2.F8.17.m17.3.3.3.cmml" xref="S2.F8.17.m17.3.3.2"><apply id="S2.F8.17.m17.2.2.1.1.cmml" xref="S2.F8.17.m17.2.2.1.1"><csymbol cd="ambiguous" id="S2.F8.17.m17.2.2.1.1.1.cmml" xref="S2.F8.17.m17.2.2.1.1">subscript</csymbol><ci id="S2.F8.17.m17.2.2.1.1.2.cmml" xref="S2.F8.17.m17.2.2.1.1.2">𝑧</ci><cn type="integer" id="S2.F8.17.m17.2.2.1.1.3.cmml" xref="S2.F8.17.m17.2.2.1.1.3">1</cn></apply><ci id="S2.F8.17.m17.1.1.cmml" xref="S2.F8.17.m17.1.1">…</ci><apply id="S2.F8.17.m17.3.3.2.2.cmml" xref="S2.F8.17.m17.3.3.2.2"><csymbol cd="ambiguous" id="S2.F8.17.m17.3.3.2.2.1.cmml" xref="S2.F8.17.m17.3.3.2.2">subscript</csymbol><ci id="S2.F8.17.m17.3.3.2.2.2.cmml" xref="S2.F8.17.m17.3.3.2.2.2">𝑧</ci><ci id="S2.F8.17.m17.3.3.2.2.3.cmml" xref="S2.F8.17.m17.3.3.2.2.3">𝑇</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.F8.17.m17.3d">z_{1},\dots,z_{T}</annotation></semantics></math> becomes dominant.
</figcaption>
</figure>
</section>
<section id="S2.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.5 </span>Dual form</h3>

<div id="S2.SS5.p1" class="ltx_para">
<p id="S2.SS5.p1.1" class="ltx_p">The parallelization introduced above is necessary but not sufficient for efficiency in wall-clock time.
Modern accelerators specialize in matrix-matrix multiplications, known as <span id="S2.SS5.p1.1.1" class="ltx_text ltx_font_typewriter">matmul</span>s.
For example, the NVIDIA A100 GPU contains highly optimized units called TensorCores that can only perform a single operation – multiplying two matrices each of size <math id="S2.SS5.p1.1.m1.1" class="ltx_Math" alttext="16\times 16" display="inline"><semantics id="S2.SS5.p1.1.m1.1a"><mrow id="S2.SS5.p1.1.m1.1.1" xref="S2.SS5.p1.1.m1.1.1.cmml"><mn id="S2.SS5.p1.1.m1.1.1.2" xref="S2.SS5.p1.1.m1.1.1.2.cmml">16</mn><mo lspace="0.222em" rspace="0.222em" id="S2.SS5.p1.1.m1.1.1.1" xref="S2.SS5.p1.1.m1.1.1.1.cmml">×</mo><mn id="S2.SS5.p1.1.m1.1.1.3" xref="S2.SS5.p1.1.m1.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p1.1.m1.1b"><apply id="S2.SS5.p1.1.m1.1.1.cmml" xref="S2.SS5.p1.1.m1.1.1"><times id="S2.SS5.p1.1.m1.1.1.1.cmml" xref="S2.SS5.p1.1.m1.1.1.1"></times><cn type="integer" id="S2.SS5.p1.1.m1.1.1.2.cmml" xref="S2.SS5.p1.1.m1.1.1.2">16</cn><cn type="integer" id="S2.SS5.p1.1.m1.1.1.3.cmml" xref="S2.SS5.p1.1.m1.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p1.1.m1.1c">16\times 16</annotation></semantics></math>.
Without enough of these <span id="S2.SS5.p1.1.2" class="ltx_text ltx_font_typewriter">matmul</span>s, the TensorCores are idle, and most of the potential for the A100 is unrealized.</p>
</div>
<div id="S2.SS5.p2" class="ltx_para">
<p id="S2.SS5.p2.5" class="ltx_p">Unfortunately, the TTT layer developed so far even with mini-batch still has very few <span id="S2.SS5.p2.5.1" class="ltx_text ltx_font_typewriter">matmul</span>s.
Consider the simplest case of <math id="S2.SS5.p2.1.m1.1" class="ltx_Math" alttext="\ell" display="inline"><semantics id="S2.SS5.p2.1.m1.1a"><mi mathvariant="normal" id="S2.SS5.p2.1.m1.1.1" xref="S2.SS5.p2.1.m1.1.1.cmml">ℓ</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p2.1.m1.1b"><ci id="S2.SS5.p2.1.m1.1.1.cmml" xref="S2.SS5.p2.1.m1.1.1">ℓ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p2.1.m1.1c">\ell</annotation></semantics></math>, where <math id="S2.SS5.p2.2.m2.1" class="ltx_Math" alttext="\theta_{K}=\theta_{V}=\theta_{Q}=I" display="inline"><semantics id="S2.SS5.p2.2.m2.1a"><mrow id="S2.SS5.p2.2.m2.1.1" xref="S2.SS5.p2.2.m2.1.1.cmml"><msub id="S2.SS5.p2.2.m2.1.1.2" xref="S2.SS5.p2.2.m2.1.1.2.cmml"><mi id="S2.SS5.p2.2.m2.1.1.2.2" xref="S2.SS5.p2.2.m2.1.1.2.2.cmml">θ</mi><mi id="S2.SS5.p2.2.m2.1.1.2.3" xref="S2.SS5.p2.2.m2.1.1.2.3.cmml">K</mi></msub><mo id="S2.SS5.p2.2.m2.1.1.3" xref="S2.SS5.p2.2.m2.1.1.3.cmml">=</mo><msub id="S2.SS5.p2.2.m2.1.1.4" xref="S2.SS5.p2.2.m2.1.1.4.cmml"><mi id="S2.SS5.p2.2.m2.1.1.4.2" xref="S2.SS5.p2.2.m2.1.1.4.2.cmml">θ</mi><mi id="S2.SS5.p2.2.m2.1.1.4.3" xref="S2.SS5.p2.2.m2.1.1.4.3.cmml">V</mi></msub><mo id="S2.SS5.p2.2.m2.1.1.5" xref="S2.SS5.p2.2.m2.1.1.5.cmml">=</mo><msub id="S2.SS5.p2.2.m2.1.1.6" xref="S2.SS5.p2.2.m2.1.1.6.cmml"><mi id="S2.SS5.p2.2.m2.1.1.6.2" xref="S2.SS5.p2.2.m2.1.1.6.2.cmml">θ</mi><mi id="S2.SS5.p2.2.m2.1.1.6.3" xref="S2.SS5.p2.2.m2.1.1.6.3.cmml">Q</mi></msub><mo id="S2.SS5.p2.2.m2.1.1.7" xref="S2.SS5.p2.2.m2.1.1.7.cmml">=</mo><mi id="S2.SS5.p2.2.m2.1.1.8" xref="S2.SS5.p2.2.m2.1.1.8.cmml">I</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p2.2.m2.1b"><apply id="S2.SS5.p2.2.m2.1.1.cmml" xref="S2.SS5.p2.2.m2.1.1"><and id="S2.SS5.p2.2.m2.1.1a.cmml" xref="S2.SS5.p2.2.m2.1.1"></and><apply id="S2.SS5.p2.2.m2.1.1b.cmml" xref="S2.SS5.p2.2.m2.1.1"><eq id="S2.SS5.p2.2.m2.1.1.3.cmml" xref="S2.SS5.p2.2.m2.1.1.3"></eq><apply id="S2.SS5.p2.2.m2.1.1.2.cmml" xref="S2.SS5.p2.2.m2.1.1.2"><csymbol cd="ambiguous" id="S2.SS5.p2.2.m2.1.1.2.1.cmml" xref="S2.SS5.p2.2.m2.1.1.2">subscript</csymbol><ci id="S2.SS5.p2.2.m2.1.1.2.2.cmml" xref="S2.SS5.p2.2.m2.1.1.2.2">𝜃</ci><ci id="S2.SS5.p2.2.m2.1.1.2.3.cmml" xref="S2.SS5.p2.2.m2.1.1.2.3">𝐾</ci></apply><apply id="S2.SS5.p2.2.m2.1.1.4.cmml" xref="S2.SS5.p2.2.m2.1.1.4"><csymbol cd="ambiguous" id="S2.SS5.p2.2.m2.1.1.4.1.cmml" xref="S2.SS5.p2.2.m2.1.1.4">subscript</csymbol><ci id="S2.SS5.p2.2.m2.1.1.4.2.cmml" xref="S2.SS5.p2.2.m2.1.1.4.2">𝜃</ci><ci id="S2.SS5.p2.2.m2.1.1.4.3.cmml" xref="S2.SS5.p2.2.m2.1.1.4.3">𝑉</ci></apply></apply><apply id="S2.SS5.p2.2.m2.1.1c.cmml" xref="S2.SS5.p2.2.m2.1.1"><eq id="S2.SS5.p2.2.m2.1.1.5.cmml" xref="S2.SS5.p2.2.m2.1.1.5"></eq><share href="#S2.SS5.p2.2.m2.1.1.4.cmml" id="S2.SS5.p2.2.m2.1.1d.cmml" xref="S2.SS5.p2.2.m2.1.1"></share><apply id="S2.SS5.p2.2.m2.1.1.6.cmml" xref="S2.SS5.p2.2.m2.1.1.6"><csymbol cd="ambiguous" id="S2.SS5.p2.2.m2.1.1.6.1.cmml" xref="S2.SS5.p2.2.m2.1.1.6">subscript</csymbol><ci id="S2.SS5.p2.2.m2.1.1.6.2.cmml" xref="S2.SS5.p2.2.m2.1.1.6.2">𝜃</ci><ci id="S2.SS5.p2.2.m2.1.1.6.3.cmml" xref="S2.SS5.p2.2.m2.1.1.6.3">𝑄</ci></apply></apply><apply id="S2.SS5.p2.2.m2.1.1e.cmml" xref="S2.SS5.p2.2.m2.1.1"><eq id="S2.SS5.p2.2.m2.1.1.7.cmml" xref="S2.SS5.p2.2.m2.1.1.7"></eq><share href="#S2.SS5.p2.2.m2.1.1.6.cmml" id="S2.SS5.p2.2.m2.1.1f.cmml" xref="S2.SS5.p2.2.m2.1.1"></share><ci id="S2.SS5.p2.2.m2.1.1.8.cmml" xref="S2.SS5.p2.2.m2.1.1.8">𝐼</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p2.2.m2.1c">\theta_{K}=\theta_{V}=\theta_{Q}=I</annotation></semantics></math>, for only the first TTT mini-batch of size <math id="S2.SS5.p2.3.m3.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S2.SS5.p2.3.m3.1a"><mi id="S2.SS5.p2.3.m3.1.1" xref="S2.SS5.p2.3.m3.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p2.3.m3.1b"><ci id="S2.SS5.p2.3.m3.1.1.cmml" xref="S2.SS5.p2.3.m3.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p2.3.m3.1c">b</annotation></semantics></math>.
In addition, consider <math id="S2.SS5.p2.4.m4.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S2.SS5.p2.4.m4.1a"><mi id="S2.SS5.p2.4.m4.1.1" xref="S2.SS5.p2.4.m4.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p2.4.m4.1b"><ci id="S2.SS5.p2.4.m4.1.1.cmml" xref="S2.SS5.p2.4.m4.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p2.4.m4.1c">f</annotation></semantics></math> as a linear model.
Copying Equation <a href="#S2.E3" title="In 2.1 TTT as updating a hidden state ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>,
our loss at time <math id="S2.SS5.p2.5.m5.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S2.SS5.p2.5.m5.1a"><mi id="S2.SS5.p2.5.m5.1.1" xref="S2.SS5.p2.5.m5.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p2.5.m5.1b"><ci id="S2.SS5.p2.5.m5.1.1.cmml" xref="S2.SS5.p2.5.m5.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p2.5.m5.1c">t</annotation></semantics></math> is:</p>
<table id="S2.Ex1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.Ex1.m1.1" class="ltx_Math" alttext="\ell\left(W_{0};x_{t}\right)=\|f(x_{t};W_{0})-x_{t}\|^{2}=\|W_{0}x_{t}-x_{t}\|^{2}." display="block"><semantics id="S2.Ex1.m1.1a"><mrow id="S2.Ex1.m1.1.1.1" xref="S2.Ex1.m1.1.1.1.1.cmml"><mrow id="S2.Ex1.m1.1.1.1.1" xref="S2.Ex1.m1.1.1.1.1.cmml"><mrow id="S2.Ex1.m1.1.1.1.1.2" xref="S2.Ex1.m1.1.1.1.1.2.cmml"><mi mathvariant="normal" id="S2.Ex1.m1.1.1.1.1.2.4" xref="S2.Ex1.m1.1.1.1.1.2.4.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="S2.Ex1.m1.1.1.1.1.2.3" xref="S2.Ex1.m1.1.1.1.1.2.3.cmml">​</mo><mrow id="S2.Ex1.m1.1.1.1.1.2.2.2" xref="S2.Ex1.m1.1.1.1.1.2.2.3.cmml"><mo id="S2.Ex1.m1.1.1.1.1.2.2.2.3" xref="S2.Ex1.m1.1.1.1.1.2.2.3.cmml">(</mo><msub id="S2.Ex1.m1.1.1.1.1.1.1.1.1" xref="S2.Ex1.m1.1.1.1.1.1.1.1.1.cmml"><mi id="S2.Ex1.m1.1.1.1.1.1.1.1.1.2" xref="S2.Ex1.m1.1.1.1.1.1.1.1.1.2.cmml">W</mi><mn id="S2.Ex1.m1.1.1.1.1.1.1.1.1.3" xref="S2.Ex1.m1.1.1.1.1.1.1.1.1.3.cmml">0</mn></msub><mo id="S2.Ex1.m1.1.1.1.1.2.2.2.4" xref="S2.Ex1.m1.1.1.1.1.2.2.3.cmml">;</mo><msub id="S2.Ex1.m1.1.1.1.1.2.2.2.2" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.cmml"><mi id="S2.Ex1.m1.1.1.1.1.2.2.2.2.2" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.2.cmml">x</mi><mi id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.cmml">t</mi></msub><mo id="S2.Ex1.m1.1.1.1.1.2.2.2.5" xref="S2.Ex1.m1.1.1.1.1.2.2.3.cmml">)</mo></mrow></mrow><mo id="S2.Ex1.m1.1.1.1.1.6" xref="S2.Ex1.m1.1.1.1.1.6.cmml">=</mo><msup id="S2.Ex1.m1.1.1.1.1.3" xref="S2.Ex1.m1.1.1.1.1.3.cmml"><mrow id="S2.Ex1.m1.1.1.1.1.3.1.1" xref="S2.Ex1.m1.1.1.1.1.3.1.2.cmml"><mo stretchy="false" id="S2.Ex1.m1.1.1.1.1.3.1.1.2" xref="S2.Ex1.m1.1.1.1.1.3.1.2.1.cmml">‖</mo><mrow id="S2.Ex1.m1.1.1.1.1.3.1.1.1" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.cmml"><mrow id="S2.Ex1.m1.1.1.1.1.3.1.1.1.2" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.cmml"><mi id="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.4" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.4.cmml">f</mi><mo lspace="0em" rspace="0em" id="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.3" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.3.cmml">​</mo><mrow id="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.2" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.3.cmml"><mo stretchy="false" id="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.2.3" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.3.cmml">(</mo><msub id="S2.Ex1.m1.1.1.1.1.3.1.1.1.1.1.1.1" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.1.1.1.1.cmml"><mi id="S2.Ex1.m1.1.1.1.1.3.1.1.1.1.1.1.1.2" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="S2.Ex1.m1.1.1.1.1.3.1.1.1.1.1.1.1.3" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.1.1.1.1.3.cmml">t</mi></msub><mo id="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.2.4" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.3.cmml">;</mo><msub id="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.2.2" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.2.2.cmml"><mi id="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.2.2.2" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.2.2.2.cmml">W</mi><mn id="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.2.2.3" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.2.2.3.cmml">0</mn></msub><mo stretchy="false" id="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.2.5" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.3.cmml">)</mo></mrow></mrow><mo id="S2.Ex1.m1.1.1.1.1.3.1.1.1.3" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.3.cmml">−</mo><msub id="S2.Ex1.m1.1.1.1.1.3.1.1.1.4" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.4.cmml"><mi id="S2.Ex1.m1.1.1.1.1.3.1.1.1.4.2" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.4.2.cmml">x</mi><mi id="S2.Ex1.m1.1.1.1.1.3.1.1.1.4.3" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.4.3.cmml">t</mi></msub></mrow><mo stretchy="false" id="S2.Ex1.m1.1.1.1.1.3.1.1.3" xref="S2.Ex1.m1.1.1.1.1.3.1.2.1.cmml">‖</mo></mrow><mn id="S2.Ex1.m1.1.1.1.1.3.3" xref="S2.Ex1.m1.1.1.1.1.3.3.cmml">2</mn></msup><mo id="S2.Ex1.m1.1.1.1.1.7" xref="S2.Ex1.m1.1.1.1.1.7.cmml">=</mo><msup id="S2.Ex1.m1.1.1.1.1.4" xref="S2.Ex1.m1.1.1.1.1.4.cmml"><mrow id="S2.Ex1.m1.1.1.1.1.4.1.1" xref="S2.Ex1.m1.1.1.1.1.4.1.2.cmml"><mo stretchy="false" id="S2.Ex1.m1.1.1.1.1.4.1.1.2" xref="S2.Ex1.m1.1.1.1.1.4.1.2.1.cmml">‖</mo><mrow id="S2.Ex1.m1.1.1.1.1.4.1.1.1" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.cmml"><mrow id="S2.Ex1.m1.1.1.1.1.4.1.1.1.2" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.cmml"><msub id="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.2" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.2.cmml"><mi id="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.2.2" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.2.2.cmml">W</mi><mn id="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.2.3" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.2.3.cmml">0</mn></msub><mo lspace="0em" rspace="0em" id="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.1" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.1.cmml">​</mo><msub id="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.3" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.3.cmml"><mi id="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.3.2" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.3.2.cmml">x</mi><mi id="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.3.3" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.3.3.cmml">t</mi></msub></mrow><mo id="S2.Ex1.m1.1.1.1.1.4.1.1.1.1" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.1.cmml">−</mo><msub id="S2.Ex1.m1.1.1.1.1.4.1.1.1.3" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.3.cmml"><mi id="S2.Ex1.m1.1.1.1.1.4.1.1.1.3.2" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.3.2.cmml">x</mi><mi id="S2.Ex1.m1.1.1.1.1.4.1.1.1.3.3" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.3.3.cmml">t</mi></msub></mrow><mo stretchy="false" id="S2.Ex1.m1.1.1.1.1.4.1.1.3" xref="S2.Ex1.m1.1.1.1.1.4.1.2.1.cmml">‖</mo></mrow><mn id="S2.Ex1.m1.1.1.1.1.4.3" xref="S2.Ex1.m1.1.1.1.1.4.3.cmml">2</mn></msup></mrow><mo lspace="0em" id="S2.Ex1.m1.1.1.1.2" xref="S2.Ex1.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.Ex1.m1.1b"><apply id="S2.Ex1.m1.1.1.1.1.cmml" xref="S2.Ex1.m1.1.1.1"><and id="S2.Ex1.m1.1.1.1.1a.cmml" xref="S2.Ex1.m1.1.1.1"></and><apply id="S2.Ex1.m1.1.1.1.1b.cmml" xref="S2.Ex1.m1.1.1.1"><eq id="S2.Ex1.m1.1.1.1.1.6.cmml" xref="S2.Ex1.m1.1.1.1.1.6"></eq><apply id="S2.Ex1.m1.1.1.1.1.2.cmml" xref="S2.Ex1.m1.1.1.1.1.2"><times id="S2.Ex1.m1.1.1.1.1.2.3.cmml" xref="S2.Ex1.m1.1.1.1.1.2.3"></times><ci id="S2.Ex1.m1.1.1.1.1.2.4.cmml" xref="S2.Ex1.m1.1.1.1.1.2.4">ℓ</ci><list id="S2.Ex1.m1.1.1.1.1.2.2.3.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2"><apply id="S2.Ex1.m1.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex1.m1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex1.m1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.Ex1.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.Ex1.m1.1.1.1.1.1.1.1.1.2">𝑊</ci><cn type="integer" id="S2.Ex1.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.Ex1.m1.1.1.1.1.1.1.1.1.3">0</cn></apply><apply id="S2.Ex1.m1.1.1.1.1.2.2.2.2.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.1.1.2.2.2.2.1.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2">subscript</csymbol><ci id="S2.Ex1.m1.1.1.1.1.2.2.2.2.2.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.2">𝑥</ci><ci id="S2.Ex1.m1.1.1.1.1.2.2.2.2.3.cmml" xref="S2.Ex1.m1.1.1.1.1.2.2.2.2.3">𝑡</ci></apply></list></apply><apply id="S2.Ex1.m1.1.1.1.1.3.cmml" xref="S2.Ex1.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.1.1.3.2.cmml" xref="S2.Ex1.m1.1.1.1.1.3">superscript</csymbol><apply id="S2.Ex1.m1.1.1.1.1.3.1.2.cmml" xref="S2.Ex1.m1.1.1.1.1.3.1.1"><csymbol cd="latexml" id="S2.Ex1.m1.1.1.1.1.3.1.2.1.cmml" xref="S2.Ex1.m1.1.1.1.1.3.1.1.2">norm</csymbol><apply id="S2.Ex1.m1.1.1.1.1.3.1.1.1.cmml" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1"><minus id="S2.Ex1.m1.1.1.1.1.3.1.1.1.3.cmml" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.3"></minus><apply id="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.cmml" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.2"><times id="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.3.cmml" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.3"></times><ci id="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.4.cmml" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.4">𝑓</ci><list id="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.3.cmml" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.2"><apply id="S2.Ex1.m1.1.1.1.1.3.1.1.1.1.1.1.1.cmml" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.1.1.3.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.Ex1.m1.1.1.1.1.3.1.1.1.1.1.1.1.2.cmml" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.1.1.1.1.2">𝑥</ci><ci id="S2.Ex1.m1.1.1.1.1.3.1.1.1.1.1.1.1.3.cmml" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.1.1.1.1.3">𝑡</ci></apply><apply id="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.2.2.cmml" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.2.2.1.cmml" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.2.2">subscript</csymbol><ci id="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.2.2.2.cmml" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.2.2.2">𝑊</ci><cn type="integer" id="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.2.2.3.cmml" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.2.2.2.2.3">0</cn></apply></list></apply><apply id="S2.Ex1.m1.1.1.1.1.3.1.1.1.4.cmml" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.4"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.1.1.3.1.1.1.4.1.cmml" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.4">subscript</csymbol><ci id="S2.Ex1.m1.1.1.1.1.3.1.1.1.4.2.cmml" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.4.2">𝑥</ci><ci id="S2.Ex1.m1.1.1.1.1.3.1.1.1.4.3.cmml" xref="S2.Ex1.m1.1.1.1.1.3.1.1.1.4.3">𝑡</ci></apply></apply></apply><cn type="integer" id="S2.Ex1.m1.1.1.1.1.3.3.cmml" xref="S2.Ex1.m1.1.1.1.1.3.3">2</cn></apply></apply><apply id="S2.Ex1.m1.1.1.1.1c.cmml" xref="S2.Ex1.m1.1.1.1"><eq id="S2.Ex1.m1.1.1.1.1.7.cmml" xref="S2.Ex1.m1.1.1.1.1.7"></eq><share href="#S2.Ex1.m1.1.1.1.1.3.cmml" id="S2.Ex1.m1.1.1.1.1d.cmml" xref="S2.Ex1.m1.1.1.1"></share><apply id="S2.Ex1.m1.1.1.1.1.4.cmml" xref="S2.Ex1.m1.1.1.1.1.4"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.1.1.4.2.cmml" xref="S2.Ex1.m1.1.1.1.1.4">superscript</csymbol><apply id="S2.Ex1.m1.1.1.1.1.4.1.2.cmml" xref="S2.Ex1.m1.1.1.1.1.4.1.1"><csymbol cd="latexml" id="S2.Ex1.m1.1.1.1.1.4.1.2.1.cmml" xref="S2.Ex1.m1.1.1.1.1.4.1.1.2">norm</csymbol><apply id="S2.Ex1.m1.1.1.1.1.4.1.1.1.cmml" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1"><minus id="S2.Ex1.m1.1.1.1.1.4.1.1.1.1.cmml" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.1"></minus><apply id="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.cmml" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.2"><times id="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.1.cmml" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.1"></times><apply id="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.2.cmml" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.2"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.2.1.cmml" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.2">subscript</csymbol><ci id="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.2.2.cmml" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.2.2">𝑊</ci><cn type="integer" id="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.2.3.cmml" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.2.3">0</cn></apply><apply id="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.3.cmml" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.3"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.3.1.cmml" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.3">subscript</csymbol><ci id="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.3.2.cmml" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.3.2">𝑥</ci><ci id="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.3.3.cmml" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.2.3.3">𝑡</ci></apply></apply><apply id="S2.Ex1.m1.1.1.1.1.4.1.1.1.3.cmml" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.1.1.4.1.1.1.3.1.cmml" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.3">subscript</csymbol><ci id="S2.Ex1.m1.1.1.1.1.4.1.1.1.3.2.cmml" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.3.2">𝑥</ci><ci id="S2.Ex1.m1.1.1.1.1.4.1.1.1.3.3.cmml" xref="S2.Ex1.m1.1.1.1.1.4.1.1.1.3.3">𝑡</ci></apply></apply></apply><cn type="integer" id="S2.Ex1.m1.1.1.1.1.4.3.cmml" xref="S2.Ex1.m1.1.1.1.1.4.3">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex1.m1.1c">\ell\left(W_{0};x_{t}\right)=\|f(x_{t};W_{0})-x_{t}\|^{2}=\|W_{0}x_{t}-x_{t}\|^{2}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S2.SS5.p2.15" class="ltx_p">As discussed in Subsection <a href="#S2.SS4" title="2.4 Parallelization with mini-batch TTT ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.4</span></a>, we can parallelize the computation of:</p>
<table id="S2.Ex2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.Ex2.m1.1" class="ltx_Math" alttext="G_{t}=\nabla\ell\left(W_{0};x_{t}\right)=2(W_{0}x_{t}-x_{t})x_{t}^{T}," display="block"><semantics id="S2.Ex2.m1.1a"><mrow id="S2.Ex2.m1.1.1.1" xref="S2.Ex2.m1.1.1.1.1.cmml"><mrow id="S2.Ex2.m1.1.1.1.1" xref="S2.Ex2.m1.1.1.1.1.cmml"><msub id="S2.Ex2.m1.1.1.1.1.5" xref="S2.Ex2.m1.1.1.1.1.5.cmml"><mi id="S2.Ex2.m1.1.1.1.1.5.2" xref="S2.Ex2.m1.1.1.1.1.5.2.cmml">G</mi><mi id="S2.Ex2.m1.1.1.1.1.5.3" xref="S2.Ex2.m1.1.1.1.1.5.3.cmml">t</mi></msub><mo id="S2.Ex2.m1.1.1.1.1.6" xref="S2.Ex2.m1.1.1.1.1.6.cmml">=</mo><mrow id="S2.Ex2.m1.1.1.1.1.2" xref="S2.Ex2.m1.1.1.1.1.2.cmml"><mrow id="S2.Ex2.m1.1.1.1.1.2.4" xref="S2.Ex2.m1.1.1.1.1.2.4.cmml"><mo rspace="0.167em" id="S2.Ex2.m1.1.1.1.1.2.4.1" xref="S2.Ex2.m1.1.1.1.1.2.4.1.cmml">∇</mo><mi mathvariant="normal" id="S2.Ex2.m1.1.1.1.1.2.4.2" xref="S2.Ex2.m1.1.1.1.1.2.4.2.cmml">ℓ</mi></mrow><mo lspace="0em" rspace="0em" id="S2.Ex2.m1.1.1.1.1.2.3" xref="S2.Ex2.m1.1.1.1.1.2.3.cmml">​</mo><mrow id="S2.Ex2.m1.1.1.1.1.2.2.2" xref="S2.Ex2.m1.1.1.1.1.2.2.3.cmml"><mo id="S2.Ex2.m1.1.1.1.1.2.2.2.3" xref="S2.Ex2.m1.1.1.1.1.2.2.3.cmml">(</mo><msub id="S2.Ex2.m1.1.1.1.1.1.1.1.1" xref="S2.Ex2.m1.1.1.1.1.1.1.1.1.cmml"><mi id="S2.Ex2.m1.1.1.1.1.1.1.1.1.2" xref="S2.Ex2.m1.1.1.1.1.1.1.1.1.2.cmml">W</mi><mn id="S2.Ex2.m1.1.1.1.1.1.1.1.1.3" xref="S2.Ex2.m1.1.1.1.1.1.1.1.1.3.cmml">0</mn></msub><mo id="S2.Ex2.m1.1.1.1.1.2.2.2.4" xref="S2.Ex2.m1.1.1.1.1.2.2.3.cmml">;</mo><msub id="S2.Ex2.m1.1.1.1.1.2.2.2.2" xref="S2.Ex2.m1.1.1.1.1.2.2.2.2.cmml"><mi id="S2.Ex2.m1.1.1.1.1.2.2.2.2.2" xref="S2.Ex2.m1.1.1.1.1.2.2.2.2.2.cmml">x</mi><mi id="S2.Ex2.m1.1.1.1.1.2.2.2.2.3" xref="S2.Ex2.m1.1.1.1.1.2.2.2.2.3.cmml">t</mi></msub><mo id="S2.Ex2.m1.1.1.1.1.2.2.2.5" xref="S2.Ex2.m1.1.1.1.1.2.2.3.cmml">)</mo></mrow></mrow><mo id="S2.Ex2.m1.1.1.1.1.7" xref="S2.Ex2.m1.1.1.1.1.7.cmml">=</mo><mrow id="S2.Ex2.m1.1.1.1.1.3" xref="S2.Ex2.m1.1.1.1.1.3.cmml"><mn id="S2.Ex2.m1.1.1.1.1.3.3" xref="S2.Ex2.m1.1.1.1.1.3.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="S2.Ex2.m1.1.1.1.1.3.2" xref="S2.Ex2.m1.1.1.1.1.3.2.cmml">​</mo><mrow id="S2.Ex2.m1.1.1.1.1.3.1.1" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.cmml"><mo stretchy="false" id="S2.Ex2.m1.1.1.1.1.3.1.1.2" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.cmml">(</mo><mrow id="S2.Ex2.m1.1.1.1.1.3.1.1.1" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.cmml"><mrow id="S2.Ex2.m1.1.1.1.1.3.1.1.1.2" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.cmml"><msub id="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.2" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.2.cmml"><mi id="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.2.2" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.2.2.cmml">W</mi><mn id="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.2.3" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.2.3.cmml">0</mn></msub><mo lspace="0em" rspace="0em" id="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.1" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.1.cmml">​</mo><msub id="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.3" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.3.cmml"><mi id="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.3.2" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.3.2.cmml">x</mi><mi id="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.3.3" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.3.3.cmml">t</mi></msub></mrow><mo id="S2.Ex2.m1.1.1.1.1.3.1.1.1.1" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.1.cmml">−</mo><msub id="S2.Ex2.m1.1.1.1.1.3.1.1.1.3" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.3.cmml"><mi id="S2.Ex2.m1.1.1.1.1.3.1.1.1.3.2" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.3.2.cmml">x</mi><mi id="S2.Ex2.m1.1.1.1.1.3.1.1.1.3.3" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.3.3.cmml">t</mi></msub></mrow><mo stretchy="false" id="S2.Ex2.m1.1.1.1.1.3.1.1.3" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S2.Ex2.m1.1.1.1.1.3.2a" xref="S2.Ex2.m1.1.1.1.1.3.2.cmml">​</mo><msubsup id="S2.Ex2.m1.1.1.1.1.3.4" xref="S2.Ex2.m1.1.1.1.1.3.4.cmml"><mi id="S2.Ex2.m1.1.1.1.1.3.4.2.2" xref="S2.Ex2.m1.1.1.1.1.3.4.2.2.cmml">x</mi><mi id="S2.Ex2.m1.1.1.1.1.3.4.2.3" xref="S2.Ex2.m1.1.1.1.1.3.4.2.3.cmml">t</mi><mi id="S2.Ex2.m1.1.1.1.1.3.4.3" xref="S2.Ex2.m1.1.1.1.1.3.4.3.cmml">T</mi></msubsup></mrow></mrow><mo id="S2.Ex2.m1.1.1.1.2" xref="S2.Ex2.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.Ex2.m1.1b"><apply id="S2.Ex2.m1.1.1.1.1.cmml" xref="S2.Ex2.m1.1.1.1"><and id="S2.Ex2.m1.1.1.1.1a.cmml" xref="S2.Ex2.m1.1.1.1"></and><apply id="S2.Ex2.m1.1.1.1.1b.cmml" xref="S2.Ex2.m1.1.1.1"><eq id="S2.Ex2.m1.1.1.1.1.6.cmml" xref="S2.Ex2.m1.1.1.1.1.6"></eq><apply id="S2.Ex2.m1.1.1.1.1.5.cmml" xref="S2.Ex2.m1.1.1.1.1.5"><csymbol cd="ambiguous" id="S2.Ex2.m1.1.1.1.1.5.1.cmml" xref="S2.Ex2.m1.1.1.1.1.5">subscript</csymbol><ci id="S2.Ex2.m1.1.1.1.1.5.2.cmml" xref="S2.Ex2.m1.1.1.1.1.5.2">𝐺</ci><ci id="S2.Ex2.m1.1.1.1.1.5.3.cmml" xref="S2.Ex2.m1.1.1.1.1.5.3">𝑡</ci></apply><apply id="S2.Ex2.m1.1.1.1.1.2.cmml" xref="S2.Ex2.m1.1.1.1.1.2"><times id="S2.Ex2.m1.1.1.1.1.2.3.cmml" xref="S2.Ex2.m1.1.1.1.1.2.3"></times><apply id="S2.Ex2.m1.1.1.1.1.2.4.cmml" xref="S2.Ex2.m1.1.1.1.1.2.4"><ci id="S2.Ex2.m1.1.1.1.1.2.4.1.cmml" xref="S2.Ex2.m1.1.1.1.1.2.4.1">∇</ci><ci id="S2.Ex2.m1.1.1.1.1.2.4.2.cmml" xref="S2.Ex2.m1.1.1.1.1.2.4.2">ℓ</ci></apply><list id="S2.Ex2.m1.1.1.1.1.2.2.3.cmml" xref="S2.Ex2.m1.1.1.1.1.2.2.2"><apply id="S2.Ex2.m1.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex2.m1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.Ex2.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex2.m1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.Ex2.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.Ex2.m1.1.1.1.1.1.1.1.1.2">𝑊</ci><cn type="integer" id="S2.Ex2.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.Ex2.m1.1.1.1.1.1.1.1.1.3">0</cn></apply><apply id="S2.Ex2.m1.1.1.1.1.2.2.2.2.cmml" xref="S2.Ex2.m1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S2.Ex2.m1.1.1.1.1.2.2.2.2.1.cmml" xref="S2.Ex2.m1.1.1.1.1.2.2.2.2">subscript</csymbol><ci id="S2.Ex2.m1.1.1.1.1.2.2.2.2.2.cmml" xref="S2.Ex2.m1.1.1.1.1.2.2.2.2.2">𝑥</ci><ci id="S2.Ex2.m1.1.1.1.1.2.2.2.2.3.cmml" xref="S2.Ex2.m1.1.1.1.1.2.2.2.2.3">𝑡</ci></apply></list></apply></apply><apply id="S2.Ex2.m1.1.1.1.1c.cmml" xref="S2.Ex2.m1.1.1.1"><eq id="S2.Ex2.m1.1.1.1.1.7.cmml" xref="S2.Ex2.m1.1.1.1.1.7"></eq><share href="#S2.Ex2.m1.1.1.1.1.2.cmml" id="S2.Ex2.m1.1.1.1.1d.cmml" xref="S2.Ex2.m1.1.1.1"></share><apply id="S2.Ex2.m1.1.1.1.1.3.cmml" xref="S2.Ex2.m1.1.1.1.1.3"><times id="S2.Ex2.m1.1.1.1.1.3.2.cmml" xref="S2.Ex2.m1.1.1.1.1.3.2"></times><cn type="integer" id="S2.Ex2.m1.1.1.1.1.3.3.cmml" xref="S2.Ex2.m1.1.1.1.1.3.3">2</cn><apply id="S2.Ex2.m1.1.1.1.1.3.1.1.1.cmml" xref="S2.Ex2.m1.1.1.1.1.3.1.1"><minus id="S2.Ex2.m1.1.1.1.1.3.1.1.1.1.cmml" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.1"></minus><apply id="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.cmml" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.2"><times id="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.1.cmml" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.1"></times><apply id="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.2.cmml" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.2"><csymbol cd="ambiguous" id="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.2.1.cmml" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.2">subscript</csymbol><ci id="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.2.2.cmml" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.2.2">𝑊</ci><cn type="integer" id="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.2.3.cmml" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.2.3">0</cn></apply><apply id="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.3.cmml" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.3"><csymbol cd="ambiguous" id="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.3.1.cmml" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.3">subscript</csymbol><ci id="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.3.2.cmml" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.3.2">𝑥</ci><ci id="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.3.3.cmml" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.2.3.3">𝑡</ci></apply></apply><apply id="S2.Ex2.m1.1.1.1.1.3.1.1.1.3.cmml" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex2.m1.1.1.1.1.3.1.1.1.3.1.cmml" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.3">subscript</csymbol><ci id="S2.Ex2.m1.1.1.1.1.3.1.1.1.3.2.cmml" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.3.2">𝑥</ci><ci id="S2.Ex2.m1.1.1.1.1.3.1.1.1.3.3.cmml" xref="S2.Ex2.m1.1.1.1.1.3.1.1.1.3.3">𝑡</ci></apply></apply><apply id="S2.Ex2.m1.1.1.1.1.3.4.cmml" xref="S2.Ex2.m1.1.1.1.1.3.4"><csymbol cd="ambiguous" id="S2.Ex2.m1.1.1.1.1.3.4.1.cmml" xref="S2.Ex2.m1.1.1.1.1.3.4">superscript</csymbol><apply id="S2.Ex2.m1.1.1.1.1.3.4.2.cmml" xref="S2.Ex2.m1.1.1.1.1.3.4"><csymbol cd="ambiguous" id="S2.Ex2.m1.1.1.1.1.3.4.2.1.cmml" xref="S2.Ex2.m1.1.1.1.1.3.4">subscript</csymbol><ci id="S2.Ex2.m1.1.1.1.1.3.4.2.2.cmml" xref="S2.Ex2.m1.1.1.1.1.3.4.2.2">𝑥</ci><ci id="S2.Ex2.m1.1.1.1.1.3.4.2.3.cmml" xref="S2.Ex2.m1.1.1.1.1.3.4.2.3">𝑡</ci></apply><ci id="S2.Ex2.m1.1.1.1.1.3.4.3.cmml" xref="S2.Ex2.m1.1.1.1.1.3.4.3">𝑇</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex2.m1.1c">G_{t}=\nabla\ell\left(W_{0};x_{t}\right)=2(W_{0}x_{t}-x_{t})x_{t}^{T},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S2.SS5.p2.14" class="ltx_p">for <math id="S2.SS5.p2.6.m1.3" class="ltx_Math" alttext="t=1,\dots,b" display="inline"><semantics id="S2.SS5.p2.6.m1.3a"><mrow id="S2.SS5.p2.6.m1.3.4" xref="S2.SS5.p2.6.m1.3.4.cmml"><mi id="S2.SS5.p2.6.m1.3.4.2" xref="S2.SS5.p2.6.m1.3.4.2.cmml">t</mi><mo id="S2.SS5.p2.6.m1.3.4.1" xref="S2.SS5.p2.6.m1.3.4.1.cmml">=</mo><mrow id="S2.SS5.p2.6.m1.3.4.3.2" xref="S2.SS5.p2.6.m1.3.4.3.1.cmml"><mn id="S2.SS5.p2.6.m1.1.1" xref="S2.SS5.p2.6.m1.1.1.cmml">1</mn><mo id="S2.SS5.p2.6.m1.3.4.3.2.1" xref="S2.SS5.p2.6.m1.3.4.3.1.cmml">,</mo><mi mathvariant="normal" id="S2.SS5.p2.6.m1.2.2" xref="S2.SS5.p2.6.m1.2.2.cmml">…</mi><mo id="S2.SS5.p2.6.m1.3.4.3.2.2" xref="S2.SS5.p2.6.m1.3.4.3.1.cmml">,</mo><mi id="S2.SS5.p2.6.m1.3.3" xref="S2.SS5.p2.6.m1.3.3.cmml">b</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p2.6.m1.3b"><apply id="S2.SS5.p2.6.m1.3.4.cmml" xref="S2.SS5.p2.6.m1.3.4"><eq id="S2.SS5.p2.6.m1.3.4.1.cmml" xref="S2.SS5.p2.6.m1.3.4.1"></eq><ci id="S2.SS5.p2.6.m1.3.4.2.cmml" xref="S2.SS5.p2.6.m1.3.4.2">𝑡</ci><list id="S2.SS5.p2.6.m1.3.4.3.1.cmml" xref="S2.SS5.p2.6.m1.3.4.3.2"><cn type="integer" id="S2.SS5.p2.6.m1.1.1.cmml" xref="S2.SS5.p2.6.m1.1.1">1</cn><ci id="S2.SS5.p2.6.m1.2.2.cmml" xref="S2.SS5.p2.6.m1.2.2">…</ci><ci id="S2.SS5.p2.6.m1.3.3.cmml" xref="S2.SS5.p2.6.m1.3.3">𝑏</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p2.6.m1.3c">t=1,\dots,b</annotation></semantics></math>.
However, we cannot compute all <math id="S2.SS5.p2.7.m2.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S2.SS5.p2.7.m2.1a"><mi id="S2.SS5.p2.7.m2.1.1" xref="S2.SS5.p2.7.m2.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p2.7.m2.1b"><ci id="S2.SS5.p2.7.m2.1.1.cmml" xref="S2.SS5.p2.7.m2.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p2.7.m2.1c">b</annotation></semantics></math> of the <math id="S2.SS5.p2.8.m3.1" class="ltx_Math" alttext="G_{t}" display="inline"><semantics id="S2.SS5.p2.8.m3.1a"><msub id="S2.SS5.p2.8.m3.1.1" xref="S2.SS5.p2.8.m3.1.1.cmml"><mi id="S2.SS5.p2.8.m3.1.1.2" xref="S2.SS5.p2.8.m3.1.1.2.cmml">G</mi><mi id="S2.SS5.p2.8.m3.1.1.3" xref="S2.SS5.p2.8.m3.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS5.p2.8.m3.1b"><apply id="S2.SS5.p2.8.m3.1.1.cmml" xref="S2.SS5.p2.8.m3.1.1"><csymbol cd="ambiguous" id="S2.SS5.p2.8.m3.1.1.1.cmml" xref="S2.SS5.p2.8.m3.1.1">subscript</csymbol><ci id="S2.SS5.p2.8.m3.1.1.2.cmml" xref="S2.SS5.p2.8.m3.1.1.2">𝐺</ci><ci id="S2.SS5.p2.8.m3.1.1.3.cmml" xref="S2.SS5.p2.8.m3.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p2.8.m3.1c">G_{t}</annotation></semantics></math>s through a single <span id="S2.SS5.p2.14.1" class="ltx_text ltx_font_typewriter">matmul</span>.
Instead, we need <math id="S2.SS5.p2.9.m4.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S2.SS5.p2.9.m4.1a"><mi id="S2.SS5.p2.9.m4.1.1" xref="S2.SS5.p2.9.m4.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p2.9.m4.1b"><ci id="S2.SS5.p2.9.m4.1.1.cmml" xref="S2.SS5.p2.9.m4.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p2.9.m4.1c">b</annotation></semantics></math> outer products to compute them one by one.
To make matters worse, for each <math id="S2.SS5.p2.10.m5.1" class="ltx_Math" alttext="x_{t}\in\mathbb{R}^{d}" display="inline"><semantics id="S2.SS5.p2.10.m5.1a"><mrow id="S2.SS5.p2.10.m5.1.1" xref="S2.SS5.p2.10.m5.1.1.cmml"><msub id="S2.SS5.p2.10.m5.1.1.2" xref="S2.SS5.p2.10.m5.1.1.2.cmml"><mi id="S2.SS5.p2.10.m5.1.1.2.2" xref="S2.SS5.p2.10.m5.1.1.2.2.cmml">x</mi><mi id="S2.SS5.p2.10.m5.1.1.2.3" xref="S2.SS5.p2.10.m5.1.1.2.3.cmml">t</mi></msub><mo id="S2.SS5.p2.10.m5.1.1.1" xref="S2.SS5.p2.10.m5.1.1.1.cmml">∈</mo><msup id="S2.SS5.p2.10.m5.1.1.3" xref="S2.SS5.p2.10.m5.1.1.3.cmml"><mi id="S2.SS5.p2.10.m5.1.1.3.2" xref="S2.SS5.p2.10.m5.1.1.3.2.cmml">ℝ</mi><mi id="S2.SS5.p2.10.m5.1.1.3.3" xref="S2.SS5.p2.10.m5.1.1.3.3.cmml">d</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p2.10.m5.1b"><apply id="S2.SS5.p2.10.m5.1.1.cmml" xref="S2.SS5.p2.10.m5.1.1"><in id="S2.SS5.p2.10.m5.1.1.1.cmml" xref="S2.SS5.p2.10.m5.1.1.1"></in><apply id="S2.SS5.p2.10.m5.1.1.2.cmml" xref="S2.SS5.p2.10.m5.1.1.2"><csymbol cd="ambiguous" id="S2.SS5.p2.10.m5.1.1.2.1.cmml" xref="S2.SS5.p2.10.m5.1.1.2">subscript</csymbol><ci id="S2.SS5.p2.10.m5.1.1.2.2.cmml" xref="S2.SS5.p2.10.m5.1.1.2.2">𝑥</ci><ci id="S2.SS5.p2.10.m5.1.1.2.3.cmml" xref="S2.SS5.p2.10.m5.1.1.2.3">𝑡</ci></apply><apply id="S2.SS5.p2.10.m5.1.1.3.cmml" xref="S2.SS5.p2.10.m5.1.1.3"><csymbol cd="ambiguous" id="S2.SS5.p2.10.m5.1.1.3.1.cmml" xref="S2.SS5.p2.10.m5.1.1.3">superscript</csymbol><ci id="S2.SS5.p2.10.m5.1.1.3.2.cmml" xref="S2.SS5.p2.10.m5.1.1.3.2">ℝ</ci><ci id="S2.SS5.p2.10.m5.1.1.3.3.cmml" xref="S2.SS5.p2.10.m5.1.1.3.3">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p2.10.m5.1c">x_{t}\in\mathbb{R}^{d}</annotation></semantics></math>, <math id="S2.SS5.p2.11.m6.1" class="ltx_Math" alttext="G_{t}" display="inline"><semantics id="S2.SS5.p2.11.m6.1a"><msub id="S2.SS5.p2.11.m6.1.1" xref="S2.SS5.p2.11.m6.1.1.cmml"><mi id="S2.SS5.p2.11.m6.1.1.2" xref="S2.SS5.p2.11.m6.1.1.2.cmml">G</mi><mi id="S2.SS5.p2.11.m6.1.1.3" xref="S2.SS5.p2.11.m6.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS5.p2.11.m6.1b"><apply id="S2.SS5.p2.11.m6.1.1.cmml" xref="S2.SS5.p2.11.m6.1.1"><csymbol cd="ambiguous" id="S2.SS5.p2.11.m6.1.1.1.cmml" xref="S2.SS5.p2.11.m6.1.1">subscript</csymbol><ci id="S2.SS5.p2.11.m6.1.1.2.cmml" xref="S2.SS5.p2.11.m6.1.1.2">𝐺</ci><ci id="S2.SS5.p2.11.m6.1.1.3.cmml" xref="S2.SS5.p2.11.m6.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p2.11.m6.1c">G_{t}</annotation></semantics></math> is <math id="S2.SS5.p2.12.m7.1" class="ltx_Math" alttext="d\times d" display="inline"><semantics id="S2.SS5.p2.12.m7.1a"><mrow id="S2.SS5.p2.12.m7.1.1" xref="S2.SS5.p2.12.m7.1.1.cmml"><mi id="S2.SS5.p2.12.m7.1.1.2" xref="S2.SS5.p2.12.m7.1.1.2.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS5.p2.12.m7.1.1.1" xref="S2.SS5.p2.12.m7.1.1.1.cmml">×</mo><mi id="S2.SS5.p2.12.m7.1.1.3" xref="S2.SS5.p2.12.m7.1.1.3.cmml">d</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p2.12.m7.1b"><apply id="S2.SS5.p2.12.m7.1.1.cmml" xref="S2.SS5.p2.12.m7.1.1"><times id="S2.SS5.p2.12.m7.1.1.1.cmml" xref="S2.SS5.p2.12.m7.1.1.1"></times><ci id="S2.SS5.p2.12.m7.1.1.2.cmml" xref="S2.SS5.p2.12.m7.1.1.2">𝑑</ci><ci id="S2.SS5.p2.12.m7.1.1.3.cmml" xref="S2.SS5.p2.12.m7.1.1.3">𝑑</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p2.12.m7.1c">d\times d</annotation></semantics></math>, which incurs much heavier memory footprint and I/O cost than <math id="S2.SS5.p2.13.m8.1" class="ltx_Math" alttext="x_{t}" display="inline"><semantics id="S2.SS5.p2.13.m8.1a"><msub id="S2.SS5.p2.13.m8.1.1" xref="S2.SS5.p2.13.m8.1.1.cmml"><mi id="S2.SS5.p2.13.m8.1.1.2" xref="S2.SS5.p2.13.m8.1.1.2.cmml">x</mi><mi id="S2.SS5.p2.13.m8.1.1.3" xref="S2.SS5.p2.13.m8.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS5.p2.13.m8.1b"><apply id="S2.SS5.p2.13.m8.1.1.cmml" xref="S2.SS5.p2.13.m8.1.1"><csymbol cd="ambiguous" id="S2.SS5.p2.13.m8.1.1.1.cmml" xref="S2.SS5.p2.13.m8.1.1">subscript</csymbol><ci id="S2.SS5.p2.13.m8.1.1.2.cmml" xref="S2.SS5.p2.13.m8.1.1.2">𝑥</ci><ci id="S2.SS5.p2.13.m8.1.1.3.cmml" xref="S2.SS5.p2.13.m8.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p2.13.m8.1c">x_{t}</annotation></semantics></math> for large <math id="S2.SS5.p2.14.m9.1" class="ltx_Math" alttext="d" display="inline"><semantics id="S2.SS5.p2.14.m9.1a"><mi id="S2.SS5.p2.14.m9.1.1" xref="S2.SS5.p2.14.m9.1.1.cmml">d</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p2.14.m9.1b"><ci id="S2.SS5.p2.14.m9.1.1.cmml" xref="S2.SS5.p2.14.m9.1.1">𝑑</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p2.14.m9.1c">d</annotation></semantics></math>.</p>
</div>
<div id="S2.SS5.p3" class="ltx_para">
<p id="S2.SS5.p3.4" class="ltx_p">To solve these two problems, we make a simple observation: We do not actually need to materialize <math id="S2.SS5.p3.1.m1.3" class="ltx_Math" alttext="G_{1},\dots,G_{b}" display="inline"><semantics id="S2.SS5.p3.1.m1.3a"><mrow id="S2.SS5.p3.1.m1.3.3.2" xref="S2.SS5.p3.1.m1.3.3.3.cmml"><msub id="S2.SS5.p3.1.m1.2.2.1.1" xref="S2.SS5.p3.1.m1.2.2.1.1.cmml"><mi id="S2.SS5.p3.1.m1.2.2.1.1.2" xref="S2.SS5.p3.1.m1.2.2.1.1.2.cmml">G</mi><mn id="S2.SS5.p3.1.m1.2.2.1.1.3" xref="S2.SS5.p3.1.m1.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.SS5.p3.1.m1.3.3.2.3" xref="S2.SS5.p3.1.m1.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS5.p3.1.m1.1.1" xref="S2.SS5.p3.1.m1.1.1.cmml">…</mi><mo id="S2.SS5.p3.1.m1.3.3.2.4" xref="S2.SS5.p3.1.m1.3.3.3.cmml">,</mo><msub id="S2.SS5.p3.1.m1.3.3.2.2" xref="S2.SS5.p3.1.m1.3.3.2.2.cmml"><mi id="S2.SS5.p3.1.m1.3.3.2.2.2" xref="S2.SS5.p3.1.m1.3.3.2.2.2.cmml">G</mi><mi id="S2.SS5.p3.1.m1.3.3.2.2.3" xref="S2.SS5.p3.1.m1.3.3.2.2.3.cmml">b</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p3.1.m1.3b"><list id="S2.SS5.p3.1.m1.3.3.3.cmml" xref="S2.SS5.p3.1.m1.3.3.2"><apply id="S2.SS5.p3.1.m1.2.2.1.1.cmml" xref="S2.SS5.p3.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS5.p3.1.m1.2.2.1.1.1.cmml" xref="S2.SS5.p3.1.m1.2.2.1.1">subscript</csymbol><ci id="S2.SS5.p3.1.m1.2.2.1.1.2.cmml" xref="S2.SS5.p3.1.m1.2.2.1.1.2">𝐺</ci><cn type="integer" id="S2.SS5.p3.1.m1.2.2.1.1.3.cmml" xref="S2.SS5.p3.1.m1.2.2.1.1.3">1</cn></apply><ci id="S2.SS5.p3.1.m1.1.1.cmml" xref="S2.SS5.p3.1.m1.1.1">…</ci><apply id="S2.SS5.p3.1.m1.3.3.2.2.cmml" xref="S2.SS5.p3.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S2.SS5.p3.1.m1.3.3.2.2.1.cmml" xref="S2.SS5.p3.1.m1.3.3.2.2">subscript</csymbol><ci id="S2.SS5.p3.1.m1.3.3.2.2.2.cmml" xref="S2.SS5.p3.1.m1.3.3.2.2.2">𝐺</ci><ci id="S2.SS5.p3.1.m1.3.3.2.2.3.cmml" xref="S2.SS5.p3.1.m1.3.3.2.2.3">𝑏</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p3.1.m1.3c">G_{1},\dots,G_{b}</annotation></semantics></math> as long as we can compute <math id="S2.SS5.p3.2.m2.1" class="ltx_Math" alttext="W_{b}" display="inline"><semantics id="S2.SS5.p3.2.m2.1a"><msub id="S2.SS5.p3.2.m2.1.1" xref="S2.SS5.p3.2.m2.1.1.cmml"><mi id="S2.SS5.p3.2.m2.1.1.2" xref="S2.SS5.p3.2.m2.1.1.2.cmml">W</mi><mi id="S2.SS5.p3.2.m2.1.1.3" xref="S2.SS5.p3.2.m2.1.1.3.cmml">b</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS5.p3.2.m2.1b"><apply id="S2.SS5.p3.2.m2.1.1.cmml" xref="S2.SS5.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS5.p3.2.m2.1.1.1.cmml" xref="S2.SS5.p3.2.m2.1.1">subscript</csymbol><ci id="S2.SS5.p3.2.m2.1.1.2.cmml" xref="S2.SS5.p3.2.m2.1.1.2">𝑊</ci><ci id="S2.SS5.p3.2.m2.1.1.3.cmml" xref="S2.SS5.p3.2.m2.1.1.3">𝑏</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p3.2.m2.1c">W_{b}</annotation></semantics></math> at the end of the mini-batch, and the output tokens <math id="S2.SS5.p3.3.m3.3" class="ltx_Math" alttext="z_{1},\dots,z_{b}" display="inline"><semantics id="S2.SS5.p3.3.m3.3a"><mrow id="S2.SS5.p3.3.m3.3.3.2" xref="S2.SS5.p3.3.m3.3.3.3.cmml"><msub id="S2.SS5.p3.3.m3.2.2.1.1" xref="S2.SS5.p3.3.m3.2.2.1.1.cmml"><mi id="S2.SS5.p3.3.m3.2.2.1.1.2" xref="S2.SS5.p3.3.m3.2.2.1.1.2.cmml">z</mi><mn id="S2.SS5.p3.3.m3.2.2.1.1.3" xref="S2.SS5.p3.3.m3.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.SS5.p3.3.m3.3.3.2.3" xref="S2.SS5.p3.3.m3.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS5.p3.3.m3.1.1" xref="S2.SS5.p3.3.m3.1.1.cmml">…</mi><mo id="S2.SS5.p3.3.m3.3.3.2.4" xref="S2.SS5.p3.3.m3.3.3.3.cmml">,</mo><msub id="S2.SS5.p3.3.m3.3.3.2.2" xref="S2.SS5.p3.3.m3.3.3.2.2.cmml"><mi id="S2.SS5.p3.3.m3.3.3.2.2.2" xref="S2.SS5.p3.3.m3.3.3.2.2.2.cmml">z</mi><mi id="S2.SS5.p3.3.m3.3.3.2.2.3" xref="S2.SS5.p3.3.m3.3.3.2.2.3.cmml">b</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p3.3.m3.3b"><list id="S2.SS5.p3.3.m3.3.3.3.cmml" xref="S2.SS5.p3.3.m3.3.3.2"><apply id="S2.SS5.p3.3.m3.2.2.1.1.cmml" xref="S2.SS5.p3.3.m3.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS5.p3.3.m3.2.2.1.1.1.cmml" xref="S2.SS5.p3.3.m3.2.2.1.1">subscript</csymbol><ci id="S2.SS5.p3.3.m3.2.2.1.1.2.cmml" xref="S2.SS5.p3.3.m3.2.2.1.1.2">𝑧</ci><cn type="integer" id="S2.SS5.p3.3.m3.2.2.1.1.3.cmml" xref="S2.SS5.p3.3.m3.2.2.1.1.3">1</cn></apply><ci id="S2.SS5.p3.3.m3.1.1.cmml" xref="S2.SS5.p3.3.m3.1.1">…</ci><apply id="S2.SS5.p3.3.m3.3.3.2.2.cmml" xref="S2.SS5.p3.3.m3.3.3.2.2"><csymbol cd="ambiguous" id="S2.SS5.p3.3.m3.3.3.2.2.1.cmml" xref="S2.SS5.p3.3.m3.3.3.2.2">subscript</csymbol><ci id="S2.SS5.p3.3.m3.3.3.2.2.2.cmml" xref="S2.SS5.p3.3.m3.3.3.2.2.2">𝑧</ci><ci id="S2.SS5.p3.3.m3.3.3.2.2.3.cmml" xref="S2.SS5.p3.3.m3.3.3.2.2.3">𝑏</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p3.3.m3.3c">z_{1},\dots,z_{b}</annotation></semantics></math> (see Figure <a href="#S2.F7" title="Figure 7 ‣ 2.4 Parallelization with mini-batch TTT ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>).
Now we demonstrate these computations with the simplified TTT-Linear case above.
Denote <math id="S2.SS5.p3.4.m4.3" class="ltx_Math" alttext="X=[x_{1},\dots,x_{b}]" display="inline"><semantics id="S2.SS5.p3.4.m4.3a"><mrow id="S2.SS5.p3.4.m4.3.3" xref="S2.SS5.p3.4.m4.3.3.cmml"><mi id="S2.SS5.p3.4.m4.3.3.4" xref="S2.SS5.p3.4.m4.3.3.4.cmml">X</mi><mo id="S2.SS5.p3.4.m4.3.3.3" xref="S2.SS5.p3.4.m4.3.3.3.cmml">=</mo><mrow id="S2.SS5.p3.4.m4.3.3.2.2" xref="S2.SS5.p3.4.m4.3.3.2.3.cmml"><mo stretchy="false" id="S2.SS5.p3.4.m4.3.3.2.2.3" xref="S2.SS5.p3.4.m4.3.3.2.3.cmml">[</mo><msub id="S2.SS5.p3.4.m4.2.2.1.1.1" xref="S2.SS5.p3.4.m4.2.2.1.1.1.cmml"><mi id="S2.SS5.p3.4.m4.2.2.1.1.1.2" xref="S2.SS5.p3.4.m4.2.2.1.1.1.2.cmml">x</mi><mn id="S2.SS5.p3.4.m4.2.2.1.1.1.3" xref="S2.SS5.p3.4.m4.2.2.1.1.1.3.cmml">1</mn></msub><mo id="S2.SS5.p3.4.m4.3.3.2.2.4" xref="S2.SS5.p3.4.m4.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS5.p3.4.m4.1.1" xref="S2.SS5.p3.4.m4.1.1.cmml">…</mi><mo id="S2.SS5.p3.4.m4.3.3.2.2.5" xref="S2.SS5.p3.4.m4.3.3.2.3.cmml">,</mo><msub id="S2.SS5.p3.4.m4.3.3.2.2.2" xref="S2.SS5.p3.4.m4.3.3.2.2.2.cmml"><mi id="S2.SS5.p3.4.m4.3.3.2.2.2.2" xref="S2.SS5.p3.4.m4.3.3.2.2.2.2.cmml">x</mi><mi id="S2.SS5.p3.4.m4.3.3.2.2.2.3" xref="S2.SS5.p3.4.m4.3.3.2.2.2.3.cmml">b</mi></msub><mo stretchy="false" id="S2.SS5.p3.4.m4.3.3.2.2.6" xref="S2.SS5.p3.4.m4.3.3.2.3.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p3.4.m4.3b"><apply id="S2.SS5.p3.4.m4.3.3.cmml" xref="S2.SS5.p3.4.m4.3.3"><eq id="S2.SS5.p3.4.m4.3.3.3.cmml" xref="S2.SS5.p3.4.m4.3.3.3"></eq><ci id="S2.SS5.p3.4.m4.3.3.4.cmml" xref="S2.SS5.p3.4.m4.3.3.4">𝑋</ci><list id="S2.SS5.p3.4.m4.3.3.2.3.cmml" xref="S2.SS5.p3.4.m4.3.3.2.2"><apply id="S2.SS5.p3.4.m4.2.2.1.1.1.cmml" xref="S2.SS5.p3.4.m4.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.SS5.p3.4.m4.2.2.1.1.1.1.cmml" xref="S2.SS5.p3.4.m4.2.2.1.1.1">subscript</csymbol><ci id="S2.SS5.p3.4.m4.2.2.1.1.1.2.cmml" xref="S2.SS5.p3.4.m4.2.2.1.1.1.2">𝑥</ci><cn type="integer" id="S2.SS5.p3.4.m4.2.2.1.1.1.3.cmml" xref="S2.SS5.p3.4.m4.2.2.1.1.1.3">1</cn></apply><ci id="S2.SS5.p3.4.m4.1.1.cmml" xref="S2.SS5.p3.4.m4.1.1">…</ci><apply id="S2.SS5.p3.4.m4.3.3.2.2.2.cmml" xref="S2.SS5.p3.4.m4.3.3.2.2.2"><csymbol cd="ambiguous" id="S2.SS5.p3.4.m4.3.3.2.2.2.1.cmml" xref="S2.SS5.p3.4.m4.3.3.2.2.2">subscript</csymbol><ci id="S2.SS5.p3.4.m4.3.3.2.2.2.2.cmml" xref="S2.SS5.p3.4.m4.3.3.2.2.2.2">𝑥</ci><ci id="S2.SS5.p3.4.m4.3.3.2.2.2.3.cmml" xref="S2.SS5.p3.4.m4.3.3.2.2.2.3">𝑏</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p3.4.m4.3c">X=[x_{1},\dots,x_{b}]</annotation></semantics></math>, then:</p>
<table id="S2.Ex3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.Ex3.m1.1" class="ltx_Math" alttext="W_{b}=W_{0}-\eta\sum_{t=1}^{b}G_{t}=W_{0}-2\eta\sum_{t=1}^{b}(W_{0}x_{t}-x_{t})x^{T}=W_{0}-2\eta(W_{0}X-X)X^{T}." display="block"><semantics id="S2.Ex3.m1.1a"><mrow id="S2.Ex3.m1.1.1.1" xref="S2.Ex3.m1.1.1.1.1.cmml"><mrow id="S2.Ex3.m1.1.1.1.1" xref="S2.Ex3.m1.1.1.1.1.cmml"><msub id="S2.Ex3.m1.1.1.1.1.4" xref="S2.Ex3.m1.1.1.1.1.4.cmml"><mi id="S2.Ex3.m1.1.1.1.1.4.2" xref="S2.Ex3.m1.1.1.1.1.4.2.cmml">W</mi><mi id="S2.Ex3.m1.1.1.1.1.4.3" xref="S2.Ex3.m1.1.1.1.1.4.3.cmml">b</mi></msub><mo id="S2.Ex3.m1.1.1.1.1.5" xref="S2.Ex3.m1.1.1.1.1.5.cmml">=</mo><mrow id="S2.Ex3.m1.1.1.1.1.6" xref="S2.Ex3.m1.1.1.1.1.6.cmml"><msub id="S2.Ex3.m1.1.1.1.1.6.2" xref="S2.Ex3.m1.1.1.1.1.6.2.cmml"><mi id="S2.Ex3.m1.1.1.1.1.6.2.2" xref="S2.Ex3.m1.1.1.1.1.6.2.2.cmml">W</mi><mn id="S2.Ex3.m1.1.1.1.1.6.2.3" xref="S2.Ex3.m1.1.1.1.1.6.2.3.cmml">0</mn></msub><mo id="S2.Ex3.m1.1.1.1.1.6.1" xref="S2.Ex3.m1.1.1.1.1.6.1.cmml">−</mo><mrow id="S2.Ex3.m1.1.1.1.1.6.3" xref="S2.Ex3.m1.1.1.1.1.6.3.cmml"><mi id="S2.Ex3.m1.1.1.1.1.6.3.2" xref="S2.Ex3.m1.1.1.1.1.6.3.2.cmml">η</mi><mo lspace="0em" rspace="0em" id="S2.Ex3.m1.1.1.1.1.6.3.1" xref="S2.Ex3.m1.1.1.1.1.6.3.1.cmml">​</mo><mrow id="S2.Ex3.m1.1.1.1.1.6.3.3" xref="S2.Ex3.m1.1.1.1.1.6.3.3.cmml"><munderover id="S2.Ex3.m1.1.1.1.1.6.3.3.1" xref="S2.Ex3.m1.1.1.1.1.6.3.3.1.cmml"><mo movablelimits="false" id="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.2" xref="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.2.cmml">∑</mo><mrow id="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.3" xref="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.3.cmml"><mi id="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.3.2" xref="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.3.2.cmml">t</mi><mo id="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.3.1" xref="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.3.1.cmml">=</mo><mn id="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.3.3" xref="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.3.3.cmml">1</mn></mrow><mi id="S2.Ex3.m1.1.1.1.1.6.3.3.1.3" xref="S2.Ex3.m1.1.1.1.1.6.3.3.1.3.cmml">b</mi></munderover><msub id="S2.Ex3.m1.1.1.1.1.6.3.3.2" xref="S2.Ex3.m1.1.1.1.1.6.3.3.2.cmml"><mi id="S2.Ex3.m1.1.1.1.1.6.3.3.2.2" xref="S2.Ex3.m1.1.1.1.1.6.3.3.2.2.cmml">G</mi><mi id="S2.Ex3.m1.1.1.1.1.6.3.3.2.3" xref="S2.Ex3.m1.1.1.1.1.6.3.3.2.3.cmml">t</mi></msub></mrow></mrow></mrow><mo id="S2.Ex3.m1.1.1.1.1.7" xref="S2.Ex3.m1.1.1.1.1.7.cmml">=</mo><mrow id="S2.Ex3.m1.1.1.1.1.1" xref="S2.Ex3.m1.1.1.1.1.1.cmml"><msub id="S2.Ex3.m1.1.1.1.1.1.3" xref="S2.Ex3.m1.1.1.1.1.1.3.cmml"><mi id="S2.Ex3.m1.1.1.1.1.1.3.2" xref="S2.Ex3.m1.1.1.1.1.1.3.2.cmml">W</mi><mn id="S2.Ex3.m1.1.1.1.1.1.3.3" xref="S2.Ex3.m1.1.1.1.1.1.3.3.cmml">0</mn></msub><mo id="S2.Ex3.m1.1.1.1.1.1.2" xref="S2.Ex3.m1.1.1.1.1.1.2.cmml">−</mo><mrow id="S2.Ex3.m1.1.1.1.1.1.1" xref="S2.Ex3.m1.1.1.1.1.1.1.cmml"><mn id="S2.Ex3.m1.1.1.1.1.1.1.3" xref="S2.Ex3.m1.1.1.1.1.1.1.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="S2.Ex3.m1.1.1.1.1.1.1.2" xref="S2.Ex3.m1.1.1.1.1.1.1.2.cmml">​</mo><mi id="S2.Ex3.m1.1.1.1.1.1.1.4" xref="S2.Ex3.m1.1.1.1.1.1.1.4.cmml">η</mi><mo lspace="0em" rspace="0em" id="S2.Ex3.m1.1.1.1.1.1.1.2a" xref="S2.Ex3.m1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S2.Ex3.m1.1.1.1.1.1.1.1" xref="S2.Ex3.m1.1.1.1.1.1.1.1.cmml"><munderover id="S2.Ex3.m1.1.1.1.1.1.1.1.2" xref="S2.Ex3.m1.1.1.1.1.1.1.1.2.cmml"><mo movablelimits="false" rspace="0em" id="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.2" xref="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.2.cmml">∑</mo><mrow id="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.3" xref="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.3.cmml"><mi id="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.3.2" xref="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.3.2.cmml">t</mi><mo id="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.3.1" xref="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.3.1.cmml">=</mo><mn id="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.3.3" xref="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.3.3.cmml">1</mn></mrow><mi id="S2.Ex3.m1.1.1.1.1.1.1.1.2.3" xref="S2.Ex3.m1.1.1.1.1.1.1.1.2.3.cmml">b</mi></munderover><mrow id="S2.Ex3.m1.1.1.1.1.1.1.1.1" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.cmml"><mrow id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.2" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.cmml"><msub id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.2" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml"><mi id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.2" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.2.cmml">W</mi><mn id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.3" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.3.cmml">0</mn></msub><mo lspace="0em" rspace="0em" id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.1" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml">​</mo><msub id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.3" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml"><mi id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.3.2" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.3.2.cmml">x</mi><mi id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.3.3" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.3.3.cmml">t</mi></msub></mrow><mo id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">−</mo><msub id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.3.2" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml">x</mi><mi id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.3.3" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml">t</mi></msub></mrow><mo stretchy="false" id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.3" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S2.Ex3.m1.1.1.1.1.1.1.1.1.2" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.2.cmml">​</mo><msup id="S2.Ex3.m1.1.1.1.1.1.1.1.1.3" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S2.Ex3.m1.1.1.1.1.1.1.1.1.3.2" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.3.2.cmml">x</mi><mi id="S2.Ex3.m1.1.1.1.1.1.1.1.1.3.3" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.3.3.cmml">T</mi></msup></mrow></mrow></mrow></mrow><mo id="S2.Ex3.m1.1.1.1.1.8" xref="S2.Ex3.m1.1.1.1.1.8.cmml">=</mo><mrow id="S2.Ex3.m1.1.1.1.1.2" xref="S2.Ex3.m1.1.1.1.1.2.cmml"><msub id="S2.Ex3.m1.1.1.1.1.2.3" xref="S2.Ex3.m1.1.1.1.1.2.3.cmml"><mi id="S2.Ex3.m1.1.1.1.1.2.3.2" xref="S2.Ex3.m1.1.1.1.1.2.3.2.cmml">W</mi><mn id="S2.Ex3.m1.1.1.1.1.2.3.3" xref="S2.Ex3.m1.1.1.1.1.2.3.3.cmml">0</mn></msub><mo id="S2.Ex3.m1.1.1.1.1.2.2" xref="S2.Ex3.m1.1.1.1.1.2.2.cmml">−</mo><mrow id="S2.Ex3.m1.1.1.1.1.2.1" xref="S2.Ex3.m1.1.1.1.1.2.1.cmml"><mn id="S2.Ex3.m1.1.1.1.1.2.1.3" xref="S2.Ex3.m1.1.1.1.1.2.1.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="S2.Ex3.m1.1.1.1.1.2.1.2" xref="S2.Ex3.m1.1.1.1.1.2.1.2.cmml">​</mo><mi id="S2.Ex3.m1.1.1.1.1.2.1.4" xref="S2.Ex3.m1.1.1.1.1.2.1.4.cmml">η</mi><mo lspace="0em" rspace="0em" id="S2.Ex3.m1.1.1.1.1.2.1.2a" xref="S2.Ex3.m1.1.1.1.1.2.1.2.cmml">​</mo><mrow id="S2.Ex3.m1.1.1.1.1.2.1.1.1" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.cmml"><mo stretchy="false" id="S2.Ex3.m1.1.1.1.1.2.1.1.1.2" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.cmml">(</mo><mrow id="S2.Ex3.m1.1.1.1.1.2.1.1.1.1" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.cmml"><mrow id="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.cmml"><msub id="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.2" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.2.cmml"><mi id="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.2.2" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.2.2.cmml">W</mi><mn id="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.2.3" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.2.3.cmml">0</mn></msub><mo lspace="0em" rspace="0em" id="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.1" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.1.cmml">​</mo><mi id="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.3" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.3.cmml">X</mi></mrow><mo id="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.1" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.1.cmml">−</mo><mi id="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.3" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.3.cmml">X</mi></mrow><mo stretchy="false" id="S2.Ex3.m1.1.1.1.1.2.1.1.1.3" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S2.Ex3.m1.1.1.1.1.2.1.2b" xref="S2.Ex3.m1.1.1.1.1.2.1.2.cmml">​</mo><msup id="S2.Ex3.m1.1.1.1.1.2.1.5" xref="S2.Ex3.m1.1.1.1.1.2.1.5.cmml"><mi id="S2.Ex3.m1.1.1.1.1.2.1.5.2" xref="S2.Ex3.m1.1.1.1.1.2.1.5.2.cmml">X</mi><mi id="S2.Ex3.m1.1.1.1.1.2.1.5.3" xref="S2.Ex3.m1.1.1.1.1.2.1.5.3.cmml">T</mi></msup></mrow></mrow></mrow><mo lspace="0em" id="S2.Ex3.m1.1.1.1.2" xref="S2.Ex3.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.Ex3.m1.1b"><apply id="S2.Ex3.m1.1.1.1.1.cmml" xref="S2.Ex3.m1.1.1.1"><and id="S2.Ex3.m1.1.1.1.1a.cmml" xref="S2.Ex3.m1.1.1.1"></and><apply id="S2.Ex3.m1.1.1.1.1b.cmml" xref="S2.Ex3.m1.1.1.1"><eq id="S2.Ex3.m1.1.1.1.1.5.cmml" xref="S2.Ex3.m1.1.1.1.1.5"></eq><apply id="S2.Ex3.m1.1.1.1.1.4.cmml" xref="S2.Ex3.m1.1.1.1.1.4"><csymbol cd="ambiguous" id="S2.Ex3.m1.1.1.1.1.4.1.cmml" xref="S2.Ex3.m1.1.1.1.1.4">subscript</csymbol><ci id="S2.Ex3.m1.1.1.1.1.4.2.cmml" xref="S2.Ex3.m1.1.1.1.1.4.2">𝑊</ci><ci id="S2.Ex3.m1.1.1.1.1.4.3.cmml" xref="S2.Ex3.m1.1.1.1.1.4.3">𝑏</ci></apply><apply id="S2.Ex3.m1.1.1.1.1.6.cmml" xref="S2.Ex3.m1.1.1.1.1.6"><minus id="S2.Ex3.m1.1.1.1.1.6.1.cmml" xref="S2.Ex3.m1.1.1.1.1.6.1"></minus><apply id="S2.Ex3.m1.1.1.1.1.6.2.cmml" xref="S2.Ex3.m1.1.1.1.1.6.2"><csymbol cd="ambiguous" id="S2.Ex3.m1.1.1.1.1.6.2.1.cmml" xref="S2.Ex3.m1.1.1.1.1.6.2">subscript</csymbol><ci id="S2.Ex3.m1.1.1.1.1.6.2.2.cmml" xref="S2.Ex3.m1.1.1.1.1.6.2.2">𝑊</ci><cn type="integer" id="S2.Ex3.m1.1.1.1.1.6.2.3.cmml" xref="S2.Ex3.m1.1.1.1.1.6.2.3">0</cn></apply><apply id="S2.Ex3.m1.1.1.1.1.6.3.cmml" xref="S2.Ex3.m1.1.1.1.1.6.3"><times id="S2.Ex3.m1.1.1.1.1.6.3.1.cmml" xref="S2.Ex3.m1.1.1.1.1.6.3.1"></times><ci id="S2.Ex3.m1.1.1.1.1.6.3.2.cmml" xref="S2.Ex3.m1.1.1.1.1.6.3.2">𝜂</ci><apply id="S2.Ex3.m1.1.1.1.1.6.3.3.cmml" xref="S2.Ex3.m1.1.1.1.1.6.3.3"><apply id="S2.Ex3.m1.1.1.1.1.6.3.3.1.cmml" xref="S2.Ex3.m1.1.1.1.1.6.3.3.1"><csymbol cd="ambiguous" id="S2.Ex3.m1.1.1.1.1.6.3.3.1.1.cmml" xref="S2.Ex3.m1.1.1.1.1.6.3.3.1">superscript</csymbol><apply id="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.cmml" xref="S2.Ex3.m1.1.1.1.1.6.3.3.1"><csymbol cd="ambiguous" id="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.1.cmml" xref="S2.Ex3.m1.1.1.1.1.6.3.3.1">subscript</csymbol><sum id="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.2.cmml" xref="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.2"></sum><apply id="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.3.cmml" xref="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.3"><eq id="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.3.1.cmml" xref="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.3.1"></eq><ci id="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.3.2.cmml" xref="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.3.2">𝑡</ci><cn type="integer" id="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.3.3.cmml" xref="S2.Ex3.m1.1.1.1.1.6.3.3.1.2.3.3">1</cn></apply></apply><ci id="S2.Ex3.m1.1.1.1.1.6.3.3.1.3.cmml" xref="S2.Ex3.m1.1.1.1.1.6.3.3.1.3">𝑏</ci></apply><apply id="S2.Ex3.m1.1.1.1.1.6.3.3.2.cmml" xref="S2.Ex3.m1.1.1.1.1.6.3.3.2"><csymbol cd="ambiguous" id="S2.Ex3.m1.1.1.1.1.6.3.3.2.1.cmml" xref="S2.Ex3.m1.1.1.1.1.6.3.3.2">subscript</csymbol><ci id="S2.Ex3.m1.1.1.1.1.6.3.3.2.2.cmml" xref="S2.Ex3.m1.1.1.1.1.6.3.3.2.2">𝐺</ci><ci id="S2.Ex3.m1.1.1.1.1.6.3.3.2.3.cmml" xref="S2.Ex3.m1.1.1.1.1.6.3.3.2.3">𝑡</ci></apply></apply></apply></apply></apply><apply id="S2.Ex3.m1.1.1.1.1c.cmml" xref="S2.Ex3.m1.1.1.1"><eq id="S2.Ex3.m1.1.1.1.1.7.cmml" xref="S2.Ex3.m1.1.1.1.1.7"></eq><share href="#S2.Ex3.m1.1.1.1.1.6.cmml" id="S2.Ex3.m1.1.1.1.1d.cmml" xref="S2.Ex3.m1.1.1.1"></share><apply id="S2.Ex3.m1.1.1.1.1.1.cmml" xref="S2.Ex3.m1.1.1.1.1.1"><minus id="S2.Ex3.m1.1.1.1.1.1.2.cmml" xref="S2.Ex3.m1.1.1.1.1.1.2"></minus><apply id="S2.Ex3.m1.1.1.1.1.1.3.cmml" xref="S2.Ex3.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex3.m1.1.1.1.1.1.3.1.cmml" xref="S2.Ex3.m1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.Ex3.m1.1.1.1.1.1.3.2.cmml" xref="S2.Ex3.m1.1.1.1.1.1.3.2">𝑊</ci><cn type="integer" id="S2.Ex3.m1.1.1.1.1.1.3.3.cmml" xref="S2.Ex3.m1.1.1.1.1.1.3.3">0</cn></apply><apply id="S2.Ex3.m1.1.1.1.1.1.1.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1"><times id="S2.Ex3.m1.1.1.1.1.1.1.2.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.2"></times><cn type="integer" id="S2.Ex3.m1.1.1.1.1.1.1.3.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.3">2</cn><ci id="S2.Ex3.m1.1.1.1.1.1.1.4.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.4">𝜂</ci><apply id="S2.Ex3.m1.1.1.1.1.1.1.1.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1"><apply id="S2.Ex3.m1.1.1.1.1.1.1.1.2.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.Ex3.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.2">superscript</csymbol><apply id="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.1.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.2">subscript</csymbol><sum id="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.2.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.2"></sum><apply id="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.3.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.3"><eq id="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.3.1.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.3.1"></eq><ci id="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.3.2.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.3.2">𝑡</ci><cn type="integer" id="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.3.3.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.2.2.3.3">1</cn></apply></apply><ci id="S2.Ex3.m1.1.1.1.1.1.1.1.2.3.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.2.3">𝑏</ci></apply><apply id="S2.Ex3.m1.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1"><times id="S2.Ex3.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.2"></times><apply id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1"><minus id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.1"></minus><apply id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2"><times id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.1"></times><apply id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.1.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.2">subscript</csymbol><ci id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.2.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.2">𝑊</ci><cn type="integer" id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.3.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.3">0</cn></apply><apply id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.3.1.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.3">subscript</csymbol><ci id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.3.2.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.3.2">𝑥</ci><ci id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.3.3.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.2.3.3">𝑡</ci></apply></apply><apply id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.3.2">𝑥</ci><ci id="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.1.1.1.3.3">𝑡</ci></apply></apply><apply id="S2.Ex3.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex3.m1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.3">superscript</csymbol><ci id="S2.Ex3.m1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.3.2">𝑥</ci><ci id="S2.Ex3.m1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S2.Ex3.m1.1.1.1.1.1.1.1.1.3.3">𝑇</ci></apply></apply></apply></apply></apply></apply><apply id="S2.Ex3.m1.1.1.1.1e.cmml" xref="S2.Ex3.m1.1.1.1"><eq id="S2.Ex3.m1.1.1.1.1.8.cmml" xref="S2.Ex3.m1.1.1.1.1.8"></eq><share href="#S2.Ex3.m1.1.1.1.1.1.cmml" id="S2.Ex3.m1.1.1.1.1f.cmml" xref="S2.Ex3.m1.1.1.1"></share><apply id="S2.Ex3.m1.1.1.1.1.2.cmml" xref="S2.Ex3.m1.1.1.1.1.2"><minus id="S2.Ex3.m1.1.1.1.1.2.2.cmml" xref="S2.Ex3.m1.1.1.1.1.2.2"></minus><apply id="S2.Ex3.m1.1.1.1.1.2.3.cmml" xref="S2.Ex3.m1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S2.Ex3.m1.1.1.1.1.2.3.1.cmml" xref="S2.Ex3.m1.1.1.1.1.2.3">subscript</csymbol><ci id="S2.Ex3.m1.1.1.1.1.2.3.2.cmml" xref="S2.Ex3.m1.1.1.1.1.2.3.2">𝑊</ci><cn type="integer" id="S2.Ex3.m1.1.1.1.1.2.3.3.cmml" xref="S2.Ex3.m1.1.1.1.1.2.3.3">0</cn></apply><apply id="S2.Ex3.m1.1.1.1.1.2.1.cmml" xref="S2.Ex3.m1.1.1.1.1.2.1"><times id="S2.Ex3.m1.1.1.1.1.2.1.2.cmml" xref="S2.Ex3.m1.1.1.1.1.2.1.2"></times><cn type="integer" id="S2.Ex3.m1.1.1.1.1.2.1.3.cmml" xref="S2.Ex3.m1.1.1.1.1.2.1.3">2</cn><ci id="S2.Ex3.m1.1.1.1.1.2.1.4.cmml" xref="S2.Ex3.m1.1.1.1.1.2.1.4">𝜂</ci><apply id="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.cmml" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1"><minus id="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.1.cmml" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.1"></minus><apply id="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.cmml" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2"><times id="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.1.cmml" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.1"></times><apply id="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.2.cmml" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.2.1.cmml" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.2">subscript</csymbol><ci id="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.2.2.cmml" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.2.2">𝑊</ci><cn type="integer" id="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.2.3.cmml" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.2.3">0</cn></apply><ci id="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.3.cmml" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.2.3">𝑋</ci></apply><ci id="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.3.cmml" xref="S2.Ex3.m1.1.1.1.1.2.1.1.1.1.3">𝑋</ci></apply><apply id="S2.Ex3.m1.1.1.1.1.2.1.5.cmml" xref="S2.Ex3.m1.1.1.1.1.2.1.5"><csymbol cd="ambiguous" id="S2.Ex3.m1.1.1.1.1.2.1.5.1.cmml" xref="S2.Ex3.m1.1.1.1.1.2.1.5">superscript</csymbol><ci id="S2.Ex3.m1.1.1.1.1.2.1.5.2.cmml" xref="S2.Ex3.m1.1.1.1.1.2.1.5.2">𝑋</ci><ci id="S2.Ex3.m1.1.1.1.1.2.1.5.3.cmml" xref="S2.Ex3.m1.1.1.1.1.2.1.5.3">𝑇</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex3.m1.1c">W_{b}=W_{0}-\eta\sum_{t=1}^{b}G_{t}=W_{0}-2\eta\sum_{t=1}^{b}(W_{0}x_{t}-x_{t})x^{T}=W_{0}-2\eta(W_{0}X-X)X^{T}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S2.SS5.p3.6" class="ltx_p">So <math id="S2.SS5.p3.5.m1.1" class="ltx_Math" alttext="W_{b}" display="inline"><semantics id="S2.SS5.p3.5.m1.1a"><msub id="S2.SS5.p3.5.m1.1.1" xref="S2.SS5.p3.5.m1.1.1.cmml"><mi id="S2.SS5.p3.5.m1.1.1.2" xref="S2.SS5.p3.5.m1.1.1.2.cmml">W</mi><mi id="S2.SS5.p3.5.m1.1.1.3" xref="S2.SS5.p3.5.m1.1.1.3.cmml">b</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS5.p3.5.m1.1b"><apply id="S2.SS5.p3.5.m1.1.1.cmml" xref="S2.SS5.p3.5.m1.1.1"><csymbol cd="ambiguous" id="S2.SS5.p3.5.m1.1.1.1.cmml" xref="S2.SS5.p3.5.m1.1.1">subscript</csymbol><ci id="S2.SS5.p3.5.m1.1.1.2.cmml" xref="S2.SS5.p3.5.m1.1.1.2">𝑊</ci><ci id="S2.SS5.p3.5.m1.1.1.3.cmml" xref="S2.SS5.p3.5.m1.1.1.3">𝑏</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p3.5.m1.1c">W_{b}</annotation></semantics></math> can be conveniently computed with a <span id="S2.SS5.p3.6.1" class="ltx_text ltx_font_typewriter">matmul</span>.
To compute <math id="S2.SS5.p3.6.m2.3" class="ltx_Math" alttext="Z=[z_{1},\dots,z_{b}]" display="inline"><semantics id="S2.SS5.p3.6.m2.3a"><mrow id="S2.SS5.p3.6.m2.3.3" xref="S2.SS5.p3.6.m2.3.3.cmml"><mi id="S2.SS5.p3.6.m2.3.3.4" xref="S2.SS5.p3.6.m2.3.3.4.cmml">Z</mi><mo id="S2.SS5.p3.6.m2.3.3.3" xref="S2.SS5.p3.6.m2.3.3.3.cmml">=</mo><mrow id="S2.SS5.p3.6.m2.3.3.2.2" xref="S2.SS5.p3.6.m2.3.3.2.3.cmml"><mo stretchy="false" id="S2.SS5.p3.6.m2.3.3.2.2.3" xref="S2.SS5.p3.6.m2.3.3.2.3.cmml">[</mo><msub id="S2.SS5.p3.6.m2.2.2.1.1.1" xref="S2.SS5.p3.6.m2.2.2.1.1.1.cmml"><mi id="S2.SS5.p3.6.m2.2.2.1.1.1.2" xref="S2.SS5.p3.6.m2.2.2.1.1.1.2.cmml">z</mi><mn id="S2.SS5.p3.6.m2.2.2.1.1.1.3" xref="S2.SS5.p3.6.m2.2.2.1.1.1.3.cmml">1</mn></msub><mo id="S2.SS5.p3.6.m2.3.3.2.2.4" xref="S2.SS5.p3.6.m2.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS5.p3.6.m2.1.1" xref="S2.SS5.p3.6.m2.1.1.cmml">…</mi><mo id="S2.SS5.p3.6.m2.3.3.2.2.5" xref="S2.SS5.p3.6.m2.3.3.2.3.cmml">,</mo><msub id="S2.SS5.p3.6.m2.3.3.2.2.2" xref="S2.SS5.p3.6.m2.3.3.2.2.2.cmml"><mi id="S2.SS5.p3.6.m2.3.3.2.2.2.2" xref="S2.SS5.p3.6.m2.3.3.2.2.2.2.cmml">z</mi><mi id="S2.SS5.p3.6.m2.3.3.2.2.2.3" xref="S2.SS5.p3.6.m2.3.3.2.2.2.3.cmml">b</mi></msub><mo stretchy="false" id="S2.SS5.p3.6.m2.3.3.2.2.6" xref="S2.SS5.p3.6.m2.3.3.2.3.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p3.6.m2.3b"><apply id="S2.SS5.p3.6.m2.3.3.cmml" xref="S2.SS5.p3.6.m2.3.3"><eq id="S2.SS5.p3.6.m2.3.3.3.cmml" xref="S2.SS5.p3.6.m2.3.3.3"></eq><ci id="S2.SS5.p3.6.m2.3.3.4.cmml" xref="S2.SS5.p3.6.m2.3.3.4">𝑍</ci><list id="S2.SS5.p3.6.m2.3.3.2.3.cmml" xref="S2.SS5.p3.6.m2.3.3.2.2"><apply id="S2.SS5.p3.6.m2.2.2.1.1.1.cmml" xref="S2.SS5.p3.6.m2.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.SS5.p3.6.m2.2.2.1.1.1.1.cmml" xref="S2.SS5.p3.6.m2.2.2.1.1.1">subscript</csymbol><ci id="S2.SS5.p3.6.m2.2.2.1.1.1.2.cmml" xref="S2.SS5.p3.6.m2.2.2.1.1.1.2">𝑧</ci><cn type="integer" id="S2.SS5.p3.6.m2.2.2.1.1.1.3.cmml" xref="S2.SS5.p3.6.m2.2.2.1.1.1.3">1</cn></apply><ci id="S2.SS5.p3.6.m2.1.1.cmml" xref="S2.SS5.p3.6.m2.1.1">…</ci><apply id="S2.SS5.p3.6.m2.3.3.2.2.2.cmml" xref="S2.SS5.p3.6.m2.3.3.2.2.2"><csymbol cd="ambiguous" id="S2.SS5.p3.6.m2.3.3.2.2.2.1.cmml" xref="S2.SS5.p3.6.m2.3.3.2.2.2">subscript</csymbol><ci id="S2.SS5.p3.6.m2.3.3.2.2.2.2.cmml" xref="S2.SS5.p3.6.m2.3.3.2.2.2.2">𝑧</ci><ci id="S2.SS5.p3.6.m2.3.3.2.2.2.3.cmml" xref="S2.SS5.p3.6.m2.3.3.2.2.2.3">𝑏</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p3.6.m2.3c">Z=[z_{1},\dots,z_{b}]</annotation></semantics></math>, we know that:</p>
<table id="S2.E7" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E7.m1.1" class="ltx_Math" alttext="z_{t}=f(x_{t};W_{t})=W_{t}x_{t}=\left(W_{0}-\eta\sum_{s=1}^{t}G_{t}\right)x_{t}=W_{0}x_{t}-2\eta\sum_{s=1}^{t}(W_{0}x_{s}-x_{s})x_{s}^{T}x_{s}." display="block"><semantics id="S2.E7.m1.1a"><mrow id="S2.E7.m1.1.1.1" xref="S2.E7.m1.1.1.1.1.cmml"><mrow id="S2.E7.m1.1.1.1.1" xref="S2.E7.m1.1.1.1.1.cmml"><msub id="S2.E7.m1.1.1.1.1.6" xref="S2.E7.m1.1.1.1.1.6.cmml"><mi id="S2.E7.m1.1.1.1.1.6.2" xref="S2.E7.m1.1.1.1.1.6.2.cmml">z</mi><mi id="S2.E7.m1.1.1.1.1.6.3" xref="S2.E7.m1.1.1.1.1.6.3.cmml">t</mi></msub><mo id="S2.E7.m1.1.1.1.1.7" xref="S2.E7.m1.1.1.1.1.7.cmml">=</mo><mrow id="S2.E7.m1.1.1.1.1.2" xref="S2.E7.m1.1.1.1.1.2.cmml"><mi id="S2.E7.m1.1.1.1.1.2.4" xref="S2.E7.m1.1.1.1.1.2.4.cmml">f</mi><mo lspace="0em" rspace="0em" id="S2.E7.m1.1.1.1.1.2.3" xref="S2.E7.m1.1.1.1.1.2.3.cmml">​</mo><mrow id="S2.E7.m1.1.1.1.1.2.2.2" xref="S2.E7.m1.1.1.1.1.2.2.3.cmml"><mo stretchy="false" id="S2.E7.m1.1.1.1.1.2.2.2.3" xref="S2.E7.m1.1.1.1.1.2.2.3.cmml">(</mo><msub id="S2.E7.m1.1.1.1.1.1.1.1.1" xref="S2.E7.m1.1.1.1.1.1.1.1.1.cmml"><mi id="S2.E7.m1.1.1.1.1.1.1.1.1.2" xref="S2.E7.m1.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="S2.E7.m1.1.1.1.1.1.1.1.1.3" xref="S2.E7.m1.1.1.1.1.1.1.1.1.3.cmml">t</mi></msub><mo id="S2.E7.m1.1.1.1.1.2.2.2.4" xref="S2.E7.m1.1.1.1.1.2.2.3.cmml">;</mo><msub id="S2.E7.m1.1.1.1.1.2.2.2.2" xref="S2.E7.m1.1.1.1.1.2.2.2.2.cmml"><mi id="S2.E7.m1.1.1.1.1.2.2.2.2.2" xref="S2.E7.m1.1.1.1.1.2.2.2.2.2.cmml">W</mi><mi id="S2.E7.m1.1.1.1.1.2.2.2.2.3" xref="S2.E7.m1.1.1.1.1.2.2.2.2.3.cmml">t</mi></msub><mo stretchy="false" id="S2.E7.m1.1.1.1.1.2.2.2.5" xref="S2.E7.m1.1.1.1.1.2.2.3.cmml">)</mo></mrow></mrow><mo id="S2.E7.m1.1.1.1.1.8" xref="S2.E7.m1.1.1.1.1.8.cmml">=</mo><mrow id="S2.E7.m1.1.1.1.1.9" xref="S2.E7.m1.1.1.1.1.9.cmml"><msub id="S2.E7.m1.1.1.1.1.9.2" xref="S2.E7.m1.1.1.1.1.9.2.cmml"><mi id="S2.E7.m1.1.1.1.1.9.2.2" xref="S2.E7.m1.1.1.1.1.9.2.2.cmml">W</mi><mi id="S2.E7.m1.1.1.1.1.9.2.3" xref="S2.E7.m1.1.1.1.1.9.2.3.cmml">t</mi></msub><mo lspace="0em" rspace="0em" id="S2.E7.m1.1.1.1.1.9.1" xref="S2.E7.m1.1.1.1.1.9.1.cmml">​</mo><msub id="S2.E7.m1.1.1.1.1.9.3" xref="S2.E7.m1.1.1.1.1.9.3.cmml"><mi id="S2.E7.m1.1.1.1.1.9.3.2" xref="S2.E7.m1.1.1.1.1.9.3.2.cmml">x</mi><mi id="S2.E7.m1.1.1.1.1.9.3.3" xref="S2.E7.m1.1.1.1.1.9.3.3.cmml">t</mi></msub></mrow><mo id="S2.E7.m1.1.1.1.1.10" xref="S2.E7.m1.1.1.1.1.10.cmml">=</mo><mrow id="S2.E7.m1.1.1.1.1.3" xref="S2.E7.m1.1.1.1.1.3.cmml"><mrow id="S2.E7.m1.1.1.1.1.3.1.1" xref="S2.E7.m1.1.1.1.1.3.1.1.1.cmml"><mo id="S2.E7.m1.1.1.1.1.3.1.1.2" xref="S2.E7.m1.1.1.1.1.3.1.1.1.cmml">(</mo><mrow id="S2.E7.m1.1.1.1.1.3.1.1.1" xref="S2.E7.m1.1.1.1.1.3.1.1.1.cmml"><msub id="S2.E7.m1.1.1.1.1.3.1.1.1.2" xref="S2.E7.m1.1.1.1.1.3.1.1.1.2.cmml"><mi id="S2.E7.m1.1.1.1.1.3.1.1.1.2.2" xref="S2.E7.m1.1.1.1.1.3.1.1.1.2.2.cmml">W</mi><mn id="S2.E7.m1.1.1.1.1.3.1.1.1.2.3" xref="S2.E7.m1.1.1.1.1.3.1.1.1.2.3.cmml">0</mn></msub><mo id="S2.E7.m1.1.1.1.1.3.1.1.1.1" xref="S2.E7.m1.1.1.1.1.3.1.1.1.1.cmml">−</mo><mrow id="S2.E7.m1.1.1.1.1.3.1.1.1.3" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.cmml"><mi id="S2.E7.m1.1.1.1.1.3.1.1.1.3.2" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.2.cmml">η</mi><mo lspace="0em" rspace="0em" id="S2.E7.m1.1.1.1.1.3.1.1.1.3.1" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.1.cmml">​</mo><mrow id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.cmml"><munderover id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.cmml"><mo movablelimits="false" id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.2" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.2.cmml">∑</mo><mrow id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.3" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.3.cmml"><mi id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.3.2" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.3.2.cmml">s</mi><mo id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.3.1" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.3.1.cmml">=</mo><mn id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.3.3" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.3.3.cmml">1</mn></mrow><mi id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.3" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.3.cmml">t</mi></munderover><msub id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.2" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.2.cmml"><mi id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.2.2" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.2.2.cmml">G</mi><mi id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.2.3" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.2.3.cmml">t</mi></msub></mrow></mrow></mrow><mo id="S2.E7.m1.1.1.1.1.3.1.1.3" xref="S2.E7.m1.1.1.1.1.3.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S2.E7.m1.1.1.1.1.3.2" xref="S2.E7.m1.1.1.1.1.3.2.cmml">​</mo><msub id="S2.E7.m1.1.1.1.1.3.3" xref="S2.E7.m1.1.1.1.1.3.3.cmml"><mi id="S2.E7.m1.1.1.1.1.3.3.2" xref="S2.E7.m1.1.1.1.1.3.3.2.cmml">x</mi><mi id="S2.E7.m1.1.1.1.1.3.3.3" xref="S2.E7.m1.1.1.1.1.3.3.3.cmml">t</mi></msub></mrow><mo id="S2.E7.m1.1.1.1.1.11" xref="S2.E7.m1.1.1.1.1.11.cmml">=</mo><mrow id="S2.E7.m1.1.1.1.1.4" xref="S2.E7.m1.1.1.1.1.4.cmml"><mrow id="S2.E7.m1.1.1.1.1.4.3" xref="S2.E7.m1.1.1.1.1.4.3.cmml"><msub id="S2.E7.m1.1.1.1.1.4.3.2" xref="S2.E7.m1.1.1.1.1.4.3.2.cmml"><mi id="S2.E7.m1.1.1.1.1.4.3.2.2" xref="S2.E7.m1.1.1.1.1.4.3.2.2.cmml">W</mi><mn id="S2.E7.m1.1.1.1.1.4.3.2.3" xref="S2.E7.m1.1.1.1.1.4.3.2.3.cmml">0</mn></msub><mo lspace="0em" rspace="0em" id="S2.E7.m1.1.1.1.1.4.3.1" xref="S2.E7.m1.1.1.1.1.4.3.1.cmml">​</mo><msub id="S2.E7.m1.1.1.1.1.4.3.3" xref="S2.E7.m1.1.1.1.1.4.3.3.cmml"><mi id="S2.E7.m1.1.1.1.1.4.3.3.2" xref="S2.E7.m1.1.1.1.1.4.3.3.2.cmml">x</mi><mi id="S2.E7.m1.1.1.1.1.4.3.3.3" xref="S2.E7.m1.1.1.1.1.4.3.3.3.cmml">t</mi></msub></mrow><mo id="S2.E7.m1.1.1.1.1.4.2" xref="S2.E7.m1.1.1.1.1.4.2.cmml">−</mo><mrow id="S2.E7.m1.1.1.1.1.4.1" xref="S2.E7.m1.1.1.1.1.4.1.cmml"><mn id="S2.E7.m1.1.1.1.1.4.1.3" xref="S2.E7.m1.1.1.1.1.4.1.3.cmml">2</mn><mo lspace="0em" rspace="0em" id="S2.E7.m1.1.1.1.1.4.1.2" xref="S2.E7.m1.1.1.1.1.4.1.2.cmml">​</mo><mi id="S2.E7.m1.1.1.1.1.4.1.4" xref="S2.E7.m1.1.1.1.1.4.1.4.cmml">η</mi><mo lspace="0em" rspace="0em" id="S2.E7.m1.1.1.1.1.4.1.2a" xref="S2.E7.m1.1.1.1.1.4.1.2.cmml">​</mo><mrow id="S2.E7.m1.1.1.1.1.4.1.1" xref="S2.E7.m1.1.1.1.1.4.1.1.cmml"><munderover id="S2.E7.m1.1.1.1.1.4.1.1.2" xref="S2.E7.m1.1.1.1.1.4.1.1.2.cmml"><mo movablelimits="false" rspace="0em" id="S2.E7.m1.1.1.1.1.4.1.1.2.2.2" xref="S2.E7.m1.1.1.1.1.4.1.1.2.2.2.cmml">∑</mo><mrow id="S2.E7.m1.1.1.1.1.4.1.1.2.2.3" xref="S2.E7.m1.1.1.1.1.4.1.1.2.2.3.cmml"><mi id="S2.E7.m1.1.1.1.1.4.1.1.2.2.3.2" xref="S2.E7.m1.1.1.1.1.4.1.1.2.2.3.2.cmml">s</mi><mo id="S2.E7.m1.1.1.1.1.4.1.1.2.2.3.1" xref="S2.E7.m1.1.1.1.1.4.1.1.2.2.3.1.cmml">=</mo><mn id="S2.E7.m1.1.1.1.1.4.1.1.2.2.3.3" xref="S2.E7.m1.1.1.1.1.4.1.1.2.2.3.3.cmml">1</mn></mrow><mi id="S2.E7.m1.1.1.1.1.4.1.1.2.3" xref="S2.E7.m1.1.1.1.1.4.1.1.2.3.cmml">t</mi></munderover><mrow id="S2.E7.m1.1.1.1.1.4.1.1.1" xref="S2.E7.m1.1.1.1.1.4.1.1.1.cmml"><mrow id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.2" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.cmml"><mrow id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.cmml"><msub id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.2" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.2.cmml"><mi id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.2.2" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.2.2.cmml">W</mi><mn id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.2.3" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.2.3.cmml">0</mn></msub><mo lspace="0em" rspace="0em" id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.1" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.1.cmml">​</mo><msub id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.3" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.3.cmml"><mi id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.3.2" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.3.2.cmml">x</mi><mi id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.3.3" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.3.3.cmml">s</mi></msub></mrow><mo id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.1" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.1.cmml">−</mo><msub id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.3" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.3.cmml"><mi id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.3.2" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.3.2.cmml">x</mi><mi id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.3.3" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.3.3.cmml">s</mi></msub></mrow><mo stretchy="false" id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.3" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S2.E7.m1.1.1.1.1.4.1.1.1.2" xref="S2.E7.m1.1.1.1.1.4.1.1.1.2.cmml">​</mo><msubsup id="S2.E7.m1.1.1.1.1.4.1.1.1.3" xref="S2.E7.m1.1.1.1.1.4.1.1.1.3.cmml"><mi id="S2.E7.m1.1.1.1.1.4.1.1.1.3.2.2" xref="S2.E7.m1.1.1.1.1.4.1.1.1.3.2.2.cmml">x</mi><mi id="S2.E7.m1.1.1.1.1.4.1.1.1.3.2.3" xref="S2.E7.m1.1.1.1.1.4.1.1.1.3.2.3.cmml">s</mi><mi id="S2.E7.m1.1.1.1.1.4.1.1.1.3.3" xref="S2.E7.m1.1.1.1.1.4.1.1.1.3.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="S2.E7.m1.1.1.1.1.4.1.1.1.2a" xref="S2.E7.m1.1.1.1.1.4.1.1.1.2.cmml">​</mo><msub id="S2.E7.m1.1.1.1.1.4.1.1.1.4" xref="S2.E7.m1.1.1.1.1.4.1.1.1.4.cmml"><mi id="S2.E7.m1.1.1.1.1.4.1.1.1.4.2" xref="S2.E7.m1.1.1.1.1.4.1.1.1.4.2.cmml">x</mi><mi id="S2.E7.m1.1.1.1.1.4.1.1.1.4.3" xref="S2.E7.m1.1.1.1.1.4.1.1.1.4.3.cmml">s</mi></msub></mrow></mrow></mrow></mrow></mrow><mo lspace="0em" id="S2.E7.m1.1.1.1.2" xref="S2.E7.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E7.m1.1b"><apply id="S2.E7.m1.1.1.1.1.cmml" xref="S2.E7.m1.1.1.1"><and id="S2.E7.m1.1.1.1.1a.cmml" xref="S2.E7.m1.1.1.1"></and><apply id="S2.E7.m1.1.1.1.1b.cmml" xref="S2.E7.m1.1.1.1"><eq id="S2.E7.m1.1.1.1.1.7.cmml" xref="S2.E7.m1.1.1.1.1.7"></eq><apply id="S2.E7.m1.1.1.1.1.6.cmml" xref="S2.E7.m1.1.1.1.1.6"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.1.1.6.1.cmml" xref="S2.E7.m1.1.1.1.1.6">subscript</csymbol><ci id="S2.E7.m1.1.1.1.1.6.2.cmml" xref="S2.E7.m1.1.1.1.1.6.2">𝑧</ci><ci id="S2.E7.m1.1.1.1.1.6.3.cmml" xref="S2.E7.m1.1.1.1.1.6.3">𝑡</ci></apply><apply id="S2.E7.m1.1.1.1.1.2.cmml" xref="S2.E7.m1.1.1.1.1.2"><times id="S2.E7.m1.1.1.1.1.2.3.cmml" xref="S2.E7.m1.1.1.1.1.2.3"></times><ci id="S2.E7.m1.1.1.1.1.2.4.cmml" xref="S2.E7.m1.1.1.1.1.2.4">𝑓</ci><list id="S2.E7.m1.1.1.1.1.2.2.3.cmml" xref="S2.E7.m1.1.1.1.1.2.2.2"><apply id="S2.E7.m1.1.1.1.1.1.1.1.1.cmml" xref="S2.E7.m1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.E7.m1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E7.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.E7.m1.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="S2.E7.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.E7.m1.1.1.1.1.1.1.1.1.3">𝑡</ci></apply><apply id="S2.E7.m1.1.1.1.1.2.2.2.2.cmml" xref="S2.E7.m1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.1.1.2.2.2.2.1.cmml" xref="S2.E7.m1.1.1.1.1.2.2.2.2">subscript</csymbol><ci id="S2.E7.m1.1.1.1.1.2.2.2.2.2.cmml" xref="S2.E7.m1.1.1.1.1.2.2.2.2.2">𝑊</ci><ci id="S2.E7.m1.1.1.1.1.2.2.2.2.3.cmml" xref="S2.E7.m1.1.1.1.1.2.2.2.2.3">𝑡</ci></apply></list></apply></apply><apply id="S2.E7.m1.1.1.1.1c.cmml" xref="S2.E7.m1.1.1.1"><eq id="S2.E7.m1.1.1.1.1.8.cmml" xref="S2.E7.m1.1.1.1.1.8"></eq><share href="#S2.E7.m1.1.1.1.1.2.cmml" id="S2.E7.m1.1.1.1.1d.cmml" xref="S2.E7.m1.1.1.1"></share><apply id="S2.E7.m1.1.1.1.1.9.cmml" xref="S2.E7.m1.1.1.1.1.9"><times id="S2.E7.m1.1.1.1.1.9.1.cmml" xref="S2.E7.m1.1.1.1.1.9.1"></times><apply id="S2.E7.m1.1.1.1.1.9.2.cmml" xref="S2.E7.m1.1.1.1.1.9.2"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.1.1.9.2.1.cmml" xref="S2.E7.m1.1.1.1.1.9.2">subscript</csymbol><ci id="S2.E7.m1.1.1.1.1.9.2.2.cmml" xref="S2.E7.m1.1.1.1.1.9.2.2">𝑊</ci><ci id="S2.E7.m1.1.1.1.1.9.2.3.cmml" xref="S2.E7.m1.1.1.1.1.9.2.3">𝑡</ci></apply><apply id="S2.E7.m1.1.1.1.1.9.3.cmml" xref="S2.E7.m1.1.1.1.1.9.3"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.1.1.9.3.1.cmml" xref="S2.E7.m1.1.1.1.1.9.3">subscript</csymbol><ci id="S2.E7.m1.1.1.1.1.9.3.2.cmml" xref="S2.E7.m1.1.1.1.1.9.3.2">𝑥</ci><ci id="S2.E7.m1.1.1.1.1.9.3.3.cmml" xref="S2.E7.m1.1.1.1.1.9.3.3">𝑡</ci></apply></apply></apply><apply id="S2.E7.m1.1.1.1.1e.cmml" xref="S2.E7.m1.1.1.1"><eq id="S2.E7.m1.1.1.1.1.10.cmml" xref="S2.E7.m1.1.1.1.1.10"></eq><share href="#S2.E7.m1.1.1.1.1.9.cmml" id="S2.E7.m1.1.1.1.1f.cmml" xref="S2.E7.m1.1.1.1"></share><apply id="S2.E7.m1.1.1.1.1.3.cmml" xref="S2.E7.m1.1.1.1.1.3"><times id="S2.E7.m1.1.1.1.1.3.2.cmml" xref="S2.E7.m1.1.1.1.1.3.2"></times><apply id="S2.E7.m1.1.1.1.1.3.1.1.1.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1"><minus id="S2.E7.m1.1.1.1.1.3.1.1.1.1.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.1"></minus><apply id="S2.E7.m1.1.1.1.1.3.1.1.1.2.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.2"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.1.1.3.1.1.1.2.1.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.2">subscript</csymbol><ci id="S2.E7.m1.1.1.1.1.3.1.1.1.2.2.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.2.2">𝑊</ci><cn type="integer" id="S2.E7.m1.1.1.1.1.3.1.1.1.2.3.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.2.3">0</cn></apply><apply id="S2.E7.m1.1.1.1.1.3.1.1.1.3.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3"><times id="S2.E7.m1.1.1.1.1.3.1.1.1.3.1.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.1"></times><ci id="S2.E7.m1.1.1.1.1.3.1.1.1.3.2.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.2">𝜂</ci><apply id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3"><apply id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.1.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1">superscript</csymbol><apply id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.1.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1">subscript</csymbol><sum id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.2.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.2"></sum><apply id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.3.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.3"><eq id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.3.1.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.3.1"></eq><ci id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.3.2.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.3.2">𝑠</ci><cn type="integer" id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.3.3.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.2.3.3">1</cn></apply></apply><ci id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.3.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.1.3">𝑡</ci></apply><apply id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.2.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.2"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.2.1.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.2">subscript</csymbol><ci id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.2.2.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.2.2">𝐺</ci><ci id="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.2.3.cmml" xref="S2.E7.m1.1.1.1.1.3.1.1.1.3.3.2.3">𝑡</ci></apply></apply></apply></apply><apply id="S2.E7.m1.1.1.1.1.3.3.cmml" xref="S2.E7.m1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.1.1.3.3.1.cmml" xref="S2.E7.m1.1.1.1.1.3.3">subscript</csymbol><ci id="S2.E7.m1.1.1.1.1.3.3.2.cmml" xref="S2.E7.m1.1.1.1.1.3.3.2">𝑥</ci><ci id="S2.E7.m1.1.1.1.1.3.3.3.cmml" xref="S2.E7.m1.1.1.1.1.3.3.3">𝑡</ci></apply></apply></apply><apply id="S2.E7.m1.1.1.1.1g.cmml" xref="S2.E7.m1.1.1.1"><eq id="S2.E7.m1.1.1.1.1.11.cmml" xref="S2.E7.m1.1.1.1.1.11"></eq><share href="#S2.E7.m1.1.1.1.1.3.cmml" id="S2.E7.m1.1.1.1.1h.cmml" xref="S2.E7.m1.1.1.1"></share><apply id="S2.E7.m1.1.1.1.1.4.cmml" xref="S2.E7.m1.1.1.1.1.4"><minus id="S2.E7.m1.1.1.1.1.4.2.cmml" xref="S2.E7.m1.1.1.1.1.4.2"></minus><apply id="S2.E7.m1.1.1.1.1.4.3.cmml" xref="S2.E7.m1.1.1.1.1.4.3"><times id="S2.E7.m1.1.1.1.1.4.3.1.cmml" xref="S2.E7.m1.1.1.1.1.4.3.1"></times><apply id="S2.E7.m1.1.1.1.1.4.3.2.cmml" xref="S2.E7.m1.1.1.1.1.4.3.2"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.1.1.4.3.2.1.cmml" xref="S2.E7.m1.1.1.1.1.4.3.2">subscript</csymbol><ci id="S2.E7.m1.1.1.1.1.4.3.2.2.cmml" xref="S2.E7.m1.1.1.1.1.4.3.2.2">𝑊</ci><cn type="integer" id="S2.E7.m1.1.1.1.1.4.3.2.3.cmml" xref="S2.E7.m1.1.1.1.1.4.3.2.3">0</cn></apply><apply id="S2.E7.m1.1.1.1.1.4.3.3.cmml" xref="S2.E7.m1.1.1.1.1.4.3.3"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.1.1.4.3.3.1.cmml" xref="S2.E7.m1.1.1.1.1.4.3.3">subscript</csymbol><ci id="S2.E7.m1.1.1.1.1.4.3.3.2.cmml" xref="S2.E7.m1.1.1.1.1.4.3.3.2">𝑥</ci><ci id="S2.E7.m1.1.1.1.1.4.3.3.3.cmml" xref="S2.E7.m1.1.1.1.1.4.3.3.3">𝑡</ci></apply></apply><apply id="S2.E7.m1.1.1.1.1.4.1.cmml" xref="S2.E7.m1.1.1.1.1.4.1"><times id="S2.E7.m1.1.1.1.1.4.1.2.cmml" xref="S2.E7.m1.1.1.1.1.4.1.2"></times><cn type="integer" id="S2.E7.m1.1.1.1.1.4.1.3.cmml" xref="S2.E7.m1.1.1.1.1.4.1.3">2</cn><ci id="S2.E7.m1.1.1.1.1.4.1.4.cmml" xref="S2.E7.m1.1.1.1.1.4.1.4">𝜂</ci><apply id="S2.E7.m1.1.1.1.1.4.1.1.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1"><apply id="S2.E7.m1.1.1.1.1.4.1.1.2.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.2"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.1.1.4.1.1.2.1.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.2">superscript</csymbol><apply id="S2.E7.m1.1.1.1.1.4.1.1.2.2.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.2"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.1.1.4.1.1.2.2.1.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.2">subscript</csymbol><sum id="S2.E7.m1.1.1.1.1.4.1.1.2.2.2.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.2.2.2"></sum><apply id="S2.E7.m1.1.1.1.1.4.1.1.2.2.3.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.2.2.3"><eq id="S2.E7.m1.1.1.1.1.4.1.1.2.2.3.1.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.2.2.3.1"></eq><ci id="S2.E7.m1.1.1.1.1.4.1.1.2.2.3.2.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.2.2.3.2">𝑠</ci><cn type="integer" id="S2.E7.m1.1.1.1.1.4.1.1.2.2.3.3.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.2.2.3.3">1</cn></apply></apply><ci id="S2.E7.m1.1.1.1.1.4.1.1.2.3.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.2.3">𝑡</ci></apply><apply id="S2.E7.m1.1.1.1.1.4.1.1.1.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1"><times id="S2.E7.m1.1.1.1.1.4.1.1.1.2.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.2"></times><apply id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1"><minus id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.1.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.1"></minus><apply id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2"><times id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.1.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.1"></times><apply id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.2.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.2.1.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.2">subscript</csymbol><ci id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.2.2.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.2.2">𝑊</ci><cn type="integer" id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.2.3.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.2.3">0</cn></apply><apply id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.3.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.3.1.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.3">subscript</csymbol><ci id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.3.2.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.3.2">𝑥</ci><ci id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.3.3.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.2.3.3">𝑠</ci></apply></apply><apply id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.3.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.3.1.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.3.2.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.3.2">𝑥</ci><ci id="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.3.3.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.1.1.1.3.3">𝑠</ci></apply></apply><apply id="S2.E7.m1.1.1.1.1.4.1.1.1.3.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.3"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.1.1.4.1.1.1.3.1.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.3">superscript</csymbol><apply id="S2.E7.m1.1.1.1.1.4.1.1.1.3.2.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.3"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.1.1.4.1.1.1.3.2.1.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.3">subscript</csymbol><ci id="S2.E7.m1.1.1.1.1.4.1.1.1.3.2.2.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.3.2.2">𝑥</ci><ci id="S2.E7.m1.1.1.1.1.4.1.1.1.3.2.3.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.3.2.3">𝑠</ci></apply><ci id="S2.E7.m1.1.1.1.1.4.1.1.1.3.3.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.3.3">𝑇</ci></apply><apply id="S2.E7.m1.1.1.1.1.4.1.1.1.4.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.4"><csymbol cd="ambiguous" id="S2.E7.m1.1.1.1.1.4.1.1.1.4.1.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.4">subscript</csymbol><ci id="S2.E7.m1.1.1.1.1.4.1.1.1.4.2.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.4.2">𝑥</ci><ci id="S2.E7.m1.1.1.1.1.4.1.1.1.4.3.cmml" xref="S2.E7.m1.1.1.1.1.4.1.1.1.4.3">𝑠</ci></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E7.m1.1c">z_{t}=f(x_{t};W_{t})=W_{t}x_{t}=\left(W_{0}-\eta\sum_{s=1}^{t}G_{t}\right)x_{t}=W_{0}x_{t}-2\eta\sum_{s=1}^{t}(W_{0}x_{s}-x_{s})x_{s}^{T}x_{s}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(7)</span></td>
</tr></tbody>
</table>
<p id="S2.SS5.p3.8" class="ltx_p">Denote <math id="S2.SS5.p3.7.m1.1" class="ltx_Math" alttext="\delta_{t}=\sum_{s=1}^{t}(W_{0}x_{s}-x_{s})x_{s}^{T}x_{s}" display="inline"><semantics id="S2.SS5.p3.7.m1.1a"><mrow id="S2.SS5.p3.7.m1.1.1" xref="S2.SS5.p3.7.m1.1.1.cmml"><msub id="S2.SS5.p3.7.m1.1.1.3" xref="S2.SS5.p3.7.m1.1.1.3.cmml"><mi id="S2.SS5.p3.7.m1.1.1.3.2" xref="S2.SS5.p3.7.m1.1.1.3.2.cmml">δ</mi><mi id="S2.SS5.p3.7.m1.1.1.3.3" xref="S2.SS5.p3.7.m1.1.1.3.3.cmml">t</mi></msub><mo rspace="0.111em" id="S2.SS5.p3.7.m1.1.1.2" xref="S2.SS5.p3.7.m1.1.1.2.cmml">=</mo><mrow id="S2.SS5.p3.7.m1.1.1.1" xref="S2.SS5.p3.7.m1.1.1.1.cmml"><msubsup id="S2.SS5.p3.7.m1.1.1.1.2" xref="S2.SS5.p3.7.m1.1.1.1.2.cmml"><mo rspace="0em" id="S2.SS5.p3.7.m1.1.1.1.2.2.2" xref="S2.SS5.p3.7.m1.1.1.1.2.2.2.cmml">∑</mo><mrow id="S2.SS5.p3.7.m1.1.1.1.2.2.3" xref="S2.SS5.p3.7.m1.1.1.1.2.2.3.cmml"><mi id="S2.SS5.p3.7.m1.1.1.1.2.2.3.2" xref="S2.SS5.p3.7.m1.1.1.1.2.2.3.2.cmml">s</mi><mo id="S2.SS5.p3.7.m1.1.1.1.2.2.3.1" xref="S2.SS5.p3.7.m1.1.1.1.2.2.3.1.cmml">=</mo><mn id="S2.SS5.p3.7.m1.1.1.1.2.2.3.3" xref="S2.SS5.p3.7.m1.1.1.1.2.2.3.3.cmml">1</mn></mrow><mi id="S2.SS5.p3.7.m1.1.1.1.2.3" xref="S2.SS5.p3.7.m1.1.1.1.2.3.cmml">t</mi></msubsup><mrow id="S2.SS5.p3.7.m1.1.1.1.1" xref="S2.SS5.p3.7.m1.1.1.1.1.cmml"><mrow id="S2.SS5.p3.7.m1.1.1.1.1.1.1" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS5.p3.7.m1.1.1.1.1.1.1.2" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.cmml"><mrow id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.cmml"><msub id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.2" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.2.cmml"><mi id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.2.2" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.2.2.cmml">W</mi><mn id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.2.3" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.2.3.cmml">0</mn></msub><mo lspace="0em" rspace="0em" id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.1" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.1.cmml">​</mo><msub id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.3" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.3.cmml"><mi id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.3.2" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.3.2.cmml">x</mi><mi id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.3.3" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.3.3.cmml">s</mi></msub></mrow><mo id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.1" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.1.cmml">−</mo><msub id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.3" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.3.cmml"><mi id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.3.2" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.3.2.cmml">x</mi><mi id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.3.3" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.3.3.cmml">s</mi></msub></mrow><mo stretchy="false" id="S2.SS5.p3.7.m1.1.1.1.1.1.1.3" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S2.SS5.p3.7.m1.1.1.1.1.2" xref="S2.SS5.p3.7.m1.1.1.1.1.2.cmml">​</mo><msubsup id="S2.SS5.p3.7.m1.1.1.1.1.3" xref="S2.SS5.p3.7.m1.1.1.1.1.3.cmml"><mi id="S2.SS5.p3.7.m1.1.1.1.1.3.2.2" xref="S2.SS5.p3.7.m1.1.1.1.1.3.2.2.cmml">x</mi><mi id="S2.SS5.p3.7.m1.1.1.1.1.3.2.3" xref="S2.SS5.p3.7.m1.1.1.1.1.3.2.3.cmml">s</mi><mi id="S2.SS5.p3.7.m1.1.1.1.1.3.3" xref="S2.SS5.p3.7.m1.1.1.1.1.3.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="S2.SS5.p3.7.m1.1.1.1.1.2a" xref="S2.SS5.p3.7.m1.1.1.1.1.2.cmml">​</mo><msub id="S2.SS5.p3.7.m1.1.1.1.1.4" xref="S2.SS5.p3.7.m1.1.1.1.1.4.cmml"><mi id="S2.SS5.p3.7.m1.1.1.1.1.4.2" xref="S2.SS5.p3.7.m1.1.1.1.1.4.2.cmml">x</mi><mi id="S2.SS5.p3.7.m1.1.1.1.1.4.3" xref="S2.SS5.p3.7.m1.1.1.1.1.4.3.cmml">s</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p3.7.m1.1b"><apply id="S2.SS5.p3.7.m1.1.1.cmml" xref="S2.SS5.p3.7.m1.1.1"><eq id="S2.SS5.p3.7.m1.1.1.2.cmml" xref="S2.SS5.p3.7.m1.1.1.2"></eq><apply id="S2.SS5.p3.7.m1.1.1.3.cmml" xref="S2.SS5.p3.7.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS5.p3.7.m1.1.1.3.1.cmml" xref="S2.SS5.p3.7.m1.1.1.3">subscript</csymbol><ci id="S2.SS5.p3.7.m1.1.1.3.2.cmml" xref="S2.SS5.p3.7.m1.1.1.3.2">𝛿</ci><ci id="S2.SS5.p3.7.m1.1.1.3.3.cmml" xref="S2.SS5.p3.7.m1.1.1.3.3">𝑡</ci></apply><apply id="S2.SS5.p3.7.m1.1.1.1.cmml" xref="S2.SS5.p3.7.m1.1.1.1"><apply id="S2.SS5.p3.7.m1.1.1.1.2.cmml" xref="S2.SS5.p3.7.m1.1.1.1.2"><csymbol cd="ambiguous" id="S2.SS5.p3.7.m1.1.1.1.2.1.cmml" xref="S2.SS5.p3.7.m1.1.1.1.2">superscript</csymbol><apply id="S2.SS5.p3.7.m1.1.1.1.2.2.cmml" xref="S2.SS5.p3.7.m1.1.1.1.2"><csymbol cd="ambiguous" id="S2.SS5.p3.7.m1.1.1.1.2.2.1.cmml" xref="S2.SS5.p3.7.m1.1.1.1.2">subscript</csymbol><sum id="S2.SS5.p3.7.m1.1.1.1.2.2.2.cmml" xref="S2.SS5.p3.7.m1.1.1.1.2.2.2"></sum><apply id="S2.SS5.p3.7.m1.1.1.1.2.2.3.cmml" xref="S2.SS5.p3.7.m1.1.1.1.2.2.3"><eq id="S2.SS5.p3.7.m1.1.1.1.2.2.3.1.cmml" xref="S2.SS5.p3.7.m1.1.1.1.2.2.3.1"></eq><ci id="S2.SS5.p3.7.m1.1.1.1.2.2.3.2.cmml" xref="S2.SS5.p3.7.m1.1.1.1.2.2.3.2">𝑠</ci><cn type="integer" id="S2.SS5.p3.7.m1.1.1.1.2.2.3.3.cmml" xref="S2.SS5.p3.7.m1.1.1.1.2.2.3.3">1</cn></apply></apply><ci id="S2.SS5.p3.7.m1.1.1.1.2.3.cmml" xref="S2.SS5.p3.7.m1.1.1.1.2.3">𝑡</ci></apply><apply id="S2.SS5.p3.7.m1.1.1.1.1.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1"><times id="S2.SS5.p3.7.m1.1.1.1.1.2.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.2"></times><apply id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1"><minus id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.1.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.1"></minus><apply id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2"><times id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.1"></times><apply id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.2.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.2.1.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.2">subscript</csymbol><ci id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.2.2.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.2.2">𝑊</ci><cn type="integer" id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.2.3.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.2.3">0</cn></apply><apply id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.3.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.3.1.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.3">subscript</csymbol><ci id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.3.2.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.3.2">𝑥</ci><ci id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.3.3.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.2.3.3">𝑠</ci></apply></apply><apply id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.3.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.3.1.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.3.2.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.3.2">𝑥</ci><ci id="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.3.3.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.1.1.1.3.3">𝑠</ci></apply></apply><apply id="S2.SS5.p3.7.m1.1.1.1.1.3.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.SS5.p3.7.m1.1.1.1.1.3.1.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.3">superscript</csymbol><apply id="S2.SS5.p3.7.m1.1.1.1.1.3.2.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.SS5.p3.7.m1.1.1.1.1.3.2.1.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.3">subscript</csymbol><ci id="S2.SS5.p3.7.m1.1.1.1.1.3.2.2.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.3.2.2">𝑥</ci><ci id="S2.SS5.p3.7.m1.1.1.1.1.3.2.3.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.3.2.3">𝑠</ci></apply><ci id="S2.SS5.p3.7.m1.1.1.1.1.3.3.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.3.3">𝑇</ci></apply><apply id="S2.SS5.p3.7.m1.1.1.1.1.4.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.4"><csymbol cd="ambiguous" id="S2.SS5.p3.7.m1.1.1.1.1.4.1.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.4">subscript</csymbol><ci id="S2.SS5.p3.7.m1.1.1.1.1.4.2.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.4.2">𝑥</ci><ci id="S2.SS5.p3.7.m1.1.1.1.1.4.3.cmml" xref="S2.SS5.p3.7.m1.1.1.1.1.4.3">𝑠</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p3.7.m1.1c">\delta_{t}=\sum_{s=1}^{t}(W_{0}x_{s}-x_{s})x_{s}^{T}x_{s}</annotation></semantics></math> and the matrix <math id="S2.SS5.p3.8.m2.3" class="ltx_Math" alttext="\Delta=[\delta_{1},\dots,\delta_{b}]" display="inline"><semantics id="S2.SS5.p3.8.m2.3a"><mrow id="S2.SS5.p3.8.m2.3.3" xref="S2.SS5.p3.8.m2.3.3.cmml"><mi mathvariant="normal" id="S2.SS5.p3.8.m2.3.3.4" xref="S2.SS5.p3.8.m2.3.3.4.cmml">Δ</mi><mo id="S2.SS5.p3.8.m2.3.3.3" xref="S2.SS5.p3.8.m2.3.3.3.cmml">=</mo><mrow id="S2.SS5.p3.8.m2.3.3.2.2" xref="S2.SS5.p3.8.m2.3.3.2.3.cmml"><mo stretchy="false" id="S2.SS5.p3.8.m2.3.3.2.2.3" xref="S2.SS5.p3.8.m2.3.3.2.3.cmml">[</mo><msub id="S2.SS5.p3.8.m2.2.2.1.1.1" xref="S2.SS5.p3.8.m2.2.2.1.1.1.cmml"><mi id="S2.SS5.p3.8.m2.2.2.1.1.1.2" xref="S2.SS5.p3.8.m2.2.2.1.1.1.2.cmml">δ</mi><mn id="S2.SS5.p3.8.m2.2.2.1.1.1.3" xref="S2.SS5.p3.8.m2.2.2.1.1.1.3.cmml">1</mn></msub><mo id="S2.SS5.p3.8.m2.3.3.2.2.4" xref="S2.SS5.p3.8.m2.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS5.p3.8.m2.1.1" xref="S2.SS5.p3.8.m2.1.1.cmml">…</mi><mo id="S2.SS5.p3.8.m2.3.3.2.2.5" xref="S2.SS5.p3.8.m2.3.3.2.3.cmml">,</mo><msub id="S2.SS5.p3.8.m2.3.3.2.2.2" xref="S2.SS5.p3.8.m2.3.3.2.2.2.cmml"><mi id="S2.SS5.p3.8.m2.3.3.2.2.2.2" xref="S2.SS5.p3.8.m2.3.3.2.2.2.2.cmml">δ</mi><mi id="S2.SS5.p3.8.m2.3.3.2.2.2.3" xref="S2.SS5.p3.8.m2.3.3.2.2.2.3.cmml">b</mi></msub><mo stretchy="false" id="S2.SS5.p3.8.m2.3.3.2.2.6" xref="S2.SS5.p3.8.m2.3.3.2.3.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p3.8.m2.3b"><apply id="S2.SS5.p3.8.m2.3.3.cmml" xref="S2.SS5.p3.8.m2.3.3"><eq id="S2.SS5.p3.8.m2.3.3.3.cmml" xref="S2.SS5.p3.8.m2.3.3.3"></eq><ci id="S2.SS5.p3.8.m2.3.3.4.cmml" xref="S2.SS5.p3.8.m2.3.3.4">Δ</ci><list id="S2.SS5.p3.8.m2.3.3.2.3.cmml" xref="S2.SS5.p3.8.m2.3.3.2.2"><apply id="S2.SS5.p3.8.m2.2.2.1.1.1.cmml" xref="S2.SS5.p3.8.m2.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.SS5.p3.8.m2.2.2.1.1.1.1.cmml" xref="S2.SS5.p3.8.m2.2.2.1.1.1">subscript</csymbol><ci id="S2.SS5.p3.8.m2.2.2.1.1.1.2.cmml" xref="S2.SS5.p3.8.m2.2.2.1.1.1.2">𝛿</ci><cn type="integer" id="S2.SS5.p3.8.m2.2.2.1.1.1.3.cmml" xref="S2.SS5.p3.8.m2.2.2.1.1.1.3">1</cn></apply><ci id="S2.SS5.p3.8.m2.1.1.cmml" xref="S2.SS5.p3.8.m2.1.1">…</ci><apply id="S2.SS5.p3.8.m2.3.3.2.2.2.cmml" xref="S2.SS5.p3.8.m2.3.3.2.2.2"><csymbol cd="ambiguous" id="S2.SS5.p3.8.m2.3.3.2.2.2.1.cmml" xref="S2.SS5.p3.8.m2.3.3.2.2.2">subscript</csymbol><ci id="S2.SS5.p3.8.m2.3.3.2.2.2.2.cmml" xref="S2.SS5.p3.8.m2.3.3.2.2.2.2">𝛿</ci><ci id="S2.SS5.p3.8.m2.3.3.2.2.2.3.cmml" xref="S2.SS5.p3.8.m2.3.3.2.2.2.3">𝑏</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p3.8.m2.3c">\Delta=[\delta_{1},\dots,\delta_{b}]</annotation></semantics></math>.
We can derive that:</p>
<table id="S2.Ex4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.Ex4.m1.1" class="ltx_Math" alttext="\Delta=\texttt{mask}\left(X^{T}X\right)\left(W_{0}X-X\right)," display="block"><semantics id="S2.Ex4.m1.1a"><mrow id="S2.Ex4.m1.1.1.1" xref="S2.Ex4.m1.1.1.1.1.cmml"><mrow id="S2.Ex4.m1.1.1.1.1" xref="S2.Ex4.m1.1.1.1.1.cmml"><mi mathvariant="normal" id="S2.Ex4.m1.1.1.1.1.4" xref="S2.Ex4.m1.1.1.1.1.4.cmml">Δ</mi><mo id="S2.Ex4.m1.1.1.1.1.3" xref="S2.Ex4.m1.1.1.1.1.3.cmml">=</mo><mrow id="S2.Ex4.m1.1.1.1.1.2" xref="S2.Ex4.m1.1.1.1.1.2.cmml"><mtext class="ltx_mathvariant_monospace" id="S2.Ex4.m1.1.1.1.1.2.4" xref="S2.Ex4.m1.1.1.1.1.2.4a.cmml">mask</mtext><mo lspace="0em" rspace="0em" id="S2.Ex4.m1.1.1.1.1.2.3" xref="S2.Ex4.m1.1.1.1.1.2.3.cmml">​</mo><mrow id="S2.Ex4.m1.1.1.1.1.1.1.1" xref="S2.Ex4.m1.1.1.1.1.1.1.1.1.cmml"><mo id="S2.Ex4.m1.1.1.1.1.1.1.1.2" xref="S2.Ex4.m1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.Ex4.m1.1.1.1.1.1.1.1.1" xref="S2.Ex4.m1.1.1.1.1.1.1.1.1.cmml"><msup id="S2.Ex4.m1.1.1.1.1.1.1.1.1.2" xref="S2.Ex4.m1.1.1.1.1.1.1.1.1.2.cmml"><mi id="S2.Ex4.m1.1.1.1.1.1.1.1.1.2.2" xref="S2.Ex4.m1.1.1.1.1.1.1.1.1.2.2.cmml">X</mi><mi id="S2.Ex4.m1.1.1.1.1.1.1.1.1.2.3" xref="S2.Ex4.m1.1.1.1.1.1.1.1.1.2.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="S2.Ex4.m1.1.1.1.1.1.1.1.1.1" xref="S2.Ex4.m1.1.1.1.1.1.1.1.1.1.cmml">​</mo><mi id="S2.Ex4.m1.1.1.1.1.1.1.1.1.3" xref="S2.Ex4.m1.1.1.1.1.1.1.1.1.3.cmml">X</mi></mrow><mo id="S2.Ex4.m1.1.1.1.1.1.1.1.3" xref="S2.Ex4.m1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S2.Ex4.m1.1.1.1.1.2.3a" xref="S2.Ex4.m1.1.1.1.1.2.3.cmml">​</mo><mrow id="S2.Ex4.m1.1.1.1.1.2.2.1" xref="S2.Ex4.m1.1.1.1.1.2.2.1.1.cmml"><mo id="S2.Ex4.m1.1.1.1.1.2.2.1.2" xref="S2.Ex4.m1.1.1.1.1.2.2.1.1.cmml">(</mo><mrow id="S2.Ex4.m1.1.1.1.1.2.2.1.1" xref="S2.Ex4.m1.1.1.1.1.2.2.1.1.cmml"><mrow id="S2.Ex4.m1.1.1.1.1.2.2.1.1.2" xref="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.cmml"><msub id="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.2" xref="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.2.cmml"><mi id="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.2.2" xref="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.2.2.cmml">W</mi><mn id="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.2.3" xref="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.2.3.cmml">0</mn></msub><mo lspace="0em" rspace="0em" id="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.1" xref="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.1.cmml">​</mo><mi id="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.3" xref="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.3.cmml">X</mi></mrow><mo id="S2.Ex4.m1.1.1.1.1.2.2.1.1.1" xref="S2.Ex4.m1.1.1.1.1.2.2.1.1.1.cmml">−</mo><mi id="S2.Ex4.m1.1.1.1.1.2.2.1.1.3" xref="S2.Ex4.m1.1.1.1.1.2.2.1.1.3.cmml">X</mi></mrow><mo id="S2.Ex4.m1.1.1.1.1.2.2.1.3" xref="S2.Ex4.m1.1.1.1.1.2.2.1.1.cmml">)</mo></mrow></mrow></mrow><mo id="S2.Ex4.m1.1.1.1.2" xref="S2.Ex4.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.Ex4.m1.1b"><apply id="S2.Ex4.m1.1.1.1.1.cmml" xref="S2.Ex4.m1.1.1.1"><eq id="S2.Ex4.m1.1.1.1.1.3.cmml" xref="S2.Ex4.m1.1.1.1.1.3"></eq><ci id="S2.Ex4.m1.1.1.1.1.4.cmml" xref="S2.Ex4.m1.1.1.1.1.4">Δ</ci><apply id="S2.Ex4.m1.1.1.1.1.2.cmml" xref="S2.Ex4.m1.1.1.1.1.2"><times id="S2.Ex4.m1.1.1.1.1.2.3.cmml" xref="S2.Ex4.m1.1.1.1.1.2.3"></times><ci id="S2.Ex4.m1.1.1.1.1.2.4a.cmml" xref="S2.Ex4.m1.1.1.1.1.2.4"><mtext class="ltx_mathvariant_monospace" id="S2.Ex4.m1.1.1.1.1.2.4.cmml" xref="S2.Ex4.m1.1.1.1.1.2.4">mask</mtext></ci><apply id="S2.Ex4.m1.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex4.m1.1.1.1.1.1.1.1"><times id="S2.Ex4.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex4.m1.1.1.1.1.1.1.1.1.1"></times><apply id="S2.Ex4.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.Ex4.m1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.Ex4.m1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.Ex4.m1.1.1.1.1.1.1.1.1.2">superscript</csymbol><ci id="S2.Ex4.m1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S2.Ex4.m1.1.1.1.1.1.1.1.1.2.2">𝑋</ci><ci id="S2.Ex4.m1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S2.Ex4.m1.1.1.1.1.1.1.1.1.2.3">𝑇</ci></apply><ci id="S2.Ex4.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.Ex4.m1.1.1.1.1.1.1.1.1.3">𝑋</ci></apply><apply id="S2.Ex4.m1.1.1.1.1.2.2.1.1.cmml" xref="S2.Ex4.m1.1.1.1.1.2.2.1"><minus id="S2.Ex4.m1.1.1.1.1.2.2.1.1.1.cmml" xref="S2.Ex4.m1.1.1.1.1.2.2.1.1.1"></minus><apply id="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.cmml" xref="S2.Ex4.m1.1.1.1.1.2.2.1.1.2"><times id="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.1.cmml" xref="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.1"></times><apply id="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.2.cmml" xref="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.2"><csymbol cd="ambiguous" id="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.2.1.cmml" xref="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.2">subscript</csymbol><ci id="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.2.2.cmml" xref="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.2.2">𝑊</ci><cn type="integer" id="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.2.3.cmml" xref="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.2.3">0</cn></apply><ci id="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.3.cmml" xref="S2.Ex4.m1.1.1.1.1.2.2.1.1.2.3">𝑋</ci></apply><ci id="S2.Ex4.m1.1.1.1.1.2.2.1.1.3.cmml" xref="S2.Ex4.m1.1.1.1.1.2.2.1.1.3">𝑋</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex4.m1.1c">\Delta=\texttt{mask}\left(X^{T}X\right)\left(W_{0}X-X\right),</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S2.SS5.p3.13" class="ltx_p">where <span id="S2.SS5.p3.13.1" class="ltx_text ltx_font_typewriter">mask</span> is the lower triangular mask with zeros (similar to the attention mask, but with zeros instead of infinities), and the term <math id="S2.SS5.p3.9.m1.1" class="ltx_Math" alttext="W_{0}X-X" display="inline"><semantics id="S2.SS5.p3.9.m1.1a"><mrow id="S2.SS5.p3.9.m1.1.1" xref="S2.SS5.p3.9.m1.1.1.cmml"><mrow id="S2.SS5.p3.9.m1.1.1.2" xref="S2.SS5.p3.9.m1.1.1.2.cmml"><msub id="S2.SS5.p3.9.m1.1.1.2.2" xref="S2.SS5.p3.9.m1.1.1.2.2.cmml"><mi id="S2.SS5.p3.9.m1.1.1.2.2.2" xref="S2.SS5.p3.9.m1.1.1.2.2.2.cmml">W</mi><mn id="S2.SS5.p3.9.m1.1.1.2.2.3" xref="S2.SS5.p3.9.m1.1.1.2.2.3.cmml">0</mn></msub><mo lspace="0em" rspace="0em" id="S2.SS5.p3.9.m1.1.1.2.1" xref="S2.SS5.p3.9.m1.1.1.2.1.cmml">​</mo><mi id="S2.SS5.p3.9.m1.1.1.2.3" xref="S2.SS5.p3.9.m1.1.1.2.3.cmml">X</mi></mrow><mo id="S2.SS5.p3.9.m1.1.1.1" xref="S2.SS5.p3.9.m1.1.1.1.cmml">−</mo><mi id="S2.SS5.p3.9.m1.1.1.3" xref="S2.SS5.p3.9.m1.1.1.3.cmml">X</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p3.9.m1.1b"><apply id="S2.SS5.p3.9.m1.1.1.cmml" xref="S2.SS5.p3.9.m1.1.1"><minus id="S2.SS5.p3.9.m1.1.1.1.cmml" xref="S2.SS5.p3.9.m1.1.1.1"></minus><apply id="S2.SS5.p3.9.m1.1.1.2.cmml" xref="S2.SS5.p3.9.m1.1.1.2"><times id="S2.SS5.p3.9.m1.1.1.2.1.cmml" xref="S2.SS5.p3.9.m1.1.1.2.1"></times><apply id="S2.SS5.p3.9.m1.1.1.2.2.cmml" xref="S2.SS5.p3.9.m1.1.1.2.2"><csymbol cd="ambiguous" id="S2.SS5.p3.9.m1.1.1.2.2.1.cmml" xref="S2.SS5.p3.9.m1.1.1.2.2">subscript</csymbol><ci id="S2.SS5.p3.9.m1.1.1.2.2.2.cmml" xref="S2.SS5.p3.9.m1.1.1.2.2.2">𝑊</ci><cn type="integer" id="S2.SS5.p3.9.m1.1.1.2.2.3.cmml" xref="S2.SS5.p3.9.m1.1.1.2.2.3">0</cn></apply><ci id="S2.SS5.p3.9.m1.1.1.2.3.cmml" xref="S2.SS5.p3.9.m1.1.1.2.3">𝑋</ci></apply><ci id="S2.SS5.p3.9.m1.1.1.3.cmml" xref="S2.SS5.p3.9.m1.1.1.3">𝑋</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p3.9.m1.1c">W_{0}X-X</annotation></semantics></math> can be reused from the computation of <math id="S2.SS5.p3.10.m2.1" class="ltx_Math" alttext="W_{b}" display="inline"><semantics id="S2.SS5.p3.10.m2.1a"><msub id="S2.SS5.p3.10.m2.1.1" xref="S2.SS5.p3.10.m2.1.1.cmml"><mi id="S2.SS5.p3.10.m2.1.1.2" xref="S2.SS5.p3.10.m2.1.1.2.cmml">W</mi><mi id="S2.SS5.p3.10.m2.1.1.3" xref="S2.SS5.p3.10.m2.1.1.3.cmml">b</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS5.p3.10.m2.1b"><apply id="S2.SS5.p3.10.m2.1.1.cmml" xref="S2.SS5.p3.10.m2.1.1"><csymbol cd="ambiguous" id="S2.SS5.p3.10.m2.1.1.1.cmml" xref="S2.SS5.p3.10.m2.1.1">subscript</csymbol><ci id="S2.SS5.p3.10.m2.1.1.2.cmml" xref="S2.SS5.p3.10.m2.1.1.2">𝑊</ci><ci id="S2.SS5.p3.10.m2.1.1.3.cmml" xref="S2.SS5.p3.10.m2.1.1.3">𝑏</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p3.10.m2.1c">W_{b}</annotation></semantics></math>.
Now <math id="S2.SS5.p3.11.m3.1" class="ltx_Math" alttext="\Delta" display="inline"><semantics id="S2.SS5.p3.11.m3.1a"><mi mathvariant="normal" id="S2.SS5.p3.11.m3.1.1" xref="S2.SS5.p3.11.m3.1.1.cmml">Δ</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p3.11.m3.1b"><ci id="S2.SS5.p3.11.m3.1.1.cmml" xref="S2.SS5.p3.11.m3.1.1">Δ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p3.11.m3.1c">\Delta</annotation></semantics></math> is also conveniently computed with <span id="S2.SS5.p3.13.2" class="ltx_text ltx_font_typewriter">matmul</span>s.
Plugging <math id="S2.SS5.p3.12.m4.1" class="ltx_Math" alttext="\Delta" display="inline"><semantics id="S2.SS5.p3.12.m4.1a"><mi mathvariant="normal" id="S2.SS5.p3.12.m4.1.1" xref="S2.SS5.p3.12.m4.1.1.cmml">Δ</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p3.12.m4.1b"><ci id="S2.SS5.p3.12.m4.1.1.cmml" xref="S2.SS5.p3.12.m4.1.1">Δ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p3.12.m4.1c">\Delta</annotation></semantics></math> back into Equation <a href="#S2.E7" title="In 2.5 Dual form ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>, we obtain <math id="S2.SS5.p3.13.m5.1" class="ltx_Math" alttext="Z=W_{0}X-2\eta\Delta" display="inline"><semantics id="S2.SS5.p3.13.m5.1a"><mrow id="S2.SS5.p3.13.m5.1.1" xref="S2.SS5.p3.13.m5.1.1.cmml"><mi id="S2.SS5.p3.13.m5.1.1.2" xref="S2.SS5.p3.13.m5.1.1.2.cmml">Z</mi><mo id="S2.SS5.p3.13.m5.1.1.1" xref="S2.SS5.p3.13.m5.1.1.1.cmml">=</mo><mrow id="S2.SS5.p3.13.m5.1.1.3" xref="S2.SS5.p3.13.m5.1.1.3.cmml"><mrow id="S2.SS5.p3.13.m5.1.1.3.2" xref="S2.SS5.p3.13.m5.1.1.3.2.cmml"><msub id="S2.SS5.p3.13.m5.1.1.3.2.2" xref="S2.SS5.p3.13.m5.1.1.3.2.2.cmml"><mi id="S2.SS5.p3.13.m5.1.1.3.2.2.2" xref="S2.SS5.p3.13.m5.1.1.3.2.2.2.cmml">W</mi><mn id="S2.SS5.p3.13.m5.1.1.3.2.2.3" xref="S2.SS5.p3.13.m5.1.1.3.2.2.3.cmml">0</mn></msub><mo lspace="0em" rspace="0em" id="S2.SS5.p3.13.m5.1.1.3.2.1" xref="S2.SS5.p3.13.m5.1.1.3.2.1.cmml">​</mo><mi id="S2.SS5.p3.13.m5.1.1.3.2.3" xref="S2.SS5.p3.13.m5.1.1.3.2.3.cmml">X</mi></mrow><mo id="S2.SS5.p3.13.m5.1.1.3.1" xref="S2.SS5.p3.13.m5.1.1.3.1.cmml">−</mo><mrow id="S2.SS5.p3.13.m5.1.1.3.3" xref="S2.SS5.p3.13.m5.1.1.3.3.cmml"><mn id="S2.SS5.p3.13.m5.1.1.3.3.2" xref="S2.SS5.p3.13.m5.1.1.3.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="S2.SS5.p3.13.m5.1.1.3.3.1" xref="S2.SS5.p3.13.m5.1.1.3.3.1.cmml">​</mo><mi id="S2.SS5.p3.13.m5.1.1.3.3.3" xref="S2.SS5.p3.13.m5.1.1.3.3.3.cmml">η</mi><mo lspace="0em" rspace="0em" id="S2.SS5.p3.13.m5.1.1.3.3.1a" xref="S2.SS5.p3.13.m5.1.1.3.3.1.cmml">​</mo><mi mathvariant="normal" id="S2.SS5.p3.13.m5.1.1.3.3.4" xref="S2.SS5.p3.13.m5.1.1.3.3.4.cmml">Δ</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p3.13.m5.1b"><apply id="S2.SS5.p3.13.m5.1.1.cmml" xref="S2.SS5.p3.13.m5.1.1"><eq id="S2.SS5.p3.13.m5.1.1.1.cmml" xref="S2.SS5.p3.13.m5.1.1.1"></eq><ci id="S2.SS5.p3.13.m5.1.1.2.cmml" xref="S2.SS5.p3.13.m5.1.1.2">𝑍</ci><apply id="S2.SS5.p3.13.m5.1.1.3.cmml" xref="S2.SS5.p3.13.m5.1.1.3"><minus id="S2.SS5.p3.13.m5.1.1.3.1.cmml" xref="S2.SS5.p3.13.m5.1.1.3.1"></minus><apply id="S2.SS5.p3.13.m5.1.1.3.2.cmml" xref="S2.SS5.p3.13.m5.1.1.3.2"><times id="S2.SS5.p3.13.m5.1.1.3.2.1.cmml" xref="S2.SS5.p3.13.m5.1.1.3.2.1"></times><apply id="S2.SS5.p3.13.m5.1.1.3.2.2.cmml" xref="S2.SS5.p3.13.m5.1.1.3.2.2"><csymbol cd="ambiguous" id="S2.SS5.p3.13.m5.1.1.3.2.2.1.cmml" xref="S2.SS5.p3.13.m5.1.1.3.2.2">subscript</csymbol><ci id="S2.SS5.p3.13.m5.1.1.3.2.2.2.cmml" xref="S2.SS5.p3.13.m5.1.1.3.2.2.2">𝑊</ci><cn type="integer" id="S2.SS5.p3.13.m5.1.1.3.2.2.3.cmml" xref="S2.SS5.p3.13.m5.1.1.3.2.2.3">0</cn></apply><ci id="S2.SS5.p3.13.m5.1.1.3.2.3.cmml" xref="S2.SS5.p3.13.m5.1.1.3.2.3">𝑋</ci></apply><apply id="S2.SS5.p3.13.m5.1.1.3.3.cmml" xref="S2.SS5.p3.13.m5.1.1.3.3"><times id="S2.SS5.p3.13.m5.1.1.3.3.1.cmml" xref="S2.SS5.p3.13.m5.1.1.3.3.1"></times><cn type="integer" id="S2.SS5.p3.13.m5.1.1.3.3.2.cmml" xref="S2.SS5.p3.13.m5.1.1.3.3.2">2</cn><ci id="S2.SS5.p3.13.m5.1.1.3.3.3.cmml" xref="S2.SS5.p3.13.m5.1.1.3.3.3">𝜂</ci><ci id="S2.SS5.p3.13.m5.1.1.3.3.4.cmml" xref="S2.SS5.p3.13.m5.1.1.3.3.4">Δ</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p3.13.m5.1c">Z=W_{0}X-2\eta\Delta</annotation></semantics></math>.</p>
</div>
<div id="S2.SS5.p4" class="ltx_para">
<p id="S2.SS5.p4.3" class="ltx_p">We call this procedure the <em id="S2.SS5.p4.3.1" class="ltx_emph ltx_font_italic">dual form</em>, in contrast to the <em id="S2.SS5.p4.3.2" class="ltx_emph ltx_font_italic">primal form</em> before this subsection, where the <math id="S2.SS5.p4.1.m1.1" class="ltx_Math" alttext="G" display="inline"><semantics id="S2.SS5.p4.1.m1.1a"><mi id="S2.SS5.p4.1.m1.1.1" xref="S2.SS5.p4.1.m1.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p4.1.m1.1b"><ci id="S2.SS5.p4.1.m1.1.1.cmml" xref="S2.SS5.p4.1.m1.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p4.1.m1.1c">G</annotation></semantics></math>s and <math id="S2.SS5.p4.2.m2.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.SS5.p4.2.m2.1a"><mi id="S2.SS5.p4.2.m2.1.1" xref="S2.SS5.p4.2.m2.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p4.2.m2.1b"><ci id="S2.SS5.p4.2.m2.1.1.cmml" xref="S2.SS5.p4.2.m2.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p4.2.m2.1c">W</annotation></semantics></math>s are explicitly materialized.
As discussed, the two forms are equivalent in output.
The terminology of primal and dual follows prior work that has explored similar mathematical formulations outside of TTT <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib36" title="" class="ltx_ref">36</a>, <a href="#bib.bib8" title="" class="ltx_ref">8</a>, <a href="#bib.bib59" title="" class="ltx_ref">59</a>]</cite>.
In Appendix <a href="#A1" title="Appendix A Dual Form ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>, we show that the dual form still works when <math id="S2.SS5.p4.3.m3.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S2.SS5.p4.3.m3.1a"><mi id="S2.SS5.p4.3.m3.1.1" xref="S2.SS5.p4.3.m3.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p4.3.m3.1b"><ci id="S2.SS5.p4.3.m3.1.1.cmml" xref="S2.SS5.p4.3.m3.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p4.3.m3.1c">f</annotation></semantics></math> is a neural network with nonlinear layers, except with more complicated notation.</p>
</div>
<div id="S2.SS5.p5" class="ltx_para">
<p id="S2.SS5.p5.9" class="ltx_p">Time complexity of the primal form within a TTT mini-batch is <math id="S2.SS5.p5.1.m1.1" class="ltx_Math" alttext="O(b\times d^{2})" display="inline"><semantics id="S2.SS5.p5.1.m1.1a"><mrow id="S2.SS5.p5.1.m1.1.1" xref="S2.SS5.p5.1.m1.1.1.cmml"><mi id="S2.SS5.p5.1.m1.1.1.3" xref="S2.SS5.p5.1.m1.1.1.3.cmml">O</mi><mo lspace="0em" rspace="0em" id="S2.SS5.p5.1.m1.1.1.2" xref="S2.SS5.p5.1.m1.1.1.2.cmml">​</mo><mrow id="S2.SS5.p5.1.m1.1.1.1.1" xref="S2.SS5.p5.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS5.p5.1.m1.1.1.1.1.2" xref="S2.SS5.p5.1.m1.1.1.1.1.1.cmml">(</mo><mrow id="S2.SS5.p5.1.m1.1.1.1.1.1" xref="S2.SS5.p5.1.m1.1.1.1.1.1.cmml"><mi id="S2.SS5.p5.1.m1.1.1.1.1.1.2" xref="S2.SS5.p5.1.m1.1.1.1.1.1.2.cmml">b</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS5.p5.1.m1.1.1.1.1.1.1" xref="S2.SS5.p5.1.m1.1.1.1.1.1.1.cmml">×</mo><msup id="S2.SS5.p5.1.m1.1.1.1.1.1.3" xref="S2.SS5.p5.1.m1.1.1.1.1.1.3.cmml"><mi id="S2.SS5.p5.1.m1.1.1.1.1.1.3.2" xref="S2.SS5.p5.1.m1.1.1.1.1.1.3.2.cmml">d</mi><mn id="S2.SS5.p5.1.m1.1.1.1.1.1.3.3" xref="S2.SS5.p5.1.m1.1.1.1.1.1.3.3.cmml">2</mn></msup></mrow><mo stretchy="false" id="S2.SS5.p5.1.m1.1.1.1.1.3" xref="S2.SS5.p5.1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p5.1.m1.1b"><apply id="S2.SS5.p5.1.m1.1.1.cmml" xref="S2.SS5.p5.1.m1.1.1"><times id="S2.SS5.p5.1.m1.1.1.2.cmml" xref="S2.SS5.p5.1.m1.1.1.2"></times><ci id="S2.SS5.p5.1.m1.1.1.3.cmml" xref="S2.SS5.p5.1.m1.1.1.3">𝑂</ci><apply id="S2.SS5.p5.1.m1.1.1.1.1.1.cmml" xref="S2.SS5.p5.1.m1.1.1.1.1"><times id="S2.SS5.p5.1.m1.1.1.1.1.1.1.cmml" xref="S2.SS5.p5.1.m1.1.1.1.1.1.1"></times><ci id="S2.SS5.p5.1.m1.1.1.1.1.1.2.cmml" xref="S2.SS5.p5.1.m1.1.1.1.1.1.2">𝑏</ci><apply id="S2.SS5.p5.1.m1.1.1.1.1.1.3.cmml" xref="S2.SS5.p5.1.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.SS5.p5.1.m1.1.1.1.1.1.3.1.cmml" xref="S2.SS5.p5.1.m1.1.1.1.1.1.3">superscript</csymbol><ci id="S2.SS5.p5.1.m1.1.1.1.1.1.3.2.cmml" xref="S2.SS5.p5.1.m1.1.1.1.1.1.3.2">𝑑</ci><cn type="integer" id="S2.SS5.p5.1.m1.1.1.1.1.1.3.3.cmml" xref="S2.SS5.p5.1.m1.1.1.1.1.1.3.3">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p5.1.m1.1c">O(b\times d^{2})</annotation></semantics></math>.
Time complexity of the dual form is <math id="S2.SS5.p5.2.m2.1" class="ltx_Math" alttext="O(b\times d^{2})" display="inline"><semantics id="S2.SS5.p5.2.m2.1a"><mrow id="S2.SS5.p5.2.m2.1.1" xref="S2.SS5.p5.2.m2.1.1.cmml"><mi id="S2.SS5.p5.2.m2.1.1.3" xref="S2.SS5.p5.2.m2.1.1.3.cmml">O</mi><mo lspace="0em" rspace="0em" id="S2.SS5.p5.2.m2.1.1.2" xref="S2.SS5.p5.2.m2.1.1.2.cmml">​</mo><mrow id="S2.SS5.p5.2.m2.1.1.1.1" xref="S2.SS5.p5.2.m2.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS5.p5.2.m2.1.1.1.1.2" xref="S2.SS5.p5.2.m2.1.1.1.1.1.cmml">(</mo><mrow id="S2.SS5.p5.2.m2.1.1.1.1.1" xref="S2.SS5.p5.2.m2.1.1.1.1.1.cmml"><mi id="S2.SS5.p5.2.m2.1.1.1.1.1.2" xref="S2.SS5.p5.2.m2.1.1.1.1.1.2.cmml">b</mi><mo lspace="0.222em" rspace="0.222em" id="S2.SS5.p5.2.m2.1.1.1.1.1.1" xref="S2.SS5.p5.2.m2.1.1.1.1.1.1.cmml">×</mo><msup id="S2.SS5.p5.2.m2.1.1.1.1.1.3" xref="S2.SS5.p5.2.m2.1.1.1.1.1.3.cmml"><mi id="S2.SS5.p5.2.m2.1.1.1.1.1.3.2" xref="S2.SS5.p5.2.m2.1.1.1.1.1.3.2.cmml">d</mi><mn id="S2.SS5.p5.2.m2.1.1.1.1.1.3.3" xref="S2.SS5.p5.2.m2.1.1.1.1.1.3.3.cmml">2</mn></msup></mrow><mo stretchy="false" id="S2.SS5.p5.2.m2.1.1.1.1.3" xref="S2.SS5.p5.2.m2.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p5.2.m2.1b"><apply id="S2.SS5.p5.2.m2.1.1.cmml" xref="S2.SS5.p5.2.m2.1.1"><times id="S2.SS5.p5.2.m2.1.1.2.cmml" xref="S2.SS5.p5.2.m2.1.1.2"></times><ci id="S2.SS5.p5.2.m2.1.1.3.cmml" xref="S2.SS5.p5.2.m2.1.1.3">𝑂</ci><apply id="S2.SS5.p5.2.m2.1.1.1.1.1.cmml" xref="S2.SS5.p5.2.m2.1.1.1.1"><times id="S2.SS5.p5.2.m2.1.1.1.1.1.1.cmml" xref="S2.SS5.p5.2.m2.1.1.1.1.1.1"></times><ci id="S2.SS5.p5.2.m2.1.1.1.1.1.2.cmml" xref="S2.SS5.p5.2.m2.1.1.1.1.1.2">𝑏</ci><apply id="S2.SS5.p5.2.m2.1.1.1.1.1.3.cmml" xref="S2.SS5.p5.2.m2.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.SS5.p5.2.m2.1.1.1.1.1.3.1.cmml" xref="S2.SS5.p5.2.m2.1.1.1.1.1.3">superscript</csymbol><ci id="S2.SS5.p5.2.m2.1.1.1.1.1.3.2.cmml" xref="S2.SS5.p5.2.m2.1.1.1.1.1.3.2">𝑑</ci><cn type="integer" id="S2.SS5.p5.2.m2.1.1.1.1.1.3.3.cmml" xref="S2.SS5.p5.2.m2.1.1.1.1.1.3.3">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p5.2.m2.1c">O(b\times d^{2})</annotation></semantics></math> for computing <math id="S2.SS5.p5.3.m3.1" class="ltx_Math" alttext="W_{b}" display="inline"><semantics id="S2.SS5.p5.3.m3.1a"><msub id="S2.SS5.p5.3.m3.1.1" xref="S2.SS5.p5.3.m3.1.1.cmml"><mi id="S2.SS5.p5.3.m3.1.1.2" xref="S2.SS5.p5.3.m3.1.1.2.cmml">W</mi><mi id="S2.SS5.p5.3.m3.1.1.3" xref="S2.SS5.p5.3.m3.1.1.3.cmml">b</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS5.p5.3.m3.1b"><apply id="S2.SS5.p5.3.m3.1.1.cmml" xref="S2.SS5.p5.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS5.p5.3.m3.1.1.1.cmml" xref="S2.SS5.p5.3.m3.1.1">subscript</csymbol><ci id="S2.SS5.p5.3.m3.1.1.2.cmml" xref="S2.SS5.p5.3.m3.1.1.2">𝑊</ci><ci id="S2.SS5.p5.3.m3.1.1.3.cmml" xref="S2.SS5.p5.3.m3.1.1.3">𝑏</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p5.3.m3.1c">W_{b}</annotation></semantics></math> alone, then an additional <math id="S2.SS5.p5.4.m4.1" class="ltx_Math" alttext="O(b^{2}\times d)" display="inline"><semantics id="S2.SS5.p5.4.m4.1a"><mrow id="S2.SS5.p5.4.m4.1.1" xref="S2.SS5.p5.4.m4.1.1.cmml"><mi id="S2.SS5.p5.4.m4.1.1.3" xref="S2.SS5.p5.4.m4.1.1.3.cmml">O</mi><mo lspace="0em" rspace="0em" id="S2.SS5.p5.4.m4.1.1.2" xref="S2.SS5.p5.4.m4.1.1.2.cmml">​</mo><mrow id="S2.SS5.p5.4.m4.1.1.1.1" xref="S2.SS5.p5.4.m4.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS5.p5.4.m4.1.1.1.1.2" xref="S2.SS5.p5.4.m4.1.1.1.1.1.cmml">(</mo><mrow id="S2.SS5.p5.4.m4.1.1.1.1.1" xref="S2.SS5.p5.4.m4.1.1.1.1.1.cmml"><msup id="S2.SS5.p5.4.m4.1.1.1.1.1.2" xref="S2.SS5.p5.4.m4.1.1.1.1.1.2.cmml"><mi id="S2.SS5.p5.4.m4.1.1.1.1.1.2.2" xref="S2.SS5.p5.4.m4.1.1.1.1.1.2.2.cmml">b</mi><mn id="S2.SS5.p5.4.m4.1.1.1.1.1.2.3" xref="S2.SS5.p5.4.m4.1.1.1.1.1.2.3.cmml">2</mn></msup><mo lspace="0.222em" rspace="0.222em" id="S2.SS5.p5.4.m4.1.1.1.1.1.1" xref="S2.SS5.p5.4.m4.1.1.1.1.1.1.cmml">×</mo><mi id="S2.SS5.p5.4.m4.1.1.1.1.1.3" xref="S2.SS5.p5.4.m4.1.1.1.1.1.3.cmml">d</mi></mrow><mo stretchy="false" id="S2.SS5.p5.4.m4.1.1.1.1.3" xref="S2.SS5.p5.4.m4.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p5.4.m4.1b"><apply id="S2.SS5.p5.4.m4.1.1.cmml" xref="S2.SS5.p5.4.m4.1.1"><times id="S2.SS5.p5.4.m4.1.1.2.cmml" xref="S2.SS5.p5.4.m4.1.1.2"></times><ci id="S2.SS5.p5.4.m4.1.1.3.cmml" xref="S2.SS5.p5.4.m4.1.1.3">𝑂</ci><apply id="S2.SS5.p5.4.m4.1.1.1.1.1.cmml" xref="S2.SS5.p5.4.m4.1.1.1.1"><times id="S2.SS5.p5.4.m4.1.1.1.1.1.1.cmml" xref="S2.SS5.p5.4.m4.1.1.1.1.1.1"></times><apply id="S2.SS5.p5.4.m4.1.1.1.1.1.2.cmml" xref="S2.SS5.p5.4.m4.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.SS5.p5.4.m4.1.1.1.1.1.2.1.cmml" xref="S2.SS5.p5.4.m4.1.1.1.1.1.2">superscript</csymbol><ci id="S2.SS5.p5.4.m4.1.1.1.1.1.2.2.cmml" xref="S2.SS5.p5.4.m4.1.1.1.1.1.2.2">𝑏</ci><cn type="integer" id="S2.SS5.p5.4.m4.1.1.1.1.1.2.3.cmml" xref="S2.SS5.p5.4.m4.1.1.1.1.1.2.3">2</cn></apply><ci id="S2.SS5.p5.4.m4.1.1.1.1.1.3.cmml" xref="S2.SS5.p5.4.m4.1.1.1.1.1.3">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p5.4.m4.1c">O(b^{2}\times d)</annotation></semantics></math> for computing <math id="S2.SS5.p5.5.m5.3" class="ltx_Math" alttext="z_{1},\dots,z_{b}" display="inline"><semantics id="S2.SS5.p5.5.m5.3a"><mrow id="S2.SS5.p5.5.m5.3.3.2" xref="S2.SS5.p5.5.m5.3.3.3.cmml"><msub id="S2.SS5.p5.5.m5.2.2.1.1" xref="S2.SS5.p5.5.m5.2.2.1.1.cmml"><mi id="S2.SS5.p5.5.m5.2.2.1.1.2" xref="S2.SS5.p5.5.m5.2.2.1.1.2.cmml">z</mi><mn id="S2.SS5.p5.5.m5.2.2.1.1.3" xref="S2.SS5.p5.5.m5.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.SS5.p5.5.m5.3.3.2.3" xref="S2.SS5.p5.5.m5.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS5.p5.5.m5.1.1" xref="S2.SS5.p5.5.m5.1.1.cmml">…</mi><mo id="S2.SS5.p5.5.m5.3.3.2.4" xref="S2.SS5.p5.5.m5.3.3.3.cmml">,</mo><msub id="S2.SS5.p5.5.m5.3.3.2.2" xref="S2.SS5.p5.5.m5.3.3.2.2.cmml"><mi id="S2.SS5.p5.5.m5.3.3.2.2.2" xref="S2.SS5.p5.5.m5.3.3.2.2.2.cmml">z</mi><mi id="S2.SS5.p5.5.m5.3.3.2.2.3" xref="S2.SS5.p5.5.m5.3.3.2.2.3.cmml">b</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p5.5.m5.3b"><list id="S2.SS5.p5.5.m5.3.3.3.cmml" xref="S2.SS5.p5.5.m5.3.3.2"><apply id="S2.SS5.p5.5.m5.2.2.1.1.cmml" xref="S2.SS5.p5.5.m5.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS5.p5.5.m5.2.2.1.1.1.cmml" xref="S2.SS5.p5.5.m5.2.2.1.1">subscript</csymbol><ci id="S2.SS5.p5.5.m5.2.2.1.1.2.cmml" xref="S2.SS5.p5.5.m5.2.2.1.1.2">𝑧</ci><cn type="integer" id="S2.SS5.p5.5.m5.2.2.1.1.3.cmml" xref="S2.SS5.p5.5.m5.2.2.1.1.3">1</cn></apply><ci id="S2.SS5.p5.5.m5.1.1.cmml" xref="S2.SS5.p5.5.m5.1.1">…</ci><apply id="S2.SS5.p5.5.m5.3.3.2.2.cmml" xref="S2.SS5.p5.5.m5.3.3.2.2"><csymbol cd="ambiguous" id="S2.SS5.p5.5.m5.3.3.2.2.1.cmml" xref="S2.SS5.p5.5.m5.3.3.2.2">subscript</csymbol><ci id="S2.SS5.p5.5.m5.3.3.2.2.2.cmml" xref="S2.SS5.p5.5.m5.3.3.2.2.2">𝑧</ci><ci id="S2.SS5.p5.5.m5.3.3.2.2.3.cmml" xref="S2.SS5.p5.5.m5.3.3.2.2.3">𝑏</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p5.5.m5.3c">z_{1},\dots,z_{b}</annotation></semantics></math>.
Compared to the primal, the dual form sacrifices theoretical complexity for hardware utilization.
In practice, <math id="S2.SS5.p5.6.m6.1" class="ltx_Math" alttext="d" display="inline"><semantics id="S2.SS5.p5.6.m6.1a"><mi id="S2.SS5.p5.6.m6.1.1" xref="S2.SS5.p5.6.m6.1.1.cmml">d</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p5.6.m6.1b"><ci id="S2.SS5.p5.6.m6.1.1.cmml" xref="S2.SS5.p5.6.m6.1.1">𝑑</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p5.6.m6.1c">d</annotation></semantics></math> is typically a few hundred and <math id="S2.SS5.p5.7.m7.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S2.SS5.p5.7.m7.1a"><mi id="S2.SS5.p5.7.m7.1.1" xref="S2.SS5.p5.7.m7.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S2.SS5.p5.7.m7.1b"><ci id="S2.SS5.p5.7.m7.1.1.cmml" xref="S2.SS5.p5.7.m7.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p5.7.m7.1c">b</annotation></semantics></math> is chosen to be only 16. As a consequence, wall-clock time for computing <math id="S2.SS5.p5.8.m8.3" class="ltx_Math" alttext="z_{1},\dots,z_{b}" display="inline"><semantics id="S2.SS5.p5.8.m8.3a"><mrow id="S2.SS5.p5.8.m8.3.3.2" xref="S2.SS5.p5.8.m8.3.3.3.cmml"><msub id="S2.SS5.p5.8.m8.2.2.1.1" xref="S2.SS5.p5.8.m8.2.2.1.1.cmml"><mi id="S2.SS5.p5.8.m8.2.2.1.1.2" xref="S2.SS5.p5.8.m8.2.2.1.1.2.cmml">z</mi><mn id="S2.SS5.p5.8.m8.2.2.1.1.3" xref="S2.SS5.p5.8.m8.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.SS5.p5.8.m8.3.3.2.3" xref="S2.SS5.p5.8.m8.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS5.p5.8.m8.1.1" xref="S2.SS5.p5.8.m8.1.1.cmml">…</mi><mo id="S2.SS5.p5.8.m8.3.3.2.4" xref="S2.SS5.p5.8.m8.3.3.3.cmml">,</mo><msub id="S2.SS5.p5.8.m8.3.3.2.2" xref="S2.SS5.p5.8.m8.3.3.2.2.cmml"><mi id="S2.SS5.p5.8.m8.3.3.2.2.2" xref="S2.SS5.p5.8.m8.3.3.2.2.2.cmml">z</mi><mi id="S2.SS5.p5.8.m8.3.3.2.2.3" xref="S2.SS5.p5.8.m8.3.3.2.2.3.cmml">b</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS5.p5.8.m8.3b"><list id="S2.SS5.p5.8.m8.3.3.3.cmml" xref="S2.SS5.p5.8.m8.3.3.2"><apply id="S2.SS5.p5.8.m8.2.2.1.1.cmml" xref="S2.SS5.p5.8.m8.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS5.p5.8.m8.2.2.1.1.1.cmml" xref="S2.SS5.p5.8.m8.2.2.1.1">subscript</csymbol><ci id="S2.SS5.p5.8.m8.2.2.1.1.2.cmml" xref="S2.SS5.p5.8.m8.2.2.1.1.2">𝑧</ci><cn type="integer" id="S2.SS5.p5.8.m8.2.2.1.1.3.cmml" xref="S2.SS5.p5.8.m8.2.2.1.1.3">1</cn></apply><ci id="S2.SS5.p5.8.m8.1.1.cmml" xref="S2.SS5.p5.8.m8.1.1">…</ci><apply id="S2.SS5.p5.8.m8.3.3.2.2.cmml" xref="S2.SS5.p5.8.m8.3.3.2.2"><csymbol cd="ambiguous" id="S2.SS5.p5.8.m8.3.3.2.2.1.cmml" xref="S2.SS5.p5.8.m8.3.3.2.2">subscript</csymbol><ci id="S2.SS5.p5.8.m8.3.3.2.2.2.cmml" xref="S2.SS5.p5.8.m8.3.3.2.2.2">𝑧</ci><ci id="S2.SS5.p5.8.m8.3.3.2.2.3.cmml" xref="S2.SS5.p5.8.m8.3.3.2.2.3">𝑏</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS5.p5.8.m8.3c">z_{1},\dots,z_{b}</annotation></semantics></math> is relatively small, as observed in the right panel of Figure <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:fig:batch-size</span>.
In our JAX implementation, training with the dual form is more than <math id="S2.SS5.p5.9.m9.1" class="ltx_math_unparsed" alttext="5\times" display="inline"><semantics id="S2.SS5.p5.9.m9.1a"><mrow id="S2.SS5.p5.9.m9.1b"><mn id="S2.SS5.p5.9.m9.1.1">5</mn><mo lspace="0.222em" id="S2.SS5.p5.9.m9.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S2.SS5.p5.9.m9.1c">5\times</annotation></semantics></math> faster than with primal.</p>
</div>
</section>
<section id="S2.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.6 </span>Theoretical equivalences</h3>

<figure id="S2.F9" class="ltx_figure"><img src="/html/2407.04620/assets/x9.png" id="S2.F9.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="346" height="118" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 9: </span>Parametric learners need to define two attributes: model and optimizer (left), and each learner uniquely induces a TTT layer (right).
Two of the induced TTT layers: TTT-Linear and TTT-MLP, are proposed in this paper.
The TTT layer with a linear model and batch GD is equivalent to linear attention <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib41" title="" class="ltx_ref">41</a>]</cite>.
</figcaption>
</figure>
<div id="S2.SS6.p1" class="ltx_para">
<p id="S2.SS6.p1.2" class="ltx_p">In Subsection <a href="#S2.SS1" title="2.1 TTT as updating a hidden state ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.1</span></a>, we mentioned that <math id="S2.SS6.p1.1.m1.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S2.SS6.p1.1.m1.1a"><mi id="S2.SS6.p1.1.m1.1.1" xref="S2.SS6.p1.1.m1.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S2.SS6.p1.1.m1.1b"><ci id="S2.SS6.p1.1.m1.1.1.cmml" xref="S2.SS6.p1.1.m1.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.p1.1.m1.1c">f</annotation></semantics></math> can be a linear model or a neural network.
In Subsection <a href="#S2.SS4" title="2.4 Parallelization with mini-batch TTT ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.4</span></a>, we also discussed three variants of the update rule: online GD, batch GD, and mini-batch GD.
Each of these <math id="S2.SS6.p1.2.m2.1" class="ltx_Math" alttext="2\times 3" display="inline"><semantics id="S2.SS6.p1.2.m2.1a"><mrow id="S2.SS6.p1.2.m2.1.1" xref="S2.SS6.p1.2.m2.1.1.cmml"><mn id="S2.SS6.p1.2.m2.1.1.2" xref="S2.SS6.p1.2.m2.1.1.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="S2.SS6.p1.2.m2.1.1.1" xref="S2.SS6.p1.2.m2.1.1.1.cmml">×</mo><mn id="S2.SS6.p1.2.m2.1.1.3" xref="S2.SS6.p1.2.m2.1.1.3.cmml">3</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.SS6.p1.2.m2.1b"><apply id="S2.SS6.p1.2.m2.1.1.cmml" xref="S2.SS6.p1.2.m2.1.1"><times id="S2.SS6.p1.2.m2.1.1.1.cmml" xref="S2.SS6.p1.2.m2.1.1.1"></times><cn type="integer" id="S2.SS6.p1.2.m2.1.1.2.cmml" xref="S2.SS6.p1.2.m2.1.1.2">2</cn><cn type="integer" id="S2.SS6.p1.2.m2.1.1.3.cmml" xref="S2.SS6.p1.2.m2.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.p1.2.m2.1c">2\times 3</annotation></semantics></math> combinations induces a different instantiation of the TTT layer, as illustrated in Figure <a href="#S2.F9" title="Figure 9 ‣ 2.6 Theoretical equivalences ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>.
We now show that among these induced instantiations, the TTT layer with a linear model and batch GD is equivalent to linear attention <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib41" title="" class="ltx_ref">41</a>]</cite>, a widely known RNN layer.<span id="footnote7" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">7</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">7</sup><span class="ltx_tag ltx_tag_note">7</span>
In a nutshell, linear attention <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib41" title="" class="ltx_ref">41</a>]</cite> is simply self-attention without the softmax.
Recall the definition of self-attention:
<math id="footnote7.m1.1" class="ltx_Math" alttext="z_{t}=V_{t}\texttt{softmax}\left(K^{T}_{t}q_{t}\right)" display="inline"><semantics id="footnote7.m1.1b"><mrow id="footnote7.m1.1.1" xref="footnote7.m1.1.1.cmml"><msub id="footnote7.m1.1.1.3" xref="footnote7.m1.1.1.3.cmml"><mi id="footnote7.m1.1.1.3.2" xref="footnote7.m1.1.1.3.2.cmml">z</mi><mi id="footnote7.m1.1.1.3.3" xref="footnote7.m1.1.1.3.3.cmml">t</mi></msub><mo id="footnote7.m1.1.1.2" xref="footnote7.m1.1.1.2.cmml">=</mo><mrow id="footnote7.m1.1.1.1" xref="footnote7.m1.1.1.1.cmml"><msub id="footnote7.m1.1.1.1.3" xref="footnote7.m1.1.1.1.3.cmml"><mi id="footnote7.m1.1.1.1.3.2" xref="footnote7.m1.1.1.1.3.2.cmml">V</mi><mi id="footnote7.m1.1.1.1.3.3" xref="footnote7.m1.1.1.1.3.3.cmml">t</mi></msub><mo lspace="0em" rspace="0em" id="footnote7.m1.1.1.1.2" xref="footnote7.m1.1.1.1.2.cmml">​</mo><mtext class="ltx_mathvariant_monospace" id="footnote7.m1.1.1.1.4" xref="footnote7.m1.1.1.1.4a.cmml">softmax</mtext><mo lspace="0em" rspace="0em" id="footnote7.m1.1.1.1.2b" xref="footnote7.m1.1.1.1.2.cmml">​</mo><mrow id="footnote7.m1.1.1.1.1.1" xref="footnote7.m1.1.1.1.1.1.1.cmml"><mo id="footnote7.m1.1.1.1.1.1.2" xref="footnote7.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="footnote7.m1.1.1.1.1.1.1" xref="footnote7.m1.1.1.1.1.1.1.cmml"><msubsup id="footnote7.m1.1.1.1.1.1.1.2" xref="footnote7.m1.1.1.1.1.1.1.2.cmml"><mi id="footnote7.m1.1.1.1.1.1.1.2.2.2" xref="footnote7.m1.1.1.1.1.1.1.2.2.2.cmml">K</mi><mi id="footnote7.m1.1.1.1.1.1.1.2.3" xref="footnote7.m1.1.1.1.1.1.1.2.3.cmml">t</mi><mi id="footnote7.m1.1.1.1.1.1.1.2.2.3" xref="footnote7.m1.1.1.1.1.1.1.2.2.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="footnote7.m1.1.1.1.1.1.1.1" xref="footnote7.m1.1.1.1.1.1.1.1.cmml">​</mo><msub id="footnote7.m1.1.1.1.1.1.1.3" xref="footnote7.m1.1.1.1.1.1.1.3.cmml"><mi id="footnote7.m1.1.1.1.1.1.1.3.2" xref="footnote7.m1.1.1.1.1.1.1.3.2.cmml">q</mi><mi id="footnote7.m1.1.1.1.1.1.1.3.3" xref="footnote7.m1.1.1.1.1.1.1.3.3.cmml">t</mi></msub></mrow><mo id="footnote7.m1.1.1.1.1.1.3" xref="footnote7.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="footnote7.m1.1c"><apply id="footnote7.m1.1.1.cmml" xref="footnote7.m1.1.1"><eq id="footnote7.m1.1.1.2.cmml" xref="footnote7.m1.1.1.2"></eq><apply id="footnote7.m1.1.1.3.cmml" xref="footnote7.m1.1.1.3"><csymbol cd="ambiguous" id="footnote7.m1.1.1.3.1.cmml" xref="footnote7.m1.1.1.3">subscript</csymbol><ci id="footnote7.m1.1.1.3.2.cmml" xref="footnote7.m1.1.1.3.2">𝑧</ci><ci id="footnote7.m1.1.1.3.3.cmml" xref="footnote7.m1.1.1.3.3">𝑡</ci></apply><apply id="footnote7.m1.1.1.1.cmml" xref="footnote7.m1.1.1.1"><times id="footnote7.m1.1.1.1.2.cmml" xref="footnote7.m1.1.1.1.2"></times><apply id="footnote7.m1.1.1.1.3.cmml" xref="footnote7.m1.1.1.1.3"><csymbol cd="ambiguous" id="footnote7.m1.1.1.1.3.1.cmml" xref="footnote7.m1.1.1.1.3">subscript</csymbol><ci id="footnote7.m1.1.1.1.3.2.cmml" xref="footnote7.m1.1.1.1.3.2">𝑉</ci><ci id="footnote7.m1.1.1.1.3.3.cmml" xref="footnote7.m1.1.1.1.3.3">𝑡</ci></apply><ci id="footnote7.m1.1.1.1.4a.cmml" xref="footnote7.m1.1.1.1.4"><mtext class="ltx_mathvariant_monospace" id="footnote7.m1.1.1.1.4.cmml" xref="footnote7.m1.1.1.1.4">softmax</mtext></ci><apply id="footnote7.m1.1.1.1.1.1.1.cmml" xref="footnote7.m1.1.1.1.1.1"><times id="footnote7.m1.1.1.1.1.1.1.1.cmml" xref="footnote7.m1.1.1.1.1.1.1.1"></times><apply id="footnote7.m1.1.1.1.1.1.1.2.cmml" xref="footnote7.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="footnote7.m1.1.1.1.1.1.1.2.1.cmml" xref="footnote7.m1.1.1.1.1.1.1.2">subscript</csymbol><apply id="footnote7.m1.1.1.1.1.1.1.2.2.cmml" xref="footnote7.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="footnote7.m1.1.1.1.1.1.1.2.2.1.cmml" xref="footnote7.m1.1.1.1.1.1.1.2">superscript</csymbol><ci id="footnote7.m1.1.1.1.1.1.1.2.2.2.cmml" xref="footnote7.m1.1.1.1.1.1.1.2.2.2">𝐾</ci><ci id="footnote7.m1.1.1.1.1.1.1.2.2.3.cmml" xref="footnote7.m1.1.1.1.1.1.1.2.2.3">𝑇</ci></apply><ci id="footnote7.m1.1.1.1.1.1.1.2.3.cmml" xref="footnote7.m1.1.1.1.1.1.1.2.3">𝑡</ci></apply><apply id="footnote7.m1.1.1.1.1.1.1.3.cmml" xref="footnote7.m1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="footnote7.m1.1.1.1.1.1.1.3.1.cmml" xref="footnote7.m1.1.1.1.1.1.1.3">subscript</csymbol><ci id="footnote7.m1.1.1.1.1.1.1.3.2.cmml" xref="footnote7.m1.1.1.1.1.1.1.3.2">𝑞</ci><ci id="footnote7.m1.1.1.1.1.1.1.3.3.cmml" xref="footnote7.m1.1.1.1.1.1.1.3.3">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote7.m1.1d">z_{t}=V_{t}\texttt{softmax}\left(K^{T}_{t}q_{t}\right)</annotation></semantics></math>.
Without <span id="footnote7.1" class="ltx_text ltx_font_typewriter">softmax</span>, this becomes
<math id="footnote7.m2.1" class="ltx_Math" alttext="z_{t}=V_{t}\left(K^{T}_{t}q_{t}\right)=\sum_{s=1}^{t}v_{s}k_{s}^{T}q_{t}" display="inline"><semantics id="footnote7.m2.1b"><mrow id="footnote7.m2.1.1" xref="footnote7.m2.1.1.cmml"><msub id="footnote7.m2.1.1.3" xref="footnote7.m2.1.1.3.cmml"><mi id="footnote7.m2.1.1.3.2" xref="footnote7.m2.1.1.3.2.cmml">z</mi><mi id="footnote7.m2.1.1.3.3" xref="footnote7.m2.1.1.3.3.cmml">t</mi></msub><mo id="footnote7.m2.1.1.4" xref="footnote7.m2.1.1.4.cmml">=</mo><mrow id="footnote7.m2.1.1.1" xref="footnote7.m2.1.1.1.cmml"><msub id="footnote7.m2.1.1.1.3" xref="footnote7.m2.1.1.1.3.cmml"><mi id="footnote7.m2.1.1.1.3.2" xref="footnote7.m2.1.1.1.3.2.cmml">V</mi><mi id="footnote7.m2.1.1.1.3.3" xref="footnote7.m2.1.1.1.3.3.cmml">t</mi></msub><mo lspace="0em" rspace="0em" id="footnote7.m2.1.1.1.2" xref="footnote7.m2.1.1.1.2.cmml">​</mo><mrow id="footnote7.m2.1.1.1.1.1" xref="footnote7.m2.1.1.1.1.1.1.cmml"><mo id="footnote7.m2.1.1.1.1.1.2" xref="footnote7.m2.1.1.1.1.1.1.cmml">(</mo><mrow id="footnote7.m2.1.1.1.1.1.1" xref="footnote7.m2.1.1.1.1.1.1.cmml"><msubsup id="footnote7.m2.1.1.1.1.1.1.2" xref="footnote7.m2.1.1.1.1.1.1.2.cmml"><mi id="footnote7.m2.1.1.1.1.1.1.2.2.2" xref="footnote7.m2.1.1.1.1.1.1.2.2.2.cmml">K</mi><mi id="footnote7.m2.1.1.1.1.1.1.2.3" xref="footnote7.m2.1.1.1.1.1.1.2.3.cmml">t</mi><mi id="footnote7.m2.1.1.1.1.1.1.2.2.3" xref="footnote7.m2.1.1.1.1.1.1.2.2.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="footnote7.m2.1.1.1.1.1.1.1" xref="footnote7.m2.1.1.1.1.1.1.1.cmml">​</mo><msub id="footnote7.m2.1.1.1.1.1.1.3" xref="footnote7.m2.1.1.1.1.1.1.3.cmml"><mi id="footnote7.m2.1.1.1.1.1.1.3.2" xref="footnote7.m2.1.1.1.1.1.1.3.2.cmml">q</mi><mi id="footnote7.m2.1.1.1.1.1.1.3.3" xref="footnote7.m2.1.1.1.1.1.1.3.3.cmml">t</mi></msub></mrow><mo id="footnote7.m2.1.1.1.1.1.3" xref="footnote7.m2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo rspace="0.111em" id="footnote7.m2.1.1.5" xref="footnote7.m2.1.1.5.cmml">=</mo><mrow id="footnote7.m2.1.1.6" xref="footnote7.m2.1.1.6.cmml"><msubsup id="footnote7.m2.1.1.6.1" xref="footnote7.m2.1.1.6.1.cmml"><mo id="footnote7.m2.1.1.6.1.2.2" xref="footnote7.m2.1.1.6.1.2.2.cmml">∑</mo><mrow id="footnote7.m2.1.1.6.1.2.3" xref="footnote7.m2.1.1.6.1.2.3.cmml"><mi id="footnote7.m2.1.1.6.1.2.3.2" xref="footnote7.m2.1.1.6.1.2.3.2.cmml">s</mi><mo id="footnote7.m2.1.1.6.1.2.3.1" xref="footnote7.m2.1.1.6.1.2.3.1.cmml">=</mo><mn id="footnote7.m2.1.1.6.1.2.3.3" xref="footnote7.m2.1.1.6.1.2.3.3.cmml">1</mn></mrow><mi id="footnote7.m2.1.1.6.1.3" xref="footnote7.m2.1.1.6.1.3.cmml">t</mi></msubsup><mrow id="footnote7.m2.1.1.6.2" xref="footnote7.m2.1.1.6.2.cmml"><msub id="footnote7.m2.1.1.6.2.2" xref="footnote7.m2.1.1.6.2.2.cmml"><mi id="footnote7.m2.1.1.6.2.2.2" xref="footnote7.m2.1.1.6.2.2.2.cmml">v</mi><mi id="footnote7.m2.1.1.6.2.2.3" xref="footnote7.m2.1.1.6.2.2.3.cmml">s</mi></msub><mo lspace="0em" rspace="0em" id="footnote7.m2.1.1.6.2.1" xref="footnote7.m2.1.1.6.2.1.cmml">​</mo><msubsup id="footnote7.m2.1.1.6.2.3" xref="footnote7.m2.1.1.6.2.3.cmml"><mi id="footnote7.m2.1.1.6.2.3.2.2" xref="footnote7.m2.1.1.6.2.3.2.2.cmml">k</mi><mi id="footnote7.m2.1.1.6.2.3.2.3" xref="footnote7.m2.1.1.6.2.3.2.3.cmml">s</mi><mi id="footnote7.m2.1.1.6.2.3.3" xref="footnote7.m2.1.1.6.2.3.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="footnote7.m2.1.1.6.2.1b" xref="footnote7.m2.1.1.6.2.1.cmml">​</mo><msub id="footnote7.m2.1.1.6.2.4" xref="footnote7.m2.1.1.6.2.4.cmml"><mi id="footnote7.m2.1.1.6.2.4.2" xref="footnote7.m2.1.1.6.2.4.2.cmml">q</mi><mi id="footnote7.m2.1.1.6.2.4.3" xref="footnote7.m2.1.1.6.2.4.3.cmml">t</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="footnote7.m2.1c"><apply id="footnote7.m2.1.1.cmml" xref="footnote7.m2.1.1"><and id="footnote7.m2.1.1a.cmml" xref="footnote7.m2.1.1"></and><apply id="footnote7.m2.1.1b.cmml" xref="footnote7.m2.1.1"><eq id="footnote7.m2.1.1.4.cmml" xref="footnote7.m2.1.1.4"></eq><apply id="footnote7.m2.1.1.3.cmml" xref="footnote7.m2.1.1.3"><csymbol cd="ambiguous" id="footnote7.m2.1.1.3.1.cmml" xref="footnote7.m2.1.1.3">subscript</csymbol><ci id="footnote7.m2.1.1.3.2.cmml" xref="footnote7.m2.1.1.3.2">𝑧</ci><ci id="footnote7.m2.1.1.3.3.cmml" xref="footnote7.m2.1.1.3.3">𝑡</ci></apply><apply id="footnote7.m2.1.1.1.cmml" xref="footnote7.m2.1.1.1"><times id="footnote7.m2.1.1.1.2.cmml" xref="footnote7.m2.1.1.1.2"></times><apply id="footnote7.m2.1.1.1.3.cmml" xref="footnote7.m2.1.1.1.3"><csymbol cd="ambiguous" id="footnote7.m2.1.1.1.3.1.cmml" xref="footnote7.m2.1.1.1.3">subscript</csymbol><ci id="footnote7.m2.1.1.1.3.2.cmml" xref="footnote7.m2.1.1.1.3.2">𝑉</ci><ci id="footnote7.m2.1.1.1.3.3.cmml" xref="footnote7.m2.1.1.1.3.3">𝑡</ci></apply><apply id="footnote7.m2.1.1.1.1.1.1.cmml" xref="footnote7.m2.1.1.1.1.1"><times id="footnote7.m2.1.1.1.1.1.1.1.cmml" xref="footnote7.m2.1.1.1.1.1.1.1"></times><apply id="footnote7.m2.1.1.1.1.1.1.2.cmml" xref="footnote7.m2.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="footnote7.m2.1.1.1.1.1.1.2.1.cmml" xref="footnote7.m2.1.1.1.1.1.1.2">subscript</csymbol><apply id="footnote7.m2.1.1.1.1.1.1.2.2.cmml" xref="footnote7.m2.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="footnote7.m2.1.1.1.1.1.1.2.2.1.cmml" xref="footnote7.m2.1.1.1.1.1.1.2">superscript</csymbol><ci id="footnote7.m2.1.1.1.1.1.1.2.2.2.cmml" xref="footnote7.m2.1.1.1.1.1.1.2.2.2">𝐾</ci><ci id="footnote7.m2.1.1.1.1.1.1.2.2.3.cmml" xref="footnote7.m2.1.1.1.1.1.1.2.2.3">𝑇</ci></apply><ci id="footnote7.m2.1.1.1.1.1.1.2.3.cmml" xref="footnote7.m2.1.1.1.1.1.1.2.3">𝑡</ci></apply><apply id="footnote7.m2.1.1.1.1.1.1.3.cmml" xref="footnote7.m2.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="footnote7.m2.1.1.1.1.1.1.3.1.cmml" xref="footnote7.m2.1.1.1.1.1.1.3">subscript</csymbol><ci id="footnote7.m2.1.1.1.1.1.1.3.2.cmml" xref="footnote7.m2.1.1.1.1.1.1.3.2">𝑞</ci><ci id="footnote7.m2.1.1.1.1.1.1.3.3.cmml" xref="footnote7.m2.1.1.1.1.1.1.3.3">𝑡</ci></apply></apply></apply></apply><apply id="footnote7.m2.1.1c.cmml" xref="footnote7.m2.1.1"><eq id="footnote7.m2.1.1.5.cmml" xref="footnote7.m2.1.1.5"></eq><share href="#footnote7.m2.1.1.1.cmml" id="footnote7.m2.1.1d.cmml" xref="footnote7.m2.1.1"></share><apply id="footnote7.m2.1.1.6.cmml" xref="footnote7.m2.1.1.6"><apply id="footnote7.m2.1.1.6.1.cmml" xref="footnote7.m2.1.1.6.1"><csymbol cd="ambiguous" id="footnote7.m2.1.1.6.1.1.cmml" xref="footnote7.m2.1.1.6.1">superscript</csymbol><apply id="footnote7.m2.1.1.6.1.2.cmml" xref="footnote7.m2.1.1.6.1"><csymbol cd="ambiguous" id="footnote7.m2.1.1.6.1.2.1.cmml" xref="footnote7.m2.1.1.6.1">subscript</csymbol><sum id="footnote7.m2.1.1.6.1.2.2.cmml" xref="footnote7.m2.1.1.6.1.2.2"></sum><apply id="footnote7.m2.1.1.6.1.2.3.cmml" xref="footnote7.m2.1.1.6.1.2.3"><eq id="footnote7.m2.1.1.6.1.2.3.1.cmml" xref="footnote7.m2.1.1.6.1.2.3.1"></eq><ci id="footnote7.m2.1.1.6.1.2.3.2.cmml" xref="footnote7.m2.1.1.6.1.2.3.2">𝑠</ci><cn type="integer" id="footnote7.m2.1.1.6.1.2.3.3.cmml" xref="footnote7.m2.1.1.6.1.2.3.3">1</cn></apply></apply><ci id="footnote7.m2.1.1.6.1.3.cmml" xref="footnote7.m2.1.1.6.1.3">𝑡</ci></apply><apply id="footnote7.m2.1.1.6.2.cmml" xref="footnote7.m2.1.1.6.2"><times id="footnote7.m2.1.1.6.2.1.cmml" xref="footnote7.m2.1.1.6.2.1"></times><apply id="footnote7.m2.1.1.6.2.2.cmml" xref="footnote7.m2.1.1.6.2.2"><csymbol cd="ambiguous" id="footnote7.m2.1.1.6.2.2.1.cmml" xref="footnote7.m2.1.1.6.2.2">subscript</csymbol><ci id="footnote7.m2.1.1.6.2.2.2.cmml" xref="footnote7.m2.1.1.6.2.2.2">𝑣</ci><ci id="footnote7.m2.1.1.6.2.2.3.cmml" xref="footnote7.m2.1.1.6.2.2.3">𝑠</ci></apply><apply id="footnote7.m2.1.1.6.2.3.cmml" xref="footnote7.m2.1.1.6.2.3"><csymbol cd="ambiguous" id="footnote7.m2.1.1.6.2.3.1.cmml" xref="footnote7.m2.1.1.6.2.3">superscript</csymbol><apply id="footnote7.m2.1.1.6.2.3.2.cmml" xref="footnote7.m2.1.1.6.2.3"><csymbol cd="ambiguous" id="footnote7.m2.1.1.6.2.3.2.1.cmml" xref="footnote7.m2.1.1.6.2.3">subscript</csymbol><ci id="footnote7.m2.1.1.6.2.3.2.2.cmml" xref="footnote7.m2.1.1.6.2.3.2.2">𝑘</ci><ci id="footnote7.m2.1.1.6.2.3.2.3.cmml" xref="footnote7.m2.1.1.6.2.3.2.3">𝑠</ci></apply><ci id="footnote7.m2.1.1.6.2.3.3.cmml" xref="footnote7.m2.1.1.6.2.3.3">𝑇</ci></apply><apply id="footnote7.m2.1.1.6.2.4.cmml" xref="footnote7.m2.1.1.6.2.4"><csymbol cd="ambiguous" id="footnote7.m2.1.1.6.2.4.1.cmml" xref="footnote7.m2.1.1.6.2.4">subscript</csymbol><ci id="footnote7.m2.1.1.6.2.4.2.cmml" xref="footnote7.m2.1.1.6.2.4.2">𝑞</ci><ci id="footnote7.m2.1.1.6.2.4.3.cmml" xref="footnote7.m2.1.1.6.2.4.3">𝑡</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote7.m2.1d">z_{t}=V_{t}\left(K^{T}_{t}q_{t}\right)=\sum_{s=1}^{t}v_{s}k_{s}^{T}q_{t}</annotation></semantics></math>, which is the simplest formulation of linear attention.
Similar to other RNN layers, it can be written in a recurrent form, where <math id="footnote7.m3.1" class="ltx_Math" alttext="\sum_{s=1}^{t}v_{s}k_{s}^{T}" display="inline"><semantics id="footnote7.m3.1b"><mrow id="footnote7.m3.1.1" xref="footnote7.m3.1.1.cmml"><msubsup id="footnote7.m3.1.1.1" xref="footnote7.m3.1.1.1.cmml"><mo id="footnote7.m3.1.1.1.2.2" xref="footnote7.m3.1.1.1.2.2.cmml">∑</mo><mrow id="footnote7.m3.1.1.1.2.3" xref="footnote7.m3.1.1.1.2.3.cmml"><mi id="footnote7.m3.1.1.1.2.3.2" xref="footnote7.m3.1.1.1.2.3.2.cmml">s</mi><mo id="footnote7.m3.1.1.1.2.3.1" xref="footnote7.m3.1.1.1.2.3.1.cmml">=</mo><mn id="footnote7.m3.1.1.1.2.3.3" xref="footnote7.m3.1.1.1.2.3.3.cmml">1</mn></mrow><mi id="footnote7.m3.1.1.1.3" xref="footnote7.m3.1.1.1.3.cmml">t</mi></msubsup><mrow id="footnote7.m3.1.1.2" xref="footnote7.m3.1.1.2.cmml"><msub id="footnote7.m3.1.1.2.2" xref="footnote7.m3.1.1.2.2.cmml"><mi id="footnote7.m3.1.1.2.2.2" xref="footnote7.m3.1.1.2.2.2.cmml">v</mi><mi id="footnote7.m3.1.1.2.2.3" xref="footnote7.m3.1.1.2.2.3.cmml">s</mi></msub><mo lspace="0em" rspace="0em" id="footnote7.m3.1.1.2.1" xref="footnote7.m3.1.1.2.1.cmml">​</mo><msubsup id="footnote7.m3.1.1.2.3" xref="footnote7.m3.1.1.2.3.cmml"><mi id="footnote7.m3.1.1.2.3.2.2" xref="footnote7.m3.1.1.2.3.2.2.cmml">k</mi><mi id="footnote7.m3.1.1.2.3.2.3" xref="footnote7.m3.1.1.2.3.2.3.cmml">s</mi><mi id="footnote7.m3.1.1.2.3.3" xref="footnote7.m3.1.1.2.3.3.cmml">T</mi></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="footnote7.m3.1c"><apply id="footnote7.m3.1.1.cmml" xref="footnote7.m3.1.1"><apply id="footnote7.m3.1.1.1.cmml" xref="footnote7.m3.1.1.1"><csymbol cd="ambiguous" id="footnote7.m3.1.1.1.1.cmml" xref="footnote7.m3.1.1.1">superscript</csymbol><apply id="footnote7.m3.1.1.1.2.cmml" xref="footnote7.m3.1.1.1"><csymbol cd="ambiguous" id="footnote7.m3.1.1.1.2.1.cmml" xref="footnote7.m3.1.1.1">subscript</csymbol><sum id="footnote7.m3.1.1.1.2.2.cmml" xref="footnote7.m3.1.1.1.2.2"></sum><apply id="footnote7.m3.1.1.1.2.3.cmml" xref="footnote7.m3.1.1.1.2.3"><eq id="footnote7.m3.1.1.1.2.3.1.cmml" xref="footnote7.m3.1.1.1.2.3.1"></eq><ci id="footnote7.m3.1.1.1.2.3.2.cmml" xref="footnote7.m3.1.1.1.2.3.2">𝑠</ci><cn type="integer" id="footnote7.m3.1.1.1.2.3.3.cmml" xref="footnote7.m3.1.1.1.2.3.3">1</cn></apply></apply><ci id="footnote7.m3.1.1.1.3.cmml" xref="footnote7.m3.1.1.1.3">𝑡</ci></apply><apply id="footnote7.m3.1.1.2.cmml" xref="footnote7.m3.1.1.2"><times id="footnote7.m3.1.1.2.1.cmml" xref="footnote7.m3.1.1.2.1"></times><apply id="footnote7.m3.1.1.2.2.cmml" xref="footnote7.m3.1.1.2.2"><csymbol cd="ambiguous" id="footnote7.m3.1.1.2.2.1.cmml" xref="footnote7.m3.1.1.2.2">subscript</csymbol><ci id="footnote7.m3.1.1.2.2.2.cmml" xref="footnote7.m3.1.1.2.2.2">𝑣</ci><ci id="footnote7.m3.1.1.2.2.3.cmml" xref="footnote7.m3.1.1.2.2.3">𝑠</ci></apply><apply id="footnote7.m3.1.1.2.3.cmml" xref="footnote7.m3.1.1.2.3"><csymbol cd="ambiguous" id="footnote7.m3.1.1.2.3.1.cmml" xref="footnote7.m3.1.1.2.3">superscript</csymbol><apply id="footnote7.m3.1.1.2.3.2.cmml" xref="footnote7.m3.1.1.2.3"><csymbol cd="ambiguous" id="footnote7.m3.1.1.2.3.2.1.cmml" xref="footnote7.m3.1.1.2.3">subscript</csymbol><ci id="footnote7.m3.1.1.2.3.2.2.cmml" xref="footnote7.m3.1.1.2.3.2.2">𝑘</ci><ci id="footnote7.m3.1.1.2.3.2.3.cmml" xref="footnote7.m3.1.1.2.3.2.3">𝑠</ci></apply><ci id="footnote7.m3.1.1.2.3.3.cmml" xref="footnote7.m3.1.1.2.3.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote7.m3.1d">\sum_{s=1}^{t}v_{s}k_{s}^{T}</annotation></semantics></math> is the hidden state.
Since <math id="footnote7.m4.1" class="ltx_Math" alttext="\sum_{s=1}^{t}v_{s}k_{s}^{T}" display="inline"><semantics id="footnote7.m4.1b"><mrow id="footnote7.m4.1.1" xref="footnote7.m4.1.1.cmml"><msubsup id="footnote7.m4.1.1.1" xref="footnote7.m4.1.1.1.cmml"><mo id="footnote7.m4.1.1.1.2.2" xref="footnote7.m4.1.1.1.2.2.cmml">∑</mo><mrow id="footnote7.m4.1.1.1.2.3" xref="footnote7.m4.1.1.1.2.3.cmml"><mi id="footnote7.m4.1.1.1.2.3.2" xref="footnote7.m4.1.1.1.2.3.2.cmml">s</mi><mo id="footnote7.m4.1.1.1.2.3.1" xref="footnote7.m4.1.1.1.2.3.1.cmml">=</mo><mn id="footnote7.m4.1.1.1.2.3.3" xref="footnote7.m4.1.1.1.2.3.3.cmml">1</mn></mrow><mi id="footnote7.m4.1.1.1.3" xref="footnote7.m4.1.1.1.3.cmml">t</mi></msubsup><mrow id="footnote7.m4.1.1.2" xref="footnote7.m4.1.1.2.cmml"><msub id="footnote7.m4.1.1.2.2" xref="footnote7.m4.1.1.2.2.cmml"><mi id="footnote7.m4.1.1.2.2.2" xref="footnote7.m4.1.1.2.2.2.cmml">v</mi><mi id="footnote7.m4.1.1.2.2.3" xref="footnote7.m4.1.1.2.2.3.cmml">s</mi></msub><mo lspace="0em" rspace="0em" id="footnote7.m4.1.1.2.1" xref="footnote7.m4.1.1.2.1.cmml">​</mo><msubsup id="footnote7.m4.1.1.2.3" xref="footnote7.m4.1.1.2.3.cmml"><mi id="footnote7.m4.1.1.2.3.2.2" xref="footnote7.m4.1.1.2.3.2.2.cmml">k</mi><mi id="footnote7.m4.1.1.2.3.2.3" xref="footnote7.m4.1.1.2.3.2.3.cmml">s</mi><mi id="footnote7.m4.1.1.2.3.3" xref="footnote7.m4.1.1.2.3.3.cmml">T</mi></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="footnote7.m4.1c"><apply id="footnote7.m4.1.1.cmml" xref="footnote7.m4.1.1"><apply id="footnote7.m4.1.1.1.cmml" xref="footnote7.m4.1.1.1"><csymbol cd="ambiguous" id="footnote7.m4.1.1.1.1.cmml" xref="footnote7.m4.1.1.1">superscript</csymbol><apply id="footnote7.m4.1.1.1.2.cmml" xref="footnote7.m4.1.1.1"><csymbol cd="ambiguous" id="footnote7.m4.1.1.1.2.1.cmml" xref="footnote7.m4.1.1.1">subscript</csymbol><sum id="footnote7.m4.1.1.1.2.2.cmml" xref="footnote7.m4.1.1.1.2.2"></sum><apply id="footnote7.m4.1.1.1.2.3.cmml" xref="footnote7.m4.1.1.1.2.3"><eq id="footnote7.m4.1.1.1.2.3.1.cmml" xref="footnote7.m4.1.1.1.2.3.1"></eq><ci id="footnote7.m4.1.1.1.2.3.2.cmml" xref="footnote7.m4.1.1.1.2.3.2">𝑠</ci><cn type="integer" id="footnote7.m4.1.1.1.2.3.3.cmml" xref="footnote7.m4.1.1.1.2.3.3">1</cn></apply></apply><ci id="footnote7.m4.1.1.1.3.cmml" xref="footnote7.m4.1.1.1.3">𝑡</ci></apply><apply id="footnote7.m4.1.1.2.cmml" xref="footnote7.m4.1.1.2"><times id="footnote7.m4.1.1.2.1.cmml" xref="footnote7.m4.1.1.2.1"></times><apply id="footnote7.m4.1.1.2.2.cmml" xref="footnote7.m4.1.1.2.2"><csymbol cd="ambiguous" id="footnote7.m4.1.1.2.2.1.cmml" xref="footnote7.m4.1.1.2.2">subscript</csymbol><ci id="footnote7.m4.1.1.2.2.2.cmml" xref="footnote7.m4.1.1.2.2.2">𝑣</ci><ci id="footnote7.m4.1.1.2.2.3.cmml" xref="footnote7.m4.1.1.2.2.3">𝑠</ci></apply><apply id="footnote7.m4.1.1.2.3.cmml" xref="footnote7.m4.1.1.2.3"><csymbol cd="ambiguous" id="footnote7.m4.1.1.2.3.1.cmml" xref="footnote7.m4.1.1.2.3">superscript</csymbol><apply id="footnote7.m4.1.1.2.3.2.cmml" xref="footnote7.m4.1.1.2.3"><csymbol cd="ambiguous" id="footnote7.m4.1.1.2.3.2.1.cmml" xref="footnote7.m4.1.1.2.3">subscript</csymbol><ci id="footnote7.m4.1.1.2.3.2.2.cmml" xref="footnote7.m4.1.1.2.3.2.2">𝑘</ci><ci id="footnote7.m4.1.1.2.3.2.3.cmml" xref="footnote7.m4.1.1.2.3.2.3">𝑠</ci></apply><ci id="footnote7.m4.1.1.2.3.3.cmml" xref="footnote7.m4.1.1.2.3.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote7.m4.1d">\sum_{s=1}^{t}v_{s}k_{s}^{T}</annotation></semantics></math> can be computed in a <span id="footnote7.2" class="ltx_text ltx_font_typewriter">cumsum</span> for every <math id="footnote7.m5.3" class="ltx_Math" alttext="t=1,\dots,T" display="inline"><semantics id="footnote7.m5.3b"><mrow id="footnote7.m5.3.4" xref="footnote7.m5.3.4.cmml"><mi id="footnote7.m5.3.4.2" xref="footnote7.m5.3.4.2.cmml">t</mi><mo id="footnote7.m5.3.4.1" xref="footnote7.m5.3.4.1.cmml">=</mo><mrow id="footnote7.m5.3.4.3.2" xref="footnote7.m5.3.4.3.1.cmml"><mn id="footnote7.m5.1.1" xref="footnote7.m5.1.1.cmml">1</mn><mo id="footnote7.m5.3.4.3.2.1" xref="footnote7.m5.3.4.3.1.cmml">,</mo><mi mathvariant="normal" id="footnote7.m5.2.2" xref="footnote7.m5.2.2.cmml">…</mi><mo id="footnote7.m5.3.4.3.2.2" xref="footnote7.m5.3.4.3.1.cmml">,</mo><mi id="footnote7.m5.3.3" xref="footnote7.m5.3.3.cmml">T</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="footnote7.m5.3c"><apply id="footnote7.m5.3.4.cmml" xref="footnote7.m5.3.4"><eq id="footnote7.m5.3.4.1.cmml" xref="footnote7.m5.3.4.1"></eq><ci id="footnote7.m5.3.4.2.cmml" xref="footnote7.m5.3.4.2">𝑡</ci><list id="footnote7.m5.3.4.3.1.cmml" xref="footnote7.m5.3.4.3.2"><cn type="integer" id="footnote7.m5.1.1.cmml" xref="footnote7.m5.1.1">1</cn><ci id="footnote7.m5.2.2.cmml" xref="footnote7.m5.2.2">…</ci><ci id="footnote7.m5.3.3.cmml" xref="footnote7.m5.3.3">𝑇</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote7.m5.3d">t=1,\dots,T</annotation></semantics></math>, linear attention also has linear complexity w.r.t. <math id="footnote7.m6.1" class="ltx_Math" alttext="T" display="inline"><semantics id="footnote7.m6.1b"><mi id="footnote7.m6.1.1" xref="footnote7.m6.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="footnote7.m6.1c"><ci id="footnote7.m6.1.1.cmml" xref="footnote7.m6.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote7.m6.1d">T</annotation></semantics></math>.
</span></span></span></p>
</div>
<div id="Thmtheorem1" class="ltx_theorem ltx_theorem_theorem">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmtheorem1.1.1.1" class="ltx_text ltx_font_bold">Theorem 1</span></span><span id="Thmtheorem1.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmtheorem1.p1" class="ltx_para">
<p id="Thmtheorem1.p1.5" class="ltx_p"><span id="Thmtheorem1.p1.5.5" class="ltx_text ltx_font_italic">Consider the TTT layer with <math id="Thmtheorem1.p1.1.1.m1.1" class="ltx_Math" alttext="f(x)=Wx" display="inline"><semantics id="Thmtheorem1.p1.1.1.m1.1a"><mrow id="Thmtheorem1.p1.1.1.m1.1.2" xref="Thmtheorem1.p1.1.1.m1.1.2.cmml"><mrow id="Thmtheorem1.p1.1.1.m1.1.2.2" xref="Thmtheorem1.p1.1.1.m1.1.2.2.cmml"><mi id="Thmtheorem1.p1.1.1.m1.1.2.2.2" xref="Thmtheorem1.p1.1.1.m1.1.2.2.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="Thmtheorem1.p1.1.1.m1.1.2.2.1" xref="Thmtheorem1.p1.1.1.m1.1.2.2.1.cmml">​</mo><mrow id="Thmtheorem1.p1.1.1.m1.1.2.2.3.2" xref="Thmtheorem1.p1.1.1.m1.1.2.2.cmml"><mo stretchy="false" id="Thmtheorem1.p1.1.1.m1.1.2.2.3.2.1" xref="Thmtheorem1.p1.1.1.m1.1.2.2.cmml">(</mo><mi id="Thmtheorem1.p1.1.1.m1.1.1" xref="Thmtheorem1.p1.1.1.m1.1.1.cmml">x</mi><mo stretchy="false" id="Thmtheorem1.p1.1.1.m1.1.2.2.3.2.2" xref="Thmtheorem1.p1.1.1.m1.1.2.2.cmml">)</mo></mrow></mrow><mo id="Thmtheorem1.p1.1.1.m1.1.2.1" xref="Thmtheorem1.p1.1.1.m1.1.2.1.cmml">=</mo><mrow id="Thmtheorem1.p1.1.1.m1.1.2.3" xref="Thmtheorem1.p1.1.1.m1.1.2.3.cmml"><mi id="Thmtheorem1.p1.1.1.m1.1.2.3.2" xref="Thmtheorem1.p1.1.1.m1.1.2.3.2.cmml">W</mi><mo lspace="0em" rspace="0em" id="Thmtheorem1.p1.1.1.m1.1.2.3.1" xref="Thmtheorem1.p1.1.1.m1.1.2.3.1.cmml">​</mo><mi id="Thmtheorem1.p1.1.1.m1.1.2.3.3" xref="Thmtheorem1.p1.1.1.m1.1.2.3.3.cmml">x</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmtheorem1.p1.1.1.m1.1b"><apply id="Thmtheorem1.p1.1.1.m1.1.2.cmml" xref="Thmtheorem1.p1.1.1.m1.1.2"><eq id="Thmtheorem1.p1.1.1.m1.1.2.1.cmml" xref="Thmtheorem1.p1.1.1.m1.1.2.1"></eq><apply id="Thmtheorem1.p1.1.1.m1.1.2.2.cmml" xref="Thmtheorem1.p1.1.1.m1.1.2.2"><times id="Thmtheorem1.p1.1.1.m1.1.2.2.1.cmml" xref="Thmtheorem1.p1.1.1.m1.1.2.2.1"></times><ci id="Thmtheorem1.p1.1.1.m1.1.2.2.2.cmml" xref="Thmtheorem1.p1.1.1.m1.1.2.2.2">𝑓</ci><ci id="Thmtheorem1.p1.1.1.m1.1.1.cmml" xref="Thmtheorem1.p1.1.1.m1.1.1">𝑥</ci></apply><apply id="Thmtheorem1.p1.1.1.m1.1.2.3.cmml" xref="Thmtheorem1.p1.1.1.m1.1.2.3"><times id="Thmtheorem1.p1.1.1.m1.1.2.3.1.cmml" xref="Thmtheorem1.p1.1.1.m1.1.2.3.1"></times><ci id="Thmtheorem1.p1.1.1.m1.1.2.3.2.cmml" xref="Thmtheorem1.p1.1.1.m1.1.2.3.2">𝑊</ci><ci id="Thmtheorem1.p1.1.1.m1.1.2.3.3.cmml" xref="Thmtheorem1.p1.1.1.m1.1.2.3.3">𝑥</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem1.p1.1.1.m1.1c">f(x)=Wx</annotation></semantics></math> as the inner-loop model, batch gradient descent with <math id="Thmtheorem1.p1.2.2.m2.1" class="ltx_Math" alttext="\eta=1/2" display="inline"><semantics id="Thmtheorem1.p1.2.2.m2.1a"><mrow id="Thmtheorem1.p1.2.2.m2.1.1" xref="Thmtheorem1.p1.2.2.m2.1.1.cmml"><mi id="Thmtheorem1.p1.2.2.m2.1.1.2" xref="Thmtheorem1.p1.2.2.m2.1.1.2.cmml">η</mi><mo id="Thmtheorem1.p1.2.2.m2.1.1.1" xref="Thmtheorem1.p1.2.2.m2.1.1.1.cmml">=</mo><mrow id="Thmtheorem1.p1.2.2.m2.1.1.3" xref="Thmtheorem1.p1.2.2.m2.1.1.3.cmml"><mn id="Thmtheorem1.p1.2.2.m2.1.1.3.2" xref="Thmtheorem1.p1.2.2.m2.1.1.3.2.cmml">1</mn><mo id="Thmtheorem1.p1.2.2.m2.1.1.3.1" xref="Thmtheorem1.p1.2.2.m2.1.1.3.1.cmml">/</mo><mn id="Thmtheorem1.p1.2.2.m2.1.1.3.3" xref="Thmtheorem1.p1.2.2.m2.1.1.3.3.cmml">2</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmtheorem1.p1.2.2.m2.1b"><apply id="Thmtheorem1.p1.2.2.m2.1.1.cmml" xref="Thmtheorem1.p1.2.2.m2.1.1"><eq id="Thmtheorem1.p1.2.2.m2.1.1.1.cmml" xref="Thmtheorem1.p1.2.2.m2.1.1.1"></eq><ci id="Thmtheorem1.p1.2.2.m2.1.1.2.cmml" xref="Thmtheorem1.p1.2.2.m2.1.1.2">𝜂</ci><apply id="Thmtheorem1.p1.2.2.m2.1.1.3.cmml" xref="Thmtheorem1.p1.2.2.m2.1.1.3"><divide id="Thmtheorem1.p1.2.2.m2.1.1.3.1.cmml" xref="Thmtheorem1.p1.2.2.m2.1.1.3.1"></divide><cn type="integer" id="Thmtheorem1.p1.2.2.m2.1.1.3.2.cmml" xref="Thmtheorem1.p1.2.2.m2.1.1.3.2">1</cn><cn type="integer" id="Thmtheorem1.p1.2.2.m2.1.1.3.3.cmml" xref="Thmtheorem1.p1.2.2.m2.1.1.3.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem1.p1.2.2.m2.1c">\eta=1/2</annotation></semantics></math> as the update rule, and <math id="Thmtheorem1.p1.3.3.m3.1" class="ltx_Math" alttext="W_{0}=0" display="inline"><semantics id="Thmtheorem1.p1.3.3.m3.1a"><mrow id="Thmtheorem1.p1.3.3.m3.1.1" xref="Thmtheorem1.p1.3.3.m3.1.1.cmml"><msub id="Thmtheorem1.p1.3.3.m3.1.1.2" xref="Thmtheorem1.p1.3.3.m3.1.1.2.cmml"><mi id="Thmtheorem1.p1.3.3.m3.1.1.2.2" xref="Thmtheorem1.p1.3.3.m3.1.1.2.2.cmml">W</mi><mn id="Thmtheorem1.p1.3.3.m3.1.1.2.3" xref="Thmtheorem1.p1.3.3.m3.1.1.2.3.cmml">0</mn></msub><mo id="Thmtheorem1.p1.3.3.m3.1.1.1" xref="Thmtheorem1.p1.3.3.m3.1.1.1.cmml">=</mo><mn id="Thmtheorem1.p1.3.3.m3.1.1.3" xref="Thmtheorem1.p1.3.3.m3.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="Thmtheorem1.p1.3.3.m3.1b"><apply id="Thmtheorem1.p1.3.3.m3.1.1.cmml" xref="Thmtheorem1.p1.3.3.m3.1.1"><eq id="Thmtheorem1.p1.3.3.m3.1.1.1.cmml" xref="Thmtheorem1.p1.3.3.m3.1.1.1"></eq><apply id="Thmtheorem1.p1.3.3.m3.1.1.2.cmml" xref="Thmtheorem1.p1.3.3.m3.1.1.2"><csymbol cd="ambiguous" id="Thmtheorem1.p1.3.3.m3.1.1.2.1.cmml" xref="Thmtheorem1.p1.3.3.m3.1.1.2">subscript</csymbol><ci id="Thmtheorem1.p1.3.3.m3.1.1.2.2.cmml" xref="Thmtheorem1.p1.3.3.m3.1.1.2.2">𝑊</ci><cn type="integer" id="Thmtheorem1.p1.3.3.m3.1.1.2.3.cmml" xref="Thmtheorem1.p1.3.3.m3.1.1.2.3">0</cn></apply><cn type="integer" id="Thmtheorem1.p1.3.3.m3.1.1.3.cmml" xref="Thmtheorem1.p1.3.3.m3.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem1.p1.3.3.m3.1c">W_{0}=0</annotation></semantics></math>. Then, given the same input sequence <math id="Thmtheorem1.p1.4.4.m4.3" class="ltx_Math" alttext="x_{1},\dots,x_{T}" display="inline"><semantics id="Thmtheorem1.p1.4.4.m4.3a"><mrow id="Thmtheorem1.p1.4.4.m4.3.3.2" xref="Thmtheorem1.p1.4.4.m4.3.3.3.cmml"><msub id="Thmtheorem1.p1.4.4.m4.2.2.1.1" xref="Thmtheorem1.p1.4.4.m4.2.2.1.1.cmml"><mi id="Thmtheorem1.p1.4.4.m4.2.2.1.1.2" xref="Thmtheorem1.p1.4.4.m4.2.2.1.1.2.cmml">x</mi><mn id="Thmtheorem1.p1.4.4.m4.2.2.1.1.3" xref="Thmtheorem1.p1.4.4.m4.2.2.1.1.3.cmml">1</mn></msub><mo id="Thmtheorem1.p1.4.4.m4.3.3.2.3" xref="Thmtheorem1.p1.4.4.m4.3.3.3.cmml">,</mo><mi mathvariant="normal" id="Thmtheorem1.p1.4.4.m4.1.1" xref="Thmtheorem1.p1.4.4.m4.1.1.cmml">…</mi><mo id="Thmtheorem1.p1.4.4.m4.3.3.2.4" xref="Thmtheorem1.p1.4.4.m4.3.3.3.cmml">,</mo><msub id="Thmtheorem1.p1.4.4.m4.3.3.2.2" xref="Thmtheorem1.p1.4.4.m4.3.3.2.2.cmml"><mi id="Thmtheorem1.p1.4.4.m4.3.3.2.2.2" xref="Thmtheorem1.p1.4.4.m4.3.3.2.2.2.cmml">x</mi><mi id="Thmtheorem1.p1.4.4.m4.3.3.2.2.3" xref="Thmtheorem1.p1.4.4.m4.3.3.2.2.3.cmml">T</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="Thmtheorem1.p1.4.4.m4.3b"><list id="Thmtheorem1.p1.4.4.m4.3.3.3.cmml" xref="Thmtheorem1.p1.4.4.m4.3.3.2"><apply id="Thmtheorem1.p1.4.4.m4.2.2.1.1.cmml" xref="Thmtheorem1.p1.4.4.m4.2.2.1.1"><csymbol cd="ambiguous" id="Thmtheorem1.p1.4.4.m4.2.2.1.1.1.cmml" xref="Thmtheorem1.p1.4.4.m4.2.2.1.1">subscript</csymbol><ci id="Thmtheorem1.p1.4.4.m4.2.2.1.1.2.cmml" xref="Thmtheorem1.p1.4.4.m4.2.2.1.1.2">𝑥</ci><cn type="integer" id="Thmtheorem1.p1.4.4.m4.2.2.1.1.3.cmml" xref="Thmtheorem1.p1.4.4.m4.2.2.1.1.3">1</cn></apply><ci id="Thmtheorem1.p1.4.4.m4.1.1.cmml" xref="Thmtheorem1.p1.4.4.m4.1.1">…</ci><apply id="Thmtheorem1.p1.4.4.m4.3.3.2.2.cmml" xref="Thmtheorem1.p1.4.4.m4.3.3.2.2"><csymbol cd="ambiguous" id="Thmtheorem1.p1.4.4.m4.3.3.2.2.1.cmml" xref="Thmtheorem1.p1.4.4.m4.3.3.2.2">subscript</csymbol><ci id="Thmtheorem1.p1.4.4.m4.3.3.2.2.2.cmml" xref="Thmtheorem1.p1.4.4.m4.3.3.2.2.2">𝑥</ci><ci id="Thmtheorem1.p1.4.4.m4.3.3.2.2.3.cmml" xref="Thmtheorem1.p1.4.4.m4.3.3.2.2.3">𝑇</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem1.p1.4.4.m4.3c">x_{1},\dots,x_{T}</annotation></semantics></math>, the output rule defined in Equation <a href="#S2.E5" title="In 2.3 Learning a self-supervised task for TTT ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> produces the same output sequence <math id="Thmtheorem1.p1.5.5.m5.3" class="ltx_Math" alttext="z_{1},\dots,z_{T}" display="inline"><semantics id="Thmtheorem1.p1.5.5.m5.3a"><mrow id="Thmtheorem1.p1.5.5.m5.3.3.2" xref="Thmtheorem1.p1.5.5.m5.3.3.3.cmml"><msub id="Thmtheorem1.p1.5.5.m5.2.2.1.1" xref="Thmtheorem1.p1.5.5.m5.2.2.1.1.cmml"><mi id="Thmtheorem1.p1.5.5.m5.2.2.1.1.2" xref="Thmtheorem1.p1.5.5.m5.2.2.1.1.2.cmml">z</mi><mn id="Thmtheorem1.p1.5.5.m5.2.2.1.1.3" xref="Thmtheorem1.p1.5.5.m5.2.2.1.1.3.cmml">1</mn></msub><mo id="Thmtheorem1.p1.5.5.m5.3.3.2.3" xref="Thmtheorem1.p1.5.5.m5.3.3.3.cmml">,</mo><mi mathvariant="normal" id="Thmtheorem1.p1.5.5.m5.1.1" xref="Thmtheorem1.p1.5.5.m5.1.1.cmml">…</mi><mo id="Thmtheorem1.p1.5.5.m5.3.3.2.4" xref="Thmtheorem1.p1.5.5.m5.3.3.3.cmml">,</mo><msub id="Thmtheorem1.p1.5.5.m5.3.3.2.2" xref="Thmtheorem1.p1.5.5.m5.3.3.2.2.cmml"><mi id="Thmtheorem1.p1.5.5.m5.3.3.2.2.2" xref="Thmtheorem1.p1.5.5.m5.3.3.2.2.2.cmml">z</mi><mi id="Thmtheorem1.p1.5.5.m5.3.3.2.2.3" xref="Thmtheorem1.p1.5.5.m5.3.3.2.2.3.cmml">T</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="Thmtheorem1.p1.5.5.m5.3b"><list id="Thmtheorem1.p1.5.5.m5.3.3.3.cmml" xref="Thmtheorem1.p1.5.5.m5.3.3.2"><apply id="Thmtheorem1.p1.5.5.m5.2.2.1.1.cmml" xref="Thmtheorem1.p1.5.5.m5.2.2.1.1"><csymbol cd="ambiguous" id="Thmtheorem1.p1.5.5.m5.2.2.1.1.1.cmml" xref="Thmtheorem1.p1.5.5.m5.2.2.1.1">subscript</csymbol><ci id="Thmtheorem1.p1.5.5.m5.2.2.1.1.2.cmml" xref="Thmtheorem1.p1.5.5.m5.2.2.1.1.2">𝑧</ci><cn type="integer" id="Thmtheorem1.p1.5.5.m5.2.2.1.1.3.cmml" xref="Thmtheorem1.p1.5.5.m5.2.2.1.1.3">1</cn></apply><ci id="Thmtheorem1.p1.5.5.m5.1.1.cmml" xref="Thmtheorem1.p1.5.5.m5.1.1">…</ci><apply id="Thmtheorem1.p1.5.5.m5.3.3.2.2.cmml" xref="Thmtheorem1.p1.5.5.m5.3.3.2.2"><csymbol cd="ambiguous" id="Thmtheorem1.p1.5.5.m5.3.3.2.2.1.cmml" xref="Thmtheorem1.p1.5.5.m5.3.3.2.2">subscript</csymbol><ci id="Thmtheorem1.p1.5.5.m5.3.3.2.2.2.cmml" xref="Thmtheorem1.p1.5.5.m5.3.3.2.2.2">𝑧</ci><ci id="Thmtheorem1.p1.5.5.m5.3.3.2.2.3.cmml" xref="Thmtheorem1.p1.5.5.m5.3.3.2.2.3">𝑇</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem1.p1.5.5.m5.3c">z_{1},\dots,z_{T}</annotation></semantics></math> as linear attention.</span></p>
</div>
</div>
<div id="S2.SS6.1" class="ltx_proof">
<h6 class="ltx_title ltx_runin ltx_font_italic ltx_title_proof">Proof.</h6>
<div id="S2.SS6.1.p1" class="ltx_para">
<p id="S2.SS6.1.p1.2" class="ltx_p">By definition of <math id="S2.SS6.1.p1.1.m1.1" class="ltx_Math" alttext="\ell" display="inline"><semantics id="S2.SS6.1.p1.1.m1.1a"><mi mathvariant="normal" id="S2.SS6.1.p1.1.m1.1.1" xref="S2.SS6.1.p1.1.m1.1.1.cmml">ℓ</mi><annotation-xml encoding="MathML-Content" id="S2.SS6.1.p1.1.m1.1b"><ci id="S2.SS6.1.p1.1.m1.1.1.cmml" xref="S2.SS6.1.p1.1.m1.1.1">ℓ</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.1.p1.1.m1.1c">\ell</annotation></semantics></math> in Equation <a href="#S2.E4" title="In 2.3 Learning a self-supervised task for TTT ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>,
<math id="S2.SS6.1.p1.2.m2.4" class="ltx_Math" alttext="\nabla\ell\left(W_{0};x_{t}\right)=-2(\theta_{V}x_{t})(\theta_{K}x_{t})^{T}" display="inline"><semantics id="S2.SS6.1.p1.2.m2.4a"><mrow id="S2.SS6.1.p1.2.m2.4.4" xref="S2.SS6.1.p1.2.m2.4.4.cmml"><mrow id="S2.SS6.1.p1.2.m2.2.2.2" xref="S2.SS6.1.p1.2.m2.2.2.2.cmml"><mrow id="S2.SS6.1.p1.2.m2.2.2.2.4" xref="S2.SS6.1.p1.2.m2.2.2.2.4.cmml"><mo rspace="0.167em" id="S2.SS6.1.p1.2.m2.2.2.2.4.1" xref="S2.SS6.1.p1.2.m2.2.2.2.4.1.cmml">∇</mo><mi mathvariant="normal" id="S2.SS6.1.p1.2.m2.2.2.2.4.2" xref="S2.SS6.1.p1.2.m2.2.2.2.4.2.cmml">ℓ</mi></mrow><mo lspace="0em" rspace="0em" id="S2.SS6.1.p1.2.m2.2.2.2.3" xref="S2.SS6.1.p1.2.m2.2.2.2.3.cmml">​</mo><mrow id="S2.SS6.1.p1.2.m2.2.2.2.2.2" xref="S2.SS6.1.p1.2.m2.2.2.2.2.3.cmml"><mo id="S2.SS6.1.p1.2.m2.2.2.2.2.2.3" xref="S2.SS6.1.p1.2.m2.2.2.2.2.3.cmml">(</mo><msub id="S2.SS6.1.p1.2.m2.1.1.1.1.1.1" xref="S2.SS6.1.p1.2.m2.1.1.1.1.1.1.cmml"><mi id="S2.SS6.1.p1.2.m2.1.1.1.1.1.1.2" xref="S2.SS6.1.p1.2.m2.1.1.1.1.1.1.2.cmml">W</mi><mn id="S2.SS6.1.p1.2.m2.1.1.1.1.1.1.3" xref="S2.SS6.1.p1.2.m2.1.1.1.1.1.1.3.cmml">0</mn></msub><mo id="S2.SS6.1.p1.2.m2.2.2.2.2.2.4" xref="S2.SS6.1.p1.2.m2.2.2.2.2.3.cmml">;</mo><msub id="S2.SS6.1.p1.2.m2.2.2.2.2.2.2" xref="S2.SS6.1.p1.2.m2.2.2.2.2.2.2.cmml"><mi id="S2.SS6.1.p1.2.m2.2.2.2.2.2.2.2" xref="S2.SS6.1.p1.2.m2.2.2.2.2.2.2.2.cmml">x</mi><mi id="S2.SS6.1.p1.2.m2.2.2.2.2.2.2.3" xref="S2.SS6.1.p1.2.m2.2.2.2.2.2.2.3.cmml">t</mi></msub><mo id="S2.SS6.1.p1.2.m2.2.2.2.2.2.5" xref="S2.SS6.1.p1.2.m2.2.2.2.2.3.cmml">)</mo></mrow></mrow><mo id="S2.SS6.1.p1.2.m2.4.4.5" xref="S2.SS6.1.p1.2.m2.4.4.5.cmml">=</mo><mrow id="S2.SS6.1.p1.2.m2.4.4.4" xref="S2.SS6.1.p1.2.m2.4.4.4.cmml"><mo id="S2.SS6.1.p1.2.m2.4.4.4a" xref="S2.SS6.1.p1.2.m2.4.4.4.cmml">−</mo><mrow id="S2.SS6.1.p1.2.m2.4.4.4.2" xref="S2.SS6.1.p1.2.m2.4.4.4.2.cmml"><mn id="S2.SS6.1.p1.2.m2.4.4.4.2.4" xref="S2.SS6.1.p1.2.m2.4.4.4.2.4.cmml">2</mn><mo lspace="0em" rspace="0em" id="S2.SS6.1.p1.2.m2.4.4.4.2.3" xref="S2.SS6.1.p1.2.m2.4.4.4.2.3.cmml">​</mo><mrow id="S2.SS6.1.p1.2.m2.3.3.3.1.1.1" xref="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.2" xref="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.cmml">(</mo><mrow id="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1" xref="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.cmml"><msub id="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.2" xref="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.2.cmml"><mi id="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.2.2" xref="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.2.2.cmml">θ</mi><mi id="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.2.3" xref="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.2.3.cmml">V</mi></msub><mo lspace="0em" rspace="0em" id="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.1" xref="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.1.cmml">​</mo><msub id="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.3" xref="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.3.cmml"><mi id="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.3.2" xref="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.3.2.cmml">x</mi><mi id="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.3.3" xref="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.3.3.cmml">t</mi></msub></mrow><mo stretchy="false" id="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.3" xref="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S2.SS6.1.p1.2.m2.4.4.4.2.3a" xref="S2.SS6.1.p1.2.m2.4.4.4.2.3.cmml">​</mo><msup id="S2.SS6.1.p1.2.m2.4.4.4.2.2" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.cmml"><mrow id="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.cmml"><mo stretchy="false" id="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.2" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.cmml">(</mo><mrow id="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.cmml"><msub id="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.2" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.2.cmml"><mi id="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.2.2" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.2.2.cmml">θ</mi><mi id="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.2.3" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.2.3.cmml">K</mi></msub><mo lspace="0em" rspace="0em" id="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.1" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.1.cmml">​</mo><msub id="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.3" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.3.cmml"><mi id="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.3.2" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.3.2.cmml">x</mi><mi id="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.3.3" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.3.3.cmml">t</mi></msub></mrow><mo stretchy="false" id="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.3" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.cmml">)</mo></mrow><mi id="S2.SS6.1.p1.2.m2.4.4.4.2.2.3" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.3.cmml">T</mi></msup></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS6.1.p1.2.m2.4b"><apply id="S2.SS6.1.p1.2.m2.4.4.cmml" xref="S2.SS6.1.p1.2.m2.4.4"><eq id="S2.SS6.1.p1.2.m2.4.4.5.cmml" xref="S2.SS6.1.p1.2.m2.4.4.5"></eq><apply id="S2.SS6.1.p1.2.m2.2.2.2.cmml" xref="S2.SS6.1.p1.2.m2.2.2.2"><times id="S2.SS6.1.p1.2.m2.2.2.2.3.cmml" xref="S2.SS6.1.p1.2.m2.2.2.2.3"></times><apply id="S2.SS6.1.p1.2.m2.2.2.2.4.cmml" xref="S2.SS6.1.p1.2.m2.2.2.2.4"><ci id="S2.SS6.1.p1.2.m2.2.2.2.4.1.cmml" xref="S2.SS6.1.p1.2.m2.2.2.2.4.1">∇</ci><ci id="S2.SS6.1.p1.2.m2.2.2.2.4.2.cmml" xref="S2.SS6.1.p1.2.m2.2.2.2.4.2">ℓ</ci></apply><list id="S2.SS6.1.p1.2.m2.2.2.2.2.3.cmml" xref="S2.SS6.1.p1.2.m2.2.2.2.2.2"><apply id="S2.SS6.1.p1.2.m2.1.1.1.1.1.1.cmml" xref="S2.SS6.1.p1.2.m2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS6.1.p1.2.m2.1.1.1.1.1.1.1.cmml" xref="S2.SS6.1.p1.2.m2.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS6.1.p1.2.m2.1.1.1.1.1.1.2.cmml" xref="S2.SS6.1.p1.2.m2.1.1.1.1.1.1.2">𝑊</ci><cn type="integer" id="S2.SS6.1.p1.2.m2.1.1.1.1.1.1.3.cmml" xref="S2.SS6.1.p1.2.m2.1.1.1.1.1.1.3">0</cn></apply><apply id="S2.SS6.1.p1.2.m2.2.2.2.2.2.2.cmml" xref="S2.SS6.1.p1.2.m2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS6.1.p1.2.m2.2.2.2.2.2.2.1.cmml" xref="S2.SS6.1.p1.2.m2.2.2.2.2.2.2">subscript</csymbol><ci id="S2.SS6.1.p1.2.m2.2.2.2.2.2.2.2.cmml" xref="S2.SS6.1.p1.2.m2.2.2.2.2.2.2.2">𝑥</ci><ci id="S2.SS6.1.p1.2.m2.2.2.2.2.2.2.3.cmml" xref="S2.SS6.1.p1.2.m2.2.2.2.2.2.2.3">𝑡</ci></apply></list></apply><apply id="S2.SS6.1.p1.2.m2.4.4.4.cmml" xref="S2.SS6.1.p1.2.m2.4.4.4"><minus id="S2.SS6.1.p1.2.m2.4.4.4.3.cmml" xref="S2.SS6.1.p1.2.m2.4.4.4"></minus><apply id="S2.SS6.1.p1.2.m2.4.4.4.2.cmml" xref="S2.SS6.1.p1.2.m2.4.4.4.2"><times id="S2.SS6.1.p1.2.m2.4.4.4.2.3.cmml" xref="S2.SS6.1.p1.2.m2.4.4.4.2.3"></times><cn type="integer" id="S2.SS6.1.p1.2.m2.4.4.4.2.4.cmml" xref="S2.SS6.1.p1.2.m2.4.4.4.2.4">2</cn><apply id="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.cmml" xref="S2.SS6.1.p1.2.m2.3.3.3.1.1.1"><times id="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.1.cmml" xref="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.1"></times><apply id="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.2.cmml" xref="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.2.1.cmml" xref="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.2">subscript</csymbol><ci id="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.2.2.cmml" xref="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.2.2">𝜃</ci><ci id="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.2.3.cmml" xref="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.2.3">𝑉</ci></apply><apply id="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.3.cmml" xref="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.3.1.cmml" xref="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.3">subscript</csymbol><ci id="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.3.2.cmml" xref="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.3.2">𝑥</ci><ci id="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.3.3.cmml" xref="S2.SS6.1.p1.2.m2.3.3.3.1.1.1.1.3.3">𝑡</ci></apply></apply><apply id="S2.SS6.1.p1.2.m2.4.4.4.2.2.cmml" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2"><csymbol cd="ambiguous" id="S2.SS6.1.p1.2.m2.4.4.4.2.2.2.cmml" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2">superscript</csymbol><apply id="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.cmml" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1"><times id="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.1.cmml" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.1"></times><apply id="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.2.cmml" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.2"><csymbol cd="ambiguous" id="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.2.1.cmml" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.2">subscript</csymbol><ci id="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.2.2.cmml" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.2.2">𝜃</ci><ci id="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.2.3.cmml" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.2.3">𝐾</ci></apply><apply id="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.3.cmml" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.3"><csymbol cd="ambiguous" id="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.3.1.cmml" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.3">subscript</csymbol><ci id="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.3.2.cmml" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.3.2">𝑥</ci><ci id="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.3.3.cmml" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.1.1.1.3.3">𝑡</ci></apply></apply><ci id="S2.SS6.1.p1.2.m2.4.4.4.2.2.3.cmml" xref="S2.SS6.1.p1.2.m2.4.4.4.2.2.3">𝑇</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.1.p1.2.m2.4c">\nabla\ell\left(W_{0};x_{t}\right)=-2(\theta_{V}x_{t})(\theta_{K}x_{t})^{T}</annotation></semantics></math>.
By definition of batch GD in Equation <a href="#S2.E6" title="In 2.4 Parallelization with mini-batch TTT ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>
:</p>
<table id="S2.Ex5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.Ex5.m1.1" class="ltx_Math" alttext="W_{t}=W_{t-1}-\eta\nabla\ell\left(W_{0};x_{t}\right)=W_{0}-\eta\sum_{s=1}^{t}\nabla\ell\left(W_{0};x_{s}\right)=\sum_{s=1}^{t}(\theta_{V}x_{s})(\theta_{K}x_{s})^{T}." display="block"><semantics id="S2.Ex5.m1.1a"><mrow id="S2.Ex5.m1.1.1.1" xref="S2.Ex5.m1.1.1.1.1.cmml"><mrow id="S2.Ex5.m1.1.1.1.1" xref="S2.Ex5.m1.1.1.1.1.cmml"><msub id="S2.Ex5.m1.1.1.1.1.8" xref="S2.Ex5.m1.1.1.1.1.8.cmml"><mi id="S2.Ex5.m1.1.1.1.1.8.2" xref="S2.Ex5.m1.1.1.1.1.8.2.cmml">W</mi><mi id="S2.Ex5.m1.1.1.1.1.8.3" xref="S2.Ex5.m1.1.1.1.1.8.3.cmml">t</mi></msub><mo id="S2.Ex5.m1.1.1.1.1.9" xref="S2.Ex5.m1.1.1.1.1.9.cmml">=</mo><mrow id="S2.Ex5.m1.1.1.1.1.2" xref="S2.Ex5.m1.1.1.1.1.2.cmml"><msub id="S2.Ex5.m1.1.1.1.1.2.4" xref="S2.Ex5.m1.1.1.1.1.2.4.cmml"><mi id="S2.Ex5.m1.1.1.1.1.2.4.2" xref="S2.Ex5.m1.1.1.1.1.2.4.2.cmml">W</mi><mrow id="S2.Ex5.m1.1.1.1.1.2.4.3" xref="S2.Ex5.m1.1.1.1.1.2.4.3.cmml"><mi id="S2.Ex5.m1.1.1.1.1.2.4.3.2" xref="S2.Ex5.m1.1.1.1.1.2.4.3.2.cmml">t</mi><mo id="S2.Ex5.m1.1.1.1.1.2.4.3.1" xref="S2.Ex5.m1.1.1.1.1.2.4.3.1.cmml">−</mo><mn id="S2.Ex5.m1.1.1.1.1.2.4.3.3" xref="S2.Ex5.m1.1.1.1.1.2.4.3.3.cmml">1</mn></mrow></msub><mo id="S2.Ex5.m1.1.1.1.1.2.3" xref="S2.Ex5.m1.1.1.1.1.2.3.cmml">−</mo><mrow id="S2.Ex5.m1.1.1.1.1.2.2" xref="S2.Ex5.m1.1.1.1.1.2.2.cmml"><mi id="S2.Ex5.m1.1.1.1.1.2.2.4" xref="S2.Ex5.m1.1.1.1.1.2.2.4.cmml">η</mi><mo lspace="0.167em" rspace="0em" id="S2.Ex5.m1.1.1.1.1.2.2.3" xref="S2.Ex5.m1.1.1.1.1.2.2.3.cmml">​</mo><mrow id="S2.Ex5.m1.1.1.1.1.2.2.5" xref="S2.Ex5.m1.1.1.1.1.2.2.5.cmml"><mo rspace="0.167em" id="S2.Ex5.m1.1.1.1.1.2.2.5.1" xref="S2.Ex5.m1.1.1.1.1.2.2.5.1.cmml">∇</mo><mi mathvariant="normal" id="S2.Ex5.m1.1.1.1.1.2.2.5.2" xref="S2.Ex5.m1.1.1.1.1.2.2.5.2.cmml">ℓ</mi></mrow><mo lspace="0em" rspace="0em" id="S2.Ex5.m1.1.1.1.1.2.2.3a" xref="S2.Ex5.m1.1.1.1.1.2.2.3.cmml">​</mo><mrow id="S2.Ex5.m1.1.1.1.1.2.2.2.2" xref="S2.Ex5.m1.1.1.1.1.2.2.2.3.cmml"><mo id="S2.Ex5.m1.1.1.1.1.2.2.2.2.3" xref="S2.Ex5.m1.1.1.1.1.2.2.2.3.cmml">(</mo><msub id="S2.Ex5.m1.1.1.1.1.1.1.1.1.1" xref="S2.Ex5.m1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S2.Ex5.m1.1.1.1.1.1.1.1.1.1.2" xref="S2.Ex5.m1.1.1.1.1.1.1.1.1.1.2.cmml">W</mi><mn id="S2.Ex5.m1.1.1.1.1.1.1.1.1.1.3" xref="S2.Ex5.m1.1.1.1.1.1.1.1.1.1.3.cmml">0</mn></msub><mo id="S2.Ex5.m1.1.1.1.1.2.2.2.2.4" xref="S2.Ex5.m1.1.1.1.1.2.2.2.3.cmml">;</mo><msub id="S2.Ex5.m1.1.1.1.1.2.2.2.2.2" xref="S2.Ex5.m1.1.1.1.1.2.2.2.2.2.cmml"><mi id="S2.Ex5.m1.1.1.1.1.2.2.2.2.2.2" xref="S2.Ex5.m1.1.1.1.1.2.2.2.2.2.2.cmml">x</mi><mi id="S2.Ex5.m1.1.1.1.1.2.2.2.2.2.3" xref="S2.Ex5.m1.1.1.1.1.2.2.2.2.2.3.cmml">t</mi></msub><mo id="S2.Ex5.m1.1.1.1.1.2.2.2.2.5" xref="S2.Ex5.m1.1.1.1.1.2.2.2.3.cmml">)</mo></mrow></mrow></mrow><mo id="S2.Ex5.m1.1.1.1.1.10" xref="S2.Ex5.m1.1.1.1.1.10.cmml">=</mo><mrow id="S2.Ex5.m1.1.1.1.1.4" xref="S2.Ex5.m1.1.1.1.1.4.cmml"><msub id="S2.Ex5.m1.1.1.1.1.4.4" xref="S2.Ex5.m1.1.1.1.1.4.4.cmml"><mi id="S2.Ex5.m1.1.1.1.1.4.4.2" xref="S2.Ex5.m1.1.1.1.1.4.4.2.cmml">W</mi><mn id="S2.Ex5.m1.1.1.1.1.4.4.3" xref="S2.Ex5.m1.1.1.1.1.4.4.3.cmml">0</mn></msub><mo id="S2.Ex5.m1.1.1.1.1.4.3" xref="S2.Ex5.m1.1.1.1.1.4.3.cmml">−</mo><mrow id="S2.Ex5.m1.1.1.1.1.4.2" xref="S2.Ex5.m1.1.1.1.1.4.2.cmml"><mi id="S2.Ex5.m1.1.1.1.1.4.2.4" xref="S2.Ex5.m1.1.1.1.1.4.2.4.cmml">η</mi><mo lspace="0em" rspace="0em" id="S2.Ex5.m1.1.1.1.1.4.2.3" xref="S2.Ex5.m1.1.1.1.1.4.2.3.cmml">​</mo><mrow id="S2.Ex5.m1.1.1.1.1.4.2.2" xref="S2.Ex5.m1.1.1.1.1.4.2.2.cmml"><munderover id="S2.Ex5.m1.1.1.1.1.4.2.2.3" xref="S2.Ex5.m1.1.1.1.1.4.2.2.3.cmml"><mo movablelimits="false" id="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.2" xref="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.2.cmml">∑</mo><mrow id="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.3" xref="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.3.cmml"><mi id="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.3.2" xref="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.3.2.cmml">s</mi><mo id="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.3.1" xref="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.3.1.cmml">=</mo><mn id="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.3.3" xref="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.3.3.cmml">1</mn></mrow><mi id="S2.Ex5.m1.1.1.1.1.4.2.2.3.3" xref="S2.Ex5.m1.1.1.1.1.4.2.2.3.3.cmml">t</mi></munderover><mrow id="S2.Ex5.m1.1.1.1.1.4.2.2.2" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2.cmml"><mrow id="S2.Ex5.m1.1.1.1.1.4.2.2.2.4" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2.4.cmml"><mo rspace="0.167em" id="S2.Ex5.m1.1.1.1.1.4.2.2.2.4.1" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2.4.1.cmml">∇</mo><mi mathvariant="normal" id="S2.Ex5.m1.1.1.1.1.4.2.2.2.4.2" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2.4.2.cmml">ℓ</mi></mrow><mo lspace="0em" rspace="0em" id="S2.Ex5.m1.1.1.1.1.4.2.2.2.3" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2.3.cmml">​</mo><mrow id="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.2" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.3.cmml"><mo id="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.2.3" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.3.cmml">(</mo><msub id="S2.Ex5.m1.1.1.1.1.3.1.1.1.1.1.1" xref="S2.Ex5.m1.1.1.1.1.3.1.1.1.1.1.1.cmml"><mi id="S2.Ex5.m1.1.1.1.1.3.1.1.1.1.1.1.2" xref="S2.Ex5.m1.1.1.1.1.3.1.1.1.1.1.1.2.cmml">W</mi><mn id="S2.Ex5.m1.1.1.1.1.3.1.1.1.1.1.1.3" xref="S2.Ex5.m1.1.1.1.1.3.1.1.1.1.1.1.3.cmml">0</mn></msub><mo id="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.2.4" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.3.cmml">;</mo><msub id="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.2.2" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.2.2.cmml"><mi id="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.2.2.2" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.2.2.2.cmml">x</mi><mi id="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.2.2.3" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.2.2.3.cmml">s</mi></msub><mo id="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.2.5" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.3.cmml">)</mo></mrow></mrow></mrow></mrow></mrow><mo rspace="0.111em" id="S2.Ex5.m1.1.1.1.1.11" xref="S2.Ex5.m1.1.1.1.1.11.cmml">=</mo><mrow id="S2.Ex5.m1.1.1.1.1.6" xref="S2.Ex5.m1.1.1.1.1.6.cmml"><munderover id="S2.Ex5.m1.1.1.1.1.6.3" xref="S2.Ex5.m1.1.1.1.1.6.3.cmml"><mo movablelimits="false" rspace="0em" id="S2.Ex5.m1.1.1.1.1.6.3.2.2" xref="S2.Ex5.m1.1.1.1.1.6.3.2.2.cmml">∑</mo><mrow id="S2.Ex5.m1.1.1.1.1.6.3.2.3" xref="S2.Ex5.m1.1.1.1.1.6.3.2.3.cmml"><mi id="S2.Ex5.m1.1.1.1.1.6.3.2.3.2" xref="S2.Ex5.m1.1.1.1.1.6.3.2.3.2.cmml">s</mi><mo id="S2.Ex5.m1.1.1.1.1.6.3.2.3.1" xref="S2.Ex5.m1.1.1.1.1.6.3.2.3.1.cmml">=</mo><mn id="S2.Ex5.m1.1.1.1.1.6.3.2.3.3" xref="S2.Ex5.m1.1.1.1.1.6.3.2.3.3.cmml">1</mn></mrow><mi id="S2.Ex5.m1.1.1.1.1.6.3.3" xref="S2.Ex5.m1.1.1.1.1.6.3.3.cmml">t</mi></munderover><mrow id="S2.Ex5.m1.1.1.1.1.6.2" xref="S2.Ex5.m1.1.1.1.1.6.2.cmml"><mrow id="S2.Ex5.m1.1.1.1.1.5.1.1.1" xref="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.cmml"><mo stretchy="false" id="S2.Ex5.m1.1.1.1.1.5.1.1.1.2" xref="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.cmml">(</mo><mrow id="S2.Ex5.m1.1.1.1.1.5.1.1.1.1" xref="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.cmml"><msub id="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.2" xref="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.2.cmml"><mi id="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.2.2" xref="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.2.2.cmml">θ</mi><mi id="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.2.3" xref="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.2.3.cmml">V</mi></msub><mo lspace="0em" rspace="0em" id="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.1" xref="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.1.cmml">​</mo><msub id="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.3" xref="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.3.cmml"><mi id="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.3.2" xref="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.3.2.cmml">x</mi><mi id="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.3.3" xref="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.3.3.cmml">s</mi></msub></mrow><mo stretchy="false" id="S2.Ex5.m1.1.1.1.1.5.1.1.1.3" xref="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S2.Ex5.m1.1.1.1.1.6.2.3" xref="S2.Ex5.m1.1.1.1.1.6.2.3.cmml">​</mo><msup id="S2.Ex5.m1.1.1.1.1.6.2.2" xref="S2.Ex5.m1.1.1.1.1.6.2.2.cmml"><mrow id="S2.Ex5.m1.1.1.1.1.6.2.2.1.1" xref="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.cmml"><mo stretchy="false" id="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.2" xref="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.cmml">(</mo><mrow id="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1" xref="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.cmml"><msub id="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.2" xref="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.2.cmml"><mi id="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.2.2" xref="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.2.2.cmml">θ</mi><mi id="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.2.3" xref="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.2.3.cmml">K</mi></msub><mo lspace="0em" rspace="0em" id="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.1" xref="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.1.cmml">​</mo><msub id="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.3" xref="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.3.cmml"><mi id="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.3.2" xref="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.3.2.cmml">x</mi><mi id="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.3.3" xref="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.3.3.cmml">s</mi></msub></mrow><mo stretchy="false" id="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.3" xref="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.cmml">)</mo></mrow><mi id="S2.Ex5.m1.1.1.1.1.6.2.2.3" xref="S2.Ex5.m1.1.1.1.1.6.2.2.3.cmml">T</mi></msup></mrow></mrow></mrow><mo lspace="0em" id="S2.Ex5.m1.1.1.1.2" xref="S2.Ex5.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.Ex5.m1.1b"><apply id="S2.Ex5.m1.1.1.1.1.cmml" xref="S2.Ex5.m1.1.1.1"><and id="S2.Ex5.m1.1.1.1.1a.cmml" xref="S2.Ex5.m1.1.1.1"></and><apply id="S2.Ex5.m1.1.1.1.1b.cmml" xref="S2.Ex5.m1.1.1.1"><eq id="S2.Ex5.m1.1.1.1.1.9.cmml" xref="S2.Ex5.m1.1.1.1.1.9"></eq><apply id="S2.Ex5.m1.1.1.1.1.8.cmml" xref="S2.Ex5.m1.1.1.1.1.8"><csymbol cd="ambiguous" id="S2.Ex5.m1.1.1.1.1.8.1.cmml" xref="S2.Ex5.m1.1.1.1.1.8">subscript</csymbol><ci id="S2.Ex5.m1.1.1.1.1.8.2.cmml" xref="S2.Ex5.m1.1.1.1.1.8.2">𝑊</ci><ci id="S2.Ex5.m1.1.1.1.1.8.3.cmml" xref="S2.Ex5.m1.1.1.1.1.8.3">𝑡</ci></apply><apply id="S2.Ex5.m1.1.1.1.1.2.cmml" xref="S2.Ex5.m1.1.1.1.1.2"><minus id="S2.Ex5.m1.1.1.1.1.2.3.cmml" xref="S2.Ex5.m1.1.1.1.1.2.3"></minus><apply id="S2.Ex5.m1.1.1.1.1.2.4.cmml" xref="S2.Ex5.m1.1.1.1.1.2.4"><csymbol cd="ambiguous" id="S2.Ex5.m1.1.1.1.1.2.4.1.cmml" xref="S2.Ex5.m1.1.1.1.1.2.4">subscript</csymbol><ci id="S2.Ex5.m1.1.1.1.1.2.4.2.cmml" xref="S2.Ex5.m1.1.1.1.1.2.4.2">𝑊</ci><apply id="S2.Ex5.m1.1.1.1.1.2.4.3.cmml" xref="S2.Ex5.m1.1.1.1.1.2.4.3"><minus id="S2.Ex5.m1.1.1.1.1.2.4.3.1.cmml" xref="S2.Ex5.m1.1.1.1.1.2.4.3.1"></minus><ci id="S2.Ex5.m1.1.1.1.1.2.4.3.2.cmml" xref="S2.Ex5.m1.1.1.1.1.2.4.3.2">𝑡</ci><cn type="integer" id="S2.Ex5.m1.1.1.1.1.2.4.3.3.cmml" xref="S2.Ex5.m1.1.1.1.1.2.4.3.3">1</cn></apply></apply><apply id="S2.Ex5.m1.1.1.1.1.2.2.cmml" xref="S2.Ex5.m1.1.1.1.1.2.2"><times id="S2.Ex5.m1.1.1.1.1.2.2.3.cmml" xref="S2.Ex5.m1.1.1.1.1.2.2.3"></times><ci id="S2.Ex5.m1.1.1.1.1.2.2.4.cmml" xref="S2.Ex5.m1.1.1.1.1.2.2.4">𝜂</ci><apply id="S2.Ex5.m1.1.1.1.1.2.2.5.cmml" xref="S2.Ex5.m1.1.1.1.1.2.2.5"><ci id="S2.Ex5.m1.1.1.1.1.2.2.5.1.cmml" xref="S2.Ex5.m1.1.1.1.1.2.2.5.1">∇</ci><ci id="S2.Ex5.m1.1.1.1.1.2.2.5.2.cmml" xref="S2.Ex5.m1.1.1.1.1.2.2.5.2">ℓ</ci></apply><list id="S2.Ex5.m1.1.1.1.1.2.2.2.3.cmml" xref="S2.Ex5.m1.1.1.1.1.2.2.2.2"><apply id="S2.Ex5.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex5.m1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.Ex5.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex5.m1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.Ex5.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.Ex5.m1.1.1.1.1.1.1.1.1.1.2">𝑊</ci><cn type="integer" id="S2.Ex5.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.Ex5.m1.1.1.1.1.1.1.1.1.1.3">0</cn></apply><apply id="S2.Ex5.m1.1.1.1.1.2.2.2.2.2.cmml" xref="S2.Ex5.m1.1.1.1.1.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.Ex5.m1.1.1.1.1.2.2.2.2.2.1.cmml" xref="S2.Ex5.m1.1.1.1.1.2.2.2.2.2">subscript</csymbol><ci id="S2.Ex5.m1.1.1.1.1.2.2.2.2.2.2.cmml" xref="S2.Ex5.m1.1.1.1.1.2.2.2.2.2.2">𝑥</ci><ci id="S2.Ex5.m1.1.1.1.1.2.2.2.2.2.3.cmml" xref="S2.Ex5.m1.1.1.1.1.2.2.2.2.2.3">𝑡</ci></apply></list></apply></apply></apply><apply id="S2.Ex5.m1.1.1.1.1c.cmml" xref="S2.Ex5.m1.1.1.1"><eq id="S2.Ex5.m1.1.1.1.1.10.cmml" xref="S2.Ex5.m1.1.1.1.1.10"></eq><share href="#S2.Ex5.m1.1.1.1.1.2.cmml" id="S2.Ex5.m1.1.1.1.1d.cmml" xref="S2.Ex5.m1.1.1.1"></share><apply id="S2.Ex5.m1.1.1.1.1.4.cmml" xref="S2.Ex5.m1.1.1.1.1.4"><minus id="S2.Ex5.m1.1.1.1.1.4.3.cmml" xref="S2.Ex5.m1.1.1.1.1.4.3"></minus><apply id="S2.Ex5.m1.1.1.1.1.4.4.cmml" xref="S2.Ex5.m1.1.1.1.1.4.4"><csymbol cd="ambiguous" id="S2.Ex5.m1.1.1.1.1.4.4.1.cmml" xref="S2.Ex5.m1.1.1.1.1.4.4">subscript</csymbol><ci id="S2.Ex5.m1.1.1.1.1.4.4.2.cmml" xref="S2.Ex5.m1.1.1.1.1.4.4.2">𝑊</ci><cn type="integer" id="S2.Ex5.m1.1.1.1.1.4.4.3.cmml" xref="S2.Ex5.m1.1.1.1.1.4.4.3">0</cn></apply><apply id="S2.Ex5.m1.1.1.1.1.4.2.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2"><times id="S2.Ex5.m1.1.1.1.1.4.2.3.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.3"></times><ci id="S2.Ex5.m1.1.1.1.1.4.2.4.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.4">𝜂</ci><apply id="S2.Ex5.m1.1.1.1.1.4.2.2.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.2"><apply id="S2.Ex5.m1.1.1.1.1.4.2.2.3.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.2.3"><csymbol cd="ambiguous" id="S2.Ex5.m1.1.1.1.1.4.2.2.3.1.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.2.3">superscript</csymbol><apply id="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.2.3"><csymbol cd="ambiguous" id="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.1.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.2.3">subscript</csymbol><sum id="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.2.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.2"></sum><apply id="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.3.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.3"><eq id="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.3.1.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.3.1"></eq><ci id="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.3.2.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.3.2">𝑠</ci><cn type="integer" id="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.3.3.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.2.3.2.3.3">1</cn></apply></apply><ci id="S2.Ex5.m1.1.1.1.1.4.2.2.3.3.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.2.3.3">𝑡</ci></apply><apply id="S2.Ex5.m1.1.1.1.1.4.2.2.2.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2"><times id="S2.Ex5.m1.1.1.1.1.4.2.2.2.3.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2.3"></times><apply id="S2.Ex5.m1.1.1.1.1.4.2.2.2.4.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2.4"><ci id="S2.Ex5.m1.1.1.1.1.4.2.2.2.4.1.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2.4.1">∇</ci><ci id="S2.Ex5.m1.1.1.1.1.4.2.2.2.4.2.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2.4.2">ℓ</ci></apply><list id="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.3.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.2"><apply id="S2.Ex5.m1.1.1.1.1.3.1.1.1.1.1.1.cmml" xref="S2.Ex5.m1.1.1.1.1.3.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.Ex5.m1.1.1.1.1.3.1.1.1.1.1.1.1.cmml" xref="S2.Ex5.m1.1.1.1.1.3.1.1.1.1.1.1">subscript</csymbol><ci id="S2.Ex5.m1.1.1.1.1.3.1.1.1.1.1.1.2.cmml" xref="S2.Ex5.m1.1.1.1.1.3.1.1.1.1.1.1.2">𝑊</ci><cn type="integer" id="S2.Ex5.m1.1.1.1.1.3.1.1.1.1.1.1.3.cmml" xref="S2.Ex5.m1.1.1.1.1.3.1.1.1.1.1.1.3">0</cn></apply><apply id="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.2.2.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.2.2.1.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.2.2">subscript</csymbol><ci id="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.2.2.2.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.2.2.2">𝑥</ci><ci id="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.2.2.3.cmml" xref="S2.Ex5.m1.1.1.1.1.4.2.2.2.2.2.2.3">𝑠</ci></apply></list></apply></apply></apply></apply></apply><apply id="S2.Ex5.m1.1.1.1.1e.cmml" xref="S2.Ex5.m1.1.1.1"><eq id="S2.Ex5.m1.1.1.1.1.11.cmml" xref="S2.Ex5.m1.1.1.1.1.11"></eq><share href="#S2.Ex5.m1.1.1.1.1.4.cmml" id="S2.Ex5.m1.1.1.1.1f.cmml" xref="S2.Ex5.m1.1.1.1"></share><apply id="S2.Ex5.m1.1.1.1.1.6.cmml" xref="S2.Ex5.m1.1.1.1.1.6"><apply id="S2.Ex5.m1.1.1.1.1.6.3.cmml" xref="S2.Ex5.m1.1.1.1.1.6.3"><csymbol cd="ambiguous" id="S2.Ex5.m1.1.1.1.1.6.3.1.cmml" xref="S2.Ex5.m1.1.1.1.1.6.3">superscript</csymbol><apply id="S2.Ex5.m1.1.1.1.1.6.3.2.cmml" xref="S2.Ex5.m1.1.1.1.1.6.3"><csymbol cd="ambiguous" id="S2.Ex5.m1.1.1.1.1.6.3.2.1.cmml" xref="S2.Ex5.m1.1.1.1.1.6.3">subscript</csymbol><sum id="S2.Ex5.m1.1.1.1.1.6.3.2.2.cmml" xref="S2.Ex5.m1.1.1.1.1.6.3.2.2"></sum><apply id="S2.Ex5.m1.1.1.1.1.6.3.2.3.cmml" xref="S2.Ex5.m1.1.1.1.1.6.3.2.3"><eq id="S2.Ex5.m1.1.1.1.1.6.3.2.3.1.cmml" xref="S2.Ex5.m1.1.1.1.1.6.3.2.3.1"></eq><ci id="S2.Ex5.m1.1.1.1.1.6.3.2.3.2.cmml" xref="S2.Ex5.m1.1.1.1.1.6.3.2.3.2">𝑠</ci><cn type="integer" id="S2.Ex5.m1.1.1.1.1.6.3.2.3.3.cmml" xref="S2.Ex5.m1.1.1.1.1.6.3.2.3.3">1</cn></apply></apply><ci id="S2.Ex5.m1.1.1.1.1.6.3.3.cmml" xref="S2.Ex5.m1.1.1.1.1.6.3.3">𝑡</ci></apply><apply id="S2.Ex5.m1.1.1.1.1.6.2.cmml" xref="S2.Ex5.m1.1.1.1.1.6.2"><times id="S2.Ex5.m1.1.1.1.1.6.2.3.cmml" xref="S2.Ex5.m1.1.1.1.1.6.2.3"></times><apply id="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.cmml" xref="S2.Ex5.m1.1.1.1.1.5.1.1.1"><times id="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.1.cmml" xref="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.1"></times><apply id="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.2.cmml" xref="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.2.1.cmml" xref="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.2">subscript</csymbol><ci id="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.2.2.cmml" xref="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.2.2">𝜃</ci><ci id="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.2.3.cmml" xref="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.2.3">𝑉</ci></apply><apply id="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.3.cmml" xref="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.3.1.cmml" xref="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.3">subscript</csymbol><ci id="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.3.2.cmml" xref="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.3.2">𝑥</ci><ci id="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.3.3.cmml" xref="S2.Ex5.m1.1.1.1.1.5.1.1.1.1.3.3">𝑠</ci></apply></apply><apply id="S2.Ex5.m1.1.1.1.1.6.2.2.cmml" xref="S2.Ex5.m1.1.1.1.1.6.2.2"><csymbol cd="ambiguous" id="S2.Ex5.m1.1.1.1.1.6.2.2.2.cmml" xref="S2.Ex5.m1.1.1.1.1.6.2.2">superscript</csymbol><apply id="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.cmml" xref="S2.Ex5.m1.1.1.1.1.6.2.2.1.1"><times id="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.1.cmml" xref="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.1"></times><apply id="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.2.cmml" xref="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.2"><csymbol cd="ambiguous" id="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.2.1.cmml" xref="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.2">subscript</csymbol><ci id="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.2.2.cmml" xref="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.2.2">𝜃</ci><ci id="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.2.3.cmml" xref="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.2.3">𝐾</ci></apply><apply id="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.3.cmml" xref="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.3.1.cmml" xref="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.3">subscript</csymbol><ci id="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.3.2.cmml" xref="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.3.2">𝑥</ci><ci id="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.3.3.cmml" xref="S2.Ex5.m1.1.1.1.1.6.2.2.1.1.1.3.3">𝑠</ci></apply></apply><ci id="S2.Ex5.m1.1.1.1.1.6.2.2.3.cmml" xref="S2.Ex5.m1.1.1.1.1.6.2.2.3">𝑇</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex5.m1.1c">W_{t}=W_{t-1}-\eta\nabla\ell\left(W_{0};x_{t}\right)=W_{0}-\eta\sum_{s=1}^{t}\nabla\ell\left(W_{0};x_{s}\right)=\sum_{s=1}^{t}(\theta_{V}x_{s})(\theta_{K}x_{s})^{T}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S2.SS6.1.p1.3" class="ltx_p">Plugging <math id="S2.SS6.1.p1.3.m1.1" class="ltx_Math" alttext="W_{t}" display="inline"><semantics id="S2.SS6.1.p1.3.m1.1a"><msub id="S2.SS6.1.p1.3.m1.1.1" xref="S2.SS6.1.p1.3.m1.1.1.cmml"><mi id="S2.SS6.1.p1.3.m1.1.1.2" xref="S2.SS6.1.p1.3.m1.1.1.2.cmml">W</mi><mi id="S2.SS6.1.p1.3.m1.1.1.3" xref="S2.SS6.1.p1.3.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS6.1.p1.3.m1.1b"><apply id="S2.SS6.1.p1.3.m1.1.1.cmml" xref="S2.SS6.1.p1.3.m1.1.1"><csymbol cd="ambiguous" id="S2.SS6.1.p1.3.m1.1.1.1.cmml" xref="S2.SS6.1.p1.3.m1.1.1">subscript</csymbol><ci id="S2.SS6.1.p1.3.m1.1.1.2.cmml" xref="S2.SS6.1.p1.3.m1.1.1.2">𝑊</ci><ci id="S2.SS6.1.p1.3.m1.1.1.3.cmml" xref="S2.SS6.1.p1.3.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.1.p1.3.m1.1c">W_{t}</annotation></semantics></math> into the output rule in Equation <a href="#S2.E5" title="In 2.3 Learning a self-supervised task for TTT ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, we obtain the output token:</p>
<table id="S2.Ex6" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.Ex6.m1.1" class="ltx_Math" alttext="z_{t}=f\left(\theta_{Q}x_{t};W_{t}\right)=\sum_{s=1}^{t}(\theta_{V}x_{s})(\theta_{K}x_{s})^{T}(\theta_{Q}x_{t}),\vspace{-1ex}" display="block"><semantics id="S2.Ex6.m1.1a"><mrow id="S2.Ex6.m1.1.1.1" xref="S2.Ex6.m1.1.1.1.1.cmml"><mrow id="S2.Ex6.m1.1.1.1.1" xref="S2.Ex6.m1.1.1.1.1.cmml"><msub id="S2.Ex6.m1.1.1.1.1.7" xref="S2.Ex6.m1.1.1.1.1.7.cmml"><mi id="S2.Ex6.m1.1.1.1.1.7.2" xref="S2.Ex6.m1.1.1.1.1.7.2.cmml">z</mi><mi id="S2.Ex6.m1.1.1.1.1.7.3" xref="S2.Ex6.m1.1.1.1.1.7.3.cmml">t</mi></msub><mo id="S2.Ex6.m1.1.1.1.1.8" xref="S2.Ex6.m1.1.1.1.1.8.cmml">=</mo><mrow id="S2.Ex6.m1.1.1.1.1.2" xref="S2.Ex6.m1.1.1.1.1.2.cmml"><mi id="S2.Ex6.m1.1.1.1.1.2.4" xref="S2.Ex6.m1.1.1.1.1.2.4.cmml">f</mi><mo lspace="0em" rspace="0em" id="S2.Ex6.m1.1.1.1.1.2.3" xref="S2.Ex6.m1.1.1.1.1.2.3.cmml">​</mo><mrow id="S2.Ex6.m1.1.1.1.1.2.2.2" xref="S2.Ex6.m1.1.1.1.1.2.2.3.cmml"><mo id="S2.Ex6.m1.1.1.1.1.2.2.2.3" xref="S2.Ex6.m1.1.1.1.1.2.2.3.cmml">(</mo><mrow id="S2.Ex6.m1.1.1.1.1.1.1.1.1" xref="S2.Ex6.m1.1.1.1.1.1.1.1.1.cmml"><msub id="S2.Ex6.m1.1.1.1.1.1.1.1.1.2" xref="S2.Ex6.m1.1.1.1.1.1.1.1.1.2.cmml"><mi id="S2.Ex6.m1.1.1.1.1.1.1.1.1.2.2" xref="S2.Ex6.m1.1.1.1.1.1.1.1.1.2.2.cmml">θ</mi><mi id="S2.Ex6.m1.1.1.1.1.1.1.1.1.2.3" xref="S2.Ex6.m1.1.1.1.1.1.1.1.1.2.3.cmml">Q</mi></msub><mo lspace="0em" rspace="0em" id="S2.Ex6.m1.1.1.1.1.1.1.1.1.1" xref="S2.Ex6.m1.1.1.1.1.1.1.1.1.1.cmml">​</mo><msub id="S2.Ex6.m1.1.1.1.1.1.1.1.1.3" xref="S2.Ex6.m1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S2.Ex6.m1.1.1.1.1.1.1.1.1.3.2" xref="S2.Ex6.m1.1.1.1.1.1.1.1.1.3.2.cmml">x</mi><mi id="S2.Ex6.m1.1.1.1.1.1.1.1.1.3.3" xref="S2.Ex6.m1.1.1.1.1.1.1.1.1.3.3.cmml">t</mi></msub></mrow><mo id="S2.Ex6.m1.1.1.1.1.2.2.2.4" xref="S2.Ex6.m1.1.1.1.1.2.2.3.cmml">;</mo><msub id="S2.Ex6.m1.1.1.1.1.2.2.2.2" xref="S2.Ex6.m1.1.1.1.1.2.2.2.2.cmml"><mi id="S2.Ex6.m1.1.1.1.1.2.2.2.2.2" xref="S2.Ex6.m1.1.1.1.1.2.2.2.2.2.cmml">W</mi><mi id="S2.Ex6.m1.1.1.1.1.2.2.2.2.3" xref="S2.Ex6.m1.1.1.1.1.2.2.2.2.3.cmml">t</mi></msub><mo id="S2.Ex6.m1.1.1.1.1.2.2.2.5" xref="S2.Ex6.m1.1.1.1.1.2.2.3.cmml">)</mo></mrow></mrow><mo rspace="0.111em" id="S2.Ex6.m1.1.1.1.1.9" xref="S2.Ex6.m1.1.1.1.1.9.cmml">=</mo><mrow id="S2.Ex6.m1.1.1.1.1.5" xref="S2.Ex6.m1.1.1.1.1.5.cmml"><munderover id="S2.Ex6.m1.1.1.1.1.5.4" xref="S2.Ex6.m1.1.1.1.1.5.4.cmml"><mo movablelimits="false" rspace="0em" id="S2.Ex6.m1.1.1.1.1.5.4.2.2" xref="S2.Ex6.m1.1.1.1.1.5.4.2.2.cmml">∑</mo><mrow id="S2.Ex6.m1.1.1.1.1.5.4.2.3" xref="S2.Ex6.m1.1.1.1.1.5.4.2.3.cmml"><mi id="S2.Ex6.m1.1.1.1.1.5.4.2.3.2" xref="S2.Ex6.m1.1.1.1.1.5.4.2.3.2.cmml">s</mi><mo id="S2.Ex6.m1.1.1.1.1.5.4.2.3.1" xref="S2.Ex6.m1.1.1.1.1.5.4.2.3.1.cmml">=</mo><mn id="S2.Ex6.m1.1.1.1.1.5.4.2.3.3" xref="S2.Ex6.m1.1.1.1.1.5.4.2.3.3.cmml">1</mn></mrow><mi id="S2.Ex6.m1.1.1.1.1.5.4.3" xref="S2.Ex6.m1.1.1.1.1.5.4.3.cmml">t</mi></munderover><mrow id="S2.Ex6.m1.1.1.1.1.5.3" xref="S2.Ex6.m1.1.1.1.1.5.3.cmml"><mrow id="S2.Ex6.m1.1.1.1.1.3.1.1.1" xref="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.cmml"><mo stretchy="false" id="S2.Ex6.m1.1.1.1.1.3.1.1.1.2" xref="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.cmml">(</mo><mrow id="S2.Ex6.m1.1.1.1.1.3.1.1.1.1" xref="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.cmml"><msub id="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.2" xref="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.2.cmml"><mi id="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.2.2" xref="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.2.2.cmml">θ</mi><mi id="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.2.3" xref="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.2.3.cmml">V</mi></msub><mo lspace="0em" rspace="0em" id="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.1" xref="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.1.cmml">​</mo><msub id="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.3" xref="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.3.cmml"><mi id="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.3.2" xref="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.3.2.cmml">x</mi><mi id="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.3.3" xref="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.3.3.cmml">s</mi></msub></mrow><mo stretchy="false" id="S2.Ex6.m1.1.1.1.1.3.1.1.1.3" xref="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="S2.Ex6.m1.1.1.1.1.5.3.4" xref="S2.Ex6.m1.1.1.1.1.5.3.4.cmml">​</mo><msup id="S2.Ex6.m1.1.1.1.1.4.2.2" xref="S2.Ex6.m1.1.1.1.1.4.2.2.cmml"><mrow id="S2.Ex6.m1.1.1.1.1.4.2.2.1.1" xref="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.cmml"><mo stretchy="false" id="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.2" xref="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.cmml">(</mo><mrow id="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1" xref="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.cmml"><msub id="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.2" xref="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.2.cmml"><mi id="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.2.2" xref="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.2.2.cmml">θ</mi><mi id="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.2.3" xref="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.2.3.cmml">K</mi></msub><mo lspace="0em" rspace="0em" id="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.1" xref="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.1.cmml">​</mo><msub id="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.3" xref="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.3.cmml"><mi id="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.3.2" xref="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.3.2.cmml">x</mi><mi id="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.3.3" xref="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.3.3.cmml">s</mi></msub></mrow><mo stretchy="false" id="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.3" xref="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.cmml">)</mo></mrow><mi id="S2.Ex6.m1.1.1.1.1.4.2.2.3" xref="S2.Ex6.m1.1.1.1.1.4.2.2.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="S2.Ex6.m1.1.1.1.1.5.3.4a" xref="S2.Ex6.m1.1.1.1.1.5.3.4.cmml">​</mo><mrow id="S2.Ex6.m1.1.1.1.1.5.3.3.1" xref="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.cmml"><mo stretchy="false" id="S2.Ex6.m1.1.1.1.1.5.3.3.1.2" xref="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.cmml">(</mo><mrow id="S2.Ex6.m1.1.1.1.1.5.3.3.1.1" xref="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.cmml"><msub id="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.2" xref="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.2.cmml"><mi id="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.2.2" xref="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.2.2.cmml">θ</mi><mi id="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.2.3" xref="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.2.3.cmml">Q</mi></msub><mo lspace="0em" rspace="0em" id="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.1" xref="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.1.cmml">​</mo><msub id="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.3" xref="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.3.cmml"><mi id="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.3.2" xref="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.3.2.cmml">x</mi><mi id="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.3.3" xref="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.3.3.cmml">t</mi></msub></mrow><mo stretchy="false" id="S2.Ex6.m1.1.1.1.1.5.3.3.1.3" xref="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><mo id="S2.Ex6.m1.1.1.1.2" xref="S2.Ex6.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.Ex6.m1.1b"><apply id="S2.Ex6.m1.1.1.1.1.cmml" xref="S2.Ex6.m1.1.1.1"><and id="S2.Ex6.m1.1.1.1.1a.cmml" xref="S2.Ex6.m1.1.1.1"></and><apply id="S2.Ex6.m1.1.1.1.1b.cmml" xref="S2.Ex6.m1.1.1.1"><eq id="S2.Ex6.m1.1.1.1.1.8.cmml" xref="S2.Ex6.m1.1.1.1.1.8"></eq><apply id="S2.Ex6.m1.1.1.1.1.7.cmml" xref="S2.Ex6.m1.1.1.1.1.7"><csymbol cd="ambiguous" id="S2.Ex6.m1.1.1.1.1.7.1.cmml" xref="S2.Ex6.m1.1.1.1.1.7">subscript</csymbol><ci id="S2.Ex6.m1.1.1.1.1.7.2.cmml" xref="S2.Ex6.m1.1.1.1.1.7.2">𝑧</ci><ci id="S2.Ex6.m1.1.1.1.1.7.3.cmml" xref="S2.Ex6.m1.1.1.1.1.7.3">𝑡</ci></apply><apply id="S2.Ex6.m1.1.1.1.1.2.cmml" xref="S2.Ex6.m1.1.1.1.1.2"><times id="S2.Ex6.m1.1.1.1.1.2.3.cmml" xref="S2.Ex6.m1.1.1.1.1.2.3"></times><ci id="S2.Ex6.m1.1.1.1.1.2.4.cmml" xref="S2.Ex6.m1.1.1.1.1.2.4">𝑓</ci><list id="S2.Ex6.m1.1.1.1.1.2.2.3.cmml" xref="S2.Ex6.m1.1.1.1.1.2.2.2"><apply id="S2.Ex6.m1.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex6.m1.1.1.1.1.1.1.1.1"><times id="S2.Ex6.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex6.m1.1.1.1.1.1.1.1.1.1"></times><apply id="S2.Ex6.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.Ex6.m1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.Ex6.m1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.Ex6.m1.1.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S2.Ex6.m1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S2.Ex6.m1.1.1.1.1.1.1.1.1.2.2">𝜃</ci><ci id="S2.Ex6.m1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S2.Ex6.m1.1.1.1.1.1.1.1.1.2.3">𝑄</ci></apply><apply id="S2.Ex6.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.Ex6.m1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex6.m1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S2.Ex6.m1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.Ex6.m1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S2.Ex6.m1.1.1.1.1.1.1.1.1.3.2">𝑥</ci><ci id="S2.Ex6.m1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S2.Ex6.m1.1.1.1.1.1.1.1.1.3.3">𝑡</ci></apply></apply><apply id="S2.Ex6.m1.1.1.1.1.2.2.2.2.cmml" xref="S2.Ex6.m1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S2.Ex6.m1.1.1.1.1.2.2.2.2.1.cmml" xref="S2.Ex6.m1.1.1.1.1.2.2.2.2">subscript</csymbol><ci id="S2.Ex6.m1.1.1.1.1.2.2.2.2.2.cmml" xref="S2.Ex6.m1.1.1.1.1.2.2.2.2.2">𝑊</ci><ci id="S2.Ex6.m1.1.1.1.1.2.2.2.2.3.cmml" xref="S2.Ex6.m1.1.1.1.1.2.2.2.2.3">𝑡</ci></apply></list></apply></apply><apply id="S2.Ex6.m1.1.1.1.1c.cmml" xref="S2.Ex6.m1.1.1.1"><eq id="S2.Ex6.m1.1.1.1.1.9.cmml" xref="S2.Ex6.m1.1.1.1.1.9"></eq><share href="#S2.Ex6.m1.1.1.1.1.2.cmml" id="S2.Ex6.m1.1.1.1.1d.cmml" xref="S2.Ex6.m1.1.1.1"></share><apply id="S2.Ex6.m1.1.1.1.1.5.cmml" xref="S2.Ex6.m1.1.1.1.1.5"><apply id="S2.Ex6.m1.1.1.1.1.5.4.cmml" xref="S2.Ex6.m1.1.1.1.1.5.4"><csymbol cd="ambiguous" id="S2.Ex6.m1.1.1.1.1.5.4.1.cmml" xref="S2.Ex6.m1.1.1.1.1.5.4">superscript</csymbol><apply id="S2.Ex6.m1.1.1.1.1.5.4.2.cmml" xref="S2.Ex6.m1.1.1.1.1.5.4"><csymbol cd="ambiguous" id="S2.Ex6.m1.1.1.1.1.5.4.2.1.cmml" xref="S2.Ex6.m1.1.1.1.1.5.4">subscript</csymbol><sum id="S2.Ex6.m1.1.1.1.1.5.4.2.2.cmml" xref="S2.Ex6.m1.1.1.1.1.5.4.2.2"></sum><apply id="S2.Ex6.m1.1.1.1.1.5.4.2.3.cmml" xref="S2.Ex6.m1.1.1.1.1.5.4.2.3"><eq id="S2.Ex6.m1.1.1.1.1.5.4.2.3.1.cmml" xref="S2.Ex6.m1.1.1.1.1.5.4.2.3.1"></eq><ci id="S2.Ex6.m1.1.1.1.1.5.4.2.3.2.cmml" xref="S2.Ex6.m1.1.1.1.1.5.4.2.3.2">𝑠</ci><cn type="integer" id="S2.Ex6.m1.1.1.1.1.5.4.2.3.3.cmml" xref="S2.Ex6.m1.1.1.1.1.5.4.2.3.3">1</cn></apply></apply><ci id="S2.Ex6.m1.1.1.1.1.5.4.3.cmml" xref="S2.Ex6.m1.1.1.1.1.5.4.3">𝑡</ci></apply><apply id="S2.Ex6.m1.1.1.1.1.5.3.cmml" xref="S2.Ex6.m1.1.1.1.1.5.3"><times id="S2.Ex6.m1.1.1.1.1.5.3.4.cmml" xref="S2.Ex6.m1.1.1.1.1.5.3.4"></times><apply id="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.cmml" xref="S2.Ex6.m1.1.1.1.1.3.1.1.1"><times id="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.1.cmml" xref="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.1"></times><apply id="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.2.cmml" xref="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.2.1.cmml" xref="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.2">subscript</csymbol><ci id="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.2.2.cmml" xref="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.2.2">𝜃</ci><ci id="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.2.3.cmml" xref="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.2.3">𝑉</ci></apply><apply id="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.3.cmml" xref="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.3.1.cmml" xref="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.3">subscript</csymbol><ci id="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.3.2.cmml" xref="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.3.2">𝑥</ci><ci id="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.3.3.cmml" xref="S2.Ex6.m1.1.1.1.1.3.1.1.1.1.3.3">𝑠</ci></apply></apply><apply id="S2.Ex6.m1.1.1.1.1.4.2.2.cmml" xref="S2.Ex6.m1.1.1.1.1.4.2.2"><csymbol cd="ambiguous" id="S2.Ex6.m1.1.1.1.1.4.2.2.2.cmml" xref="S2.Ex6.m1.1.1.1.1.4.2.2">superscript</csymbol><apply id="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.cmml" xref="S2.Ex6.m1.1.1.1.1.4.2.2.1.1"><times id="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.1.cmml" xref="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.1"></times><apply id="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.2.cmml" xref="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.2"><csymbol cd="ambiguous" id="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.2.1.cmml" xref="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.2">subscript</csymbol><ci id="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.2.2.cmml" xref="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.2.2">𝜃</ci><ci id="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.2.3.cmml" xref="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.2.3">𝐾</ci></apply><apply id="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.3.cmml" xref="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.3.1.cmml" xref="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.3">subscript</csymbol><ci id="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.3.2.cmml" xref="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.3.2">𝑥</ci><ci id="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.3.3.cmml" xref="S2.Ex6.m1.1.1.1.1.4.2.2.1.1.1.3.3">𝑠</ci></apply></apply><ci id="S2.Ex6.m1.1.1.1.1.4.2.2.3.cmml" xref="S2.Ex6.m1.1.1.1.1.4.2.2.3">𝑇</ci></apply><apply id="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.cmml" xref="S2.Ex6.m1.1.1.1.1.5.3.3.1"><times id="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.1.cmml" xref="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.1"></times><apply id="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.2.cmml" xref="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.2"><csymbol cd="ambiguous" id="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.2.1.cmml" xref="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.2">subscript</csymbol><ci id="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.2.2.cmml" xref="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.2.2">𝜃</ci><ci id="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.2.3.cmml" xref="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.2.3">𝑄</ci></apply><apply id="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.3.cmml" xref="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.3"><csymbol cd="ambiguous" id="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.3.1.cmml" xref="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.3">subscript</csymbol><ci id="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.3.2.cmml" xref="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.3.2">𝑥</ci><ci id="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.3.3.cmml" xref="S2.Ex6.m1.1.1.1.1.5.3.3.1.1.3.3">𝑡</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex6.m1.1c">z_{t}=f\left(\theta_{Q}x_{t};W_{t}\right)=\sum_{s=1}^{t}(\theta_{V}x_{s})(\theta_{K}x_{s})^{T}(\theta_{Q}x_{t}),\vspace{-1ex}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S2.SS6.1.p1.4" class="ltx_p">which is the definition of linear attention.
∎</p>
</div>
</div>
<div id="S2.SS6.p2" class="ltx_para">
<p id="S2.SS6.p2.1" class="ltx_p">In Table <a href="#S2.T1" title="Table 1 ‣ 2.6 Theoretical equivalences ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, we first empirically verify the equivalence above with an improved implementation of linear attention.<span id="footnote8" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">8</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">8</sup><span class="ltx_tag ltx_tag_note">8</span>
The original formulation of linear attention in <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib41" title="" class="ltx_ref">41</a>]</cite> contains a normalizer and a feature expansion on <math id="footnote8.m1.1" class="ltx_Math" alttext="x_{t}" display="inline"><semantics id="footnote8.m1.1b"><msub id="footnote8.m1.1.1" xref="footnote8.m1.1.1.cmml"><mi id="footnote8.m1.1.1.2" xref="footnote8.m1.1.1.2.cmml">x</mi><mi id="footnote8.m1.1.1.3" xref="footnote8.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="footnote8.m1.1c"><apply id="footnote8.m1.1.1.cmml" xref="footnote8.m1.1.1"><csymbol cd="ambiguous" id="footnote8.m1.1.1.1.cmml" xref="footnote8.m1.1.1">subscript</csymbol><ci id="footnote8.m1.1.1.2.cmml" xref="footnote8.m1.1.1.2">𝑥</ci><ci id="footnote8.m1.1.1.3.cmml" xref="footnote8.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote8.m1.1d">x_{t}</annotation></semantics></math>, which can still be included in an equivalent TTT layer.
However, prior work has found that these two additions can hurt performance <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib58" title="" class="ltx_ref">58</a>]</cite>, which we have verified in our own experiment (first vs. second row of Table <a href="#S2.T1" title="Table 1 ‣ 2.6 Theoretical equivalences ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>).
Therefore, we only construct a TTT layer equivalent to the simplest formulation of linear attention without the two additions.
</span></span></span>
Then, to illustrate the contribution of each of our components (including some that will be introduced in the next subsection), we add them row by row to the TTT layer that is equivalent to linear attention, and ultimately obtain our proposed instantiation called <em id="S2.SS6.p2.1.1" class="ltx_emph ltx_font_italic">TTT-Linear</em>.
The change from batch GD to mini-batch GD contributes the most improvement by a large margin.</p>
</div>
<figure id="S2.F10.fig1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:164.8pt;">
<div id="S2.F10.1" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:260.2pt;">
<img src="/html/2407.04620/assets/x10.png" id="S2.F10.1.g1" class="ltx_graphics ltx_img_landscape" width="424" height="194" alt="Refer to caption">
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 10: </span>
RNN layers and TTT layers are both subsets of <span id="S2.F10.fig1.3.1" class="ltx_text">sequence modeling layers.</span>
RNN layers have a hidden state that is fixed in size across time.
TTT layers with parametric learners are also RNN layers, since their hidden state is also fixed in size.
TTT layers with nonparametric learners can represent self-attention, as discussed in Subsection <a href="#S2.SS6" title="2.6 Theoretical equivalences ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.6</span></a>.
</figcaption>
</figure>
<div id="S2.SS6.p3" class="ltx_para">
<p id="S2.SS6.p3.6" class="ltx_p">While the space of models <math id="S2.SS6.p3.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S2.SS6.p3.1.m1.1a"><mo id="S2.SS6.p3.1.m1.1.1" xref="S2.SS6.p3.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S2.SS6.p3.1.m1.1b"><times id="S2.SS6.p3.1.m1.1.1.cmml" xref="S2.SS6.p3.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.p3.1.m1.1c">\times</annotation></semantics></math> optimizers in Figure <a href="#S2.F9" title="Figure 9 ‣ 2.6 Theoretical equivalences ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a> is already large,
machine learning is much richer than optimizing the parameters <math id="S2.SS6.p3.2.m2.1" class="ltx_Math" alttext="W_{t}" display="inline"><semantics id="S2.SS6.p3.2.m2.1a"><msub id="S2.SS6.p3.2.m2.1.1" xref="S2.SS6.p3.2.m2.1.1.cmml"><mi id="S2.SS6.p3.2.m2.1.1.2" xref="S2.SS6.p3.2.m2.1.1.2.cmml">W</mi><mi id="S2.SS6.p3.2.m2.1.1.3" xref="S2.SS6.p3.2.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS6.p3.2.m2.1b"><apply id="S2.SS6.p3.2.m2.1.1.cmml" xref="S2.SS6.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS6.p3.2.m2.1.1.1.cmml" xref="S2.SS6.p3.2.m2.1.1">subscript</csymbol><ci id="S2.SS6.p3.2.m2.1.1.2.cmml" xref="S2.SS6.p3.2.m2.1.1.2">𝑊</ci><ci id="S2.SS6.p3.2.m2.1.1.3.cmml" xref="S2.SS6.p3.2.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.p3.2.m2.1c">W_{t}</annotation></semantics></math> of a model <math id="S2.SS6.p3.3.m3.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S2.SS6.p3.3.m3.1a"><mi id="S2.SS6.p3.3.m3.1.1" xref="S2.SS6.p3.3.m3.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S2.SS6.p3.3.m3.1b"><ci id="S2.SS6.p3.3.m3.1.1.cmml" xref="S2.SS6.p3.3.m3.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.p3.3.m3.1c">f</annotation></semantics></math>.
There are also nonparametric learners, such as nearest neighbors, support vector machines (SVMs), and kernel ridge regression.
By definition, nonparametric learners do not have parameters <math id="S2.SS6.p3.4.m4.1" class="ltx_Math" alttext="W_{t}" display="inline"><semantics id="S2.SS6.p3.4.m4.1a"><msub id="S2.SS6.p3.4.m4.1.1" xref="S2.SS6.p3.4.m4.1.1.cmml"><mi id="S2.SS6.p3.4.m4.1.1.2" xref="S2.SS6.p3.4.m4.1.1.2.cmml">W</mi><mi id="S2.SS6.p3.4.m4.1.1.3" xref="S2.SS6.p3.4.m4.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS6.p3.4.m4.1b"><apply id="S2.SS6.p3.4.m4.1.1.cmml" xref="S2.SS6.p3.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS6.p3.4.m4.1.1.1.cmml" xref="S2.SS6.p3.4.m4.1.1">subscript</csymbol><ci id="S2.SS6.p3.4.m4.1.1.2.cmml" xref="S2.SS6.p3.4.m4.1.1.2">𝑊</ci><ci id="S2.SS6.p3.4.m4.1.1.3.cmml" xref="S2.SS6.p3.4.m4.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.p3.4.m4.1c">W_{t}</annotation></semantics></math>, and instead directly uses training data <math id="S2.SS6.p3.5.m5.3" class="ltx_Math" alttext="x_{1},\dots,x_{t}" display="inline"><semantics id="S2.SS6.p3.5.m5.3a"><mrow id="S2.SS6.p3.5.m5.3.3.2" xref="S2.SS6.p3.5.m5.3.3.3.cmml"><msub id="S2.SS6.p3.5.m5.2.2.1.1" xref="S2.SS6.p3.5.m5.2.2.1.1.cmml"><mi id="S2.SS6.p3.5.m5.2.2.1.1.2" xref="S2.SS6.p3.5.m5.2.2.1.1.2.cmml">x</mi><mn id="S2.SS6.p3.5.m5.2.2.1.1.3" xref="S2.SS6.p3.5.m5.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.SS6.p3.5.m5.3.3.2.3" xref="S2.SS6.p3.5.m5.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS6.p3.5.m5.1.1" xref="S2.SS6.p3.5.m5.1.1.cmml">…</mi><mo id="S2.SS6.p3.5.m5.3.3.2.4" xref="S2.SS6.p3.5.m5.3.3.3.cmml">,</mo><msub id="S2.SS6.p3.5.m5.3.3.2.2" xref="S2.SS6.p3.5.m5.3.3.2.2.cmml"><mi id="S2.SS6.p3.5.m5.3.3.2.2.2" xref="S2.SS6.p3.5.m5.3.3.2.2.2.cmml">x</mi><mi id="S2.SS6.p3.5.m5.3.3.2.2.3" xref="S2.SS6.p3.5.m5.3.3.2.2.3.cmml">t</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS6.p3.5.m5.3b"><list id="S2.SS6.p3.5.m5.3.3.3.cmml" xref="S2.SS6.p3.5.m5.3.3.2"><apply id="S2.SS6.p3.5.m5.2.2.1.1.cmml" xref="S2.SS6.p3.5.m5.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS6.p3.5.m5.2.2.1.1.1.cmml" xref="S2.SS6.p3.5.m5.2.2.1.1">subscript</csymbol><ci id="S2.SS6.p3.5.m5.2.2.1.1.2.cmml" xref="S2.SS6.p3.5.m5.2.2.1.1.2">𝑥</ci><cn type="integer" id="S2.SS6.p3.5.m5.2.2.1.1.3.cmml" xref="S2.SS6.p3.5.m5.2.2.1.1.3">1</cn></apply><ci id="S2.SS6.p3.5.m5.1.1.cmml" xref="S2.SS6.p3.5.m5.1.1">…</ci><apply id="S2.SS6.p3.5.m5.3.3.2.2.cmml" xref="S2.SS6.p3.5.m5.3.3.2.2"><csymbol cd="ambiguous" id="S2.SS6.p3.5.m5.3.3.2.2.1.cmml" xref="S2.SS6.p3.5.m5.3.3.2.2">subscript</csymbol><ci id="S2.SS6.p3.5.m5.3.3.2.2.2.cmml" xref="S2.SS6.p3.5.m5.3.3.2.2.2">𝑥</ci><ci id="S2.SS6.p3.5.m5.3.3.2.2.3.cmml" xref="S2.SS6.p3.5.m5.3.3.2.2.3">𝑡</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.p3.5.m5.3c">x_{1},\dots,x_{t}</annotation></semantics></math>.
Hence we use the notation <math id="S2.SS6.p3.6.m6.4" class="ltx_Math" alttext="f(x;x_{1},\dots,x_{t})" display="inline"><semantics id="S2.SS6.p3.6.m6.4a"><mrow id="S2.SS6.p3.6.m6.4.4" xref="S2.SS6.p3.6.m6.4.4.cmml"><mi id="S2.SS6.p3.6.m6.4.4.4" xref="S2.SS6.p3.6.m6.4.4.4.cmml">f</mi><mo lspace="0em" rspace="0em" id="S2.SS6.p3.6.m6.4.4.3" xref="S2.SS6.p3.6.m6.4.4.3.cmml">​</mo><mrow id="S2.SS6.p3.6.m6.4.4.2.2" xref="S2.SS6.p3.6.m6.4.4.2.3.cmml"><mo stretchy="false" id="S2.SS6.p3.6.m6.4.4.2.2.3" xref="S2.SS6.p3.6.m6.4.4.2.3.cmml">(</mo><mi id="S2.SS6.p3.6.m6.1.1" xref="S2.SS6.p3.6.m6.1.1.cmml">x</mi><mo id="S2.SS6.p3.6.m6.4.4.2.2.4" xref="S2.SS6.p3.6.m6.4.4.2.3.cmml">;</mo><msub id="S2.SS6.p3.6.m6.3.3.1.1.1" xref="S2.SS6.p3.6.m6.3.3.1.1.1.cmml"><mi id="S2.SS6.p3.6.m6.3.3.1.1.1.2" xref="S2.SS6.p3.6.m6.3.3.1.1.1.2.cmml">x</mi><mn id="S2.SS6.p3.6.m6.3.3.1.1.1.3" xref="S2.SS6.p3.6.m6.3.3.1.1.1.3.cmml">1</mn></msub><mo id="S2.SS6.p3.6.m6.4.4.2.2.5" xref="S2.SS6.p3.6.m6.4.4.2.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS6.p3.6.m6.2.2" xref="S2.SS6.p3.6.m6.2.2.cmml">…</mi><mo id="S2.SS6.p3.6.m6.4.4.2.2.6" xref="S2.SS6.p3.6.m6.4.4.2.3.cmml">,</mo><msub id="S2.SS6.p3.6.m6.4.4.2.2.2" xref="S2.SS6.p3.6.m6.4.4.2.2.2.cmml"><mi id="S2.SS6.p3.6.m6.4.4.2.2.2.2" xref="S2.SS6.p3.6.m6.4.4.2.2.2.2.cmml">x</mi><mi id="S2.SS6.p3.6.m6.4.4.2.2.2.3" xref="S2.SS6.p3.6.m6.4.4.2.2.2.3.cmml">t</mi></msub><mo stretchy="false" id="S2.SS6.p3.6.m6.4.4.2.2.7" xref="S2.SS6.p3.6.m6.4.4.2.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS6.p3.6.m6.4b"><apply id="S2.SS6.p3.6.m6.4.4.cmml" xref="S2.SS6.p3.6.m6.4.4"><times id="S2.SS6.p3.6.m6.4.4.3.cmml" xref="S2.SS6.p3.6.m6.4.4.3"></times><ci id="S2.SS6.p3.6.m6.4.4.4.cmml" xref="S2.SS6.p3.6.m6.4.4.4">𝑓</ci><list id="S2.SS6.p3.6.m6.4.4.2.3.cmml" xref="S2.SS6.p3.6.m6.4.4.2.2"><ci id="S2.SS6.p3.6.m6.1.1.cmml" xref="S2.SS6.p3.6.m6.1.1">𝑥</ci><apply id="S2.SS6.p3.6.m6.3.3.1.1.1.cmml" xref="S2.SS6.p3.6.m6.3.3.1.1.1"><csymbol cd="ambiguous" id="S2.SS6.p3.6.m6.3.3.1.1.1.1.cmml" xref="S2.SS6.p3.6.m6.3.3.1.1.1">subscript</csymbol><ci id="S2.SS6.p3.6.m6.3.3.1.1.1.2.cmml" xref="S2.SS6.p3.6.m6.3.3.1.1.1.2">𝑥</ci><cn type="integer" id="S2.SS6.p3.6.m6.3.3.1.1.1.3.cmml" xref="S2.SS6.p3.6.m6.3.3.1.1.1.3">1</cn></apply><ci id="S2.SS6.p3.6.m6.2.2.cmml" xref="S2.SS6.p3.6.m6.2.2">…</ci><apply id="S2.SS6.p3.6.m6.4.4.2.2.2.cmml" xref="S2.SS6.p3.6.m6.4.4.2.2.2"><csymbol cd="ambiguous" id="S2.SS6.p3.6.m6.4.4.2.2.2.1.cmml" xref="S2.SS6.p3.6.m6.4.4.2.2.2">subscript</csymbol><ci id="S2.SS6.p3.6.m6.4.4.2.2.2.2.cmml" xref="S2.SS6.p3.6.m6.4.4.2.2.2.2">𝑥</ci><ci id="S2.SS6.p3.6.m6.4.4.2.2.2.3.cmml" xref="S2.SS6.p3.6.m6.4.4.2.2.2.3">𝑡</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.p3.6.m6.4c">f(x;x_{1},\dots,x_{t})</annotation></semantics></math>.
We now show that for a particular nonparametric learner, the induced TTT layer is equivalent to self-attention.</p>
</div>
<div id="Thmtheorem2" class="ltx_theorem ltx_theorem_theorem">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmtheorem2.1.1.1" class="ltx_text ltx_font_bold">Theorem 2</span></span><span id="Thmtheorem2.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmtheorem2.p1" class="ltx_para">
<p id="Thmtheorem2.p1.6" class="ltx_p"><span id="Thmtheorem2.p1.6.1" class="ltx_text ltx_font_italic">Consider the TTT layer with the Nadaraya-Watson estimator <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib7" title="" class="ltx_ref">7</a>, <a href="#bib.bib12" title="" class="ltx_ref">12</a>]</cite>, defined as:</span></p>
<table id="S2.E8" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E8.m1.6" class="ltx_Math" alttext="f(x;x_{1},\dots,x_{t})=\frac{1}{\sum_{s=1}^{t}\kappa(x,x_{s})}\sum_{s=1}^{t}\kappa(x,x_{s})~{}y_{s}," display="block"><semantics id="S2.E8.m1.6a"><mrow id="S2.E8.m1.6.6.1" xref="S2.E8.m1.6.6.1.1.cmml"><mrow id="S2.E8.m1.6.6.1.1" xref="S2.E8.m1.6.6.1.1.cmml"><mrow id="S2.E8.m1.6.6.1.1.2" xref="S2.E8.m1.6.6.1.1.2.cmml"><mi id="S2.E8.m1.6.6.1.1.2.4" xref="S2.E8.m1.6.6.1.1.2.4.cmml">f</mi><mo lspace="0em" rspace="0em" id="S2.E8.m1.6.6.1.1.2.3" xref="S2.E8.m1.6.6.1.1.2.3.cmml">​</mo><mrow id="S2.E8.m1.6.6.1.1.2.2.2" xref="S2.E8.m1.6.6.1.1.2.2.3.cmml"><mo stretchy="false" id="S2.E8.m1.6.6.1.1.2.2.2.3" xref="S2.E8.m1.6.6.1.1.2.2.3.cmml">(</mo><mi id="S2.E8.m1.3.3" xref="S2.E8.m1.3.3.cmml">x</mi><mo id="S2.E8.m1.6.6.1.1.2.2.2.4" xref="S2.E8.m1.6.6.1.1.2.2.3.cmml">;</mo><msub id="S2.E8.m1.6.6.1.1.1.1.1.1" xref="S2.E8.m1.6.6.1.1.1.1.1.1.cmml"><mi id="S2.E8.m1.6.6.1.1.1.1.1.1.2" xref="S2.E8.m1.6.6.1.1.1.1.1.1.2.cmml">x</mi><mn id="S2.E8.m1.6.6.1.1.1.1.1.1.3" xref="S2.E8.m1.6.6.1.1.1.1.1.1.3.cmml">1</mn></msub><mo id="S2.E8.m1.6.6.1.1.2.2.2.5" xref="S2.E8.m1.6.6.1.1.2.2.3.cmml">,</mo><mi mathvariant="normal" id="S2.E8.m1.4.4" xref="S2.E8.m1.4.4.cmml">…</mi><mo id="S2.E8.m1.6.6.1.1.2.2.2.6" xref="S2.E8.m1.6.6.1.1.2.2.3.cmml">,</mo><msub id="S2.E8.m1.6.6.1.1.2.2.2.2" xref="S2.E8.m1.6.6.1.1.2.2.2.2.cmml"><mi id="S2.E8.m1.6.6.1.1.2.2.2.2.2" xref="S2.E8.m1.6.6.1.1.2.2.2.2.2.cmml">x</mi><mi id="S2.E8.m1.6.6.1.1.2.2.2.2.3" xref="S2.E8.m1.6.6.1.1.2.2.2.2.3.cmml">t</mi></msub><mo stretchy="false" id="S2.E8.m1.6.6.1.1.2.2.2.7" xref="S2.E8.m1.6.6.1.1.2.2.3.cmml">)</mo></mrow></mrow><mo id="S2.E8.m1.6.6.1.1.4" xref="S2.E8.m1.6.6.1.1.4.cmml">=</mo><mrow id="S2.E8.m1.6.6.1.1.3" xref="S2.E8.m1.6.6.1.1.3.cmml"><mfrac id="S2.E8.m1.2.2" xref="S2.E8.m1.2.2.cmml"><mn id="S2.E8.m1.2.2.4" xref="S2.E8.m1.2.2.4.cmml">1</mn><mrow id="S2.E8.m1.2.2.2" xref="S2.E8.m1.2.2.2.cmml"><msubsup id="S2.E8.m1.2.2.2.3" xref="S2.E8.m1.2.2.2.3.cmml"><mo id="S2.E8.m1.2.2.2.3.2.2" xref="S2.E8.m1.2.2.2.3.2.2.cmml">∑</mo><mrow id="S2.E8.m1.2.2.2.3.2.3" xref="S2.E8.m1.2.2.2.3.2.3.cmml"><mi id="S2.E8.m1.2.2.2.3.2.3.2" xref="S2.E8.m1.2.2.2.3.2.3.2.cmml">s</mi><mo id="S2.E8.m1.2.2.2.3.2.3.1" xref="S2.E8.m1.2.2.2.3.2.3.1.cmml">=</mo><mn id="S2.E8.m1.2.2.2.3.2.3.3" xref="S2.E8.m1.2.2.2.3.2.3.3.cmml">1</mn></mrow><mi id="S2.E8.m1.2.2.2.3.3" xref="S2.E8.m1.2.2.2.3.3.cmml">t</mi></msubsup><mrow id="S2.E8.m1.2.2.2.2" xref="S2.E8.m1.2.2.2.2.cmml"><mi id="S2.E8.m1.2.2.2.2.3" xref="S2.E8.m1.2.2.2.2.3.cmml">κ</mi><mo lspace="0em" rspace="0em" id="S2.E8.m1.2.2.2.2.2" xref="S2.E8.m1.2.2.2.2.2.cmml">​</mo><mrow id="S2.E8.m1.2.2.2.2.1.1" xref="S2.E8.m1.2.2.2.2.1.2.cmml"><mo stretchy="false" id="S2.E8.m1.2.2.2.2.1.1.2" xref="S2.E8.m1.2.2.2.2.1.2.cmml">(</mo><mi id="S2.E8.m1.1.1.1.1" xref="S2.E8.m1.1.1.1.1.cmml">x</mi><mo id="S2.E8.m1.2.2.2.2.1.1.3" xref="S2.E8.m1.2.2.2.2.1.2.cmml">,</mo><msub id="S2.E8.m1.2.2.2.2.1.1.1" xref="S2.E8.m1.2.2.2.2.1.1.1.cmml"><mi id="S2.E8.m1.2.2.2.2.1.1.1.2" xref="S2.E8.m1.2.2.2.2.1.1.1.2.cmml">x</mi><mi id="S2.E8.m1.2.2.2.2.1.1.1.3" xref="S2.E8.m1.2.2.2.2.1.1.1.3.cmml">s</mi></msub><mo stretchy="false" id="S2.E8.m1.2.2.2.2.1.1.4" xref="S2.E8.m1.2.2.2.2.1.2.cmml">)</mo></mrow></mrow></mrow></mfrac><mo lspace="0em" rspace="0em" id="S2.E8.m1.6.6.1.1.3.2" xref="S2.E8.m1.6.6.1.1.3.2.cmml">​</mo><mrow id="S2.E8.m1.6.6.1.1.3.1" xref="S2.E8.m1.6.6.1.1.3.1.cmml"><munderover id="S2.E8.m1.6.6.1.1.3.1.2" xref="S2.E8.m1.6.6.1.1.3.1.2.cmml"><mo movablelimits="false" id="S2.E8.m1.6.6.1.1.3.1.2.2.2" xref="S2.E8.m1.6.6.1.1.3.1.2.2.2.cmml">∑</mo><mrow id="S2.E8.m1.6.6.1.1.3.1.2.2.3" xref="S2.E8.m1.6.6.1.1.3.1.2.2.3.cmml"><mi id="S2.E8.m1.6.6.1.1.3.1.2.2.3.2" xref="S2.E8.m1.6.6.1.1.3.1.2.2.3.2.cmml">s</mi><mo id="S2.E8.m1.6.6.1.1.3.1.2.2.3.1" xref="S2.E8.m1.6.6.1.1.3.1.2.2.3.1.cmml">=</mo><mn id="S2.E8.m1.6.6.1.1.3.1.2.2.3.3" xref="S2.E8.m1.6.6.1.1.3.1.2.2.3.3.cmml">1</mn></mrow><mi id="S2.E8.m1.6.6.1.1.3.1.2.3" xref="S2.E8.m1.6.6.1.1.3.1.2.3.cmml">t</mi></munderover><mrow id="S2.E8.m1.6.6.1.1.3.1.1" xref="S2.E8.m1.6.6.1.1.3.1.1.cmml"><mi id="S2.E8.m1.6.6.1.1.3.1.1.3" xref="S2.E8.m1.6.6.1.1.3.1.1.3.cmml">κ</mi><mo lspace="0em" rspace="0em" id="S2.E8.m1.6.6.1.1.3.1.1.2" xref="S2.E8.m1.6.6.1.1.3.1.1.2.cmml">​</mo><mrow id="S2.E8.m1.6.6.1.1.3.1.1.1.1" xref="S2.E8.m1.6.6.1.1.3.1.1.1.2.cmml"><mo stretchy="false" id="S2.E8.m1.6.6.1.1.3.1.1.1.1.2" xref="S2.E8.m1.6.6.1.1.3.1.1.1.2.cmml">(</mo><mi id="S2.E8.m1.5.5" xref="S2.E8.m1.5.5.cmml">x</mi><mo id="S2.E8.m1.6.6.1.1.3.1.1.1.1.3" xref="S2.E8.m1.6.6.1.1.3.1.1.1.2.cmml">,</mo><msub id="S2.E8.m1.6.6.1.1.3.1.1.1.1.1" xref="S2.E8.m1.6.6.1.1.3.1.1.1.1.1.cmml"><mi id="S2.E8.m1.6.6.1.1.3.1.1.1.1.1.2" xref="S2.E8.m1.6.6.1.1.3.1.1.1.1.1.2.cmml">x</mi><mi id="S2.E8.m1.6.6.1.1.3.1.1.1.1.1.3" xref="S2.E8.m1.6.6.1.1.3.1.1.1.1.1.3.cmml">s</mi></msub><mo stretchy="false" id="S2.E8.m1.6.6.1.1.3.1.1.1.1.4" xref="S2.E8.m1.6.6.1.1.3.1.1.1.2.cmml">)</mo></mrow><mo lspace="0.330em" rspace="0em" id="S2.E8.m1.6.6.1.1.3.1.1.2a" xref="S2.E8.m1.6.6.1.1.3.1.1.2.cmml">​</mo><msub id="S2.E8.m1.6.6.1.1.3.1.1.4" xref="S2.E8.m1.6.6.1.1.3.1.1.4.cmml"><mi id="S2.E8.m1.6.6.1.1.3.1.1.4.2" xref="S2.E8.m1.6.6.1.1.3.1.1.4.2.cmml">y</mi><mi id="S2.E8.m1.6.6.1.1.3.1.1.4.3" xref="S2.E8.m1.6.6.1.1.3.1.1.4.3.cmml">s</mi></msub></mrow></mrow></mrow></mrow><mo id="S2.E8.m1.6.6.1.2" xref="S2.E8.m1.6.6.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.E8.m1.6b"><apply id="S2.E8.m1.6.6.1.1.cmml" xref="S2.E8.m1.6.6.1"><eq id="S2.E8.m1.6.6.1.1.4.cmml" xref="S2.E8.m1.6.6.1.1.4"></eq><apply id="S2.E8.m1.6.6.1.1.2.cmml" xref="S2.E8.m1.6.6.1.1.2"><times id="S2.E8.m1.6.6.1.1.2.3.cmml" xref="S2.E8.m1.6.6.1.1.2.3"></times><ci id="S2.E8.m1.6.6.1.1.2.4.cmml" xref="S2.E8.m1.6.6.1.1.2.4">𝑓</ci><list id="S2.E8.m1.6.6.1.1.2.2.3.cmml" xref="S2.E8.m1.6.6.1.1.2.2.2"><ci id="S2.E8.m1.3.3.cmml" xref="S2.E8.m1.3.3">𝑥</ci><apply id="S2.E8.m1.6.6.1.1.1.1.1.1.cmml" xref="S2.E8.m1.6.6.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E8.m1.6.6.1.1.1.1.1.1.1.cmml" xref="S2.E8.m1.6.6.1.1.1.1.1.1">subscript</csymbol><ci id="S2.E8.m1.6.6.1.1.1.1.1.1.2.cmml" xref="S2.E8.m1.6.6.1.1.1.1.1.1.2">𝑥</ci><cn type="integer" id="S2.E8.m1.6.6.1.1.1.1.1.1.3.cmml" xref="S2.E8.m1.6.6.1.1.1.1.1.1.3">1</cn></apply><ci id="S2.E8.m1.4.4.cmml" xref="S2.E8.m1.4.4">…</ci><apply id="S2.E8.m1.6.6.1.1.2.2.2.2.cmml" xref="S2.E8.m1.6.6.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S2.E8.m1.6.6.1.1.2.2.2.2.1.cmml" xref="S2.E8.m1.6.6.1.1.2.2.2.2">subscript</csymbol><ci id="S2.E8.m1.6.6.1.1.2.2.2.2.2.cmml" xref="S2.E8.m1.6.6.1.1.2.2.2.2.2">𝑥</ci><ci id="S2.E8.m1.6.6.1.1.2.2.2.2.3.cmml" xref="S2.E8.m1.6.6.1.1.2.2.2.2.3">𝑡</ci></apply></list></apply><apply id="S2.E8.m1.6.6.1.1.3.cmml" xref="S2.E8.m1.6.6.1.1.3"><times id="S2.E8.m1.6.6.1.1.3.2.cmml" xref="S2.E8.m1.6.6.1.1.3.2"></times><apply id="S2.E8.m1.2.2.cmml" xref="S2.E8.m1.2.2"><divide id="S2.E8.m1.2.2.3.cmml" xref="S2.E8.m1.2.2"></divide><cn type="integer" id="S2.E8.m1.2.2.4.cmml" xref="S2.E8.m1.2.2.4">1</cn><apply id="S2.E8.m1.2.2.2.cmml" xref="S2.E8.m1.2.2.2"><apply id="S2.E8.m1.2.2.2.3.cmml" xref="S2.E8.m1.2.2.2.3"><csymbol cd="ambiguous" id="S2.E8.m1.2.2.2.3.1.cmml" xref="S2.E8.m1.2.2.2.3">superscript</csymbol><apply id="S2.E8.m1.2.2.2.3.2.cmml" xref="S2.E8.m1.2.2.2.3"><csymbol cd="ambiguous" id="S2.E8.m1.2.2.2.3.2.1.cmml" xref="S2.E8.m1.2.2.2.3">subscript</csymbol><sum id="S2.E8.m1.2.2.2.3.2.2.cmml" xref="S2.E8.m1.2.2.2.3.2.2"></sum><apply id="S2.E8.m1.2.2.2.3.2.3.cmml" xref="S2.E8.m1.2.2.2.3.2.3"><eq id="S2.E8.m1.2.2.2.3.2.3.1.cmml" xref="S2.E8.m1.2.2.2.3.2.3.1"></eq><ci id="S2.E8.m1.2.2.2.3.2.3.2.cmml" xref="S2.E8.m1.2.2.2.3.2.3.2">𝑠</ci><cn type="integer" id="S2.E8.m1.2.2.2.3.2.3.3.cmml" xref="S2.E8.m1.2.2.2.3.2.3.3">1</cn></apply></apply><ci id="S2.E8.m1.2.2.2.3.3.cmml" xref="S2.E8.m1.2.2.2.3.3">𝑡</ci></apply><apply id="S2.E8.m1.2.2.2.2.cmml" xref="S2.E8.m1.2.2.2.2"><times id="S2.E8.m1.2.2.2.2.2.cmml" xref="S2.E8.m1.2.2.2.2.2"></times><ci id="S2.E8.m1.2.2.2.2.3.cmml" xref="S2.E8.m1.2.2.2.2.3">𝜅</ci><interval closure="open" id="S2.E8.m1.2.2.2.2.1.2.cmml" xref="S2.E8.m1.2.2.2.2.1.1"><ci id="S2.E8.m1.1.1.1.1.cmml" xref="S2.E8.m1.1.1.1.1">𝑥</ci><apply id="S2.E8.m1.2.2.2.2.1.1.1.cmml" xref="S2.E8.m1.2.2.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.E8.m1.2.2.2.2.1.1.1.1.cmml" xref="S2.E8.m1.2.2.2.2.1.1.1">subscript</csymbol><ci id="S2.E8.m1.2.2.2.2.1.1.1.2.cmml" xref="S2.E8.m1.2.2.2.2.1.1.1.2">𝑥</ci><ci id="S2.E8.m1.2.2.2.2.1.1.1.3.cmml" xref="S2.E8.m1.2.2.2.2.1.1.1.3">𝑠</ci></apply></interval></apply></apply></apply><apply id="S2.E8.m1.6.6.1.1.3.1.cmml" xref="S2.E8.m1.6.6.1.1.3.1"><apply id="S2.E8.m1.6.6.1.1.3.1.2.cmml" xref="S2.E8.m1.6.6.1.1.3.1.2"><csymbol cd="ambiguous" id="S2.E8.m1.6.6.1.1.3.1.2.1.cmml" xref="S2.E8.m1.6.6.1.1.3.1.2">superscript</csymbol><apply id="S2.E8.m1.6.6.1.1.3.1.2.2.cmml" xref="S2.E8.m1.6.6.1.1.3.1.2"><csymbol cd="ambiguous" id="S2.E8.m1.6.6.1.1.3.1.2.2.1.cmml" xref="S2.E8.m1.6.6.1.1.3.1.2">subscript</csymbol><sum id="S2.E8.m1.6.6.1.1.3.1.2.2.2.cmml" xref="S2.E8.m1.6.6.1.1.3.1.2.2.2"></sum><apply id="S2.E8.m1.6.6.1.1.3.1.2.2.3.cmml" xref="S2.E8.m1.6.6.1.1.3.1.2.2.3"><eq id="S2.E8.m1.6.6.1.1.3.1.2.2.3.1.cmml" xref="S2.E8.m1.6.6.1.1.3.1.2.2.3.1"></eq><ci id="S2.E8.m1.6.6.1.1.3.1.2.2.3.2.cmml" xref="S2.E8.m1.6.6.1.1.3.1.2.2.3.2">𝑠</ci><cn type="integer" id="S2.E8.m1.6.6.1.1.3.1.2.2.3.3.cmml" xref="S2.E8.m1.6.6.1.1.3.1.2.2.3.3">1</cn></apply></apply><ci id="S2.E8.m1.6.6.1.1.3.1.2.3.cmml" xref="S2.E8.m1.6.6.1.1.3.1.2.3">𝑡</ci></apply><apply id="S2.E8.m1.6.6.1.1.3.1.1.cmml" xref="S2.E8.m1.6.6.1.1.3.1.1"><times id="S2.E8.m1.6.6.1.1.3.1.1.2.cmml" xref="S2.E8.m1.6.6.1.1.3.1.1.2"></times><ci id="S2.E8.m1.6.6.1.1.3.1.1.3.cmml" xref="S2.E8.m1.6.6.1.1.3.1.1.3">𝜅</ci><interval closure="open" id="S2.E8.m1.6.6.1.1.3.1.1.1.2.cmml" xref="S2.E8.m1.6.6.1.1.3.1.1.1.1"><ci id="S2.E8.m1.5.5.cmml" xref="S2.E8.m1.5.5">𝑥</ci><apply id="S2.E8.m1.6.6.1.1.3.1.1.1.1.1.cmml" xref="S2.E8.m1.6.6.1.1.3.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E8.m1.6.6.1.1.3.1.1.1.1.1.1.cmml" xref="S2.E8.m1.6.6.1.1.3.1.1.1.1.1">subscript</csymbol><ci id="S2.E8.m1.6.6.1.1.3.1.1.1.1.1.2.cmml" xref="S2.E8.m1.6.6.1.1.3.1.1.1.1.1.2">𝑥</ci><ci id="S2.E8.m1.6.6.1.1.3.1.1.1.1.1.3.cmml" xref="S2.E8.m1.6.6.1.1.3.1.1.1.1.1.3">𝑠</ci></apply></interval><apply id="S2.E8.m1.6.6.1.1.3.1.1.4.cmml" xref="S2.E8.m1.6.6.1.1.3.1.1.4"><csymbol cd="ambiguous" id="S2.E8.m1.6.6.1.1.3.1.1.4.1.cmml" xref="S2.E8.m1.6.6.1.1.3.1.1.4">subscript</csymbol><ci id="S2.E8.m1.6.6.1.1.3.1.1.4.2.cmml" xref="S2.E8.m1.6.6.1.1.3.1.1.4.2">𝑦</ci><ci id="S2.E8.m1.6.6.1.1.3.1.1.4.3.cmml" xref="S2.E8.m1.6.6.1.1.3.1.1.4.3">𝑠</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E8.m1.6c">f(x;x_{1},\dots,x_{t})=\frac{1}{\sum_{s=1}^{t}\kappa(x,x_{s})}\sum_{s=1}^{t}\kappa(x,x_{s})~{}y_{s},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(8)</span></td>
</tr></tbody>
</table>
<p id="Thmtheorem2.p1.1" class="ltx_p"><span id="Thmtheorem2.p1.1.1" class="ltx_text ltx_font_italic">where <math id="Thmtheorem2.p1.1.1.m1.1" class="ltx_Math" alttext="y_{s}=\theta_{V}x_{s}" display="inline"><semantics id="Thmtheorem2.p1.1.1.m1.1a"><mrow id="Thmtheorem2.p1.1.1.m1.1.1" xref="Thmtheorem2.p1.1.1.m1.1.1.cmml"><msub id="Thmtheorem2.p1.1.1.m1.1.1.2" xref="Thmtheorem2.p1.1.1.m1.1.1.2.cmml"><mi id="Thmtheorem2.p1.1.1.m1.1.1.2.2" xref="Thmtheorem2.p1.1.1.m1.1.1.2.2.cmml">y</mi><mi id="Thmtheorem2.p1.1.1.m1.1.1.2.3" xref="Thmtheorem2.p1.1.1.m1.1.1.2.3.cmml">s</mi></msub><mo id="Thmtheorem2.p1.1.1.m1.1.1.1" xref="Thmtheorem2.p1.1.1.m1.1.1.1.cmml">=</mo><mrow id="Thmtheorem2.p1.1.1.m1.1.1.3" xref="Thmtheorem2.p1.1.1.m1.1.1.3.cmml"><msub id="Thmtheorem2.p1.1.1.m1.1.1.3.2" xref="Thmtheorem2.p1.1.1.m1.1.1.3.2.cmml"><mi id="Thmtheorem2.p1.1.1.m1.1.1.3.2.2" xref="Thmtheorem2.p1.1.1.m1.1.1.3.2.2.cmml">θ</mi><mi id="Thmtheorem2.p1.1.1.m1.1.1.3.2.3" xref="Thmtheorem2.p1.1.1.m1.1.1.3.2.3.cmml">V</mi></msub><mo lspace="0em" rspace="0em" id="Thmtheorem2.p1.1.1.m1.1.1.3.1" xref="Thmtheorem2.p1.1.1.m1.1.1.3.1.cmml">​</mo><msub id="Thmtheorem2.p1.1.1.m1.1.1.3.3" xref="Thmtheorem2.p1.1.1.m1.1.1.3.3.cmml"><mi id="Thmtheorem2.p1.1.1.m1.1.1.3.3.2" xref="Thmtheorem2.p1.1.1.m1.1.1.3.3.2.cmml">x</mi><mi id="Thmtheorem2.p1.1.1.m1.1.1.3.3.3" xref="Thmtheorem2.p1.1.1.m1.1.1.3.3.3.cmml">s</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmtheorem2.p1.1.1.m1.1b"><apply id="Thmtheorem2.p1.1.1.m1.1.1.cmml" xref="Thmtheorem2.p1.1.1.m1.1.1"><eq id="Thmtheorem2.p1.1.1.m1.1.1.1.cmml" xref="Thmtheorem2.p1.1.1.m1.1.1.1"></eq><apply id="Thmtheorem2.p1.1.1.m1.1.1.2.cmml" xref="Thmtheorem2.p1.1.1.m1.1.1.2"><csymbol cd="ambiguous" id="Thmtheorem2.p1.1.1.m1.1.1.2.1.cmml" xref="Thmtheorem2.p1.1.1.m1.1.1.2">subscript</csymbol><ci id="Thmtheorem2.p1.1.1.m1.1.1.2.2.cmml" xref="Thmtheorem2.p1.1.1.m1.1.1.2.2">𝑦</ci><ci id="Thmtheorem2.p1.1.1.m1.1.1.2.3.cmml" xref="Thmtheorem2.p1.1.1.m1.1.1.2.3">𝑠</ci></apply><apply id="Thmtheorem2.p1.1.1.m1.1.1.3.cmml" xref="Thmtheorem2.p1.1.1.m1.1.1.3"><times id="Thmtheorem2.p1.1.1.m1.1.1.3.1.cmml" xref="Thmtheorem2.p1.1.1.m1.1.1.3.1"></times><apply id="Thmtheorem2.p1.1.1.m1.1.1.3.2.cmml" xref="Thmtheorem2.p1.1.1.m1.1.1.3.2"><csymbol cd="ambiguous" id="Thmtheorem2.p1.1.1.m1.1.1.3.2.1.cmml" xref="Thmtheorem2.p1.1.1.m1.1.1.3.2">subscript</csymbol><ci id="Thmtheorem2.p1.1.1.m1.1.1.3.2.2.cmml" xref="Thmtheorem2.p1.1.1.m1.1.1.3.2.2">𝜃</ci><ci id="Thmtheorem2.p1.1.1.m1.1.1.3.2.3.cmml" xref="Thmtheorem2.p1.1.1.m1.1.1.3.2.3">𝑉</ci></apply><apply id="Thmtheorem2.p1.1.1.m1.1.1.3.3.cmml" xref="Thmtheorem2.p1.1.1.m1.1.1.3.3"><csymbol cd="ambiguous" id="Thmtheorem2.p1.1.1.m1.1.1.3.3.1.cmml" xref="Thmtheorem2.p1.1.1.m1.1.1.3.3">subscript</csymbol><ci id="Thmtheorem2.p1.1.1.m1.1.1.3.3.2.cmml" xref="Thmtheorem2.p1.1.1.m1.1.1.3.3.2">𝑥</ci><ci id="Thmtheorem2.p1.1.1.m1.1.1.3.3.3.cmml" xref="Thmtheorem2.p1.1.1.m1.1.1.3.3.3">𝑠</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem2.p1.1.1.m1.1c">y_{s}=\theta_{V}x_{s}</annotation></semantics></math> is the label view discussed in Subsection <a href="#S2.SS3" title="2.3 Learning a self-supervised task for TTT ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.3</span></a>, and</span></p>
<table id="S2.E9" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.E9.m1.5" class="ltx_Math" alttext="\kappa\left(x,x^{\prime};\theta_{K},\theta_{Q}\right)\propto e^{(\theta_{K}x)^{T}\theta_{Q}x^{\prime}}" display="block"><semantics id="S2.E9.m1.5a"><mrow id="S2.E9.m1.5.5" xref="S2.E9.m1.5.5.cmml"><mrow id="S2.E9.m1.5.5.3" xref="S2.E9.m1.5.5.3.cmml"><mi id="S2.E9.m1.5.5.3.5" xref="S2.E9.m1.5.5.3.5.cmml">κ</mi><mo lspace="0em" rspace="0em" id="S2.E9.m1.5.5.3.4" xref="S2.E9.m1.5.5.3.4.cmml">​</mo><mrow id="S2.E9.m1.5.5.3.3.3" xref="S2.E9.m1.5.5.3.3.4.cmml"><mo id="S2.E9.m1.5.5.3.3.3.4" xref="S2.E9.m1.5.5.3.3.4.cmml">(</mo><mi id="S2.E9.m1.2.2" xref="S2.E9.m1.2.2.cmml">x</mi><mo id="S2.E9.m1.5.5.3.3.3.5" xref="S2.E9.m1.5.5.3.3.4.cmml">,</mo><msup id="S2.E9.m1.3.3.1.1.1.1" xref="S2.E9.m1.3.3.1.1.1.1.cmml"><mi id="S2.E9.m1.3.3.1.1.1.1.2" xref="S2.E9.m1.3.3.1.1.1.1.2.cmml">x</mi><mo id="S2.E9.m1.3.3.1.1.1.1.3" xref="S2.E9.m1.3.3.1.1.1.1.3.cmml">′</mo></msup><mo id="S2.E9.m1.5.5.3.3.3.6" xref="S2.E9.m1.5.5.3.3.4.cmml">;</mo><msub id="S2.E9.m1.4.4.2.2.2.2" xref="S2.E9.m1.4.4.2.2.2.2.cmml"><mi id="S2.E9.m1.4.4.2.2.2.2.2" xref="S2.E9.m1.4.4.2.2.2.2.2.cmml">θ</mi><mi id="S2.E9.m1.4.4.2.2.2.2.3" xref="S2.E9.m1.4.4.2.2.2.2.3.cmml">K</mi></msub><mo id="S2.E9.m1.5.5.3.3.3.7" xref="S2.E9.m1.5.5.3.3.4.cmml">,</mo><msub id="S2.E9.m1.5.5.3.3.3.3" xref="S2.E9.m1.5.5.3.3.3.3.cmml"><mi id="S2.E9.m1.5.5.3.3.3.3.2" xref="S2.E9.m1.5.5.3.3.3.3.2.cmml">θ</mi><mi id="S2.E9.m1.5.5.3.3.3.3.3" xref="S2.E9.m1.5.5.3.3.3.3.3.cmml">Q</mi></msub><mo id="S2.E9.m1.5.5.3.3.3.8" xref="S2.E9.m1.5.5.3.3.4.cmml">)</mo></mrow></mrow><mo id="S2.E9.m1.5.5.4" xref="S2.E9.m1.5.5.4.cmml">∝</mo><msup id="S2.E9.m1.5.5.5" xref="S2.E9.m1.5.5.5.cmml"><mi id="S2.E9.m1.5.5.5.2" xref="S2.E9.m1.5.5.5.2.cmml">e</mi><mrow id="S2.E9.m1.1.1.1" xref="S2.E9.m1.1.1.1.cmml"><msup id="S2.E9.m1.1.1.1.1" xref="S2.E9.m1.1.1.1.1.cmml"><mrow id="S2.E9.m1.1.1.1.1.1.1" xref="S2.E9.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.E9.m1.1.1.1.1.1.1.2" xref="S2.E9.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.E9.m1.1.1.1.1.1.1.1" xref="S2.E9.m1.1.1.1.1.1.1.1.cmml"><msub id="S2.E9.m1.1.1.1.1.1.1.1.2" xref="S2.E9.m1.1.1.1.1.1.1.1.2.cmml"><mi id="S2.E9.m1.1.1.1.1.1.1.1.2.2" xref="S2.E9.m1.1.1.1.1.1.1.1.2.2.cmml">θ</mi><mi id="S2.E9.m1.1.1.1.1.1.1.1.2.3" xref="S2.E9.m1.1.1.1.1.1.1.1.2.3.cmml">K</mi></msub><mo lspace="0em" rspace="0em" id="S2.E9.m1.1.1.1.1.1.1.1.1" xref="S2.E9.m1.1.1.1.1.1.1.1.1.cmml">​</mo><mi id="S2.E9.m1.1.1.1.1.1.1.1.3" xref="S2.E9.m1.1.1.1.1.1.1.1.3.cmml">x</mi></mrow><mo stretchy="false" id="S2.E9.m1.1.1.1.1.1.1.3" xref="S2.E9.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mi id="S2.E9.m1.1.1.1.1.3" xref="S2.E9.m1.1.1.1.1.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="S2.E9.m1.1.1.1.2" xref="S2.E9.m1.1.1.1.2.cmml">​</mo><msub id="S2.E9.m1.1.1.1.3" xref="S2.E9.m1.1.1.1.3.cmml"><mi id="S2.E9.m1.1.1.1.3.2" xref="S2.E9.m1.1.1.1.3.2.cmml">θ</mi><mi id="S2.E9.m1.1.1.1.3.3" xref="S2.E9.m1.1.1.1.3.3.cmml">Q</mi></msub><mo lspace="0em" rspace="0em" id="S2.E9.m1.1.1.1.2a" xref="S2.E9.m1.1.1.1.2.cmml">​</mo><msup id="S2.E9.m1.1.1.1.4" xref="S2.E9.m1.1.1.1.4.cmml"><mi id="S2.E9.m1.1.1.1.4.2" xref="S2.E9.m1.1.1.1.4.2.cmml">x</mi><mo id="S2.E9.m1.1.1.1.4.3" xref="S2.E9.m1.1.1.1.4.3.cmml">′</mo></msup></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.E9.m1.5b"><apply id="S2.E9.m1.5.5.cmml" xref="S2.E9.m1.5.5"><csymbol cd="latexml" id="S2.E9.m1.5.5.4.cmml" xref="S2.E9.m1.5.5.4">proportional-to</csymbol><apply id="S2.E9.m1.5.5.3.cmml" xref="S2.E9.m1.5.5.3"><times id="S2.E9.m1.5.5.3.4.cmml" xref="S2.E9.m1.5.5.3.4"></times><ci id="S2.E9.m1.5.5.3.5.cmml" xref="S2.E9.m1.5.5.3.5">𝜅</ci><vector id="S2.E9.m1.5.5.3.3.4.cmml" xref="S2.E9.m1.5.5.3.3.3"><ci id="S2.E9.m1.2.2.cmml" xref="S2.E9.m1.2.2">𝑥</ci><apply id="S2.E9.m1.3.3.1.1.1.1.cmml" xref="S2.E9.m1.3.3.1.1.1.1"><csymbol cd="ambiguous" id="S2.E9.m1.3.3.1.1.1.1.1.cmml" xref="S2.E9.m1.3.3.1.1.1.1">superscript</csymbol><ci id="S2.E9.m1.3.3.1.1.1.1.2.cmml" xref="S2.E9.m1.3.3.1.1.1.1.2">𝑥</ci><ci id="S2.E9.m1.3.3.1.1.1.1.3.cmml" xref="S2.E9.m1.3.3.1.1.1.1.3">′</ci></apply><apply id="S2.E9.m1.4.4.2.2.2.2.cmml" xref="S2.E9.m1.4.4.2.2.2.2"><csymbol cd="ambiguous" id="S2.E9.m1.4.4.2.2.2.2.1.cmml" xref="S2.E9.m1.4.4.2.2.2.2">subscript</csymbol><ci id="S2.E9.m1.4.4.2.2.2.2.2.cmml" xref="S2.E9.m1.4.4.2.2.2.2.2">𝜃</ci><ci id="S2.E9.m1.4.4.2.2.2.2.3.cmml" xref="S2.E9.m1.4.4.2.2.2.2.3">𝐾</ci></apply><apply id="S2.E9.m1.5.5.3.3.3.3.cmml" xref="S2.E9.m1.5.5.3.3.3.3"><csymbol cd="ambiguous" id="S2.E9.m1.5.5.3.3.3.3.1.cmml" xref="S2.E9.m1.5.5.3.3.3.3">subscript</csymbol><ci id="S2.E9.m1.5.5.3.3.3.3.2.cmml" xref="S2.E9.m1.5.5.3.3.3.3.2">𝜃</ci><ci id="S2.E9.m1.5.5.3.3.3.3.3.cmml" xref="S2.E9.m1.5.5.3.3.3.3.3">𝑄</ci></apply></vector></apply><apply id="S2.E9.m1.5.5.5.cmml" xref="S2.E9.m1.5.5.5"><csymbol cd="ambiguous" id="S2.E9.m1.5.5.5.1.cmml" xref="S2.E9.m1.5.5.5">superscript</csymbol><ci id="S2.E9.m1.5.5.5.2.cmml" xref="S2.E9.m1.5.5.5.2">𝑒</ci><apply id="S2.E9.m1.1.1.1.cmml" xref="S2.E9.m1.1.1.1"><times id="S2.E9.m1.1.1.1.2.cmml" xref="S2.E9.m1.1.1.1.2"></times><apply id="S2.E9.m1.1.1.1.1.cmml" xref="S2.E9.m1.1.1.1.1"><csymbol cd="ambiguous" id="S2.E9.m1.1.1.1.1.2.cmml" xref="S2.E9.m1.1.1.1.1">superscript</csymbol><apply id="S2.E9.m1.1.1.1.1.1.1.1.cmml" xref="S2.E9.m1.1.1.1.1.1.1"><times id="S2.E9.m1.1.1.1.1.1.1.1.1.cmml" xref="S2.E9.m1.1.1.1.1.1.1.1.1"></times><apply id="S2.E9.m1.1.1.1.1.1.1.1.2.cmml" xref="S2.E9.m1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.E9.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.E9.m1.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S2.E9.m1.1.1.1.1.1.1.1.2.2.cmml" xref="S2.E9.m1.1.1.1.1.1.1.1.2.2">𝜃</ci><ci id="S2.E9.m1.1.1.1.1.1.1.1.2.3.cmml" xref="S2.E9.m1.1.1.1.1.1.1.1.2.3">𝐾</ci></apply><ci id="S2.E9.m1.1.1.1.1.1.1.1.3.cmml" xref="S2.E9.m1.1.1.1.1.1.1.1.3">𝑥</ci></apply><ci id="S2.E9.m1.1.1.1.1.3.cmml" xref="S2.E9.m1.1.1.1.1.3">𝑇</ci></apply><apply id="S2.E9.m1.1.1.1.3.cmml" xref="S2.E9.m1.1.1.1.3"><csymbol cd="ambiguous" id="S2.E9.m1.1.1.1.3.1.cmml" xref="S2.E9.m1.1.1.1.3">subscript</csymbol><ci id="S2.E9.m1.1.1.1.3.2.cmml" xref="S2.E9.m1.1.1.1.3.2">𝜃</ci><ci id="S2.E9.m1.1.1.1.3.3.cmml" xref="S2.E9.m1.1.1.1.3.3">𝑄</ci></apply><apply id="S2.E9.m1.1.1.1.4.cmml" xref="S2.E9.m1.1.1.1.4"><csymbol cd="ambiguous" id="S2.E9.m1.1.1.1.4.1.cmml" xref="S2.E9.m1.1.1.1.4">superscript</csymbol><ci id="S2.E9.m1.1.1.1.4.2.cmml" xref="S2.E9.m1.1.1.1.4.2">𝑥</ci><ci id="S2.E9.m1.1.1.1.4.3.cmml" xref="S2.E9.m1.1.1.1.4.3">′</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.E9.m1.5c">\kappa\left(x,x^{\prime};\theta_{K},\theta_{Q}\right)\propto e^{(\theta_{K}x)^{T}\theta_{Q}x^{\prime}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(9)</span></td>
</tr></tbody>
</table>
<p id="Thmtheorem2.p1.5" class="ltx_p"><span id="Thmtheorem2.p1.5.4" class="ltx_text ltx_font_italic">is a kernel with bandwidth hyper-parameters <math id="Thmtheorem2.p1.2.1.m1.1" class="ltx_Math" alttext="\theta_{K}" display="inline"><semantics id="Thmtheorem2.p1.2.1.m1.1a"><msub id="Thmtheorem2.p1.2.1.m1.1.1" xref="Thmtheorem2.p1.2.1.m1.1.1.cmml"><mi id="Thmtheorem2.p1.2.1.m1.1.1.2" xref="Thmtheorem2.p1.2.1.m1.1.1.2.cmml">θ</mi><mi id="Thmtheorem2.p1.2.1.m1.1.1.3" xref="Thmtheorem2.p1.2.1.m1.1.1.3.cmml">K</mi></msub><annotation-xml encoding="MathML-Content" id="Thmtheorem2.p1.2.1.m1.1b"><apply id="Thmtheorem2.p1.2.1.m1.1.1.cmml" xref="Thmtheorem2.p1.2.1.m1.1.1"><csymbol cd="ambiguous" id="Thmtheorem2.p1.2.1.m1.1.1.1.cmml" xref="Thmtheorem2.p1.2.1.m1.1.1">subscript</csymbol><ci id="Thmtheorem2.p1.2.1.m1.1.1.2.cmml" xref="Thmtheorem2.p1.2.1.m1.1.1.2">𝜃</ci><ci id="Thmtheorem2.p1.2.1.m1.1.1.3.cmml" xref="Thmtheorem2.p1.2.1.m1.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem2.p1.2.1.m1.1c">\theta_{K}</annotation></semantics></math> and <math id="Thmtheorem2.p1.3.2.m2.1" class="ltx_Math" alttext="\theta_{Q}" display="inline"><semantics id="Thmtheorem2.p1.3.2.m2.1a"><msub id="Thmtheorem2.p1.3.2.m2.1.1" xref="Thmtheorem2.p1.3.2.m2.1.1.cmml"><mi id="Thmtheorem2.p1.3.2.m2.1.1.2" xref="Thmtheorem2.p1.3.2.m2.1.1.2.cmml">θ</mi><mi id="Thmtheorem2.p1.3.2.m2.1.1.3" xref="Thmtheorem2.p1.3.2.m2.1.1.3.cmml">Q</mi></msub><annotation-xml encoding="MathML-Content" id="Thmtheorem2.p1.3.2.m2.1b"><apply id="Thmtheorem2.p1.3.2.m2.1.1.cmml" xref="Thmtheorem2.p1.3.2.m2.1.1"><csymbol cd="ambiguous" id="Thmtheorem2.p1.3.2.m2.1.1.1.cmml" xref="Thmtheorem2.p1.3.2.m2.1.1">subscript</csymbol><ci id="Thmtheorem2.p1.3.2.m2.1.1.2.cmml" xref="Thmtheorem2.p1.3.2.m2.1.1.2">𝜃</ci><ci id="Thmtheorem2.p1.3.2.m2.1.1.3.cmml" xref="Thmtheorem2.p1.3.2.m2.1.1.3">𝑄</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem2.p1.3.2.m2.1c">\theta_{Q}</annotation></semantics></math>.
Then given the same input sequence <math id="Thmtheorem2.p1.4.3.m3.3" class="ltx_Math" alttext="x_{1},\dots,x_{T}" display="inline"><semantics id="Thmtheorem2.p1.4.3.m3.3a"><mrow id="Thmtheorem2.p1.4.3.m3.3.3.2" xref="Thmtheorem2.p1.4.3.m3.3.3.3.cmml"><msub id="Thmtheorem2.p1.4.3.m3.2.2.1.1" xref="Thmtheorem2.p1.4.3.m3.2.2.1.1.cmml"><mi id="Thmtheorem2.p1.4.3.m3.2.2.1.1.2" xref="Thmtheorem2.p1.4.3.m3.2.2.1.1.2.cmml">x</mi><mn id="Thmtheorem2.p1.4.3.m3.2.2.1.1.3" xref="Thmtheorem2.p1.4.3.m3.2.2.1.1.3.cmml">1</mn></msub><mo id="Thmtheorem2.p1.4.3.m3.3.3.2.3" xref="Thmtheorem2.p1.4.3.m3.3.3.3.cmml">,</mo><mi mathvariant="normal" id="Thmtheorem2.p1.4.3.m3.1.1" xref="Thmtheorem2.p1.4.3.m3.1.1.cmml">…</mi><mo id="Thmtheorem2.p1.4.3.m3.3.3.2.4" xref="Thmtheorem2.p1.4.3.m3.3.3.3.cmml">,</mo><msub id="Thmtheorem2.p1.4.3.m3.3.3.2.2" xref="Thmtheorem2.p1.4.3.m3.3.3.2.2.cmml"><mi id="Thmtheorem2.p1.4.3.m3.3.3.2.2.2" xref="Thmtheorem2.p1.4.3.m3.3.3.2.2.2.cmml">x</mi><mi id="Thmtheorem2.p1.4.3.m3.3.3.2.2.3" xref="Thmtheorem2.p1.4.3.m3.3.3.2.2.3.cmml">T</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="Thmtheorem2.p1.4.3.m3.3b"><list id="Thmtheorem2.p1.4.3.m3.3.3.3.cmml" xref="Thmtheorem2.p1.4.3.m3.3.3.2"><apply id="Thmtheorem2.p1.4.3.m3.2.2.1.1.cmml" xref="Thmtheorem2.p1.4.3.m3.2.2.1.1"><csymbol cd="ambiguous" id="Thmtheorem2.p1.4.3.m3.2.2.1.1.1.cmml" xref="Thmtheorem2.p1.4.3.m3.2.2.1.1">subscript</csymbol><ci id="Thmtheorem2.p1.4.3.m3.2.2.1.1.2.cmml" xref="Thmtheorem2.p1.4.3.m3.2.2.1.1.2">𝑥</ci><cn type="integer" id="Thmtheorem2.p1.4.3.m3.2.2.1.1.3.cmml" xref="Thmtheorem2.p1.4.3.m3.2.2.1.1.3">1</cn></apply><ci id="Thmtheorem2.p1.4.3.m3.1.1.cmml" xref="Thmtheorem2.p1.4.3.m3.1.1">…</ci><apply id="Thmtheorem2.p1.4.3.m3.3.3.2.2.cmml" xref="Thmtheorem2.p1.4.3.m3.3.3.2.2"><csymbol cd="ambiguous" id="Thmtheorem2.p1.4.3.m3.3.3.2.2.1.cmml" xref="Thmtheorem2.p1.4.3.m3.3.3.2.2">subscript</csymbol><ci id="Thmtheorem2.p1.4.3.m3.3.3.2.2.2.cmml" xref="Thmtheorem2.p1.4.3.m3.3.3.2.2.2">𝑥</ci><ci id="Thmtheorem2.p1.4.3.m3.3.3.2.2.3.cmml" xref="Thmtheorem2.p1.4.3.m3.3.3.2.2.3">𝑇</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem2.p1.4.3.m3.3c">x_{1},\dots,x_{T}</annotation></semantics></math>, the output rule defined in Equation <a href="#S2.E5" title="In 2.3 Learning a self-supervised task for TTT ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> produces the same output sequence <math id="Thmtheorem2.p1.5.4.m4.3" class="ltx_Math" alttext="z_{1},\dots,z_{T}" display="inline"><semantics id="Thmtheorem2.p1.5.4.m4.3a"><mrow id="Thmtheorem2.p1.5.4.m4.3.3.2" xref="Thmtheorem2.p1.5.4.m4.3.3.3.cmml"><msub id="Thmtheorem2.p1.5.4.m4.2.2.1.1" xref="Thmtheorem2.p1.5.4.m4.2.2.1.1.cmml"><mi id="Thmtheorem2.p1.5.4.m4.2.2.1.1.2" xref="Thmtheorem2.p1.5.4.m4.2.2.1.1.2.cmml">z</mi><mn id="Thmtheorem2.p1.5.4.m4.2.2.1.1.3" xref="Thmtheorem2.p1.5.4.m4.2.2.1.1.3.cmml">1</mn></msub><mo id="Thmtheorem2.p1.5.4.m4.3.3.2.3" xref="Thmtheorem2.p1.5.4.m4.3.3.3.cmml">,</mo><mi mathvariant="normal" id="Thmtheorem2.p1.5.4.m4.1.1" xref="Thmtheorem2.p1.5.4.m4.1.1.cmml">…</mi><mo id="Thmtheorem2.p1.5.4.m4.3.3.2.4" xref="Thmtheorem2.p1.5.4.m4.3.3.3.cmml">,</mo><msub id="Thmtheorem2.p1.5.4.m4.3.3.2.2" xref="Thmtheorem2.p1.5.4.m4.3.3.2.2.cmml"><mi id="Thmtheorem2.p1.5.4.m4.3.3.2.2.2" xref="Thmtheorem2.p1.5.4.m4.3.3.2.2.2.cmml">z</mi><mi id="Thmtheorem2.p1.5.4.m4.3.3.2.2.3" xref="Thmtheorem2.p1.5.4.m4.3.3.2.2.3.cmml">T</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="Thmtheorem2.p1.5.4.m4.3b"><list id="Thmtheorem2.p1.5.4.m4.3.3.3.cmml" xref="Thmtheorem2.p1.5.4.m4.3.3.2"><apply id="Thmtheorem2.p1.5.4.m4.2.2.1.1.cmml" xref="Thmtheorem2.p1.5.4.m4.2.2.1.1"><csymbol cd="ambiguous" id="Thmtheorem2.p1.5.4.m4.2.2.1.1.1.cmml" xref="Thmtheorem2.p1.5.4.m4.2.2.1.1">subscript</csymbol><ci id="Thmtheorem2.p1.5.4.m4.2.2.1.1.2.cmml" xref="Thmtheorem2.p1.5.4.m4.2.2.1.1.2">𝑧</ci><cn type="integer" id="Thmtheorem2.p1.5.4.m4.2.2.1.1.3.cmml" xref="Thmtheorem2.p1.5.4.m4.2.2.1.1.3">1</cn></apply><ci id="Thmtheorem2.p1.5.4.m4.1.1.cmml" xref="Thmtheorem2.p1.5.4.m4.1.1">…</ci><apply id="Thmtheorem2.p1.5.4.m4.3.3.2.2.cmml" xref="Thmtheorem2.p1.5.4.m4.3.3.2.2"><csymbol cd="ambiguous" id="Thmtheorem2.p1.5.4.m4.3.3.2.2.1.cmml" xref="Thmtheorem2.p1.5.4.m4.3.3.2.2">subscript</csymbol><ci id="Thmtheorem2.p1.5.4.m4.3.3.2.2.2.cmml" xref="Thmtheorem2.p1.5.4.m4.3.3.2.2.2">𝑧</ci><ci id="Thmtheorem2.p1.5.4.m4.3.3.2.2.3.cmml" xref="Thmtheorem2.p1.5.4.m4.3.3.2.2.3">𝑇</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="Thmtheorem2.p1.5.4.m4.3c">z_{1},\dots,z_{T}</annotation></semantics></math> as self-attention.</span></p>
</div>
</div>
<div id="S2.SS6.2" class="ltx_proof">
<h6 class="ltx_title ltx_runin ltx_font_italic ltx_title_proof">Proof.</h6>
<div id="S2.SS6.2.p1" class="ltx_para">
<p id="S2.SS6.2.p1.2" class="ltx_p">Plugging <math id="S2.SS6.2.p1.1.m1.1" class="ltx_Math" alttext="y_{s}" display="inline"><semantics id="S2.SS6.2.p1.1.m1.1a"><msub id="S2.SS6.2.p1.1.m1.1.1" xref="S2.SS6.2.p1.1.m1.1.1.cmml"><mi id="S2.SS6.2.p1.1.m1.1.1.2" xref="S2.SS6.2.p1.1.m1.1.1.2.cmml">y</mi><mi id="S2.SS6.2.p1.1.m1.1.1.3" xref="S2.SS6.2.p1.1.m1.1.1.3.cmml">s</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS6.2.p1.1.m1.1b"><apply id="S2.SS6.2.p1.1.m1.1.1.cmml" xref="S2.SS6.2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS6.2.p1.1.m1.1.1.1.cmml" xref="S2.SS6.2.p1.1.m1.1.1">subscript</csymbol><ci id="S2.SS6.2.p1.1.m1.1.1.2.cmml" xref="S2.SS6.2.p1.1.m1.1.1.2">𝑦</ci><ci id="S2.SS6.2.p1.1.m1.1.1.3.cmml" xref="S2.SS6.2.p1.1.m1.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.2.p1.1.m1.1c">y_{s}</annotation></semantics></math> and <math id="S2.SS6.2.p1.2.m2.1" class="ltx_Math" alttext="\kappa" display="inline"><semantics id="S2.SS6.2.p1.2.m2.1a"><mi id="S2.SS6.2.p1.2.m2.1.1" xref="S2.SS6.2.p1.2.m2.1.1.cmml">κ</mi><annotation-xml encoding="MathML-Content" id="S2.SS6.2.p1.2.m2.1b"><ci id="S2.SS6.2.p1.2.m2.1.1.cmml" xref="S2.SS6.2.p1.2.m2.1.1">𝜅</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.2.p1.2.m2.1c">\kappa</annotation></semantics></math> above into Equation <a href="#S2.E8" title="In Theorem 2. ‣ 2.6 Theoretical equivalences ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> gives us the definition of self-attention.
∎</p>
</div>
</div>
<div id="S2.SS6.p4" class="ltx_para">
<p id="S2.SS6.p4.1" class="ltx_p">Appendix <a href="#A2" title="Appendix B Nadaraya-Watson estimator ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a> contains a detailed explanation of the Nadaraya-Watson estimator and kernel <math id="S2.SS6.p4.1.m1.1" class="ltx_Math" alttext="\kappa" display="inline"><semantics id="S2.SS6.p4.1.m1.1a"><mi id="S2.SS6.p4.1.m1.1.1" xref="S2.SS6.p4.1.m1.1.1.cmml">κ</mi><annotation-xml encoding="MathML-Content" id="S2.SS6.p4.1.m1.1b"><ci id="S2.SS6.p4.1.m1.1.1.cmml" xref="S2.SS6.p4.1.m1.1.1">𝜅</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.p4.1.m1.1c">\kappa</annotation></semantics></math> above.
In contrast to Theorem <a href="#Thmtheorem1" title="Theorem 1. ‣ 2.6 Theoretical equivalences ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, Theorem <a href="#Thmtheorem2" title="Theorem 2. ‣ 2.6 Theoretical equivalences ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> does not produce a different implementation from attention.</p>
</div>
<div id="S2.SS6.p5" class="ltx_para">
<p id="S2.SS6.p5.5" class="ltx_p">For the TTT layer above, the hidden state is <math id="S2.SS6.p5.1.m1.3" class="ltx_Math" alttext="x_{1},\dots,x_{t}" display="inline"><semantics id="S2.SS6.p5.1.m1.3a"><mrow id="S2.SS6.p5.1.m1.3.3.2" xref="S2.SS6.p5.1.m1.3.3.3.cmml"><msub id="S2.SS6.p5.1.m1.2.2.1.1" xref="S2.SS6.p5.1.m1.2.2.1.1.cmml"><mi id="S2.SS6.p5.1.m1.2.2.1.1.2" xref="S2.SS6.p5.1.m1.2.2.1.1.2.cmml">x</mi><mn id="S2.SS6.p5.1.m1.2.2.1.1.3" xref="S2.SS6.p5.1.m1.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.SS6.p5.1.m1.3.3.2.3" xref="S2.SS6.p5.1.m1.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS6.p5.1.m1.1.1" xref="S2.SS6.p5.1.m1.1.1.cmml">…</mi><mo id="S2.SS6.p5.1.m1.3.3.2.4" xref="S2.SS6.p5.1.m1.3.3.3.cmml">,</mo><msub id="S2.SS6.p5.1.m1.3.3.2.2" xref="S2.SS6.p5.1.m1.3.3.2.2.cmml"><mi id="S2.SS6.p5.1.m1.3.3.2.2.2" xref="S2.SS6.p5.1.m1.3.3.2.2.2.cmml">x</mi><mi id="S2.SS6.p5.1.m1.3.3.2.2.3" xref="S2.SS6.p5.1.m1.3.3.2.2.3.cmml">t</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS6.p5.1.m1.3b"><list id="S2.SS6.p5.1.m1.3.3.3.cmml" xref="S2.SS6.p5.1.m1.3.3.2"><apply id="S2.SS6.p5.1.m1.2.2.1.1.cmml" xref="S2.SS6.p5.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS6.p5.1.m1.2.2.1.1.1.cmml" xref="S2.SS6.p5.1.m1.2.2.1.1">subscript</csymbol><ci id="S2.SS6.p5.1.m1.2.2.1.1.2.cmml" xref="S2.SS6.p5.1.m1.2.2.1.1.2">𝑥</ci><cn type="integer" id="S2.SS6.p5.1.m1.2.2.1.1.3.cmml" xref="S2.SS6.p5.1.m1.2.2.1.1.3">1</cn></apply><ci id="S2.SS6.p5.1.m1.1.1.cmml" xref="S2.SS6.p5.1.m1.1.1">…</ci><apply id="S2.SS6.p5.1.m1.3.3.2.2.cmml" xref="S2.SS6.p5.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S2.SS6.p5.1.m1.3.3.2.2.1.cmml" xref="S2.SS6.p5.1.m1.3.3.2.2">subscript</csymbol><ci id="S2.SS6.p5.1.m1.3.3.2.2.2.cmml" xref="S2.SS6.p5.1.m1.3.3.2.2.2">𝑥</ci><ci id="S2.SS6.p5.1.m1.3.3.2.2.3.cmml" xref="S2.SS6.p5.1.m1.3.3.2.2.3">𝑡</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.p5.1.m1.3c">x_{1},\dots,x_{t}</annotation></semantics></math> or a similar list of processed training data, the update rule adds <math id="S2.SS6.p5.2.m2.1" class="ltx_Math" alttext="x_{t}" display="inline"><semantics id="S2.SS6.p5.2.m2.1a"><msub id="S2.SS6.p5.2.m2.1.1" xref="S2.SS6.p5.2.m2.1.1.cmml"><mi id="S2.SS6.p5.2.m2.1.1.2" xref="S2.SS6.p5.2.m2.1.1.2.cmml">x</mi><mi id="S2.SS6.p5.2.m2.1.1.3" xref="S2.SS6.p5.2.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS6.p5.2.m2.1b"><apply id="S2.SS6.p5.2.m2.1.1.cmml" xref="S2.SS6.p5.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS6.p5.2.m2.1.1.1.cmml" xref="S2.SS6.p5.2.m2.1.1">subscript</csymbol><ci id="S2.SS6.p5.2.m2.1.1.2.cmml" xref="S2.SS6.p5.2.m2.1.1.2">𝑥</ci><ci id="S2.SS6.p5.2.m2.1.1.3.cmml" xref="S2.SS6.p5.2.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.p5.2.m2.1c">x_{t}</annotation></semantics></math> to the list, and the output rule scans the list with <math id="S2.SS6.p5.3.m3.1" class="ltx_Math" alttext="\kappa" display="inline"><semantics id="S2.SS6.p5.3.m3.1a"><mi id="S2.SS6.p5.3.m3.1.1" xref="S2.SS6.p5.3.m3.1.1.cmml">κ</mi><annotation-xml encoding="MathML-Content" id="S2.SS6.p5.3.m3.1b"><ci id="S2.SS6.p5.3.m3.1.1.cmml" xref="S2.SS6.p5.3.m3.1.1">𝜅</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.p5.3.m3.1c">\kappa</annotation></semantics></math>.
In previous subsections, our hidden state has been defined as <math id="S2.SS6.p5.4.m4.1" class="ltx_Math" alttext="W_{t}" display="inline"><semantics id="S2.SS6.p5.4.m4.1a"><msub id="S2.SS6.p5.4.m4.1.1" xref="S2.SS6.p5.4.m4.1.1.cmml"><mi id="S2.SS6.p5.4.m4.1.1.2" xref="S2.SS6.p5.4.m4.1.1.2.cmml">W</mi><mi id="S2.SS6.p5.4.m4.1.1.3" xref="S2.SS6.p5.4.m4.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS6.p5.4.m4.1b"><apply id="S2.SS6.p5.4.m4.1.1.cmml" xref="S2.SS6.p5.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS6.p5.4.m4.1.1.1.cmml" xref="S2.SS6.p5.4.m4.1.1">subscript</csymbol><ci id="S2.SS6.p5.4.m4.1.1.2.cmml" xref="S2.SS6.p5.4.m4.1.1.2">𝑊</ci><ci id="S2.SS6.p5.4.m4.1.1.3.cmml" xref="S2.SS6.p5.4.m4.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.p5.4.m4.1c">W_{t}</annotation></semantics></math>, the update rule a gradient step, and the output rule a call to <math id="S2.SS6.p5.5.m5.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S2.SS6.p5.5.m5.1a"><mi id="S2.SS6.p5.5.m5.1.1" xref="S2.SS6.p5.5.m5.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S2.SS6.p5.5.m5.1b"><ci id="S2.SS6.p5.5.m5.1.1.cmml" xref="S2.SS6.p5.5.m5.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.p5.5.m5.1c">f</annotation></semantics></math>.
To unify these two constructions, we define a new abstraction called a learner, which uniquely induces a TTT layer.</p>
</div>
<div id="S2.SS6.p6" class="ltx_para">
<p id="S2.SS6.p6.1" class="ltx_p">Similar to its definition in standard machine learning packages <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite>, all learners need to implement two methods: <span id="S2.SS6.p6.1.1" class="ltx_text">train and predict.</span>
Now we redefine the hidden state of the induced TTT layer as the internal storage of the learner, and the update and output rules as the train and predict methods.
Under this new definition of TTT layers, both parametric learners such as that in Theorem <a href="#Thmtheorem1" title="Theorem 1. ‣ 2.6 Theoretical equivalences ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> and nonparametric learners such as that in Theorem <a href="#Thmtheorem2" title="Theorem 2. ‣ 2.6 Theoretical equivalences ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> can be included.
Figure <a href="#S2.F10.fig1" title="Figure 10 ‣ 2.6 Theoretical equivalences ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a> summarizes this general definition of TTT layers in the broader scope of all sequence modeling layers.</p>
</div>
<div id="S2.SS6.p7" class="ltx_para">
<p id="S2.SS6.p7.1" class="ltx_p">This general definition has an additional benefit for parametric learners:
There can be more objects other than <math id="S2.SS6.p7.1.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.SS6.p7.1.m1.1a"><mi id="S2.SS6.p7.1.m1.1.1" xref="S2.SS6.p7.1.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.SS6.p7.1.m1.1b"><ci id="S2.SS6.p7.1.m1.1.1.cmml" xref="S2.SS6.p7.1.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS6.p7.1.m1.1c">W</annotation></semantics></math> in the internal storage of parametric learners, such as the optimizer state, which will also be included in the hidden state of the induced TTT layer.
This extension allows TTT layers to use more sophisticated optimizers such as Adam <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib42" title="" class="ltx_ref">42</a>]</cite> in future work.</p>
</div>
<figure id="S2.T1" class="ltx_table">
<div class="ltx_flex_figure ltx_flex_table">
<div class="ltx_flex_cell ltx_flex_size_1">
<div id="S2.T1.8.8" class="ltx_inline-block ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle ltx_transformed_outer" style="width:195.1pt;height:149pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-12.9pt,6.5pt) scale(0.92,0.92) ;">
<table id="S2.T1.8.8.8" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S2.T1.8.8.8.9.1" class="ltx_tr">
<th id="S2.T1.8.8.8.9.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="S2.T1.8.8.8.9.1.1.1" class="ltx_text ltx_font_bold">Configuration</span></th>
<td id="S2.T1.8.8.8.9.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="S2.T1.8.8.8.9.1.2.1" class="ltx_text ltx_font_bold">Ppl.</span></td>
<td id="S2.T1.8.8.8.9.1.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;"><span id="S2.T1.8.8.8.9.1.3.1" class="ltx_text ltx_font_bold">Diff.</span></td>
</tr>
<tr id="S2.T1.8.8.8.10.2" class="ltx_tr">
<th id="S2.T1.8.8.8.10.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">Linear attention <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib41" title="" class="ltx_ref">41</a>]</cite>
</th>
<td id="S2.T1.8.8.8.10.2.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">15.91</td>
<td id="S2.T1.8.8.8.10.2.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">-</td>
</tr>
<tr id="S2.T1.1.1.1.1" class="ltx_tr">
<th id="S2.T1.1.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r" style="padding-top:2.5pt;padding-bottom:2.5pt;">Linear attn. improved</th>
<td id="S2.T1.1.1.1.1.3" class="ltx_td ltx_align_center ltx_border_r" style="padding-top:2.5pt;padding-bottom:2.5pt;">15.23</td>
<td id="S2.T1.1.1.1.1.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-top:2.5pt;padding-bottom:2.5pt;">
<math id="S2.T1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="-" display="inline"><semantics id="S2.T1.1.1.1.1.1.m1.1a"><mo id="S2.T1.1.1.1.1.1.m1.1.1" xref="S2.T1.1.1.1.1.1.m1.1.1.cmml">−</mo><annotation-xml encoding="MathML-Content" id="S2.T1.1.1.1.1.1.m1.1b"><minus id="S2.T1.1.1.1.1.1.m1.1.1.cmml" xref="S2.T1.1.1.1.1.1.m1.1.1"></minus></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.1.1.1.1.1.m1.1c">-</annotation></semantics></math>0.68</td>
</tr>
<tr id="S2.T1.8.8.8.11.3" class="ltx_tr">
<th id="S2.T1.8.8.8.11.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">TTT equivalence</th>
<td id="S2.T1.8.8.8.11.3.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">15.23</td>
<td id="S2.T1.8.8.8.11.3.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">0</td>
</tr>
<tr id="S2.T1.2.2.2.2" class="ltx_tr">
<th id="S2.T1.2.2.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r" style="padding-top:2.5pt;padding-bottom:2.5pt;">+ learnable <math id="S2.T1.2.2.2.2.1.m1.1" class="ltx_Math" alttext="W_{0}" display="inline"><semantics id="S2.T1.2.2.2.2.1.m1.1a"><msub id="S2.T1.2.2.2.2.1.m1.1.1" xref="S2.T1.2.2.2.2.1.m1.1.1.cmml"><mi id="S2.T1.2.2.2.2.1.m1.1.1.2" xref="S2.T1.2.2.2.2.1.m1.1.1.2.cmml">W</mi><mn id="S2.T1.2.2.2.2.1.m1.1.1.3" xref="S2.T1.2.2.2.2.1.m1.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.2.2.2.2.1.m1.1b"><apply id="S2.T1.2.2.2.2.1.m1.1.1.cmml" xref="S2.T1.2.2.2.2.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.2.2.2.2.1.m1.1.1.1.cmml" xref="S2.T1.2.2.2.2.1.m1.1.1">subscript</csymbol><ci id="S2.T1.2.2.2.2.1.m1.1.1.2.cmml" xref="S2.T1.2.2.2.2.1.m1.1.1.2">𝑊</ci><cn type="integer" id="S2.T1.2.2.2.2.1.m1.1.1.3.cmml" xref="S2.T1.2.2.2.2.1.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.2.2.2.2.1.m1.1c">W_{0}</annotation></semantics></math>
</th>
<td id="S2.T1.2.2.2.2.2" class="ltx_td ltx_align_center ltx_border_r" style="padding-top:2.5pt;padding-bottom:2.5pt;">15.27</td>
<td id="S2.T1.2.2.2.2.3" class="ltx_td ltx_align_center ltx_border_r" style="padding-top:2.5pt;padding-bottom:2.5pt;">+0.04</td>
</tr>
<tr id="S2.T1.4.4.4.4" class="ltx_tr">
<th id="S2.T1.3.3.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r" style="padding-top:2.5pt;padding-bottom:2.5pt;">+ LN and residual in <math id="S2.T1.3.3.3.3.1.m1.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S2.T1.3.3.3.3.1.m1.1a"><mi id="S2.T1.3.3.3.3.1.m1.1.1" xref="S2.T1.3.3.3.3.1.m1.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S2.T1.3.3.3.3.1.m1.1b"><ci id="S2.T1.3.3.3.3.1.m1.1.1.cmml" xref="S2.T1.3.3.3.3.1.m1.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.3.3.3.3.1.m1.1c">f</annotation></semantics></math>
</th>
<td id="S2.T1.4.4.4.4.3" class="ltx_td ltx_align_center ltx_border_r" style="padding-top:2.5pt;padding-bottom:2.5pt;">14.05</td>
<td id="S2.T1.4.4.4.4.2" class="ltx_td ltx_align_center ltx_border_r" style="padding-top:2.5pt;padding-bottom:2.5pt;">
<math id="S2.T1.4.4.4.4.2.m1.1" class="ltx_Math" alttext="-" display="inline"><semantics id="S2.T1.4.4.4.4.2.m1.1a"><mo id="S2.T1.4.4.4.4.2.m1.1.1" xref="S2.T1.4.4.4.4.2.m1.1.1.cmml">−</mo><annotation-xml encoding="MathML-Content" id="S2.T1.4.4.4.4.2.m1.1b"><minus id="S2.T1.4.4.4.4.2.m1.1.1.cmml" xref="S2.T1.4.4.4.4.2.m1.1.1"></minus></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.4.4.4.4.2.m1.1c">-</annotation></semantics></math>1.22</td>
</tr>
<tr id="S2.T1.5.5.5.5" class="ltx_tr">
<th id="S2.T1.5.5.5.5.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r" style="padding-top:2.5pt;padding-bottom:2.5pt;">+ mini-batch TTT</th>
<td id="S2.T1.5.5.5.5.3" class="ltx_td ltx_align_center ltx_border_r" style="padding-top:2.5pt;padding-bottom:2.5pt;">12.35</td>
<td id="S2.T1.5.5.5.5.1" class="ltx_td ltx_align_center ltx_border_r" style="padding-top:2.5pt;padding-bottom:2.5pt;">
<math id="S2.T1.5.5.5.5.1.m1.1" class="ltx_Math" alttext="-" display="inline"><semantics id="S2.T1.5.5.5.5.1.m1.1a"><mo id="S2.T1.5.5.5.5.1.m1.1.1" xref="S2.T1.5.5.5.5.1.m1.1.1.cmml">−</mo><annotation-xml encoding="MathML-Content" id="S2.T1.5.5.5.5.1.m1.1b"><minus id="S2.T1.5.5.5.5.1.m1.1.1.cmml" xref="S2.T1.5.5.5.5.1.m1.1.1"></minus></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.5.5.5.5.1.m1.1c">-</annotation></semantics></math>1.70</td>
</tr>
<tr id="S2.T1.7.7.7.7" class="ltx_tr">
<th id="S2.T1.6.6.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_l ltx_border_r" style="padding-top:2.5pt;padding-bottom:2.5pt;">+ learnable <math id="S2.T1.6.6.6.6.1.m1.1" class="ltx_Math" alttext="\eta" display="inline"><semantics id="S2.T1.6.6.6.6.1.m1.1a"><mi id="S2.T1.6.6.6.6.1.m1.1.1" xref="S2.T1.6.6.6.6.1.m1.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S2.T1.6.6.6.6.1.m1.1b"><ci id="S2.T1.6.6.6.6.1.m1.1.1.cmml" xref="S2.T1.6.6.6.6.1.m1.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.6.6.6.6.1.m1.1c">\eta</annotation></semantics></math>
</th>
<td id="S2.T1.7.7.7.7.3" class="ltx_td ltx_align_center ltx_border_r" style="padding-top:2.5pt;padding-bottom:2.5pt;">11.99</td>
<td id="S2.T1.7.7.7.7.2" class="ltx_td ltx_align_center ltx_border_r" style="padding-top:2.5pt;padding-bottom:2.5pt;">
<math id="S2.T1.7.7.7.7.2.m1.1" class="ltx_Math" alttext="-" display="inline"><semantics id="S2.T1.7.7.7.7.2.m1.1a"><mo id="S2.T1.7.7.7.7.2.m1.1.1" xref="S2.T1.7.7.7.7.2.m1.1.1.cmml">−</mo><annotation-xml encoding="MathML-Content" id="S2.T1.7.7.7.7.2.m1.1b"><minus id="S2.T1.7.7.7.7.2.m1.1.1.cmml" xref="S2.T1.7.7.7.7.2.m1.1.1"></minus></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.7.7.7.7.2.m1.1c">-</annotation></semantics></math>0.36</td>
</tr>
<tr id="S2.T1.8.8.8.8" class="ltx_tr">
<th id="S2.T1.8.8.8.8.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_b ltx_border_l ltx_border_r" style="padding-top:2.5pt;padding-bottom:2.5pt;">+ Mamba backbone</th>
<td id="S2.T1.8.8.8.8.3" class="ltx_td ltx_align_center ltx_border_b ltx_border_r" style="padding-top:2.5pt;padding-bottom:2.5pt;">11.09</td>
<td id="S2.T1.8.8.8.8.1" class="ltx_td ltx_align_center ltx_border_b ltx_border_r" style="padding-top:2.5pt;padding-bottom:2.5pt;">
<math id="S2.T1.8.8.8.8.1.m1.1" class="ltx_Math" alttext="-" display="inline"><semantics id="S2.T1.8.8.8.8.1.m1.1a"><mo id="S2.T1.8.8.8.8.1.m1.1.1" xref="S2.T1.8.8.8.8.1.m1.1.1.cmml">−</mo><annotation-xml encoding="MathML-Content" id="S2.T1.8.8.8.8.1.m1.1b"><minus id="S2.T1.8.8.8.8.1.m1.1.1.cmml" xref="S2.T1.8.8.8.8.1.m1.1.1"></minus></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.8.8.8.8.1.m1.1c">-</annotation></semantics></math>0.90</td>
</tr>
</tbody>
</table>
</span></div>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S2.T1.16" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:216.8pt;">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Table 1: </span>Ablations on improving from linear attention.
All models here have 125M parameters, and are trained according to the recipe in Subsection <a href="#S3.SS1" title="3.1 Short context: the Pile ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a>.
The last row, with perplexity 11.09, is the final result of TTT-Linear in Figure <a href="#S3.F11" title="Figure 11 ‣ 3.1 Short context: the Pile ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a>.
Starting from the equivalence discussed in Subsection <a href="#S2.SS6" title="2.6 Theoretical equivalences ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.6</span></a>, learnable <math id="S2.T1.13.5.m1.1" class="ltx_Math" alttext="W_{0}" display="inline"><semantics id="S2.T1.13.5.m1.1b"><msub id="S2.T1.13.5.m1.1.1" xref="S2.T1.13.5.m1.1.1.cmml"><mi id="S2.T1.13.5.m1.1.1.2" xref="S2.T1.13.5.m1.1.1.2.cmml">W</mi><mn id="S2.T1.13.5.m1.1.1.3" xref="S2.T1.13.5.m1.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.13.5.m1.1c"><apply id="S2.T1.13.5.m1.1.1.cmml" xref="S2.T1.13.5.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.13.5.m1.1.1.1.cmml" xref="S2.T1.13.5.m1.1.1">subscript</csymbol><ci id="S2.T1.13.5.m1.1.1.2.cmml" xref="S2.T1.13.5.m1.1.1.2">𝑊</ci><cn type="integer" id="S2.T1.13.5.m1.1.1.3.cmml" xref="S2.T1.13.5.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.13.5.m1.1d">W_{0}</annotation></semantics></math> hurts slightly, but the rows below cannot train stably without it.
The biggest improvement comes from mini-batch TTT (changing from <math id="S2.T1.14.6.m2.1" class="ltx_Math" alttext="b=T=2048" display="inline"><semantics id="S2.T1.14.6.m2.1b"><mrow id="S2.T1.14.6.m2.1.1" xref="S2.T1.14.6.m2.1.1.cmml"><mi id="S2.T1.14.6.m2.1.1.2" xref="S2.T1.14.6.m2.1.1.2.cmml">b</mi><mo id="S2.T1.14.6.m2.1.1.3" xref="S2.T1.14.6.m2.1.1.3.cmml">=</mo><mi id="S2.T1.14.6.m2.1.1.4" xref="S2.T1.14.6.m2.1.1.4.cmml">T</mi><mo id="S2.T1.14.6.m2.1.1.5" xref="S2.T1.14.6.m2.1.1.5.cmml">=</mo><mn id="S2.T1.14.6.m2.1.1.6" xref="S2.T1.14.6.m2.1.1.6.cmml">2048</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.T1.14.6.m2.1c"><apply id="S2.T1.14.6.m2.1.1.cmml" xref="S2.T1.14.6.m2.1.1"><and id="S2.T1.14.6.m2.1.1a.cmml" xref="S2.T1.14.6.m2.1.1"></and><apply id="S2.T1.14.6.m2.1.1b.cmml" xref="S2.T1.14.6.m2.1.1"><eq id="S2.T1.14.6.m2.1.1.3.cmml" xref="S2.T1.14.6.m2.1.1.3"></eq><ci id="S2.T1.14.6.m2.1.1.2.cmml" xref="S2.T1.14.6.m2.1.1.2">𝑏</ci><ci id="S2.T1.14.6.m2.1.1.4.cmml" xref="S2.T1.14.6.m2.1.1.4">𝑇</ci></apply><apply id="S2.T1.14.6.m2.1.1c.cmml" xref="S2.T1.14.6.m2.1.1"><eq id="S2.T1.14.6.m2.1.1.5.cmml" xref="S2.T1.14.6.m2.1.1.5"></eq><share href="#S2.T1.14.6.m2.1.1.4.cmml" id="S2.T1.14.6.m2.1.1d.cmml" xref="S2.T1.14.6.m2.1.1"></share><cn type="integer" id="S2.T1.14.6.m2.1.1.6.cmml" xref="S2.T1.14.6.m2.1.1.6">2048</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.14.6.m2.1d">b=T=2048</annotation></semantics></math> to <math id="S2.T1.15.7.m3.1" class="ltx_Math" alttext="b=16" display="inline"><semantics id="S2.T1.15.7.m3.1b"><mrow id="S2.T1.15.7.m3.1.1" xref="S2.T1.15.7.m3.1.1.cmml"><mi id="S2.T1.15.7.m3.1.1.2" xref="S2.T1.15.7.m3.1.1.2.cmml">b</mi><mo id="S2.T1.15.7.m3.1.1.1" xref="S2.T1.15.7.m3.1.1.1.cmml">=</mo><mn id="S2.T1.15.7.m3.1.1.3" xref="S2.T1.15.7.m3.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.T1.15.7.m3.1c"><apply id="S2.T1.15.7.m3.1.1.cmml" xref="S2.T1.15.7.m3.1.1"><eq id="S2.T1.15.7.m3.1.1.1.cmml" xref="S2.T1.15.7.m3.1.1.1"></eq><ci id="S2.T1.15.7.m3.1.1.2.cmml" xref="S2.T1.15.7.m3.1.1.2">𝑏</ci><cn type="integer" id="S2.T1.15.7.m3.1.1.3.cmml" xref="S2.T1.15.7.m3.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.15.7.m3.1d">b=16</annotation></semantics></math>).
The second comes from instantiating the inner model <math id="S2.T1.16.8.m4.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S2.T1.16.8.m4.1b"><mi id="S2.T1.16.8.m4.1.1" xref="S2.T1.16.8.m4.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S2.T1.16.8.m4.1c"><ci id="S2.T1.16.8.m4.1.1.cmml" xref="S2.T1.16.8.m4.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.16.8.m4.1d">f</annotation></semantics></math> with LN and residual connection.
Both of these designs would be difficult to come across without the conceptual framework of TTT.
</figcaption>
</figure>
</div>
</div>
</figure>
</section>
<section id="S2.SS7" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.7 </span>Implementation details</h3>

<section id="S2.SS7.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Instantiations of <math id="S2.SS7.SSS0.Px1.1.m1.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S2.SS7.SSS0.Px1.1.m1.1b"><mi id="S2.SS7.SSS0.Px1.1.m1.1.1" xref="S2.SS7.SSS0.Px1.1.m1.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px1.1.m1.1c"><ci id="S2.SS7.SSS0.Px1.1.m1.1.1.cmml" xref="S2.SS7.SSS0.Px1.1.m1.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px1.1.m1.1d">f</annotation></semantics></math>.</h5>

<div id="S2.SS7.SSS0.Px1.p1" class="ltx_para">
<p id="S2.SS7.SSS0.Px1.p1.10" class="ltx_p">We propose two variants of TTT layers – TTT-Linear and TTT-MLP, differing only in their instantiations of <math id="S2.SS7.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S2.SS7.SSS0.Px1.p1.1.m1.1a"><mi id="S2.SS7.SSS0.Px1.p1.1.m1.1.1" xref="S2.SS7.SSS0.Px1.p1.1.m1.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px1.p1.1.m1.1b"><ci id="S2.SS7.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S2.SS7.SSS0.Px1.p1.1.m1.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px1.p1.1.m1.1c">f</annotation></semantics></math>.
For TTT-Linear, <math id="S2.SS7.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="f_{\,\texttt{lin}}(x)=Wx" display="inline"><semantics id="S2.SS7.SSS0.Px1.p1.2.m2.1a"><mrow id="S2.SS7.SSS0.Px1.p1.2.m2.1.2" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.cmml"><mrow id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.cmml"><msub id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.2" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.2.cmml"><mi id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.2.2" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.2.2.cmml">f</mi><mtext class="ltx_mathvariant_monospace" id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.2.3" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.2.3a.cmml">lin</mtext></msub><mo lspace="0em" rspace="0em" id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.1" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.1.cmml">​</mo><mrow id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.3.2" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.cmml"><mo stretchy="false" id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.3.2.1" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.cmml">(</mo><mi id="S2.SS7.SSS0.Px1.p1.2.m2.1.1" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.1.cmml">x</mi><mo stretchy="false" id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.3.2.2" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.cmml">)</mo></mrow></mrow><mo id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.1" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.1.cmml">=</mo><mrow id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.3" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.3.cmml"><mi id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.3.2" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.3.2.cmml">W</mi><mo lspace="0em" rspace="0em" id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.3.1" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.3.1.cmml">​</mo><mi id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.3.3" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.3.3.cmml">x</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px1.p1.2.m2.1b"><apply id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.cmml" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2"><eq id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.1.cmml" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.1"></eq><apply id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.cmml" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2"><times id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.1.cmml" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.1"></times><apply id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.2.cmml" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.2"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.2.1.cmml" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.2">subscript</csymbol><ci id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.2.2.cmml" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.2.2">𝑓</ci><ci id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.2.3a.cmml" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.2.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.2.3.cmml" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.2.2.3">lin</mtext></ci></apply><ci id="S2.SS7.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.1">𝑥</ci></apply><apply id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.3.cmml" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.3"><times id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.3.1.cmml" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.3.1"></times><ci id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.3.2.cmml" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.3.2">𝑊</ci><ci id="S2.SS7.SSS0.Px1.p1.2.m2.1.2.3.3.cmml" xref="S2.SS7.SSS0.Px1.p1.2.m2.1.2.3.3">𝑥</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px1.p1.2.m2.1c">f_{\,\texttt{lin}}(x)=Wx</annotation></semantics></math>, where <math id="S2.SS7.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.SS7.SSS0.Px1.p1.3.m3.1a"><mi id="S2.SS7.SSS0.Px1.p1.3.m3.1.1" xref="S2.SS7.SSS0.Px1.p1.3.m3.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px1.p1.3.m3.1b"><ci id="S2.SS7.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S2.SS7.SSS0.Px1.p1.3.m3.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px1.p1.3.m3.1c">W</annotation></semantics></math> is square.
For TTT-MLP, <math id="S2.SS7.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="f_{\,\texttt{MLP}}" display="inline"><semantics id="S2.SS7.SSS0.Px1.p1.4.m4.1a"><msub id="S2.SS7.SSS0.Px1.p1.4.m4.1.1" xref="S2.SS7.SSS0.Px1.p1.4.m4.1.1.cmml"><mi id="S2.SS7.SSS0.Px1.p1.4.m4.1.1.2" xref="S2.SS7.SSS0.Px1.p1.4.m4.1.1.2.cmml">f</mi><mtext class="ltx_mathvariant_monospace" id="S2.SS7.SSS0.Px1.p1.4.m4.1.1.3" xref="S2.SS7.SSS0.Px1.p1.4.m4.1.1.3a.cmml">MLP</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px1.p1.4.m4.1b"><apply id="S2.SS7.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S2.SS7.SSS0.Px1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px1.p1.4.m4.1.1.1.cmml" xref="S2.SS7.SSS0.Px1.p1.4.m4.1.1">subscript</csymbol><ci id="S2.SS7.SSS0.Px1.p1.4.m4.1.1.2.cmml" xref="S2.SS7.SSS0.Px1.p1.4.m4.1.1.2">𝑓</ci><ci id="S2.SS7.SSS0.Px1.p1.4.m4.1.1.3a.cmml" xref="S2.SS7.SSS0.Px1.p1.4.m4.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S2.SS7.SSS0.Px1.p1.4.m4.1.1.3.cmml" xref="S2.SS7.SSS0.Px1.p1.4.m4.1.1.3">MLP</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px1.p1.4.m4.1c">f_{\,\texttt{MLP}}</annotation></semantics></math> has two layers similar to the MLPs in Transformers.
Specifically, the hidden dimension is <math id="S2.SS7.SSS0.Px1.p1.5.m5.1" class="ltx_math_unparsed" alttext="4\times" display="inline"><semantics id="S2.SS7.SSS0.Px1.p1.5.m5.1a"><mrow id="S2.SS7.SSS0.Px1.p1.5.m5.1b"><mn id="S2.SS7.SSS0.Px1.p1.5.m5.1.1">4</mn><mo lspace="0.222em" id="S2.SS7.SSS0.Px1.p1.5.m5.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px1.p1.5.m5.1c">4\times</annotation></semantics></math> the input dimension, followed by a GELU activation <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib31" title="" class="ltx_ref">31</a>]</cite>.
For better stability during TTT, <math id="S2.SS7.SSS0.Px1.p1.6.m6.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S2.SS7.SSS0.Px1.p1.6.m6.1a"><mi id="S2.SS7.SSS0.Px1.p1.6.m6.1.1" xref="S2.SS7.SSS0.Px1.p1.6.m6.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px1.p1.6.m6.1b"><ci id="S2.SS7.SSS0.Px1.p1.6.m6.1.1.cmml" xref="S2.SS7.SSS0.Px1.p1.6.m6.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px1.p1.6.m6.1c">f</annotation></semantics></math> always contains a Layer Normalization (LN) and residual connection. That is,
<math id="S2.SS7.SSS0.Px1.p1.7.m7.3" class="ltx_Math" alttext="f(x)=x+\texttt{LN}(f_{\,\texttt{res}}(x))," display="inline"><semantics id="S2.SS7.SSS0.Px1.p1.7.m7.3a"><mrow id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.cmml"><mrow id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.cmml"><mrow id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.3" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.3.cmml"><mi id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.3.2" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.3.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.3.1" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.3.1.cmml">​</mo><mrow id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.3.3.2" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.3.cmml"><mo stretchy="false" id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.3.3.2.1" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.3.cmml">(</mo><mi id="S2.SS7.SSS0.Px1.p1.7.m7.1.1" xref="S2.SS7.SSS0.Px1.p1.7.m7.1.1.cmml">x</mi><mo stretchy="false" id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.3.3.2.2" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.3.cmml">)</mo></mrow></mrow><mo id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.2" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.2.cmml">=</mo><mrow id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.cmml"><mi id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.3" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.3.cmml">x</mi><mo id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.2" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.2.cmml">+</mo><mrow id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.cmml"><mtext class="ltx_mathvariant_monospace" id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.3" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.3a.cmml">LN</mtext><mo lspace="0em" rspace="0em" id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.2" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.2.cmml">​</mo><mrow id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.2" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.cmml"><msub id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.2" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.2.cmml"><mi id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.2.2" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.2.2.cmml">f</mi><mtext class="ltx_mathvariant_monospace" id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.2.3" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.2.3a.cmml">res</mtext></msub><mo lspace="0em" rspace="0em" id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.1" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.1.cmml">​</mo><mrow id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.3.2" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.3.2.1" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.cmml">(</mo><mi id="S2.SS7.SSS0.Px1.p1.7.m7.2.2" xref="S2.SS7.SSS0.Px1.p1.7.m7.2.2.cmml">x</mi><mo stretchy="false" id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.3.2.2" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.3" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><mo id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.2" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px1.p1.7.m7.3b"><apply id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.cmml" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1"><eq id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.2.cmml" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.2"></eq><apply id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.3.cmml" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.3"><times id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.3.1.cmml" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.3.1"></times><ci id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.3.2.cmml" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.3.2">𝑓</ci><ci id="S2.SS7.SSS0.Px1.p1.7.m7.1.1.cmml" xref="S2.SS7.SSS0.Px1.p1.7.m7.1.1">𝑥</ci></apply><apply id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.cmml" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1"><plus id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.2.cmml" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.2"></plus><ci id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.3.cmml" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.3">𝑥</ci><apply id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.cmml" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1"><times id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.2.cmml" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.2"></times><ci id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.3a.cmml" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.3"><mtext class="ltx_mathvariant_monospace" id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.3.cmml" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.3">LN</mtext></ci><apply id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.cmml" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1"><times id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.1.cmml" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.1"></times><apply id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.2.cmml" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.2.1.cmml" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.2.2.cmml" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.2.2">𝑓</ci><ci id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.2.3a.cmml" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.2.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.2.3.cmml" xref="S2.SS7.SSS0.Px1.p1.7.m7.3.3.1.1.1.1.1.1.1.2.3">res</mtext></ci></apply><ci id="S2.SS7.SSS0.Px1.p1.7.m7.2.2.cmml" xref="S2.SS7.SSS0.Px1.p1.7.m7.2.2">𝑥</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px1.p1.7.m7.3c">f(x)=x+\texttt{LN}(f_{\,\texttt{res}}(x)),</annotation></semantics></math>
where <math id="S2.SS7.SSS0.Px1.p1.8.m8.1" class="ltx_Math" alttext="f_{\,\texttt{res}}" display="inline"><semantics id="S2.SS7.SSS0.Px1.p1.8.m8.1a"><msub id="S2.SS7.SSS0.Px1.p1.8.m8.1.1" xref="S2.SS7.SSS0.Px1.p1.8.m8.1.1.cmml"><mi id="S2.SS7.SSS0.Px1.p1.8.m8.1.1.2" xref="S2.SS7.SSS0.Px1.p1.8.m8.1.1.2.cmml">f</mi><mtext class="ltx_mathvariant_monospace" id="S2.SS7.SSS0.Px1.p1.8.m8.1.1.3" xref="S2.SS7.SSS0.Px1.p1.8.m8.1.1.3a.cmml">res</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px1.p1.8.m8.1b"><apply id="S2.SS7.SSS0.Px1.p1.8.m8.1.1.cmml" xref="S2.SS7.SSS0.Px1.p1.8.m8.1.1"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px1.p1.8.m8.1.1.1.cmml" xref="S2.SS7.SSS0.Px1.p1.8.m8.1.1">subscript</csymbol><ci id="S2.SS7.SSS0.Px1.p1.8.m8.1.1.2.cmml" xref="S2.SS7.SSS0.Px1.p1.8.m8.1.1.2">𝑓</ci><ci id="S2.SS7.SSS0.Px1.p1.8.m8.1.1.3a.cmml" xref="S2.SS7.SSS0.Px1.p1.8.m8.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S2.SS7.SSS0.Px1.p1.8.m8.1.1.3.cmml" xref="S2.SS7.SSS0.Px1.p1.8.m8.1.1.3">res</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px1.p1.8.m8.1c">f_{\,\texttt{res}}</annotation></semantics></math> can be <math id="S2.SS7.SSS0.Px1.p1.9.m9.1" class="ltx_Math" alttext="f_{\,\texttt{lin}}" display="inline"><semantics id="S2.SS7.SSS0.Px1.p1.9.m9.1a"><msub id="S2.SS7.SSS0.Px1.p1.9.m9.1.1" xref="S2.SS7.SSS0.Px1.p1.9.m9.1.1.cmml"><mi id="S2.SS7.SSS0.Px1.p1.9.m9.1.1.2" xref="S2.SS7.SSS0.Px1.p1.9.m9.1.1.2.cmml">f</mi><mtext class="ltx_mathvariant_monospace" id="S2.SS7.SSS0.Px1.p1.9.m9.1.1.3" xref="S2.SS7.SSS0.Px1.p1.9.m9.1.1.3a.cmml">lin</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px1.p1.9.m9.1b"><apply id="S2.SS7.SSS0.Px1.p1.9.m9.1.1.cmml" xref="S2.SS7.SSS0.Px1.p1.9.m9.1.1"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px1.p1.9.m9.1.1.1.cmml" xref="S2.SS7.SSS0.Px1.p1.9.m9.1.1">subscript</csymbol><ci id="S2.SS7.SSS0.Px1.p1.9.m9.1.1.2.cmml" xref="S2.SS7.SSS0.Px1.p1.9.m9.1.1.2">𝑓</ci><ci id="S2.SS7.SSS0.Px1.p1.9.m9.1.1.3a.cmml" xref="S2.SS7.SSS0.Px1.p1.9.m9.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S2.SS7.SSS0.Px1.p1.9.m9.1.1.3.cmml" xref="S2.SS7.SSS0.Px1.p1.9.m9.1.1.3">lin</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px1.p1.9.m9.1c">f_{\,\texttt{lin}}</annotation></semantics></math> or <math id="S2.SS7.SSS0.Px1.p1.10.m10.1" class="ltx_Math" alttext="f_{\,\texttt{MLP}}" display="inline"><semantics id="S2.SS7.SSS0.Px1.p1.10.m10.1a"><msub id="S2.SS7.SSS0.Px1.p1.10.m10.1.1" xref="S2.SS7.SSS0.Px1.p1.10.m10.1.1.cmml"><mi id="S2.SS7.SSS0.Px1.p1.10.m10.1.1.2" xref="S2.SS7.SSS0.Px1.p1.10.m10.1.1.2.cmml">f</mi><mtext class="ltx_mathvariant_monospace" id="S2.SS7.SSS0.Px1.p1.10.m10.1.1.3" xref="S2.SS7.SSS0.Px1.p1.10.m10.1.1.3a.cmml">MLP</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px1.p1.10.m10.1b"><apply id="S2.SS7.SSS0.Px1.p1.10.m10.1.1.cmml" xref="S2.SS7.SSS0.Px1.p1.10.m10.1.1"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px1.p1.10.m10.1.1.1.cmml" xref="S2.SS7.SSS0.Px1.p1.10.m10.1.1">subscript</csymbol><ci id="S2.SS7.SSS0.Px1.p1.10.m10.1.1.2.cmml" xref="S2.SS7.SSS0.Px1.p1.10.m10.1.1.2">𝑓</ci><ci id="S2.SS7.SSS0.Px1.p1.10.m10.1.1.3a.cmml" xref="S2.SS7.SSS0.Px1.p1.10.m10.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S2.SS7.SSS0.Px1.p1.10.m10.1.1.3.cmml" xref="S2.SS7.SSS0.Px1.p1.10.m10.1.1.3">MLP</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px1.p1.10.m10.1c">f_{\,\texttt{MLP}}</annotation></semantics></math>.</p>
</div>
</section>
<section id="S2.SS7.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Learnable <math id="S2.SS7.SSS0.Px2.1.m1.1" class="ltx_Math" alttext="W_{0}" display="inline"><semantics id="S2.SS7.SSS0.Px2.1.m1.1b"><msub id="S2.SS7.SSS0.Px2.1.m1.1.1" xref="S2.SS7.SSS0.Px2.1.m1.1.1.cmml"><mi id="S2.SS7.SSS0.Px2.1.m1.1.1.2" xref="S2.SS7.SSS0.Px2.1.m1.1.1.2.cmml">W</mi><mn id="S2.SS7.SSS0.Px2.1.m1.1.1.3" xref="S2.SS7.SSS0.Px2.1.m1.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px2.1.m1.1c"><apply id="S2.SS7.SSS0.Px2.1.m1.1.1.cmml" xref="S2.SS7.SSS0.Px2.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px2.1.m1.1.1.1.cmml" xref="S2.SS7.SSS0.Px2.1.m1.1.1">subscript</csymbol><ci id="S2.SS7.SSS0.Px2.1.m1.1.1.2.cmml" xref="S2.SS7.SSS0.Px2.1.m1.1.1.2">𝑊</ci><cn type="integer" id="S2.SS7.SSS0.Px2.1.m1.1.1.3.cmml" xref="S2.SS7.SSS0.Px2.1.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px2.1.m1.1d">W_{0}</annotation></semantics></math>.</h5>

<div id="S2.SS7.SSS0.Px2.p1" class="ltx_para">
<p id="S2.SS7.SSS0.Px2.p1.9" class="ltx_p">The TTT initialization <math id="S2.SS7.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="W_{0}" display="inline"><semantics id="S2.SS7.SSS0.Px2.p1.1.m1.1a"><msub id="S2.SS7.SSS0.Px2.p1.1.m1.1.1" xref="S2.SS7.SSS0.Px2.p1.1.m1.1.1.cmml"><mi id="S2.SS7.SSS0.Px2.p1.1.m1.1.1.2" xref="S2.SS7.SSS0.Px2.p1.1.m1.1.1.2.cmml">W</mi><mn id="S2.SS7.SSS0.Px2.p1.1.m1.1.1.3" xref="S2.SS7.SSS0.Px2.p1.1.m1.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px2.p1.1.m1.1b"><apply id="S2.SS7.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S2.SS7.SSS0.Px2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px2.p1.1.m1.1.1.1.cmml" xref="S2.SS7.SSS0.Px2.p1.1.m1.1.1">subscript</csymbol><ci id="S2.SS7.SSS0.Px2.p1.1.m1.1.1.2.cmml" xref="S2.SS7.SSS0.Px2.p1.1.m1.1.1.2">𝑊</ci><cn type="integer" id="S2.SS7.SSS0.Px2.p1.1.m1.1.1.3.cmml" xref="S2.SS7.SSS0.Px2.p1.1.m1.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px2.p1.1.m1.1c">W_{0}</annotation></semantics></math> is shared between all sequences, even though subsequent weights <math id="S2.SS7.SSS0.Px2.p1.2.m2.3" class="ltx_Math" alttext="W_{1},\dots,W_{T}" display="inline"><semantics id="S2.SS7.SSS0.Px2.p1.2.m2.3a"><mrow id="S2.SS7.SSS0.Px2.p1.2.m2.3.3.2" xref="S2.SS7.SSS0.Px2.p1.2.m2.3.3.3.cmml"><msub id="S2.SS7.SSS0.Px2.p1.2.m2.2.2.1.1" xref="S2.SS7.SSS0.Px2.p1.2.m2.2.2.1.1.cmml"><mi id="S2.SS7.SSS0.Px2.p1.2.m2.2.2.1.1.2" xref="S2.SS7.SSS0.Px2.p1.2.m2.2.2.1.1.2.cmml">W</mi><mn id="S2.SS7.SSS0.Px2.p1.2.m2.2.2.1.1.3" xref="S2.SS7.SSS0.Px2.p1.2.m2.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.SS7.SSS0.Px2.p1.2.m2.3.3.2.3" xref="S2.SS7.SSS0.Px2.p1.2.m2.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS7.SSS0.Px2.p1.2.m2.1.1" xref="S2.SS7.SSS0.Px2.p1.2.m2.1.1.cmml">…</mi><mo id="S2.SS7.SSS0.Px2.p1.2.m2.3.3.2.4" xref="S2.SS7.SSS0.Px2.p1.2.m2.3.3.3.cmml">,</mo><msub id="S2.SS7.SSS0.Px2.p1.2.m2.3.3.2.2" xref="S2.SS7.SSS0.Px2.p1.2.m2.3.3.2.2.cmml"><mi id="S2.SS7.SSS0.Px2.p1.2.m2.3.3.2.2.2" xref="S2.SS7.SSS0.Px2.p1.2.m2.3.3.2.2.2.cmml">W</mi><mi id="S2.SS7.SSS0.Px2.p1.2.m2.3.3.2.2.3" xref="S2.SS7.SSS0.Px2.p1.2.m2.3.3.2.2.3.cmml">T</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px2.p1.2.m2.3b"><list id="S2.SS7.SSS0.Px2.p1.2.m2.3.3.3.cmml" xref="S2.SS7.SSS0.Px2.p1.2.m2.3.3.2"><apply id="S2.SS7.SSS0.Px2.p1.2.m2.2.2.1.1.cmml" xref="S2.SS7.SSS0.Px2.p1.2.m2.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px2.p1.2.m2.2.2.1.1.1.cmml" xref="S2.SS7.SSS0.Px2.p1.2.m2.2.2.1.1">subscript</csymbol><ci id="S2.SS7.SSS0.Px2.p1.2.m2.2.2.1.1.2.cmml" xref="S2.SS7.SSS0.Px2.p1.2.m2.2.2.1.1.2">𝑊</ci><cn type="integer" id="S2.SS7.SSS0.Px2.p1.2.m2.2.2.1.1.3.cmml" xref="S2.SS7.SSS0.Px2.p1.2.m2.2.2.1.1.3">1</cn></apply><ci id="S2.SS7.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S2.SS7.SSS0.Px2.p1.2.m2.1.1">…</ci><apply id="S2.SS7.SSS0.Px2.p1.2.m2.3.3.2.2.cmml" xref="S2.SS7.SSS0.Px2.p1.2.m2.3.3.2.2"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px2.p1.2.m2.3.3.2.2.1.cmml" xref="S2.SS7.SSS0.Px2.p1.2.m2.3.3.2.2">subscript</csymbol><ci id="S2.SS7.SSS0.Px2.p1.2.m2.3.3.2.2.2.cmml" xref="S2.SS7.SSS0.Px2.p1.2.m2.3.3.2.2.2">𝑊</ci><ci id="S2.SS7.SSS0.Px2.p1.2.m2.3.3.2.2.3.cmml" xref="S2.SS7.SSS0.Px2.p1.2.m2.3.3.2.2.3">𝑇</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px2.p1.2.m2.3c">W_{1},\dots,W_{T}</annotation></semantics></math> are different for each input sequence.
Instead of setting <math id="S2.SS7.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="W_{0}=0" display="inline"><semantics id="S2.SS7.SSS0.Px2.p1.3.m3.1a"><mrow id="S2.SS7.SSS0.Px2.p1.3.m3.1.1" xref="S2.SS7.SSS0.Px2.p1.3.m3.1.1.cmml"><msub id="S2.SS7.SSS0.Px2.p1.3.m3.1.1.2" xref="S2.SS7.SSS0.Px2.p1.3.m3.1.1.2.cmml"><mi id="S2.SS7.SSS0.Px2.p1.3.m3.1.1.2.2" xref="S2.SS7.SSS0.Px2.p1.3.m3.1.1.2.2.cmml">W</mi><mn id="S2.SS7.SSS0.Px2.p1.3.m3.1.1.2.3" xref="S2.SS7.SSS0.Px2.p1.3.m3.1.1.2.3.cmml">0</mn></msub><mo id="S2.SS7.SSS0.Px2.p1.3.m3.1.1.1" xref="S2.SS7.SSS0.Px2.p1.3.m3.1.1.1.cmml">=</mo><mn id="S2.SS7.SSS0.Px2.p1.3.m3.1.1.3" xref="S2.SS7.SSS0.Px2.p1.3.m3.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px2.p1.3.m3.1b"><apply id="S2.SS7.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S2.SS7.SSS0.Px2.p1.3.m3.1.1"><eq id="S2.SS7.SSS0.Px2.p1.3.m3.1.1.1.cmml" xref="S2.SS7.SSS0.Px2.p1.3.m3.1.1.1"></eq><apply id="S2.SS7.SSS0.Px2.p1.3.m3.1.1.2.cmml" xref="S2.SS7.SSS0.Px2.p1.3.m3.1.1.2"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px2.p1.3.m3.1.1.2.1.cmml" xref="S2.SS7.SSS0.Px2.p1.3.m3.1.1.2">subscript</csymbol><ci id="S2.SS7.SSS0.Px2.p1.3.m3.1.1.2.2.cmml" xref="S2.SS7.SSS0.Px2.p1.3.m3.1.1.2.2">𝑊</ci><cn type="integer" id="S2.SS7.SSS0.Px2.p1.3.m3.1.1.2.3.cmml" xref="S2.SS7.SSS0.Px2.p1.3.m3.1.1.2.3">0</cn></apply><cn type="integer" id="S2.SS7.SSS0.Px2.p1.3.m3.1.1.3.cmml" xref="S2.SS7.SSS0.Px2.p1.3.m3.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px2.p1.3.m3.1c">W_{0}=0</annotation></semantics></math>, we can learn it as part of the outer loop.
Since outer-loop parameters are always denoted by <math id="S2.SS7.SSS0.Px2.p1.4.m4.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="S2.SS7.SSS0.Px2.p1.4.m4.1a"><mi id="S2.SS7.SSS0.Px2.p1.4.m4.1.1" xref="S2.SS7.SSS0.Px2.p1.4.m4.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px2.p1.4.m4.1b"><ci id="S2.SS7.SSS0.Px2.p1.4.m4.1.1.cmml" xref="S2.SS7.SSS0.Px2.p1.4.m4.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px2.p1.4.m4.1c">\theta</annotation></semantics></math>s instead of <math id="S2.SS7.SSS0.Px2.p1.5.m5.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S2.SS7.SSS0.Px2.p1.5.m5.1a"><mi id="S2.SS7.SSS0.Px2.p1.5.m5.1.1" xref="S2.SS7.SSS0.Px2.p1.5.m5.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px2.p1.5.m5.1b"><ci id="S2.SS7.SSS0.Px2.p1.5.m5.1.1.cmml" xref="S2.SS7.SSS0.Px2.p1.5.m5.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px2.p1.5.m5.1c">W</annotation></semantics></math>s, we assign an alias <math id="S2.SS7.SSS0.Px2.p1.6.m6.1" class="ltx_Math" alttext="\theta_{\text{init}}=W_{0}" display="inline"><semantics id="S2.SS7.SSS0.Px2.p1.6.m6.1a"><mrow id="S2.SS7.SSS0.Px2.p1.6.m6.1.1" xref="S2.SS7.SSS0.Px2.p1.6.m6.1.1.cmml"><msub id="S2.SS7.SSS0.Px2.p1.6.m6.1.1.2" xref="S2.SS7.SSS0.Px2.p1.6.m6.1.1.2.cmml"><mi id="S2.SS7.SSS0.Px2.p1.6.m6.1.1.2.2" xref="S2.SS7.SSS0.Px2.p1.6.m6.1.1.2.2.cmml">θ</mi><mtext id="S2.SS7.SSS0.Px2.p1.6.m6.1.1.2.3" xref="S2.SS7.SSS0.Px2.p1.6.m6.1.1.2.3a.cmml">init</mtext></msub><mo id="S2.SS7.SSS0.Px2.p1.6.m6.1.1.1" xref="S2.SS7.SSS0.Px2.p1.6.m6.1.1.1.cmml">=</mo><msub id="S2.SS7.SSS0.Px2.p1.6.m6.1.1.3" xref="S2.SS7.SSS0.Px2.p1.6.m6.1.1.3.cmml"><mi id="S2.SS7.SSS0.Px2.p1.6.m6.1.1.3.2" xref="S2.SS7.SSS0.Px2.p1.6.m6.1.1.3.2.cmml">W</mi><mn id="S2.SS7.SSS0.Px2.p1.6.m6.1.1.3.3" xref="S2.SS7.SSS0.Px2.p1.6.m6.1.1.3.3.cmml">0</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px2.p1.6.m6.1b"><apply id="S2.SS7.SSS0.Px2.p1.6.m6.1.1.cmml" xref="S2.SS7.SSS0.Px2.p1.6.m6.1.1"><eq id="S2.SS7.SSS0.Px2.p1.6.m6.1.1.1.cmml" xref="S2.SS7.SSS0.Px2.p1.6.m6.1.1.1"></eq><apply id="S2.SS7.SSS0.Px2.p1.6.m6.1.1.2.cmml" xref="S2.SS7.SSS0.Px2.p1.6.m6.1.1.2"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px2.p1.6.m6.1.1.2.1.cmml" xref="S2.SS7.SSS0.Px2.p1.6.m6.1.1.2">subscript</csymbol><ci id="S2.SS7.SSS0.Px2.p1.6.m6.1.1.2.2.cmml" xref="S2.SS7.SSS0.Px2.p1.6.m6.1.1.2.2">𝜃</ci><ci id="S2.SS7.SSS0.Px2.p1.6.m6.1.1.2.3a.cmml" xref="S2.SS7.SSS0.Px2.p1.6.m6.1.1.2.3"><mtext mathsize="70%" id="S2.SS7.SSS0.Px2.p1.6.m6.1.1.2.3.cmml" xref="S2.SS7.SSS0.Px2.p1.6.m6.1.1.2.3">init</mtext></ci></apply><apply id="S2.SS7.SSS0.Px2.p1.6.m6.1.1.3.cmml" xref="S2.SS7.SSS0.Px2.p1.6.m6.1.1.3"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px2.p1.6.m6.1.1.3.1.cmml" xref="S2.SS7.SSS0.Px2.p1.6.m6.1.1.3">subscript</csymbol><ci id="S2.SS7.SSS0.Px2.p1.6.m6.1.1.3.2.cmml" xref="S2.SS7.SSS0.Px2.p1.6.m6.1.1.3.2">𝑊</ci><cn type="integer" id="S2.SS7.SSS0.Px2.p1.6.m6.1.1.3.3.cmml" xref="S2.SS7.SSS0.Px2.p1.6.m6.1.1.3.3">0</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px2.p1.6.m6.1c">\theta_{\text{init}}=W_{0}</annotation></semantics></math>.
In practice,
<math id="S2.SS7.SSS0.Px2.p1.7.m7.1" class="ltx_Math" alttext="\theta_{\text{init}}" display="inline"><semantics id="S2.SS7.SSS0.Px2.p1.7.m7.1a"><msub id="S2.SS7.SSS0.Px2.p1.7.m7.1.1" xref="S2.SS7.SSS0.Px2.p1.7.m7.1.1.cmml"><mi id="S2.SS7.SSS0.Px2.p1.7.m7.1.1.2" xref="S2.SS7.SSS0.Px2.p1.7.m7.1.1.2.cmml">θ</mi><mtext id="S2.SS7.SSS0.Px2.p1.7.m7.1.1.3" xref="S2.SS7.SSS0.Px2.p1.7.m7.1.1.3a.cmml">init</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px2.p1.7.m7.1b"><apply id="S2.SS7.SSS0.Px2.p1.7.m7.1.1.cmml" xref="S2.SS7.SSS0.Px2.p1.7.m7.1.1"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px2.p1.7.m7.1.1.1.cmml" xref="S2.SS7.SSS0.Px2.p1.7.m7.1.1">subscript</csymbol><ci id="S2.SS7.SSS0.Px2.p1.7.m7.1.1.2.cmml" xref="S2.SS7.SSS0.Px2.p1.7.m7.1.1.2">𝜃</ci><ci id="S2.SS7.SSS0.Px2.p1.7.m7.1.1.3a.cmml" xref="S2.SS7.SSS0.Px2.p1.7.m7.1.1.3"><mtext mathsize="70%" id="S2.SS7.SSS0.Px2.p1.7.m7.1.1.3.cmml" xref="S2.SS7.SSS0.Px2.p1.7.m7.1.1.3">init</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px2.p1.7.m7.1c">\theta_{\text{init}}</annotation></semantics></math> adds a negligible amount of parameters comparing to the reconstruction views <math id="S2.SS7.SSS0.Px2.p1.8.m8.3" class="ltx_Math" alttext="\theta_{K},\theta_{Q},\theta_{V}" display="inline"><semantics id="S2.SS7.SSS0.Px2.p1.8.m8.3a"><mrow id="S2.SS7.SSS0.Px2.p1.8.m8.3.3.3" xref="S2.SS7.SSS0.Px2.p1.8.m8.3.3.4.cmml"><msub id="S2.SS7.SSS0.Px2.p1.8.m8.1.1.1.1" xref="S2.SS7.SSS0.Px2.p1.8.m8.1.1.1.1.cmml"><mi id="S2.SS7.SSS0.Px2.p1.8.m8.1.1.1.1.2" xref="S2.SS7.SSS0.Px2.p1.8.m8.1.1.1.1.2.cmml">θ</mi><mi id="S2.SS7.SSS0.Px2.p1.8.m8.1.1.1.1.3" xref="S2.SS7.SSS0.Px2.p1.8.m8.1.1.1.1.3.cmml">K</mi></msub><mo id="S2.SS7.SSS0.Px2.p1.8.m8.3.3.3.4" xref="S2.SS7.SSS0.Px2.p1.8.m8.3.3.4.cmml">,</mo><msub id="S2.SS7.SSS0.Px2.p1.8.m8.2.2.2.2" xref="S2.SS7.SSS0.Px2.p1.8.m8.2.2.2.2.cmml"><mi id="S2.SS7.SSS0.Px2.p1.8.m8.2.2.2.2.2" xref="S2.SS7.SSS0.Px2.p1.8.m8.2.2.2.2.2.cmml">θ</mi><mi id="S2.SS7.SSS0.Px2.p1.8.m8.2.2.2.2.3" xref="S2.SS7.SSS0.Px2.p1.8.m8.2.2.2.2.3.cmml">Q</mi></msub><mo id="S2.SS7.SSS0.Px2.p1.8.m8.3.3.3.5" xref="S2.SS7.SSS0.Px2.p1.8.m8.3.3.4.cmml">,</mo><msub id="S2.SS7.SSS0.Px2.p1.8.m8.3.3.3.3" xref="S2.SS7.SSS0.Px2.p1.8.m8.3.3.3.3.cmml"><mi id="S2.SS7.SSS0.Px2.p1.8.m8.3.3.3.3.2" xref="S2.SS7.SSS0.Px2.p1.8.m8.3.3.3.3.2.cmml">θ</mi><mi id="S2.SS7.SSS0.Px2.p1.8.m8.3.3.3.3.3" xref="S2.SS7.SSS0.Px2.p1.8.m8.3.3.3.3.3.cmml">V</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px2.p1.8.m8.3b"><list id="S2.SS7.SSS0.Px2.p1.8.m8.3.3.4.cmml" xref="S2.SS7.SSS0.Px2.p1.8.m8.3.3.3"><apply id="S2.SS7.SSS0.Px2.p1.8.m8.1.1.1.1.cmml" xref="S2.SS7.SSS0.Px2.p1.8.m8.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px2.p1.8.m8.1.1.1.1.1.cmml" xref="S2.SS7.SSS0.Px2.p1.8.m8.1.1.1.1">subscript</csymbol><ci id="S2.SS7.SSS0.Px2.p1.8.m8.1.1.1.1.2.cmml" xref="S2.SS7.SSS0.Px2.p1.8.m8.1.1.1.1.2">𝜃</ci><ci id="S2.SS7.SSS0.Px2.p1.8.m8.1.1.1.1.3.cmml" xref="S2.SS7.SSS0.Px2.p1.8.m8.1.1.1.1.3">𝐾</ci></apply><apply id="S2.SS7.SSS0.Px2.p1.8.m8.2.2.2.2.cmml" xref="S2.SS7.SSS0.Px2.p1.8.m8.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px2.p1.8.m8.2.2.2.2.1.cmml" xref="S2.SS7.SSS0.Px2.p1.8.m8.2.2.2.2">subscript</csymbol><ci id="S2.SS7.SSS0.Px2.p1.8.m8.2.2.2.2.2.cmml" xref="S2.SS7.SSS0.Px2.p1.8.m8.2.2.2.2.2">𝜃</ci><ci id="S2.SS7.SSS0.Px2.p1.8.m8.2.2.2.2.3.cmml" xref="S2.SS7.SSS0.Px2.p1.8.m8.2.2.2.2.3">𝑄</ci></apply><apply id="S2.SS7.SSS0.Px2.p1.8.m8.3.3.3.3.cmml" xref="S2.SS7.SSS0.Px2.p1.8.m8.3.3.3.3"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px2.p1.8.m8.3.3.3.3.1.cmml" xref="S2.SS7.SSS0.Px2.p1.8.m8.3.3.3.3">subscript</csymbol><ci id="S2.SS7.SSS0.Px2.p1.8.m8.3.3.3.3.2.cmml" xref="S2.SS7.SSS0.Px2.p1.8.m8.3.3.3.3.2">𝜃</ci><ci id="S2.SS7.SSS0.Px2.p1.8.m8.3.3.3.3.3.cmml" xref="S2.SS7.SSS0.Px2.p1.8.m8.3.3.3.3.3">𝑉</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px2.p1.8.m8.3c">\theta_{K},\theta_{Q},\theta_{V}</annotation></semantics></math>, because both its input and output are low dimensional.
Empirically, we observe that learning <math id="S2.SS7.SSS0.Px2.p1.9.m9.1" class="ltx_Math" alttext="W_{0}" display="inline"><semantics id="S2.SS7.SSS0.Px2.p1.9.m9.1a"><msub id="S2.SS7.SSS0.Px2.p1.9.m9.1.1" xref="S2.SS7.SSS0.Px2.p1.9.m9.1.1.cmml"><mi id="S2.SS7.SSS0.Px2.p1.9.m9.1.1.2" xref="S2.SS7.SSS0.Px2.p1.9.m9.1.1.2.cmml">W</mi><mn id="S2.SS7.SSS0.Px2.p1.9.m9.1.1.3" xref="S2.SS7.SSS0.Px2.p1.9.m9.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px2.p1.9.m9.1b"><apply id="S2.SS7.SSS0.Px2.p1.9.m9.1.1.cmml" xref="S2.SS7.SSS0.Px2.p1.9.m9.1.1"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px2.p1.9.m9.1.1.1.cmml" xref="S2.SS7.SSS0.Px2.p1.9.m9.1.1">subscript</csymbol><ci id="S2.SS7.SSS0.Px2.p1.9.m9.1.1.2.cmml" xref="S2.SS7.SSS0.Px2.p1.9.m9.1.1.2">𝑊</ci><cn type="integer" id="S2.SS7.SSS0.Px2.p1.9.m9.1.1.3.cmml" xref="S2.SS7.SSS0.Px2.p1.9.m9.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px2.p1.9.m9.1c">W_{0}</annotation></semantics></math> significantly improves training stability.</p>
</div>
</section>
<section id="S2.SS7.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Learnable <math id="S2.SS7.SSS0.Px3.1.m1.1" class="ltx_Math" alttext="\eta" display="inline"><semantics id="S2.SS7.SSS0.Px3.1.m1.1b"><mi id="S2.SS7.SSS0.Px3.1.m1.1.1" xref="S2.SS7.SSS0.Px3.1.m1.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px3.1.m1.1c"><ci id="S2.SS7.SSS0.Px3.1.m1.1.1.cmml" xref="S2.SS7.SSS0.Px3.1.m1.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px3.1.m1.1d">\eta</annotation></semantics></math>.</h5>

<div id="S2.SS7.SSS0.Px3.p1" class="ltx_para">
<p id="S2.SS7.SSS0.Px3.p1.8" class="ltx_p">The learning rate is usually the most important hyper-parameter for gradient descent, so we experiment with learning the inner-loop learning rate <math id="S2.SS7.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="\eta" display="inline"><semantics id="S2.SS7.SSS0.Px3.p1.1.m1.1a"><mi id="S2.SS7.SSS0.Px3.p1.1.m1.1.1" xref="S2.SS7.SSS0.Px3.p1.1.m1.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px3.p1.1.m1.1b"><ci id="S2.SS7.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S2.SS7.SSS0.Px3.p1.1.m1.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px3.p1.1.m1.1c">\eta</annotation></semantics></math> in Equation <a href="#S2.E6" title="In 2.4 Parallelization with mini-batch TTT ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> as part of the outer loop.
We make <math id="S2.SS7.SSS0.Px3.p1.2.m2.1" class="ltx_Math" alttext="\eta" display="inline"><semantics id="S2.SS7.SSS0.Px3.p1.2.m2.1a"><mi id="S2.SS7.SSS0.Px3.p1.2.m2.1.1" xref="S2.SS7.SSS0.Px3.p1.2.m2.1.1.cmml">η</mi><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px3.p1.2.m2.1b"><ci id="S2.SS7.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S2.SS7.SSS0.Px3.p1.2.m2.1.1">𝜂</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px3.p1.2.m2.1c">\eta</annotation></semantics></math> a function of the input token (therefore different across time) for additional flexibility.
Concretely, we design <math id="S2.SS7.SSS0.Px3.p1.3.m3.2" class="ltx_Math" alttext="\eta(x)=\eta_{\text{base}}\,\sigma(\theta_{\text{lr}}\cdot x)" display="inline"><semantics id="S2.SS7.SSS0.Px3.p1.3.m3.2a"><mrow id="S2.SS7.SSS0.Px3.p1.3.m3.2.2" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.cmml"><mrow id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.3" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.3.cmml"><mi id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.3.2" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.3.2.cmml">η</mi><mo lspace="0em" rspace="0em" id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.3.1" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.3.1.cmml">​</mo><mrow id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.3.3.2" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.3.cmml"><mo stretchy="false" id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.3.3.2.1" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.3.cmml">(</mo><mi id="S2.SS7.SSS0.Px3.p1.3.m3.1.1" xref="S2.SS7.SSS0.Px3.p1.3.m3.1.1.cmml">x</mi><mo stretchy="false" id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.3.3.2.2" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.3.cmml">)</mo></mrow></mrow><mo id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.2" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.2.cmml">=</mo><mrow id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.cmml"><msub id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.3" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.3.cmml"><mi id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.3.2" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.3.2.cmml">η</mi><mtext id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.3.3" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.3.3a.cmml">base</mtext></msub><mo lspace="0em" rspace="0em" id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.2" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.2.cmml">​</mo><mi id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.4" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.4.cmml">σ</mi><mo lspace="0em" rspace="0em" id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.2a" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.2.cmml">​</mo><mrow id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.2" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.cmml">(</mo><mrow id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.cmml"><msub id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.2" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.2.cmml"><mi id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.2.2" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.2.2.cmml">θ</mi><mtext id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.2.3" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.2.3a.cmml">lr</mtext></msub><mo lspace="0.222em" rspace="0.222em" id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.1" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.1.cmml">⋅</mo><mi id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.3" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.3.cmml">x</mi></mrow><mo stretchy="false" id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.3" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px3.p1.3.m3.2b"><apply id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2"><eq id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.2.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.2"></eq><apply id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.3.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.3"><times id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.3.1.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.3.1"></times><ci id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.3.2.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.3.2">𝜂</ci><ci id="S2.SS7.SSS0.Px3.p1.3.m3.1.1.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.1.1">𝑥</ci></apply><apply id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1"><times id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.2.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.2"></times><apply id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.3.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.3"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.3.1.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.3">subscript</csymbol><ci id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.3.2.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.3.2">𝜂</ci><ci id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.3.3a.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.3.3"><mtext mathsize="70%" id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.3.3.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.3.3">base</mtext></ci></apply><ci id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.4.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.4">𝜎</ci><apply id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1"><ci id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.1.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.1">⋅</ci><apply id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.2.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.2.1.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.2">subscript</csymbol><ci id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.2.2.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.2.2">𝜃</ci><ci id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.2.3a.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.2.3"><mtext mathsize="70%" id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.2.3.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.2.3">lr</mtext></ci></apply><ci id="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.3.cmml" xref="S2.SS7.SSS0.Px3.p1.3.m3.2.2.1.1.1.1.3">𝑥</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px3.p1.3.m3.2c">\eta(x)=\eta_{\text{base}}\,\sigma(\theta_{\text{lr}}\cdot x)</annotation></semantics></math>, where the learnable vector <math id="S2.SS7.SSS0.Px3.p1.4.m4.1" class="ltx_Math" alttext="\theta_{\text{lr}}" display="inline"><semantics id="S2.SS7.SSS0.Px3.p1.4.m4.1a"><msub id="S2.SS7.SSS0.Px3.p1.4.m4.1.1" xref="S2.SS7.SSS0.Px3.p1.4.m4.1.1.cmml"><mi id="S2.SS7.SSS0.Px3.p1.4.m4.1.1.2" xref="S2.SS7.SSS0.Px3.p1.4.m4.1.1.2.cmml">θ</mi><mtext id="S2.SS7.SSS0.Px3.p1.4.m4.1.1.3" xref="S2.SS7.SSS0.Px3.p1.4.m4.1.1.3a.cmml">lr</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px3.p1.4.m4.1b"><apply id="S2.SS7.SSS0.Px3.p1.4.m4.1.1.cmml" xref="S2.SS7.SSS0.Px3.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px3.p1.4.m4.1.1.1.cmml" xref="S2.SS7.SSS0.Px3.p1.4.m4.1.1">subscript</csymbol><ci id="S2.SS7.SSS0.Px3.p1.4.m4.1.1.2.cmml" xref="S2.SS7.SSS0.Px3.p1.4.m4.1.1.2">𝜃</ci><ci id="S2.SS7.SSS0.Px3.p1.4.m4.1.1.3a.cmml" xref="S2.SS7.SSS0.Px3.p1.4.m4.1.1.3"><mtext mathsize="70%" id="S2.SS7.SSS0.Px3.p1.4.m4.1.1.3.cmml" xref="S2.SS7.SSS0.Px3.p1.4.m4.1.1.3">lr</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px3.p1.4.m4.1c">\theta_{\text{lr}}</annotation></semantics></math> is an outer-loop parameter, <math id="S2.SS7.SSS0.Px3.p1.5.m5.1" class="ltx_Math" alttext="\sigma" display="inline"><semantics id="S2.SS7.SSS0.Px3.p1.5.m5.1a"><mi id="S2.SS7.SSS0.Px3.p1.5.m5.1.1" xref="S2.SS7.SSS0.Px3.p1.5.m5.1.1.cmml">σ</mi><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px3.p1.5.m5.1b"><ci id="S2.SS7.SSS0.Px3.p1.5.m5.1.1.cmml" xref="S2.SS7.SSS0.Px3.p1.5.m5.1.1">𝜎</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px3.p1.5.m5.1c">\sigma</annotation></semantics></math> is the sigmoid function, and the scalar <math id="S2.SS7.SSS0.Px3.p1.6.m6.1" class="ltx_Math" alttext="\eta_{\text{base}}" display="inline"><semantics id="S2.SS7.SSS0.Px3.p1.6.m6.1a"><msub id="S2.SS7.SSS0.Px3.p1.6.m6.1.1" xref="S2.SS7.SSS0.Px3.p1.6.m6.1.1.cmml"><mi id="S2.SS7.SSS0.Px3.p1.6.m6.1.1.2" xref="S2.SS7.SSS0.Px3.p1.6.m6.1.1.2.cmml">η</mi><mtext id="S2.SS7.SSS0.Px3.p1.6.m6.1.1.3" xref="S2.SS7.SSS0.Px3.p1.6.m6.1.1.3a.cmml">base</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px3.p1.6.m6.1b"><apply id="S2.SS7.SSS0.Px3.p1.6.m6.1.1.cmml" xref="S2.SS7.SSS0.Px3.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S2.SS7.SSS0.Px3.p1.6.m6.1.1.1.cmml" xref="S2.SS7.SSS0.Px3.p1.6.m6.1.1">subscript</csymbol><ci id="S2.SS7.SSS0.Px3.p1.6.m6.1.1.2.cmml" xref="S2.SS7.SSS0.Px3.p1.6.m6.1.1.2">𝜂</ci><ci id="S2.SS7.SSS0.Px3.p1.6.m6.1.1.3a.cmml" xref="S2.SS7.SSS0.Px3.p1.6.m6.1.1.3"><mtext mathsize="70%" id="S2.SS7.SSS0.Px3.p1.6.m6.1.1.3.cmml" xref="S2.SS7.SSS0.Px3.p1.6.m6.1.1.3">base</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px3.p1.6.m6.1c">\eta_{\text{base}}</annotation></semantics></math> is the base learning rate, set to 1 for TTT-Linear and 0.1 for TTT-MLP.
Alternatively, <math id="S2.SS7.SSS0.Px3.p1.7.m7.1" class="ltx_Math" alttext="\eta(x)" display="inline"><semantics id="S2.SS7.SSS0.Px3.p1.7.m7.1a"><mrow id="S2.SS7.SSS0.Px3.p1.7.m7.1.2" xref="S2.SS7.SSS0.Px3.p1.7.m7.1.2.cmml"><mi id="S2.SS7.SSS0.Px3.p1.7.m7.1.2.2" xref="S2.SS7.SSS0.Px3.p1.7.m7.1.2.2.cmml">η</mi><mo lspace="0em" rspace="0em" id="S2.SS7.SSS0.Px3.p1.7.m7.1.2.1" xref="S2.SS7.SSS0.Px3.p1.7.m7.1.2.1.cmml">​</mo><mrow id="S2.SS7.SSS0.Px3.p1.7.m7.1.2.3.2" xref="S2.SS7.SSS0.Px3.p1.7.m7.1.2.cmml"><mo stretchy="false" id="S2.SS7.SSS0.Px3.p1.7.m7.1.2.3.2.1" xref="S2.SS7.SSS0.Px3.p1.7.m7.1.2.cmml">(</mo><mi id="S2.SS7.SSS0.Px3.p1.7.m7.1.1" xref="S2.SS7.SSS0.Px3.p1.7.m7.1.1.cmml">x</mi><mo stretchy="false" id="S2.SS7.SSS0.Px3.p1.7.m7.1.2.3.2.2" xref="S2.SS7.SSS0.Px3.p1.7.m7.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px3.p1.7.m7.1b"><apply id="S2.SS7.SSS0.Px3.p1.7.m7.1.2.cmml" xref="S2.SS7.SSS0.Px3.p1.7.m7.1.2"><times id="S2.SS7.SSS0.Px3.p1.7.m7.1.2.1.cmml" xref="S2.SS7.SSS0.Px3.p1.7.m7.1.2.1"></times><ci id="S2.SS7.SSS0.Px3.p1.7.m7.1.2.2.cmml" xref="S2.SS7.SSS0.Px3.p1.7.m7.1.2.2">𝜂</ci><ci id="S2.SS7.SSS0.Px3.p1.7.m7.1.1.cmml" xref="S2.SS7.SSS0.Px3.p1.7.m7.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px3.p1.7.m7.1c">\eta(x)</annotation></semantics></math> can also be interpreted as a gate for <math id="S2.SS7.SSS0.Px3.p1.8.m8.1" class="ltx_Math" alttext="\nabla\ell" display="inline"><semantics id="S2.SS7.SSS0.Px3.p1.8.m8.1a"><mrow id="S2.SS7.SSS0.Px3.p1.8.m8.1.1" xref="S2.SS7.SSS0.Px3.p1.8.m8.1.1.cmml"><mo rspace="0.167em" id="S2.SS7.SSS0.Px3.p1.8.m8.1.1.1" xref="S2.SS7.SSS0.Px3.p1.8.m8.1.1.1.cmml">∇</mo><mi mathvariant="normal" id="S2.SS7.SSS0.Px3.p1.8.m8.1.1.2" xref="S2.SS7.SSS0.Px3.p1.8.m8.1.1.2.cmml">ℓ</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS7.SSS0.Px3.p1.8.m8.1b"><apply id="S2.SS7.SSS0.Px3.p1.8.m8.1.1.cmml" xref="S2.SS7.SSS0.Px3.p1.8.m8.1.1"><ci id="S2.SS7.SSS0.Px3.p1.8.m8.1.1.1.cmml" xref="S2.SS7.SSS0.Px3.p1.8.m8.1.1.1">∇</ci><ci id="S2.SS7.SSS0.Px3.p1.8.m8.1.1.2.cmml" xref="S2.SS7.SSS0.Px3.p1.8.m8.1.1.2">ℓ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS7.SSS0.Px3.p1.8.m8.1c">\nabla\ell</annotation></semantics></math>.</p>
</div>
</section>
<section id="S2.SS7.SSS0.Px4" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Backbone architecture.</h5>

<div id="S2.SS7.SSS0.Px4.p1" class="ltx_para">
<p id="S2.SS7.SSS0.Px4.p1.1" class="ltx_p">The cleanest way to integrate any RNN layer into a larger architecture would be to directly replace self-attention in a Transformer, known in this context as a backbone.
However, existing RNNs such as Mamba <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib26" title="" class="ltx_ref">26</a>]</cite> and Griffin <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib18" title="" class="ltx_ref">18</a>]</cite> all use a different backbone from Transformers.
Most notably, their backbone contains temporal convolutions before the RNN layers, which might help collect local information across time.
After experimenting with the Mamba backbone, we find that it also improves perplexity for TTT layers, so we incorporate it into our proposed method.
See Figure <a href="#A2.F16" title="Figure 16 ‣ Asymmetric kernels. ‣ Appendix B Nadaraya-Watson estimator ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">16</span></a> (in Appendix) for details.</p>
</div>
</section>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Experiments</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">We evaluate TTT-Linear and TTT-MLP by comparing with two baselines
– Transformer and Mamba, a modern RNN.
Our main codebase is based on EasyLM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib25" title="" class="ltx_ref">25</a>]</cite>, an open-source project for training and serving LLMs in JAX.
All experiments can be reproduced using the publicly available code and datasets provided at the bottom of the first page.</p>
</div>
<section id="S3.SS0.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Datasets.</h5>

<div id="S3.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS0.SSS0.Px1.p1.1" class="ltx_p">Following the Mamba paper <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib26" title="" class="ltx_ref">26</a>]</cite>, we perform standard experiments with 2k and 8k context lengths on the Pile <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib24" title="" class="ltx_ref">24</a>]</cite>, a popular dataset of documents for training open-source LLMs <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib9" title="" class="ltx_ref">9</a>]</cite>.
However, the Pile contains few sequences of length greater than 8k <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib19" title="" class="ltx_ref">19</a>]</cite>.
To evaluate capabilities in long context, we also experiment with context lengths ranging from 1k to 32k in <math id="S3.SS0.SSS0.Px1.p1.1.m1.1" class="ltx_math_unparsed" alttext="2\times" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.1.m1.1a"><mrow id="S3.SS0.SSS0.Px1.p1.1.m1.1b"><mn id="S3.SS0.SSS0.Px1.p1.1.m1.1.1">2</mn><mo lspace="0.222em" id="S3.SS0.SSS0.Px1.p1.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.1.m1.1c">2\times</annotation></semantics></math> increments, using a subset of the Pile called Books3, which has been widely used to train LLMs in long context <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib49" title="" class="ltx_ref">49</a>, <a href="#bib.bib3" title="" class="ltx_ref">3</a>]</cite>.</p>
</div>
</section>
<section id="S3.SS0.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Backbone architecture.</h5>

<div id="S3.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S3.SS0.SSS0.Px2.p1.1" class="ltx_p">As discussed in Subsection <a href="#S2.SS7" title="2.7 Implementation details ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.7</span></a>, Transformer and Mamba use different backbones, and TTT-Linear and TTT-MLP always use the Mamba backbone unless noted otherwise.
As an ablation study, Figure <a href="#S3.F11" title="Figure 11 ‣ 3.1 Short context: the Pile ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a> and Figure <a href="#S3.F12" title="Figure 12 ‣ 3.2 Long context: Books ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12</span></a> contain TTT layers within the Transformer backbone.
When a figure contains both the Transformer backbone and Mamba backbone, we denote them by <em id="S3.SS0.SSS0.Px2.p1.1.1" class="ltx_emph ltx_font_italic">(T)</em> and <em id="S3.SS0.SSS0.Px2.p1.1.2" class="ltx_emph ltx_font_italic">(M)</em>, respectively.</p>
</div>
</section>
<section id="S3.SS0.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Protocols.</h5>

<div id="S3.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="S3.SS0.SSS0.Px3.p1.1" class="ltx_p">To ensure fairness to our baselines, we strictly follow the evaluation protocols in the Mamba paper when possible:</p>
<ul id="S3.I1" class="ltx_itemize">
<li id="S3.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i1.p1" class="ltx_para">
<p id="S3.I1.i1.p1.1" class="ltx_p">For each evaluation setting (e.g., dataset, context length, and method), we experiment with four model sizes: 125M, 350M, 760M, and 1.3B parameters. For Mamba, the corresponding sizes are 130M, 370M, 790M, and 1.4B, as Mamba does not follow the Transformer configurations.</p>
</div>
</li>
<li id="S3.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i2.p1" class="ltx_para">
<p id="S3.I1.i2.p1.1" class="ltx_p">All models are trained with the Chinchilla recipe<span id="footnote9" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">9</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">9</sup><span class="ltx_tag ltx_tag_note">9</span>
The Chinchilla paper is another highly influential study of empirical scaling laws <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib34" title="" class="ltx_ref">34</a>]</cite>.
From large-scale experiments with many hyper-parameters, they observe that the compute-optimal models follow a particular training recipe.
We only follow the Chinchilla recipe used in the Mamba paper, which may be slightly different from the original recipe in <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib34" title="" class="ltx_ref">34</a>]</cite>.
</span></span></span> described in the Mamba paper and reproduced in our Appendix <a href="#A3" title="Appendix C Experiment details ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>.
Our Transformer baseline, based on the Llama architecture <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib73" title="" class="ltx_ref">73</a>]</cite>, also follows the baseline in the Mamba paper.
As verification, our baselines can reproduce the numbers reported in the Mamba paper in their evaluation settings.<span id="footnote10" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">10</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">10</sup><span class="ltx_tag ltx_tag_note">10</span>
The only difference between our protocol and that in the Mamba paper is the tokenizer.
The Mamba paper uses two different tokenizers – GPT-2 and GPT-NeoX – for various experiments.
For consistency, we adhere to a single tokenizer throughout this paper and choose the Llama tokenizer <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib73" title="" class="ltx_ref">73</a>]</cite>, which is the modern state-of-the-art.
</span></span></span></p>
</div>
</li>
<li id="S3.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i3.p1" class="ltx_para">
<p id="S3.I1.i3.p1.1" class="ltx_p">We do not experiment with hybrid architectures (e.g. Griffin <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib18" title="" class="ltx_ref">18</a>]</cite>), because our baselines are not hybrid.
While hybrid architectures that use both self-attention and TTT layers may improve performance, they would reduce the clarity of our academic evaluation.</p>
</div>
</li>
</ul>
</div>
</section>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Short context: the Pile</h3>

<figure id="S3.F11" class="ltx_figure"><img src="/html/2407.04620/assets/x11.png" id="S3.F11.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="184" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 11: </span>Evaluations for context lengths 2k and 8k on the Pile.
Details in Subsection <a href="#S3.SS1" title="3.1 Short context: the Pile ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a>.
TTT-Linear has comparable performance as Mamba at 2k context, and better performance at 8k.
</figcaption>
</figure>
<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">From Figure <a href="#S3.F11" title="Figure 11 ‣ 3.1 Short context: the Pile ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a>, we make a few observations:</p>
<ul id="S3.I2" class="ltx_itemize">
<li id="S3.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I2.i1.p1" class="ltx_para">
<p id="S3.I2.i1.p1.1" class="ltx_p">At 2k context, TTT-Linear (M), Mamba, and Transformer have comparable performance, as the lines mostly overlap.
TTT-MLP (M) performs slightly worse under large FLOP budgets.
Even though TTT-MLP has better perplexity than TTT-Linear at every model size, the extra cost in FLOPs offsets the advantage.</p>
</div>
</li>
<li id="S3.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I2.i2.p1" class="ltx_para">
<p id="S3.I2.i2.p1.1" class="ltx_p">At 8k context, both TTT-Linear (M) and TTT-MLP (M) perform significantly better than Mamba, in contrast to the observation at 2k. Even TTT-MLP (T) with the Transformer backbone performs slightly better than Mamba around 1.3B.
A robust phenomenon we observe throughout this paper is that as context length grows longer, the advantage of TTT layers over Mamba widens.</p>
</div>
</li>
<li id="S3.I2.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I2.i3.p1" class="ltx_para">
<p id="S3.I2.i3.p1.1" class="ltx_p">At 8k context, Transformer still has good (if not the best) perplexity at every model size, but its line is not competitive because of the cost in FLOPs.</p>
</div>
</li>
</ul>
</div>
<section id="S3.SS1.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Effect of backbone.</h5>

<div id="S3.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS1.SSS0.Px1.p1.1" class="ltx_p">Switching the TTT layers from Mamba backbone into Transformer backbone has two effects.
First, TTT layers with Mamba backbone perform better in our evaluations so far.
Second, with Mamba backbone, TTT-MLP at best is only comparable to TTT-Linear; but with Transformer backbone, TTT-MLP is clearly better.
We hypothesize that the temporal convolutions in the Mamba backbone help more when the sequence modeling layer has a less expressive hidden state.
The linear model is less expressive than the MLP, therefore benefits more from the convolutions.
<span id="S3.SS1.SSS0.Px1.p1.1.1" class="ltx_text">We will revisit this hypothesis in the next subsection.</span></p>
</div>
</section>
<section id="S3.SS1.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Lack of linear fit.</h5>

<div id="S3.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="S3.SS1.SSS0.Px2.p1.1" class="ltx_p">The Chinchilla paper empirically observed that the compute-optimal models following their recipe fall onto a line in the log-log plot of FLOPs vs. perplexity, as is often the case for scaling law experiments <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib34" title="" class="ltx_ref">34</a>]</cite>.
However, we do not observe a clean linear fit in Figure <a href="#S3.F11" title="Figure 11 ‣ 3.1 Short context: the Pile ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a> or Figure <a href="#S3.F12" title="Figure 12 ‣ 3.2 Long context: Books ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12</span></a> (the analogous experiments in Books), not even for Transformers.
This is not surprising given the differences in dataset, context length, tokenizer, and architecture.
Following the Mamba paper, we connect the points instead of fitting them with linear regression due to the large error.<span id="footnote11" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">11</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">11</sup><span class="ltx_tag ltx_tag_note">11</span>
Ideally, we would have rerun all the hyper-parameters and derived a potentially new recipe for each method based on our evaluation setting, following the process in the Chinchilla paper.
If the new compute-optimal models do fall onto a line, we could then predict performance beyond the current FLOPs regime <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib40" title="" class="ltx_ref">40</a>, <a href="#bib.bib34" title="" class="ltx_ref">34</a>]</cite>. However, this empirical study would require orders of magnitude more resources than ours.
</span></span></span></p>
</div>
</section>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Long context: Books</h3>

<figure id="S3.F12" class="ltx_figure"><img src="/html/2407.04620/assets/x12.png" id="S3.F12.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="184" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 12: </span>Evaluations for context lengths 2k and 32k on Books.
Details in Subsection <a href="#S3.SS2" title="3.2 Long context: Books ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>.
Our complete results for context lengths 1k, 2k, 4k, 8k, 16k, 32k, including Transformer finetuning, are in Figure <a href="#A3.F18" title="Figure 18 ‣ Gradient checkpointing through time. ‣ Appendix C Experiment details ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">18</span></a> (in Appendix).
Most observations from the Pile still hold.</figcaption>
</figure>
<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">To evaluate capabilities in long context, we experiment with context lengths ranging from 1k to 32k in <math id="S3.SS2.p1.1.m1.1" class="ltx_math_unparsed" alttext="2\times" display="inline"><semantics id="S3.SS2.p1.1.m1.1a"><mrow id="S3.SS2.p1.1.m1.1b"><mn id="S3.SS2.p1.1.m1.1.1">2</mn><mo lspace="0.222em" id="S3.SS2.p1.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S3.SS2.p1.1.m1.1c">2\times</annotation></semantics></math> increments, using a popular subset of the Pile called Books3.
The training recipe here is the same as for the Pile, and all experiments for the TTT layers are <span id="S3.SS2.p1.1.1" class="ltx_text">performed in one training run.</span><span id="footnote12" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">12</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">12</sup><span class="ltx_tag ltx_tag_note">12</span>
Following the Mamba paper, we always use 0.5M tokens per training batch regardless of context length. That means for context length <math id="footnote12.m1.1" class="ltx_Math" alttext="T" display="inline"><semantics id="footnote12.m1.1b"><mi id="footnote12.m1.1.1" xref="footnote12.m1.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="footnote12.m1.1c"><ci id="footnote12.m1.1.1.cmml" xref="footnote12.m1.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote12.m1.1d">T</annotation></semantics></math> we have <math id="footnote12.m2.1" class="ltx_Math" alttext="0.5\text{M}\,/\,T" display="inline"><semantics id="footnote12.m2.1b"><mrow id="footnote12.m2.1.1" xref="footnote12.m2.1.1.cmml"><mrow id="footnote12.m2.1.1.2" xref="footnote12.m2.1.1.2.cmml"><mn id="footnote12.m2.1.1.2.2" xref="footnote12.m2.1.1.2.2.cmml">0.5</mn><mo lspace="0em" rspace="0em" id="footnote12.m2.1.1.2.1" xref="footnote12.m2.1.1.2.1.cmml">​</mo><mtext id="footnote12.m2.1.1.2.3" xref="footnote12.m2.1.1.2.3a.cmml">M</mtext></mrow><mo lspace="0.392em" rspace="0.392em" id="footnote12.m2.1.1.1" xref="footnote12.m2.1.1.1.cmml">/</mo><mi id="footnote12.m2.1.1.3" xref="footnote12.m2.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="footnote12.m2.1c"><apply id="footnote12.m2.1.1.cmml" xref="footnote12.m2.1.1"><divide id="footnote12.m2.1.1.1.cmml" xref="footnote12.m2.1.1.1"></divide><apply id="footnote12.m2.1.1.2.cmml" xref="footnote12.m2.1.1.2"><times id="footnote12.m2.1.1.2.1.cmml" xref="footnote12.m2.1.1.2.1"></times><cn type="float" id="footnote12.m2.1.1.2.2.cmml" xref="footnote12.m2.1.1.2.2">0.5</cn><ci id="footnote12.m2.1.1.2.3a.cmml" xref="footnote12.m2.1.1.2.3"><mtext id="footnote12.m2.1.1.2.3.cmml" xref="footnote12.m2.1.1.2.3">M</mtext></ci></apply><ci id="footnote12.m2.1.1.3.cmml" xref="footnote12.m2.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote12.m2.1d">0.5\text{M}\,/\,T</annotation></semantics></math> sequences per batch (assume divisible).
</span></span></span>
From the subset of results in Figure <a href="#S3.F12" title="Figure 12 ‣ 3.2 Long context: Books ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12</span></a>, we make a few observations:</p>
<ul id="S3.I3" class="ltx_itemize">
<li id="S3.I3.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I3.i1.p1" class="ltx_para">
<p id="S3.I3.i1.p1.1" class="ltx_p">At 2k context on Books, all the observations from Pile 2k still hold, except that Mamba now performs slightly better than TTT-Linear (whereas their lines roughly overlapped for Pile 2k).</p>
</div>
</li>
<li id="S3.I3.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I3.i2.p1" class="ltx_para">
<p id="S3.I3.i2.p1.1" class="ltx_p">At 32k context, both TTT-Linear (M) and TTT-MLP (M) perform better than Mamba, similar to the observation from Pile 8k. Even TTT-MLP (T) with the Transformer backbone performs slightly better than Mamba at 32k context.</p>
</div>
</li>
<li id="S3.I3.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I3.i3.p1" class="ltx_para">
<p id="S3.I3.i3.p1.1" class="ltx_p">TTT-MLP (T) is only slightly worse than TTT-MLP (M) at 1.3B scale.
As discussed, it is hard to derive an empirical scaling law due to the lack of a clean linear fit. However, the strong trend for TTT-MLP (T) suggests that the Transformer backbone might be more suitable for larger models and longer context beyond our evaluations.</p>
</div>
</li>
</ul>
<p id="S3.SS2.p1.2" class="ltx_p">We only ablate the backbones for 2k and 32k due to the cost of training LLMs. For future work, we believe that given TTT layers with even more expressive hidden states, the Mamba backbone with temporal convolutions will become unnecessary.</p>
</div>
<section id="S3.SS2.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Transformer finetuning.</h5>

<div id="S3.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS2.SSS0.Px1.p1.1" class="ltx_p">While we have been training Transformers from scratch following the Mamba paper, in practice this approach is rarely used for long context.
The standard practice is to train a Transformer in short context, then finetune in long context.
To reflect this practice, we add another baseline, <em id="S3.SS2.SSS0.Px1.p1.1.1" class="ltx_emph ltx_font_italic">TF finetune</em>, for context lengths 4k and above.
This baseline starts from the model trained (according to the Chinchilla recipe) on Books 2k, then uses 20% more tokens to finetune at the designated context length, following the Llama Long paper <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib78" title="" class="ltx_ref">78</a>]</cite>.
See details of the TF finetune recipe in Appendix <a href="#A3" title="Appendix C Experiment details ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>.</p>
</div>
<div id="S3.SS2.SSS0.Px1.p2" class="ltx_para">
<p id="S3.SS2.SSS0.Px1.p2.1" class="ltx_p">Our complete results for context lengths 1k, 2k, 4k, 8k, 16k, 32k, including TF finetune, are in Figure <a href="#A3.F18" title="Figure 18 ‣ Gradient checkpointing through time. ‣ Appendix C Experiment details ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">18</span></a> (in Appendix).</p>
</div>
<figure id="S3.F13" class="ltx_figure"><img src="/html/2407.04620/assets/x13.png" id="S3.F13.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="207" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 13: </span>
Experiments on Books with context lengths ranging from 1k to 32k.
We treat context length as a hyper-parameter and connect the selected points.
Since we have Transformers trained from scratch and finetuned, we label them as <em id="S3.F13.5.1" class="ltx_emph ltx_font_italic">TF pretrain</em> and <em id="S3.F13.6.2" class="ltx_emph ltx_font_italic">TF finetune</em>.
The left panel of Figure <a href="#S0.F2" title="Figure 2 ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> is a zoomed-in view between 350M and 1.3B parameters, where <em id="S3.F13.7.3" class="ltx_emph ltx_font_italic">Transformer</em> is <em id="S3.F13.8.4" class="ltx_emph ltx_font_italic">TF finetune</em>, the stronger Transformer baseline.
</figcaption>
</figure>
</section>
<section id="S3.SS2.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Context length as a hyper-parameter.</h5>

<div id="S3.SS2.SSS0.Px2.p1" class="ltx_para">
<p id="S3.SS2.SSS0.Px2.p1.1" class="ltx_p">While the length of the input sequence is determined by the user, the context length at which the language model processes the input is determined by the engineer as a design choice.
Therefore, context length is a hyper-parameter that can be selected just as other ones.<span id="footnote13" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">13</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">13</sup><span class="ltx_tag ltx_tag_note">13</span>
To be precise, there are two hyper-parameters: the context length at which the LLM is trained, and one at which the LLM is evaluated.
Both of them can be different from the sequence length, which is determined by the user.
Transformers tend to perform poorly when the evaluation context is longer than the training context <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib18" title="" class="ltx_ref">18</a>]</cite>.
Therefore, we always evaluate at the training context length, making the two hyper-parameters the same.
</span></span></span>
For LLMs with linear complexity, we select the argmin in perplexity, since every context length has the same FLOPs.
For Transformers, longer context costs more FLOPs, so we form a convex hull of all the points in the log-log plot and connect those on the boundary.</p>
</div>
<div id="S3.SS2.SSS0.Px2.p2" class="ltx_para">
<p id="S3.SS2.SSS0.Px2.p2.1" class="ltx_p">From Figure <a href="#S3.F13" title="Figure 13 ‣ Transformer finetuning. ‣ 3.2 Long context: Books ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13</span></a>, we make a few observations:</p>
<ul id="S3.I4" class="ltx_itemize">
<li id="S3.I4.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I4.i1.p1" class="ltx_para">
<p id="S3.I4.i1.p1.1" class="ltx_p">The lines of TTT-Linear and TTT-MLP, the best-performing methods, almost completely overlap.
The lines of Mamba and TF finetune also mostly overlap after <math id="S3.I4.i1.p1.1.m1.1" class="ltx_Math" alttext="10^{20}" display="inline"><semantics id="S3.I4.i1.p1.1.m1.1a"><msup id="S3.I4.i1.p1.1.m1.1.1" xref="S3.I4.i1.p1.1.m1.1.1.cmml"><mn id="S3.I4.i1.p1.1.m1.1.1.2" xref="S3.I4.i1.p1.1.m1.1.1.2.cmml">10</mn><mn id="S3.I4.i1.p1.1.m1.1.1.3" xref="S3.I4.i1.p1.1.m1.1.1.3.cmml">20</mn></msup><annotation-xml encoding="MathML-Content" id="S3.I4.i1.p1.1.m1.1b"><apply id="S3.I4.i1.p1.1.m1.1.1.cmml" xref="S3.I4.i1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.I4.i1.p1.1.m1.1.1.1.cmml" xref="S3.I4.i1.p1.1.m1.1.1">superscript</csymbol><cn type="integer" id="S3.I4.i1.p1.1.m1.1.1.2.cmml" xref="S3.I4.i1.p1.1.m1.1.1.2">10</cn><cn type="integer" id="S3.I4.i1.p1.1.m1.1.1.3.cmml" xref="S3.I4.i1.p1.1.m1.1.1.3">20</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.I4.i1.p1.1.m1.1c">10^{20}</annotation></semantics></math> FLOPs.</p>
</div>
</li>
<li id="S3.I4.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I4.i2.p1" class="ltx_para">
<p id="S3.I4.i2.p1.1" class="ltx_p">TF finetune performs significantly better than TF pretrain, as it benefits from long context without incurring extremely large cost in training FLOPs.
Note that the inference FLOPs of TF finetune and pretrain are equally poor, which is not reflected in this plot.</p>
</div>
</li>
<li id="S3.I4.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I4.i3.p1" class="ltx_para">
<p id="S3.I4.i3.p1.1" class="ltx_p">For all methods trained from scratch (including TF pretrain), perplexity becomes worse once the context length becomes too large.
This trend is highlighted in Figure <a href="#A3.F19" title="Figure 19 ‣ Gradient checkpointing through time. ‣ Appendix C Experiment details ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">19</span></a> (in Appendix).
We leave further investigation of this trend to future work.</p>
</div>
</li>
</ul>
<p id="S3.SS2.SSS0.Px2.p2.2" class="ltx_p">The left panel of Figure <a href="#S0.F2" title="Figure 2 ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> is a zoomed-in view of Figure <a href="#S3.F13" title="Figure 13 ‣ Transformer finetuning. ‣ 3.2 Long context: Books ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13</span></a>.
For clarity, we leave TF pretrain out of Figure <a href="#S0.F2" title="Figure 2 ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> and only show TF finetune (labeled as <em id="S3.SS2.SSS0.Px2.p2.2.1" class="ltx_emph ltx_font_italic">Transformer</em>) since it is the stronger baseline.
Figure <a href="#S3.F14.3" title="Figure 14 ‣ Context length as a hyper-parameter. ‣ 3.2 Long context: Books ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">14</span></a> reproduces the right panel of Figure <a href="#S0.F2" title="Figure 2 ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, now with TTT-MLP and additional discussion.</p>
</div>
<figure id="S3.F14.3" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:186.5pt;">
<div id="S3.F14.1" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:238.5pt;">
<img src="/html/2407.04620/assets/x14.png" id="S3.F14.1.g1" class="ltx_graphics ltx_img_landscape" width="438" height="292" alt="Refer to caption">
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 14: </span>
Reproduction of the right panel of Figure <a href="#S0.F2" title="Figure 2 ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, now with TTT-MLP.
Comparing to TTT-Linear, TTT-MLP performs slightly worse at short context but better at long context.
This observation matches our expectation that the MLP as hidden state is more expressive than the linear model.
Again, all methods have matched training FLOPs as Mamba 1.4B.
For TTT-Linear and TTT-MLP, this protocol implies matched inference FLOPs.
Transformer (TF finetune) has <math id="S3.F14.3.2.m1.1" class="ltx_math_unparsed" alttext="2.8\times" display="inline"><semantics id="S3.F14.3.2.m1.1b"><mrow id="S3.F14.3.2.m1.1c"><mn id="S3.F14.3.2.m1.1.1">2.8</mn><mo lspace="0.222em" id="S3.F14.3.2.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S3.F14.3.2.m1.1d">2.8\times</annotation></semantics></math> the inference FLOPs, giving it an advantage as our baseline.
</figcaption>
</figure>
</section>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Wall-clock time</h3>

<figure id="S3.F15" class="ltx_figure"><img src="/html/2407.04620/assets/x15.png" id="S3.F15.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="438" height="131" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 15: </span>
Benchmark on an NVIDIA A100 GPU with 80G HBM and PCIe connections.
<span id="S3.F15.3.1" class="ltx_text ltx_font_bold">Left</span>: Same as Figure <a href="#S0.F3.fig1" title="Figure 3 ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, but now with TTT-MLP.
Note that our Transformer baseline is significantly faster then that in the Mamba paper, because we use vLLM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib46" title="" class="ltx_ref">46</a>]</cite>, a state-of-the-art serving system, instead of the HuggingFace Transformer <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib77" title="" class="ltx_ref">77</a>]</cite>.
<span id="S3.F15.4.2" class="ltx_text ltx_font_bold">Right</span>: For generate (decode), TTT-Linear and Mamba have almost the same latency, which is significantly smaller than that of Transformer and TTT-MLP.
</figcaption>
</figure>
<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">LLM training and inference can be decomposed into forward, backward, and generate.
Prompt processing during inference, also known as prefill, is the same operation as forward during training, except that the intermediate activations do not need to be stored for backward.
Since both forward (during training and inference) and backward can be parallelized, we use the dual form.
Generating new tokens, also known as decode, is inherently sequential, so we use the primal form.</p>
</div>
<div id="S3.SS3.p2" class="ltx_para">
<p id="S3.SS3.p2.1" class="ltx_p">Due to resource constraints, our experiments are written in JAX and run on TPUs.
On a v5e-256 TPU pod, the Transformer baseline takes 0.30s per iteration of training at context 2k, while TTT-Linear takes 0.27s per iteration, already 10% faster without any systems optimization.
However, Mamba (implemented in PyTorch, Triton, and CUDA) can only run on GPUs, so for fair comparison, we also rewrite our method with preliminary systems optimization to run on GPUs.</p>
</div>
<div id="S3.SS3.p3" class="ltx_para">
<p id="S3.SS3.p3.1" class="ltx_p">Specifically, we write a GPU kernel for forward in ThunderKittens <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib66" title="" class="ltx_ref">66</a>]</cite>.
Historically, RNNs have been inefficient during forward and backward due to poor use of parallelism and <span id="S3.SS3.p3.1.1" class="ltx_text ltx_font_typewriter">matmul</span>s.
Our goal with the forward kernel is to demonstrate the effectiveness of mini-batch TTT and the dual form for these problems.
A kernel for backward should have the same properties in efficiency as forward, but requires more complex logic for manual differentiation, therefore is left for future work.</p>
</div>
<div id="S3.SS3.p4" class="ltx_para">
<p id="S3.SS3.p4.1" class="ltx_p">The left panel of Figure <a href="#S3.F15" title="Figure 15 ‣ 3.3 Wall-clock time ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">15</span></a> shows the latency for batch size 16 of our forward kernel.
All models are 1.3B (1.4B for Mamba).
Time per token grows linearly for Transformer as context length increases, but stays roughly constant for the other methods.<span id="footnote14" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">14</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">14</sup><span class="ltx_tag ltx_tag_note">14</span>
We observe that forward latency of the network increases slightly for TTT-Linear, TTT-MLP, and Mamba, even though latency of each sequence modeling layer alone stays constant.
Consider the operation <math id="footnote14.m1.1" class="ltx_Math" alttext="\theta X" display="inline"><semantics id="footnote14.m1.1b"><mrow id="footnote14.m1.1.1" xref="footnote14.m1.1.1.cmml"><mi id="footnote14.m1.1.1.2" xref="footnote14.m1.1.1.2.cmml">θ</mi><mo lspace="0em" rspace="0em" id="footnote14.m1.1.1.1" xref="footnote14.m1.1.1.1.cmml">​</mo><mi id="footnote14.m1.1.1.3" xref="footnote14.m1.1.1.3.cmml">X</mi></mrow><annotation-xml encoding="MathML-Content" id="footnote14.m1.1c"><apply id="footnote14.m1.1.1.cmml" xref="footnote14.m1.1.1"><times id="footnote14.m1.1.1.1.cmml" xref="footnote14.m1.1.1.1"></times><ci id="footnote14.m1.1.1.2.cmml" xref="footnote14.m1.1.1.2">𝜃</ci><ci id="footnote14.m1.1.1.3.cmml" xref="footnote14.m1.1.1.3">𝑋</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote14.m1.1d">\theta X</annotation></semantics></math>, where <math id="footnote14.m2.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="footnote14.m2.1b"><mi id="footnote14.m2.1.1" xref="footnote14.m2.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="footnote14.m2.1c"><ci id="footnote14.m2.1.1.cmml" xref="footnote14.m2.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote14.m2.1d">\theta</annotation></semantics></math> is <math id="footnote14.m3.1" class="ltx_Math" alttext="d\times d" display="inline"><semantics id="footnote14.m3.1b"><mrow id="footnote14.m3.1.1" xref="footnote14.m3.1.1.cmml"><mi id="footnote14.m3.1.1.2" xref="footnote14.m3.1.1.2.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="footnote14.m3.1.1.1" xref="footnote14.m3.1.1.1.cmml">×</mo><mi id="footnote14.m3.1.1.3" xref="footnote14.m3.1.1.3.cmml">d</mi></mrow><annotation-xml encoding="MathML-Content" id="footnote14.m3.1c"><apply id="footnote14.m3.1.1.cmml" xref="footnote14.m3.1.1"><times id="footnote14.m3.1.1.1.cmml" xref="footnote14.m3.1.1.1"></times><ci id="footnote14.m3.1.1.2.cmml" xref="footnote14.m3.1.1.2">𝑑</ci><ci id="footnote14.m3.1.1.3.cmml" xref="footnote14.m3.1.1.3">𝑑</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote14.m3.1d">d\times d</annotation></semantics></math> and <math id="footnote14.m4.1" class="ltx_Math" alttext="X" display="inline"><semantics id="footnote14.m4.1b"><mi id="footnote14.m4.1.1" xref="footnote14.m4.1.1.cmml">X</mi><annotation-xml encoding="MathML-Content" id="footnote14.m4.1c"><ci id="footnote14.m4.1.1.cmml" xref="footnote14.m4.1.1">𝑋</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote14.m4.1d">X</annotation></semantics></math> is <math id="footnote14.m5.1" class="ltx_Math" alttext="d\times T" display="inline"><semantics id="footnote14.m5.1b"><mrow id="footnote14.m5.1.1" xref="footnote14.m5.1.1.cmml"><mi id="footnote14.m5.1.1.2" xref="footnote14.m5.1.1.2.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="footnote14.m5.1.1.1" xref="footnote14.m5.1.1.1.cmml">×</mo><mi id="footnote14.m5.1.1.3" xref="footnote14.m5.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="footnote14.m5.1c"><apply id="footnote14.m5.1.1.cmml" xref="footnote14.m5.1.1"><times id="footnote14.m5.1.1.1.cmml" xref="footnote14.m5.1.1.1"></times><ci id="footnote14.m5.1.1.2.cmml" xref="footnote14.m5.1.1.2">𝑑</ci><ci id="footnote14.m5.1.1.3.cmml" xref="footnote14.m5.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote14.m5.1d">d\times T</annotation></semantics></math>.
Its latency (normalized over <math id="footnote14.m6.1" class="ltx_Math" alttext="T" display="inline"><semantics id="footnote14.m6.1b"><mi id="footnote14.m6.1.1" xref="footnote14.m6.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="footnote14.m6.1c"><ci id="footnote14.m6.1.1.cmml" xref="footnote14.m6.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote14.m6.1d">T</annotation></semantics></math>) is expected to be constant, but in practice grows slightly with <math id="footnote14.m7.1" class="ltx_Math" alttext="T" display="inline"><semantics id="footnote14.m7.1b"><mi id="footnote14.m7.1.1" xref="footnote14.m7.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="footnote14.m7.1c"><ci id="footnote14.m7.1.1.cmml" xref="footnote14.m7.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote14.m7.1d">T</annotation></semantics></math>.
One possible cause of this phenomenon is the GPU throttling after <math id="footnote14.m8.1" class="ltx_Math" alttext="T" display="inline"><semantics id="footnote14.m8.1b"><mi id="footnote14.m8.1.1" xref="footnote14.m8.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="footnote14.m8.1c"><ci id="footnote14.m8.1.1.cmml" xref="footnote14.m8.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote14.m8.1d">T</annotation></semantics></math> gets very large <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib30" title="" class="ltx_ref">30</a>]</cite>.
</span></span></span>
Note that our Transformer baseline is significantly faster that in the Mamba paper, because we use vLLM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib46" title="" class="ltx_ref">46</a>]</cite>, a state-of-the-art serving system, instead of the HuggingFace Transformer <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib77" title="" class="ltx_ref">77</a>]</cite>.</p>
</div>
<div id="S3.SS3.p5" class="ltx_para">
<p id="S3.SS3.p5.1" class="ltx_p">In addition, we write another GPU kernel for generate in Triton <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib72" title="" class="ltx_ref">72</a>]</cite>, and benchmark its speed in the right panel of Figure <a href="#S3.F15" title="Figure 15 ‣ 3.3 Wall-clock time ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">15</span></a> for batch size 512.
Another popular metric for wall-clock time is throughput, which takes into account the potential benefit of being able to use a larger batch size.<span id="footnote15" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">15</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">15</sup><span class="ltx_tag ltx_tag_note">15</span>
To calculate throughput for each method, we increase its batch size in <math id="footnote15.m1.1" class="ltx_math_unparsed" alttext="2\times" display="inline"><semantics id="footnote15.m1.1b"><mrow id="footnote15.m1.1c"><mn id="footnote15.m1.1.1">2</mn><mo lspace="0.222em" id="footnote15.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="footnote15.m1.1d">2\times</annotation></semantics></math> increments until GPU runs out of memory, measure the tokens per second for every batch size, and select the highest.</span></span></span>
For completeness, we report the throughput for forward and generate in Figure <a href="#A3.F20" title="Figure 20 ‣ Gradient checkpointing through time. ‣ Appendix C Experiment details ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">20</span></a> (in Appendix). All the observations and ordering between methods above still hold for throughput.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Related Work</h2>

<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Modern RNNs</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">Mamba is one of the many Structured State-Space Models <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib27" title="" class="ltx_ref">27</a>, <a href="#bib.bib21" title="" class="ltx_ref">21</a>, <a href="#bib.bib57" title="" class="ltx_ref">57</a>, <a href="#bib.bib18" title="" class="ltx_ref">18</a>]</cite>. The hidden state in these models is a vector, similar to in LSTMs.
For TTT-Linear or TTT-MLP, the hidden state is a matrix or two matrices, therefore larger.
In Figure <a href="#S3.F14.3" title="Figure 14 ‣ Context length as a hyper-parameter. ‣ 3.2 Long context: Books ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">14</span></a>, we find that TTT layers can take advantage of their larger hidden states to compress more information in long context, where TTT-MLP outperforms TTT-Linear, which in turn outperforms Mamba.</p>
</div>
<div id="S4.SS1.p2" class="ltx_para">
<p id="S4.SS1.p2.1" class="ltx_p">Similar to TTT-Linear, RWKV <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib55" title="" class="ltx_ref">55</a>, <a href="#bib.bib56" title="" class="ltx_ref">56</a>]</cite>, xLSTM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib5" title="" class="ltx_ref">5</a>]</cite>, and Gated Linear Attention (GLA) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib79" title="" class="ltx_ref">79</a>]</cite> also have matrix hidden states, which are inherited from linear attention <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib41" title="" class="ltx_ref">41</a>]</cite>.
Modern RNNs such as GLA use chunk-wise parallelism to improve hardware efficiency, so tokens inside a chunk can be processed with <span id="S4.SS1.p2.1.1" class="ltx_text ltx_font_typewriter">matmul</span>s instead of a <span id="S4.SS1.p2.1.2" class="ltx_text ltx_font_typewriter">cumsum</span>.
However, chunk-wise parallelism does not change the expressiveness of the model, since all temporal dependencies are still equivalent to a <span id="S4.SS1.p2.1.3" class="ltx_text ltx_font_typewriter">cumsum</span>.</p>
</div>
<div id="S4.SS1.p3" class="ltx_para">
<p id="S4.SS1.p3.6" class="ltx_p">In contrast, mini-batch TTT allows more complex temporal dependencies across mini-batches. Each hidden state <math id="S4.SS1.p3.1.m1.1" class="ltx_Math" alttext="W_{t}" display="inline"><semantics id="S4.SS1.p3.1.m1.1a"><msub id="S4.SS1.p3.1.m1.1.1" xref="S4.SS1.p3.1.m1.1.1.cmml"><mi id="S4.SS1.p3.1.m1.1.1.2" xref="S4.SS1.p3.1.m1.1.1.2.cmml">W</mi><mi id="S4.SS1.p3.1.m1.1.1.3" xref="S4.SS1.p3.1.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.p3.1.m1.1b"><apply id="S4.SS1.p3.1.m1.1.1.cmml" xref="S4.SS1.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS1.p3.1.m1.1.1.1.cmml" xref="S4.SS1.p3.1.m1.1.1">subscript</csymbol><ci id="S4.SS1.p3.1.m1.1.1.2.cmml" xref="S4.SS1.p3.1.m1.1.1.2">𝑊</ci><ci id="S4.SS1.p3.1.m1.1.1.3.cmml" xref="S4.SS1.p3.1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p3.1.m1.1c">W_{t}</annotation></semantics></math> depends on previous <math id="S4.SS1.p3.2.m2.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S4.SS1.p3.2.m2.1a"><mi id="S4.SS1.p3.2.m2.1.1" xref="S4.SS1.p3.2.m2.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p3.2.m2.1b"><ci id="S4.SS1.p3.2.m2.1.1.cmml" xref="S4.SS1.p3.2.m2.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p3.2.m2.1c">W</annotation></semantics></math>s within its mini-batch still through a <span id="S4.SS1.p3.6.1" class="ltx_text ltx_font_typewriter">cumsum</span>, but depends on <math id="S4.SS1.p3.3.m3.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S4.SS1.p3.3.m3.1a"><mi id="S4.SS1.p3.3.m3.1.1" xref="S4.SS1.p3.3.m3.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p3.3.m3.1b"><ci id="S4.SS1.p3.3.m3.1.1.cmml" xref="S4.SS1.p3.3.m3.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p3.3.m3.1c">W</annotation></semantics></math>s in previous mini-batches also through the gradient operator.
As illustrated Figure <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:fig:batch-size</span>, mini-batch TTT enables a trade-off between expressiveness and hardware efficiency, since a smaller batch size <math id="S4.SS1.p3.4.m4.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S4.SS1.p3.4.m4.1a"><mi id="S4.SS1.p3.4.m4.1.1" xref="S4.SS1.p3.4.m4.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p3.4.m4.1b"><ci id="S4.SS1.p3.4.m4.1.1.cmml" xref="S4.SS1.p3.4.m4.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p3.4.m4.1c">b</annotation></semantics></math> leads to better perplexity at the cost of higher latency.
This trade-off is a unique and important feature of TTT. As shown in Table <a href="#S2.T1" title="Table 1 ‣ 2.6 Theoretical equivalences ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, the intermediate batch size <math id="S4.SS1.p3.5.m5.1" class="ltx_Math" alttext="b=16" display="inline"><semantics id="S4.SS1.p3.5.m5.1a"><mrow id="S4.SS1.p3.5.m5.1.1" xref="S4.SS1.p3.5.m5.1.1.cmml"><mi id="S4.SS1.p3.5.m5.1.1.2" xref="S4.SS1.p3.5.m5.1.1.2.cmml">b</mi><mo id="S4.SS1.p3.5.m5.1.1.1" xref="S4.SS1.p3.5.m5.1.1.1.cmml">=</mo><mn id="S4.SS1.p3.5.m5.1.1.3" xref="S4.SS1.p3.5.m5.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p3.5.m5.1b"><apply id="S4.SS1.p3.5.m5.1.1.cmml" xref="S4.SS1.p3.5.m5.1.1"><eq id="S4.SS1.p3.5.m5.1.1.1.cmml" xref="S4.SS1.p3.5.m5.1.1.1"></eq><ci id="S4.SS1.p3.5.m5.1.1.2.cmml" xref="S4.SS1.p3.5.m5.1.1.2">𝑏</ci><cn type="integer" id="S4.SS1.p3.5.m5.1.1.3.cmml" xref="S4.SS1.p3.5.m5.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p3.5.m5.1c">b=16</annotation></semantics></math> significantly outperforms <math id="S4.SS1.p3.6.m6.1" class="ltx_Math" alttext="b=T" display="inline"><semantics id="S4.SS1.p3.6.m6.1a"><mrow id="S4.SS1.p3.6.m6.1.1" xref="S4.SS1.p3.6.m6.1.1.cmml"><mi id="S4.SS1.p3.6.m6.1.1.2" xref="S4.SS1.p3.6.m6.1.1.2.cmml">b</mi><mo id="S4.SS1.p3.6.m6.1.1.1" xref="S4.SS1.p3.6.m6.1.1.1.cmml">=</mo><mi id="S4.SS1.p3.6.m6.1.1.3" xref="S4.SS1.p3.6.m6.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p3.6.m6.1b"><apply id="S4.SS1.p3.6.m6.1.1.cmml" xref="S4.SS1.p3.6.m6.1.1"><eq id="S4.SS1.p3.6.m6.1.1.1.cmml" xref="S4.SS1.p3.6.m6.1.1.1"></eq><ci id="S4.SS1.p3.6.m6.1.1.2.cmml" xref="S4.SS1.p3.6.m6.1.1.2">𝑏</ci><ci id="S4.SS1.p3.6.m6.1.1.3.cmml" xref="S4.SS1.p3.6.m6.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p3.6.m6.1c">b=T</annotation></semantics></math> which is fully <span id="S4.SS1.p3.6.2" class="ltx_text ltx_font_typewriter">cumsum</span>.</p>
</div>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Learning at Test Time</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">The idea of learning at test time has a long history in machine learning.
One of the earliest versions of this idea is called local learning (Bottou and Vapnik <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib10" title="" class="ltx_ref">10</a>]</cite>):
For each test input, train on its neighbors before making a prediction.
This procedure has been effectively applied to models ranging from SVMs <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib81" title="" class="ltx_ref">81</a>]</cite> to modern LLMs <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib29" title="" class="ltx_ref">29</a>]</cite>.</p>
</div>
<div id="S4.SS2.p2" class="ltx_para">
<p id="S4.SS2.p2.1" class="ltx_p">Another early version of learning at test time is called <em id="S4.SS2.p2.1.1" class="ltx_emph ltx_font_italic">transductive learning</em> <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib22" title="" class="ltx_ref">22</a>]</cite>.
The principle of transduction, as stated by Vladimir Vapnik <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib74" title="" class="ltx_ref">74</a>]</cite>, is to
"… get the answer that you really need, but not a more general one."
Practical implementations of transductive learning use test data to add constraints to the margin of SVMs <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib39" title="" class="ltx_ref">39</a>, <a href="#bib.bib17" title="" class="ltx_ref">17</a>]</cite>.
However, transductive learning usually needs multiple test instances to be empirically effective, unlike many instantiations of test-time training, which only need a test single instance (image, video, or natural language sequence) at a time.</p>
</div>
<div id="S4.SS2.p3" class="ltx_para">
<p id="S4.SS2.p3.1" class="ltx_p">In computer vision, the idea of learning at test time has been applied for decades to applications such as face detection <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib38" title="" class="ltx_ref">38</a>]</cite>, object detection <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib53" title="" class="ltx_ref">53</a>]</cite>, image super-resolution <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib65" title="" class="ltx_ref">65</a>]</cite>, and 3D reconstruction <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib50" title="" class="ltx_ref">50</a>]</cite>.
More recently, the same idea has also been applied to natural language processing, where it is called dynamic evaluation <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>, <a href="#bib.bib45" title="" class="ltx_ref">45</a>]</cite>.
The basic approach is to directly finetune a language model on the test sequence, which often comes in the form of a prompt.</p>
</div>
<div id="S4.SS2.p4" class="ltx_para">
<p id="S4.SS2.p4.1" class="ltx_p">Next, we discuss two relevant lines of work in detail: test-time training and fast weights.</p>
</div>
<section id="S4.SS2.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.2.1 </span>Test-Time Training</h4>

<div id="S4.SS2.SSS1.p1" class="ltx_para">
<p id="S4.SS2.SSS1.p1.8" class="ltx_p">The core idea of <em id="S4.SS2.SSS1.p1.8.1" class="ltx_emph ltx_font_italic">Test-Time Training</em> (TTT) is that each test instance defines its own learning problem, where this test instance alone is the target of generalization <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib69" title="" class="ltx_ref">69</a>]</cite>.
Concretely, for each test instance <math id="S4.SS2.SSS1.p1.1.m1.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S4.SS2.SSS1.p1.1.m1.1a"><mi id="S4.SS2.SSS1.p1.1.m1.1.1" xref="S4.SS2.SSS1.p1.1.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS1.p1.1.m1.1b"><ci id="S4.SS2.SSS1.p1.1.m1.1.1.cmml" xref="S4.SS2.SSS1.p1.1.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS1.p1.1.m1.1c">x</annotation></semantics></math>, the conventional practice is to predict <math id="S4.SS2.SSS1.p1.2.m2.1" class="ltx_Math" alttext="f(x)" display="inline"><semantics id="S4.SS2.SSS1.p1.2.m2.1a"><mrow id="S4.SS2.SSS1.p1.2.m2.1.2" xref="S4.SS2.SSS1.p1.2.m2.1.2.cmml"><mi id="S4.SS2.SSS1.p1.2.m2.1.2.2" xref="S4.SS2.SSS1.p1.2.m2.1.2.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="S4.SS2.SSS1.p1.2.m2.1.2.1" xref="S4.SS2.SSS1.p1.2.m2.1.2.1.cmml">​</mo><mrow id="S4.SS2.SSS1.p1.2.m2.1.2.3.2" xref="S4.SS2.SSS1.p1.2.m2.1.2.cmml"><mo stretchy="false" id="S4.SS2.SSS1.p1.2.m2.1.2.3.2.1" xref="S4.SS2.SSS1.p1.2.m2.1.2.cmml">(</mo><mi id="S4.SS2.SSS1.p1.2.m2.1.1" xref="S4.SS2.SSS1.p1.2.m2.1.1.cmml">x</mi><mo stretchy="false" id="S4.SS2.SSS1.p1.2.m2.1.2.3.2.2" xref="S4.SS2.SSS1.p1.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS1.p1.2.m2.1b"><apply id="S4.SS2.SSS1.p1.2.m2.1.2.cmml" xref="S4.SS2.SSS1.p1.2.m2.1.2"><times id="S4.SS2.SSS1.p1.2.m2.1.2.1.cmml" xref="S4.SS2.SSS1.p1.2.m2.1.2.1"></times><ci id="S4.SS2.SSS1.p1.2.m2.1.2.2.cmml" xref="S4.SS2.SSS1.p1.2.m2.1.2.2">𝑓</ci><ci id="S4.SS2.SSS1.p1.2.m2.1.1.cmml" xref="S4.SS2.SSS1.p1.2.m2.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS1.p1.2.m2.1c">f(x)</annotation></semantics></math>, using a predictor <math id="S4.SS2.SSS1.p1.3.m3.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S4.SS2.SSS1.p1.3.m3.1a"><mi id="S4.SS2.SSS1.p1.3.m3.1.1" xref="S4.SS2.SSS1.p1.3.m3.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS1.p1.3.m3.1b"><ci id="S4.SS2.SSS1.p1.3.m3.1.1.cmml" xref="S4.SS2.SSS1.p1.3.m3.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS1.p1.3.m3.1c">f</annotation></semantics></math> that is optimized for all training instances on average.
TTT first formulates a learning problem defined by <math id="S4.SS2.SSS1.p1.4.m4.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S4.SS2.SSS1.p1.4.m4.1a"><mi id="S4.SS2.SSS1.p1.4.m4.1.1" xref="S4.SS2.SSS1.p1.4.m4.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS1.p1.4.m4.1b"><ci id="S4.SS2.SSS1.p1.4.m4.1.1.cmml" xref="S4.SS2.SSS1.p1.4.m4.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS1.p1.4.m4.1c">x</annotation></semantics></math>, then trains a model <math id="S4.SS2.SSS1.p1.5.m5.1" class="ltx_Math" alttext="f_{x}" display="inline"><semantics id="S4.SS2.SSS1.p1.5.m5.1a"><msub id="S4.SS2.SSS1.p1.5.m5.1.1" xref="S4.SS2.SSS1.p1.5.m5.1.1.cmml"><mi id="S4.SS2.SSS1.p1.5.m5.1.1.2" xref="S4.SS2.SSS1.p1.5.m5.1.1.2.cmml">f</mi><mi id="S4.SS2.SSS1.p1.5.m5.1.1.3" xref="S4.SS2.SSS1.p1.5.m5.1.1.3.cmml">x</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS1.p1.5.m5.1b"><apply id="S4.SS2.SSS1.p1.5.m5.1.1.cmml" xref="S4.SS2.SSS1.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS1.p1.5.m5.1.1.1.cmml" xref="S4.SS2.SSS1.p1.5.m5.1.1">subscript</csymbol><ci id="S4.SS2.SSS1.p1.5.m5.1.1.2.cmml" xref="S4.SS2.SSS1.p1.5.m5.1.1.2">𝑓</ci><ci id="S4.SS2.SSS1.p1.5.m5.1.1.3.cmml" xref="S4.SS2.SSS1.p1.5.m5.1.1.3">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS1.p1.5.m5.1c">f_{x}</annotation></semantics></math> on <math id="S4.SS2.SSS1.p1.6.m6.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S4.SS2.SSS1.p1.6.m6.1a"><mi id="S4.SS2.SSS1.p1.6.m6.1.1" xref="S4.SS2.SSS1.p1.6.m6.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS1.p1.6.m6.1b"><ci id="S4.SS2.SSS1.p1.6.m6.1.1.cmml" xref="S4.SS2.SSS1.p1.6.m6.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS1.p1.6.m6.1c">x</annotation></semantics></math> (often with <math id="S4.SS2.SSS1.p1.7.m7.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S4.SS2.SSS1.p1.7.m7.1a"><mi id="S4.SS2.SSS1.p1.7.m7.1.1" xref="S4.SS2.SSS1.p1.7.m7.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS1.p1.7.m7.1b"><ci id="S4.SS2.SSS1.p1.7.m7.1.1.cmml" xref="S4.SS2.SSS1.p1.7.m7.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS1.p1.7.m7.1c">f</annotation></semantics></math> as initialization), and predicts <math id="S4.SS2.SSS1.p1.8.m8.1" class="ltx_Math" alttext="f_{x}(x)" display="inline"><semantics id="S4.SS2.SSS1.p1.8.m8.1a"><mrow id="S4.SS2.SSS1.p1.8.m8.1.2" xref="S4.SS2.SSS1.p1.8.m8.1.2.cmml"><msub id="S4.SS2.SSS1.p1.8.m8.1.2.2" xref="S4.SS2.SSS1.p1.8.m8.1.2.2.cmml"><mi id="S4.SS2.SSS1.p1.8.m8.1.2.2.2" xref="S4.SS2.SSS1.p1.8.m8.1.2.2.2.cmml">f</mi><mi id="S4.SS2.SSS1.p1.8.m8.1.2.2.3" xref="S4.SS2.SSS1.p1.8.m8.1.2.2.3.cmml">x</mi></msub><mo lspace="0em" rspace="0em" id="S4.SS2.SSS1.p1.8.m8.1.2.1" xref="S4.SS2.SSS1.p1.8.m8.1.2.1.cmml">​</mo><mrow id="S4.SS2.SSS1.p1.8.m8.1.2.3.2" xref="S4.SS2.SSS1.p1.8.m8.1.2.cmml"><mo stretchy="false" id="S4.SS2.SSS1.p1.8.m8.1.2.3.2.1" xref="S4.SS2.SSS1.p1.8.m8.1.2.cmml">(</mo><mi id="S4.SS2.SSS1.p1.8.m8.1.1" xref="S4.SS2.SSS1.p1.8.m8.1.1.cmml">x</mi><mo stretchy="false" id="S4.SS2.SSS1.p1.8.m8.1.2.3.2.2" xref="S4.SS2.SSS1.p1.8.m8.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS1.p1.8.m8.1b"><apply id="S4.SS2.SSS1.p1.8.m8.1.2.cmml" xref="S4.SS2.SSS1.p1.8.m8.1.2"><times id="S4.SS2.SSS1.p1.8.m8.1.2.1.cmml" xref="S4.SS2.SSS1.p1.8.m8.1.2.1"></times><apply id="S4.SS2.SSS1.p1.8.m8.1.2.2.cmml" xref="S4.SS2.SSS1.p1.8.m8.1.2.2"><csymbol cd="ambiguous" id="S4.SS2.SSS1.p1.8.m8.1.2.2.1.cmml" xref="S4.SS2.SSS1.p1.8.m8.1.2.2">subscript</csymbol><ci id="S4.SS2.SSS1.p1.8.m8.1.2.2.2.cmml" xref="S4.SS2.SSS1.p1.8.m8.1.2.2.2">𝑓</ci><ci id="S4.SS2.SSS1.p1.8.m8.1.2.2.3.cmml" xref="S4.SS2.SSS1.p1.8.m8.1.2.2.3">𝑥</ci></apply><ci id="S4.SS2.SSS1.p1.8.m8.1.1.cmml" xref="S4.SS2.SSS1.p1.8.m8.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS1.p1.8.m8.1c">f_{x}(x)</annotation></semantics></math>.</p>
</div>
<div id="S4.SS2.SSS1.p2" class="ltx_para">
<p id="S4.SS2.SSS1.p2.2" class="ltx_p">Since the test instance comes without its label, the learning problem can only be formulated with a self-supervised task. Prior work has shown that TTT with reconstruction significantly improves performance especially on outliers <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib23" title="" class="ltx_ref">23</a>]</cite>.
Improvements become even more pronounced when testing on video frames that arrive in a stream and TTT is autoregressive <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib76" title="" class="ltx_ref">76</a>]</cite>, as <math id="S4.SS2.SSS1.p2.1.m1.1" class="ltx_Math" alttext="f_{t}" display="inline"><semantics id="S4.SS2.SSS1.p2.1.m1.1a"><msub id="S4.SS2.SSS1.p2.1.m1.1.1" xref="S4.SS2.SSS1.p2.1.m1.1.1.cmml"><mi id="S4.SS2.SSS1.p2.1.m1.1.1.2" xref="S4.SS2.SSS1.p2.1.m1.1.1.2.cmml">f</mi><mi id="S4.SS2.SSS1.p2.1.m1.1.1.3" xref="S4.SS2.SSS1.p2.1.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS1.p2.1.m1.1b"><apply id="S4.SS2.SSS1.p2.1.m1.1.1.cmml" xref="S4.SS2.SSS1.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS1.p2.1.m1.1.1.1.cmml" xref="S4.SS2.SSS1.p2.1.m1.1.1">subscript</csymbol><ci id="S4.SS2.SSS1.p2.1.m1.1.1.2.cmml" xref="S4.SS2.SSS1.p2.1.m1.1.1.2">𝑓</ci><ci id="S4.SS2.SSS1.p2.1.m1.1.1.3.cmml" xref="S4.SS2.SSS1.p2.1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS1.p2.1.m1.1c">f_{t}</annotation></semantics></math> is trained on past frames <math id="S4.SS2.SSS1.p2.2.m2.3" class="ltx_Math" alttext="x_{1},\dots,x_{t}" display="inline"><semantics id="S4.SS2.SSS1.p2.2.m2.3a"><mrow id="S4.SS2.SSS1.p2.2.m2.3.3.2" xref="S4.SS2.SSS1.p2.2.m2.3.3.3.cmml"><msub id="S4.SS2.SSS1.p2.2.m2.2.2.1.1" xref="S4.SS2.SSS1.p2.2.m2.2.2.1.1.cmml"><mi id="S4.SS2.SSS1.p2.2.m2.2.2.1.1.2" xref="S4.SS2.SSS1.p2.2.m2.2.2.1.1.2.cmml">x</mi><mn id="S4.SS2.SSS1.p2.2.m2.2.2.1.1.3" xref="S4.SS2.SSS1.p2.2.m2.2.2.1.1.3.cmml">1</mn></msub><mo id="S4.SS2.SSS1.p2.2.m2.3.3.2.3" xref="S4.SS2.SSS1.p2.2.m2.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S4.SS2.SSS1.p2.2.m2.1.1" xref="S4.SS2.SSS1.p2.2.m2.1.1.cmml">…</mi><mo id="S4.SS2.SSS1.p2.2.m2.3.3.2.4" xref="S4.SS2.SSS1.p2.2.m2.3.3.3.cmml">,</mo><msub id="S4.SS2.SSS1.p2.2.m2.3.3.2.2" xref="S4.SS2.SSS1.p2.2.m2.3.3.2.2.cmml"><mi id="S4.SS2.SSS1.p2.2.m2.3.3.2.2.2" xref="S4.SS2.SSS1.p2.2.m2.3.3.2.2.2.cmml">x</mi><mi id="S4.SS2.SSS1.p2.2.m2.3.3.2.2.3" xref="S4.SS2.SSS1.p2.2.m2.3.3.2.2.3.cmml">t</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS1.p2.2.m2.3b"><list id="S4.SS2.SSS1.p2.2.m2.3.3.3.cmml" xref="S4.SS2.SSS1.p2.2.m2.3.3.2"><apply id="S4.SS2.SSS1.p2.2.m2.2.2.1.1.cmml" xref="S4.SS2.SSS1.p2.2.m2.2.2.1.1"><csymbol cd="ambiguous" id="S4.SS2.SSS1.p2.2.m2.2.2.1.1.1.cmml" xref="S4.SS2.SSS1.p2.2.m2.2.2.1.1">subscript</csymbol><ci id="S4.SS2.SSS1.p2.2.m2.2.2.1.1.2.cmml" xref="S4.SS2.SSS1.p2.2.m2.2.2.1.1.2">𝑥</ci><cn type="integer" id="S4.SS2.SSS1.p2.2.m2.2.2.1.1.3.cmml" xref="S4.SS2.SSS1.p2.2.m2.2.2.1.1.3">1</cn></apply><ci id="S4.SS2.SSS1.p2.2.m2.1.1.cmml" xref="S4.SS2.SSS1.p2.2.m2.1.1">…</ci><apply id="S4.SS2.SSS1.p2.2.m2.3.3.2.2.cmml" xref="S4.SS2.SSS1.p2.2.m2.3.3.2.2"><csymbol cd="ambiguous" id="S4.SS2.SSS1.p2.2.m2.3.3.2.2.1.cmml" xref="S4.SS2.SSS1.p2.2.m2.3.3.2.2">subscript</csymbol><ci id="S4.SS2.SSS1.p2.2.m2.3.3.2.2.2.cmml" xref="S4.SS2.SSS1.p2.2.m2.3.3.2.2.2">𝑥</ci><ci id="S4.SS2.SSS1.p2.2.m2.3.3.2.2.3.cmml" xref="S4.SS2.SSS1.p2.2.m2.3.3.2.2.3">𝑡</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS1.p2.2.m2.3c">x_{1},\dots,x_{t}</annotation></semantics></math>.
The autoregressive connection makes <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib76" title="" class="ltx_ref">76</a>]</cite> most relevant to our paper.</p>
</div>
<div id="S4.SS2.SSS1.p3" class="ltx_para">
<p id="S4.SS2.SSS1.p3.1" class="ltx_p">Conceptually, the biggest difference between our paper and prior work is that our reconstruction task is learned in an outer loop, instead of handcrafted with human priors.
Follow-up work to TTT has explored applications such as
robot manipulation <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib28" title="" class="ltx_ref">28</a>]</cite> and locomotion <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib68" title="" class="ltx_ref">68</a>]</cite>,
among others, that often require different designs for the self-supervised task.</p>
</div>
</section>
<section id="S4.SS2.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.2.2 </span>Fast Weights</h4>

<div id="S4.SS2.SSS2.p1" class="ltx_para">
<p id="S4.SS2.SSS2.p1.1" class="ltx_p">The general idea of <em id="S4.SS2.SSS2.p1.1.1" class="ltx_emph ltx_font_italic">fast weights</em> is to update the parameters of a “fast” model on only the most relevant data, as opposed to the conventional practice of updating a “slow” model on all data <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib71" title="" class="ltx_ref">71</a>]</cite>.
This idea has existed since the 1980s <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib32" title="" class="ltx_ref">32</a>]</cite>.
The most relevant data can be the test instance itself, therefore TTT can be viewed as a special case of fast weights.</p>
</div>
<div id="S4.SS2.SSS2.p2" class="ltx_para">
<p id="S4.SS2.SSS2.p2.2" class="ltx_p">Prior work in fast weights usually avoids forming an explicit learning problem that optimizes some objective on data.
For example, the update rule of Hebbian learning and Hopfield networks <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib35" title="" class="ltx_ref">35</a>]</cite> simply adds <math id="S4.SS2.SSS2.p2.1.m1.1" class="ltx_Math" alttext="xx^{T}" display="inline"><semantics id="S4.SS2.SSS2.p2.1.m1.1a"><mrow id="S4.SS2.SSS2.p2.1.m1.1.1" xref="S4.SS2.SSS2.p2.1.m1.1.1.cmml"><mi id="S4.SS2.SSS2.p2.1.m1.1.1.2" xref="S4.SS2.SSS2.p2.1.m1.1.1.2.cmml">x</mi><mo lspace="0em" rspace="0em" id="S4.SS2.SSS2.p2.1.m1.1.1.1" xref="S4.SS2.SSS2.p2.1.m1.1.1.1.cmml">​</mo><msup id="S4.SS2.SSS2.p2.1.m1.1.1.3" xref="S4.SS2.SSS2.p2.1.m1.1.1.3.cmml"><mi id="S4.SS2.SSS2.p2.1.m1.1.1.3.2" xref="S4.SS2.SSS2.p2.1.m1.1.1.3.2.cmml">x</mi><mi id="S4.SS2.SSS2.p2.1.m1.1.1.3.3" xref="S4.SS2.SSS2.p2.1.m1.1.1.3.3.cmml">T</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS2.p2.1.m1.1b"><apply id="S4.SS2.SSS2.p2.1.m1.1.1.cmml" xref="S4.SS2.SSS2.p2.1.m1.1.1"><times id="S4.SS2.SSS2.p2.1.m1.1.1.1.cmml" xref="S4.SS2.SSS2.p2.1.m1.1.1.1"></times><ci id="S4.SS2.SSS2.p2.1.m1.1.1.2.cmml" xref="S4.SS2.SSS2.p2.1.m1.1.1.2">𝑥</ci><apply id="S4.SS2.SSS2.p2.1.m1.1.1.3.cmml" xref="S4.SS2.SSS2.p2.1.m1.1.1.3"><csymbol cd="ambiguous" id="S4.SS2.SSS2.p2.1.m1.1.1.3.1.cmml" xref="S4.SS2.SSS2.p2.1.m1.1.1.3">superscript</csymbol><ci id="S4.SS2.SSS2.p2.1.m1.1.1.3.2.cmml" xref="S4.SS2.SSS2.p2.1.m1.1.1.3.2">𝑥</ci><ci id="S4.SS2.SSS2.p2.1.m1.1.1.3.3.cmml" xref="S4.SS2.SSS2.p2.1.m1.1.1.3.3">𝑇</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS2.p2.1.m1.1c">xx^{T}</annotation></semantics></math> (or some variant thereof) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib4" title="" class="ltx_ref">4</a>]</cite> to the fast weights given each input <math id="S4.SS2.SSS2.p2.2.m2.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S4.SS2.SSS2.p2.2.m2.1a"><mi id="S4.SS2.SSS2.p2.2.m2.1.1" xref="S4.SS2.SSS2.p2.2.m2.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS2.p2.2.m2.1b"><ci id="S4.SS2.SSS2.p2.2.m2.1.1.cmml" xref="S4.SS2.SSS2.p2.2.m2.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS2.p2.2.m2.1c">x</annotation></semantics></math>.
In contrast, TTT embraces the idea of formulating an explicit learning problem, where the test instance is the target of generalization.
Our update rule is also an explicit step of optimization.</p>
</div>
<div id="S4.SS2.SSS2.p3" class="ltx_para">
<p id="S4.SS2.SSS2.p3.2" class="ltx_p">The idea of <em id="S4.SS2.SSS2.p3.2.1" class="ltx_emph ltx_font_italic">fast weight programmers</em> (FWPs) is to update the fast weights with a “slow” model <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib62" title="" class="ltx_ref">62</a>]</cite>.
Our inner-loop weights <math id="S4.SS2.SSS2.p3.1.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S4.SS2.SSS2.p3.1.m1.1a"><mi id="S4.SS2.SSS2.p3.1.m1.1.1" xref="S4.SS2.SSS2.p3.1.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS2.p3.1.m1.1b"><ci id="S4.SS2.SSS2.p3.1.m1.1.1.cmml" xref="S4.SS2.SSS2.p3.1.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS2.p3.1.m1.1c">W</annotation></semantics></math> can be viewed as “fast” and outer-loop weights <math id="S4.SS2.SSS2.p3.2.m2.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="S4.SS2.SSS2.p3.2.m2.1a"><mi id="S4.SS2.SSS2.p3.2.m2.1.1" xref="S4.SS2.SSS2.p3.2.m2.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS2.p3.2.m2.1b"><ci id="S4.SS2.SSS2.p3.2.m2.1.1.cmml" xref="S4.SS2.SSS2.p3.2.m2.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS2.p3.2.m2.1c">\theta</annotation></semantics></math> as “slow”.
Therefore, networks containing TTT layers can be viewed as a special case of FWPs <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib43" title="" class="ltx_ref">43</a>]</cite>, similar to how TTT can be viewed as a special case of fast weights.
The FWP with the Hebbian update rule above is equivalent to linear attention <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib60" title="" class="ltx_ref">60</a>]</cite>, therefore also to naive TTT-Linear with batch gradient descent.</p>
</div>
<div id="S4.SS2.SSS2.p4" class="ltx_para">
<p id="S4.SS2.SSS2.p4.1" class="ltx_p">The definition of FWPs is very broad.
In fact, all networks with some gating mechanism, such as Transformers with SwiGLU blocks <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib63" title="" class="ltx_ref">63</a>]</cite>, can also be viewed as a special case of FWPs<span id="footnote16" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">16</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">16</sup><span class="ltx_tag ltx_tag_note">16</span>
Consider a simple gate <math id="footnote16.m1.2" class="ltx_Math" alttext="z=\sigma(\theta x)\odot(\theta^{\prime}x)" display="inline"><semantics id="footnote16.m1.2b"><mrow id="footnote16.m1.2.2" xref="footnote16.m1.2.2.cmml"><mi id="footnote16.m1.2.2.4" xref="footnote16.m1.2.2.4.cmml">z</mi><mo id="footnote16.m1.2.2.3" xref="footnote16.m1.2.2.3.cmml">=</mo><mrow id="footnote16.m1.2.2.2" xref="footnote16.m1.2.2.2.cmml"><mrow id="footnote16.m1.1.1.1.1" xref="footnote16.m1.1.1.1.1.cmml"><mi id="footnote16.m1.1.1.1.1.3" xref="footnote16.m1.1.1.1.1.3.cmml">σ</mi><mo lspace="0em" rspace="0em" id="footnote16.m1.1.1.1.1.2" xref="footnote16.m1.1.1.1.1.2.cmml">​</mo><mrow id="footnote16.m1.1.1.1.1.1.1" xref="footnote16.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="footnote16.m1.1.1.1.1.1.1.2" xref="footnote16.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="footnote16.m1.1.1.1.1.1.1.1" xref="footnote16.m1.1.1.1.1.1.1.1.cmml"><mi id="footnote16.m1.1.1.1.1.1.1.1.2" xref="footnote16.m1.1.1.1.1.1.1.1.2.cmml">θ</mi><mo lspace="0em" rspace="0em" id="footnote16.m1.1.1.1.1.1.1.1.1" xref="footnote16.m1.1.1.1.1.1.1.1.1.cmml">​</mo><mi id="footnote16.m1.1.1.1.1.1.1.1.3" xref="footnote16.m1.1.1.1.1.1.1.1.3.cmml">x</mi></mrow><mo rspace="0.055em" stretchy="false" id="footnote16.m1.1.1.1.1.1.1.3" xref="footnote16.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo rspace="0.222em" id="footnote16.m1.2.2.2.3" xref="footnote16.m1.2.2.2.3.cmml">⊙</mo><mrow id="footnote16.m1.2.2.2.2.1" xref="footnote16.m1.2.2.2.2.1.1.cmml"><mo stretchy="false" id="footnote16.m1.2.2.2.2.1.2" xref="footnote16.m1.2.2.2.2.1.1.cmml">(</mo><mrow id="footnote16.m1.2.2.2.2.1.1" xref="footnote16.m1.2.2.2.2.1.1.cmml"><msup id="footnote16.m1.2.2.2.2.1.1.2" xref="footnote16.m1.2.2.2.2.1.1.2.cmml"><mi id="footnote16.m1.2.2.2.2.1.1.2.2" xref="footnote16.m1.2.2.2.2.1.1.2.2.cmml">θ</mi><mo id="footnote16.m1.2.2.2.2.1.1.2.3" xref="footnote16.m1.2.2.2.2.1.1.2.3.cmml">′</mo></msup><mo lspace="0em" rspace="0em" id="footnote16.m1.2.2.2.2.1.1.1" xref="footnote16.m1.2.2.2.2.1.1.1.cmml">​</mo><mi id="footnote16.m1.2.2.2.2.1.1.3" xref="footnote16.m1.2.2.2.2.1.1.3.cmml">x</mi></mrow><mo stretchy="false" id="footnote16.m1.2.2.2.2.1.3" xref="footnote16.m1.2.2.2.2.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="footnote16.m1.2c"><apply id="footnote16.m1.2.2.cmml" xref="footnote16.m1.2.2"><eq id="footnote16.m1.2.2.3.cmml" xref="footnote16.m1.2.2.3"></eq><ci id="footnote16.m1.2.2.4.cmml" xref="footnote16.m1.2.2.4">𝑧</ci><apply id="footnote16.m1.2.2.2.cmml" xref="footnote16.m1.2.2.2"><csymbol cd="latexml" id="footnote16.m1.2.2.2.3.cmml" xref="footnote16.m1.2.2.2.3">direct-product</csymbol><apply id="footnote16.m1.1.1.1.1.cmml" xref="footnote16.m1.1.1.1.1"><times id="footnote16.m1.1.1.1.1.2.cmml" xref="footnote16.m1.1.1.1.1.2"></times><ci id="footnote16.m1.1.1.1.1.3.cmml" xref="footnote16.m1.1.1.1.1.3">𝜎</ci><apply id="footnote16.m1.1.1.1.1.1.1.1.cmml" xref="footnote16.m1.1.1.1.1.1.1"><times id="footnote16.m1.1.1.1.1.1.1.1.1.cmml" xref="footnote16.m1.1.1.1.1.1.1.1.1"></times><ci id="footnote16.m1.1.1.1.1.1.1.1.2.cmml" xref="footnote16.m1.1.1.1.1.1.1.1.2">𝜃</ci><ci id="footnote16.m1.1.1.1.1.1.1.1.3.cmml" xref="footnote16.m1.1.1.1.1.1.1.1.3">𝑥</ci></apply></apply><apply id="footnote16.m1.2.2.2.2.1.1.cmml" xref="footnote16.m1.2.2.2.2.1"><times id="footnote16.m1.2.2.2.2.1.1.1.cmml" xref="footnote16.m1.2.2.2.2.1.1.1"></times><apply id="footnote16.m1.2.2.2.2.1.1.2.cmml" xref="footnote16.m1.2.2.2.2.1.1.2"><csymbol cd="ambiguous" id="footnote16.m1.2.2.2.2.1.1.2.1.cmml" xref="footnote16.m1.2.2.2.2.1.1.2">superscript</csymbol><ci id="footnote16.m1.2.2.2.2.1.1.2.2.cmml" xref="footnote16.m1.2.2.2.2.1.1.2.2">𝜃</ci><ci id="footnote16.m1.2.2.2.2.1.1.2.3.cmml" xref="footnote16.m1.2.2.2.2.1.1.2.3">′</ci></apply><ci id="footnote16.m1.2.2.2.2.1.1.3.cmml" xref="footnote16.m1.2.2.2.2.1.1.3">𝑥</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote16.m1.2d">z=\sigma(\theta x)\odot(\theta^{\prime}x)</annotation></semantics></math>, where <math id="footnote16.m2.1" class="ltx_Math" alttext="x" display="inline"><semantics id="footnote16.m2.1b"><mi id="footnote16.m2.1.1" xref="footnote16.m2.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="footnote16.m2.1c"><ci id="footnote16.m2.1.1.cmml" xref="footnote16.m2.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote16.m2.1d">x</annotation></semantics></math> is the input, <math id="footnote16.m3.1" class="ltx_Math" alttext="z" display="inline"><semantics id="footnote16.m3.1b"><mi id="footnote16.m3.1.1" xref="footnote16.m3.1.1.cmml">z</mi><annotation-xml encoding="MathML-Content" id="footnote16.m3.1c"><ci id="footnote16.m3.1.1.cmml" xref="footnote16.m3.1.1">𝑧</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote16.m3.1d">z</annotation></semantics></math> is the output, <math id="footnote16.m4.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="footnote16.m4.1b"><mi id="footnote16.m4.1.1" xref="footnote16.m4.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="footnote16.m4.1c"><ci id="footnote16.m4.1.1.cmml" xref="footnote16.m4.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote16.m4.1d">\theta</annotation></semantics></math> and <math id="footnote16.m5.1" class="ltx_Math" alttext="\theta^{\prime}" display="inline"><semantics id="footnote16.m5.1b"><msup id="footnote16.m5.1.1" xref="footnote16.m5.1.1.cmml"><mi id="footnote16.m5.1.1.2" xref="footnote16.m5.1.1.2.cmml">θ</mi><mo id="footnote16.m5.1.1.3" xref="footnote16.m5.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="footnote16.m5.1c"><apply id="footnote16.m5.1.1.cmml" xref="footnote16.m5.1.1"><csymbol cd="ambiguous" id="footnote16.m5.1.1.1.cmml" xref="footnote16.m5.1.1">superscript</csymbol><ci id="footnote16.m5.1.1.2.cmml" xref="footnote16.m5.1.1.2">𝜃</ci><ci id="footnote16.m5.1.1.3.cmml" xref="footnote16.m5.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote16.m5.1d">\theta^{\prime}</annotation></semantics></math> are learnable weight matrices, <math id="footnote16.m6.1" class="ltx_Math" alttext="\odot" display="inline"><semantics id="footnote16.m6.1b"><mo id="footnote16.m6.1.1" xref="footnote16.m6.1.1.cmml">⊙</mo><annotation-xml encoding="MathML-Content" id="footnote16.m6.1c"><csymbol cd="latexml" id="footnote16.m6.1.1.cmml" xref="footnote16.m6.1.1">direct-product</csymbol></annotation-xml><annotation encoding="application/x-tex" id="footnote16.m6.1d">\odot</annotation></semantics></math> is element-wise multiplication, and <math id="footnote16.m7.1" class="ltx_Math" alttext="\sigma" display="inline"><semantics id="footnote16.m7.1b"><mi id="footnote16.m7.1.1" xref="footnote16.m7.1.1.cmml">σ</mi><annotation-xml encoding="MathML-Content" id="footnote16.m7.1c"><ci id="footnote16.m7.1.1.cmml" xref="footnote16.m7.1.1">𝜎</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote16.m7.1d">\sigma</annotation></semantics></math> is the sigmoid function.
A well known interpretation is to view <math id="footnote16.m8.1" class="ltx_Math" alttext="W=\texttt{diag}(\theta^{\prime}x)" display="inline"><semantics id="footnote16.m8.1b"><mrow id="footnote16.m8.1.1" xref="footnote16.m8.1.1.cmml"><mi id="footnote16.m8.1.1.3" xref="footnote16.m8.1.1.3.cmml">W</mi><mo id="footnote16.m8.1.1.2" xref="footnote16.m8.1.1.2.cmml">=</mo><mrow id="footnote16.m8.1.1.1" xref="footnote16.m8.1.1.1.cmml"><mtext class="ltx_mathvariant_monospace" id="footnote16.m8.1.1.1.3" xref="footnote16.m8.1.1.1.3a.cmml">diag</mtext><mo lspace="0em" rspace="0em" id="footnote16.m8.1.1.1.2" xref="footnote16.m8.1.1.1.2.cmml">​</mo><mrow id="footnote16.m8.1.1.1.1.1" xref="footnote16.m8.1.1.1.1.1.1.cmml"><mo stretchy="false" id="footnote16.m8.1.1.1.1.1.2" xref="footnote16.m8.1.1.1.1.1.1.cmml">(</mo><mrow id="footnote16.m8.1.1.1.1.1.1" xref="footnote16.m8.1.1.1.1.1.1.cmml"><msup id="footnote16.m8.1.1.1.1.1.1.2" xref="footnote16.m8.1.1.1.1.1.1.2.cmml"><mi id="footnote16.m8.1.1.1.1.1.1.2.2" xref="footnote16.m8.1.1.1.1.1.1.2.2.cmml">θ</mi><mo id="footnote16.m8.1.1.1.1.1.1.2.3" xref="footnote16.m8.1.1.1.1.1.1.2.3.cmml">′</mo></msup><mo lspace="0em" rspace="0em" id="footnote16.m8.1.1.1.1.1.1.1" xref="footnote16.m8.1.1.1.1.1.1.1.cmml">​</mo><mi id="footnote16.m8.1.1.1.1.1.1.3" xref="footnote16.m8.1.1.1.1.1.1.3.cmml">x</mi></mrow><mo stretchy="false" id="footnote16.m8.1.1.1.1.1.3" xref="footnote16.m8.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="footnote16.m8.1c"><apply id="footnote16.m8.1.1.cmml" xref="footnote16.m8.1.1"><eq id="footnote16.m8.1.1.2.cmml" xref="footnote16.m8.1.1.2"></eq><ci id="footnote16.m8.1.1.3.cmml" xref="footnote16.m8.1.1.3">𝑊</ci><apply id="footnote16.m8.1.1.1.cmml" xref="footnote16.m8.1.1.1"><times id="footnote16.m8.1.1.1.2.cmml" xref="footnote16.m8.1.1.1.2"></times><ci id="footnote16.m8.1.1.1.3a.cmml" xref="footnote16.m8.1.1.1.3"><mtext class="ltx_mathvariant_monospace" id="footnote16.m8.1.1.1.3.cmml" xref="footnote16.m8.1.1.1.3">diag</mtext></ci><apply id="footnote16.m8.1.1.1.1.1.1.cmml" xref="footnote16.m8.1.1.1.1.1"><times id="footnote16.m8.1.1.1.1.1.1.1.cmml" xref="footnote16.m8.1.1.1.1.1.1.1"></times><apply id="footnote16.m8.1.1.1.1.1.1.2.cmml" xref="footnote16.m8.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="footnote16.m8.1.1.1.1.1.1.2.1.cmml" xref="footnote16.m8.1.1.1.1.1.1.2">superscript</csymbol><ci id="footnote16.m8.1.1.1.1.1.1.2.2.cmml" xref="footnote16.m8.1.1.1.1.1.1.2.2">𝜃</ci><ci id="footnote16.m8.1.1.1.1.1.1.2.3.cmml" xref="footnote16.m8.1.1.1.1.1.1.2.3">′</ci></apply><ci id="footnote16.m8.1.1.1.1.1.1.3.cmml" xref="footnote16.m8.1.1.1.1.1.1.3">𝑥</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote16.m8.1d">W=\texttt{diag}(\theta^{\prime}x)</annotation></semantics></math> as the fast weights controlled by slow weights <math id="footnote16.m9.1" class="ltx_Math" alttext="\theta^{\prime}" display="inline"><semantics id="footnote16.m9.1b"><msup id="footnote16.m9.1.1" xref="footnote16.m9.1.1.cmml"><mi id="footnote16.m9.1.1.2" xref="footnote16.m9.1.1.2.cmml">θ</mi><mo id="footnote16.m9.1.1.3" xref="footnote16.m9.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="footnote16.m9.1c"><apply id="footnote16.m9.1.1.cmml" xref="footnote16.m9.1.1"><csymbol cd="ambiguous" id="footnote16.m9.1.1.1.cmml" xref="footnote16.m9.1.1">superscript</csymbol><ci id="footnote16.m9.1.1.2.cmml" xref="footnote16.m9.1.1.2">𝜃</ci><ci id="footnote16.m9.1.1.3.cmml" xref="footnote16.m9.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote16.m9.1d">\theta^{\prime}</annotation></semantics></math>, then equivalently, <math id="footnote16.m10.1" class="ltx_Math" alttext="z=W\sigma(\theta x)" display="inline"><semantics id="footnote16.m10.1b"><mrow id="footnote16.m10.1.1" xref="footnote16.m10.1.1.cmml"><mi id="footnote16.m10.1.1.3" xref="footnote16.m10.1.1.3.cmml">z</mi><mo id="footnote16.m10.1.1.2" xref="footnote16.m10.1.1.2.cmml">=</mo><mrow id="footnote16.m10.1.1.1" xref="footnote16.m10.1.1.1.cmml"><mi id="footnote16.m10.1.1.1.3" xref="footnote16.m10.1.1.1.3.cmml">W</mi><mo lspace="0em" rspace="0em" id="footnote16.m10.1.1.1.2" xref="footnote16.m10.1.1.1.2.cmml">​</mo><mi id="footnote16.m10.1.1.1.4" xref="footnote16.m10.1.1.1.4.cmml">σ</mi><mo lspace="0em" rspace="0em" id="footnote16.m10.1.1.1.2b" xref="footnote16.m10.1.1.1.2.cmml">​</mo><mrow id="footnote16.m10.1.1.1.1.1" xref="footnote16.m10.1.1.1.1.1.1.cmml"><mo stretchy="false" id="footnote16.m10.1.1.1.1.1.2" xref="footnote16.m10.1.1.1.1.1.1.cmml">(</mo><mrow id="footnote16.m10.1.1.1.1.1.1" xref="footnote16.m10.1.1.1.1.1.1.cmml"><mi id="footnote16.m10.1.1.1.1.1.1.2" xref="footnote16.m10.1.1.1.1.1.1.2.cmml">θ</mi><mo lspace="0em" rspace="0em" id="footnote16.m10.1.1.1.1.1.1.1" xref="footnote16.m10.1.1.1.1.1.1.1.cmml">​</mo><mi id="footnote16.m10.1.1.1.1.1.1.3" xref="footnote16.m10.1.1.1.1.1.1.3.cmml">x</mi></mrow><mo stretchy="false" id="footnote16.m10.1.1.1.1.1.3" xref="footnote16.m10.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="footnote16.m10.1c"><apply id="footnote16.m10.1.1.cmml" xref="footnote16.m10.1.1"><eq id="footnote16.m10.1.1.2.cmml" xref="footnote16.m10.1.1.2"></eq><ci id="footnote16.m10.1.1.3.cmml" xref="footnote16.m10.1.1.3">𝑧</ci><apply id="footnote16.m10.1.1.1.cmml" xref="footnote16.m10.1.1.1"><times id="footnote16.m10.1.1.1.2.cmml" xref="footnote16.m10.1.1.1.2"></times><ci id="footnote16.m10.1.1.1.3.cmml" xref="footnote16.m10.1.1.1.3">𝑊</ci><ci id="footnote16.m10.1.1.1.4.cmml" xref="footnote16.m10.1.1.1.4">𝜎</ci><apply id="footnote16.m10.1.1.1.1.1.1.cmml" xref="footnote16.m10.1.1.1.1.1"><times id="footnote16.m10.1.1.1.1.1.1.1.cmml" xref="footnote16.m10.1.1.1.1.1.1.1"></times><ci id="footnote16.m10.1.1.1.1.1.1.2.cmml" xref="footnote16.m10.1.1.1.1.1.1.2">𝜃</ci><ci id="footnote16.m10.1.1.1.1.1.1.3.cmml" xref="footnote16.m10.1.1.1.1.1.1.3">𝑥</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote16.m10.1d">z=W\sigma(\theta x)</annotation></semantics></math> is simply a two-layer MLP with fast weights <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib26" title="" class="ltx_ref">26</a>]</cite>.
</span></span></span>.
Recent work has been experimenting with FWPs for language modeling: Irie et al. <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib37" title="" class="ltx_ref">37</a>]</cite> design “fast” networks with weights produced as output of a “slow” networks.
Clark et al. <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite> give a Transformer a final layer of fast weights, whose initialization is trained as slow weights.
Our contribution relative to existing work on FWPs, again, is formulating an explicit learning problem for the update, which enables us to borrow tools from learning such as mini-batch and LN.</p>
</div>
</section>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Learning to Learn</h3>

<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p">For decades, researchers have been arguing that learning to learn, also known as meta-learning or bi-level optimization, should be a critical component of intelligence <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib61" title="" class="ltx_ref">61</a>, <a href="#bib.bib6" title="" class="ltx_ref">6</a>, <a href="#bib.bib70" title="" class="ltx_ref">70</a>, <a href="#bib.bib47" title="" class="ltx_ref">47</a>]</cite>.
In prior work such as <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib2" title="" class="ltx_ref">2</a>]</cite>, <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib20" title="" class="ltx_ref">20</a>]</cite> and <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib52" title="" class="ltx_ref">52</a>]</cite>, the inner loop learns from an entire dataset at a time instead of a sequence, so the outer loop needs a collection of datasets or tasks.
In short, the outer loop is “one level above” regular training.
Since it is hard to collect millions of datasets, this outer loop is hard to scale.</p>
</div>
<div id="S4.SS3.p2" class="ltx_para">
<p id="S4.SS3.p2.1" class="ltx_p">In contrast, for TTT, each sequence itself is a dataset and defines its own generalization problem. The inner loop is “one level below” regular training, so our outer loop is only another solution to the canonical problem of supervised learning,
instead of a new problem setting like generalization across datasets.
As illustrated in Table <a href="#S4.T2" title="Table 2 ‣ 4.3 Learning to Learn ‣ 4 Related Work ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, our outer loop is “at the same level” as regular training.
This makes our outer loop easier to scale.</p>
</div>
<figure id="S4.T2" class="ltx_table">
<div id="S4.T2.10" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:362.6pt;height:113.4pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-20.1pt,6.3pt) scale(0.9,0.9) ;">
<table id="S4.T2.10.10" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T2.10.10.11.1" class="ltx_tr">
<td id="S4.T2.10.10.11.1.1" class="ltx_td ltx_border_l ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;"></td>
<td id="S4.T2.10.10.11.1.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;"><span id="S4.T2.10.10.11.1.2.1" class="ltx_text ltx_font_bold">Inner loop</span></td>
<td id="S4.T2.10.10.11.1.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;"><span id="S4.T2.10.10.11.1.3.1" class="ltx_text ltx_font_bold">Outer loop</span></td>
<td id="S4.T2.10.10.11.1.4" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;"><span id="S4.T2.10.10.11.1.4.1" class="ltx_text ltx_font_bold">Subsection</span></td>
</tr>
<tr id="S4.T2.2.2.2" class="ltx_tr">
<td id="S4.T2.2.2.2.3" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;"><span id="S4.T2.2.2.2.3.1" class="ltx_text ltx_font_bold">Piece of data</span></td>
<td id="S4.T2.1.1.1.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;">Token <math id="S4.T2.1.1.1.1.m1.1" class="ltx_Math" alttext="x_{t}" display="inline"><semantics id="S4.T2.1.1.1.1.m1.1a"><msub id="S4.T2.1.1.1.1.m1.1.1" xref="S4.T2.1.1.1.1.m1.1.1.cmml"><mi id="S4.T2.1.1.1.1.m1.1.1.2" xref="S4.T2.1.1.1.1.m1.1.1.2.cmml">x</mi><mi id="S4.T2.1.1.1.1.m1.1.1.3" xref="S4.T2.1.1.1.1.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S4.T2.1.1.1.1.m1.1b"><apply id="S4.T2.1.1.1.1.m1.1.1.cmml" xref="S4.T2.1.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T2.1.1.1.1.m1.1.1.1.cmml" xref="S4.T2.1.1.1.1.m1.1.1">subscript</csymbol><ci id="S4.T2.1.1.1.1.m1.1.1.2.cmml" xref="S4.T2.1.1.1.1.m1.1.1.2">𝑥</ci><ci id="S4.T2.1.1.1.1.m1.1.1.3.cmml" xref="S4.T2.1.1.1.1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.1.1.1.1.m1.1c">x_{t}</annotation></semantics></math>
</td>
<td id="S4.T2.2.2.2.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="background-color:#FCF8ED;padding-top:4pt;padding-bottom:4pt;"><span id="S4.T2.2.2.2.2.1" class="ltx_text" style="background-color:#FCF8ED;">Sequence <math id="S4.T2.2.2.2.2.1.m1.3" class="ltx_Math" alttext="x_{1},\dots,x_{T}" display="inline"><semantics id="S4.T2.2.2.2.2.1.m1.3a"><mrow id="S4.T2.2.2.2.2.1.m1.3.3.2" xref="S4.T2.2.2.2.2.1.m1.3.3.3.cmml"><msub id="S4.T2.2.2.2.2.1.m1.2.2.1.1" xref="S4.T2.2.2.2.2.1.m1.2.2.1.1.cmml"><mi mathbackground="#FCF8ED" id="S4.T2.2.2.2.2.1.m1.2.2.1.1.2" xref="S4.T2.2.2.2.2.1.m1.2.2.1.1.2.cmml">x</mi><mn mathbackground="#FCF8ED" id="S4.T2.2.2.2.2.1.m1.2.2.1.1.3" xref="S4.T2.2.2.2.2.1.m1.2.2.1.1.3.cmml">1</mn></msub><mo mathbackground="#FCF8ED" id="S4.T2.2.2.2.2.1.m1.3.3.2.3" xref="S4.T2.2.2.2.2.1.m1.3.3.3.cmml">,</mo><mi mathbackground="#FCF8ED" mathvariant="normal" id="S4.T2.2.2.2.2.1.m1.1.1" xref="S4.T2.2.2.2.2.1.m1.1.1.cmml">…</mi><mo mathbackground="#FCF8ED" id="S4.T2.2.2.2.2.1.m1.3.3.2.4" xref="S4.T2.2.2.2.2.1.m1.3.3.3.cmml">,</mo><msub id="S4.T2.2.2.2.2.1.m1.3.3.2.2" xref="S4.T2.2.2.2.2.1.m1.3.3.2.2.cmml"><mi mathbackground="#FCF8ED" id="S4.T2.2.2.2.2.1.m1.3.3.2.2.2" xref="S4.T2.2.2.2.2.1.m1.3.3.2.2.2.cmml">x</mi><mi mathbackground="#FCF8ED" id="S4.T2.2.2.2.2.1.m1.3.3.2.2.3" xref="S4.T2.2.2.2.2.1.m1.3.3.2.2.3.cmml">T</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.2.2.2.2.1.m1.3b"><list id="S4.T2.2.2.2.2.1.m1.3.3.3.cmml" xref="S4.T2.2.2.2.2.1.m1.3.3.2"><apply id="S4.T2.2.2.2.2.1.m1.2.2.1.1.cmml" xref="S4.T2.2.2.2.2.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S4.T2.2.2.2.2.1.m1.2.2.1.1.1.cmml" xref="S4.T2.2.2.2.2.1.m1.2.2.1.1">subscript</csymbol><ci id="S4.T2.2.2.2.2.1.m1.2.2.1.1.2.cmml" xref="S4.T2.2.2.2.2.1.m1.2.2.1.1.2">𝑥</ci><cn type="integer" id="S4.T2.2.2.2.2.1.m1.2.2.1.1.3.cmml" xref="S4.T2.2.2.2.2.1.m1.2.2.1.1.3">1</cn></apply><ci id="S4.T2.2.2.2.2.1.m1.1.1.cmml" xref="S4.T2.2.2.2.2.1.m1.1.1">…</ci><apply id="S4.T2.2.2.2.2.1.m1.3.3.2.2.cmml" xref="S4.T2.2.2.2.2.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S4.T2.2.2.2.2.1.m1.3.3.2.2.1.cmml" xref="S4.T2.2.2.2.2.1.m1.3.3.2.2">subscript</csymbol><ci id="S4.T2.2.2.2.2.1.m1.3.3.2.2.2.cmml" xref="S4.T2.2.2.2.2.1.m1.3.3.2.2.2">𝑥</ci><ci id="S4.T2.2.2.2.2.1.m1.3.3.2.2.3.cmml" xref="S4.T2.2.2.2.2.1.m1.3.3.2.2.3">𝑇</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.2.2.2.2.1.m1.3c">x_{1},\dots,x_{T}</annotation></semantics></math></span></td>
<td id="S4.T2.2.2.2.4" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;" rowspan="4"><span id="S4.T2.2.2.2.4.1" class="ltx_text"><a href="#S2.SS1" title="2.1 TTT as updating a hidden state ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.1</span></a>, <a href="#S2.SS2" title="2.2 Training a network with TTT layers ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.2</span></a></span></td>
</tr>
<tr id="S4.T2.3.3.3" class="ltx_tr">
<td id="S4.T2.3.3.3.2" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;"><span id="S4.T2.3.3.3.2.1" class="ltx_text ltx_font_bold">Training set</span></td>
<td id="S4.T2.3.3.3.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;">Sequence <math id="S4.T2.3.3.3.1.m1.3" class="ltx_Math" alttext="x_{1},\dots,x_{T}" display="inline"><semantics id="S4.T2.3.3.3.1.m1.3a"><mrow id="S4.T2.3.3.3.1.m1.3.3.2" xref="S4.T2.3.3.3.1.m1.3.3.3.cmml"><msub id="S4.T2.3.3.3.1.m1.2.2.1.1" xref="S4.T2.3.3.3.1.m1.2.2.1.1.cmml"><mi id="S4.T2.3.3.3.1.m1.2.2.1.1.2" xref="S4.T2.3.3.3.1.m1.2.2.1.1.2.cmml">x</mi><mn id="S4.T2.3.3.3.1.m1.2.2.1.1.3" xref="S4.T2.3.3.3.1.m1.2.2.1.1.3.cmml">1</mn></msub><mo id="S4.T2.3.3.3.1.m1.3.3.2.3" xref="S4.T2.3.3.3.1.m1.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S4.T2.3.3.3.1.m1.1.1" xref="S4.T2.3.3.3.1.m1.1.1.cmml">…</mi><mo id="S4.T2.3.3.3.1.m1.3.3.2.4" xref="S4.T2.3.3.3.1.m1.3.3.3.cmml">,</mo><msub id="S4.T2.3.3.3.1.m1.3.3.2.2" xref="S4.T2.3.3.3.1.m1.3.3.2.2.cmml"><mi id="S4.T2.3.3.3.1.m1.3.3.2.2.2" xref="S4.T2.3.3.3.1.m1.3.3.2.2.2.cmml">x</mi><mi id="S4.T2.3.3.3.1.m1.3.3.2.2.3" xref="S4.T2.3.3.3.1.m1.3.3.2.2.3.cmml">T</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.3.3.3.1.m1.3b"><list id="S4.T2.3.3.3.1.m1.3.3.3.cmml" xref="S4.T2.3.3.3.1.m1.3.3.2"><apply id="S4.T2.3.3.3.1.m1.2.2.1.1.cmml" xref="S4.T2.3.3.3.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S4.T2.3.3.3.1.m1.2.2.1.1.1.cmml" xref="S4.T2.3.3.3.1.m1.2.2.1.1">subscript</csymbol><ci id="S4.T2.3.3.3.1.m1.2.2.1.1.2.cmml" xref="S4.T2.3.3.3.1.m1.2.2.1.1.2">𝑥</ci><cn type="integer" id="S4.T2.3.3.3.1.m1.2.2.1.1.3.cmml" xref="S4.T2.3.3.3.1.m1.2.2.1.1.3">1</cn></apply><ci id="S4.T2.3.3.3.1.m1.1.1.cmml" xref="S4.T2.3.3.3.1.m1.1.1">…</ci><apply id="S4.T2.3.3.3.1.m1.3.3.2.2.cmml" xref="S4.T2.3.3.3.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S4.T2.3.3.3.1.m1.3.3.2.2.1.cmml" xref="S4.T2.3.3.3.1.m1.3.3.2.2">subscript</csymbol><ci id="S4.T2.3.3.3.1.m1.3.3.2.2.2.cmml" xref="S4.T2.3.3.3.1.m1.3.3.2.2.2">𝑥</ci><ci id="S4.T2.3.3.3.1.m1.3.3.2.2.3.cmml" xref="S4.T2.3.3.3.1.m1.3.3.2.2.3">𝑇</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.3.3.3.1.m1.3c">x_{1},\dots,x_{T}</annotation></semantics></math>
</td>
<td id="S4.T2.3.3.3.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="background-color:#FCF8ED;padding-top:4pt;padding-bottom:4pt;"><span id="S4.T2.3.3.3.3.1" class="ltx_text" style="background-color:#FCF8ED;">Dataset of sequences, e.g., Books</span></td>
</tr>
<tr id="S4.T2.4.4.4" class="ltx_tr">
<td id="S4.T2.4.4.4.2" class="ltx_td ltx_align_left ltx_border_l ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;"><span id="S4.T2.4.4.4.2.1" class="ltx_text ltx_font_bold">Objective</span></td>
<td id="S4.T2.4.4.4.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;">Reconstruction (loss <math id="S4.T2.4.4.4.1.m1.1" class="ltx_math_unparsed" alttext="\ell)" display="inline"><semantics id="S4.T2.4.4.4.1.m1.1a"><mrow id="S4.T2.4.4.4.1.m1.1b"><mi mathvariant="normal" id="S4.T2.4.4.4.1.m1.1.1">ℓ</mi><mo stretchy="false" id="S4.T2.4.4.4.1.m1.1.2">)</mo></mrow><annotation encoding="application/x-tex" id="S4.T2.4.4.4.1.m1.1c">\ell)</annotation></semantics></math>
</td>
<td id="S4.T2.4.4.4.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="background-color:#FCF8ED;padding-top:4pt;padding-bottom:4pt;"><span id="S4.T2.4.4.4.3.1" class="ltx_text" style="background-color:#FCF8ED;">Next-token prediction</span></td>
</tr>
<tr id="S4.T2.7.7.7" class="ltx_tr">
<td id="S4.T2.7.7.7.4" class="ltx_td ltx_align_left ltx_border_b ltx_border_l ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;" rowspan="3"><span id="S4.T2.7.7.7.4.1" class="ltx_text ltx_font_bold">Parameters</span></td>
<td id="S4.T2.6.6.6.2" class="ltx_td ltx_align_left ltx_border_b ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;" rowspan="3"><span id="S4.T2.6.6.6.2.2" class="ltx_text"><math id="S4.T2.5.5.5.1.1.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S4.T2.5.5.5.1.1.m1.1a"><mi id="S4.T2.5.5.5.1.1.m1.1.1" xref="S4.T2.5.5.5.1.1.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S4.T2.5.5.5.1.1.m1.1b"><ci id="S4.T2.5.5.5.1.1.m1.1.1.cmml" xref="S4.T2.5.5.5.1.1.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.5.5.5.1.1.m1.1c">W</annotation></semantics></math> (weights of <math id="S4.T2.6.6.6.2.2.m2.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S4.T2.6.6.6.2.2.m2.1a"><mi id="S4.T2.6.6.6.2.2.m2.1.1" xref="S4.T2.6.6.6.2.2.m2.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S4.T2.6.6.6.2.2.m2.1b"><ci id="S4.T2.6.6.6.2.2.m2.1.1.cmml" xref="S4.T2.6.6.6.2.2.m2.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.6.6.6.2.2.m2.1c">f</annotation></semantics></math>)</span></td>
<td id="S4.T2.7.7.7.3" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="background-color:#FCF8ED;padding-top:4pt;padding-bottom:4pt;">
<math id="S4.T2.7.7.7.3.m1.1" class="ltx_Math" style="background-color:#FCF8ED;" alttext="\theta_{\text{rest}}" display="inline"><semantics id="S4.T2.7.7.7.3.m1.1a"><msub id="S4.T2.7.7.7.3.m1.1.1" xref="S4.T2.7.7.7.3.m1.1.1.cmml"><mi mathbackground="#FCF8ED" id="S4.T2.7.7.7.3.m1.1.1.2" xref="S4.T2.7.7.7.3.m1.1.1.2.cmml">θ</mi><mtext mathbackground="#FCF8ED" id="S4.T2.7.7.7.3.m1.1.1.3" xref="S4.T2.7.7.7.3.m1.1.1.3a.cmml">rest</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.T2.7.7.7.3.m1.1b"><apply id="S4.T2.7.7.7.3.m1.1.1.cmml" xref="S4.T2.7.7.7.3.m1.1.1"><csymbol cd="ambiguous" id="S4.T2.7.7.7.3.m1.1.1.1.cmml" xref="S4.T2.7.7.7.3.m1.1.1">subscript</csymbol><ci id="S4.T2.7.7.7.3.m1.1.1.2.cmml" xref="S4.T2.7.7.7.3.m1.1.1.2">𝜃</ci><ci id="S4.T2.7.7.7.3.m1.1.1.3a.cmml" xref="S4.T2.7.7.7.3.m1.1.1.3"><mtext mathbackground="#FCF8ED" mathsize="70%" id="S4.T2.7.7.7.3.m1.1.1.3.cmml" xref="S4.T2.7.7.7.3.m1.1.1.3">rest</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.7.7.7.3.m1.1c">\theta_{\text{rest}}</annotation></semantics></math><span id="S4.T2.7.7.7.3.1" class="ltx_text" style="background-color:#FCF8ED;"> (rest of the network)</span>
</td>
</tr>
<tr id="S4.T2.8.8.8" class="ltx_tr">
<td id="S4.T2.8.8.8.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;">
<math id="S4.T2.8.8.8.1.m1.3" class="ltx_Math" alttext="\theta_{K},\theta_{Q},\theta_{V}" display="inline"><semantics id="S4.T2.8.8.8.1.m1.3a"><mrow id="S4.T2.8.8.8.1.m1.3.3.3" xref="S4.T2.8.8.8.1.m1.3.3.4.cmml"><msub id="S4.T2.8.8.8.1.m1.1.1.1.1" xref="S4.T2.8.8.8.1.m1.1.1.1.1.cmml"><mi id="S4.T2.8.8.8.1.m1.1.1.1.1.2" xref="S4.T2.8.8.8.1.m1.1.1.1.1.2.cmml">θ</mi><mi id="S4.T2.8.8.8.1.m1.1.1.1.1.3" xref="S4.T2.8.8.8.1.m1.1.1.1.1.3.cmml">K</mi></msub><mo id="S4.T2.8.8.8.1.m1.3.3.3.4" xref="S4.T2.8.8.8.1.m1.3.3.4.cmml">,</mo><msub id="S4.T2.8.8.8.1.m1.2.2.2.2" xref="S4.T2.8.8.8.1.m1.2.2.2.2.cmml"><mi id="S4.T2.8.8.8.1.m1.2.2.2.2.2" xref="S4.T2.8.8.8.1.m1.2.2.2.2.2.cmml">θ</mi><mi id="S4.T2.8.8.8.1.m1.2.2.2.2.3" xref="S4.T2.8.8.8.1.m1.2.2.2.2.3.cmml">Q</mi></msub><mo id="S4.T2.8.8.8.1.m1.3.3.3.5" xref="S4.T2.8.8.8.1.m1.3.3.4.cmml">,</mo><msub id="S4.T2.8.8.8.1.m1.3.3.3.3" xref="S4.T2.8.8.8.1.m1.3.3.3.3.cmml"><mi id="S4.T2.8.8.8.1.m1.3.3.3.3.2" xref="S4.T2.8.8.8.1.m1.3.3.3.3.2.cmml">θ</mi><mi id="S4.T2.8.8.8.1.m1.3.3.3.3.3" xref="S4.T2.8.8.8.1.m1.3.3.3.3.3.cmml">V</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.8.8.8.1.m1.3b"><list id="S4.T2.8.8.8.1.m1.3.3.4.cmml" xref="S4.T2.8.8.8.1.m1.3.3.3"><apply id="S4.T2.8.8.8.1.m1.1.1.1.1.cmml" xref="S4.T2.8.8.8.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S4.T2.8.8.8.1.m1.1.1.1.1.1.cmml" xref="S4.T2.8.8.8.1.m1.1.1.1.1">subscript</csymbol><ci id="S4.T2.8.8.8.1.m1.1.1.1.1.2.cmml" xref="S4.T2.8.8.8.1.m1.1.1.1.1.2">𝜃</ci><ci id="S4.T2.8.8.8.1.m1.1.1.1.1.3.cmml" xref="S4.T2.8.8.8.1.m1.1.1.1.1.3">𝐾</ci></apply><apply id="S4.T2.8.8.8.1.m1.2.2.2.2.cmml" xref="S4.T2.8.8.8.1.m1.2.2.2.2"><csymbol cd="ambiguous" id="S4.T2.8.8.8.1.m1.2.2.2.2.1.cmml" xref="S4.T2.8.8.8.1.m1.2.2.2.2">subscript</csymbol><ci id="S4.T2.8.8.8.1.m1.2.2.2.2.2.cmml" xref="S4.T2.8.8.8.1.m1.2.2.2.2.2">𝜃</ci><ci id="S4.T2.8.8.8.1.m1.2.2.2.2.3.cmml" xref="S4.T2.8.8.8.1.m1.2.2.2.2.3">𝑄</ci></apply><apply id="S4.T2.8.8.8.1.m1.3.3.3.3.cmml" xref="S4.T2.8.8.8.1.m1.3.3.3.3"><csymbol cd="ambiguous" id="S4.T2.8.8.8.1.m1.3.3.3.3.1.cmml" xref="S4.T2.8.8.8.1.m1.3.3.3.3">subscript</csymbol><ci id="S4.T2.8.8.8.1.m1.3.3.3.3.2.cmml" xref="S4.T2.8.8.8.1.m1.3.3.3.3.2">𝜃</ci><ci id="S4.T2.8.8.8.1.m1.3.3.3.3.3.cmml" xref="S4.T2.8.8.8.1.m1.3.3.3.3.3">𝑉</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.8.8.8.1.m1.3c">\theta_{K},\theta_{Q},\theta_{V}</annotation></semantics></math> (reconstruction views)</td>
<td id="S4.T2.8.8.8.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;"><a href="#S2.SS3" title="2.3 Learning a self-supervised task for TTT ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.3</span></a></td>
</tr>
<tr id="S4.T2.10.10.10" class="ltx_tr">
<td id="S4.T2.10.10.10.2" class="ltx_td ltx_align_left ltx_border_b ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;">
<math id="S4.T2.9.9.9.1.m1.1" class="ltx_Math" alttext="\theta_{\text{init}}" display="inline"><semantics id="S4.T2.9.9.9.1.m1.1a"><msub id="S4.T2.9.9.9.1.m1.1.1" xref="S4.T2.9.9.9.1.m1.1.1.cmml"><mi id="S4.T2.9.9.9.1.m1.1.1.2" xref="S4.T2.9.9.9.1.m1.1.1.2.cmml">θ</mi><mtext id="S4.T2.9.9.9.1.m1.1.1.3" xref="S4.T2.9.9.9.1.m1.1.1.3a.cmml">init</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.T2.9.9.9.1.m1.1b"><apply id="S4.T2.9.9.9.1.m1.1.1.cmml" xref="S4.T2.9.9.9.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T2.9.9.9.1.m1.1.1.1.cmml" xref="S4.T2.9.9.9.1.m1.1.1">subscript</csymbol><ci id="S4.T2.9.9.9.1.m1.1.1.2.cmml" xref="S4.T2.9.9.9.1.m1.1.1.2">𝜃</ci><ci id="S4.T2.9.9.9.1.m1.1.1.3a.cmml" xref="S4.T2.9.9.9.1.m1.1.1.3"><mtext mathsize="70%" id="S4.T2.9.9.9.1.m1.1.1.3.cmml" xref="S4.T2.9.9.9.1.m1.1.1.3">init</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.9.9.9.1.m1.1c">\theta_{\text{init}}</annotation></semantics></math> and <math id="S4.T2.10.10.10.2.m2.1" class="ltx_Math" alttext="\theta_{\text{lr}}" display="inline"><semantics id="S4.T2.10.10.10.2.m2.1a"><msub id="S4.T2.10.10.10.2.m2.1.1" xref="S4.T2.10.10.10.2.m2.1.1.cmml"><mi id="S4.T2.10.10.10.2.m2.1.1.2" xref="S4.T2.10.10.10.2.m2.1.1.2.cmml">θ</mi><mtext id="S4.T2.10.10.10.2.m2.1.1.3" xref="S4.T2.10.10.10.2.m2.1.1.3a.cmml">lr</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.T2.10.10.10.2.m2.1b"><apply id="S4.T2.10.10.10.2.m2.1.1.cmml" xref="S4.T2.10.10.10.2.m2.1.1"><csymbol cd="ambiguous" id="S4.T2.10.10.10.2.m2.1.1.1.cmml" xref="S4.T2.10.10.10.2.m2.1.1">subscript</csymbol><ci id="S4.T2.10.10.10.2.m2.1.1.2.cmml" xref="S4.T2.10.10.10.2.m2.1.1.2">𝜃</ci><ci id="S4.T2.10.10.10.2.m2.1.1.3a.cmml" xref="S4.T2.10.10.10.2.m2.1.1.3"><mtext mathsize="70%" id="S4.T2.10.10.10.2.m2.1.1.3.cmml" xref="S4.T2.10.10.10.2.m2.1.1.3">lr</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.10.10.10.2.m2.1c">\theta_{\text{lr}}</annotation></semantics></math>
</td>
<td id="S4.T2.10.10.10.3" class="ltx_td ltx_align_left ltx_border_b ltx_border_r ltx_border_t" style="padding-top:4pt;padding-bottom:4pt;"><a href="#S2.SS7" title="2.7 Implementation details ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.7</span></a></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>
In summary, our paper reformulates supervised learning as learning to learn, with two nested loops.
Highlighted rows of the outer loop are the same as in the regular training.
Parameters of the outer loop become hyper-parameters of the inner loop.
Intuitively, the inner loop, <span id="S4.T2.12.1" class="ltx_text ltx_font_italic">i.e.</span> TTT, is “one level below” regular training.
</figcaption>
</figure>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Discussion</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">We have reformulated the canonical problem of supervised learning as learning to (learn at test time).
Our formulation produces an alternative conceptual framework for building what is traditionally known as network architectures.
We summarize our current instantiation in Table <a href="#S4.T2" title="Table 2 ‣ 4.3 Learning to Learn ‣ 4 Related Work ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.</p>
</div>
<div id="S5.p2" class="ltx_para">
<p id="S5.p2.1" class="ltx_p">The search space for effective instantiations inside this framework is huge, and our paper has only taken a baby step.
Fortunately, if our perspective holds, then heuristics from regular training can transfer to test-time training, and search can be efficient.
Next we outline some especially promising directions for future work.</p>
<ul id="S5.I1" class="ltx_itemize">
<li id="S5.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i1.p1" class="ltx_para">
<p id="S5.I1.i1.p1.1" class="ltx_p"><span id="S5.I1.i1.p1.1.1" class="ltx_text ltx_font_bold">Outer-loop parameterization.</span>
There are many other ways to parameterize a family of multi-view reconstruction tasks, or perhaps a more general family of self-supervised tasks. It would be a big coincidence if the first one we have tried turns out to be the best.</p>
</div>
</li>
<li id="S5.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i2.p1" class="ltx_para">
<p id="S5.I1.i2.p1.1" class="ltx_p"><span id="S5.I1.i2.p1.1.1" class="ltx_text ltx_font_bold">Systems optimization.</span> Our systems optimization in Subsection <a href="#S3.SS3" title="3.3 Wall-clock time ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a> has been preliminary at best, and there are many ways to improve it.
In addition, pipeline parallelism through time might allow us to process long sequences of millions of tokens on multiple devices together.</p>
</div>
</li>
<li id="S5.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i3.p1" class="ltx_para">
<p id="S5.I1.i3.p1.1" class="ltx_p"><span id="S5.I1.i3.p1.1.1" class="ltx_text ltx_font_bold">Longer context and larger models.</span> Constrained by our academic resources, we have not trained with millions or billions in context length, which would also require larger models according to Figure <a href="#A3.F19" title="Figure 19 ‣ Gradient checkpointing through time. ‣ Appendix C Experiment details ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">19</span></a>.
The advantage of TTT layers should become more pronounced in longer context.</p>
</div>
</li>
<li id="S5.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i4.p1" class="ltx_para">
<p id="S5.I1.i4.p1.3" class="ltx_p"><span id="S5.I1.i4.p1.1.1" class="ltx_text ltx_font_bold">More ambitious instantiations of <math id="S5.I1.i4.p1.1.1.m1.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S5.I1.i4.p1.1.1.m1.1a"><mi id="S5.I1.i4.p1.1.1.m1.1.1" xref="S5.I1.i4.p1.1.1.m1.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S5.I1.i4.p1.1.1.m1.1b"><ci id="S5.I1.i4.p1.1.1.m1.1.1.cmml" xref="S5.I1.i4.p1.1.1.m1.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.I1.i4.p1.1.1.m1.1c">f</annotation></semantics></math>.</span>
When context length becomes longer, <math id="S5.I1.i4.p1.2.m1.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S5.I1.i4.p1.2.m1.1a"><mi id="S5.I1.i4.p1.2.m1.1.1" xref="S5.I1.i4.p1.2.m1.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S5.I1.i4.p1.2.m1.1b"><ci id="S5.I1.i4.p1.2.m1.1.1.cmml" xref="S5.I1.i4.p1.2.m1.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.I1.i4.p1.2.m1.1c">f</annotation></semantics></math> would also need to be larger.
For video tasks and embodied agents, whose context length can easily scale up to millions or billions, <math id="S5.I1.i4.p1.3.m2.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S5.I1.i4.p1.3.m2.1a"><mi id="S5.I1.i4.p1.3.m2.1.1" xref="S5.I1.i4.p1.3.m2.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S5.I1.i4.p1.3.m2.1b"><ci id="S5.I1.i4.p1.3.m2.1.1.cmml" xref="S5.I1.i4.p1.3.m2.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.I1.i4.p1.3.m2.1c">f</annotation></semantics></math> could be a convolutional neural network.</p>
</div>
</li>
<li id="S5.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S5.I1.i5.p1" class="ltx_para">
<p id="S5.I1.i5.p1.1" class="ltx_p"><span id="S5.I1.i5.p1.1.1" class="ltx_text ltx_font_bold">Multi-level learning to learn.</span> If <math id="S5.I1.i5.p1.1.m1.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S5.I1.i5.p1.1.m1.1a"><mi id="S5.I1.i5.p1.1.m1.1.1" xref="S5.I1.i5.p1.1.m1.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S5.I1.i5.p1.1.m1.1b"><ci id="S5.I1.i5.p1.1.m1.1.1.cmml" xref="S5.I1.i5.p1.1.m1.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.I1.i5.p1.1.m1.1c">f</annotation></semantics></math> itself is a self-attention layer, then by Theorem <a href="#Thmtheorem2" title="Theorem 2. ‣ 2.6 Theoretical equivalences ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> it can be interpreted as yet another inner loop nested inside the existing one. In this fashion, we can potentially build many levels of nested learning problems.</p>
</div>
</li>
</ul>
</div>
<div id="S5.p3" class="ltx_para">
<p id="S5.p3.1" class="ltx_p">Why do we study TTT? First a more basic question: Why study AI? For some of us, AI is a playground to probe about the nature of human intelligence.
Prior work often tries to model human learning with machine learning, where training is on a shuffled dataset with i.i.d. instances, and inference is on a separate test set.
However, humans do not naturally learn with i.i.d. instances or have a train-test split.
We believe that human learning has a more promising connection with TTT, our inner loop, whose data is a potentially very long sequence with strong temporal dependencies, and any piece of data can be used for both training and testing. This is why we study TTT.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Author Contributions</h2>

<div id="Sx1.p1" class="ltx_para">
<p id="Sx1.p1.1" class="ltx_p"><span id="Sx1.p1.1.1" class="ltx_text ltx_font_bold">Yu Sun</span> started this project with Xinhao Li in November 2022, and has been working on it full-time since June 2023.
Yu proposed the conceptual framework of the project, designed mini-batch TTT and the dual form, wrote the paper with help from others, and led the daily operations of the team.</p>
</div>
<div id="Sx1.p2" class="ltx_para">
<p id="Sx1.p2.1" class="ltx_p"><span id="Sx1.p2.1.1" class="ltx_text ltx_font_bold">Xinhao Li</span> started this project with Yu Sun in November 2022, and has been working on it full-time since then.
Xinhao and Karan co-led the development of our current codebase.
Before March 2024, Xinhao was the primary contributor to our earlier codebases that shaped this project.
Xinhao made significant contributions to the project direction in discussions.</p>
</div>
<div id="Sx1.p3" class="ltx_para">
<p id="Sx1.p3.1" class="ltx_p"><span id="Sx1.p3.1.1" class="ltx_text ltx_font_bold">Karan Dalal</span> joined this project full-time in June 2023.
In collaboration with Xinhao, he co-led the development of our current codebase.
Karan managed the experiments in Section <a href="#S3" title="3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, helped write the paper, and made significant contributions to the project direction in discussions.</p>
</div>
<div id="Sx1.p4" class="ltx_para">
<p id="Sx1.p4.1" class="ltx_p"><span id="Sx1.p4.1.1" class="ltx_text ltx_font_bold">Jiarui Xu</span> joined this project in March 2024. He led our architectural development since he joined, and made significant contributions to the project direction in discussions.</p>
</div>
<div id="Sx1.p5" class="ltx_para">
<p id="Sx1.p5.1" class="ltx_p"><span id="Sx1.p5.1.1" class="ltx_text ltx_font_bold">Arjun Vikram</span> joined this project in September 2023. He made significant contributions to our systems optimization, as well as current and earlier codebases.</p>
</div>
<div id="Sx1.p6" class="ltx_para">
<p id="Sx1.p6.1" class="ltx_p"><span id="Sx1.p6.1.1" class="ltx_text ltx_font_bold">Genghan Zhang</span> joined this project in January 2024. He provided critical insights and made significant improvements to our systems optimization.</p>
</div>
<div id="Sx1.p7" class="ltx_para">
<p id="Sx1.p7.1" class="ltx_p"><span id="Sx1.p7.1.1" class="ltx_text ltx_font_bold">Yann Dubois</span> joined this project in February 2024. He proposed our current instantiation of <math id="Sx1.p7.1.m1.1" class="ltx_Math" alttext="f" display="inline"><semantics id="Sx1.p7.1.m1.1a"><mi id="Sx1.p7.1.m1.1.1" xref="Sx1.p7.1.m1.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="Sx1.p7.1.m1.1b"><ci id="Sx1.p7.1.m1.1.1.cmml" xref="Sx1.p7.1.m1.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="Sx1.p7.1.m1.1c">f</annotation></semantics></math>, and made significant contributions to the project direction in discussions.</p>
</div>
<div id="Sx1.p8" class="ltx_para">
<p id="Sx1.p8.1" class="ltx_p"><span id="Sx1.p8.1.1" class="ltx_text ltx_font_bold">Xinlei Chen and Xiaolong Wang</span> have been supporting this project since November 2022, and the direction of test-time training for many years. Without their support in compute and organization, this project could not have survived its early stage.
They gave invaluable advice to our experiments.</p>
</div>
<div id="Sx1.p9" class="ltx_para">
<p id="Sx1.p9.1" class="ltx_p"><span id="Sx1.p9.1.1" class="ltx_text ltx_font_bold">Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin</span> have been supporting this project since May 2023. They gave invaluable advice to our experiments and presentation.
For example, Sanmi suggested us to focus on TTT-Linear,
Tatsu suggested the experiments in Figure <a href="#S0.F2" title="Figure 2 ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> (left), and Carlos outlined Section <a href="#S2" title="2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.</p>
</div>
</section>
<section id="Sx2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Acknowledgements</h2>

<div id="Sx2.p1" class="ltx_para">
<p id="Sx2.p1.1" class="ltx_p">Part of the compute for this project is generously supported by the Google TPU Research Cloud program.
XW is supported, in part, by the Amazon Research Award, the Cisco Faculty Award and the Qualcomm Innovation Fellowship.
SK acknowledges support by NSF 2046795 and 2205329, NIFA award 2020-67021-32799, the Alfred P. Sloan Foundation, and Google Inc.
TH is supported by a Sony Faculty Innovation Award and a gift from Panasonic.
CG acknowledges support by the Air Force Office of Scientific Research (AFOSR), FA9550-20-1-0427, Stanford Human-Centered Artificial Intelligence (HAI) Institute, and gifts from Google and IBM.</p>
</div>
<div id="Sx2.p2" class="ltx_para">
<p id="Sx2.p2.1" class="ltx_p">We would like to thank Rohan Taori, Xuechen Li, Allan Zhou, Ke Chen, and Guandao Yang for many helpful discussions,
Menghao Guo for help with code release,
Xinyang Geng for help with EasyLM,
Hao Liu for help with the LWM codebase,
David Hall for help with Levanter,
Yossi Gandelsman and Yutong Bai for help at an early stage of the project,
Mert Yuksekgonul for help with figures in the paper,
Horace He and Azalia Mirhoseini for help with systems,
Sharad Vikram and Roy Frostig for answering our questions about JAX and Pallas,
Albert Gu and Tri Dao for helping us reproduce experiments in the Mamba paper,
and Kilian Weinberger and Percy Liang for advice on presentation.
Yu Sun is grateful to his PhD advisors,
Alexei A. Efros and Moritz Hardt, for their many insights from years ago that eventually became part of this paper.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[1]</span>
<span class="ltx_bibblock">
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.

</span>
<span class="ltx_bibblock">Gpt-4 technical report.

</span>
<span class="ltx_bibblock"><span id="bib.bib1.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2303.08774</span>, 2023.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[2]</span>
<span class="ltx_bibblock">
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas.

</span>
<span class="ltx_bibblock">Learning to learn by gradient descent by gradient descent.

</span>
<span class="ltx_bibblock"><span id="bib.bib2.1.1" class="ltx_text ltx_font_italic">Advances in neural information processing systems</span>, 29, 2016.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[3]</span>
<span class="ltx_bibblock">
Authors Guild.

</span>
<span class="ltx_bibblock">You just found out your book was used to train ai. now what?, 2023.

</span>
<span class="ltx_bibblock">Accessed: 2024-06-24.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[4]</span>
<span class="ltx_bibblock">
Jimmy Ba, Geoffrey E Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin Ionescu.

</span>
<span class="ltx_bibblock">Using fast weights to attend to the recent past.

</span>
<span class="ltx_bibblock"><span id="bib.bib4.1.1" class="ltx_text ltx_font_italic">Advances in neural information processing systems</span>, 29, 2016.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[5]</span>
<span class="ltx_bibblock">
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter.

</span>
<span class="ltx_bibblock">xlstm: Extended long short-term memory.

</span>
<span class="ltx_bibblock"><span id="bib.bib5.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2405.04517</span>, 2024.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[6]</span>
<span class="ltx_bibblock">
Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier.

</span>
<span class="ltx_bibblock"><span id="bib.bib6.1.1" class="ltx_text ltx_font_italic">Learning a synaptic learning rule</span>.

</span>
<span class="ltx_bibblock">Citeseer, 1990.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[7]</span>
<span class="ltx_bibblock">
Hermanus Josephus Bierens.

</span>
<span class="ltx_bibblock">The nadaraya-watson kernel regression function estimator.

</span>
<span class="ltx_bibblock"><span id="bib.bib7.1.1" class="ltx_text ltx_font_italic">(Serie Research Memoranda; No. 1988-58). Faculty of Economics and Business Administration, Vrije Universiteit Amsterdam.</span>, 1988.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[8]</span>
<span class="ltx_bibblock">
Christopher M Bishop and Nasser M Nasrabadi.

</span>
<span class="ltx_bibblock"><span id="bib.bib8.1.1" class="ltx_text ltx_font_italic">Pattern recognition and machine learning</span>, volume 4.

</span>
<span class="ltx_bibblock">Springer, 2006.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[9]</span>
<span class="ltx_bibblock">
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al.

</span>
<span class="ltx_bibblock">Gpt-neox-20b: An open-source autoregressive language model.

</span>
<span class="ltx_bibblock"><span id="bib.bib9.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2204.06745</span>, 2022.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[10]</span>
<span class="ltx_bibblock">
Léon Bottou and Vladimir Vapnik.

</span>
<span class="ltx_bibblock">Local learning algorithms.

</span>
<span class="ltx_bibblock"><span id="bib.bib10.1.1" class="ltx_text ltx_font_italic">Neural computation</span>, 4(6):888–900, 1992.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[11]</span>
<span class="ltx_bibblock">
Leo Breiman, William Meisel, and Edward Purcell.

</span>
<span class="ltx_bibblock">Variable kernel estimates of multivariate densities.

</span>
<span class="ltx_bibblock"><span id="bib.bib11.1.1" class="ltx_text ltx_font_italic">Technometrics</span>, 19(2):135–144, 1977.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[12]</span>
<span class="ltx_bibblock">
Zongwu Cai.

</span>
<span class="ltx_bibblock">Weighted nadaraya–watson regression estimation.

</span>
<span class="ltx_bibblock"><span id="bib.bib12.1.1" class="ltx_text ltx_font_italic">Statistics &amp; probability letters</span>, 51(3):307–318, 2001.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[13]</span>
<span class="ltx_bibblock">
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin.

</span>
<span class="ltx_bibblock">Training deep nets with sublinear memory cost, 2016.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[14]</span>
<span class="ltx_bibblock">
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.

</span>
<span class="ltx_bibblock">Improved baselines with momentum contrastive learning.

</span>
<span class="ltx_bibblock"><span id="bib.bib14.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2003.04297</span>, 2020.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[15]</span>
<span class="ltx_bibblock">
Yen-Chi Chen.

</span>
<span class="ltx_bibblock">A tutorial on kernel density estimation and recent advances.

</span>
<span class="ltx_bibblock"><span id="bib.bib15.1.1" class="ltx_text ltx_font_italic">Biostatistics &amp; Epidemiology</span>, 1(1):161–187, 2017.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[16]</span>
<span class="ltx_bibblock">
Kevin Clark, Kelvin Guu, Ming-Wei Chang, Panupong Pasupat, Geoffrey Hinton, and Mohammad Norouzi.

</span>
<span class="ltx_bibblock">Meta-learning fast weight language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib16.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2212.02475</span>, 2022.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[17]</span>
<span class="ltx_bibblock">
Ronan Collobert, Fabian Sinz, Jason Weston, Léon Bottou, and Thorsten Joachims.

</span>
<span class="ltx_bibblock">Large scale transductive svms.

</span>
<span class="ltx_bibblock"><span id="bib.bib17.1.1" class="ltx_text ltx_font_italic">Journal of Machine Learning Research</span>, 7(8), 2006.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[18]</span>
<span class="ltx_bibblock">
Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al.

</span>
<span class="ltx_bibblock">Griffin: Mixing gated linear recurrences with local attention for efficient language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib18.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2402.19427</span>, 2024.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[19]</span>
<span class="ltx_bibblock">
Harm de Vries.

</span>
<span class="ltx_bibblock">In the long (context) run, 2023.

</span>
<span class="ltx_bibblock">Accessed: 2024-06-24.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[20]</span>
<span class="ltx_bibblock">
Chelsea Finn, Pieter Abbeel, and Sergey Levine.

</span>
<span class="ltx_bibblock">Model-agnostic meta-learning for fast adaptation of deep networks.

</span>
<span class="ltx_bibblock">In <span id="bib.bib20.1.1" class="ltx_text ltx_font_italic">International conference on machine learning</span>, pages 1126–1135. PMLR, 2017.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[21]</span>
<span class="ltx_bibblock">
Daniel Y Fu, Tri Dao, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré.

</span>
<span class="ltx_bibblock">Hungry hungry hippos: Towards language modeling with state space models.

</span>
<span class="ltx_bibblock"><span id="bib.bib21.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2212.14052</span>, 2022.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[22]</span>
<span class="ltx_bibblock">
A. Gammerman, V. Vovk, and V. Vapnik.

</span>
<span class="ltx_bibblock">Learning by transduction.

</span>
<span class="ltx_bibblock">In <span id="bib.bib22.1.1" class="ltx_text ltx_font_italic">In Uncertainty in Artificial Intelligence</span>, pages 148–155. Morgan Kaufmann, 1998.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[23]</span>
<span class="ltx_bibblock">
Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei A. Efros.

</span>
<span class="ltx_bibblock">Test-time training with masked autoencoders.

</span>
<span class="ltx_bibblock"><span id="bib.bib23.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, 2022.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[24]</span>
<span class="ltx_bibblock">
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy.

</span>
<span class="ltx_bibblock">The pile: An 800gb dataset of diverse text for language modeling, 2020.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[25]</span>
<span class="ltx_bibblock">
Xinyang Geng.

</span>
<span class="ltx_bibblock">EasyLM: A Simple And Scalable Training Framework for Large Language Models.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/young-geng/EasyLM" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/young-geng/EasyLM</a>, mar 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/young-geng/EasyLM" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/young-geng/EasyLM</a>.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[26]</span>
<span class="ltx_bibblock">
Albert Gu and Tri Dao.

</span>
<span class="ltx_bibblock">Mamba: Linear-time sequence modeling with selective state spaces.

</span>
<span class="ltx_bibblock"><span id="bib.bib26.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2312.00752</span>, 2023.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[27]</span>
<span class="ltx_bibblock">
Albert Gu, Karan Goel, and Christopher Ré.

</span>
<span class="ltx_bibblock">Efficiently modeling long sequences with structured state spaces.

</span>
<span class="ltx_bibblock"><span id="bib.bib27.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2111.00396</span>, 2021.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[28]</span>
<span class="ltx_bibblock">
Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyà, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang.

</span>
<span class="ltx_bibblock">Self-supervised policy adaptation during deployment.

</span>
<span class="ltx_bibblock"><span id="bib.bib28.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2007.04309</span>, 2020.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[29]</span>
<span class="ltx_bibblock">
Moritz Hardt and Yu Sun.

</span>
<span class="ltx_bibblock">Test-time training on nearest neighbors for large language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib29.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2305.18466</span>, 2023.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[30]</span>
<span class="ltx_bibblock">
Horace He.

</span>
<span class="ltx_bibblock">Strangely, matrix multiplications on gpus run faster when given "predictable" data! [short], 2024.

</span>
<span class="ltx_bibblock">Accessed: 2024-06-30.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[31]</span>
<span class="ltx_bibblock">
Dan Hendrycks and Kevin Gimpel.

</span>
<span class="ltx_bibblock">Gaussian error linear units (gelus).

</span>
<span class="ltx_bibblock"><span id="bib.bib31.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:1606.08415</span>, 2016.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[32]</span>
<span class="ltx_bibblock">
Geoffrey E Hinton and David C Plaut.

</span>
<span class="ltx_bibblock">Using fast weights to deblur old memories.

</span>
<span class="ltx_bibblock">In <span id="bib.bib32.1.1" class="ltx_text ltx_font_italic">Proceedings of the ninth annual conference of the Cognitive Science Society</span>, pages 177–186, 1987.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[33]</span>
<span class="ltx_bibblock">
Sepp Hochreiter and Jürgen Schmidhuber.

</span>
<span class="ltx_bibblock">Long short-term memory.

</span>
<span class="ltx_bibblock"><span id="bib.bib33.1.1" class="ltx_text ltx_font_italic">Neural computation</span>, 9(8):1735–1780, 1997.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[34]</span>
<span class="ltx_bibblock">
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.

</span>
<span class="ltx_bibblock">Training compute-optimal large language models, 2022.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[35]</span>
<span class="ltx_bibblock">
John J Hopfield.

</span>
<span class="ltx_bibblock">Neural networks and physical systems with emergent collective computational abilities.

</span>
<span class="ltx_bibblock"><span id="bib.bib35.1.1" class="ltx_text ltx_font_italic">Proceedings of the national academy of sciences</span>, 79(8):2554–2558, 1982.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[36]</span>
<span class="ltx_bibblock">
Kazuki Irie, Róbert Csordás, and Jürgen Schmidhuber.

</span>
<span class="ltx_bibblock">The dual form of neural networks revisited: Connecting test time predictions to training patterns via spotlights of attention.

</span>
<span class="ltx_bibblock">In <span id="bib.bib36.1.1" class="ltx_text ltx_font_italic">International Conference on Machine Learning</span>, pages 9639–9659. PMLR, 2022.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[37]</span>
<span class="ltx_bibblock">
Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber.

</span>
<span class="ltx_bibblock">Going beyond linear transformers with recurrent fast weight programmers.

</span>
<span class="ltx_bibblock"><span id="bib.bib37.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, 34:7703–7717, 2021.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[38]</span>
<span class="ltx_bibblock">
Vidit Jain and Erik Learned-Miller.

</span>
<span class="ltx_bibblock">Online domain adaptation of a pre-trained cascade of classifiers.

</span>
<span class="ltx_bibblock">In <span id="bib.bib38.1.1" class="ltx_text ltx_font_italic">CVPR 2011</span>, pages 577–584. IEEE, 2011.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[39]</span>
<span class="ltx_bibblock">
Thorsten Joachims.

</span>
<span class="ltx_bibblock"><span id="bib.bib39.1.1" class="ltx_text ltx_font_italic">Learning to classify text using support vector machines</span>, volume 668.

</span>
<span class="ltx_bibblock">Springer Science &amp; Business Media, 2002.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[40]</span>
<span class="ltx_bibblock">
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.

</span>
<span class="ltx_bibblock">Scaling laws for neural language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib40.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2001.08361</span>, 2020.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[41]</span>
<span class="ltx_bibblock">
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret.

</span>
<span class="ltx_bibblock">Transformers are rnns: Fast autoregressive transformers with linear attention.

</span>
<span class="ltx_bibblock">In <span id="bib.bib41.1.1" class="ltx_text ltx_font_italic">International conference on machine learning</span>, pages 5156–5165. PMLR, 2020.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[42]</span>
<span class="ltx_bibblock">
Diederik P Kingma and Jimmy Ba.

</span>
<span class="ltx_bibblock">Adam: A method for stochastic optimization.

</span>
<span class="ltx_bibblock"><span id="bib.bib42.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:1412.6980</span>, 2014.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[43]</span>
<span class="ltx_bibblock">
Louis Kirsch and Jürgen Schmidhuber.

</span>
<span class="ltx_bibblock">Meta learning backpropagation and improving it.

</span>
<span class="ltx_bibblock"><span id="bib.bib43.1.1" class="ltx_text ltx_font_italic">Advances in Neural Information Processing Systems</span>, 34:14122–14134, 2021.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[44]</span>
<span class="ltx_bibblock">
Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals.

</span>
<span class="ltx_bibblock">Dynamic evaluation of neural sequence models.

</span>
<span class="ltx_bibblock">In <span id="bib.bib44.1.1" class="ltx_text ltx_font_italic">International Conference on Machine Learning</span>, pages 2766–2775. PMLR, 2018.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[45]</span>
<span class="ltx_bibblock">
Ben Krause, Emmanuel Kahembwe, Iain Murray, and Steve Renals.

</span>
<span class="ltx_bibblock">Dynamic evaluation of transformer language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib45.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:1904.08378</span>, 2019.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[46]</span>
<span class="ltx_bibblock">
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica.

</span>
<span class="ltx_bibblock">Efficient memory management for large language model serving with pagedattention.

</span>
<span class="ltx_bibblock">In <span id="bib.bib46.1.1" class="ltx_text ltx_font_italic">Proceedings of the 29th Symposium on Operating Systems Principles</span>, pages 611–626, 2023.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[47]</span>
<span class="ltx_bibblock">
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman.

</span>
<span class="ltx_bibblock">Building machines that learn and think like people.

</span>
<span class="ltx_bibblock"><span id="bib.bib47.1.1" class="ltx_text ltx_font_italic">Behavioral and brain sciences</span>, 40:e253, 2017.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[48]</span>
<span class="ltx_bibblock">
Quoc V Le.

</span>
<span class="ltx_bibblock">Building high-level features using large scale unsupervised learning.

</span>
<span class="ltx_bibblock">In <span id="bib.bib48.1.1" class="ltx_text ltx_font_italic">2013 IEEE international conference on acoustics, speech and signal processing</span>, pages 8595–8598. IEEE, 2013.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[49]</span>
<span class="ltx_bibblock">
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel.

</span>
<span class="ltx_bibblock">World model on million-length video and language with blockwise ringattention.

</span>
<span class="ltx_bibblock"><span id="bib.bib49.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2402.08268</span>, 2024.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[50]</span>
<span class="ltx_bibblock">
Xuan Luo, Jia-Bin Huang, Richard Szeliski, Kevin Matzen, and Johannes Kopf.

</span>
<span class="ltx_bibblock">Consistent video depth estimation.

</span>
<span class="ltx_bibblock"><span id="bib.bib50.1.1" class="ltx_text ltx_font_italic">ACM Transactions on Graphics (ToG)</span>, 39(4):71–1, 2020.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[51]</span>
<span class="ltx_bibblock">
Dougal Maclaurin, David Duvenaud, and Ryan Adams.

</span>
<span class="ltx_bibblock">Gradient-based hyperparameter optimization through reversible learning.

</span>
<span class="ltx_bibblock">In <span id="bib.bib51.1.1" class="ltx_text ltx_font_italic">International conference on machine learning</span>, pages 2113–2122. PMLR, 2015.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[52]</span>
<span class="ltx_bibblock">
Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein.

</span>
<span class="ltx_bibblock">Meta-learning update rules for unsupervised representation learning.

</span>
<span class="ltx_bibblock"><span id="bib.bib52.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:1804.00222</span>, 2018.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[53]</span>
<span class="ltx_bibblock">
Ravi Teja Mullapudi, Steven Chen, Keyi Zhang, Deva Ramanan, and Kayvon Fatahalian.

</span>
<span class="ltx_bibblock">Online model distillation for efficient video inference.

</span>
<span class="ltx_bibblock"><span id="bib.bib53.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:1812.02699</span>, 2018.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[54]</span>
<span class="ltx_bibblock">
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.

</span>
<span class="ltx_bibblock">Scikit-learn: Machine learning in Python.

</span>
<span class="ltx_bibblock"><span id="bib.bib54.1.1" class="ltx_text ltx_font_italic">Journal of Machine Learning Research</span>, 12:2825–2830, 2011.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[55]</span>
<span class="ltx_bibblock">
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al.

</span>
<span class="ltx_bibblock">Rwkv: Reinventing rnns for the transformer era.

</span>
<span class="ltx_bibblock"><span id="bib.bib55.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2305.13048</span>, 2023.

</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[56]</span>
<span class="ltx_bibblock">
Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Teddy Ferdinan, Haowen Hou, Przemysław Kazienko, et al.

</span>
<span class="ltx_bibblock">Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence.

</span>
<span class="ltx_bibblock"><span id="bib.bib56.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2404.05892</span>, 2024.

</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[57]</span>
<span class="ltx_bibblock">
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré.

</span>
<span class="ltx_bibblock">Hyena hierarchy: Towards larger convolutional language models.

</span>
<span class="ltx_bibblock"><span id="bib.bib57.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2302.10866</span>, 2023.

</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[58]</span>
<span class="ltx_bibblock">
Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong.

</span>
<span class="ltx_bibblock">The devil in linear transformer.

</span>
<span class="ltx_bibblock"><span id="bib.bib58.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2210.10340</span>, 2022.

</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[59]</span>
<span class="ltx_bibblock">
Frank Rosenblatt.

</span>
<span class="ltx_bibblock">The perceptron: a probabilistic model for information storage and organization in the brain.

</span>
<span class="ltx_bibblock"><span id="bib.bib59.1.1" class="ltx_text ltx_font_italic">Psychological review</span>, 65(6):386, 1958.

</span>
</li>
<li id="bib.bib60" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[60]</span>
<span class="ltx_bibblock">
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber.

</span>
<span class="ltx_bibblock">Linear transformers are secretly fast weight programmers.

</span>
<span class="ltx_bibblock">In <span id="bib.bib60.1.1" class="ltx_text ltx_font_italic">International Conference on Machine Learning</span>, pages 9355–9366. PMLR, 2021.

</span>
</li>
<li id="bib.bib61" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[61]</span>
<span class="ltx_bibblock">
Jürgen Schmidhuber.

</span>
<span class="ltx_bibblock"><span id="bib.bib61.1.1" class="ltx_text ltx_font_italic">Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-… hook</span>.

</span>
<span class="ltx_bibblock">PhD thesis, Technische Universität München, 1987.

</span>
</li>
<li id="bib.bib62" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[62]</span>
<span class="ltx_bibblock">
Jürgen Schmidhuber.

</span>
<span class="ltx_bibblock">Learning to control fast-weight memories: An alternative to dynamic recurrent networks.

</span>
<span class="ltx_bibblock"><span id="bib.bib62.1.1" class="ltx_text ltx_font_italic">Neural Computation</span>, 4(1):131–139, 1992.

</span>
</li>
<li id="bib.bib63" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[63]</span>
<span class="ltx_bibblock">
Noam Shazeer.

</span>
<span class="ltx_bibblock">Glu variants improve transformer, 2020.

</span>
</li>
<li id="bib.bib64" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[64]</span>
<span class="ltx_bibblock">
Sam Shleifer, Jason Weston, and Myle Ott.

</span>
<span class="ltx_bibblock">Normformer: Improved transformer pretraining with extra normalization.

</span>
<span class="ltx_bibblock"><span id="bib.bib64.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2110.09456</span>, 2021.

</span>
</li>
<li id="bib.bib65" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[65]</span>
<span class="ltx_bibblock">
Assaf Shocher, Nadav Cohen, and Michal Irani.

</span>
<span class="ltx_bibblock">“zero-shot” super-resolution using deep internal learning.

</span>
<span class="ltx_bibblock">In <span id="bib.bib65.1.1" class="ltx_text ltx_font_italic">Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition</span>, pages 3118–3126, 2018.

</span>
</li>
<li id="bib.bib66" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[66]</span>
<span class="ltx_bibblock">
Benjamin Spector, Aaryan Singhal, Simran Arora, and Chris Re.

</span>
<span class="ltx_bibblock">Thunderkittens.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/HazyResearch/ThunderKittens" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/HazyResearch/ThunderKittens</a>, 2023.

</span>
</li>
<li id="bib.bib67" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[67]</span>
<span class="ltx_bibblock">
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu.

</span>
<span class="ltx_bibblock">Roformer: Enhanced transformer with rotary position embedding, 2023.

</span>
</li>
<li id="bib.bib68" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[68]</span>
<span class="ltx_bibblock">
Yu Sun, Wyatt L Ubellacker, Wen-Loong Ma, Xiang Zhang, Changhao Wang, Noel V Csomay-Shanklin, Masayoshi Tomizuka, Koushil Sreenath, and Aaron D Ames.

</span>
<span class="ltx_bibblock">Online learning of unknown dynamics for model-based controllers in legged locomotion.

</span>
<span class="ltx_bibblock"><span id="bib.bib68.1.1" class="ltx_text ltx_font_italic">IEEE Robotics and Automation Letters</span>, 6(4):8442–8449, 2021.

</span>
</li>
<li id="bib.bib69" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[69]</span>
<span class="ltx_bibblock">
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt.

</span>
<span class="ltx_bibblock">Test-time training with self-supervision for generalization under distribution shifts.

</span>
<span class="ltx_bibblock">In <span id="bib.bib69.1.1" class="ltx_text ltx_font_italic">International Conference on Machine Learning</span>, pages 9229–9248. PMLR, 2020.

</span>
</li>
<li id="bib.bib70" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[70]</span>
<span class="ltx_bibblock">
Sebastian Thrun and Lorien Pratt.

</span>
<span class="ltx_bibblock">Learning to learn: Introduction and overview.

</span>
<span class="ltx_bibblock">In <span id="bib.bib70.1.1" class="ltx_text ltx_font_italic">Learning to learn</span>, pages 3–17. Springer, 1998.

</span>
</li>
<li id="bib.bib71" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[71]</span>
<span class="ltx_bibblock">
Tijmen Tieleman and Geoffrey Hinton.

</span>
<span class="ltx_bibblock">Using fast weights to improve persistent contrastive divergence.

</span>
<span class="ltx_bibblock">In <span id="bib.bib71.1.1" class="ltx_text ltx_font_italic">Proceedings of the 26th annual international conference on machine learning</span>, pages 1033–1040, 2009.

</span>
</li>
<li id="bib.bib72" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[72]</span>
<span class="ltx_bibblock">
Philippe Tillet, Hsiang-Tsung Kung, and David Cox.

</span>
<span class="ltx_bibblock">Triton: an intermediate language and compiler for tiled neural network computations.

</span>
<span class="ltx_bibblock">In <span id="bib.bib72.1.1" class="ltx_text ltx_font_italic">Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages</span>, pages 10–19, 2019.

</span>
</li>
<li id="bib.bib73" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[73]</span>
<span class="ltx_bibblock">
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas
Scialom.

</span>
<span class="ltx_bibblock">Llama 2: Open foundation and fine-tuned chat models, 2023.

</span>
</li>
<li id="bib.bib74" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[74]</span>
<span class="ltx_bibblock">
Vladimir Vapnik.

</span>
<span class="ltx_bibblock"><span id="bib.bib74.1.1" class="ltx_text ltx_font_italic">The nature of statistical learning theory</span>.

</span>
<span class="ltx_bibblock">Springer science &amp; business media, 2013.

</span>
</li>
<li id="bib.bib75" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[75]</span>
<span class="ltx_bibblock">
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol.

</span>
<span class="ltx_bibblock">Extracting and composing robust features with denoising autoencoders.

</span>
<span class="ltx_bibblock">In <span id="bib.bib75.1.1" class="ltx_text ltx_font_italic">ICML</span>, page 1096–1103, 2008.

</span>
</li>
<li id="bib.bib76" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[76]</span>
<span class="ltx_bibblock">
Renhao Wang, Yu Sun, Yossi Gandelsman, Xinlei Chen, Alexei A Efros, and Xiaolong Wang.

</span>
<span class="ltx_bibblock">Test-time training on video streams.

</span>
<span class="ltx_bibblock"><span id="bib.bib76.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2307.05014</span>, 2023.

</span>
</li>
<li id="bib.bib77" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[77]</span>
<span class="ltx_bibblock">
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.

</span>
<span class="ltx_bibblock">Huggingface’s transformers: State-of-the-art natural language processing.

</span>
<span class="ltx_bibblock"><span id="bib.bib77.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:1910.03771</span>, 2019.

</span>
</li>
<li id="bib.bib78" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[78]</span>
<span class="ltx_bibblock">
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma.

</span>
<span class="ltx_bibblock">Effective long-context scaling of foundation models, 2023.

</span>
</li>
<li id="bib.bib79" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[79]</span>
<span class="ltx_bibblock">
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim.

</span>
<span class="ltx_bibblock">Gated linear attention transformers with hardware-efficient training.

</span>
<span class="ltx_bibblock"><span id="bib.bib79.1.1" class="ltx_text ltx_font_italic">arXiv preprint arXiv:2312.06635</span>, 2023.

</span>
</li>
<li id="bib.bib80" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[80]</span>
<span class="ltx_bibblock">
Biao Zhang and Rico Sennrich.

</span>
<span class="ltx_bibblock">Root mean square layer normalization, 2019.

</span>
</li>
<li id="bib.bib81" class="ltx_bibitem">
<span class="ltx_tag ltx_tag_bibitem">[81]</span>
<span class="ltx_bibblock">
Hao Zhang, Alexander C Berg, Michael Maire, and Jitendra Malik.

</span>
<span class="ltx_bibblock">Svm-knn: Discriminative nearest neighbor classification for visual category recognition.

</span>
<span class="ltx_bibblock">In <span id="bib.bib81.1.1" class="ltx_text ltx_font_italic">2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)</span>, volume 2, pages 2126–2136. IEEE, 2006.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Dual Form</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.1" class="ltx_p">The goal of this section is to derive the dual form for general MLPs of arbitrary depth, with nonlinear activations.</p>
</div>
<div id="A1.p2" class="ltx_para">
<p id="A1.p2.2" class="ltx_p">Without loss of generality, consider <math id="A1.p2.1.m1.1" class="ltx_Math" alttext="\eta=1" display="inline"><semantics id="A1.p2.1.m1.1a"><mrow id="A1.p2.1.m1.1.1" xref="A1.p2.1.m1.1.1.cmml"><mi id="A1.p2.1.m1.1.1.2" xref="A1.p2.1.m1.1.1.2.cmml">η</mi><mo id="A1.p2.1.m1.1.1.1" xref="A1.p2.1.m1.1.1.1.cmml">=</mo><mn id="A1.p2.1.m1.1.1.3" xref="A1.p2.1.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.p2.1.m1.1b"><apply id="A1.p2.1.m1.1.1.cmml" xref="A1.p2.1.m1.1.1"><eq id="A1.p2.1.m1.1.1.1.cmml" xref="A1.p2.1.m1.1.1.1"></eq><ci id="A1.p2.1.m1.1.1.2.cmml" xref="A1.p2.1.m1.1.1.2">𝜂</ci><cn type="integer" id="A1.p2.1.m1.1.1.3.cmml" xref="A1.p2.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.1.m1.1c">\eta=1</annotation></semantics></math> for convenience, and consider only the first mini-batch, where <math id="A1.p2.2.m2.3" class="ltx_Math" alttext="t=1,\dots,b" display="inline"><semantics id="A1.p2.2.m2.3a"><mrow id="A1.p2.2.m2.3.4" xref="A1.p2.2.m2.3.4.cmml"><mi id="A1.p2.2.m2.3.4.2" xref="A1.p2.2.m2.3.4.2.cmml">t</mi><mo id="A1.p2.2.m2.3.4.1" xref="A1.p2.2.m2.3.4.1.cmml">=</mo><mrow id="A1.p2.2.m2.3.4.3.2" xref="A1.p2.2.m2.3.4.3.1.cmml"><mn id="A1.p2.2.m2.1.1" xref="A1.p2.2.m2.1.1.cmml">1</mn><mo id="A1.p2.2.m2.3.4.3.2.1" xref="A1.p2.2.m2.3.4.3.1.cmml">,</mo><mi mathvariant="normal" id="A1.p2.2.m2.2.2" xref="A1.p2.2.m2.2.2.cmml">…</mi><mo id="A1.p2.2.m2.3.4.3.2.2" xref="A1.p2.2.m2.3.4.3.1.cmml">,</mo><mi id="A1.p2.2.m2.3.3" xref="A1.p2.2.m2.3.3.cmml">b</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.p2.2.m2.3b"><apply id="A1.p2.2.m2.3.4.cmml" xref="A1.p2.2.m2.3.4"><eq id="A1.p2.2.m2.3.4.1.cmml" xref="A1.p2.2.m2.3.4.1"></eq><ci id="A1.p2.2.m2.3.4.2.cmml" xref="A1.p2.2.m2.3.4.2">𝑡</ci><list id="A1.p2.2.m2.3.4.3.1.cmml" xref="A1.p2.2.m2.3.4.3.2"><cn type="integer" id="A1.p2.2.m2.1.1.cmml" xref="A1.p2.2.m2.1.1">1</cn><ci id="A1.p2.2.m2.2.2.cmml" xref="A1.p2.2.m2.2.2">…</ci><ci id="A1.p2.2.m2.3.3.cmml" xref="A1.p2.2.m2.3.3">𝑏</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.2.m2.3c">t=1,\dots,b</annotation></semantics></math>.
Denote</p>
<table id="A1.Ex7" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.Ex7.m1.1" class="ltx_Math" alttext="\hat{x}_{t}=\theta_{K}x_{t},~{}~{}~{}y_{t}=\theta_{V}x_{t},~{}~{}~{}\bar{x}_{t}=\theta_{Q}x_{t}." display="block"><semantics id="A1.Ex7.m1.1a"><mrow id="A1.Ex7.m1.1.1.1"><mrow id="A1.Ex7.m1.1.1.1.1.2" xref="A1.Ex7.m1.1.1.1.1.3.cmml"><mrow id="A1.Ex7.m1.1.1.1.1.1.1" xref="A1.Ex7.m1.1.1.1.1.1.1.cmml"><msub id="A1.Ex7.m1.1.1.1.1.1.1.2" xref="A1.Ex7.m1.1.1.1.1.1.1.2.cmml"><mover accent="true" id="A1.Ex7.m1.1.1.1.1.1.1.2.2" xref="A1.Ex7.m1.1.1.1.1.1.1.2.2.cmml"><mi id="A1.Ex7.m1.1.1.1.1.1.1.2.2.2" xref="A1.Ex7.m1.1.1.1.1.1.1.2.2.2.cmml">x</mi><mo id="A1.Ex7.m1.1.1.1.1.1.1.2.2.1" xref="A1.Ex7.m1.1.1.1.1.1.1.2.2.1.cmml">^</mo></mover><mi id="A1.Ex7.m1.1.1.1.1.1.1.2.3" xref="A1.Ex7.m1.1.1.1.1.1.1.2.3.cmml">t</mi></msub><mo id="A1.Ex7.m1.1.1.1.1.1.1.1" xref="A1.Ex7.m1.1.1.1.1.1.1.1.cmml">=</mo><mrow id="A1.Ex7.m1.1.1.1.1.1.1.3" xref="A1.Ex7.m1.1.1.1.1.1.1.3.cmml"><msub id="A1.Ex7.m1.1.1.1.1.1.1.3.2" xref="A1.Ex7.m1.1.1.1.1.1.1.3.2.cmml"><mi id="A1.Ex7.m1.1.1.1.1.1.1.3.2.2" xref="A1.Ex7.m1.1.1.1.1.1.1.3.2.2.cmml">θ</mi><mi id="A1.Ex7.m1.1.1.1.1.1.1.3.2.3" xref="A1.Ex7.m1.1.1.1.1.1.1.3.2.3.cmml">K</mi></msub><mo lspace="0em" rspace="0em" id="A1.Ex7.m1.1.1.1.1.1.1.3.1" xref="A1.Ex7.m1.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="A1.Ex7.m1.1.1.1.1.1.1.3.3" xref="A1.Ex7.m1.1.1.1.1.1.1.3.3.cmml"><mi id="A1.Ex7.m1.1.1.1.1.1.1.3.3.2" xref="A1.Ex7.m1.1.1.1.1.1.1.3.3.2.cmml">x</mi><mi id="A1.Ex7.m1.1.1.1.1.1.1.3.3.3" xref="A1.Ex7.m1.1.1.1.1.1.1.3.3.3.cmml">t</mi></msub></mrow></mrow><mo rspace="1.157em" id="A1.Ex7.m1.1.1.1.1.2.3" xref="A1.Ex7.m1.1.1.1.1.3a.cmml">,</mo><mrow id="A1.Ex7.m1.1.1.1.1.2.2.2" xref="A1.Ex7.m1.1.1.1.1.2.2.3.cmml"><mrow id="A1.Ex7.m1.1.1.1.1.2.2.1.1" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.cmml"><msub id="A1.Ex7.m1.1.1.1.1.2.2.1.1.2" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.2.cmml"><mi id="A1.Ex7.m1.1.1.1.1.2.2.1.1.2.2" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.2.2.cmml">y</mi><mi id="A1.Ex7.m1.1.1.1.1.2.2.1.1.2.3" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.2.3.cmml">t</mi></msub><mo id="A1.Ex7.m1.1.1.1.1.2.2.1.1.1" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.1.cmml">=</mo><mrow id="A1.Ex7.m1.1.1.1.1.2.2.1.1.3" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.cmml"><msub id="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.2" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.2.cmml"><mi id="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.2.2" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.2.2.cmml">θ</mi><mi id="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.2.3" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.2.3.cmml">V</mi></msub><mo lspace="0em" rspace="0em" id="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.1" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.1.cmml">​</mo><msub id="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.3" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.3.cmml"><mi id="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.3.2" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.3.2.cmml">x</mi><mi id="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.3.3" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.3.3.cmml">t</mi></msub></mrow></mrow><mo rspace="1.157em" id="A1.Ex7.m1.1.1.1.1.2.2.2.3" xref="A1.Ex7.m1.1.1.1.1.2.2.3a.cmml">,</mo><mrow id="A1.Ex7.m1.1.1.1.1.2.2.2.2" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.cmml"><msub id="A1.Ex7.m1.1.1.1.1.2.2.2.2.2" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.2.cmml"><mover accent="true" id="A1.Ex7.m1.1.1.1.1.2.2.2.2.2.2" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.2.2.cmml"><mi id="A1.Ex7.m1.1.1.1.1.2.2.2.2.2.2.2" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.2.2.2.cmml">x</mi><mo id="A1.Ex7.m1.1.1.1.1.2.2.2.2.2.2.1" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.2.2.1.cmml">¯</mo></mover><mi id="A1.Ex7.m1.1.1.1.1.2.2.2.2.2.3" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.2.3.cmml">t</mi></msub><mo id="A1.Ex7.m1.1.1.1.1.2.2.2.2.1" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.1.cmml">=</mo><mrow id="A1.Ex7.m1.1.1.1.1.2.2.2.2.3" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.cmml"><msub id="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.2" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.2.cmml"><mi id="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.2.2" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.2.2.cmml">θ</mi><mi id="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.2.3" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.2.3.cmml">Q</mi></msub><mo lspace="0em" rspace="0em" id="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.1" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.1.cmml">​</mo><msub id="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.3" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.3.cmml"><mi id="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.3.2" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.3.2.cmml">x</mi><mi id="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.3.3" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.3.3.cmml">t</mi></msub></mrow></mrow></mrow></mrow><mo lspace="0em" id="A1.Ex7.m1.1.1.1.2">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex7.m1.1b"><apply id="A1.Ex7.m1.1.1.1.1.3.cmml" xref="A1.Ex7.m1.1.1.1.1.2"><csymbol cd="ambiguous" id="A1.Ex7.m1.1.1.1.1.3a.cmml" xref="A1.Ex7.m1.1.1.1.1.2.3">formulae-sequence</csymbol><apply id="A1.Ex7.m1.1.1.1.1.1.1.cmml" xref="A1.Ex7.m1.1.1.1.1.1.1"><eq id="A1.Ex7.m1.1.1.1.1.1.1.1.cmml" xref="A1.Ex7.m1.1.1.1.1.1.1.1"></eq><apply id="A1.Ex7.m1.1.1.1.1.1.1.2.cmml" xref="A1.Ex7.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A1.Ex7.m1.1.1.1.1.1.1.2.1.cmml" xref="A1.Ex7.m1.1.1.1.1.1.1.2">subscript</csymbol><apply id="A1.Ex7.m1.1.1.1.1.1.1.2.2.cmml" xref="A1.Ex7.m1.1.1.1.1.1.1.2.2"><ci id="A1.Ex7.m1.1.1.1.1.1.1.2.2.1.cmml" xref="A1.Ex7.m1.1.1.1.1.1.1.2.2.1">^</ci><ci id="A1.Ex7.m1.1.1.1.1.1.1.2.2.2.cmml" xref="A1.Ex7.m1.1.1.1.1.1.1.2.2.2">𝑥</ci></apply><ci id="A1.Ex7.m1.1.1.1.1.1.1.2.3.cmml" xref="A1.Ex7.m1.1.1.1.1.1.1.2.3">𝑡</ci></apply><apply id="A1.Ex7.m1.1.1.1.1.1.1.3.cmml" xref="A1.Ex7.m1.1.1.1.1.1.1.3"><times id="A1.Ex7.m1.1.1.1.1.1.1.3.1.cmml" xref="A1.Ex7.m1.1.1.1.1.1.1.3.1"></times><apply id="A1.Ex7.m1.1.1.1.1.1.1.3.2.cmml" xref="A1.Ex7.m1.1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="A1.Ex7.m1.1.1.1.1.1.1.3.2.1.cmml" xref="A1.Ex7.m1.1.1.1.1.1.1.3.2">subscript</csymbol><ci id="A1.Ex7.m1.1.1.1.1.1.1.3.2.2.cmml" xref="A1.Ex7.m1.1.1.1.1.1.1.3.2.2">𝜃</ci><ci id="A1.Ex7.m1.1.1.1.1.1.1.3.2.3.cmml" xref="A1.Ex7.m1.1.1.1.1.1.1.3.2.3">𝐾</ci></apply><apply id="A1.Ex7.m1.1.1.1.1.1.1.3.3.cmml" xref="A1.Ex7.m1.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="A1.Ex7.m1.1.1.1.1.1.1.3.3.1.cmml" xref="A1.Ex7.m1.1.1.1.1.1.1.3.3">subscript</csymbol><ci id="A1.Ex7.m1.1.1.1.1.1.1.3.3.2.cmml" xref="A1.Ex7.m1.1.1.1.1.1.1.3.3.2">𝑥</ci><ci id="A1.Ex7.m1.1.1.1.1.1.1.3.3.3.cmml" xref="A1.Ex7.m1.1.1.1.1.1.1.3.3.3">𝑡</ci></apply></apply></apply><apply id="A1.Ex7.m1.1.1.1.1.2.2.3.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.2"><csymbol cd="ambiguous" id="A1.Ex7.m1.1.1.1.1.2.2.3a.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.2.3">formulae-sequence</csymbol><apply id="A1.Ex7.m1.1.1.1.1.2.2.1.1.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1"><eq id="A1.Ex7.m1.1.1.1.1.2.2.1.1.1.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.1"></eq><apply id="A1.Ex7.m1.1.1.1.1.2.2.1.1.2.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.2"><csymbol cd="ambiguous" id="A1.Ex7.m1.1.1.1.1.2.2.1.1.2.1.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.2">subscript</csymbol><ci id="A1.Ex7.m1.1.1.1.1.2.2.1.1.2.2.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.2.2">𝑦</ci><ci id="A1.Ex7.m1.1.1.1.1.2.2.1.1.2.3.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.2.3">𝑡</ci></apply><apply id="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.3"><times id="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.1.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.1"></times><apply id="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.2.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.2"><csymbol cd="ambiguous" id="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.2.1.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.2">subscript</csymbol><ci id="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.2.2.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.2.2">𝜃</ci><ci id="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.2.3.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.2.3">𝑉</ci></apply><apply id="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.3.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.3"><csymbol cd="ambiguous" id="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.3.1.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.3">subscript</csymbol><ci id="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.3.2.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.3.2">𝑥</ci><ci id="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.3.3.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.1.1.3.3.3">𝑡</ci></apply></apply></apply><apply id="A1.Ex7.m1.1.1.1.1.2.2.2.2.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2"><eq id="A1.Ex7.m1.1.1.1.1.2.2.2.2.1.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.1"></eq><apply id="A1.Ex7.m1.1.1.1.1.2.2.2.2.2.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.2"><csymbol cd="ambiguous" id="A1.Ex7.m1.1.1.1.1.2.2.2.2.2.1.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.2">subscript</csymbol><apply id="A1.Ex7.m1.1.1.1.1.2.2.2.2.2.2.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.2.2"><ci id="A1.Ex7.m1.1.1.1.1.2.2.2.2.2.2.1.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.2.2.1">¯</ci><ci id="A1.Ex7.m1.1.1.1.1.2.2.2.2.2.2.2.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.2.2.2">𝑥</ci></apply><ci id="A1.Ex7.m1.1.1.1.1.2.2.2.2.2.3.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.2.3">𝑡</ci></apply><apply id="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.3"><times id="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.1.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.1"></times><apply id="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.2.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.2"><csymbol cd="ambiguous" id="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.2.1.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.2">subscript</csymbol><ci id="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.2.2.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.2.2">𝜃</ci><ci id="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.2.3.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.2.3">𝑄</ci></apply><apply id="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.3.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.3"><csymbol cd="ambiguous" id="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.3.1.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.3">subscript</csymbol><ci id="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.3.2.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.3.2">𝑥</ci><ci id="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.3.3.cmml" xref="A1.Ex7.m1.1.1.1.1.2.2.2.2.3.3.3">𝑡</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex7.m1.1c">\hat{x}_{t}=\theta_{K}x_{t},~{}~{}~{}y_{t}=\theta_{V}x_{t},~{}~{}~{}\bar{x}_{t}=\theta_{Q}x_{t}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="A1.p2.5" class="ltx_p">Also denote
<math id="A1.p2.3.m1.3" class="ltx_Math" alttext="\hat{X}=[\hat{x}_{1},\dots,\hat{x}_{b}]" display="inline"><semantics id="A1.p2.3.m1.3a"><mrow id="A1.p2.3.m1.3.3" xref="A1.p2.3.m1.3.3.cmml"><mover accent="true" id="A1.p2.3.m1.3.3.4" xref="A1.p2.3.m1.3.3.4.cmml"><mi id="A1.p2.3.m1.3.3.4.2" xref="A1.p2.3.m1.3.3.4.2.cmml">X</mi><mo id="A1.p2.3.m1.3.3.4.1" xref="A1.p2.3.m1.3.3.4.1.cmml">^</mo></mover><mo id="A1.p2.3.m1.3.3.3" xref="A1.p2.3.m1.3.3.3.cmml">=</mo><mrow id="A1.p2.3.m1.3.3.2.2" xref="A1.p2.3.m1.3.3.2.3.cmml"><mo stretchy="false" id="A1.p2.3.m1.3.3.2.2.3" xref="A1.p2.3.m1.3.3.2.3.cmml">[</mo><msub id="A1.p2.3.m1.2.2.1.1.1" xref="A1.p2.3.m1.2.2.1.1.1.cmml"><mover accent="true" id="A1.p2.3.m1.2.2.1.1.1.2" xref="A1.p2.3.m1.2.2.1.1.1.2.cmml"><mi id="A1.p2.3.m1.2.2.1.1.1.2.2" xref="A1.p2.3.m1.2.2.1.1.1.2.2.cmml">x</mi><mo id="A1.p2.3.m1.2.2.1.1.1.2.1" xref="A1.p2.3.m1.2.2.1.1.1.2.1.cmml">^</mo></mover><mn id="A1.p2.3.m1.2.2.1.1.1.3" xref="A1.p2.3.m1.2.2.1.1.1.3.cmml">1</mn></msub><mo id="A1.p2.3.m1.3.3.2.2.4" xref="A1.p2.3.m1.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="A1.p2.3.m1.1.1" xref="A1.p2.3.m1.1.1.cmml">…</mi><mo id="A1.p2.3.m1.3.3.2.2.5" xref="A1.p2.3.m1.3.3.2.3.cmml">,</mo><msub id="A1.p2.3.m1.3.3.2.2.2" xref="A1.p2.3.m1.3.3.2.2.2.cmml"><mover accent="true" id="A1.p2.3.m1.3.3.2.2.2.2" xref="A1.p2.3.m1.3.3.2.2.2.2.cmml"><mi id="A1.p2.3.m1.3.3.2.2.2.2.2" xref="A1.p2.3.m1.3.3.2.2.2.2.2.cmml">x</mi><mo id="A1.p2.3.m1.3.3.2.2.2.2.1" xref="A1.p2.3.m1.3.3.2.2.2.2.1.cmml">^</mo></mover><mi id="A1.p2.3.m1.3.3.2.2.2.3" xref="A1.p2.3.m1.3.3.2.2.2.3.cmml">b</mi></msub><mo stretchy="false" id="A1.p2.3.m1.3.3.2.2.6" xref="A1.p2.3.m1.3.3.2.3.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.p2.3.m1.3b"><apply id="A1.p2.3.m1.3.3.cmml" xref="A1.p2.3.m1.3.3"><eq id="A1.p2.3.m1.3.3.3.cmml" xref="A1.p2.3.m1.3.3.3"></eq><apply id="A1.p2.3.m1.3.3.4.cmml" xref="A1.p2.3.m1.3.3.4"><ci id="A1.p2.3.m1.3.3.4.1.cmml" xref="A1.p2.3.m1.3.3.4.1">^</ci><ci id="A1.p2.3.m1.3.3.4.2.cmml" xref="A1.p2.3.m1.3.3.4.2">𝑋</ci></apply><list id="A1.p2.3.m1.3.3.2.3.cmml" xref="A1.p2.3.m1.3.3.2.2"><apply id="A1.p2.3.m1.2.2.1.1.1.cmml" xref="A1.p2.3.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="A1.p2.3.m1.2.2.1.1.1.1.cmml" xref="A1.p2.3.m1.2.2.1.1.1">subscript</csymbol><apply id="A1.p2.3.m1.2.2.1.1.1.2.cmml" xref="A1.p2.3.m1.2.2.1.1.1.2"><ci id="A1.p2.3.m1.2.2.1.1.1.2.1.cmml" xref="A1.p2.3.m1.2.2.1.1.1.2.1">^</ci><ci id="A1.p2.3.m1.2.2.1.1.1.2.2.cmml" xref="A1.p2.3.m1.2.2.1.1.1.2.2">𝑥</ci></apply><cn type="integer" id="A1.p2.3.m1.2.2.1.1.1.3.cmml" xref="A1.p2.3.m1.2.2.1.1.1.3">1</cn></apply><ci id="A1.p2.3.m1.1.1.cmml" xref="A1.p2.3.m1.1.1">…</ci><apply id="A1.p2.3.m1.3.3.2.2.2.cmml" xref="A1.p2.3.m1.3.3.2.2.2"><csymbol cd="ambiguous" id="A1.p2.3.m1.3.3.2.2.2.1.cmml" xref="A1.p2.3.m1.3.3.2.2.2">subscript</csymbol><apply id="A1.p2.3.m1.3.3.2.2.2.2.cmml" xref="A1.p2.3.m1.3.3.2.2.2.2"><ci id="A1.p2.3.m1.3.3.2.2.2.2.1.cmml" xref="A1.p2.3.m1.3.3.2.2.2.2.1">^</ci><ci id="A1.p2.3.m1.3.3.2.2.2.2.2.cmml" xref="A1.p2.3.m1.3.3.2.2.2.2.2">𝑥</ci></apply><ci id="A1.p2.3.m1.3.3.2.2.2.3.cmml" xref="A1.p2.3.m1.3.3.2.2.2.3">𝑏</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.3.m1.3c">\hat{X}=[\hat{x}_{1},\dots,\hat{x}_{b}]</annotation></semantics></math>,
and <math id="A1.p2.4.m2.1" class="ltx_Math" alttext="Y" display="inline"><semantics id="A1.p2.4.m2.1a"><mi id="A1.p2.4.m2.1.1" xref="A1.p2.4.m2.1.1.cmml">Y</mi><annotation-xml encoding="MathML-Content" id="A1.p2.4.m2.1b"><ci id="A1.p2.4.m2.1.1.cmml" xref="A1.p2.4.m2.1.1">𝑌</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.4.m2.1c">Y</annotation></semantics></math> and <math id="A1.p2.5.m3.1" class="ltx_Math" alttext="\bar{X}" display="inline"><semantics id="A1.p2.5.m3.1a"><mover accent="true" id="A1.p2.5.m3.1.1" xref="A1.p2.5.m3.1.1.cmml"><mi id="A1.p2.5.m3.1.1.2" xref="A1.p2.5.m3.1.1.2.cmml">X</mi><mo id="A1.p2.5.m3.1.1.1" xref="A1.p2.5.m3.1.1.1.cmml">¯</mo></mover><annotation-xml encoding="MathML-Content" id="A1.p2.5.m3.1b"><apply id="A1.p2.5.m3.1.1.cmml" xref="A1.p2.5.m3.1.1"><ci id="A1.p2.5.m3.1.1.1.cmml" xref="A1.p2.5.m3.1.1.1">¯</ci><ci id="A1.p2.5.m3.1.1.2.cmml" xref="A1.p2.5.m3.1.1.2">𝑋</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.5.m3.1c">\bar{X}</annotation></semantics></math> analogously.
In general, uppercase letters denote matrices whose columns are vectors denoted by the corresponding lowercase letter.</p>
</div>
<div id="A1.p3" class="ltx_para">
<p id="A1.p3.3" class="ltx_p">For a network with <math id="A1.p3.1.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="A1.p3.1.m1.1a"><mi id="A1.p3.1.m1.1.1" xref="A1.p3.1.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="A1.p3.1.m1.1b"><ci id="A1.p3.1.m1.1.1.cmml" xref="A1.p3.1.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p3.1.m1.1c">K</annotation></semantics></math> layers, denote the initial parameters in layer <math id="A1.p3.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A1.p3.2.m2.1a"><mi id="A1.p3.2.m2.1.1" xref="A1.p3.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A1.p3.2.m2.1b"><ci id="A1.p3.2.m2.1.1.cmml" xref="A1.p3.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p3.2.m2.1c">k</annotation></semantics></math> by <math id="A1.p3.3.m3.1" class="ltx_Math" alttext="W_{0}^{k}" display="inline"><semantics id="A1.p3.3.m3.1a"><msubsup id="A1.p3.3.m3.1.1" xref="A1.p3.3.m3.1.1.cmml"><mi id="A1.p3.3.m3.1.1.2.2" xref="A1.p3.3.m3.1.1.2.2.cmml">W</mi><mn id="A1.p3.3.m3.1.1.2.3" xref="A1.p3.3.m3.1.1.2.3.cmml">0</mn><mi id="A1.p3.3.m3.1.1.3" xref="A1.p3.3.m3.1.1.3.cmml">k</mi></msubsup><annotation-xml encoding="MathML-Content" id="A1.p3.3.m3.1b"><apply id="A1.p3.3.m3.1.1.cmml" xref="A1.p3.3.m3.1.1"><csymbol cd="ambiguous" id="A1.p3.3.m3.1.1.1.cmml" xref="A1.p3.3.m3.1.1">superscript</csymbol><apply id="A1.p3.3.m3.1.1.2.cmml" xref="A1.p3.3.m3.1.1"><csymbol cd="ambiguous" id="A1.p3.3.m3.1.1.2.1.cmml" xref="A1.p3.3.m3.1.1">subscript</csymbol><ci id="A1.p3.3.m3.1.1.2.2.cmml" xref="A1.p3.3.m3.1.1.2.2">𝑊</ci><cn type="integer" id="A1.p3.3.m3.1.1.2.3.cmml" xref="A1.p3.3.m3.1.1.2.3">0</cn></apply><ci id="A1.p3.3.m3.1.1.3.cmml" xref="A1.p3.3.m3.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p3.3.m3.1c">W_{0}^{k}</annotation></semantics></math>.
Our convention is to use superscripts for the layer and subscripts for time.</p>
</div>
<section id="A1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.1 </span>Forward pass</h3>

<div id="A1.SS1.p1" class="ltx_para">
<p id="A1.SS1.p1.3" class="ltx_p">During the initial forward pass of TTT,
we denote the input to layer <math id="A1.SS1.p1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A1.SS1.p1.1.m1.1a"><mi id="A1.SS1.p1.1.m1.1.1" xref="A1.SS1.p1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A1.SS1.p1.1.m1.1b"><ci id="A1.SS1.p1.1.m1.1.1.cmml" xref="A1.SS1.p1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p1.1.m1.1c">k</annotation></semantics></math> by
<math id="A1.SS1.p1.2.m2.3" class="ltx_Math" alttext="\hat{X}^{k}=[\hat{x}^{k}_{1},\dots,\hat{x}^{k}_{b}]" display="inline"><semantics id="A1.SS1.p1.2.m2.3a"><mrow id="A1.SS1.p1.2.m2.3.3" xref="A1.SS1.p1.2.m2.3.3.cmml"><msup id="A1.SS1.p1.2.m2.3.3.4" xref="A1.SS1.p1.2.m2.3.3.4.cmml"><mover accent="true" id="A1.SS1.p1.2.m2.3.3.4.2" xref="A1.SS1.p1.2.m2.3.3.4.2.cmml"><mi id="A1.SS1.p1.2.m2.3.3.4.2.2" xref="A1.SS1.p1.2.m2.3.3.4.2.2.cmml">X</mi><mo id="A1.SS1.p1.2.m2.3.3.4.2.1" xref="A1.SS1.p1.2.m2.3.3.4.2.1.cmml">^</mo></mover><mi id="A1.SS1.p1.2.m2.3.3.4.3" xref="A1.SS1.p1.2.m2.3.3.4.3.cmml">k</mi></msup><mo id="A1.SS1.p1.2.m2.3.3.3" xref="A1.SS1.p1.2.m2.3.3.3.cmml">=</mo><mrow id="A1.SS1.p1.2.m2.3.3.2.2" xref="A1.SS1.p1.2.m2.3.3.2.3.cmml"><mo stretchy="false" id="A1.SS1.p1.2.m2.3.3.2.2.3" xref="A1.SS1.p1.2.m2.3.3.2.3.cmml">[</mo><msubsup id="A1.SS1.p1.2.m2.2.2.1.1.1" xref="A1.SS1.p1.2.m2.2.2.1.1.1.cmml"><mover accent="true" id="A1.SS1.p1.2.m2.2.2.1.1.1.2.2" xref="A1.SS1.p1.2.m2.2.2.1.1.1.2.2.cmml"><mi id="A1.SS1.p1.2.m2.2.2.1.1.1.2.2.2" xref="A1.SS1.p1.2.m2.2.2.1.1.1.2.2.2.cmml">x</mi><mo id="A1.SS1.p1.2.m2.2.2.1.1.1.2.2.1" xref="A1.SS1.p1.2.m2.2.2.1.1.1.2.2.1.cmml">^</mo></mover><mn id="A1.SS1.p1.2.m2.2.2.1.1.1.3" xref="A1.SS1.p1.2.m2.2.2.1.1.1.3.cmml">1</mn><mi id="A1.SS1.p1.2.m2.2.2.1.1.1.2.3" xref="A1.SS1.p1.2.m2.2.2.1.1.1.2.3.cmml">k</mi></msubsup><mo id="A1.SS1.p1.2.m2.3.3.2.2.4" xref="A1.SS1.p1.2.m2.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="A1.SS1.p1.2.m2.1.1" xref="A1.SS1.p1.2.m2.1.1.cmml">…</mi><mo id="A1.SS1.p1.2.m2.3.3.2.2.5" xref="A1.SS1.p1.2.m2.3.3.2.3.cmml">,</mo><msubsup id="A1.SS1.p1.2.m2.3.3.2.2.2" xref="A1.SS1.p1.2.m2.3.3.2.2.2.cmml"><mover accent="true" id="A1.SS1.p1.2.m2.3.3.2.2.2.2.2" xref="A1.SS1.p1.2.m2.3.3.2.2.2.2.2.cmml"><mi id="A1.SS1.p1.2.m2.3.3.2.2.2.2.2.2" xref="A1.SS1.p1.2.m2.3.3.2.2.2.2.2.2.cmml">x</mi><mo id="A1.SS1.p1.2.m2.3.3.2.2.2.2.2.1" xref="A1.SS1.p1.2.m2.3.3.2.2.2.2.2.1.cmml">^</mo></mover><mi id="A1.SS1.p1.2.m2.3.3.2.2.2.3" xref="A1.SS1.p1.2.m2.3.3.2.2.2.3.cmml">b</mi><mi id="A1.SS1.p1.2.m2.3.3.2.2.2.2.3" xref="A1.SS1.p1.2.m2.3.3.2.2.2.2.3.cmml">k</mi></msubsup><mo stretchy="false" id="A1.SS1.p1.2.m2.3.3.2.2.6" xref="A1.SS1.p1.2.m2.3.3.2.3.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS1.p1.2.m2.3b"><apply id="A1.SS1.p1.2.m2.3.3.cmml" xref="A1.SS1.p1.2.m2.3.3"><eq id="A1.SS1.p1.2.m2.3.3.3.cmml" xref="A1.SS1.p1.2.m2.3.3.3"></eq><apply id="A1.SS1.p1.2.m2.3.3.4.cmml" xref="A1.SS1.p1.2.m2.3.3.4"><csymbol cd="ambiguous" id="A1.SS1.p1.2.m2.3.3.4.1.cmml" xref="A1.SS1.p1.2.m2.3.3.4">superscript</csymbol><apply id="A1.SS1.p1.2.m2.3.3.4.2.cmml" xref="A1.SS1.p1.2.m2.3.3.4.2"><ci id="A1.SS1.p1.2.m2.3.3.4.2.1.cmml" xref="A1.SS1.p1.2.m2.3.3.4.2.1">^</ci><ci id="A1.SS1.p1.2.m2.3.3.4.2.2.cmml" xref="A1.SS1.p1.2.m2.3.3.4.2.2">𝑋</ci></apply><ci id="A1.SS1.p1.2.m2.3.3.4.3.cmml" xref="A1.SS1.p1.2.m2.3.3.4.3">𝑘</ci></apply><list id="A1.SS1.p1.2.m2.3.3.2.3.cmml" xref="A1.SS1.p1.2.m2.3.3.2.2"><apply id="A1.SS1.p1.2.m2.2.2.1.1.1.cmml" xref="A1.SS1.p1.2.m2.2.2.1.1.1"><csymbol cd="ambiguous" id="A1.SS1.p1.2.m2.2.2.1.1.1.1.cmml" xref="A1.SS1.p1.2.m2.2.2.1.1.1">subscript</csymbol><apply id="A1.SS1.p1.2.m2.2.2.1.1.1.2.cmml" xref="A1.SS1.p1.2.m2.2.2.1.1.1"><csymbol cd="ambiguous" id="A1.SS1.p1.2.m2.2.2.1.1.1.2.1.cmml" xref="A1.SS1.p1.2.m2.2.2.1.1.1">superscript</csymbol><apply id="A1.SS1.p1.2.m2.2.2.1.1.1.2.2.cmml" xref="A1.SS1.p1.2.m2.2.2.1.1.1.2.2"><ci id="A1.SS1.p1.2.m2.2.2.1.1.1.2.2.1.cmml" xref="A1.SS1.p1.2.m2.2.2.1.1.1.2.2.1">^</ci><ci id="A1.SS1.p1.2.m2.2.2.1.1.1.2.2.2.cmml" xref="A1.SS1.p1.2.m2.2.2.1.1.1.2.2.2">𝑥</ci></apply><ci id="A1.SS1.p1.2.m2.2.2.1.1.1.2.3.cmml" xref="A1.SS1.p1.2.m2.2.2.1.1.1.2.3">𝑘</ci></apply><cn type="integer" id="A1.SS1.p1.2.m2.2.2.1.1.1.3.cmml" xref="A1.SS1.p1.2.m2.2.2.1.1.1.3">1</cn></apply><ci id="A1.SS1.p1.2.m2.1.1.cmml" xref="A1.SS1.p1.2.m2.1.1">…</ci><apply id="A1.SS1.p1.2.m2.3.3.2.2.2.cmml" xref="A1.SS1.p1.2.m2.3.3.2.2.2"><csymbol cd="ambiguous" id="A1.SS1.p1.2.m2.3.3.2.2.2.1.cmml" xref="A1.SS1.p1.2.m2.3.3.2.2.2">subscript</csymbol><apply id="A1.SS1.p1.2.m2.3.3.2.2.2.2.cmml" xref="A1.SS1.p1.2.m2.3.3.2.2.2"><csymbol cd="ambiguous" id="A1.SS1.p1.2.m2.3.3.2.2.2.2.1.cmml" xref="A1.SS1.p1.2.m2.3.3.2.2.2">superscript</csymbol><apply id="A1.SS1.p1.2.m2.3.3.2.2.2.2.2.cmml" xref="A1.SS1.p1.2.m2.3.3.2.2.2.2.2"><ci id="A1.SS1.p1.2.m2.3.3.2.2.2.2.2.1.cmml" xref="A1.SS1.p1.2.m2.3.3.2.2.2.2.2.1">^</ci><ci id="A1.SS1.p1.2.m2.3.3.2.2.2.2.2.2.cmml" xref="A1.SS1.p1.2.m2.3.3.2.2.2.2.2.2">𝑥</ci></apply><ci id="A1.SS1.p1.2.m2.3.3.2.2.2.2.3.cmml" xref="A1.SS1.p1.2.m2.3.3.2.2.2.2.3">𝑘</ci></apply><ci id="A1.SS1.p1.2.m2.3.3.2.2.2.3.cmml" xref="A1.SS1.p1.2.m2.3.3.2.2.2.3">𝑏</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p1.2.m2.3c">\hat{X}^{k}=[\hat{x}^{k}_{1},\dots,\hat{x}^{k}_{b}]</annotation></semantics></math>, with <math id="A1.SS1.p1.3.m3.1" class="ltx_Math" alttext="\hat{X}^{1}=\hat{X}" display="inline"><semantics id="A1.SS1.p1.3.m3.1a"><mrow id="A1.SS1.p1.3.m3.1.1" xref="A1.SS1.p1.3.m3.1.1.cmml"><msup id="A1.SS1.p1.3.m3.1.1.2" xref="A1.SS1.p1.3.m3.1.1.2.cmml"><mover accent="true" id="A1.SS1.p1.3.m3.1.1.2.2" xref="A1.SS1.p1.3.m3.1.1.2.2.cmml"><mi id="A1.SS1.p1.3.m3.1.1.2.2.2" xref="A1.SS1.p1.3.m3.1.1.2.2.2.cmml">X</mi><mo id="A1.SS1.p1.3.m3.1.1.2.2.1" xref="A1.SS1.p1.3.m3.1.1.2.2.1.cmml">^</mo></mover><mn id="A1.SS1.p1.3.m3.1.1.2.3" xref="A1.SS1.p1.3.m3.1.1.2.3.cmml">1</mn></msup><mo id="A1.SS1.p1.3.m3.1.1.1" xref="A1.SS1.p1.3.m3.1.1.1.cmml">=</mo><mover accent="true" id="A1.SS1.p1.3.m3.1.1.3" xref="A1.SS1.p1.3.m3.1.1.3.cmml"><mi id="A1.SS1.p1.3.m3.1.1.3.2" xref="A1.SS1.p1.3.m3.1.1.3.2.cmml">X</mi><mo id="A1.SS1.p1.3.m3.1.1.3.1" xref="A1.SS1.p1.3.m3.1.1.3.1.cmml">^</mo></mover></mrow><annotation-xml encoding="MathML-Content" id="A1.SS1.p1.3.m3.1b"><apply id="A1.SS1.p1.3.m3.1.1.cmml" xref="A1.SS1.p1.3.m3.1.1"><eq id="A1.SS1.p1.3.m3.1.1.1.cmml" xref="A1.SS1.p1.3.m3.1.1.1"></eq><apply id="A1.SS1.p1.3.m3.1.1.2.cmml" xref="A1.SS1.p1.3.m3.1.1.2"><csymbol cd="ambiguous" id="A1.SS1.p1.3.m3.1.1.2.1.cmml" xref="A1.SS1.p1.3.m3.1.1.2">superscript</csymbol><apply id="A1.SS1.p1.3.m3.1.1.2.2.cmml" xref="A1.SS1.p1.3.m3.1.1.2.2"><ci id="A1.SS1.p1.3.m3.1.1.2.2.1.cmml" xref="A1.SS1.p1.3.m3.1.1.2.2.1">^</ci><ci id="A1.SS1.p1.3.m3.1.1.2.2.2.cmml" xref="A1.SS1.p1.3.m3.1.1.2.2.2">𝑋</ci></apply><cn type="integer" id="A1.SS1.p1.3.m3.1.1.2.3.cmml" xref="A1.SS1.p1.3.m3.1.1.2.3">1</cn></apply><apply id="A1.SS1.p1.3.m3.1.1.3.cmml" xref="A1.SS1.p1.3.m3.1.1.3"><ci id="A1.SS1.p1.3.m3.1.1.3.1.cmml" xref="A1.SS1.p1.3.m3.1.1.3.1">^</ci><ci id="A1.SS1.p1.3.m3.1.1.3.2.cmml" xref="A1.SS1.p1.3.m3.1.1.3.2">𝑋</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p1.3.m3.1c">\hat{X}^{1}=\hat{X}</annotation></semantics></math>.
Now we write the forward pass of TTT using these notations.</p>
</div>
<div id="A1.SS1.p2" class="ltx_para">
<p id="A1.SS1.p2.1" class="ltx_p">For <math id="A1.SS1.p2.1.m1.3" class="ltx_Math" alttext="k=1,\dots,K" display="inline"><semantics id="A1.SS1.p2.1.m1.3a"><mrow id="A1.SS1.p2.1.m1.3.4" xref="A1.SS1.p2.1.m1.3.4.cmml"><mi id="A1.SS1.p2.1.m1.3.4.2" xref="A1.SS1.p2.1.m1.3.4.2.cmml">k</mi><mo id="A1.SS1.p2.1.m1.3.4.1" xref="A1.SS1.p2.1.m1.3.4.1.cmml">=</mo><mrow id="A1.SS1.p2.1.m1.3.4.3.2" xref="A1.SS1.p2.1.m1.3.4.3.1.cmml"><mn id="A1.SS1.p2.1.m1.1.1" xref="A1.SS1.p2.1.m1.1.1.cmml">1</mn><mo id="A1.SS1.p2.1.m1.3.4.3.2.1" xref="A1.SS1.p2.1.m1.3.4.3.1.cmml">,</mo><mi mathvariant="normal" id="A1.SS1.p2.1.m1.2.2" xref="A1.SS1.p2.1.m1.2.2.cmml">…</mi><mo id="A1.SS1.p2.1.m1.3.4.3.2.2" xref="A1.SS1.p2.1.m1.3.4.3.1.cmml">,</mo><mi id="A1.SS1.p2.1.m1.3.3" xref="A1.SS1.p2.1.m1.3.3.cmml">K</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS1.p2.1.m1.3b"><apply id="A1.SS1.p2.1.m1.3.4.cmml" xref="A1.SS1.p2.1.m1.3.4"><eq id="A1.SS1.p2.1.m1.3.4.1.cmml" xref="A1.SS1.p2.1.m1.3.4.1"></eq><ci id="A1.SS1.p2.1.m1.3.4.2.cmml" xref="A1.SS1.p2.1.m1.3.4.2">𝑘</ci><list id="A1.SS1.p2.1.m1.3.4.3.1.cmml" xref="A1.SS1.p2.1.m1.3.4.3.2"><cn type="integer" id="A1.SS1.p2.1.m1.1.1.cmml" xref="A1.SS1.p2.1.m1.1.1">1</cn><ci id="A1.SS1.p2.1.m1.2.2.cmml" xref="A1.SS1.p2.1.m1.2.2">…</ci><ci id="A1.SS1.p2.1.m1.3.3.cmml" xref="A1.SS1.p2.1.m1.3.3">𝐾</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p2.1.m1.3c">k=1,\dots,K</annotation></semantics></math>:</p>
<ul id="A1.I1" class="ltx_itemize">
<li id="A1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I1.i1.p1" class="ltx_para">
<p id="A1.I1.i1.p1.1" class="ltx_p"><math id="A1.I1.i1.p1.1.m1.1" class="ltx_Math" alttext="Z^{k}=W_{0}^{k}\,\hat{X}^{k}" display="inline"><semantics id="A1.I1.i1.p1.1.m1.1a"><mrow id="A1.I1.i1.p1.1.m1.1.1" xref="A1.I1.i1.p1.1.m1.1.1.cmml"><msup id="A1.I1.i1.p1.1.m1.1.1.2" xref="A1.I1.i1.p1.1.m1.1.1.2.cmml"><mi id="A1.I1.i1.p1.1.m1.1.1.2.2" xref="A1.I1.i1.p1.1.m1.1.1.2.2.cmml">Z</mi><mi id="A1.I1.i1.p1.1.m1.1.1.2.3" xref="A1.I1.i1.p1.1.m1.1.1.2.3.cmml">k</mi></msup><mo id="A1.I1.i1.p1.1.m1.1.1.1" xref="A1.I1.i1.p1.1.m1.1.1.1.cmml">=</mo><mrow id="A1.I1.i1.p1.1.m1.1.1.3" xref="A1.I1.i1.p1.1.m1.1.1.3.cmml"><msubsup id="A1.I1.i1.p1.1.m1.1.1.3.2" xref="A1.I1.i1.p1.1.m1.1.1.3.2.cmml"><mi id="A1.I1.i1.p1.1.m1.1.1.3.2.2.2" xref="A1.I1.i1.p1.1.m1.1.1.3.2.2.2.cmml">W</mi><mn id="A1.I1.i1.p1.1.m1.1.1.3.2.2.3" xref="A1.I1.i1.p1.1.m1.1.1.3.2.2.3.cmml">0</mn><mi id="A1.I1.i1.p1.1.m1.1.1.3.2.3" xref="A1.I1.i1.p1.1.m1.1.1.3.2.3.cmml">k</mi></msubsup><mo lspace="0em" rspace="0em" id="A1.I1.i1.p1.1.m1.1.1.3.1" xref="A1.I1.i1.p1.1.m1.1.1.3.1.cmml">​</mo><msup id="A1.I1.i1.p1.1.m1.1.1.3.3" xref="A1.I1.i1.p1.1.m1.1.1.3.3.cmml"><mover accent="true" id="A1.I1.i1.p1.1.m1.1.1.3.3.2" xref="A1.I1.i1.p1.1.m1.1.1.3.3.2.cmml"><mi id="A1.I1.i1.p1.1.m1.1.1.3.3.2.2" xref="A1.I1.i1.p1.1.m1.1.1.3.3.2.2.cmml">X</mi><mo id="A1.I1.i1.p1.1.m1.1.1.3.3.2.1" xref="A1.I1.i1.p1.1.m1.1.1.3.3.2.1.cmml">^</mo></mover><mi id="A1.I1.i1.p1.1.m1.1.1.3.3.3" xref="A1.I1.i1.p1.1.m1.1.1.3.3.3.cmml">k</mi></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.I1.i1.p1.1.m1.1b"><apply id="A1.I1.i1.p1.1.m1.1.1.cmml" xref="A1.I1.i1.p1.1.m1.1.1"><eq id="A1.I1.i1.p1.1.m1.1.1.1.cmml" xref="A1.I1.i1.p1.1.m1.1.1.1"></eq><apply id="A1.I1.i1.p1.1.m1.1.1.2.cmml" xref="A1.I1.i1.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="A1.I1.i1.p1.1.m1.1.1.2.1.cmml" xref="A1.I1.i1.p1.1.m1.1.1.2">superscript</csymbol><ci id="A1.I1.i1.p1.1.m1.1.1.2.2.cmml" xref="A1.I1.i1.p1.1.m1.1.1.2.2">𝑍</ci><ci id="A1.I1.i1.p1.1.m1.1.1.2.3.cmml" xref="A1.I1.i1.p1.1.m1.1.1.2.3">𝑘</ci></apply><apply id="A1.I1.i1.p1.1.m1.1.1.3.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3"><times id="A1.I1.i1.p1.1.m1.1.1.3.1.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3.1"></times><apply id="A1.I1.i1.p1.1.m1.1.1.3.2.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3.2"><csymbol cd="ambiguous" id="A1.I1.i1.p1.1.m1.1.1.3.2.1.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3.2">superscript</csymbol><apply id="A1.I1.i1.p1.1.m1.1.1.3.2.2.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3.2"><csymbol cd="ambiguous" id="A1.I1.i1.p1.1.m1.1.1.3.2.2.1.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3.2">subscript</csymbol><ci id="A1.I1.i1.p1.1.m1.1.1.3.2.2.2.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3.2.2.2">𝑊</ci><cn type="integer" id="A1.I1.i1.p1.1.m1.1.1.3.2.2.3.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3.2.2.3">0</cn></apply><ci id="A1.I1.i1.p1.1.m1.1.1.3.2.3.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3.2.3">𝑘</ci></apply><apply id="A1.I1.i1.p1.1.m1.1.1.3.3.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3.3"><csymbol cd="ambiguous" id="A1.I1.i1.p1.1.m1.1.1.3.3.1.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3.3">superscript</csymbol><apply id="A1.I1.i1.p1.1.m1.1.1.3.3.2.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3.3.2"><ci id="A1.I1.i1.p1.1.m1.1.1.3.3.2.1.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3.3.2.1">^</ci><ci id="A1.I1.i1.p1.1.m1.1.1.3.3.2.2.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3.3.2.2">𝑋</ci></apply><ci id="A1.I1.i1.p1.1.m1.1.1.3.3.3.cmml" xref="A1.I1.i1.p1.1.m1.1.1.3.3.3">𝑘</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i1.p1.1.m1.1c">Z^{k}=W_{0}^{k}\,\hat{X}^{k}</annotation></semantics></math></p>
</div>
</li>
<li id="A1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I1.i2.p1" class="ltx_para">
<p id="A1.I1.i2.p1.1" class="ltx_p"><math id="A1.I1.i2.p1.1.m1.1" class="ltx_Math" alttext="\hat{X}^{k+1}=\sigma_{k}(Z^{k})" display="inline"><semantics id="A1.I1.i2.p1.1.m1.1a"><mrow id="A1.I1.i2.p1.1.m1.1.1" xref="A1.I1.i2.p1.1.m1.1.1.cmml"><msup id="A1.I1.i2.p1.1.m1.1.1.3" xref="A1.I1.i2.p1.1.m1.1.1.3.cmml"><mover accent="true" id="A1.I1.i2.p1.1.m1.1.1.3.2" xref="A1.I1.i2.p1.1.m1.1.1.3.2.cmml"><mi id="A1.I1.i2.p1.1.m1.1.1.3.2.2" xref="A1.I1.i2.p1.1.m1.1.1.3.2.2.cmml">X</mi><mo id="A1.I1.i2.p1.1.m1.1.1.3.2.1" xref="A1.I1.i2.p1.1.m1.1.1.3.2.1.cmml">^</mo></mover><mrow id="A1.I1.i2.p1.1.m1.1.1.3.3" xref="A1.I1.i2.p1.1.m1.1.1.3.3.cmml"><mi id="A1.I1.i2.p1.1.m1.1.1.3.3.2" xref="A1.I1.i2.p1.1.m1.1.1.3.3.2.cmml">k</mi><mo id="A1.I1.i2.p1.1.m1.1.1.3.3.1" xref="A1.I1.i2.p1.1.m1.1.1.3.3.1.cmml">+</mo><mn id="A1.I1.i2.p1.1.m1.1.1.3.3.3" xref="A1.I1.i2.p1.1.m1.1.1.3.3.3.cmml">1</mn></mrow></msup><mo id="A1.I1.i2.p1.1.m1.1.1.2" xref="A1.I1.i2.p1.1.m1.1.1.2.cmml">=</mo><mrow id="A1.I1.i2.p1.1.m1.1.1.1" xref="A1.I1.i2.p1.1.m1.1.1.1.cmml"><msub id="A1.I1.i2.p1.1.m1.1.1.1.3" xref="A1.I1.i2.p1.1.m1.1.1.1.3.cmml"><mi id="A1.I1.i2.p1.1.m1.1.1.1.3.2" xref="A1.I1.i2.p1.1.m1.1.1.1.3.2.cmml">σ</mi><mi id="A1.I1.i2.p1.1.m1.1.1.1.3.3" xref="A1.I1.i2.p1.1.m1.1.1.1.3.3.cmml">k</mi></msub><mo lspace="0em" rspace="0em" id="A1.I1.i2.p1.1.m1.1.1.1.2" xref="A1.I1.i2.p1.1.m1.1.1.1.2.cmml">​</mo><mrow id="A1.I1.i2.p1.1.m1.1.1.1.1.1" xref="A1.I1.i2.p1.1.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.I1.i2.p1.1.m1.1.1.1.1.1.2" xref="A1.I1.i2.p1.1.m1.1.1.1.1.1.1.cmml">(</mo><msup id="A1.I1.i2.p1.1.m1.1.1.1.1.1.1" xref="A1.I1.i2.p1.1.m1.1.1.1.1.1.1.cmml"><mi id="A1.I1.i2.p1.1.m1.1.1.1.1.1.1.2" xref="A1.I1.i2.p1.1.m1.1.1.1.1.1.1.2.cmml">Z</mi><mi id="A1.I1.i2.p1.1.m1.1.1.1.1.1.1.3" xref="A1.I1.i2.p1.1.m1.1.1.1.1.1.1.3.cmml">k</mi></msup><mo stretchy="false" id="A1.I1.i2.p1.1.m1.1.1.1.1.1.3" xref="A1.I1.i2.p1.1.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.I1.i2.p1.1.m1.1b"><apply id="A1.I1.i2.p1.1.m1.1.1.cmml" xref="A1.I1.i2.p1.1.m1.1.1"><eq id="A1.I1.i2.p1.1.m1.1.1.2.cmml" xref="A1.I1.i2.p1.1.m1.1.1.2"></eq><apply id="A1.I1.i2.p1.1.m1.1.1.3.cmml" xref="A1.I1.i2.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="A1.I1.i2.p1.1.m1.1.1.3.1.cmml" xref="A1.I1.i2.p1.1.m1.1.1.3">superscript</csymbol><apply id="A1.I1.i2.p1.1.m1.1.1.3.2.cmml" xref="A1.I1.i2.p1.1.m1.1.1.3.2"><ci id="A1.I1.i2.p1.1.m1.1.1.3.2.1.cmml" xref="A1.I1.i2.p1.1.m1.1.1.3.2.1">^</ci><ci id="A1.I1.i2.p1.1.m1.1.1.3.2.2.cmml" xref="A1.I1.i2.p1.1.m1.1.1.3.2.2">𝑋</ci></apply><apply id="A1.I1.i2.p1.1.m1.1.1.3.3.cmml" xref="A1.I1.i2.p1.1.m1.1.1.3.3"><plus id="A1.I1.i2.p1.1.m1.1.1.3.3.1.cmml" xref="A1.I1.i2.p1.1.m1.1.1.3.3.1"></plus><ci id="A1.I1.i2.p1.1.m1.1.1.3.3.2.cmml" xref="A1.I1.i2.p1.1.m1.1.1.3.3.2">𝑘</ci><cn type="integer" id="A1.I1.i2.p1.1.m1.1.1.3.3.3.cmml" xref="A1.I1.i2.p1.1.m1.1.1.3.3.3">1</cn></apply></apply><apply id="A1.I1.i2.p1.1.m1.1.1.1.cmml" xref="A1.I1.i2.p1.1.m1.1.1.1"><times id="A1.I1.i2.p1.1.m1.1.1.1.2.cmml" xref="A1.I1.i2.p1.1.m1.1.1.1.2"></times><apply id="A1.I1.i2.p1.1.m1.1.1.1.3.cmml" xref="A1.I1.i2.p1.1.m1.1.1.1.3"><csymbol cd="ambiguous" id="A1.I1.i2.p1.1.m1.1.1.1.3.1.cmml" xref="A1.I1.i2.p1.1.m1.1.1.1.3">subscript</csymbol><ci id="A1.I1.i2.p1.1.m1.1.1.1.3.2.cmml" xref="A1.I1.i2.p1.1.m1.1.1.1.3.2">𝜎</ci><ci id="A1.I1.i2.p1.1.m1.1.1.1.3.3.cmml" xref="A1.I1.i2.p1.1.m1.1.1.1.3.3">𝑘</ci></apply><apply id="A1.I1.i2.p1.1.m1.1.1.1.1.1.1.cmml" xref="A1.I1.i2.p1.1.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.I1.i2.p1.1.m1.1.1.1.1.1.1.1.cmml" xref="A1.I1.i2.p1.1.m1.1.1.1.1.1">superscript</csymbol><ci id="A1.I1.i2.p1.1.m1.1.1.1.1.1.1.2.cmml" xref="A1.I1.i2.p1.1.m1.1.1.1.1.1.1.2">𝑍</ci><ci id="A1.I1.i2.p1.1.m1.1.1.1.1.1.1.3.cmml" xref="A1.I1.i2.p1.1.m1.1.1.1.1.1.1.3">𝑘</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I1.i2.p1.1.m1.1c">\hat{X}^{k+1}=\sigma_{k}(Z^{k})</annotation></semantics></math></p>
</div>
</li>
</ul>
<p id="A1.SS1.p2.5" class="ltx_p">where <math id="A1.SS1.p2.2.m1.1" class="ltx_Math" alttext="\sigma_{k}" display="inline"><semantics id="A1.SS1.p2.2.m1.1a"><msub id="A1.SS1.p2.2.m1.1.1" xref="A1.SS1.p2.2.m1.1.1.cmml"><mi id="A1.SS1.p2.2.m1.1.1.2" xref="A1.SS1.p2.2.m1.1.1.2.cmml">σ</mi><mi id="A1.SS1.p2.2.m1.1.1.3" xref="A1.SS1.p2.2.m1.1.1.3.cmml">k</mi></msub><annotation-xml encoding="MathML-Content" id="A1.SS1.p2.2.m1.1b"><apply id="A1.SS1.p2.2.m1.1.1.cmml" xref="A1.SS1.p2.2.m1.1.1"><csymbol cd="ambiguous" id="A1.SS1.p2.2.m1.1.1.1.cmml" xref="A1.SS1.p2.2.m1.1.1">subscript</csymbol><ci id="A1.SS1.p2.2.m1.1.1.2.cmml" xref="A1.SS1.p2.2.m1.1.1.2">𝜎</ci><ci id="A1.SS1.p2.2.m1.1.1.3.cmml" xref="A1.SS1.p2.2.m1.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p2.2.m1.1c">\sigma_{k}</annotation></semantics></math> for <math id="A1.SS1.p2.3.m2.3" class="ltx_Math" alttext="k=1,\dots,K" display="inline"><semantics id="A1.SS1.p2.3.m2.3a"><mrow id="A1.SS1.p2.3.m2.3.4" xref="A1.SS1.p2.3.m2.3.4.cmml"><mi id="A1.SS1.p2.3.m2.3.4.2" xref="A1.SS1.p2.3.m2.3.4.2.cmml">k</mi><mo id="A1.SS1.p2.3.m2.3.4.1" xref="A1.SS1.p2.3.m2.3.4.1.cmml">=</mo><mrow id="A1.SS1.p2.3.m2.3.4.3.2" xref="A1.SS1.p2.3.m2.3.4.3.1.cmml"><mn id="A1.SS1.p2.3.m2.1.1" xref="A1.SS1.p2.3.m2.1.1.cmml">1</mn><mo id="A1.SS1.p2.3.m2.3.4.3.2.1" xref="A1.SS1.p2.3.m2.3.4.3.1.cmml">,</mo><mi mathvariant="normal" id="A1.SS1.p2.3.m2.2.2" xref="A1.SS1.p2.3.m2.2.2.cmml">…</mi><mo id="A1.SS1.p2.3.m2.3.4.3.2.2" xref="A1.SS1.p2.3.m2.3.4.3.1.cmml">,</mo><mi id="A1.SS1.p2.3.m2.3.3" xref="A1.SS1.p2.3.m2.3.3.cmml">K</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS1.p2.3.m2.3b"><apply id="A1.SS1.p2.3.m2.3.4.cmml" xref="A1.SS1.p2.3.m2.3.4"><eq id="A1.SS1.p2.3.m2.3.4.1.cmml" xref="A1.SS1.p2.3.m2.3.4.1"></eq><ci id="A1.SS1.p2.3.m2.3.4.2.cmml" xref="A1.SS1.p2.3.m2.3.4.2">𝑘</ci><list id="A1.SS1.p2.3.m2.3.4.3.1.cmml" xref="A1.SS1.p2.3.m2.3.4.3.2"><cn type="integer" id="A1.SS1.p2.3.m2.1.1.cmml" xref="A1.SS1.p2.3.m2.1.1">1</cn><ci id="A1.SS1.p2.3.m2.2.2.cmml" xref="A1.SS1.p2.3.m2.2.2">…</ci><ci id="A1.SS1.p2.3.m2.3.3.cmml" xref="A1.SS1.p2.3.m2.3.3">𝐾</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p2.3.m2.3c">k=1,\dots,K</annotation></semantics></math> can be any element-wise operation (<math id="A1.SS1.p2.4.m3.1" class="ltx_Math" alttext="\mathbb{R}\mapsto\mathbb{R}" display="inline"><semantics id="A1.SS1.p2.4.m3.1a"><mrow id="A1.SS1.p2.4.m3.1.1" xref="A1.SS1.p2.4.m3.1.1.cmml"><mi id="A1.SS1.p2.4.m3.1.1.2" xref="A1.SS1.p2.4.m3.1.1.2.cmml">ℝ</mi><mo stretchy="false" id="A1.SS1.p2.4.m3.1.1.1" xref="A1.SS1.p2.4.m3.1.1.1.cmml">↦</mo><mi id="A1.SS1.p2.4.m3.1.1.3" xref="A1.SS1.p2.4.m3.1.1.3.cmml">ℝ</mi></mrow><annotation-xml encoding="MathML-Content" id="A1.SS1.p2.4.m3.1b"><apply id="A1.SS1.p2.4.m3.1.1.cmml" xref="A1.SS1.p2.4.m3.1.1"><csymbol cd="latexml" id="A1.SS1.p2.4.m3.1.1.1.cmml" xref="A1.SS1.p2.4.m3.1.1.1">maps-to</csymbol><ci id="A1.SS1.p2.4.m3.1.1.2.cmml" xref="A1.SS1.p2.4.m3.1.1.2">ℝ</ci><ci id="A1.SS1.p2.4.m3.1.1.3.cmml" xref="A1.SS1.p2.4.m3.1.1.3">ℝ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p2.4.m3.1c">\mathbb{R}\mapsto\mathbb{R}</annotation></semantics></math>) with derivative <math id="A1.SS1.p2.5.m4.1" class="ltx_Math" alttext="\sigma^{\prime}" display="inline"><semantics id="A1.SS1.p2.5.m4.1a"><msup id="A1.SS1.p2.5.m4.1.1" xref="A1.SS1.p2.5.m4.1.1.cmml"><mi id="A1.SS1.p2.5.m4.1.1.2" xref="A1.SS1.p2.5.m4.1.1.2.cmml">σ</mi><mo id="A1.SS1.p2.5.m4.1.1.3" xref="A1.SS1.p2.5.m4.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="A1.SS1.p2.5.m4.1b"><apply id="A1.SS1.p2.5.m4.1.1.cmml" xref="A1.SS1.p2.5.m4.1.1"><csymbol cd="ambiguous" id="A1.SS1.p2.5.m4.1.1.1.cmml" xref="A1.SS1.p2.5.m4.1.1">superscript</csymbol><ci id="A1.SS1.p2.5.m4.1.1.2.cmml" xref="A1.SS1.p2.5.m4.1.1.2">𝜎</ci><ci id="A1.SS1.p2.5.m4.1.1.3.cmml" xref="A1.SS1.p2.5.m4.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p2.5.m4.1c">\sigma^{\prime}</annotation></semantics></math>.</p>
</div>
<div id="A1.SS1.p3" class="ltx_para">
<p id="A1.SS1.p3.1" class="ltx_p">Given <math id="A1.SS1.p3.1.m1.1" class="ltx_Math" alttext="\hat{X}^{K+1}" display="inline"><semantics id="A1.SS1.p3.1.m1.1a"><msup id="A1.SS1.p3.1.m1.1.1" xref="A1.SS1.p3.1.m1.1.1.cmml"><mover accent="true" id="A1.SS1.p3.1.m1.1.1.2" xref="A1.SS1.p3.1.m1.1.1.2.cmml"><mi id="A1.SS1.p3.1.m1.1.1.2.2" xref="A1.SS1.p3.1.m1.1.1.2.2.cmml">X</mi><mo id="A1.SS1.p3.1.m1.1.1.2.1" xref="A1.SS1.p3.1.m1.1.1.2.1.cmml">^</mo></mover><mrow id="A1.SS1.p3.1.m1.1.1.3" xref="A1.SS1.p3.1.m1.1.1.3.cmml"><mi id="A1.SS1.p3.1.m1.1.1.3.2" xref="A1.SS1.p3.1.m1.1.1.3.2.cmml">K</mi><mo id="A1.SS1.p3.1.m1.1.1.3.1" xref="A1.SS1.p3.1.m1.1.1.3.1.cmml">+</mo><mn id="A1.SS1.p3.1.m1.1.1.3.3" xref="A1.SS1.p3.1.m1.1.1.3.3.cmml">1</mn></mrow></msup><annotation-xml encoding="MathML-Content" id="A1.SS1.p3.1.m1.1b"><apply id="A1.SS1.p3.1.m1.1.1.cmml" xref="A1.SS1.p3.1.m1.1.1"><csymbol cd="ambiguous" id="A1.SS1.p3.1.m1.1.1.1.cmml" xref="A1.SS1.p3.1.m1.1.1">superscript</csymbol><apply id="A1.SS1.p3.1.m1.1.1.2.cmml" xref="A1.SS1.p3.1.m1.1.1.2"><ci id="A1.SS1.p3.1.m1.1.1.2.1.cmml" xref="A1.SS1.p3.1.m1.1.1.2.1">^</ci><ci id="A1.SS1.p3.1.m1.1.1.2.2.cmml" xref="A1.SS1.p3.1.m1.1.1.2.2">𝑋</ci></apply><apply id="A1.SS1.p3.1.m1.1.1.3.cmml" xref="A1.SS1.p3.1.m1.1.1.3"><plus id="A1.SS1.p3.1.m1.1.1.3.1.cmml" xref="A1.SS1.p3.1.m1.1.1.3.1"></plus><ci id="A1.SS1.p3.1.m1.1.1.3.2.cmml" xref="A1.SS1.p3.1.m1.1.1.3.2">𝐾</ci><cn type="integer" id="A1.SS1.p3.1.m1.1.1.3.3.cmml" xref="A1.SS1.p3.1.m1.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p3.1.m1.1c">\hat{X}^{K+1}</annotation></semantics></math>, we compute the loss:</p>
<table id="A1.Ex8" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.Ex8.m1.3" class="ltx_Math" alttext="l=\frac{1}{2}\ell\left(W_{0}^{1},\dots,W_{0}^{K};\hat{X}\right)=\frac{1}{2}\big{\|}\hat{X}^{K+1}-Y\big{\|}_{F}^{2}=\sum_{t=1}^{b}l_{t}," display="block"><semantics id="A1.Ex8.m1.3a"><mrow id="A1.Ex8.m1.3.3.1" xref="A1.Ex8.m1.3.3.1.1.cmml"><mrow id="A1.Ex8.m1.3.3.1.1" xref="A1.Ex8.m1.3.3.1.1.cmml"><mi id="A1.Ex8.m1.3.3.1.1.5" xref="A1.Ex8.m1.3.3.1.1.5.cmml">l</mi><mo id="A1.Ex8.m1.3.3.1.1.6" xref="A1.Ex8.m1.3.3.1.1.6.cmml">=</mo><mrow id="A1.Ex8.m1.3.3.1.1.2" xref="A1.Ex8.m1.3.3.1.1.2.cmml"><mfrac id="A1.Ex8.m1.3.3.1.1.2.4" xref="A1.Ex8.m1.3.3.1.1.2.4.cmml"><mn id="A1.Ex8.m1.3.3.1.1.2.4.2" xref="A1.Ex8.m1.3.3.1.1.2.4.2.cmml">1</mn><mn id="A1.Ex8.m1.3.3.1.1.2.4.3" xref="A1.Ex8.m1.3.3.1.1.2.4.3.cmml">2</mn></mfrac><mo lspace="0em" rspace="0em" id="A1.Ex8.m1.3.3.1.1.2.3" xref="A1.Ex8.m1.3.3.1.1.2.3.cmml">​</mo><mi mathvariant="normal" id="A1.Ex8.m1.3.3.1.1.2.5" xref="A1.Ex8.m1.3.3.1.1.2.5.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="A1.Ex8.m1.3.3.1.1.2.3a" xref="A1.Ex8.m1.3.3.1.1.2.3.cmml">​</mo><mrow id="A1.Ex8.m1.3.3.1.1.2.2.2" xref="A1.Ex8.m1.3.3.1.1.2.2.3.cmml"><mo id="A1.Ex8.m1.3.3.1.1.2.2.2.3" xref="A1.Ex8.m1.3.3.1.1.2.2.3.cmml">(</mo><msubsup id="A1.Ex8.m1.3.3.1.1.1.1.1.1" xref="A1.Ex8.m1.3.3.1.1.1.1.1.1.cmml"><mi id="A1.Ex8.m1.3.3.1.1.1.1.1.1.2.2" xref="A1.Ex8.m1.3.3.1.1.1.1.1.1.2.2.cmml">W</mi><mn id="A1.Ex8.m1.3.3.1.1.1.1.1.1.2.3" xref="A1.Ex8.m1.3.3.1.1.1.1.1.1.2.3.cmml">0</mn><mn id="A1.Ex8.m1.3.3.1.1.1.1.1.1.3" xref="A1.Ex8.m1.3.3.1.1.1.1.1.1.3.cmml">1</mn></msubsup><mo id="A1.Ex8.m1.3.3.1.1.2.2.2.4" xref="A1.Ex8.m1.3.3.1.1.2.2.3.cmml">,</mo><mi mathvariant="normal" id="A1.Ex8.m1.1.1" xref="A1.Ex8.m1.1.1.cmml">…</mi><mo id="A1.Ex8.m1.3.3.1.1.2.2.2.5" xref="A1.Ex8.m1.3.3.1.1.2.2.3.cmml">,</mo><msubsup id="A1.Ex8.m1.3.3.1.1.2.2.2.2" xref="A1.Ex8.m1.3.3.1.1.2.2.2.2.cmml"><mi id="A1.Ex8.m1.3.3.1.1.2.2.2.2.2.2" xref="A1.Ex8.m1.3.3.1.1.2.2.2.2.2.2.cmml">W</mi><mn id="A1.Ex8.m1.3.3.1.1.2.2.2.2.2.3" xref="A1.Ex8.m1.3.3.1.1.2.2.2.2.2.3.cmml">0</mn><mi id="A1.Ex8.m1.3.3.1.1.2.2.2.2.3" xref="A1.Ex8.m1.3.3.1.1.2.2.2.2.3.cmml">K</mi></msubsup><mo id="A1.Ex8.m1.3.3.1.1.2.2.2.6" xref="A1.Ex8.m1.3.3.1.1.2.2.3.cmml">;</mo><mover accent="true" id="A1.Ex8.m1.2.2" xref="A1.Ex8.m1.2.2.cmml"><mi id="A1.Ex8.m1.2.2.2" xref="A1.Ex8.m1.2.2.2.cmml">X</mi><mo id="A1.Ex8.m1.2.2.1" xref="A1.Ex8.m1.2.2.1.cmml">^</mo></mover><mo id="A1.Ex8.m1.3.3.1.1.2.2.2.7" xref="A1.Ex8.m1.3.3.1.1.2.2.3.cmml">)</mo></mrow></mrow><mo id="A1.Ex8.m1.3.3.1.1.7" xref="A1.Ex8.m1.3.3.1.1.7.cmml">=</mo><mrow id="A1.Ex8.m1.3.3.1.1.3" xref="A1.Ex8.m1.3.3.1.1.3.cmml"><mfrac id="A1.Ex8.m1.3.3.1.1.3.3" xref="A1.Ex8.m1.3.3.1.1.3.3.cmml"><mn id="A1.Ex8.m1.3.3.1.1.3.3.2" xref="A1.Ex8.m1.3.3.1.1.3.3.2.cmml">1</mn><mn id="A1.Ex8.m1.3.3.1.1.3.3.3" xref="A1.Ex8.m1.3.3.1.1.3.3.3.cmml">2</mn></mfrac><mo lspace="0em" rspace="0em" id="A1.Ex8.m1.3.3.1.1.3.2" xref="A1.Ex8.m1.3.3.1.1.3.2.cmml">​</mo><msubsup id="A1.Ex8.m1.3.3.1.1.3.1" xref="A1.Ex8.m1.3.3.1.1.3.1.cmml"><mrow id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.2.cmml"><mo maxsize="120%" minsize="120%" id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.2" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.2.1.cmml">‖</mo><mrow id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.cmml"><msup id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.cmml"><mover accent="true" id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.2" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.2.cmml"><mi id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.2.2" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.2.2.cmml">X</mi><mo id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.2.1" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.2.1.cmml">^</mo></mover><mrow id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.3" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.3.cmml"><mi id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.3.2" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.3.2.cmml">K</mi><mo id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.3.1" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.3.1.cmml">+</mo><mn id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.3.3" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.3.3.cmml">1</mn></mrow></msup><mo id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.1" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.1.cmml">−</mo><mi id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.3" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.3.cmml">Y</mi></mrow><mo maxsize="120%" minsize="120%" id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.3" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.2.1.cmml">‖</mo></mrow><mi id="A1.Ex8.m1.3.3.1.1.3.1.1.3" xref="A1.Ex8.m1.3.3.1.1.3.1.1.3.cmml">F</mi><mn id="A1.Ex8.m1.3.3.1.1.3.1.3" xref="A1.Ex8.m1.3.3.1.1.3.1.3.cmml">2</mn></msubsup></mrow><mo rspace="0.111em" id="A1.Ex8.m1.3.3.1.1.8" xref="A1.Ex8.m1.3.3.1.1.8.cmml">=</mo><mrow id="A1.Ex8.m1.3.3.1.1.9" xref="A1.Ex8.m1.3.3.1.1.9.cmml"><munderover id="A1.Ex8.m1.3.3.1.1.9.1" xref="A1.Ex8.m1.3.3.1.1.9.1.cmml"><mo movablelimits="false" id="A1.Ex8.m1.3.3.1.1.9.1.2.2" xref="A1.Ex8.m1.3.3.1.1.9.1.2.2.cmml">∑</mo><mrow id="A1.Ex8.m1.3.3.1.1.9.1.2.3" xref="A1.Ex8.m1.3.3.1.1.9.1.2.3.cmml"><mi id="A1.Ex8.m1.3.3.1.1.9.1.2.3.2" xref="A1.Ex8.m1.3.3.1.1.9.1.2.3.2.cmml">t</mi><mo id="A1.Ex8.m1.3.3.1.1.9.1.2.3.1" xref="A1.Ex8.m1.3.3.1.1.9.1.2.3.1.cmml">=</mo><mn id="A1.Ex8.m1.3.3.1.1.9.1.2.3.3" xref="A1.Ex8.m1.3.3.1.1.9.1.2.3.3.cmml">1</mn></mrow><mi id="A1.Ex8.m1.3.3.1.1.9.1.3" xref="A1.Ex8.m1.3.3.1.1.9.1.3.cmml">b</mi></munderover><msub id="A1.Ex8.m1.3.3.1.1.9.2" xref="A1.Ex8.m1.3.3.1.1.9.2.cmml"><mi id="A1.Ex8.m1.3.3.1.1.9.2.2" xref="A1.Ex8.m1.3.3.1.1.9.2.2.cmml">l</mi><mi id="A1.Ex8.m1.3.3.1.1.9.2.3" xref="A1.Ex8.m1.3.3.1.1.9.2.3.cmml">t</mi></msub></mrow></mrow><mo id="A1.Ex8.m1.3.3.1.2" xref="A1.Ex8.m1.3.3.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex8.m1.3b"><apply id="A1.Ex8.m1.3.3.1.1.cmml" xref="A1.Ex8.m1.3.3.1"><and id="A1.Ex8.m1.3.3.1.1a.cmml" xref="A1.Ex8.m1.3.3.1"></and><apply id="A1.Ex8.m1.3.3.1.1b.cmml" xref="A1.Ex8.m1.3.3.1"><eq id="A1.Ex8.m1.3.3.1.1.6.cmml" xref="A1.Ex8.m1.3.3.1.1.6"></eq><ci id="A1.Ex8.m1.3.3.1.1.5.cmml" xref="A1.Ex8.m1.3.3.1.1.5">𝑙</ci><apply id="A1.Ex8.m1.3.3.1.1.2.cmml" xref="A1.Ex8.m1.3.3.1.1.2"><times id="A1.Ex8.m1.3.3.1.1.2.3.cmml" xref="A1.Ex8.m1.3.3.1.1.2.3"></times><apply id="A1.Ex8.m1.3.3.1.1.2.4.cmml" xref="A1.Ex8.m1.3.3.1.1.2.4"><divide id="A1.Ex8.m1.3.3.1.1.2.4.1.cmml" xref="A1.Ex8.m1.3.3.1.1.2.4"></divide><cn type="integer" id="A1.Ex8.m1.3.3.1.1.2.4.2.cmml" xref="A1.Ex8.m1.3.3.1.1.2.4.2">1</cn><cn type="integer" id="A1.Ex8.m1.3.3.1.1.2.4.3.cmml" xref="A1.Ex8.m1.3.3.1.1.2.4.3">2</cn></apply><ci id="A1.Ex8.m1.3.3.1.1.2.5.cmml" xref="A1.Ex8.m1.3.3.1.1.2.5">ℓ</ci><vector id="A1.Ex8.m1.3.3.1.1.2.2.3.cmml" xref="A1.Ex8.m1.3.3.1.1.2.2.2"><apply id="A1.Ex8.m1.3.3.1.1.1.1.1.1.cmml" xref="A1.Ex8.m1.3.3.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.Ex8.m1.3.3.1.1.1.1.1.1.1.cmml" xref="A1.Ex8.m1.3.3.1.1.1.1.1.1">superscript</csymbol><apply id="A1.Ex8.m1.3.3.1.1.1.1.1.1.2.cmml" xref="A1.Ex8.m1.3.3.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.Ex8.m1.3.3.1.1.1.1.1.1.2.1.cmml" xref="A1.Ex8.m1.3.3.1.1.1.1.1.1">subscript</csymbol><ci id="A1.Ex8.m1.3.3.1.1.1.1.1.1.2.2.cmml" xref="A1.Ex8.m1.3.3.1.1.1.1.1.1.2.2">𝑊</ci><cn type="integer" id="A1.Ex8.m1.3.3.1.1.1.1.1.1.2.3.cmml" xref="A1.Ex8.m1.3.3.1.1.1.1.1.1.2.3">0</cn></apply><cn type="integer" id="A1.Ex8.m1.3.3.1.1.1.1.1.1.3.cmml" xref="A1.Ex8.m1.3.3.1.1.1.1.1.1.3">1</cn></apply><ci id="A1.Ex8.m1.1.1.cmml" xref="A1.Ex8.m1.1.1">…</ci><apply id="A1.Ex8.m1.3.3.1.1.2.2.2.2.cmml" xref="A1.Ex8.m1.3.3.1.1.2.2.2.2"><csymbol cd="ambiguous" id="A1.Ex8.m1.3.3.1.1.2.2.2.2.1.cmml" xref="A1.Ex8.m1.3.3.1.1.2.2.2.2">superscript</csymbol><apply id="A1.Ex8.m1.3.3.1.1.2.2.2.2.2.cmml" xref="A1.Ex8.m1.3.3.1.1.2.2.2.2"><csymbol cd="ambiguous" id="A1.Ex8.m1.3.3.1.1.2.2.2.2.2.1.cmml" xref="A1.Ex8.m1.3.3.1.1.2.2.2.2">subscript</csymbol><ci id="A1.Ex8.m1.3.3.1.1.2.2.2.2.2.2.cmml" xref="A1.Ex8.m1.3.3.1.1.2.2.2.2.2.2">𝑊</ci><cn type="integer" id="A1.Ex8.m1.3.3.1.1.2.2.2.2.2.3.cmml" xref="A1.Ex8.m1.3.3.1.1.2.2.2.2.2.3">0</cn></apply><ci id="A1.Ex8.m1.3.3.1.1.2.2.2.2.3.cmml" xref="A1.Ex8.m1.3.3.1.1.2.2.2.2.3">𝐾</ci></apply><apply id="A1.Ex8.m1.2.2.cmml" xref="A1.Ex8.m1.2.2"><ci id="A1.Ex8.m1.2.2.1.cmml" xref="A1.Ex8.m1.2.2.1">^</ci><ci id="A1.Ex8.m1.2.2.2.cmml" xref="A1.Ex8.m1.2.2.2">𝑋</ci></apply></vector></apply></apply><apply id="A1.Ex8.m1.3.3.1.1c.cmml" xref="A1.Ex8.m1.3.3.1"><eq id="A1.Ex8.m1.3.3.1.1.7.cmml" xref="A1.Ex8.m1.3.3.1.1.7"></eq><share href="#A1.Ex8.m1.3.3.1.1.2.cmml" id="A1.Ex8.m1.3.3.1.1d.cmml" xref="A1.Ex8.m1.3.3.1"></share><apply id="A1.Ex8.m1.3.3.1.1.3.cmml" xref="A1.Ex8.m1.3.3.1.1.3"><times id="A1.Ex8.m1.3.3.1.1.3.2.cmml" xref="A1.Ex8.m1.3.3.1.1.3.2"></times><apply id="A1.Ex8.m1.3.3.1.1.3.3.cmml" xref="A1.Ex8.m1.3.3.1.1.3.3"><divide id="A1.Ex8.m1.3.3.1.1.3.3.1.cmml" xref="A1.Ex8.m1.3.3.1.1.3.3"></divide><cn type="integer" id="A1.Ex8.m1.3.3.1.1.3.3.2.cmml" xref="A1.Ex8.m1.3.3.1.1.3.3.2">1</cn><cn type="integer" id="A1.Ex8.m1.3.3.1.1.3.3.3.cmml" xref="A1.Ex8.m1.3.3.1.1.3.3.3">2</cn></apply><apply id="A1.Ex8.m1.3.3.1.1.3.1.cmml" xref="A1.Ex8.m1.3.3.1.1.3.1"><csymbol cd="ambiguous" id="A1.Ex8.m1.3.3.1.1.3.1.2.cmml" xref="A1.Ex8.m1.3.3.1.1.3.1">superscript</csymbol><apply id="A1.Ex8.m1.3.3.1.1.3.1.1.cmml" xref="A1.Ex8.m1.3.3.1.1.3.1"><csymbol cd="ambiguous" id="A1.Ex8.m1.3.3.1.1.3.1.1.2.cmml" xref="A1.Ex8.m1.3.3.1.1.3.1">subscript</csymbol><apply id="A1.Ex8.m1.3.3.1.1.3.1.1.1.2.cmml" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1"><csymbol cd="latexml" id="A1.Ex8.m1.3.3.1.1.3.1.1.1.2.1.cmml" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.2">norm</csymbol><apply id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.cmml" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1"><minus id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.1.cmml" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.1"></minus><apply id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.cmml" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.1.cmml" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2">superscript</csymbol><apply id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.2.cmml" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.2"><ci id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.2.1.cmml" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.2.1">^</ci><ci id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.2.2.cmml" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.2.2">𝑋</ci></apply><apply id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.3.cmml" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.3"><plus id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.3.1.cmml" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.3.1"></plus><ci id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.3.2.cmml" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.3.2">𝐾</ci><cn type="integer" id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.3.3.cmml" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.2.3.3">1</cn></apply></apply><ci id="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.3.cmml" xref="A1.Ex8.m1.3.3.1.1.3.1.1.1.1.1.3">𝑌</ci></apply></apply><ci id="A1.Ex8.m1.3.3.1.1.3.1.1.3.cmml" xref="A1.Ex8.m1.3.3.1.1.3.1.1.3">𝐹</ci></apply><cn type="integer" id="A1.Ex8.m1.3.3.1.1.3.1.3.cmml" xref="A1.Ex8.m1.3.3.1.1.3.1.3">2</cn></apply></apply></apply><apply id="A1.Ex8.m1.3.3.1.1e.cmml" xref="A1.Ex8.m1.3.3.1"><eq id="A1.Ex8.m1.3.3.1.1.8.cmml" xref="A1.Ex8.m1.3.3.1.1.8"></eq><share href="#A1.Ex8.m1.3.3.1.1.3.cmml" id="A1.Ex8.m1.3.3.1.1f.cmml" xref="A1.Ex8.m1.3.3.1"></share><apply id="A1.Ex8.m1.3.3.1.1.9.cmml" xref="A1.Ex8.m1.3.3.1.1.9"><apply id="A1.Ex8.m1.3.3.1.1.9.1.cmml" xref="A1.Ex8.m1.3.3.1.1.9.1"><csymbol cd="ambiguous" id="A1.Ex8.m1.3.3.1.1.9.1.1.cmml" xref="A1.Ex8.m1.3.3.1.1.9.1">superscript</csymbol><apply id="A1.Ex8.m1.3.3.1.1.9.1.2.cmml" xref="A1.Ex8.m1.3.3.1.1.9.1"><csymbol cd="ambiguous" id="A1.Ex8.m1.3.3.1.1.9.1.2.1.cmml" xref="A1.Ex8.m1.3.3.1.1.9.1">subscript</csymbol><sum id="A1.Ex8.m1.3.3.1.1.9.1.2.2.cmml" xref="A1.Ex8.m1.3.3.1.1.9.1.2.2"></sum><apply id="A1.Ex8.m1.3.3.1.1.9.1.2.3.cmml" xref="A1.Ex8.m1.3.3.1.1.9.1.2.3"><eq id="A1.Ex8.m1.3.3.1.1.9.1.2.3.1.cmml" xref="A1.Ex8.m1.3.3.1.1.9.1.2.3.1"></eq><ci id="A1.Ex8.m1.3.3.1.1.9.1.2.3.2.cmml" xref="A1.Ex8.m1.3.3.1.1.9.1.2.3.2">𝑡</ci><cn type="integer" id="A1.Ex8.m1.3.3.1.1.9.1.2.3.3.cmml" xref="A1.Ex8.m1.3.3.1.1.9.1.2.3.3">1</cn></apply></apply><ci id="A1.Ex8.m1.3.3.1.1.9.1.3.cmml" xref="A1.Ex8.m1.3.3.1.1.9.1.3">𝑏</ci></apply><apply id="A1.Ex8.m1.3.3.1.1.9.2.cmml" xref="A1.Ex8.m1.3.3.1.1.9.2"><csymbol cd="ambiguous" id="A1.Ex8.m1.3.3.1.1.9.2.1.cmml" xref="A1.Ex8.m1.3.3.1.1.9.2">subscript</csymbol><ci id="A1.Ex8.m1.3.3.1.1.9.2.2.cmml" xref="A1.Ex8.m1.3.3.1.1.9.2.2">𝑙</ci><ci id="A1.Ex8.m1.3.3.1.1.9.2.3.cmml" xref="A1.Ex8.m1.3.3.1.1.9.2.3">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex8.m1.3c">l=\frac{1}{2}\ell\left(W_{0}^{1},\dots,W_{0}^{K};\hat{X}\right)=\frac{1}{2}\big{\|}\hat{X}^{K+1}-Y\big{\|}_{F}^{2}=\sum_{t=1}^{b}l_{t},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="A1.SS1.p3.3" class="ltx_p">where <math id="A1.SS1.p3.2.m1.1" class="ltx_Math" alttext="l_{t}=\frac{1}{2}\|\hat{x}^{K}_{t}-y_{t}\|^{2}" display="inline"><semantics id="A1.SS1.p3.2.m1.1a"><mrow id="A1.SS1.p3.2.m1.1.1" xref="A1.SS1.p3.2.m1.1.1.cmml"><msub id="A1.SS1.p3.2.m1.1.1.3" xref="A1.SS1.p3.2.m1.1.1.3.cmml"><mi id="A1.SS1.p3.2.m1.1.1.3.2" xref="A1.SS1.p3.2.m1.1.1.3.2.cmml">l</mi><mi id="A1.SS1.p3.2.m1.1.1.3.3" xref="A1.SS1.p3.2.m1.1.1.3.3.cmml">t</mi></msub><mo id="A1.SS1.p3.2.m1.1.1.2" xref="A1.SS1.p3.2.m1.1.1.2.cmml">=</mo><mrow id="A1.SS1.p3.2.m1.1.1.1" xref="A1.SS1.p3.2.m1.1.1.1.cmml"><mfrac id="A1.SS1.p3.2.m1.1.1.1.3" xref="A1.SS1.p3.2.m1.1.1.1.3.cmml"><mn id="A1.SS1.p3.2.m1.1.1.1.3.2" xref="A1.SS1.p3.2.m1.1.1.1.3.2.cmml">1</mn><mn id="A1.SS1.p3.2.m1.1.1.1.3.3" xref="A1.SS1.p3.2.m1.1.1.1.3.3.cmml">2</mn></mfrac><mo lspace="0em" rspace="0em" id="A1.SS1.p3.2.m1.1.1.1.2" xref="A1.SS1.p3.2.m1.1.1.1.2.cmml">​</mo><msup id="A1.SS1.p3.2.m1.1.1.1.1" xref="A1.SS1.p3.2.m1.1.1.1.1.cmml"><mrow id="A1.SS1.p3.2.m1.1.1.1.1.1.1" xref="A1.SS1.p3.2.m1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.SS1.p3.2.m1.1.1.1.1.1.1.2" xref="A1.SS1.p3.2.m1.1.1.1.1.1.2.1.cmml">‖</mo><mrow id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.cmml"><msubsup id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.cmml"><mover accent="true" id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.2.2" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.2.2.cmml"><mi id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.2.2.2" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.2.2.2.cmml">x</mi><mo id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.2.2.1" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.2.2.1.cmml">^</mo></mover><mi id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.3" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.3.cmml">t</mi><mi id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.2.3" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.2.3.cmml">K</mi></msubsup><mo id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.1" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.1.cmml">−</mo><msub id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.3" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.3.cmml"><mi id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.3.2" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.3.2.cmml">y</mi><mi id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.3.3" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.3.3.cmml">t</mi></msub></mrow><mo stretchy="false" id="A1.SS1.p3.2.m1.1.1.1.1.1.1.3" xref="A1.SS1.p3.2.m1.1.1.1.1.1.2.1.cmml">‖</mo></mrow><mn id="A1.SS1.p3.2.m1.1.1.1.1.3" xref="A1.SS1.p3.2.m1.1.1.1.1.3.cmml">2</mn></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS1.p3.2.m1.1b"><apply id="A1.SS1.p3.2.m1.1.1.cmml" xref="A1.SS1.p3.2.m1.1.1"><eq id="A1.SS1.p3.2.m1.1.1.2.cmml" xref="A1.SS1.p3.2.m1.1.1.2"></eq><apply id="A1.SS1.p3.2.m1.1.1.3.cmml" xref="A1.SS1.p3.2.m1.1.1.3"><csymbol cd="ambiguous" id="A1.SS1.p3.2.m1.1.1.3.1.cmml" xref="A1.SS1.p3.2.m1.1.1.3">subscript</csymbol><ci id="A1.SS1.p3.2.m1.1.1.3.2.cmml" xref="A1.SS1.p3.2.m1.1.1.3.2">𝑙</ci><ci id="A1.SS1.p3.2.m1.1.1.3.3.cmml" xref="A1.SS1.p3.2.m1.1.1.3.3">𝑡</ci></apply><apply id="A1.SS1.p3.2.m1.1.1.1.cmml" xref="A1.SS1.p3.2.m1.1.1.1"><times id="A1.SS1.p3.2.m1.1.1.1.2.cmml" xref="A1.SS1.p3.2.m1.1.1.1.2"></times><apply id="A1.SS1.p3.2.m1.1.1.1.3.cmml" xref="A1.SS1.p3.2.m1.1.1.1.3"><divide id="A1.SS1.p3.2.m1.1.1.1.3.1.cmml" xref="A1.SS1.p3.2.m1.1.1.1.3"></divide><cn type="integer" id="A1.SS1.p3.2.m1.1.1.1.3.2.cmml" xref="A1.SS1.p3.2.m1.1.1.1.3.2">1</cn><cn type="integer" id="A1.SS1.p3.2.m1.1.1.1.3.3.cmml" xref="A1.SS1.p3.2.m1.1.1.1.3.3">2</cn></apply><apply id="A1.SS1.p3.2.m1.1.1.1.1.cmml" xref="A1.SS1.p3.2.m1.1.1.1.1"><csymbol cd="ambiguous" id="A1.SS1.p3.2.m1.1.1.1.1.2.cmml" xref="A1.SS1.p3.2.m1.1.1.1.1">superscript</csymbol><apply id="A1.SS1.p3.2.m1.1.1.1.1.1.2.cmml" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1"><csymbol cd="latexml" id="A1.SS1.p3.2.m1.1.1.1.1.1.2.1.cmml" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.2">norm</csymbol><apply id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.cmml" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1"><minus id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.1.cmml" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.1"></minus><apply id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.cmml" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.1.cmml" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2">subscript</csymbol><apply id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.2.cmml" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.2.1.cmml" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2">superscript</csymbol><apply id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.2.2.cmml" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.2.2"><ci id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.2.2.1.cmml" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.2.2.1">^</ci><ci id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.2.2.2.cmml" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.2.2.2">𝑥</ci></apply><ci id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.2.3.cmml" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.2.3">𝐾</ci></apply><ci id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.3.cmml" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.2.3">𝑡</ci></apply><apply id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.3.cmml" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.3.1.cmml" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.3.2.cmml" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.3.2">𝑦</ci><ci id="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.3.3.cmml" xref="A1.SS1.p3.2.m1.1.1.1.1.1.1.1.3.3">𝑡</ci></apply></apply></apply><cn type="integer" id="A1.SS1.p3.2.m1.1.1.1.1.3.cmml" xref="A1.SS1.p3.2.m1.1.1.1.1.3">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p3.2.m1.1c">l_{t}=\frac{1}{2}\|\hat{x}^{K}_{t}-y_{t}\|^{2}</annotation></semantics></math> is the same as defined in Equation <a href="#S2.E4" title="In 2.3 Learning a self-supervised task for TTT ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, except scaled by <math id="A1.SS1.p3.3.m2.1" class="ltx_Math" alttext="1/2" display="inline"><semantics id="A1.SS1.p3.3.m2.1a"><mrow id="A1.SS1.p3.3.m2.1.1" xref="A1.SS1.p3.3.m2.1.1.cmml"><mn id="A1.SS1.p3.3.m2.1.1.2" xref="A1.SS1.p3.3.m2.1.1.2.cmml">1</mn><mo id="A1.SS1.p3.3.m2.1.1.1" xref="A1.SS1.p3.3.m2.1.1.1.cmml">/</mo><mn id="A1.SS1.p3.3.m2.1.1.3" xref="A1.SS1.p3.3.m2.1.1.3.cmml">2</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS1.p3.3.m2.1b"><apply id="A1.SS1.p3.3.m2.1.1.cmml" xref="A1.SS1.p3.3.m2.1.1"><divide id="A1.SS1.p3.3.m2.1.1.1.cmml" xref="A1.SS1.p3.3.m2.1.1.1"></divide><cn type="integer" id="A1.SS1.p3.3.m2.1.1.2.cmml" xref="A1.SS1.p3.3.m2.1.1.2">1</cn><cn type="integer" id="A1.SS1.p3.3.m2.1.1.3.cmml" xref="A1.SS1.p3.3.m2.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p3.3.m2.1c">1/2</annotation></semantics></math> for convenience.</p>
</div>
<div id="A1.SS1.p4" class="ltx_para">
<p id="A1.SS1.p4.1" class="ltx_p">All the operations above (except <math id="A1.SS1.p4.1.m1.1" class="ltx_Math" alttext="\sigma" display="inline"><semantics id="A1.SS1.p4.1.m1.1a"><mi id="A1.SS1.p4.1.m1.1.1" xref="A1.SS1.p4.1.m1.1.1.cmml">σ</mi><annotation-xml encoding="MathML-Content" id="A1.SS1.p4.1.m1.1b"><ci id="A1.SS1.p4.1.m1.1.1.cmml" xref="A1.SS1.p4.1.m1.1.1">𝜎</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p4.1.m1.1c">\sigma</annotation></semantics></math>) are <span id="A1.SS1.p4.1.1" class="ltx_text ltx_font_typewriter">matmul</span>s and <span id="A1.SS1.p4.1.2" class="ltx_text ltx_font_typewriter">sum</span>s, therefore are hardware efficient.
Both the primal form and the dual form share these initial operations.</p>
</div>
</section>
<section id="A1.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.2 </span>Primal form</h3>

<div id="A1.SS2.p1" class="ltx_para">
<p id="A1.SS2.p1.5" class="ltx_p">The primal form first computes <math id="A1.SS2.p1.1.m1.1" class="ltx_Math" alttext="G^{k}_{t}=\nabla_{W^{k}_{0}}l_{t}" display="inline"><semantics id="A1.SS2.p1.1.m1.1a"><mrow id="A1.SS2.p1.1.m1.1.1" xref="A1.SS2.p1.1.m1.1.1.cmml"><msubsup id="A1.SS2.p1.1.m1.1.1.2" xref="A1.SS2.p1.1.m1.1.1.2.cmml"><mi id="A1.SS2.p1.1.m1.1.1.2.2.2" xref="A1.SS2.p1.1.m1.1.1.2.2.2.cmml">G</mi><mi id="A1.SS2.p1.1.m1.1.1.2.3" xref="A1.SS2.p1.1.m1.1.1.2.3.cmml">t</mi><mi id="A1.SS2.p1.1.m1.1.1.2.2.3" xref="A1.SS2.p1.1.m1.1.1.2.2.3.cmml">k</mi></msubsup><mo id="A1.SS2.p1.1.m1.1.1.1" xref="A1.SS2.p1.1.m1.1.1.1.cmml">=</mo><mrow id="A1.SS2.p1.1.m1.1.1.3" xref="A1.SS2.p1.1.m1.1.1.3.cmml"><msub id="A1.SS2.p1.1.m1.1.1.3.1" xref="A1.SS2.p1.1.m1.1.1.3.1.cmml"><mo rspace="0.167em" id="A1.SS2.p1.1.m1.1.1.3.1.2" xref="A1.SS2.p1.1.m1.1.1.3.1.2.cmml">∇</mo><msubsup id="A1.SS2.p1.1.m1.1.1.3.1.3" xref="A1.SS2.p1.1.m1.1.1.3.1.3.cmml"><mi id="A1.SS2.p1.1.m1.1.1.3.1.3.2.2" xref="A1.SS2.p1.1.m1.1.1.3.1.3.2.2.cmml">W</mi><mn id="A1.SS2.p1.1.m1.1.1.3.1.3.3" xref="A1.SS2.p1.1.m1.1.1.3.1.3.3.cmml">0</mn><mi id="A1.SS2.p1.1.m1.1.1.3.1.3.2.3" xref="A1.SS2.p1.1.m1.1.1.3.1.3.2.3.cmml">k</mi></msubsup></msub><msub id="A1.SS2.p1.1.m1.1.1.3.2" xref="A1.SS2.p1.1.m1.1.1.3.2.cmml"><mi id="A1.SS2.p1.1.m1.1.1.3.2.2" xref="A1.SS2.p1.1.m1.1.1.3.2.2.cmml">l</mi><mi id="A1.SS2.p1.1.m1.1.1.3.2.3" xref="A1.SS2.p1.1.m1.1.1.3.2.3.cmml">t</mi></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.1.m1.1b"><apply id="A1.SS2.p1.1.m1.1.1.cmml" xref="A1.SS2.p1.1.m1.1.1"><eq id="A1.SS2.p1.1.m1.1.1.1.cmml" xref="A1.SS2.p1.1.m1.1.1.1"></eq><apply id="A1.SS2.p1.1.m1.1.1.2.cmml" xref="A1.SS2.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="A1.SS2.p1.1.m1.1.1.2.1.cmml" xref="A1.SS2.p1.1.m1.1.1.2">subscript</csymbol><apply id="A1.SS2.p1.1.m1.1.1.2.2.cmml" xref="A1.SS2.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="A1.SS2.p1.1.m1.1.1.2.2.1.cmml" xref="A1.SS2.p1.1.m1.1.1.2">superscript</csymbol><ci id="A1.SS2.p1.1.m1.1.1.2.2.2.cmml" xref="A1.SS2.p1.1.m1.1.1.2.2.2">𝐺</ci><ci id="A1.SS2.p1.1.m1.1.1.2.2.3.cmml" xref="A1.SS2.p1.1.m1.1.1.2.2.3">𝑘</ci></apply><ci id="A1.SS2.p1.1.m1.1.1.2.3.cmml" xref="A1.SS2.p1.1.m1.1.1.2.3">𝑡</ci></apply><apply id="A1.SS2.p1.1.m1.1.1.3.cmml" xref="A1.SS2.p1.1.m1.1.1.3"><apply id="A1.SS2.p1.1.m1.1.1.3.1.cmml" xref="A1.SS2.p1.1.m1.1.1.3.1"><csymbol cd="ambiguous" id="A1.SS2.p1.1.m1.1.1.3.1.1.cmml" xref="A1.SS2.p1.1.m1.1.1.3.1">subscript</csymbol><ci id="A1.SS2.p1.1.m1.1.1.3.1.2.cmml" xref="A1.SS2.p1.1.m1.1.1.3.1.2">∇</ci><apply id="A1.SS2.p1.1.m1.1.1.3.1.3.cmml" xref="A1.SS2.p1.1.m1.1.1.3.1.3"><csymbol cd="ambiguous" id="A1.SS2.p1.1.m1.1.1.3.1.3.1.cmml" xref="A1.SS2.p1.1.m1.1.1.3.1.3">subscript</csymbol><apply id="A1.SS2.p1.1.m1.1.1.3.1.3.2.cmml" xref="A1.SS2.p1.1.m1.1.1.3.1.3"><csymbol cd="ambiguous" id="A1.SS2.p1.1.m1.1.1.3.1.3.2.1.cmml" xref="A1.SS2.p1.1.m1.1.1.3.1.3">superscript</csymbol><ci id="A1.SS2.p1.1.m1.1.1.3.1.3.2.2.cmml" xref="A1.SS2.p1.1.m1.1.1.3.1.3.2.2">𝑊</ci><ci id="A1.SS2.p1.1.m1.1.1.3.1.3.2.3.cmml" xref="A1.SS2.p1.1.m1.1.1.3.1.3.2.3">𝑘</ci></apply><cn type="integer" id="A1.SS2.p1.1.m1.1.1.3.1.3.3.cmml" xref="A1.SS2.p1.1.m1.1.1.3.1.3.3">0</cn></apply></apply><apply id="A1.SS2.p1.1.m1.1.1.3.2.cmml" xref="A1.SS2.p1.1.m1.1.1.3.2"><csymbol cd="ambiguous" id="A1.SS2.p1.1.m1.1.1.3.2.1.cmml" xref="A1.SS2.p1.1.m1.1.1.3.2">subscript</csymbol><ci id="A1.SS2.p1.1.m1.1.1.3.2.2.cmml" xref="A1.SS2.p1.1.m1.1.1.3.2.2">𝑙</ci><ci id="A1.SS2.p1.1.m1.1.1.3.2.3.cmml" xref="A1.SS2.p1.1.m1.1.1.3.2.3">𝑡</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.1.m1.1c">G^{k}_{t}=\nabla_{W^{k}_{0}}l_{t}</annotation></semantics></math> for <math id="A1.SS2.p1.2.m2.3" class="ltx_Math" alttext="t=1,\dots,b" display="inline"><semantics id="A1.SS2.p1.2.m2.3a"><mrow id="A1.SS2.p1.2.m2.3.4" xref="A1.SS2.p1.2.m2.3.4.cmml"><mi id="A1.SS2.p1.2.m2.3.4.2" xref="A1.SS2.p1.2.m2.3.4.2.cmml">t</mi><mo id="A1.SS2.p1.2.m2.3.4.1" xref="A1.SS2.p1.2.m2.3.4.1.cmml">=</mo><mrow id="A1.SS2.p1.2.m2.3.4.3.2" xref="A1.SS2.p1.2.m2.3.4.3.1.cmml"><mn id="A1.SS2.p1.2.m2.1.1" xref="A1.SS2.p1.2.m2.1.1.cmml">1</mn><mo id="A1.SS2.p1.2.m2.3.4.3.2.1" xref="A1.SS2.p1.2.m2.3.4.3.1.cmml">,</mo><mi mathvariant="normal" id="A1.SS2.p1.2.m2.2.2" xref="A1.SS2.p1.2.m2.2.2.cmml">…</mi><mo id="A1.SS2.p1.2.m2.3.4.3.2.2" xref="A1.SS2.p1.2.m2.3.4.3.1.cmml">,</mo><mi id="A1.SS2.p1.2.m2.3.3" xref="A1.SS2.p1.2.m2.3.3.cmml">b</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.2.m2.3b"><apply id="A1.SS2.p1.2.m2.3.4.cmml" xref="A1.SS2.p1.2.m2.3.4"><eq id="A1.SS2.p1.2.m2.3.4.1.cmml" xref="A1.SS2.p1.2.m2.3.4.1"></eq><ci id="A1.SS2.p1.2.m2.3.4.2.cmml" xref="A1.SS2.p1.2.m2.3.4.2">𝑡</ci><list id="A1.SS2.p1.2.m2.3.4.3.1.cmml" xref="A1.SS2.p1.2.m2.3.4.3.2"><cn type="integer" id="A1.SS2.p1.2.m2.1.1.cmml" xref="A1.SS2.p1.2.m2.1.1">1</cn><ci id="A1.SS2.p1.2.m2.2.2.cmml" xref="A1.SS2.p1.2.m2.2.2">…</ci><ci id="A1.SS2.p1.2.m2.3.3.cmml" xref="A1.SS2.p1.2.m2.3.3">𝑏</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.2.m2.3c">t=1,\dots,b</annotation></semantics></math>, then updates <math id="A1.SS2.p1.3.m3.1" class="ltx_Math" alttext="W^{k}_{t}=W_{0}^{k}-\sum_{s=1}^{t}G^{k}_{s}" display="inline"><semantics id="A1.SS2.p1.3.m3.1a"><mrow id="A1.SS2.p1.3.m3.1.1" xref="A1.SS2.p1.3.m3.1.1.cmml"><msubsup id="A1.SS2.p1.3.m3.1.1.2" xref="A1.SS2.p1.3.m3.1.1.2.cmml"><mi id="A1.SS2.p1.3.m3.1.1.2.2.2" xref="A1.SS2.p1.3.m3.1.1.2.2.2.cmml">W</mi><mi id="A1.SS2.p1.3.m3.1.1.2.3" xref="A1.SS2.p1.3.m3.1.1.2.3.cmml">t</mi><mi id="A1.SS2.p1.3.m3.1.1.2.2.3" xref="A1.SS2.p1.3.m3.1.1.2.2.3.cmml">k</mi></msubsup><mo id="A1.SS2.p1.3.m3.1.1.1" xref="A1.SS2.p1.3.m3.1.1.1.cmml">=</mo><mrow id="A1.SS2.p1.3.m3.1.1.3" xref="A1.SS2.p1.3.m3.1.1.3.cmml"><msubsup id="A1.SS2.p1.3.m3.1.1.3.2" xref="A1.SS2.p1.3.m3.1.1.3.2.cmml"><mi id="A1.SS2.p1.3.m3.1.1.3.2.2.2" xref="A1.SS2.p1.3.m3.1.1.3.2.2.2.cmml">W</mi><mn id="A1.SS2.p1.3.m3.1.1.3.2.2.3" xref="A1.SS2.p1.3.m3.1.1.3.2.2.3.cmml">0</mn><mi id="A1.SS2.p1.3.m3.1.1.3.2.3" xref="A1.SS2.p1.3.m3.1.1.3.2.3.cmml">k</mi></msubsup><mo rspace="0.055em" id="A1.SS2.p1.3.m3.1.1.3.1" xref="A1.SS2.p1.3.m3.1.1.3.1.cmml">−</mo><mrow id="A1.SS2.p1.3.m3.1.1.3.3" xref="A1.SS2.p1.3.m3.1.1.3.3.cmml"><msubsup id="A1.SS2.p1.3.m3.1.1.3.3.1" xref="A1.SS2.p1.3.m3.1.1.3.3.1.cmml"><mo id="A1.SS2.p1.3.m3.1.1.3.3.1.2.2" xref="A1.SS2.p1.3.m3.1.1.3.3.1.2.2.cmml">∑</mo><mrow id="A1.SS2.p1.3.m3.1.1.3.3.1.2.3" xref="A1.SS2.p1.3.m3.1.1.3.3.1.2.3.cmml"><mi id="A1.SS2.p1.3.m3.1.1.3.3.1.2.3.2" xref="A1.SS2.p1.3.m3.1.1.3.3.1.2.3.2.cmml">s</mi><mo id="A1.SS2.p1.3.m3.1.1.3.3.1.2.3.1" xref="A1.SS2.p1.3.m3.1.1.3.3.1.2.3.1.cmml">=</mo><mn id="A1.SS2.p1.3.m3.1.1.3.3.1.2.3.3" xref="A1.SS2.p1.3.m3.1.1.3.3.1.2.3.3.cmml">1</mn></mrow><mi id="A1.SS2.p1.3.m3.1.1.3.3.1.3" xref="A1.SS2.p1.3.m3.1.1.3.3.1.3.cmml">t</mi></msubsup><msubsup id="A1.SS2.p1.3.m3.1.1.3.3.2" xref="A1.SS2.p1.3.m3.1.1.3.3.2.cmml"><mi id="A1.SS2.p1.3.m3.1.1.3.3.2.2.2" xref="A1.SS2.p1.3.m3.1.1.3.3.2.2.2.cmml">G</mi><mi id="A1.SS2.p1.3.m3.1.1.3.3.2.3" xref="A1.SS2.p1.3.m3.1.1.3.3.2.3.cmml">s</mi><mi id="A1.SS2.p1.3.m3.1.1.3.3.2.2.3" xref="A1.SS2.p1.3.m3.1.1.3.3.2.2.3.cmml">k</mi></msubsup></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.3.m3.1b"><apply id="A1.SS2.p1.3.m3.1.1.cmml" xref="A1.SS2.p1.3.m3.1.1"><eq id="A1.SS2.p1.3.m3.1.1.1.cmml" xref="A1.SS2.p1.3.m3.1.1.1"></eq><apply id="A1.SS2.p1.3.m3.1.1.2.cmml" xref="A1.SS2.p1.3.m3.1.1.2"><csymbol cd="ambiguous" id="A1.SS2.p1.3.m3.1.1.2.1.cmml" xref="A1.SS2.p1.3.m3.1.1.2">subscript</csymbol><apply id="A1.SS2.p1.3.m3.1.1.2.2.cmml" xref="A1.SS2.p1.3.m3.1.1.2"><csymbol cd="ambiguous" id="A1.SS2.p1.3.m3.1.1.2.2.1.cmml" xref="A1.SS2.p1.3.m3.1.1.2">superscript</csymbol><ci id="A1.SS2.p1.3.m3.1.1.2.2.2.cmml" xref="A1.SS2.p1.3.m3.1.1.2.2.2">𝑊</ci><ci id="A1.SS2.p1.3.m3.1.1.2.2.3.cmml" xref="A1.SS2.p1.3.m3.1.1.2.2.3">𝑘</ci></apply><ci id="A1.SS2.p1.3.m3.1.1.2.3.cmml" xref="A1.SS2.p1.3.m3.1.1.2.3">𝑡</ci></apply><apply id="A1.SS2.p1.3.m3.1.1.3.cmml" xref="A1.SS2.p1.3.m3.1.1.3"><minus id="A1.SS2.p1.3.m3.1.1.3.1.cmml" xref="A1.SS2.p1.3.m3.1.1.3.1"></minus><apply id="A1.SS2.p1.3.m3.1.1.3.2.cmml" xref="A1.SS2.p1.3.m3.1.1.3.2"><csymbol cd="ambiguous" id="A1.SS2.p1.3.m3.1.1.3.2.1.cmml" xref="A1.SS2.p1.3.m3.1.1.3.2">superscript</csymbol><apply id="A1.SS2.p1.3.m3.1.1.3.2.2.cmml" xref="A1.SS2.p1.3.m3.1.1.3.2"><csymbol cd="ambiguous" id="A1.SS2.p1.3.m3.1.1.3.2.2.1.cmml" xref="A1.SS2.p1.3.m3.1.1.3.2">subscript</csymbol><ci id="A1.SS2.p1.3.m3.1.1.3.2.2.2.cmml" xref="A1.SS2.p1.3.m3.1.1.3.2.2.2">𝑊</ci><cn type="integer" id="A1.SS2.p1.3.m3.1.1.3.2.2.3.cmml" xref="A1.SS2.p1.3.m3.1.1.3.2.2.3">0</cn></apply><ci id="A1.SS2.p1.3.m3.1.1.3.2.3.cmml" xref="A1.SS2.p1.3.m3.1.1.3.2.3">𝑘</ci></apply><apply id="A1.SS2.p1.3.m3.1.1.3.3.cmml" xref="A1.SS2.p1.3.m3.1.1.3.3"><apply id="A1.SS2.p1.3.m3.1.1.3.3.1.cmml" xref="A1.SS2.p1.3.m3.1.1.3.3.1"><csymbol cd="ambiguous" id="A1.SS2.p1.3.m3.1.1.3.3.1.1.cmml" xref="A1.SS2.p1.3.m3.1.1.3.3.1">superscript</csymbol><apply id="A1.SS2.p1.3.m3.1.1.3.3.1.2.cmml" xref="A1.SS2.p1.3.m3.1.1.3.3.1"><csymbol cd="ambiguous" id="A1.SS2.p1.3.m3.1.1.3.3.1.2.1.cmml" xref="A1.SS2.p1.3.m3.1.1.3.3.1">subscript</csymbol><sum id="A1.SS2.p1.3.m3.1.1.3.3.1.2.2.cmml" xref="A1.SS2.p1.3.m3.1.1.3.3.1.2.2"></sum><apply id="A1.SS2.p1.3.m3.1.1.3.3.1.2.3.cmml" xref="A1.SS2.p1.3.m3.1.1.3.3.1.2.3"><eq id="A1.SS2.p1.3.m3.1.1.3.3.1.2.3.1.cmml" xref="A1.SS2.p1.3.m3.1.1.3.3.1.2.3.1"></eq><ci id="A1.SS2.p1.3.m3.1.1.3.3.1.2.3.2.cmml" xref="A1.SS2.p1.3.m3.1.1.3.3.1.2.3.2">𝑠</ci><cn type="integer" id="A1.SS2.p1.3.m3.1.1.3.3.1.2.3.3.cmml" xref="A1.SS2.p1.3.m3.1.1.3.3.1.2.3.3">1</cn></apply></apply><ci id="A1.SS2.p1.3.m3.1.1.3.3.1.3.cmml" xref="A1.SS2.p1.3.m3.1.1.3.3.1.3">𝑡</ci></apply><apply id="A1.SS2.p1.3.m3.1.1.3.3.2.cmml" xref="A1.SS2.p1.3.m3.1.1.3.3.2"><csymbol cd="ambiguous" id="A1.SS2.p1.3.m3.1.1.3.3.2.1.cmml" xref="A1.SS2.p1.3.m3.1.1.3.3.2">subscript</csymbol><apply id="A1.SS2.p1.3.m3.1.1.3.3.2.2.cmml" xref="A1.SS2.p1.3.m3.1.1.3.3.2"><csymbol cd="ambiguous" id="A1.SS2.p1.3.m3.1.1.3.3.2.2.1.cmml" xref="A1.SS2.p1.3.m3.1.1.3.3.2">superscript</csymbol><ci id="A1.SS2.p1.3.m3.1.1.3.3.2.2.2.cmml" xref="A1.SS2.p1.3.m3.1.1.3.3.2.2.2">𝐺</ci><ci id="A1.SS2.p1.3.m3.1.1.3.3.2.2.3.cmml" xref="A1.SS2.p1.3.m3.1.1.3.3.2.2.3">𝑘</ci></apply><ci id="A1.SS2.p1.3.m3.1.1.3.3.2.3.cmml" xref="A1.SS2.p1.3.m3.1.1.3.3.2.3">𝑠</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.3.m3.1c">W^{k}_{t}=W_{0}^{k}-\sum_{s=1}^{t}G^{k}_{s}</annotation></semantics></math>.
Finally, given <math id="A1.SS2.p1.4.m4.3" class="ltx_Math" alttext="\bar{X}^{1}=[\bar{x}_{1}^{1},\dots,\bar{x}_{b}^{1}]=\bar{X}" display="inline"><semantics id="A1.SS2.p1.4.m4.3a"><mrow id="A1.SS2.p1.4.m4.3.3" xref="A1.SS2.p1.4.m4.3.3.cmml"><msup id="A1.SS2.p1.4.m4.3.3.4" xref="A1.SS2.p1.4.m4.3.3.4.cmml"><mover accent="true" id="A1.SS2.p1.4.m4.3.3.4.2" xref="A1.SS2.p1.4.m4.3.3.4.2.cmml"><mi id="A1.SS2.p1.4.m4.3.3.4.2.2" xref="A1.SS2.p1.4.m4.3.3.4.2.2.cmml">X</mi><mo id="A1.SS2.p1.4.m4.3.3.4.2.1" xref="A1.SS2.p1.4.m4.3.3.4.2.1.cmml">¯</mo></mover><mn id="A1.SS2.p1.4.m4.3.3.4.3" xref="A1.SS2.p1.4.m4.3.3.4.3.cmml">1</mn></msup><mo id="A1.SS2.p1.4.m4.3.3.5" xref="A1.SS2.p1.4.m4.3.3.5.cmml">=</mo><mrow id="A1.SS2.p1.4.m4.3.3.2.2" xref="A1.SS2.p1.4.m4.3.3.2.3.cmml"><mo stretchy="false" id="A1.SS2.p1.4.m4.3.3.2.2.3" xref="A1.SS2.p1.4.m4.3.3.2.3.cmml">[</mo><msubsup id="A1.SS2.p1.4.m4.2.2.1.1.1" xref="A1.SS2.p1.4.m4.2.2.1.1.1.cmml"><mover accent="true" id="A1.SS2.p1.4.m4.2.2.1.1.1.2.2" xref="A1.SS2.p1.4.m4.2.2.1.1.1.2.2.cmml"><mi id="A1.SS2.p1.4.m4.2.2.1.1.1.2.2.2" xref="A1.SS2.p1.4.m4.2.2.1.1.1.2.2.2.cmml">x</mi><mo id="A1.SS2.p1.4.m4.2.2.1.1.1.2.2.1" xref="A1.SS2.p1.4.m4.2.2.1.1.1.2.2.1.cmml">¯</mo></mover><mn id="A1.SS2.p1.4.m4.2.2.1.1.1.2.3" xref="A1.SS2.p1.4.m4.2.2.1.1.1.2.3.cmml">1</mn><mn id="A1.SS2.p1.4.m4.2.2.1.1.1.3" xref="A1.SS2.p1.4.m4.2.2.1.1.1.3.cmml">1</mn></msubsup><mo id="A1.SS2.p1.4.m4.3.3.2.2.4" xref="A1.SS2.p1.4.m4.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="A1.SS2.p1.4.m4.1.1" xref="A1.SS2.p1.4.m4.1.1.cmml">…</mi><mo id="A1.SS2.p1.4.m4.3.3.2.2.5" xref="A1.SS2.p1.4.m4.3.3.2.3.cmml">,</mo><msubsup id="A1.SS2.p1.4.m4.3.3.2.2.2" xref="A1.SS2.p1.4.m4.3.3.2.2.2.cmml"><mover accent="true" id="A1.SS2.p1.4.m4.3.3.2.2.2.2.2" xref="A1.SS2.p1.4.m4.3.3.2.2.2.2.2.cmml"><mi id="A1.SS2.p1.4.m4.3.3.2.2.2.2.2.2" xref="A1.SS2.p1.4.m4.3.3.2.2.2.2.2.2.cmml">x</mi><mo id="A1.SS2.p1.4.m4.3.3.2.2.2.2.2.1" xref="A1.SS2.p1.4.m4.3.3.2.2.2.2.2.1.cmml">¯</mo></mover><mi id="A1.SS2.p1.4.m4.3.3.2.2.2.2.3" xref="A1.SS2.p1.4.m4.3.3.2.2.2.2.3.cmml">b</mi><mn id="A1.SS2.p1.4.m4.3.3.2.2.2.3" xref="A1.SS2.p1.4.m4.3.3.2.2.2.3.cmml">1</mn></msubsup><mo stretchy="false" id="A1.SS2.p1.4.m4.3.3.2.2.6" xref="A1.SS2.p1.4.m4.3.3.2.3.cmml">]</mo></mrow><mo id="A1.SS2.p1.4.m4.3.3.6" xref="A1.SS2.p1.4.m4.3.3.6.cmml">=</mo><mover accent="true" id="A1.SS2.p1.4.m4.3.3.7" xref="A1.SS2.p1.4.m4.3.3.7.cmml"><mi id="A1.SS2.p1.4.m4.3.3.7.2" xref="A1.SS2.p1.4.m4.3.3.7.2.cmml">X</mi><mo id="A1.SS2.p1.4.m4.3.3.7.1" xref="A1.SS2.p1.4.m4.3.3.7.1.cmml">¯</mo></mover></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.4.m4.3b"><apply id="A1.SS2.p1.4.m4.3.3.cmml" xref="A1.SS2.p1.4.m4.3.3"><and id="A1.SS2.p1.4.m4.3.3a.cmml" xref="A1.SS2.p1.4.m4.3.3"></and><apply id="A1.SS2.p1.4.m4.3.3b.cmml" xref="A1.SS2.p1.4.m4.3.3"><eq id="A1.SS2.p1.4.m4.3.3.5.cmml" xref="A1.SS2.p1.4.m4.3.3.5"></eq><apply id="A1.SS2.p1.4.m4.3.3.4.cmml" xref="A1.SS2.p1.4.m4.3.3.4"><csymbol cd="ambiguous" id="A1.SS2.p1.4.m4.3.3.4.1.cmml" xref="A1.SS2.p1.4.m4.3.3.4">superscript</csymbol><apply id="A1.SS2.p1.4.m4.3.3.4.2.cmml" xref="A1.SS2.p1.4.m4.3.3.4.2"><ci id="A1.SS2.p1.4.m4.3.3.4.2.1.cmml" xref="A1.SS2.p1.4.m4.3.3.4.2.1">¯</ci><ci id="A1.SS2.p1.4.m4.3.3.4.2.2.cmml" xref="A1.SS2.p1.4.m4.3.3.4.2.2">𝑋</ci></apply><cn type="integer" id="A1.SS2.p1.4.m4.3.3.4.3.cmml" xref="A1.SS2.p1.4.m4.3.3.4.3">1</cn></apply><list id="A1.SS2.p1.4.m4.3.3.2.3.cmml" xref="A1.SS2.p1.4.m4.3.3.2.2"><apply id="A1.SS2.p1.4.m4.2.2.1.1.1.cmml" xref="A1.SS2.p1.4.m4.2.2.1.1.1"><csymbol cd="ambiguous" id="A1.SS2.p1.4.m4.2.2.1.1.1.1.cmml" xref="A1.SS2.p1.4.m4.2.2.1.1.1">superscript</csymbol><apply id="A1.SS2.p1.4.m4.2.2.1.1.1.2.cmml" xref="A1.SS2.p1.4.m4.2.2.1.1.1"><csymbol cd="ambiguous" id="A1.SS2.p1.4.m4.2.2.1.1.1.2.1.cmml" xref="A1.SS2.p1.4.m4.2.2.1.1.1">subscript</csymbol><apply id="A1.SS2.p1.4.m4.2.2.1.1.1.2.2.cmml" xref="A1.SS2.p1.4.m4.2.2.1.1.1.2.2"><ci id="A1.SS2.p1.4.m4.2.2.1.1.1.2.2.1.cmml" xref="A1.SS2.p1.4.m4.2.2.1.1.1.2.2.1">¯</ci><ci id="A1.SS2.p1.4.m4.2.2.1.1.1.2.2.2.cmml" xref="A1.SS2.p1.4.m4.2.2.1.1.1.2.2.2">𝑥</ci></apply><cn type="integer" id="A1.SS2.p1.4.m4.2.2.1.1.1.2.3.cmml" xref="A1.SS2.p1.4.m4.2.2.1.1.1.2.3">1</cn></apply><cn type="integer" id="A1.SS2.p1.4.m4.2.2.1.1.1.3.cmml" xref="A1.SS2.p1.4.m4.2.2.1.1.1.3">1</cn></apply><ci id="A1.SS2.p1.4.m4.1.1.cmml" xref="A1.SS2.p1.4.m4.1.1">…</ci><apply id="A1.SS2.p1.4.m4.3.3.2.2.2.cmml" xref="A1.SS2.p1.4.m4.3.3.2.2.2"><csymbol cd="ambiguous" id="A1.SS2.p1.4.m4.3.3.2.2.2.1.cmml" xref="A1.SS2.p1.4.m4.3.3.2.2.2">superscript</csymbol><apply id="A1.SS2.p1.4.m4.3.3.2.2.2.2.cmml" xref="A1.SS2.p1.4.m4.3.3.2.2.2"><csymbol cd="ambiguous" id="A1.SS2.p1.4.m4.3.3.2.2.2.2.1.cmml" xref="A1.SS2.p1.4.m4.3.3.2.2.2">subscript</csymbol><apply id="A1.SS2.p1.4.m4.3.3.2.2.2.2.2.cmml" xref="A1.SS2.p1.4.m4.3.3.2.2.2.2.2"><ci id="A1.SS2.p1.4.m4.3.3.2.2.2.2.2.1.cmml" xref="A1.SS2.p1.4.m4.3.3.2.2.2.2.2.1">¯</ci><ci id="A1.SS2.p1.4.m4.3.3.2.2.2.2.2.2.cmml" xref="A1.SS2.p1.4.m4.3.3.2.2.2.2.2.2">𝑥</ci></apply><ci id="A1.SS2.p1.4.m4.3.3.2.2.2.2.3.cmml" xref="A1.SS2.p1.4.m4.3.3.2.2.2.2.3">𝑏</ci></apply><cn type="integer" id="A1.SS2.p1.4.m4.3.3.2.2.2.3.cmml" xref="A1.SS2.p1.4.m4.3.3.2.2.2.3">1</cn></apply></list></apply><apply id="A1.SS2.p1.4.m4.3.3c.cmml" xref="A1.SS2.p1.4.m4.3.3"><eq id="A1.SS2.p1.4.m4.3.3.6.cmml" xref="A1.SS2.p1.4.m4.3.3.6"></eq><share href="#A1.SS2.p1.4.m4.3.3.2.cmml" id="A1.SS2.p1.4.m4.3.3d.cmml" xref="A1.SS2.p1.4.m4.3.3"></share><apply id="A1.SS2.p1.4.m4.3.3.7.cmml" xref="A1.SS2.p1.4.m4.3.3.7"><ci id="A1.SS2.p1.4.m4.3.3.7.1.cmml" xref="A1.SS2.p1.4.m4.3.3.7.1">¯</ci><ci id="A1.SS2.p1.4.m4.3.3.7.2.cmml" xref="A1.SS2.p1.4.m4.3.3.7.2">𝑋</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.4.m4.3c">\bar{X}^{1}=[\bar{x}_{1}^{1},\dots,\bar{x}_{b}^{1}]=\bar{X}</annotation></semantics></math>, the primal form repeats the forward pass with the updated <math id="A1.SS2.p1.5.m5.1" class="ltx_Math" alttext="W" display="inline"><semantics id="A1.SS2.p1.5.m5.1a"><mi id="A1.SS2.p1.5.m5.1.1" xref="A1.SS2.p1.5.m5.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.5.m5.1b"><ci id="A1.SS2.p1.5.m5.1.1.cmml" xref="A1.SS2.p1.5.m5.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.5.m5.1c">W</annotation></semantics></math>s.</p>
</div>
<div id="A1.SS2.p2" class="ltx_para">
<p id="A1.SS2.p2.1" class="ltx_p">For <math id="A1.SS2.p2.1.m1.3" class="ltx_Math" alttext="k=1,\dots,K" display="inline"><semantics id="A1.SS2.p2.1.m1.3a"><mrow id="A1.SS2.p2.1.m1.3.4" xref="A1.SS2.p2.1.m1.3.4.cmml"><mi id="A1.SS2.p2.1.m1.3.4.2" xref="A1.SS2.p2.1.m1.3.4.2.cmml">k</mi><mo id="A1.SS2.p2.1.m1.3.4.1" xref="A1.SS2.p2.1.m1.3.4.1.cmml">=</mo><mrow id="A1.SS2.p2.1.m1.3.4.3.2" xref="A1.SS2.p2.1.m1.3.4.3.1.cmml"><mn id="A1.SS2.p2.1.m1.1.1" xref="A1.SS2.p2.1.m1.1.1.cmml">1</mn><mo id="A1.SS2.p2.1.m1.3.4.3.2.1" xref="A1.SS2.p2.1.m1.3.4.3.1.cmml">,</mo><mi mathvariant="normal" id="A1.SS2.p2.1.m1.2.2" xref="A1.SS2.p2.1.m1.2.2.cmml">…</mi><mo id="A1.SS2.p2.1.m1.3.4.3.2.2" xref="A1.SS2.p2.1.m1.3.4.3.1.cmml">,</mo><mi id="A1.SS2.p2.1.m1.3.3" xref="A1.SS2.p2.1.m1.3.3.cmml">K</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.p2.1.m1.3b"><apply id="A1.SS2.p2.1.m1.3.4.cmml" xref="A1.SS2.p2.1.m1.3.4"><eq id="A1.SS2.p2.1.m1.3.4.1.cmml" xref="A1.SS2.p2.1.m1.3.4.1"></eq><ci id="A1.SS2.p2.1.m1.3.4.2.cmml" xref="A1.SS2.p2.1.m1.3.4.2">𝑘</ci><list id="A1.SS2.p2.1.m1.3.4.3.1.cmml" xref="A1.SS2.p2.1.m1.3.4.3.2"><cn type="integer" id="A1.SS2.p2.1.m1.1.1.cmml" xref="A1.SS2.p2.1.m1.1.1">1</cn><ci id="A1.SS2.p2.1.m1.2.2.cmml" xref="A1.SS2.p2.1.m1.2.2">…</ci><ci id="A1.SS2.p2.1.m1.3.3.cmml" xref="A1.SS2.p2.1.m1.3.3">𝐾</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p2.1.m1.3c">k=1,\dots,K</annotation></semantics></math>:</p>
<ul id="A1.I2" class="ltx_itemize">
<li id="A1.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I2.i1.p1" class="ltx_para">
<p id="A1.I2.i1.p1.2" class="ltx_p"><math id="A1.I2.i1.p1.1.m1.1" class="ltx_Math" alttext="\bar{z}_{t}^{k}=W_{t}^{k}\,\bar{x}_{t}^{k}" display="inline"><semantics id="A1.I2.i1.p1.1.m1.1a"><mrow id="A1.I2.i1.p1.1.m1.1.1" xref="A1.I2.i1.p1.1.m1.1.1.cmml"><msubsup id="A1.I2.i1.p1.1.m1.1.1.2" xref="A1.I2.i1.p1.1.m1.1.1.2.cmml"><mover accent="true" id="A1.I2.i1.p1.1.m1.1.1.2.2.2" xref="A1.I2.i1.p1.1.m1.1.1.2.2.2.cmml"><mi id="A1.I2.i1.p1.1.m1.1.1.2.2.2.2" xref="A1.I2.i1.p1.1.m1.1.1.2.2.2.2.cmml">z</mi><mo id="A1.I2.i1.p1.1.m1.1.1.2.2.2.1" xref="A1.I2.i1.p1.1.m1.1.1.2.2.2.1.cmml">¯</mo></mover><mi id="A1.I2.i1.p1.1.m1.1.1.2.2.3" xref="A1.I2.i1.p1.1.m1.1.1.2.2.3.cmml">t</mi><mi id="A1.I2.i1.p1.1.m1.1.1.2.3" xref="A1.I2.i1.p1.1.m1.1.1.2.3.cmml">k</mi></msubsup><mo id="A1.I2.i1.p1.1.m1.1.1.1" xref="A1.I2.i1.p1.1.m1.1.1.1.cmml">=</mo><mrow id="A1.I2.i1.p1.1.m1.1.1.3" xref="A1.I2.i1.p1.1.m1.1.1.3.cmml"><msubsup id="A1.I2.i1.p1.1.m1.1.1.3.2" xref="A1.I2.i1.p1.1.m1.1.1.3.2.cmml"><mi id="A1.I2.i1.p1.1.m1.1.1.3.2.2.2" xref="A1.I2.i1.p1.1.m1.1.1.3.2.2.2.cmml">W</mi><mi id="A1.I2.i1.p1.1.m1.1.1.3.2.2.3" xref="A1.I2.i1.p1.1.m1.1.1.3.2.2.3.cmml">t</mi><mi id="A1.I2.i1.p1.1.m1.1.1.3.2.3" xref="A1.I2.i1.p1.1.m1.1.1.3.2.3.cmml">k</mi></msubsup><mo lspace="0em" rspace="0em" id="A1.I2.i1.p1.1.m1.1.1.3.1" xref="A1.I2.i1.p1.1.m1.1.1.3.1.cmml">​</mo><msubsup id="A1.I2.i1.p1.1.m1.1.1.3.3" xref="A1.I2.i1.p1.1.m1.1.1.3.3.cmml"><mover accent="true" id="A1.I2.i1.p1.1.m1.1.1.3.3.2.2" xref="A1.I2.i1.p1.1.m1.1.1.3.3.2.2.cmml"><mi id="A1.I2.i1.p1.1.m1.1.1.3.3.2.2.2" xref="A1.I2.i1.p1.1.m1.1.1.3.3.2.2.2.cmml">x</mi><mo id="A1.I2.i1.p1.1.m1.1.1.3.3.2.2.1" xref="A1.I2.i1.p1.1.m1.1.1.3.3.2.2.1.cmml">¯</mo></mover><mi id="A1.I2.i1.p1.1.m1.1.1.3.3.2.3" xref="A1.I2.i1.p1.1.m1.1.1.3.3.2.3.cmml">t</mi><mi id="A1.I2.i1.p1.1.m1.1.1.3.3.3" xref="A1.I2.i1.p1.1.m1.1.1.3.3.3.cmml">k</mi></msubsup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.I2.i1.p1.1.m1.1b"><apply id="A1.I2.i1.p1.1.m1.1.1.cmml" xref="A1.I2.i1.p1.1.m1.1.1"><eq id="A1.I2.i1.p1.1.m1.1.1.1.cmml" xref="A1.I2.i1.p1.1.m1.1.1.1"></eq><apply id="A1.I2.i1.p1.1.m1.1.1.2.cmml" xref="A1.I2.i1.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="A1.I2.i1.p1.1.m1.1.1.2.1.cmml" xref="A1.I2.i1.p1.1.m1.1.1.2">superscript</csymbol><apply id="A1.I2.i1.p1.1.m1.1.1.2.2.cmml" xref="A1.I2.i1.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="A1.I2.i1.p1.1.m1.1.1.2.2.1.cmml" xref="A1.I2.i1.p1.1.m1.1.1.2">subscript</csymbol><apply id="A1.I2.i1.p1.1.m1.1.1.2.2.2.cmml" xref="A1.I2.i1.p1.1.m1.1.1.2.2.2"><ci id="A1.I2.i1.p1.1.m1.1.1.2.2.2.1.cmml" xref="A1.I2.i1.p1.1.m1.1.1.2.2.2.1">¯</ci><ci id="A1.I2.i1.p1.1.m1.1.1.2.2.2.2.cmml" xref="A1.I2.i1.p1.1.m1.1.1.2.2.2.2">𝑧</ci></apply><ci id="A1.I2.i1.p1.1.m1.1.1.2.2.3.cmml" xref="A1.I2.i1.p1.1.m1.1.1.2.2.3">𝑡</ci></apply><ci id="A1.I2.i1.p1.1.m1.1.1.2.3.cmml" xref="A1.I2.i1.p1.1.m1.1.1.2.3">𝑘</ci></apply><apply id="A1.I2.i1.p1.1.m1.1.1.3.cmml" xref="A1.I2.i1.p1.1.m1.1.1.3"><times id="A1.I2.i1.p1.1.m1.1.1.3.1.cmml" xref="A1.I2.i1.p1.1.m1.1.1.3.1"></times><apply id="A1.I2.i1.p1.1.m1.1.1.3.2.cmml" xref="A1.I2.i1.p1.1.m1.1.1.3.2"><csymbol cd="ambiguous" id="A1.I2.i1.p1.1.m1.1.1.3.2.1.cmml" xref="A1.I2.i1.p1.1.m1.1.1.3.2">superscript</csymbol><apply id="A1.I2.i1.p1.1.m1.1.1.3.2.2.cmml" xref="A1.I2.i1.p1.1.m1.1.1.3.2"><csymbol cd="ambiguous" id="A1.I2.i1.p1.1.m1.1.1.3.2.2.1.cmml" xref="A1.I2.i1.p1.1.m1.1.1.3.2">subscript</csymbol><ci id="A1.I2.i1.p1.1.m1.1.1.3.2.2.2.cmml" xref="A1.I2.i1.p1.1.m1.1.1.3.2.2.2">𝑊</ci><ci id="A1.I2.i1.p1.1.m1.1.1.3.2.2.3.cmml" xref="A1.I2.i1.p1.1.m1.1.1.3.2.2.3">𝑡</ci></apply><ci id="A1.I2.i1.p1.1.m1.1.1.3.2.3.cmml" xref="A1.I2.i1.p1.1.m1.1.1.3.2.3">𝑘</ci></apply><apply id="A1.I2.i1.p1.1.m1.1.1.3.3.cmml" xref="A1.I2.i1.p1.1.m1.1.1.3.3"><csymbol cd="ambiguous" id="A1.I2.i1.p1.1.m1.1.1.3.3.1.cmml" xref="A1.I2.i1.p1.1.m1.1.1.3.3">superscript</csymbol><apply id="A1.I2.i1.p1.1.m1.1.1.3.3.2.cmml" xref="A1.I2.i1.p1.1.m1.1.1.3.3"><csymbol cd="ambiguous" id="A1.I2.i1.p1.1.m1.1.1.3.3.2.1.cmml" xref="A1.I2.i1.p1.1.m1.1.1.3.3">subscript</csymbol><apply id="A1.I2.i1.p1.1.m1.1.1.3.3.2.2.cmml" xref="A1.I2.i1.p1.1.m1.1.1.3.3.2.2"><ci id="A1.I2.i1.p1.1.m1.1.1.3.3.2.2.1.cmml" xref="A1.I2.i1.p1.1.m1.1.1.3.3.2.2.1">¯</ci><ci id="A1.I2.i1.p1.1.m1.1.1.3.3.2.2.2.cmml" xref="A1.I2.i1.p1.1.m1.1.1.3.3.2.2.2">𝑥</ci></apply><ci id="A1.I2.i1.p1.1.m1.1.1.3.3.2.3.cmml" xref="A1.I2.i1.p1.1.m1.1.1.3.3.2.3">𝑡</ci></apply><ci id="A1.I2.i1.p1.1.m1.1.1.3.3.3.cmml" xref="A1.I2.i1.p1.1.m1.1.1.3.3.3">𝑘</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I2.i1.p1.1.m1.1c">\bar{z}_{t}^{k}=W_{t}^{k}\,\bar{x}_{t}^{k}</annotation></semantics></math>, for <math id="A1.I2.i1.p1.2.m2.3" class="ltx_Math" alttext="t=1,\dots,T" display="inline"><semantics id="A1.I2.i1.p1.2.m2.3a"><mrow id="A1.I2.i1.p1.2.m2.3.4" xref="A1.I2.i1.p1.2.m2.3.4.cmml"><mi id="A1.I2.i1.p1.2.m2.3.4.2" xref="A1.I2.i1.p1.2.m2.3.4.2.cmml">t</mi><mo id="A1.I2.i1.p1.2.m2.3.4.1" xref="A1.I2.i1.p1.2.m2.3.4.1.cmml">=</mo><mrow id="A1.I2.i1.p1.2.m2.3.4.3.2" xref="A1.I2.i1.p1.2.m2.3.4.3.1.cmml"><mn id="A1.I2.i1.p1.2.m2.1.1" xref="A1.I2.i1.p1.2.m2.1.1.cmml">1</mn><mo id="A1.I2.i1.p1.2.m2.3.4.3.2.1" xref="A1.I2.i1.p1.2.m2.3.4.3.1.cmml">,</mo><mi mathvariant="normal" id="A1.I2.i1.p1.2.m2.2.2" xref="A1.I2.i1.p1.2.m2.2.2.cmml">…</mi><mo id="A1.I2.i1.p1.2.m2.3.4.3.2.2" xref="A1.I2.i1.p1.2.m2.3.4.3.1.cmml">,</mo><mi id="A1.I2.i1.p1.2.m2.3.3" xref="A1.I2.i1.p1.2.m2.3.3.cmml">T</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.I2.i1.p1.2.m2.3b"><apply id="A1.I2.i1.p1.2.m2.3.4.cmml" xref="A1.I2.i1.p1.2.m2.3.4"><eq id="A1.I2.i1.p1.2.m2.3.4.1.cmml" xref="A1.I2.i1.p1.2.m2.3.4.1"></eq><ci id="A1.I2.i1.p1.2.m2.3.4.2.cmml" xref="A1.I2.i1.p1.2.m2.3.4.2">𝑡</ci><list id="A1.I2.i1.p1.2.m2.3.4.3.1.cmml" xref="A1.I2.i1.p1.2.m2.3.4.3.2"><cn type="integer" id="A1.I2.i1.p1.2.m2.1.1.cmml" xref="A1.I2.i1.p1.2.m2.1.1">1</cn><ci id="A1.I2.i1.p1.2.m2.2.2.cmml" xref="A1.I2.i1.p1.2.m2.2.2">…</ci><ci id="A1.I2.i1.p1.2.m2.3.3.cmml" xref="A1.I2.i1.p1.2.m2.3.3">𝑇</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I2.i1.p1.2.m2.3c">t=1,\dots,T</annotation></semantics></math></p>
</div>
</li>
<li id="A1.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I2.i2.p1" class="ltx_para">
<p id="A1.I2.i2.p1.2" class="ltx_p"><math id="A1.I2.i2.p1.1.m1.1" class="ltx_Math" alttext="\bar{x}_{t}^{k+1}=\sigma_{k}(\bar{z}_{t}^{k})" display="inline"><semantics id="A1.I2.i2.p1.1.m1.1a"><mrow id="A1.I2.i2.p1.1.m1.1.1" xref="A1.I2.i2.p1.1.m1.1.1.cmml"><msubsup id="A1.I2.i2.p1.1.m1.1.1.3" xref="A1.I2.i2.p1.1.m1.1.1.3.cmml"><mover accent="true" id="A1.I2.i2.p1.1.m1.1.1.3.2.2" xref="A1.I2.i2.p1.1.m1.1.1.3.2.2.cmml"><mi id="A1.I2.i2.p1.1.m1.1.1.3.2.2.2" xref="A1.I2.i2.p1.1.m1.1.1.3.2.2.2.cmml">x</mi><mo id="A1.I2.i2.p1.1.m1.1.1.3.2.2.1" xref="A1.I2.i2.p1.1.m1.1.1.3.2.2.1.cmml">¯</mo></mover><mi id="A1.I2.i2.p1.1.m1.1.1.3.2.3" xref="A1.I2.i2.p1.1.m1.1.1.3.2.3.cmml">t</mi><mrow id="A1.I2.i2.p1.1.m1.1.1.3.3" xref="A1.I2.i2.p1.1.m1.1.1.3.3.cmml"><mi id="A1.I2.i2.p1.1.m1.1.1.3.3.2" xref="A1.I2.i2.p1.1.m1.1.1.3.3.2.cmml">k</mi><mo id="A1.I2.i2.p1.1.m1.1.1.3.3.1" xref="A1.I2.i2.p1.1.m1.1.1.3.3.1.cmml">+</mo><mn id="A1.I2.i2.p1.1.m1.1.1.3.3.3" xref="A1.I2.i2.p1.1.m1.1.1.3.3.3.cmml">1</mn></mrow></msubsup><mo id="A1.I2.i2.p1.1.m1.1.1.2" xref="A1.I2.i2.p1.1.m1.1.1.2.cmml">=</mo><mrow id="A1.I2.i2.p1.1.m1.1.1.1" xref="A1.I2.i2.p1.1.m1.1.1.1.cmml"><msub id="A1.I2.i2.p1.1.m1.1.1.1.3" xref="A1.I2.i2.p1.1.m1.1.1.1.3.cmml"><mi id="A1.I2.i2.p1.1.m1.1.1.1.3.2" xref="A1.I2.i2.p1.1.m1.1.1.1.3.2.cmml">σ</mi><mi id="A1.I2.i2.p1.1.m1.1.1.1.3.3" xref="A1.I2.i2.p1.1.m1.1.1.1.3.3.cmml">k</mi></msub><mo lspace="0em" rspace="0em" id="A1.I2.i2.p1.1.m1.1.1.1.2" xref="A1.I2.i2.p1.1.m1.1.1.1.2.cmml">​</mo><mrow id="A1.I2.i2.p1.1.m1.1.1.1.1.1" xref="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.I2.i2.p1.1.m1.1.1.1.1.1.2" xref="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.cmml">(</mo><msubsup id="A1.I2.i2.p1.1.m1.1.1.1.1.1.1" xref="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.cmml"><mover accent="true" id="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.2.2" xref="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.2.2.cmml"><mi id="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.2.2.2" xref="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.2.2.2.cmml">z</mi><mo id="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.2.2.1" xref="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.2.2.1.cmml">¯</mo></mover><mi id="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.2.3" xref="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.2.3.cmml">t</mi><mi id="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.3" xref="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.3.cmml">k</mi></msubsup><mo stretchy="false" id="A1.I2.i2.p1.1.m1.1.1.1.1.1.3" xref="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.I2.i2.p1.1.m1.1b"><apply id="A1.I2.i2.p1.1.m1.1.1.cmml" xref="A1.I2.i2.p1.1.m1.1.1"><eq id="A1.I2.i2.p1.1.m1.1.1.2.cmml" xref="A1.I2.i2.p1.1.m1.1.1.2"></eq><apply id="A1.I2.i2.p1.1.m1.1.1.3.cmml" xref="A1.I2.i2.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="A1.I2.i2.p1.1.m1.1.1.3.1.cmml" xref="A1.I2.i2.p1.1.m1.1.1.3">superscript</csymbol><apply id="A1.I2.i2.p1.1.m1.1.1.3.2.cmml" xref="A1.I2.i2.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="A1.I2.i2.p1.1.m1.1.1.3.2.1.cmml" xref="A1.I2.i2.p1.1.m1.1.1.3">subscript</csymbol><apply id="A1.I2.i2.p1.1.m1.1.1.3.2.2.cmml" xref="A1.I2.i2.p1.1.m1.1.1.3.2.2"><ci id="A1.I2.i2.p1.1.m1.1.1.3.2.2.1.cmml" xref="A1.I2.i2.p1.1.m1.1.1.3.2.2.1">¯</ci><ci id="A1.I2.i2.p1.1.m1.1.1.3.2.2.2.cmml" xref="A1.I2.i2.p1.1.m1.1.1.3.2.2.2">𝑥</ci></apply><ci id="A1.I2.i2.p1.1.m1.1.1.3.2.3.cmml" xref="A1.I2.i2.p1.1.m1.1.1.3.2.3">𝑡</ci></apply><apply id="A1.I2.i2.p1.1.m1.1.1.3.3.cmml" xref="A1.I2.i2.p1.1.m1.1.1.3.3"><plus id="A1.I2.i2.p1.1.m1.1.1.3.3.1.cmml" xref="A1.I2.i2.p1.1.m1.1.1.3.3.1"></plus><ci id="A1.I2.i2.p1.1.m1.1.1.3.3.2.cmml" xref="A1.I2.i2.p1.1.m1.1.1.3.3.2">𝑘</ci><cn type="integer" id="A1.I2.i2.p1.1.m1.1.1.3.3.3.cmml" xref="A1.I2.i2.p1.1.m1.1.1.3.3.3">1</cn></apply></apply><apply id="A1.I2.i2.p1.1.m1.1.1.1.cmml" xref="A1.I2.i2.p1.1.m1.1.1.1"><times id="A1.I2.i2.p1.1.m1.1.1.1.2.cmml" xref="A1.I2.i2.p1.1.m1.1.1.1.2"></times><apply id="A1.I2.i2.p1.1.m1.1.1.1.3.cmml" xref="A1.I2.i2.p1.1.m1.1.1.1.3"><csymbol cd="ambiguous" id="A1.I2.i2.p1.1.m1.1.1.1.3.1.cmml" xref="A1.I2.i2.p1.1.m1.1.1.1.3">subscript</csymbol><ci id="A1.I2.i2.p1.1.m1.1.1.1.3.2.cmml" xref="A1.I2.i2.p1.1.m1.1.1.1.3.2">𝜎</ci><ci id="A1.I2.i2.p1.1.m1.1.1.1.3.3.cmml" xref="A1.I2.i2.p1.1.m1.1.1.1.3.3">𝑘</ci></apply><apply id="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.cmml" xref="A1.I2.i2.p1.1.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.1.cmml" xref="A1.I2.i2.p1.1.m1.1.1.1.1.1">superscript</csymbol><apply id="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.2.cmml" xref="A1.I2.i2.p1.1.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.2.1.cmml" xref="A1.I2.i2.p1.1.m1.1.1.1.1.1">subscript</csymbol><apply id="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.2.2.cmml" xref="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.2.2"><ci id="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.2.2.1.cmml" xref="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.2.2.1">¯</ci><ci id="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.2.2.2.cmml" xref="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.2.2.2">𝑧</ci></apply><ci id="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.2.3.cmml" xref="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.2.3">𝑡</ci></apply><ci id="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.3.cmml" xref="A1.I2.i2.p1.1.m1.1.1.1.1.1.1.3">𝑘</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I2.i2.p1.1.m1.1c">\bar{x}_{t}^{k+1}=\sigma_{k}(\bar{z}_{t}^{k})</annotation></semantics></math>, for <math id="A1.I2.i2.p1.2.m2.3" class="ltx_Math" alttext="t=1,\dots,T" display="inline"><semantics id="A1.I2.i2.p1.2.m2.3a"><mrow id="A1.I2.i2.p1.2.m2.3.4" xref="A1.I2.i2.p1.2.m2.3.4.cmml"><mi id="A1.I2.i2.p1.2.m2.3.4.2" xref="A1.I2.i2.p1.2.m2.3.4.2.cmml">t</mi><mo id="A1.I2.i2.p1.2.m2.3.4.1" xref="A1.I2.i2.p1.2.m2.3.4.1.cmml">=</mo><mrow id="A1.I2.i2.p1.2.m2.3.4.3.2" xref="A1.I2.i2.p1.2.m2.3.4.3.1.cmml"><mn id="A1.I2.i2.p1.2.m2.1.1" xref="A1.I2.i2.p1.2.m2.1.1.cmml">1</mn><mo id="A1.I2.i2.p1.2.m2.3.4.3.2.1" xref="A1.I2.i2.p1.2.m2.3.4.3.1.cmml">,</mo><mi mathvariant="normal" id="A1.I2.i2.p1.2.m2.2.2" xref="A1.I2.i2.p1.2.m2.2.2.cmml">…</mi><mo id="A1.I2.i2.p1.2.m2.3.4.3.2.2" xref="A1.I2.i2.p1.2.m2.3.4.3.1.cmml">,</mo><mi id="A1.I2.i2.p1.2.m2.3.3" xref="A1.I2.i2.p1.2.m2.3.3.cmml">T</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.I2.i2.p1.2.m2.3b"><apply id="A1.I2.i2.p1.2.m2.3.4.cmml" xref="A1.I2.i2.p1.2.m2.3.4"><eq id="A1.I2.i2.p1.2.m2.3.4.1.cmml" xref="A1.I2.i2.p1.2.m2.3.4.1"></eq><ci id="A1.I2.i2.p1.2.m2.3.4.2.cmml" xref="A1.I2.i2.p1.2.m2.3.4.2">𝑡</ci><list id="A1.I2.i2.p1.2.m2.3.4.3.1.cmml" xref="A1.I2.i2.p1.2.m2.3.4.3.2"><cn type="integer" id="A1.I2.i2.p1.2.m2.1.1.cmml" xref="A1.I2.i2.p1.2.m2.1.1">1</cn><ci id="A1.I2.i2.p1.2.m2.2.2.cmml" xref="A1.I2.i2.p1.2.m2.2.2">…</ci><ci id="A1.I2.i2.p1.2.m2.3.3.cmml" xref="A1.I2.i2.p1.2.m2.3.3">𝑇</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I2.i2.p1.2.m2.3c">t=1,\dots,T</annotation></semantics></math></p>
</div>
</li>
</ul>
<p id="A1.SS2.p2.2" class="ltx_p">where <math id="A1.SS2.p2.2.m1.3" class="ltx_Math" alttext="\bar{X}^{K+1}=[\bar{x}_{1}^{k+1},\dots,\bar{x}_{b}^{k+1}]" display="inline"><semantics id="A1.SS2.p2.2.m1.3a"><mrow id="A1.SS2.p2.2.m1.3.3" xref="A1.SS2.p2.2.m1.3.3.cmml"><msup id="A1.SS2.p2.2.m1.3.3.4" xref="A1.SS2.p2.2.m1.3.3.4.cmml"><mover accent="true" id="A1.SS2.p2.2.m1.3.3.4.2" xref="A1.SS2.p2.2.m1.3.3.4.2.cmml"><mi id="A1.SS2.p2.2.m1.3.3.4.2.2" xref="A1.SS2.p2.2.m1.3.3.4.2.2.cmml">X</mi><mo id="A1.SS2.p2.2.m1.3.3.4.2.1" xref="A1.SS2.p2.2.m1.3.3.4.2.1.cmml">¯</mo></mover><mrow id="A1.SS2.p2.2.m1.3.3.4.3" xref="A1.SS2.p2.2.m1.3.3.4.3.cmml"><mi id="A1.SS2.p2.2.m1.3.3.4.3.2" xref="A1.SS2.p2.2.m1.3.3.4.3.2.cmml">K</mi><mo id="A1.SS2.p2.2.m1.3.3.4.3.1" xref="A1.SS2.p2.2.m1.3.3.4.3.1.cmml">+</mo><mn id="A1.SS2.p2.2.m1.3.3.4.3.3" xref="A1.SS2.p2.2.m1.3.3.4.3.3.cmml">1</mn></mrow></msup><mo id="A1.SS2.p2.2.m1.3.3.3" xref="A1.SS2.p2.2.m1.3.3.3.cmml">=</mo><mrow id="A1.SS2.p2.2.m1.3.3.2.2" xref="A1.SS2.p2.2.m1.3.3.2.3.cmml"><mo stretchy="false" id="A1.SS2.p2.2.m1.3.3.2.2.3" xref="A1.SS2.p2.2.m1.3.3.2.3.cmml">[</mo><msubsup id="A1.SS2.p2.2.m1.2.2.1.1.1" xref="A1.SS2.p2.2.m1.2.2.1.1.1.cmml"><mover accent="true" id="A1.SS2.p2.2.m1.2.2.1.1.1.2.2" xref="A1.SS2.p2.2.m1.2.2.1.1.1.2.2.cmml"><mi id="A1.SS2.p2.2.m1.2.2.1.1.1.2.2.2" xref="A1.SS2.p2.2.m1.2.2.1.1.1.2.2.2.cmml">x</mi><mo id="A1.SS2.p2.2.m1.2.2.1.1.1.2.2.1" xref="A1.SS2.p2.2.m1.2.2.1.1.1.2.2.1.cmml">¯</mo></mover><mn id="A1.SS2.p2.2.m1.2.2.1.1.1.2.3" xref="A1.SS2.p2.2.m1.2.2.1.1.1.2.3.cmml">1</mn><mrow id="A1.SS2.p2.2.m1.2.2.1.1.1.3" xref="A1.SS2.p2.2.m1.2.2.1.1.1.3.cmml"><mi id="A1.SS2.p2.2.m1.2.2.1.1.1.3.2" xref="A1.SS2.p2.2.m1.2.2.1.1.1.3.2.cmml">k</mi><mo id="A1.SS2.p2.2.m1.2.2.1.1.1.3.1" xref="A1.SS2.p2.2.m1.2.2.1.1.1.3.1.cmml">+</mo><mn id="A1.SS2.p2.2.m1.2.2.1.1.1.3.3" xref="A1.SS2.p2.2.m1.2.2.1.1.1.3.3.cmml">1</mn></mrow></msubsup><mo id="A1.SS2.p2.2.m1.3.3.2.2.4" xref="A1.SS2.p2.2.m1.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="A1.SS2.p2.2.m1.1.1" xref="A1.SS2.p2.2.m1.1.1.cmml">…</mi><mo id="A1.SS2.p2.2.m1.3.3.2.2.5" xref="A1.SS2.p2.2.m1.3.3.2.3.cmml">,</mo><msubsup id="A1.SS2.p2.2.m1.3.3.2.2.2" xref="A1.SS2.p2.2.m1.3.3.2.2.2.cmml"><mover accent="true" id="A1.SS2.p2.2.m1.3.3.2.2.2.2.2" xref="A1.SS2.p2.2.m1.3.3.2.2.2.2.2.cmml"><mi id="A1.SS2.p2.2.m1.3.3.2.2.2.2.2.2" xref="A1.SS2.p2.2.m1.3.3.2.2.2.2.2.2.cmml">x</mi><mo id="A1.SS2.p2.2.m1.3.3.2.2.2.2.2.1" xref="A1.SS2.p2.2.m1.3.3.2.2.2.2.2.1.cmml">¯</mo></mover><mi id="A1.SS2.p2.2.m1.3.3.2.2.2.2.3" xref="A1.SS2.p2.2.m1.3.3.2.2.2.2.3.cmml">b</mi><mrow id="A1.SS2.p2.2.m1.3.3.2.2.2.3" xref="A1.SS2.p2.2.m1.3.3.2.2.2.3.cmml"><mi id="A1.SS2.p2.2.m1.3.3.2.2.2.3.2" xref="A1.SS2.p2.2.m1.3.3.2.2.2.3.2.cmml">k</mi><mo id="A1.SS2.p2.2.m1.3.3.2.2.2.3.1" xref="A1.SS2.p2.2.m1.3.3.2.2.2.3.1.cmml">+</mo><mn id="A1.SS2.p2.2.m1.3.3.2.2.2.3.3" xref="A1.SS2.p2.2.m1.3.3.2.2.2.3.3.cmml">1</mn></mrow></msubsup><mo stretchy="false" id="A1.SS2.p2.2.m1.3.3.2.2.6" xref="A1.SS2.p2.2.m1.3.3.2.3.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.p2.2.m1.3b"><apply id="A1.SS2.p2.2.m1.3.3.cmml" xref="A1.SS2.p2.2.m1.3.3"><eq id="A1.SS2.p2.2.m1.3.3.3.cmml" xref="A1.SS2.p2.2.m1.3.3.3"></eq><apply id="A1.SS2.p2.2.m1.3.3.4.cmml" xref="A1.SS2.p2.2.m1.3.3.4"><csymbol cd="ambiguous" id="A1.SS2.p2.2.m1.3.3.4.1.cmml" xref="A1.SS2.p2.2.m1.3.3.4">superscript</csymbol><apply id="A1.SS2.p2.2.m1.3.3.4.2.cmml" xref="A1.SS2.p2.2.m1.3.3.4.2"><ci id="A1.SS2.p2.2.m1.3.3.4.2.1.cmml" xref="A1.SS2.p2.2.m1.3.3.4.2.1">¯</ci><ci id="A1.SS2.p2.2.m1.3.3.4.2.2.cmml" xref="A1.SS2.p2.2.m1.3.3.4.2.2">𝑋</ci></apply><apply id="A1.SS2.p2.2.m1.3.3.4.3.cmml" xref="A1.SS2.p2.2.m1.3.3.4.3"><plus id="A1.SS2.p2.2.m1.3.3.4.3.1.cmml" xref="A1.SS2.p2.2.m1.3.3.4.3.1"></plus><ci id="A1.SS2.p2.2.m1.3.3.4.3.2.cmml" xref="A1.SS2.p2.2.m1.3.3.4.3.2">𝐾</ci><cn type="integer" id="A1.SS2.p2.2.m1.3.3.4.3.3.cmml" xref="A1.SS2.p2.2.m1.3.3.4.3.3">1</cn></apply></apply><list id="A1.SS2.p2.2.m1.3.3.2.3.cmml" xref="A1.SS2.p2.2.m1.3.3.2.2"><apply id="A1.SS2.p2.2.m1.2.2.1.1.1.cmml" xref="A1.SS2.p2.2.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="A1.SS2.p2.2.m1.2.2.1.1.1.1.cmml" xref="A1.SS2.p2.2.m1.2.2.1.1.1">superscript</csymbol><apply id="A1.SS2.p2.2.m1.2.2.1.1.1.2.cmml" xref="A1.SS2.p2.2.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="A1.SS2.p2.2.m1.2.2.1.1.1.2.1.cmml" xref="A1.SS2.p2.2.m1.2.2.1.1.1">subscript</csymbol><apply id="A1.SS2.p2.2.m1.2.2.1.1.1.2.2.cmml" xref="A1.SS2.p2.2.m1.2.2.1.1.1.2.2"><ci id="A1.SS2.p2.2.m1.2.2.1.1.1.2.2.1.cmml" xref="A1.SS2.p2.2.m1.2.2.1.1.1.2.2.1">¯</ci><ci id="A1.SS2.p2.2.m1.2.2.1.1.1.2.2.2.cmml" xref="A1.SS2.p2.2.m1.2.2.1.1.1.2.2.2">𝑥</ci></apply><cn type="integer" id="A1.SS2.p2.2.m1.2.2.1.1.1.2.3.cmml" xref="A1.SS2.p2.2.m1.2.2.1.1.1.2.3">1</cn></apply><apply id="A1.SS2.p2.2.m1.2.2.1.1.1.3.cmml" xref="A1.SS2.p2.2.m1.2.2.1.1.1.3"><plus id="A1.SS2.p2.2.m1.2.2.1.1.1.3.1.cmml" xref="A1.SS2.p2.2.m1.2.2.1.1.1.3.1"></plus><ci id="A1.SS2.p2.2.m1.2.2.1.1.1.3.2.cmml" xref="A1.SS2.p2.2.m1.2.2.1.1.1.3.2">𝑘</ci><cn type="integer" id="A1.SS2.p2.2.m1.2.2.1.1.1.3.3.cmml" xref="A1.SS2.p2.2.m1.2.2.1.1.1.3.3">1</cn></apply></apply><ci id="A1.SS2.p2.2.m1.1.1.cmml" xref="A1.SS2.p2.2.m1.1.1">…</ci><apply id="A1.SS2.p2.2.m1.3.3.2.2.2.cmml" xref="A1.SS2.p2.2.m1.3.3.2.2.2"><csymbol cd="ambiguous" id="A1.SS2.p2.2.m1.3.3.2.2.2.1.cmml" xref="A1.SS2.p2.2.m1.3.3.2.2.2">superscript</csymbol><apply id="A1.SS2.p2.2.m1.3.3.2.2.2.2.cmml" xref="A1.SS2.p2.2.m1.3.3.2.2.2"><csymbol cd="ambiguous" id="A1.SS2.p2.2.m1.3.3.2.2.2.2.1.cmml" xref="A1.SS2.p2.2.m1.3.3.2.2.2">subscript</csymbol><apply id="A1.SS2.p2.2.m1.3.3.2.2.2.2.2.cmml" xref="A1.SS2.p2.2.m1.3.3.2.2.2.2.2"><ci id="A1.SS2.p2.2.m1.3.3.2.2.2.2.2.1.cmml" xref="A1.SS2.p2.2.m1.3.3.2.2.2.2.2.1">¯</ci><ci id="A1.SS2.p2.2.m1.3.3.2.2.2.2.2.2.cmml" xref="A1.SS2.p2.2.m1.3.3.2.2.2.2.2.2">𝑥</ci></apply><ci id="A1.SS2.p2.2.m1.3.3.2.2.2.2.3.cmml" xref="A1.SS2.p2.2.m1.3.3.2.2.2.2.3">𝑏</ci></apply><apply id="A1.SS2.p2.2.m1.3.3.2.2.2.3.cmml" xref="A1.SS2.p2.2.m1.3.3.2.2.2.3"><plus id="A1.SS2.p2.2.m1.3.3.2.2.2.3.1.cmml" xref="A1.SS2.p2.2.m1.3.3.2.2.2.3.1"></plus><ci id="A1.SS2.p2.2.m1.3.3.2.2.2.3.2.cmml" xref="A1.SS2.p2.2.m1.3.3.2.2.2.3.2">𝑘</ci><cn type="integer" id="A1.SS2.p2.2.m1.3.3.2.2.2.3.3.cmml" xref="A1.SS2.p2.2.m1.3.3.2.2.2.3.3">1</cn></apply></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p2.2.m1.3c">\bar{X}^{K+1}=[\bar{x}_{1}^{k+1},\dots,\bar{x}_{b}^{k+1}]</annotation></semantics></math> contains the output tokens.</p>
</div>
<div id="A1.SS2.p3" class="ltx_para">
<p id="A1.SS2.p3.5" class="ltx_p">Note that a standard backward pass only computes the sum of the gradients:</p>
<table id="A1.Ex9" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.Ex9.m1.1" class="ltx_Math" alttext="\nabla_{W_{0}^{k}}l=\sum_{t=1}^{b}\nabla_{W^{k}_{0}}l_{t}=\sum_{t=1}^{b}G^{k}_{t}," display="block"><semantics id="A1.Ex9.m1.1a"><mrow id="A1.Ex9.m1.1.1.1" xref="A1.Ex9.m1.1.1.1.1.cmml"><mrow id="A1.Ex9.m1.1.1.1.1" xref="A1.Ex9.m1.1.1.1.1.cmml"><mrow id="A1.Ex9.m1.1.1.1.1.2" xref="A1.Ex9.m1.1.1.1.1.2.cmml"><msub id="A1.Ex9.m1.1.1.1.1.2.1" xref="A1.Ex9.m1.1.1.1.1.2.1.cmml"><mo id="A1.Ex9.m1.1.1.1.1.2.1.2" xref="A1.Ex9.m1.1.1.1.1.2.1.2.cmml">∇</mo><msubsup id="A1.Ex9.m1.1.1.1.1.2.1.3" xref="A1.Ex9.m1.1.1.1.1.2.1.3.cmml"><mi id="A1.Ex9.m1.1.1.1.1.2.1.3.2.2" xref="A1.Ex9.m1.1.1.1.1.2.1.3.2.2.cmml">W</mi><mn id="A1.Ex9.m1.1.1.1.1.2.1.3.2.3" xref="A1.Ex9.m1.1.1.1.1.2.1.3.2.3.cmml">0</mn><mi id="A1.Ex9.m1.1.1.1.1.2.1.3.3" xref="A1.Ex9.m1.1.1.1.1.2.1.3.3.cmml">k</mi></msubsup></msub><mi id="A1.Ex9.m1.1.1.1.1.2.2" xref="A1.Ex9.m1.1.1.1.1.2.2.cmml">l</mi></mrow><mo rspace="0.111em" id="A1.Ex9.m1.1.1.1.1.3" xref="A1.Ex9.m1.1.1.1.1.3.cmml">=</mo><mrow id="A1.Ex9.m1.1.1.1.1.4" xref="A1.Ex9.m1.1.1.1.1.4.cmml"><munderover id="A1.Ex9.m1.1.1.1.1.4.1" xref="A1.Ex9.m1.1.1.1.1.4.1.cmml"><mo movablelimits="false" id="A1.Ex9.m1.1.1.1.1.4.1.2.2" xref="A1.Ex9.m1.1.1.1.1.4.1.2.2.cmml">∑</mo><mrow id="A1.Ex9.m1.1.1.1.1.4.1.2.3" xref="A1.Ex9.m1.1.1.1.1.4.1.2.3.cmml"><mi id="A1.Ex9.m1.1.1.1.1.4.1.2.3.2" xref="A1.Ex9.m1.1.1.1.1.4.1.2.3.2.cmml">t</mi><mo id="A1.Ex9.m1.1.1.1.1.4.1.2.3.1" xref="A1.Ex9.m1.1.1.1.1.4.1.2.3.1.cmml">=</mo><mn id="A1.Ex9.m1.1.1.1.1.4.1.2.3.3" xref="A1.Ex9.m1.1.1.1.1.4.1.2.3.3.cmml">1</mn></mrow><mi id="A1.Ex9.m1.1.1.1.1.4.1.3" xref="A1.Ex9.m1.1.1.1.1.4.1.3.cmml">b</mi></munderover><mrow id="A1.Ex9.m1.1.1.1.1.4.2" xref="A1.Ex9.m1.1.1.1.1.4.2.cmml"><msub id="A1.Ex9.m1.1.1.1.1.4.2.1" xref="A1.Ex9.m1.1.1.1.1.4.2.1.cmml"><mo rspace="0.167em" id="A1.Ex9.m1.1.1.1.1.4.2.1.2" xref="A1.Ex9.m1.1.1.1.1.4.2.1.2.cmml">∇</mo><msubsup id="A1.Ex9.m1.1.1.1.1.4.2.1.3" xref="A1.Ex9.m1.1.1.1.1.4.2.1.3.cmml"><mi id="A1.Ex9.m1.1.1.1.1.4.2.1.3.2.2" xref="A1.Ex9.m1.1.1.1.1.4.2.1.3.2.2.cmml">W</mi><mn id="A1.Ex9.m1.1.1.1.1.4.2.1.3.3" xref="A1.Ex9.m1.1.1.1.1.4.2.1.3.3.cmml">0</mn><mi id="A1.Ex9.m1.1.1.1.1.4.2.1.3.2.3" xref="A1.Ex9.m1.1.1.1.1.4.2.1.3.2.3.cmml">k</mi></msubsup></msub><msub id="A1.Ex9.m1.1.1.1.1.4.2.2" xref="A1.Ex9.m1.1.1.1.1.4.2.2.cmml"><mi id="A1.Ex9.m1.1.1.1.1.4.2.2.2" xref="A1.Ex9.m1.1.1.1.1.4.2.2.2.cmml">l</mi><mi id="A1.Ex9.m1.1.1.1.1.4.2.2.3" xref="A1.Ex9.m1.1.1.1.1.4.2.2.3.cmml">t</mi></msub></mrow></mrow><mo rspace="0.111em" id="A1.Ex9.m1.1.1.1.1.5" xref="A1.Ex9.m1.1.1.1.1.5.cmml">=</mo><mrow id="A1.Ex9.m1.1.1.1.1.6" xref="A1.Ex9.m1.1.1.1.1.6.cmml"><munderover id="A1.Ex9.m1.1.1.1.1.6.1" xref="A1.Ex9.m1.1.1.1.1.6.1.cmml"><mo movablelimits="false" id="A1.Ex9.m1.1.1.1.1.6.1.2.2" xref="A1.Ex9.m1.1.1.1.1.6.1.2.2.cmml">∑</mo><mrow id="A1.Ex9.m1.1.1.1.1.6.1.2.3" xref="A1.Ex9.m1.1.1.1.1.6.1.2.3.cmml"><mi id="A1.Ex9.m1.1.1.1.1.6.1.2.3.2" xref="A1.Ex9.m1.1.1.1.1.6.1.2.3.2.cmml">t</mi><mo id="A1.Ex9.m1.1.1.1.1.6.1.2.3.1" xref="A1.Ex9.m1.1.1.1.1.6.1.2.3.1.cmml">=</mo><mn id="A1.Ex9.m1.1.1.1.1.6.1.2.3.3" xref="A1.Ex9.m1.1.1.1.1.6.1.2.3.3.cmml">1</mn></mrow><mi id="A1.Ex9.m1.1.1.1.1.6.1.3" xref="A1.Ex9.m1.1.1.1.1.6.1.3.cmml">b</mi></munderover><msubsup id="A1.Ex9.m1.1.1.1.1.6.2" xref="A1.Ex9.m1.1.1.1.1.6.2.cmml"><mi id="A1.Ex9.m1.1.1.1.1.6.2.2.2" xref="A1.Ex9.m1.1.1.1.1.6.2.2.2.cmml">G</mi><mi id="A1.Ex9.m1.1.1.1.1.6.2.3" xref="A1.Ex9.m1.1.1.1.1.6.2.3.cmml">t</mi><mi id="A1.Ex9.m1.1.1.1.1.6.2.2.3" xref="A1.Ex9.m1.1.1.1.1.6.2.2.3.cmml">k</mi></msubsup></mrow></mrow><mo id="A1.Ex9.m1.1.1.1.2" xref="A1.Ex9.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex9.m1.1b"><apply id="A1.Ex9.m1.1.1.1.1.cmml" xref="A1.Ex9.m1.1.1.1"><and id="A1.Ex9.m1.1.1.1.1a.cmml" xref="A1.Ex9.m1.1.1.1"></and><apply id="A1.Ex9.m1.1.1.1.1b.cmml" xref="A1.Ex9.m1.1.1.1"><eq id="A1.Ex9.m1.1.1.1.1.3.cmml" xref="A1.Ex9.m1.1.1.1.1.3"></eq><apply id="A1.Ex9.m1.1.1.1.1.2.cmml" xref="A1.Ex9.m1.1.1.1.1.2"><apply id="A1.Ex9.m1.1.1.1.1.2.1.cmml" xref="A1.Ex9.m1.1.1.1.1.2.1"><csymbol cd="ambiguous" id="A1.Ex9.m1.1.1.1.1.2.1.1.cmml" xref="A1.Ex9.m1.1.1.1.1.2.1">subscript</csymbol><ci id="A1.Ex9.m1.1.1.1.1.2.1.2.cmml" xref="A1.Ex9.m1.1.1.1.1.2.1.2">∇</ci><apply id="A1.Ex9.m1.1.1.1.1.2.1.3.cmml" xref="A1.Ex9.m1.1.1.1.1.2.1.3"><csymbol cd="ambiguous" id="A1.Ex9.m1.1.1.1.1.2.1.3.1.cmml" xref="A1.Ex9.m1.1.1.1.1.2.1.3">superscript</csymbol><apply id="A1.Ex9.m1.1.1.1.1.2.1.3.2.cmml" xref="A1.Ex9.m1.1.1.1.1.2.1.3"><csymbol cd="ambiguous" id="A1.Ex9.m1.1.1.1.1.2.1.3.2.1.cmml" xref="A1.Ex9.m1.1.1.1.1.2.1.3">subscript</csymbol><ci id="A1.Ex9.m1.1.1.1.1.2.1.3.2.2.cmml" xref="A1.Ex9.m1.1.1.1.1.2.1.3.2.2">𝑊</ci><cn type="integer" id="A1.Ex9.m1.1.1.1.1.2.1.3.2.3.cmml" xref="A1.Ex9.m1.1.1.1.1.2.1.3.2.3">0</cn></apply><ci id="A1.Ex9.m1.1.1.1.1.2.1.3.3.cmml" xref="A1.Ex9.m1.1.1.1.1.2.1.3.3">𝑘</ci></apply></apply><ci id="A1.Ex9.m1.1.1.1.1.2.2.cmml" xref="A1.Ex9.m1.1.1.1.1.2.2">𝑙</ci></apply><apply id="A1.Ex9.m1.1.1.1.1.4.cmml" xref="A1.Ex9.m1.1.1.1.1.4"><apply id="A1.Ex9.m1.1.1.1.1.4.1.cmml" xref="A1.Ex9.m1.1.1.1.1.4.1"><csymbol cd="ambiguous" id="A1.Ex9.m1.1.1.1.1.4.1.1.cmml" xref="A1.Ex9.m1.1.1.1.1.4.1">superscript</csymbol><apply id="A1.Ex9.m1.1.1.1.1.4.1.2.cmml" xref="A1.Ex9.m1.1.1.1.1.4.1"><csymbol cd="ambiguous" id="A1.Ex9.m1.1.1.1.1.4.1.2.1.cmml" xref="A1.Ex9.m1.1.1.1.1.4.1">subscript</csymbol><sum id="A1.Ex9.m1.1.1.1.1.4.1.2.2.cmml" xref="A1.Ex9.m1.1.1.1.1.4.1.2.2"></sum><apply id="A1.Ex9.m1.1.1.1.1.4.1.2.3.cmml" xref="A1.Ex9.m1.1.1.1.1.4.1.2.3"><eq id="A1.Ex9.m1.1.1.1.1.4.1.2.3.1.cmml" xref="A1.Ex9.m1.1.1.1.1.4.1.2.3.1"></eq><ci id="A1.Ex9.m1.1.1.1.1.4.1.2.3.2.cmml" xref="A1.Ex9.m1.1.1.1.1.4.1.2.3.2">𝑡</ci><cn type="integer" id="A1.Ex9.m1.1.1.1.1.4.1.2.3.3.cmml" xref="A1.Ex9.m1.1.1.1.1.4.1.2.3.3">1</cn></apply></apply><ci id="A1.Ex9.m1.1.1.1.1.4.1.3.cmml" xref="A1.Ex9.m1.1.1.1.1.4.1.3">𝑏</ci></apply><apply id="A1.Ex9.m1.1.1.1.1.4.2.cmml" xref="A1.Ex9.m1.1.1.1.1.4.2"><apply id="A1.Ex9.m1.1.1.1.1.4.2.1.cmml" xref="A1.Ex9.m1.1.1.1.1.4.2.1"><csymbol cd="ambiguous" id="A1.Ex9.m1.1.1.1.1.4.2.1.1.cmml" xref="A1.Ex9.m1.1.1.1.1.4.2.1">subscript</csymbol><ci id="A1.Ex9.m1.1.1.1.1.4.2.1.2.cmml" xref="A1.Ex9.m1.1.1.1.1.4.2.1.2">∇</ci><apply id="A1.Ex9.m1.1.1.1.1.4.2.1.3.cmml" xref="A1.Ex9.m1.1.1.1.1.4.2.1.3"><csymbol cd="ambiguous" id="A1.Ex9.m1.1.1.1.1.4.2.1.3.1.cmml" xref="A1.Ex9.m1.1.1.1.1.4.2.1.3">subscript</csymbol><apply id="A1.Ex9.m1.1.1.1.1.4.2.1.3.2.cmml" xref="A1.Ex9.m1.1.1.1.1.4.2.1.3"><csymbol cd="ambiguous" id="A1.Ex9.m1.1.1.1.1.4.2.1.3.2.1.cmml" xref="A1.Ex9.m1.1.1.1.1.4.2.1.3">superscript</csymbol><ci id="A1.Ex9.m1.1.1.1.1.4.2.1.3.2.2.cmml" xref="A1.Ex9.m1.1.1.1.1.4.2.1.3.2.2">𝑊</ci><ci id="A1.Ex9.m1.1.1.1.1.4.2.1.3.2.3.cmml" xref="A1.Ex9.m1.1.1.1.1.4.2.1.3.2.3">𝑘</ci></apply><cn type="integer" id="A1.Ex9.m1.1.1.1.1.4.2.1.3.3.cmml" xref="A1.Ex9.m1.1.1.1.1.4.2.1.3.3">0</cn></apply></apply><apply id="A1.Ex9.m1.1.1.1.1.4.2.2.cmml" xref="A1.Ex9.m1.1.1.1.1.4.2.2"><csymbol cd="ambiguous" id="A1.Ex9.m1.1.1.1.1.4.2.2.1.cmml" xref="A1.Ex9.m1.1.1.1.1.4.2.2">subscript</csymbol><ci id="A1.Ex9.m1.1.1.1.1.4.2.2.2.cmml" xref="A1.Ex9.m1.1.1.1.1.4.2.2.2">𝑙</ci><ci id="A1.Ex9.m1.1.1.1.1.4.2.2.3.cmml" xref="A1.Ex9.m1.1.1.1.1.4.2.2.3">𝑡</ci></apply></apply></apply></apply><apply id="A1.Ex9.m1.1.1.1.1c.cmml" xref="A1.Ex9.m1.1.1.1"><eq id="A1.Ex9.m1.1.1.1.1.5.cmml" xref="A1.Ex9.m1.1.1.1.1.5"></eq><share href="#A1.Ex9.m1.1.1.1.1.4.cmml" id="A1.Ex9.m1.1.1.1.1d.cmml" xref="A1.Ex9.m1.1.1.1"></share><apply id="A1.Ex9.m1.1.1.1.1.6.cmml" xref="A1.Ex9.m1.1.1.1.1.6"><apply id="A1.Ex9.m1.1.1.1.1.6.1.cmml" xref="A1.Ex9.m1.1.1.1.1.6.1"><csymbol cd="ambiguous" id="A1.Ex9.m1.1.1.1.1.6.1.1.cmml" xref="A1.Ex9.m1.1.1.1.1.6.1">superscript</csymbol><apply id="A1.Ex9.m1.1.1.1.1.6.1.2.cmml" xref="A1.Ex9.m1.1.1.1.1.6.1"><csymbol cd="ambiguous" id="A1.Ex9.m1.1.1.1.1.6.1.2.1.cmml" xref="A1.Ex9.m1.1.1.1.1.6.1">subscript</csymbol><sum id="A1.Ex9.m1.1.1.1.1.6.1.2.2.cmml" xref="A1.Ex9.m1.1.1.1.1.6.1.2.2"></sum><apply id="A1.Ex9.m1.1.1.1.1.6.1.2.3.cmml" xref="A1.Ex9.m1.1.1.1.1.6.1.2.3"><eq id="A1.Ex9.m1.1.1.1.1.6.1.2.3.1.cmml" xref="A1.Ex9.m1.1.1.1.1.6.1.2.3.1"></eq><ci id="A1.Ex9.m1.1.1.1.1.6.1.2.3.2.cmml" xref="A1.Ex9.m1.1.1.1.1.6.1.2.3.2">𝑡</ci><cn type="integer" id="A1.Ex9.m1.1.1.1.1.6.1.2.3.3.cmml" xref="A1.Ex9.m1.1.1.1.1.6.1.2.3.3">1</cn></apply></apply><ci id="A1.Ex9.m1.1.1.1.1.6.1.3.cmml" xref="A1.Ex9.m1.1.1.1.1.6.1.3">𝑏</ci></apply><apply id="A1.Ex9.m1.1.1.1.1.6.2.cmml" xref="A1.Ex9.m1.1.1.1.1.6.2"><csymbol cd="ambiguous" id="A1.Ex9.m1.1.1.1.1.6.2.1.cmml" xref="A1.Ex9.m1.1.1.1.1.6.2">subscript</csymbol><apply id="A1.Ex9.m1.1.1.1.1.6.2.2.cmml" xref="A1.Ex9.m1.1.1.1.1.6.2"><csymbol cd="ambiguous" id="A1.Ex9.m1.1.1.1.1.6.2.2.1.cmml" xref="A1.Ex9.m1.1.1.1.1.6.2">superscript</csymbol><ci id="A1.Ex9.m1.1.1.1.1.6.2.2.2.cmml" xref="A1.Ex9.m1.1.1.1.1.6.2.2.2">𝐺</ci><ci id="A1.Ex9.m1.1.1.1.1.6.2.2.3.cmml" xref="A1.Ex9.m1.1.1.1.1.6.2.2.3">𝑘</ci></apply><ci id="A1.Ex9.m1.1.1.1.1.6.2.3.cmml" xref="A1.Ex9.m1.1.1.1.1.6.2.3">𝑡</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex9.m1.1c">\nabla_{W_{0}^{k}}l=\sum_{t=1}^{b}\nabla_{W^{k}_{0}}l_{t}=\sum_{t=1}^{b}G^{k}_{t},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="A1.SS2.p3.4" class="ltx_p">so the computation of the individual terms in the sum <math id="A1.SS2.p3.1.m1.1" class="ltx_Math" alttext="G^{k}_{t}" display="inline"><semantics id="A1.SS2.p3.1.m1.1a"><msubsup id="A1.SS2.p3.1.m1.1.1" xref="A1.SS2.p3.1.m1.1.1.cmml"><mi id="A1.SS2.p3.1.m1.1.1.2.2" xref="A1.SS2.p3.1.m1.1.1.2.2.cmml">G</mi><mi id="A1.SS2.p3.1.m1.1.1.3" xref="A1.SS2.p3.1.m1.1.1.3.cmml">t</mi><mi id="A1.SS2.p3.1.m1.1.1.2.3" xref="A1.SS2.p3.1.m1.1.1.2.3.cmml">k</mi></msubsup><annotation-xml encoding="MathML-Content" id="A1.SS2.p3.1.m1.1b"><apply id="A1.SS2.p3.1.m1.1.1.cmml" xref="A1.SS2.p3.1.m1.1.1"><csymbol cd="ambiguous" id="A1.SS2.p3.1.m1.1.1.1.cmml" xref="A1.SS2.p3.1.m1.1.1">subscript</csymbol><apply id="A1.SS2.p3.1.m1.1.1.2.cmml" xref="A1.SS2.p3.1.m1.1.1"><csymbol cd="ambiguous" id="A1.SS2.p3.1.m1.1.1.2.1.cmml" xref="A1.SS2.p3.1.m1.1.1">superscript</csymbol><ci id="A1.SS2.p3.1.m1.1.1.2.2.cmml" xref="A1.SS2.p3.1.m1.1.1.2.2">𝐺</ci><ci id="A1.SS2.p3.1.m1.1.1.2.3.cmml" xref="A1.SS2.p3.1.m1.1.1.2.3">𝑘</ci></apply><ci id="A1.SS2.p3.1.m1.1.1.3.cmml" xref="A1.SS2.p3.1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p3.1.m1.1c">G^{k}_{t}</annotation></semantics></math> for <math id="A1.SS2.p3.2.m2.3" class="ltx_Math" alttext="t=1,\dots,b" display="inline"><semantics id="A1.SS2.p3.2.m2.3a"><mrow id="A1.SS2.p3.2.m2.3.4" xref="A1.SS2.p3.2.m2.3.4.cmml"><mi id="A1.SS2.p3.2.m2.3.4.2" xref="A1.SS2.p3.2.m2.3.4.2.cmml">t</mi><mo id="A1.SS2.p3.2.m2.3.4.1" xref="A1.SS2.p3.2.m2.3.4.1.cmml">=</mo><mrow id="A1.SS2.p3.2.m2.3.4.3.2" xref="A1.SS2.p3.2.m2.3.4.3.1.cmml"><mn id="A1.SS2.p3.2.m2.1.1" xref="A1.SS2.p3.2.m2.1.1.cmml">1</mn><mo id="A1.SS2.p3.2.m2.3.4.3.2.1" xref="A1.SS2.p3.2.m2.3.4.3.1.cmml">,</mo><mi mathvariant="normal" id="A1.SS2.p3.2.m2.2.2" xref="A1.SS2.p3.2.m2.2.2.cmml">…</mi><mo id="A1.SS2.p3.2.m2.3.4.3.2.2" xref="A1.SS2.p3.2.m2.3.4.3.1.cmml">,</mo><mi id="A1.SS2.p3.2.m2.3.3" xref="A1.SS2.p3.2.m2.3.3.cmml">b</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.p3.2.m2.3b"><apply id="A1.SS2.p3.2.m2.3.4.cmml" xref="A1.SS2.p3.2.m2.3.4"><eq id="A1.SS2.p3.2.m2.3.4.1.cmml" xref="A1.SS2.p3.2.m2.3.4.1"></eq><ci id="A1.SS2.p3.2.m2.3.4.2.cmml" xref="A1.SS2.p3.2.m2.3.4.2">𝑡</ci><list id="A1.SS2.p3.2.m2.3.4.3.1.cmml" xref="A1.SS2.p3.2.m2.3.4.3.2"><cn type="integer" id="A1.SS2.p3.2.m2.1.1.cmml" xref="A1.SS2.p3.2.m2.1.1">1</cn><ci id="A1.SS2.p3.2.m2.2.2.cmml" xref="A1.SS2.p3.2.m2.2.2">…</ci><ci id="A1.SS2.p3.2.m2.3.3.cmml" xref="A1.SS2.p3.2.m2.3.3">𝑏</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p3.2.m2.3c">t=1,\dots,b</annotation></semantics></math> cannot be batched together into <span id="A1.SS2.p3.4.1" class="ltx_text ltx_font_typewriter">matmul</span>s.
Similarly, the forward pass in primal form uses a different <math id="A1.SS2.p3.3.m3.1" class="ltx_Math" alttext="W_{t}" display="inline"><semantics id="A1.SS2.p3.3.m3.1a"><msub id="A1.SS2.p3.3.m3.1.1" xref="A1.SS2.p3.3.m3.1.1.cmml"><mi id="A1.SS2.p3.3.m3.1.1.2" xref="A1.SS2.p3.3.m3.1.1.2.cmml">W</mi><mi id="A1.SS2.p3.3.m3.1.1.3" xref="A1.SS2.p3.3.m3.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="A1.SS2.p3.3.m3.1b"><apply id="A1.SS2.p3.3.m3.1.1.cmml" xref="A1.SS2.p3.3.m3.1.1"><csymbol cd="ambiguous" id="A1.SS2.p3.3.m3.1.1.1.cmml" xref="A1.SS2.p3.3.m3.1.1">subscript</csymbol><ci id="A1.SS2.p3.3.m3.1.1.2.cmml" xref="A1.SS2.p3.3.m3.1.1.2">𝑊</ci><ci id="A1.SS2.p3.3.m3.1.1.3.cmml" xref="A1.SS2.p3.3.m3.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p3.3.m3.1c">W_{t}</annotation></semantics></math> for each <math id="A1.SS2.p3.4.m4.1" class="ltx_Math" alttext="\bar{x}_{t}" display="inline"><semantics id="A1.SS2.p3.4.m4.1a"><msub id="A1.SS2.p3.4.m4.1.1" xref="A1.SS2.p3.4.m4.1.1.cmml"><mover accent="true" id="A1.SS2.p3.4.m4.1.1.2" xref="A1.SS2.p3.4.m4.1.1.2.cmml"><mi id="A1.SS2.p3.4.m4.1.1.2.2" xref="A1.SS2.p3.4.m4.1.1.2.2.cmml">x</mi><mo id="A1.SS2.p3.4.m4.1.1.2.1" xref="A1.SS2.p3.4.m4.1.1.2.1.cmml">¯</mo></mover><mi id="A1.SS2.p3.4.m4.1.1.3" xref="A1.SS2.p3.4.m4.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="A1.SS2.p3.4.m4.1b"><apply id="A1.SS2.p3.4.m4.1.1.cmml" xref="A1.SS2.p3.4.m4.1.1"><csymbol cd="ambiguous" id="A1.SS2.p3.4.m4.1.1.1.cmml" xref="A1.SS2.p3.4.m4.1.1">subscript</csymbol><apply id="A1.SS2.p3.4.m4.1.1.2.cmml" xref="A1.SS2.p3.4.m4.1.1.2"><ci id="A1.SS2.p3.4.m4.1.1.2.1.cmml" xref="A1.SS2.p3.4.m4.1.1.2.1">¯</ci><ci id="A1.SS2.p3.4.m4.1.1.2.2.cmml" xref="A1.SS2.p3.4.m4.1.1.2.2">𝑥</ci></apply><ci id="A1.SS2.p3.4.m4.1.1.3.cmml" xref="A1.SS2.p3.4.m4.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p3.4.m4.1c">\bar{x}_{t}</annotation></semantics></math>, therefore also cannot be batched in the same way as a standard forward pass.
These non-standard passes have poor hardware efficiency.</p>
</div>
</section>
<section id="A1.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.3 </span>Dual form</h3>

<div id="A1.SS3.p1" class="ltx_para">
<p id="A1.SS3.p1.7" class="ltx_p">As discussed in Subsection <a href="#S2.SS5" title="2.5 Dual form ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.5</span></a>, the goal of the dual form is to compute <math id="A1.SS3.p1.1.m1.1" class="ltx_Math" alttext="\bar{X}^{K+1}" display="inline"><semantics id="A1.SS3.p1.1.m1.1a"><msup id="A1.SS3.p1.1.m1.1.1" xref="A1.SS3.p1.1.m1.1.1.cmml"><mover accent="true" id="A1.SS3.p1.1.m1.1.1.2" xref="A1.SS3.p1.1.m1.1.1.2.cmml"><mi id="A1.SS3.p1.1.m1.1.1.2.2" xref="A1.SS3.p1.1.m1.1.1.2.2.cmml">X</mi><mo id="A1.SS3.p1.1.m1.1.1.2.1" xref="A1.SS3.p1.1.m1.1.1.2.1.cmml">¯</mo></mover><mrow id="A1.SS3.p1.1.m1.1.1.3" xref="A1.SS3.p1.1.m1.1.1.3.cmml"><mi id="A1.SS3.p1.1.m1.1.1.3.2" xref="A1.SS3.p1.1.m1.1.1.3.2.cmml">K</mi><mo id="A1.SS3.p1.1.m1.1.1.3.1" xref="A1.SS3.p1.1.m1.1.1.3.1.cmml">+</mo><mn id="A1.SS3.p1.1.m1.1.1.3.3" xref="A1.SS3.p1.1.m1.1.1.3.3.cmml">1</mn></mrow></msup><annotation-xml encoding="MathML-Content" id="A1.SS3.p1.1.m1.1b"><apply id="A1.SS3.p1.1.m1.1.1.cmml" xref="A1.SS3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A1.SS3.p1.1.m1.1.1.1.cmml" xref="A1.SS3.p1.1.m1.1.1">superscript</csymbol><apply id="A1.SS3.p1.1.m1.1.1.2.cmml" xref="A1.SS3.p1.1.m1.1.1.2"><ci id="A1.SS3.p1.1.m1.1.1.2.1.cmml" xref="A1.SS3.p1.1.m1.1.1.2.1">¯</ci><ci id="A1.SS3.p1.1.m1.1.1.2.2.cmml" xref="A1.SS3.p1.1.m1.1.1.2.2">𝑋</ci></apply><apply id="A1.SS3.p1.1.m1.1.1.3.cmml" xref="A1.SS3.p1.1.m1.1.1.3"><plus id="A1.SS3.p1.1.m1.1.1.3.1.cmml" xref="A1.SS3.p1.1.m1.1.1.3.1"></plus><ci id="A1.SS3.p1.1.m1.1.1.3.2.cmml" xref="A1.SS3.p1.1.m1.1.1.3.2">𝐾</ci><cn type="integer" id="A1.SS3.p1.1.m1.1.1.3.3.cmml" xref="A1.SS3.p1.1.m1.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p1.1.m1.1c">\bar{X}^{K+1}</annotation></semantics></math> and <math id="A1.SS3.p1.2.m2.3" class="ltx_Math" alttext="W_{b}^{1},\dots,W_{b}^{K}" display="inline"><semantics id="A1.SS3.p1.2.m2.3a"><mrow id="A1.SS3.p1.2.m2.3.3.2" xref="A1.SS3.p1.2.m2.3.3.3.cmml"><msubsup id="A1.SS3.p1.2.m2.2.2.1.1" xref="A1.SS3.p1.2.m2.2.2.1.1.cmml"><mi id="A1.SS3.p1.2.m2.2.2.1.1.2.2" xref="A1.SS3.p1.2.m2.2.2.1.1.2.2.cmml">W</mi><mi id="A1.SS3.p1.2.m2.2.2.1.1.2.3" xref="A1.SS3.p1.2.m2.2.2.1.1.2.3.cmml">b</mi><mn id="A1.SS3.p1.2.m2.2.2.1.1.3" xref="A1.SS3.p1.2.m2.2.2.1.1.3.cmml">1</mn></msubsup><mo id="A1.SS3.p1.2.m2.3.3.2.3" xref="A1.SS3.p1.2.m2.3.3.3.cmml">,</mo><mi mathvariant="normal" id="A1.SS3.p1.2.m2.1.1" xref="A1.SS3.p1.2.m2.1.1.cmml">…</mi><mo id="A1.SS3.p1.2.m2.3.3.2.4" xref="A1.SS3.p1.2.m2.3.3.3.cmml">,</mo><msubsup id="A1.SS3.p1.2.m2.3.3.2.2" xref="A1.SS3.p1.2.m2.3.3.2.2.cmml"><mi id="A1.SS3.p1.2.m2.3.3.2.2.2.2" xref="A1.SS3.p1.2.m2.3.3.2.2.2.2.cmml">W</mi><mi id="A1.SS3.p1.2.m2.3.3.2.2.2.3" xref="A1.SS3.p1.2.m2.3.3.2.2.2.3.cmml">b</mi><mi id="A1.SS3.p1.2.m2.3.3.2.2.3" xref="A1.SS3.p1.2.m2.3.3.2.2.3.cmml">K</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.p1.2.m2.3b"><list id="A1.SS3.p1.2.m2.3.3.3.cmml" xref="A1.SS3.p1.2.m2.3.3.2"><apply id="A1.SS3.p1.2.m2.2.2.1.1.cmml" xref="A1.SS3.p1.2.m2.2.2.1.1"><csymbol cd="ambiguous" id="A1.SS3.p1.2.m2.2.2.1.1.1.cmml" xref="A1.SS3.p1.2.m2.2.2.1.1">superscript</csymbol><apply id="A1.SS3.p1.2.m2.2.2.1.1.2.cmml" xref="A1.SS3.p1.2.m2.2.2.1.1"><csymbol cd="ambiguous" id="A1.SS3.p1.2.m2.2.2.1.1.2.1.cmml" xref="A1.SS3.p1.2.m2.2.2.1.1">subscript</csymbol><ci id="A1.SS3.p1.2.m2.2.2.1.1.2.2.cmml" xref="A1.SS3.p1.2.m2.2.2.1.1.2.2">𝑊</ci><ci id="A1.SS3.p1.2.m2.2.2.1.1.2.3.cmml" xref="A1.SS3.p1.2.m2.2.2.1.1.2.3">𝑏</ci></apply><cn type="integer" id="A1.SS3.p1.2.m2.2.2.1.1.3.cmml" xref="A1.SS3.p1.2.m2.2.2.1.1.3">1</cn></apply><ci id="A1.SS3.p1.2.m2.1.1.cmml" xref="A1.SS3.p1.2.m2.1.1">…</ci><apply id="A1.SS3.p1.2.m2.3.3.2.2.cmml" xref="A1.SS3.p1.2.m2.3.3.2.2"><csymbol cd="ambiguous" id="A1.SS3.p1.2.m2.3.3.2.2.1.cmml" xref="A1.SS3.p1.2.m2.3.3.2.2">superscript</csymbol><apply id="A1.SS3.p1.2.m2.3.3.2.2.2.cmml" xref="A1.SS3.p1.2.m2.3.3.2.2"><csymbol cd="ambiguous" id="A1.SS3.p1.2.m2.3.3.2.2.2.1.cmml" xref="A1.SS3.p1.2.m2.3.3.2.2">subscript</csymbol><ci id="A1.SS3.p1.2.m2.3.3.2.2.2.2.cmml" xref="A1.SS3.p1.2.m2.3.3.2.2.2.2">𝑊</ci><ci id="A1.SS3.p1.2.m2.3.3.2.2.2.3.cmml" xref="A1.SS3.p1.2.m2.3.3.2.2.2.3">𝑏</ci></apply><ci id="A1.SS3.p1.2.m2.3.3.2.2.3.cmml" xref="A1.SS3.p1.2.m2.3.3.2.2.3">𝐾</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p1.2.m2.3c">W_{b}^{1},\dots,W_{b}^{K}</annotation></semantics></math> with only <span id="A1.SS3.p1.7.1" class="ltx_text ltx_font_typewriter">matmul</span>s and light-weight operations such as <span id="A1.SS3.p1.7.2" class="ltx_text ltx_font_typewriter">sum</span>s, <math id="A1.SS3.p1.3.m3.1" class="ltx_Math" alttext="\sigma" display="inline"><semantics id="A1.SS3.p1.3.m3.1a"><mi id="A1.SS3.p1.3.m3.1.1" xref="A1.SS3.p1.3.m3.1.1.cmml">σ</mi><annotation-xml encoding="MathML-Content" id="A1.SS3.p1.3.m3.1b"><ci id="A1.SS3.p1.3.m3.1.1.cmml" xref="A1.SS3.p1.3.m3.1.1">𝜎</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p1.3.m3.1c">\sigma</annotation></semantics></math>, and <math id="A1.SS3.p1.4.m4.1" class="ltx_Math" alttext="\sigma^{\prime}" display="inline"><semantics id="A1.SS3.p1.4.m4.1a"><msup id="A1.SS3.p1.4.m4.1.1" xref="A1.SS3.p1.4.m4.1.1.cmml"><mi id="A1.SS3.p1.4.m4.1.1.2" xref="A1.SS3.p1.4.m4.1.1.2.cmml">σ</mi><mo id="A1.SS3.p1.4.m4.1.1.3" xref="A1.SS3.p1.4.m4.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="A1.SS3.p1.4.m4.1b"><apply id="A1.SS3.p1.4.m4.1.1.cmml" xref="A1.SS3.p1.4.m4.1.1"><csymbol cd="ambiguous" id="A1.SS3.p1.4.m4.1.1.1.cmml" xref="A1.SS3.p1.4.m4.1.1">superscript</csymbol><ci id="A1.SS3.p1.4.m4.1.1.2.cmml" xref="A1.SS3.p1.4.m4.1.1.2">𝜎</ci><ci id="A1.SS3.p1.4.m4.1.1.3.cmml" xref="A1.SS3.p1.4.m4.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p1.4.m4.1c">\sigma^{\prime}</annotation></semantics></math>.
To achieve this goal, we avoid explicitly computing the intermediate variables: <math id="A1.SS3.p1.5.m5.1" class="ltx_Math" alttext="G^{k}_{t}" display="inline"><semantics id="A1.SS3.p1.5.m5.1a"><msubsup id="A1.SS3.p1.5.m5.1.1" xref="A1.SS3.p1.5.m5.1.1.cmml"><mi id="A1.SS3.p1.5.m5.1.1.2.2" xref="A1.SS3.p1.5.m5.1.1.2.2.cmml">G</mi><mi id="A1.SS3.p1.5.m5.1.1.3" xref="A1.SS3.p1.5.m5.1.1.3.cmml">t</mi><mi id="A1.SS3.p1.5.m5.1.1.2.3" xref="A1.SS3.p1.5.m5.1.1.2.3.cmml">k</mi></msubsup><annotation-xml encoding="MathML-Content" id="A1.SS3.p1.5.m5.1b"><apply id="A1.SS3.p1.5.m5.1.1.cmml" xref="A1.SS3.p1.5.m5.1.1"><csymbol cd="ambiguous" id="A1.SS3.p1.5.m5.1.1.1.cmml" xref="A1.SS3.p1.5.m5.1.1">subscript</csymbol><apply id="A1.SS3.p1.5.m5.1.1.2.cmml" xref="A1.SS3.p1.5.m5.1.1"><csymbol cd="ambiguous" id="A1.SS3.p1.5.m5.1.1.2.1.cmml" xref="A1.SS3.p1.5.m5.1.1">superscript</csymbol><ci id="A1.SS3.p1.5.m5.1.1.2.2.cmml" xref="A1.SS3.p1.5.m5.1.1.2.2">𝐺</ci><ci id="A1.SS3.p1.5.m5.1.1.2.3.cmml" xref="A1.SS3.p1.5.m5.1.1.2.3">𝑘</ci></apply><ci id="A1.SS3.p1.5.m5.1.1.3.cmml" xref="A1.SS3.p1.5.m5.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p1.5.m5.1c">G^{k}_{t}</annotation></semantics></math> and <math id="A1.SS3.p1.6.m6.1" class="ltx_Math" alttext="W^{k}_{t}" display="inline"><semantics id="A1.SS3.p1.6.m6.1a"><msubsup id="A1.SS3.p1.6.m6.1.1" xref="A1.SS3.p1.6.m6.1.1.cmml"><mi id="A1.SS3.p1.6.m6.1.1.2.2" xref="A1.SS3.p1.6.m6.1.1.2.2.cmml">W</mi><mi id="A1.SS3.p1.6.m6.1.1.3" xref="A1.SS3.p1.6.m6.1.1.3.cmml">t</mi><mi id="A1.SS3.p1.6.m6.1.1.2.3" xref="A1.SS3.p1.6.m6.1.1.2.3.cmml">k</mi></msubsup><annotation-xml encoding="MathML-Content" id="A1.SS3.p1.6.m6.1b"><apply id="A1.SS3.p1.6.m6.1.1.cmml" xref="A1.SS3.p1.6.m6.1.1"><csymbol cd="ambiguous" id="A1.SS3.p1.6.m6.1.1.1.cmml" xref="A1.SS3.p1.6.m6.1.1">subscript</csymbol><apply id="A1.SS3.p1.6.m6.1.1.2.cmml" xref="A1.SS3.p1.6.m6.1.1"><csymbol cd="ambiguous" id="A1.SS3.p1.6.m6.1.1.2.1.cmml" xref="A1.SS3.p1.6.m6.1.1">superscript</csymbol><ci id="A1.SS3.p1.6.m6.1.1.2.2.cmml" xref="A1.SS3.p1.6.m6.1.1.2.2">𝑊</ci><ci id="A1.SS3.p1.6.m6.1.1.2.3.cmml" xref="A1.SS3.p1.6.m6.1.1.2.3">𝑘</ci></apply><ci id="A1.SS3.p1.6.m6.1.1.3.cmml" xref="A1.SS3.p1.6.m6.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p1.6.m6.1c">W^{k}_{t}</annotation></semantics></math> for <math id="A1.SS3.p1.7.m7.3" class="ltx_Math" alttext="t=1,\dots,b" display="inline"><semantics id="A1.SS3.p1.7.m7.3a"><mrow id="A1.SS3.p1.7.m7.3.4" xref="A1.SS3.p1.7.m7.3.4.cmml"><mi id="A1.SS3.p1.7.m7.3.4.2" xref="A1.SS3.p1.7.m7.3.4.2.cmml">t</mi><mo id="A1.SS3.p1.7.m7.3.4.1" xref="A1.SS3.p1.7.m7.3.4.1.cmml">=</mo><mrow id="A1.SS3.p1.7.m7.3.4.3.2" xref="A1.SS3.p1.7.m7.3.4.3.1.cmml"><mn id="A1.SS3.p1.7.m7.1.1" xref="A1.SS3.p1.7.m7.1.1.cmml">1</mn><mo id="A1.SS3.p1.7.m7.3.4.3.2.1" xref="A1.SS3.p1.7.m7.3.4.3.1.cmml">,</mo><mi mathvariant="normal" id="A1.SS3.p1.7.m7.2.2" xref="A1.SS3.p1.7.m7.2.2.cmml">…</mi><mo id="A1.SS3.p1.7.m7.3.4.3.2.2" xref="A1.SS3.p1.7.m7.3.4.3.1.cmml">,</mo><mi id="A1.SS3.p1.7.m7.3.3" xref="A1.SS3.p1.7.m7.3.3.cmml">b</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.p1.7.m7.3b"><apply id="A1.SS3.p1.7.m7.3.4.cmml" xref="A1.SS3.p1.7.m7.3.4"><eq id="A1.SS3.p1.7.m7.3.4.1.cmml" xref="A1.SS3.p1.7.m7.3.4.1"></eq><ci id="A1.SS3.p1.7.m7.3.4.2.cmml" xref="A1.SS3.p1.7.m7.3.4.2">𝑡</ci><list id="A1.SS3.p1.7.m7.3.4.3.1.cmml" xref="A1.SS3.p1.7.m7.3.4.3.2"><cn type="integer" id="A1.SS3.p1.7.m7.1.1.cmml" xref="A1.SS3.p1.7.m7.1.1">1</cn><ci id="A1.SS3.p1.7.m7.2.2.cmml" xref="A1.SS3.p1.7.m7.2.2">…</ci><ci id="A1.SS3.p1.7.m7.3.3.cmml" xref="A1.SS3.p1.7.m7.3.3">𝑏</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p1.7.m7.3c">t=1,\dots,b</annotation></semantics></math>.</p>
</div>
<div id="A1.SS3.p2" class="ltx_para">
<p id="A1.SS3.p2.1" class="ltx_p">The dual form first computes
<math id="A1.SS3.p2.1.m1.1" class="ltx_Math" alttext="\nabla_{\hat{X}^{K+1}}l=\hat{X}^{K+1}-Y" display="inline"><semantics id="A1.SS3.p2.1.m1.1a"><mrow id="A1.SS3.p2.1.m1.1.1" xref="A1.SS3.p2.1.m1.1.1.cmml"><mrow id="A1.SS3.p2.1.m1.1.1.2" xref="A1.SS3.p2.1.m1.1.1.2.cmml"><msub id="A1.SS3.p2.1.m1.1.1.2.1" xref="A1.SS3.p2.1.m1.1.1.2.1.cmml"><mo id="A1.SS3.p2.1.m1.1.1.2.1.2" xref="A1.SS3.p2.1.m1.1.1.2.1.2.cmml">∇</mo><msup id="A1.SS3.p2.1.m1.1.1.2.1.3" xref="A1.SS3.p2.1.m1.1.1.2.1.3.cmml"><mover accent="true" id="A1.SS3.p2.1.m1.1.1.2.1.3.2" xref="A1.SS3.p2.1.m1.1.1.2.1.3.2.cmml"><mi id="A1.SS3.p2.1.m1.1.1.2.1.3.2.2" xref="A1.SS3.p2.1.m1.1.1.2.1.3.2.2.cmml">X</mi><mo id="A1.SS3.p2.1.m1.1.1.2.1.3.2.1" xref="A1.SS3.p2.1.m1.1.1.2.1.3.2.1.cmml">^</mo></mover><mrow id="A1.SS3.p2.1.m1.1.1.2.1.3.3" xref="A1.SS3.p2.1.m1.1.1.2.1.3.3.cmml"><mi id="A1.SS3.p2.1.m1.1.1.2.1.3.3.2" xref="A1.SS3.p2.1.m1.1.1.2.1.3.3.2.cmml">K</mi><mo id="A1.SS3.p2.1.m1.1.1.2.1.3.3.1" xref="A1.SS3.p2.1.m1.1.1.2.1.3.3.1.cmml">+</mo><mn id="A1.SS3.p2.1.m1.1.1.2.1.3.3.3" xref="A1.SS3.p2.1.m1.1.1.2.1.3.3.3.cmml">1</mn></mrow></msup></msub><mi id="A1.SS3.p2.1.m1.1.1.2.2" xref="A1.SS3.p2.1.m1.1.1.2.2.cmml">l</mi></mrow><mo id="A1.SS3.p2.1.m1.1.1.1" xref="A1.SS3.p2.1.m1.1.1.1.cmml">=</mo><mrow id="A1.SS3.p2.1.m1.1.1.3" xref="A1.SS3.p2.1.m1.1.1.3.cmml"><msup id="A1.SS3.p2.1.m1.1.1.3.2" xref="A1.SS3.p2.1.m1.1.1.3.2.cmml"><mover accent="true" id="A1.SS3.p2.1.m1.1.1.3.2.2" xref="A1.SS3.p2.1.m1.1.1.3.2.2.cmml"><mi id="A1.SS3.p2.1.m1.1.1.3.2.2.2" xref="A1.SS3.p2.1.m1.1.1.3.2.2.2.cmml">X</mi><mo id="A1.SS3.p2.1.m1.1.1.3.2.2.1" xref="A1.SS3.p2.1.m1.1.1.3.2.2.1.cmml">^</mo></mover><mrow id="A1.SS3.p2.1.m1.1.1.3.2.3" xref="A1.SS3.p2.1.m1.1.1.3.2.3.cmml"><mi id="A1.SS3.p2.1.m1.1.1.3.2.3.2" xref="A1.SS3.p2.1.m1.1.1.3.2.3.2.cmml">K</mi><mo id="A1.SS3.p2.1.m1.1.1.3.2.3.1" xref="A1.SS3.p2.1.m1.1.1.3.2.3.1.cmml">+</mo><mn id="A1.SS3.p2.1.m1.1.1.3.2.3.3" xref="A1.SS3.p2.1.m1.1.1.3.2.3.3.cmml">1</mn></mrow></msup><mo id="A1.SS3.p2.1.m1.1.1.3.1" xref="A1.SS3.p2.1.m1.1.1.3.1.cmml">−</mo><mi id="A1.SS3.p2.1.m1.1.1.3.3" xref="A1.SS3.p2.1.m1.1.1.3.3.cmml">Y</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.p2.1.m1.1b"><apply id="A1.SS3.p2.1.m1.1.1.cmml" xref="A1.SS3.p2.1.m1.1.1"><eq id="A1.SS3.p2.1.m1.1.1.1.cmml" xref="A1.SS3.p2.1.m1.1.1.1"></eq><apply id="A1.SS3.p2.1.m1.1.1.2.cmml" xref="A1.SS3.p2.1.m1.1.1.2"><apply id="A1.SS3.p2.1.m1.1.1.2.1.cmml" xref="A1.SS3.p2.1.m1.1.1.2.1"><csymbol cd="ambiguous" id="A1.SS3.p2.1.m1.1.1.2.1.1.cmml" xref="A1.SS3.p2.1.m1.1.1.2.1">subscript</csymbol><ci id="A1.SS3.p2.1.m1.1.1.2.1.2.cmml" xref="A1.SS3.p2.1.m1.1.1.2.1.2">∇</ci><apply id="A1.SS3.p2.1.m1.1.1.2.1.3.cmml" xref="A1.SS3.p2.1.m1.1.1.2.1.3"><csymbol cd="ambiguous" id="A1.SS3.p2.1.m1.1.1.2.1.3.1.cmml" xref="A1.SS3.p2.1.m1.1.1.2.1.3">superscript</csymbol><apply id="A1.SS3.p2.1.m1.1.1.2.1.3.2.cmml" xref="A1.SS3.p2.1.m1.1.1.2.1.3.2"><ci id="A1.SS3.p2.1.m1.1.1.2.1.3.2.1.cmml" xref="A1.SS3.p2.1.m1.1.1.2.1.3.2.1">^</ci><ci id="A1.SS3.p2.1.m1.1.1.2.1.3.2.2.cmml" xref="A1.SS3.p2.1.m1.1.1.2.1.3.2.2">𝑋</ci></apply><apply id="A1.SS3.p2.1.m1.1.1.2.1.3.3.cmml" xref="A1.SS3.p2.1.m1.1.1.2.1.3.3"><plus id="A1.SS3.p2.1.m1.1.1.2.1.3.3.1.cmml" xref="A1.SS3.p2.1.m1.1.1.2.1.3.3.1"></plus><ci id="A1.SS3.p2.1.m1.1.1.2.1.3.3.2.cmml" xref="A1.SS3.p2.1.m1.1.1.2.1.3.3.2">𝐾</ci><cn type="integer" id="A1.SS3.p2.1.m1.1.1.2.1.3.3.3.cmml" xref="A1.SS3.p2.1.m1.1.1.2.1.3.3.3">1</cn></apply></apply></apply><ci id="A1.SS3.p2.1.m1.1.1.2.2.cmml" xref="A1.SS3.p2.1.m1.1.1.2.2">𝑙</ci></apply><apply id="A1.SS3.p2.1.m1.1.1.3.cmml" xref="A1.SS3.p2.1.m1.1.1.3"><minus id="A1.SS3.p2.1.m1.1.1.3.1.cmml" xref="A1.SS3.p2.1.m1.1.1.3.1"></minus><apply id="A1.SS3.p2.1.m1.1.1.3.2.cmml" xref="A1.SS3.p2.1.m1.1.1.3.2"><csymbol cd="ambiguous" id="A1.SS3.p2.1.m1.1.1.3.2.1.cmml" xref="A1.SS3.p2.1.m1.1.1.3.2">superscript</csymbol><apply id="A1.SS3.p2.1.m1.1.1.3.2.2.cmml" xref="A1.SS3.p2.1.m1.1.1.3.2.2"><ci id="A1.SS3.p2.1.m1.1.1.3.2.2.1.cmml" xref="A1.SS3.p2.1.m1.1.1.3.2.2.1">^</ci><ci id="A1.SS3.p2.1.m1.1.1.3.2.2.2.cmml" xref="A1.SS3.p2.1.m1.1.1.3.2.2.2">𝑋</ci></apply><apply id="A1.SS3.p2.1.m1.1.1.3.2.3.cmml" xref="A1.SS3.p2.1.m1.1.1.3.2.3"><plus id="A1.SS3.p2.1.m1.1.1.3.2.3.1.cmml" xref="A1.SS3.p2.1.m1.1.1.3.2.3.1"></plus><ci id="A1.SS3.p2.1.m1.1.1.3.2.3.2.cmml" xref="A1.SS3.p2.1.m1.1.1.3.2.3.2">𝐾</ci><cn type="integer" id="A1.SS3.p2.1.m1.1.1.3.2.3.3.cmml" xref="A1.SS3.p2.1.m1.1.1.3.2.3.3">1</cn></apply></apply><ci id="A1.SS3.p2.1.m1.1.1.3.3.cmml" xref="A1.SS3.p2.1.m1.1.1.3.3">𝑌</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p2.1.m1.1c">\nabla_{\hat{X}^{K+1}}l=\hat{X}^{K+1}-Y</annotation></semantics></math>, then takes a standard backward pass.</p>
</div>
<div id="A1.SS3.p3" class="ltx_para">
<p id="A1.SS3.p3.1" class="ltx_p">For <math id="A1.SS3.p3.1.m1.3" class="ltx_Math" alttext="k=K,\dots,1" display="inline"><semantics id="A1.SS3.p3.1.m1.3a"><mrow id="A1.SS3.p3.1.m1.3.4" xref="A1.SS3.p3.1.m1.3.4.cmml"><mi id="A1.SS3.p3.1.m1.3.4.2" xref="A1.SS3.p3.1.m1.3.4.2.cmml">k</mi><mo id="A1.SS3.p3.1.m1.3.4.1" xref="A1.SS3.p3.1.m1.3.4.1.cmml">=</mo><mrow id="A1.SS3.p3.1.m1.3.4.3.2" xref="A1.SS3.p3.1.m1.3.4.3.1.cmml"><mi id="A1.SS3.p3.1.m1.1.1" xref="A1.SS3.p3.1.m1.1.1.cmml">K</mi><mo id="A1.SS3.p3.1.m1.3.4.3.2.1" xref="A1.SS3.p3.1.m1.3.4.3.1.cmml">,</mo><mi mathvariant="normal" id="A1.SS3.p3.1.m1.2.2" xref="A1.SS3.p3.1.m1.2.2.cmml">…</mi><mo id="A1.SS3.p3.1.m1.3.4.3.2.2" xref="A1.SS3.p3.1.m1.3.4.3.1.cmml">,</mo><mn id="A1.SS3.p3.1.m1.3.3" xref="A1.SS3.p3.1.m1.3.3.cmml">1</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.p3.1.m1.3b"><apply id="A1.SS3.p3.1.m1.3.4.cmml" xref="A1.SS3.p3.1.m1.3.4"><eq id="A1.SS3.p3.1.m1.3.4.1.cmml" xref="A1.SS3.p3.1.m1.3.4.1"></eq><ci id="A1.SS3.p3.1.m1.3.4.2.cmml" xref="A1.SS3.p3.1.m1.3.4.2">𝑘</ci><list id="A1.SS3.p3.1.m1.3.4.3.1.cmml" xref="A1.SS3.p3.1.m1.3.4.3.2"><ci id="A1.SS3.p3.1.m1.1.1.cmml" xref="A1.SS3.p3.1.m1.1.1">𝐾</ci><ci id="A1.SS3.p3.1.m1.2.2.cmml" xref="A1.SS3.p3.1.m1.2.2">…</ci><cn type="integer" id="A1.SS3.p3.1.m1.3.3.cmml" xref="A1.SS3.p3.1.m1.3.3">1</cn></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p3.1.m1.3c">k=K,\dots,1</annotation></semantics></math>:</p>
<ul id="A1.I3" class="ltx_itemize">
<li id="A1.I3.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I3.i1.p1" class="ltx_para">
<p id="A1.I3.i1.p1.1" class="ltx_p"><math id="A1.I3.i1.p1.1.m1.1" class="ltx_Math" alttext="\nabla_{Z^{k}}l=\sigma^{\prime}_{k}\left(\nabla_{\hat{X}^{k+1}}l\right)" display="inline"><semantics id="A1.I3.i1.p1.1.m1.1a"><mrow id="A1.I3.i1.p1.1.m1.1.1" xref="A1.I3.i1.p1.1.m1.1.1.cmml"><mrow id="A1.I3.i1.p1.1.m1.1.1.3" xref="A1.I3.i1.p1.1.m1.1.1.3.cmml"><msub id="A1.I3.i1.p1.1.m1.1.1.3.1" xref="A1.I3.i1.p1.1.m1.1.1.3.1.cmml"><mo id="A1.I3.i1.p1.1.m1.1.1.3.1.2" xref="A1.I3.i1.p1.1.m1.1.1.3.1.2.cmml">∇</mo><msup id="A1.I3.i1.p1.1.m1.1.1.3.1.3" xref="A1.I3.i1.p1.1.m1.1.1.3.1.3.cmml"><mi id="A1.I3.i1.p1.1.m1.1.1.3.1.3.2" xref="A1.I3.i1.p1.1.m1.1.1.3.1.3.2.cmml">Z</mi><mi id="A1.I3.i1.p1.1.m1.1.1.3.1.3.3" xref="A1.I3.i1.p1.1.m1.1.1.3.1.3.3.cmml">k</mi></msup></msub><mi id="A1.I3.i1.p1.1.m1.1.1.3.2" xref="A1.I3.i1.p1.1.m1.1.1.3.2.cmml">l</mi></mrow><mo id="A1.I3.i1.p1.1.m1.1.1.2" xref="A1.I3.i1.p1.1.m1.1.1.2.cmml">=</mo><mrow id="A1.I3.i1.p1.1.m1.1.1.1" xref="A1.I3.i1.p1.1.m1.1.1.1.cmml"><msubsup id="A1.I3.i1.p1.1.m1.1.1.1.3" xref="A1.I3.i1.p1.1.m1.1.1.1.3.cmml"><mi id="A1.I3.i1.p1.1.m1.1.1.1.3.2.2" xref="A1.I3.i1.p1.1.m1.1.1.1.3.2.2.cmml">σ</mi><mi id="A1.I3.i1.p1.1.m1.1.1.1.3.3" xref="A1.I3.i1.p1.1.m1.1.1.1.3.3.cmml">k</mi><mo id="A1.I3.i1.p1.1.m1.1.1.1.3.2.3" xref="A1.I3.i1.p1.1.m1.1.1.1.3.2.3.cmml">′</mo></msubsup><mo lspace="0em" rspace="0em" id="A1.I3.i1.p1.1.m1.1.1.1.2" xref="A1.I3.i1.p1.1.m1.1.1.1.2.cmml">​</mo><mrow id="A1.I3.i1.p1.1.m1.1.1.1.1.1" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.cmml"><mo id="A1.I3.i1.p1.1.m1.1.1.1.1.1.2" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.cmml"><msub id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.cmml"><mo rspace="0.167em" id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.2" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.2.cmml">∇</mo><msup id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.cmml"><mover accent="true" id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.2" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.2.cmml"><mi id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.2.2" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.2.2.cmml">X</mi><mo id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.2.1" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.2.1.cmml">^</mo></mover><mrow id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.3" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.3.cmml"><mi id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.3.2" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.3.2.cmml">k</mi><mo id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.3.1" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.3.1.cmml">+</mo><mn id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.3.3" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.3.3.cmml">1</mn></mrow></msup></msub><mi id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.2" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.2.cmml">l</mi></mrow><mo id="A1.I3.i1.p1.1.m1.1.1.1.1.1.3" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.I3.i1.p1.1.m1.1b"><apply id="A1.I3.i1.p1.1.m1.1.1.cmml" xref="A1.I3.i1.p1.1.m1.1.1"><eq id="A1.I3.i1.p1.1.m1.1.1.2.cmml" xref="A1.I3.i1.p1.1.m1.1.1.2"></eq><apply id="A1.I3.i1.p1.1.m1.1.1.3.cmml" xref="A1.I3.i1.p1.1.m1.1.1.3"><apply id="A1.I3.i1.p1.1.m1.1.1.3.1.cmml" xref="A1.I3.i1.p1.1.m1.1.1.3.1"><csymbol cd="ambiguous" id="A1.I3.i1.p1.1.m1.1.1.3.1.1.cmml" xref="A1.I3.i1.p1.1.m1.1.1.3.1">subscript</csymbol><ci id="A1.I3.i1.p1.1.m1.1.1.3.1.2.cmml" xref="A1.I3.i1.p1.1.m1.1.1.3.1.2">∇</ci><apply id="A1.I3.i1.p1.1.m1.1.1.3.1.3.cmml" xref="A1.I3.i1.p1.1.m1.1.1.3.1.3"><csymbol cd="ambiguous" id="A1.I3.i1.p1.1.m1.1.1.3.1.3.1.cmml" xref="A1.I3.i1.p1.1.m1.1.1.3.1.3">superscript</csymbol><ci id="A1.I3.i1.p1.1.m1.1.1.3.1.3.2.cmml" xref="A1.I3.i1.p1.1.m1.1.1.3.1.3.2">𝑍</ci><ci id="A1.I3.i1.p1.1.m1.1.1.3.1.3.3.cmml" xref="A1.I3.i1.p1.1.m1.1.1.3.1.3.3">𝑘</ci></apply></apply><ci id="A1.I3.i1.p1.1.m1.1.1.3.2.cmml" xref="A1.I3.i1.p1.1.m1.1.1.3.2">𝑙</ci></apply><apply id="A1.I3.i1.p1.1.m1.1.1.1.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1"><times id="A1.I3.i1.p1.1.m1.1.1.1.2.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.2"></times><apply id="A1.I3.i1.p1.1.m1.1.1.1.3.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.3"><csymbol cd="ambiguous" id="A1.I3.i1.p1.1.m1.1.1.1.3.1.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.3">subscript</csymbol><apply id="A1.I3.i1.p1.1.m1.1.1.1.3.2.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.3"><csymbol cd="ambiguous" id="A1.I3.i1.p1.1.m1.1.1.1.3.2.1.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.3">superscript</csymbol><ci id="A1.I3.i1.p1.1.m1.1.1.1.3.2.2.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.3.2.2">𝜎</ci><ci id="A1.I3.i1.p1.1.m1.1.1.1.3.2.3.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.3.2.3">′</ci></apply><ci id="A1.I3.i1.p1.1.m1.1.1.1.3.3.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.3.3">𝑘</ci></apply><apply id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1"><apply id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.1.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1">subscript</csymbol><ci id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.2.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.2">∇</ci><apply id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.1.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3">superscript</csymbol><apply id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.2.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.2"><ci id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.2.1.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.2.1">^</ci><ci id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.2.2.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.2.2">𝑋</ci></apply><apply id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.3.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.3"><plus id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.3.1.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.3.1"></plus><ci id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.3.2.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.3.2">𝑘</ci><cn type="integer" id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.3.3.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.1.3.3.3">1</cn></apply></apply></apply><ci id="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.2.cmml" xref="A1.I3.i1.p1.1.m1.1.1.1.1.1.1.2">𝑙</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I3.i1.p1.1.m1.1c">\nabla_{Z^{k}}l=\sigma^{\prime}_{k}\left(\nabla_{\hat{X}^{k+1}}l\right)</annotation></semantics></math></p>
</div>
</li>
<li id="A1.I3.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I3.i2.p1" class="ltx_para">
<p id="A1.I3.i2.p1.1" class="ltx_p"><math id="A1.I3.i2.p1.1.m1.1" class="ltx_Math" alttext="\nabla_{\hat{X}^{k}}l=\left(W_{0}^{k}\right)^{T}\,\nabla_{Z^{k}}l" display="inline"><semantics id="A1.I3.i2.p1.1.m1.1a"><mrow id="A1.I3.i2.p1.1.m1.1.1" xref="A1.I3.i2.p1.1.m1.1.1.cmml"><mrow id="A1.I3.i2.p1.1.m1.1.1.3" xref="A1.I3.i2.p1.1.m1.1.1.3.cmml"><msub id="A1.I3.i2.p1.1.m1.1.1.3.1" xref="A1.I3.i2.p1.1.m1.1.1.3.1.cmml"><mo id="A1.I3.i2.p1.1.m1.1.1.3.1.2" xref="A1.I3.i2.p1.1.m1.1.1.3.1.2.cmml">∇</mo><msup id="A1.I3.i2.p1.1.m1.1.1.3.1.3" xref="A1.I3.i2.p1.1.m1.1.1.3.1.3.cmml"><mover accent="true" id="A1.I3.i2.p1.1.m1.1.1.3.1.3.2" xref="A1.I3.i2.p1.1.m1.1.1.3.1.3.2.cmml"><mi id="A1.I3.i2.p1.1.m1.1.1.3.1.3.2.2" xref="A1.I3.i2.p1.1.m1.1.1.3.1.3.2.2.cmml">X</mi><mo id="A1.I3.i2.p1.1.m1.1.1.3.1.3.2.1" xref="A1.I3.i2.p1.1.m1.1.1.3.1.3.2.1.cmml">^</mo></mover><mi id="A1.I3.i2.p1.1.m1.1.1.3.1.3.3" xref="A1.I3.i2.p1.1.m1.1.1.3.1.3.3.cmml">k</mi></msup></msub><mi id="A1.I3.i2.p1.1.m1.1.1.3.2" xref="A1.I3.i2.p1.1.m1.1.1.3.2.cmml">l</mi></mrow><mo id="A1.I3.i2.p1.1.m1.1.1.2" xref="A1.I3.i2.p1.1.m1.1.1.2.cmml">=</mo><mrow id="A1.I3.i2.p1.1.m1.1.1.1" xref="A1.I3.i2.p1.1.m1.1.1.1.cmml"><msup id="A1.I3.i2.p1.1.m1.1.1.1.1" xref="A1.I3.i2.p1.1.m1.1.1.1.1.cmml"><mrow id="A1.I3.i2.p1.1.m1.1.1.1.1.1.1" xref="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.1.cmml"><mo id="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.2" xref="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.1" xref="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.1.cmml"><mi id="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.1.2.2" xref="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.1.2.2.cmml">W</mi><mn id="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.1.2.3" xref="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.1.2.3.cmml">0</mn><mi id="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.1.3" xref="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.1.3.cmml">k</mi></msubsup><mo id="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.3" xref="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mi id="A1.I3.i2.p1.1.m1.1.1.1.1.3" xref="A1.I3.i2.p1.1.m1.1.1.1.1.3.cmml">T</mi></msup><mo lspace="0.167em" rspace="0em" id="A1.I3.i2.p1.1.m1.1.1.1.2" xref="A1.I3.i2.p1.1.m1.1.1.1.2.cmml">​</mo><mrow id="A1.I3.i2.p1.1.m1.1.1.1.3" xref="A1.I3.i2.p1.1.m1.1.1.1.3.cmml"><msub id="A1.I3.i2.p1.1.m1.1.1.1.3.1" xref="A1.I3.i2.p1.1.m1.1.1.1.3.1.cmml"><mo rspace="0.167em" id="A1.I3.i2.p1.1.m1.1.1.1.3.1.2" xref="A1.I3.i2.p1.1.m1.1.1.1.3.1.2.cmml">∇</mo><msup id="A1.I3.i2.p1.1.m1.1.1.1.3.1.3" xref="A1.I3.i2.p1.1.m1.1.1.1.3.1.3.cmml"><mi id="A1.I3.i2.p1.1.m1.1.1.1.3.1.3.2" xref="A1.I3.i2.p1.1.m1.1.1.1.3.1.3.2.cmml">Z</mi><mi id="A1.I3.i2.p1.1.m1.1.1.1.3.1.3.3" xref="A1.I3.i2.p1.1.m1.1.1.1.3.1.3.3.cmml">k</mi></msup></msub><mi id="A1.I3.i2.p1.1.m1.1.1.1.3.2" xref="A1.I3.i2.p1.1.m1.1.1.1.3.2.cmml">l</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.I3.i2.p1.1.m1.1b"><apply id="A1.I3.i2.p1.1.m1.1.1.cmml" xref="A1.I3.i2.p1.1.m1.1.1"><eq id="A1.I3.i2.p1.1.m1.1.1.2.cmml" xref="A1.I3.i2.p1.1.m1.1.1.2"></eq><apply id="A1.I3.i2.p1.1.m1.1.1.3.cmml" xref="A1.I3.i2.p1.1.m1.1.1.3"><apply id="A1.I3.i2.p1.1.m1.1.1.3.1.cmml" xref="A1.I3.i2.p1.1.m1.1.1.3.1"><csymbol cd="ambiguous" id="A1.I3.i2.p1.1.m1.1.1.3.1.1.cmml" xref="A1.I3.i2.p1.1.m1.1.1.3.1">subscript</csymbol><ci id="A1.I3.i2.p1.1.m1.1.1.3.1.2.cmml" xref="A1.I3.i2.p1.1.m1.1.1.3.1.2">∇</ci><apply id="A1.I3.i2.p1.1.m1.1.1.3.1.3.cmml" xref="A1.I3.i2.p1.1.m1.1.1.3.1.3"><csymbol cd="ambiguous" id="A1.I3.i2.p1.1.m1.1.1.3.1.3.1.cmml" xref="A1.I3.i2.p1.1.m1.1.1.3.1.3">superscript</csymbol><apply id="A1.I3.i2.p1.1.m1.1.1.3.1.3.2.cmml" xref="A1.I3.i2.p1.1.m1.1.1.3.1.3.2"><ci id="A1.I3.i2.p1.1.m1.1.1.3.1.3.2.1.cmml" xref="A1.I3.i2.p1.1.m1.1.1.3.1.3.2.1">^</ci><ci id="A1.I3.i2.p1.1.m1.1.1.3.1.3.2.2.cmml" xref="A1.I3.i2.p1.1.m1.1.1.3.1.3.2.2">𝑋</ci></apply><ci id="A1.I3.i2.p1.1.m1.1.1.3.1.3.3.cmml" xref="A1.I3.i2.p1.1.m1.1.1.3.1.3.3">𝑘</ci></apply></apply><ci id="A1.I3.i2.p1.1.m1.1.1.3.2.cmml" xref="A1.I3.i2.p1.1.m1.1.1.3.2">𝑙</ci></apply><apply id="A1.I3.i2.p1.1.m1.1.1.1.cmml" xref="A1.I3.i2.p1.1.m1.1.1.1"><times id="A1.I3.i2.p1.1.m1.1.1.1.2.cmml" xref="A1.I3.i2.p1.1.m1.1.1.1.2"></times><apply id="A1.I3.i2.p1.1.m1.1.1.1.1.cmml" xref="A1.I3.i2.p1.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="A1.I3.i2.p1.1.m1.1.1.1.1.2.cmml" xref="A1.I3.i2.p1.1.m1.1.1.1.1">superscript</csymbol><apply id="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.1.cmml" xref="A1.I3.i2.p1.1.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.1.1.cmml" xref="A1.I3.i2.p1.1.m1.1.1.1.1.1.1">superscript</csymbol><apply id="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.1.2.cmml" xref="A1.I3.i2.p1.1.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.1.2.1.cmml" xref="A1.I3.i2.p1.1.m1.1.1.1.1.1.1">subscript</csymbol><ci id="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.1.2.2.cmml" xref="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.1.2.2">𝑊</ci><cn type="integer" id="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.1.2.3.cmml" xref="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.1.2.3">0</cn></apply><ci id="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.1.3.cmml" xref="A1.I3.i2.p1.1.m1.1.1.1.1.1.1.1.3">𝑘</ci></apply><ci id="A1.I3.i2.p1.1.m1.1.1.1.1.3.cmml" xref="A1.I3.i2.p1.1.m1.1.1.1.1.3">𝑇</ci></apply><apply id="A1.I3.i2.p1.1.m1.1.1.1.3.cmml" xref="A1.I3.i2.p1.1.m1.1.1.1.3"><apply id="A1.I3.i2.p1.1.m1.1.1.1.3.1.cmml" xref="A1.I3.i2.p1.1.m1.1.1.1.3.1"><csymbol cd="ambiguous" id="A1.I3.i2.p1.1.m1.1.1.1.3.1.1.cmml" xref="A1.I3.i2.p1.1.m1.1.1.1.3.1">subscript</csymbol><ci id="A1.I3.i2.p1.1.m1.1.1.1.3.1.2.cmml" xref="A1.I3.i2.p1.1.m1.1.1.1.3.1.2">∇</ci><apply id="A1.I3.i2.p1.1.m1.1.1.1.3.1.3.cmml" xref="A1.I3.i2.p1.1.m1.1.1.1.3.1.3"><csymbol cd="ambiguous" id="A1.I3.i2.p1.1.m1.1.1.1.3.1.3.1.cmml" xref="A1.I3.i2.p1.1.m1.1.1.1.3.1.3">superscript</csymbol><ci id="A1.I3.i2.p1.1.m1.1.1.1.3.1.3.2.cmml" xref="A1.I3.i2.p1.1.m1.1.1.1.3.1.3.2">𝑍</ci><ci id="A1.I3.i2.p1.1.m1.1.1.1.3.1.3.3.cmml" xref="A1.I3.i2.p1.1.m1.1.1.1.3.1.3.3">𝑘</ci></apply></apply><ci id="A1.I3.i2.p1.1.m1.1.1.1.3.2.cmml" xref="A1.I3.i2.p1.1.m1.1.1.1.3.2">𝑙</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I3.i2.p1.1.m1.1c">\nabla_{\hat{X}^{k}}l=\left(W_{0}^{k}\right)^{T}\,\nabla_{Z^{k}}l</annotation></semantics></math></p>
</div>
</li>
<li id="A1.I3.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I3.i3.p1" class="ltx_para">
<p id="A1.I3.i3.p1.1" class="ltx_p"><math id="A1.I3.i3.p1.1.m1.1" class="ltx_Math" alttext="\nabla_{W_{0}^{k}}l=\nabla_{Z^{k}}l\,\left(\hat{X}^{k}\right)^{T}" display="inline"><semantics id="A1.I3.i3.p1.1.m1.1a"><mrow id="A1.I3.i3.p1.1.m1.1.1" xref="A1.I3.i3.p1.1.m1.1.1.cmml"><mrow id="A1.I3.i3.p1.1.m1.1.1.3" xref="A1.I3.i3.p1.1.m1.1.1.3.cmml"><msub id="A1.I3.i3.p1.1.m1.1.1.3.1" xref="A1.I3.i3.p1.1.m1.1.1.3.1.cmml"><mo id="A1.I3.i3.p1.1.m1.1.1.3.1.2" xref="A1.I3.i3.p1.1.m1.1.1.3.1.2.cmml">∇</mo><msubsup id="A1.I3.i3.p1.1.m1.1.1.3.1.3" xref="A1.I3.i3.p1.1.m1.1.1.3.1.3.cmml"><mi id="A1.I3.i3.p1.1.m1.1.1.3.1.3.2.2" xref="A1.I3.i3.p1.1.m1.1.1.3.1.3.2.2.cmml">W</mi><mn id="A1.I3.i3.p1.1.m1.1.1.3.1.3.2.3" xref="A1.I3.i3.p1.1.m1.1.1.3.1.3.2.3.cmml">0</mn><mi id="A1.I3.i3.p1.1.m1.1.1.3.1.3.3" xref="A1.I3.i3.p1.1.m1.1.1.3.1.3.3.cmml">k</mi></msubsup></msub><mi id="A1.I3.i3.p1.1.m1.1.1.3.2" xref="A1.I3.i3.p1.1.m1.1.1.3.2.cmml">l</mi></mrow><mo id="A1.I3.i3.p1.1.m1.1.1.2" xref="A1.I3.i3.p1.1.m1.1.1.2.cmml">=</mo><mrow id="A1.I3.i3.p1.1.m1.1.1.1" xref="A1.I3.i3.p1.1.m1.1.1.1.cmml"><mrow id="A1.I3.i3.p1.1.m1.1.1.1.3" xref="A1.I3.i3.p1.1.m1.1.1.1.3.cmml"><msub id="A1.I3.i3.p1.1.m1.1.1.1.3.1" xref="A1.I3.i3.p1.1.m1.1.1.1.3.1.cmml"><mo rspace="0.167em" id="A1.I3.i3.p1.1.m1.1.1.1.3.1.2" xref="A1.I3.i3.p1.1.m1.1.1.1.3.1.2.cmml">∇</mo><msup id="A1.I3.i3.p1.1.m1.1.1.1.3.1.3" xref="A1.I3.i3.p1.1.m1.1.1.1.3.1.3.cmml"><mi id="A1.I3.i3.p1.1.m1.1.1.1.3.1.3.2" xref="A1.I3.i3.p1.1.m1.1.1.1.3.1.3.2.cmml">Z</mi><mi id="A1.I3.i3.p1.1.m1.1.1.1.3.1.3.3" xref="A1.I3.i3.p1.1.m1.1.1.1.3.1.3.3.cmml">k</mi></msup></msub><mi id="A1.I3.i3.p1.1.m1.1.1.1.3.2" xref="A1.I3.i3.p1.1.m1.1.1.1.3.2.cmml">l</mi></mrow><mo lspace="0.170em" rspace="0em" id="A1.I3.i3.p1.1.m1.1.1.1.2" xref="A1.I3.i3.p1.1.m1.1.1.1.2.cmml">​</mo><msup id="A1.I3.i3.p1.1.m1.1.1.1.1" xref="A1.I3.i3.p1.1.m1.1.1.1.1.cmml"><mrow id="A1.I3.i3.p1.1.m1.1.1.1.1.1.1" xref="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.cmml"><mo id="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.2" xref="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.cmml">(</mo><msup id="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1" xref="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.cmml"><mover accent="true" id="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.2" xref="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.2.cmml"><mi id="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.2.2" xref="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.2.2.cmml">X</mi><mo id="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.2.1" xref="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.2.1.cmml">^</mo></mover><mi id="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.3" xref="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.3.cmml">k</mi></msup><mo id="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.3" xref="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mi id="A1.I3.i3.p1.1.m1.1.1.1.1.3" xref="A1.I3.i3.p1.1.m1.1.1.1.1.3.cmml">T</mi></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.I3.i3.p1.1.m1.1b"><apply id="A1.I3.i3.p1.1.m1.1.1.cmml" xref="A1.I3.i3.p1.1.m1.1.1"><eq id="A1.I3.i3.p1.1.m1.1.1.2.cmml" xref="A1.I3.i3.p1.1.m1.1.1.2"></eq><apply id="A1.I3.i3.p1.1.m1.1.1.3.cmml" xref="A1.I3.i3.p1.1.m1.1.1.3"><apply id="A1.I3.i3.p1.1.m1.1.1.3.1.cmml" xref="A1.I3.i3.p1.1.m1.1.1.3.1"><csymbol cd="ambiguous" id="A1.I3.i3.p1.1.m1.1.1.3.1.1.cmml" xref="A1.I3.i3.p1.1.m1.1.1.3.1">subscript</csymbol><ci id="A1.I3.i3.p1.1.m1.1.1.3.1.2.cmml" xref="A1.I3.i3.p1.1.m1.1.1.3.1.2">∇</ci><apply id="A1.I3.i3.p1.1.m1.1.1.3.1.3.cmml" xref="A1.I3.i3.p1.1.m1.1.1.3.1.3"><csymbol cd="ambiguous" id="A1.I3.i3.p1.1.m1.1.1.3.1.3.1.cmml" xref="A1.I3.i3.p1.1.m1.1.1.3.1.3">superscript</csymbol><apply id="A1.I3.i3.p1.1.m1.1.1.3.1.3.2.cmml" xref="A1.I3.i3.p1.1.m1.1.1.3.1.3"><csymbol cd="ambiguous" id="A1.I3.i3.p1.1.m1.1.1.3.1.3.2.1.cmml" xref="A1.I3.i3.p1.1.m1.1.1.3.1.3">subscript</csymbol><ci id="A1.I3.i3.p1.1.m1.1.1.3.1.3.2.2.cmml" xref="A1.I3.i3.p1.1.m1.1.1.3.1.3.2.2">𝑊</ci><cn type="integer" id="A1.I3.i3.p1.1.m1.1.1.3.1.3.2.3.cmml" xref="A1.I3.i3.p1.1.m1.1.1.3.1.3.2.3">0</cn></apply><ci id="A1.I3.i3.p1.1.m1.1.1.3.1.3.3.cmml" xref="A1.I3.i3.p1.1.m1.1.1.3.1.3.3">𝑘</ci></apply></apply><ci id="A1.I3.i3.p1.1.m1.1.1.3.2.cmml" xref="A1.I3.i3.p1.1.m1.1.1.3.2">𝑙</ci></apply><apply id="A1.I3.i3.p1.1.m1.1.1.1.cmml" xref="A1.I3.i3.p1.1.m1.1.1.1"><times id="A1.I3.i3.p1.1.m1.1.1.1.2.cmml" xref="A1.I3.i3.p1.1.m1.1.1.1.2"></times><apply id="A1.I3.i3.p1.1.m1.1.1.1.3.cmml" xref="A1.I3.i3.p1.1.m1.1.1.1.3"><apply id="A1.I3.i3.p1.1.m1.1.1.1.3.1.cmml" xref="A1.I3.i3.p1.1.m1.1.1.1.3.1"><csymbol cd="ambiguous" id="A1.I3.i3.p1.1.m1.1.1.1.3.1.1.cmml" xref="A1.I3.i3.p1.1.m1.1.1.1.3.1">subscript</csymbol><ci id="A1.I3.i3.p1.1.m1.1.1.1.3.1.2.cmml" xref="A1.I3.i3.p1.1.m1.1.1.1.3.1.2">∇</ci><apply id="A1.I3.i3.p1.1.m1.1.1.1.3.1.3.cmml" xref="A1.I3.i3.p1.1.m1.1.1.1.3.1.3"><csymbol cd="ambiguous" id="A1.I3.i3.p1.1.m1.1.1.1.3.1.3.1.cmml" xref="A1.I3.i3.p1.1.m1.1.1.1.3.1.3">superscript</csymbol><ci id="A1.I3.i3.p1.1.m1.1.1.1.3.1.3.2.cmml" xref="A1.I3.i3.p1.1.m1.1.1.1.3.1.3.2">𝑍</ci><ci id="A1.I3.i3.p1.1.m1.1.1.1.3.1.3.3.cmml" xref="A1.I3.i3.p1.1.m1.1.1.1.3.1.3.3">𝑘</ci></apply></apply><ci id="A1.I3.i3.p1.1.m1.1.1.1.3.2.cmml" xref="A1.I3.i3.p1.1.m1.1.1.1.3.2">𝑙</ci></apply><apply id="A1.I3.i3.p1.1.m1.1.1.1.1.cmml" xref="A1.I3.i3.p1.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="A1.I3.i3.p1.1.m1.1.1.1.1.2.cmml" xref="A1.I3.i3.p1.1.m1.1.1.1.1">superscript</csymbol><apply id="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.cmml" xref="A1.I3.i3.p1.1.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.1.cmml" xref="A1.I3.i3.p1.1.m1.1.1.1.1.1.1">superscript</csymbol><apply id="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.2.cmml" xref="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.2"><ci id="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.2.1.cmml" xref="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.2.1">^</ci><ci id="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.2.2.cmml" xref="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.2.2">𝑋</ci></apply><ci id="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.3.cmml" xref="A1.I3.i3.p1.1.m1.1.1.1.1.1.1.1.3">𝑘</ci></apply><ci id="A1.I3.i3.p1.1.m1.1.1.1.1.3.cmml" xref="A1.I3.i3.p1.1.m1.1.1.1.1.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I3.i3.p1.1.m1.1c">\nabla_{W_{0}^{k}}l=\nabla_{Z^{k}}l\,\left(\hat{X}^{k}\right)^{T}</annotation></semantics></math></p>
</div>
</li>
</ul>
</div>
<div id="A1.SS3.p4" class="ltx_para">
<p id="A1.SS3.p4.1" class="ltx_p">Now we can already compute <math id="A1.SS3.p4.1.m1.1" class="ltx_Math" alttext="W_{b}^{k}=W_{0}^{k}-\nabla_{W_{0}^{k}}l" display="inline"><semantics id="A1.SS3.p4.1.m1.1a"><mrow id="A1.SS3.p4.1.m1.1.1" xref="A1.SS3.p4.1.m1.1.1.cmml"><msubsup id="A1.SS3.p4.1.m1.1.1.2" xref="A1.SS3.p4.1.m1.1.1.2.cmml"><mi id="A1.SS3.p4.1.m1.1.1.2.2.2" xref="A1.SS3.p4.1.m1.1.1.2.2.2.cmml">W</mi><mi id="A1.SS3.p4.1.m1.1.1.2.2.3" xref="A1.SS3.p4.1.m1.1.1.2.2.3.cmml">b</mi><mi id="A1.SS3.p4.1.m1.1.1.2.3" xref="A1.SS3.p4.1.m1.1.1.2.3.cmml">k</mi></msubsup><mo id="A1.SS3.p4.1.m1.1.1.1" xref="A1.SS3.p4.1.m1.1.1.1.cmml">=</mo><mrow id="A1.SS3.p4.1.m1.1.1.3" xref="A1.SS3.p4.1.m1.1.1.3.cmml"><msubsup id="A1.SS3.p4.1.m1.1.1.3.2" xref="A1.SS3.p4.1.m1.1.1.3.2.cmml"><mi id="A1.SS3.p4.1.m1.1.1.3.2.2.2" xref="A1.SS3.p4.1.m1.1.1.3.2.2.2.cmml">W</mi><mn id="A1.SS3.p4.1.m1.1.1.3.2.2.3" xref="A1.SS3.p4.1.m1.1.1.3.2.2.3.cmml">0</mn><mi id="A1.SS3.p4.1.m1.1.1.3.2.3" xref="A1.SS3.p4.1.m1.1.1.3.2.3.cmml">k</mi></msubsup><mo id="A1.SS3.p4.1.m1.1.1.3.1" xref="A1.SS3.p4.1.m1.1.1.3.1.cmml">−</mo><mrow id="A1.SS3.p4.1.m1.1.1.3.3" xref="A1.SS3.p4.1.m1.1.1.3.3.cmml"><msub id="A1.SS3.p4.1.m1.1.1.3.3.1" xref="A1.SS3.p4.1.m1.1.1.3.3.1.cmml"><mo rspace="0.167em" id="A1.SS3.p4.1.m1.1.1.3.3.1.2" xref="A1.SS3.p4.1.m1.1.1.3.3.1.2.cmml">∇</mo><msubsup id="A1.SS3.p4.1.m1.1.1.3.3.1.3" xref="A1.SS3.p4.1.m1.1.1.3.3.1.3.cmml"><mi id="A1.SS3.p4.1.m1.1.1.3.3.1.3.2.2" xref="A1.SS3.p4.1.m1.1.1.3.3.1.3.2.2.cmml">W</mi><mn id="A1.SS3.p4.1.m1.1.1.3.3.1.3.2.3" xref="A1.SS3.p4.1.m1.1.1.3.3.1.3.2.3.cmml">0</mn><mi id="A1.SS3.p4.1.m1.1.1.3.3.1.3.3" xref="A1.SS3.p4.1.m1.1.1.3.3.1.3.3.cmml">k</mi></msubsup></msub><mi id="A1.SS3.p4.1.m1.1.1.3.3.2" xref="A1.SS3.p4.1.m1.1.1.3.3.2.cmml">l</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.p4.1.m1.1b"><apply id="A1.SS3.p4.1.m1.1.1.cmml" xref="A1.SS3.p4.1.m1.1.1"><eq id="A1.SS3.p4.1.m1.1.1.1.cmml" xref="A1.SS3.p4.1.m1.1.1.1"></eq><apply id="A1.SS3.p4.1.m1.1.1.2.cmml" xref="A1.SS3.p4.1.m1.1.1.2"><csymbol cd="ambiguous" id="A1.SS3.p4.1.m1.1.1.2.1.cmml" xref="A1.SS3.p4.1.m1.1.1.2">superscript</csymbol><apply id="A1.SS3.p4.1.m1.1.1.2.2.cmml" xref="A1.SS3.p4.1.m1.1.1.2"><csymbol cd="ambiguous" id="A1.SS3.p4.1.m1.1.1.2.2.1.cmml" xref="A1.SS3.p4.1.m1.1.1.2">subscript</csymbol><ci id="A1.SS3.p4.1.m1.1.1.2.2.2.cmml" xref="A1.SS3.p4.1.m1.1.1.2.2.2">𝑊</ci><ci id="A1.SS3.p4.1.m1.1.1.2.2.3.cmml" xref="A1.SS3.p4.1.m1.1.1.2.2.3">𝑏</ci></apply><ci id="A1.SS3.p4.1.m1.1.1.2.3.cmml" xref="A1.SS3.p4.1.m1.1.1.2.3">𝑘</ci></apply><apply id="A1.SS3.p4.1.m1.1.1.3.cmml" xref="A1.SS3.p4.1.m1.1.1.3"><minus id="A1.SS3.p4.1.m1.1.1.3.1.cmml" xref="A1.SS3.p4.1.m1.1.1.3.1"></minus><apply id="A1.SS3.p4.1.m1.1.1.3.2.cmml" xref="A1.SS3.p4.1.m1.1.1.3.2"><csymbol cd="ambiguous" id="A1.SS3.p4.1.m1.1.1.3.2.1.cmml" xref="A1.SS3.p4.1.m1.1.1.3.2">superscript</csymbol><apply id="A1.SS3.p4.1.m1.1.1.3.2.2.cmml" xref="A1.SS3.p4.1.m1.1.1.3.2"><csymbol cd="ambiguous" id="A1.SS3.p4.1.m1.1.1.3.2.2.1.cmml" xref="A1.SS3.p4.1.m1.1.1.3.2">subscript</csymbol><ci id="A1.SS3.p4.1.m1.1.1.3.2.2.2.cmml" xref="A1.SS3.p4.1.m1.1.1.3.2.2.2">𝑊</ci><cn type="integer" id="A1.SS3.p4.1.m1.1.1.3.2.2.3.cmml" xref="A1.SS3.p4.1.m1.1.1.3.2.2.3">0</cn></apply><ci id="A1.SS3.p4.1.m1.1.1.3.2.3.cmml" xref="A1.SS3.p4.1.m1.1.1.3.2.3">𝑘</ci></apply><apply id="A1.SS3.p4.1.m1.1.1.3.3.cmml" xref="A1.SS3.p4.1.m1.1.1.3.3"><apply id="A1.SS3.p4.1.m1.1.1.3.3.1.cmml" xref="A1.SS3.p4.1.m1.1.1.3.3.1"><csymbol cd="ambiguous" id="A1.SS3.p4.1.m1.1.1.3.3.1.1.cmml" xref="A1.SS3.p4.1.m1.1.1.3.3.1">subscript</csymbol><ci id="A1.SS3.p4.1.m1.1.1.3.3.1.2.cmml" xref="A1.SS3.p4.1.m1.1.1.3.3.1.2">∇</ci><apply id="A1.SS3.p4.1.m1.1.1.3.3.1.3.cmml" xref="A1.SS3.p4.1.m1.1.1.3.3.1.3"><csymbol cd="ambiguous" id="A1.SS3.p4.1.m1.1.1.3.3.1.3.1.cmml" xref="A1.SS3.p4.1.m1.1.1.3.3.1.3">superscript</csymbol><apply id="A1.SS3.p4.1.m1.1.1.3.3.1.3.2.cmml" xref="A1.SS3.p4.1.m1.1.1.3.3.1.3"><csymbol cd="ambiguous" id="A1.SS3.p4.1.m1.1.1.3.3.1.3.2.1.cmml" xref="A1.SS3.p4.1.m1.1.1.3.3.1.3">subscript</csymbol><ci id="A1.SS3.p4.1.m1.1.1.3.3.1.3.2.2.cmml" xref="A1.SS3.p4.1.m1.1.1.3.3.1.3.2.2">𝑊</ci><cn type="integer" id="A1.SS3.p4.1.m1.1.1.3.3.1.3.2.3.cmml" xref="A1.SS3.p4.1.m1.1.1.3.3.1.3.2.3">0</cn></apply><ci id="A1.SS3.p4.1.m1.1.1.3.3.1.3.3.cmml" xref="A1.SS3.p4.1.m1.1.1.3.3.1.3.3">𝑘</ci></apply></apply><ci id="A1.SS3.p4.1.m1.1.1.3.3.2.cmml" xref="A1.SS3.p4.1.m1.1.1.3.3.2">𝑙</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p4.1.m1.1c">W_{b}^{k}=W_{0}^{k}-\nabla_{W_{0}^{k}}l</annotation></semantics></math>.
To compute the output tokens, we do another forward pass.</p>
</div>
<div id="A1.SS3.p5" class="ltx_para">
<p id="A1.SS3.p5.1" class="ltx_p">For <math id="A1.SS3.p5.1.m1.3" class="ltx_Math" alttext="k=1,\dots,K" display="inline"><semantics id="A1.SS3.p5.1.m1.3a"><mrow id="A1.SS3.p5.1.m1.3.4" xref="A1.SS3.p5.1.m1.3.4.cmml"><mi id="A1.SS3.p5.1.m1.3.4.2" xref="A1.SS3.p5.1.m1.3.4.2.cmml">k</mi><mo id="A1.SS3.p5.1.m1.3.4.1" xref="A1.SS3.p5.1.m1.3.4.1.cmml">=</mo><mrow id="A1.SS3.p5.1.m1.3.4.3.2" xref="A1.SS3.p5.1.m1.3.4.3.1.cmml"><mn id="A1.SS3.p5.1.m1.1.1" xref="A1.SS3.p5.1.m1.1.1.cmml">1</mn><mo id="A1.SS3.p5.1.m1.3.4.3.2.1" xref="A1.SS3.p5.1.m1.3.4.3.1.cmml">,</mo><mi mathvariant="normal" id="A1.SS3.p5.1.m1.2.2" xref="A1.SS3.p5.1.m1.2.2.cmml">…</mi><mo id="A1.SS3.p5.1.m1.3.4.3.2.2" xref="A1.SS3.p5.1.m1.3.4.3.1.cmml">,</mo><mi id="A1.SS3.p5.1.m1.3.3" xref="A1.SS3.p5.1.m1.3.3.cmml">K</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.p5.1.m1.3b"><apply id="A1.SS3.p5.1.m1.3.4.cmml" xref="A1.SS3.p5.1.m1.3.4"><eq id="A1.SS3.p5.1.m1.3.4.1.cmml" xref="A1.SS3.p5.1.m1.3.4.1"></eq><ci id="A1.SS3.p5.1.m1.3.4.2.cmml" xref="A1.SS3.p5.1.m1.3.4.2">𝑘</ci><list id="A1.SS3.p5.1.m1.3.4.3.1.cmml" xref="A1.SS3.p5.1.m1.3.4.3.2"><cn type="integer" id="A1.SS3.p5.1.m1.1.1.cmml" xref="A1.SS3.p5.1.m1.1.1">1</cn><ci id="A1.SS3.p5.1.m1.2.2.cmml" xref="A1.SS3.p5.1.m1.2.2">…</ci><ci id="A1.SS3.p5.1.m1.3.3.cmml" xref="A1.SS3.p5.1.m1.3.3">𝐾</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p5.1.m1.3c">k=1,\dots,K</annotation></semantics></math>:</p>
<ul id="A1.I4" class="ltx_itemize">
<li id="A1.I4.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I4.i1.p1" class="ltx_para">
<p id="A1.I4.i1.p1.1" class="ltx_p"><math id="A1.I4.i1.p1.1.m1.1" class="ltx_Math" alttext="\bar{Z}^{k}=W^{k}\bar{X}^{k}-\nabla_{Z^{k}}l\cdot\texttt{mask}\left(\left(\hat{X}^{k}\right)^{T}\bar{X}^{k}\right)" display="inline"><semantics id="A1.I4.i1.p1.1.m1.1a"><mrow id="A1.I4.i1.p1.1.m1.1.1" xref="A1.I4.i1.p1.1.m1.1.1.cmml"><msup id="A1.I4.i1.p1.1.m1.1.1.3" xref="A1.I4.i1.p1.1.m1.1.1.3.cmml"><mover accent="true" id="A1.I4.i1.p1.1.m1.1.1.3.2" xref="A1.I4.i1.p1.1.m1.1.1.3.2.cmml"><mi id="A1.I4.i1.p1.1.m1.1.1.3.2.2" xref="A1.I4.i1.p1.1.m1.1.1.3.2.2.cmml">Z</mi><mo id="A1.I4.i1.p1.1.m1.1.1.3.2.1" xref="A1.I4.i1.p1.1.m1.1.1.3.2.1.cmml">¯</mo></mover><mi id="A1.I4.i1.p1.1.m1.1.1.3.3" xref="A1.I4.i1.p1.1.m1.1.1.3.3.cmml">k</mi></msup><mo id="A1.I4.i1.p1.1.m1.1.1.2" xref="A1.I4.i1.p1.1.m1.1.1.2.cmml">=</mo><mrow id="A1.I4.i1.p1.1.m1.1.1.1" xref="A1.I4.i1.p1.1.m1.1.1.1.cmml"><mrow id="A1.I4.i1.p1.1.m1.1.1.1.3" xref="A1.I4.i1.p1.1.m1.1.1.1.3.cmml"><msup id="A1.I4.i1.p1.1.m1.1.1.1.3.2" xref="A1.I4.i1.p1.1.m1.1.1.1.3.2.cmml"><mi id="A1.I4.i1.p1.1.m1.1.1.1.3.2.2" xref="A1.I4.i1.p1.1.m1.1.1.1.3.2.2.cmml">W</mi><mi id="A1.I4.i1.p1.1.m1.1.1.1.3.2.3" xref="A1.I4.i1.p1.1.m1.1.1.1.3.2.3.cmml">k</mi></msup><mo lspace="0em" rspace="0em" id="A1.I4.i1.p1.1.m1.1.1.1.3.1" xref="A1.I4.i1.p1.1.m1.1.1.1.3.1.cmml">​</mo><msup id="A1.I4.i1.p1.1.m1.1.1.1.3.3" xref="A1.I4.i1.p1.1.m1.1.1.1.3.3.cmml"><mover accent="true" id="A1.I4.i1.p1.1.m1.1.1.1.3.3.2" xref="A1.I4.i1.p1.1.m1.1.1.1.3.3.2.cmml"><mi id="A1.I4.i1.p1.1.m1.1.1.1.3.3.2.2" xref="A1.I4.i1.p1.1.m1.1.1.1.3.3.2.2.cmml">X</mi><mo id="A1.I4.i1.p1.1.m1.1.1.1.3.3.2.1" xref="A1.I4.i1.p1.1.m1.1.1.1.3.3.2.1.cmml">¯</mo></mover><mi id="A1.I4.i1.p1.1.m1.1.1.1.3.3.3" xref="A1.I4.i1.p1.1.m1.1.1.1.3.3.3.cmml">k</mi></msup></mrow><mo id="A1.I4.i1.p1.1.m1.1.1.1.2" xref="A1.I4.i1.p1.1.m1.1.1.1.2.cmml">−</mo><mrow id="A1.I4.i1.p1.1.m1.1.1.1.1" xref="A1.I4.i1.p1.1.m1.1.1.1.1.cmml"><mrow id="A1.I4.i1.p1.1.m1.1.1.1.1.3" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.cmml"><msub id="A1.I4.i1.p1.1.m1.1.1.1.1.3.1" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.1.cmml"><mo rspace="0.167em" id="A1.I4.i1.p1.1.m1.1.1.1.1.3.1.2" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.1.2.cmml">∇</mo><msup id="A1.I4.i1.p1.1.m1.1.1.1.1.3.1.3" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.1.3.cmml"><mi id="A1.I4.i1.p1.1.m1.1.1.1.1.3.1.3.2" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.1.3.2.cmml">Z</mi><mi id="A1.I4.i1.p1.1.m1.1.1.1.1.3.1.3.3" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.1.3.3.cmml">k</mi></msup></msub><mrow id="A1.I4.i1.p1.1.m1.1.1.1.1.3.2" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.2.cmml"><mi id="A1.I4.i1.p1.1.m1.1.1.1.1.3.2.2" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.2.2.cmml">l</mi><mo lspace="0.222em" rspace="0.222em" id="A1.I4.i1.p1.1.m1.1.1.1.1.3.2.1" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.2.1.cmml">⋅</mo><mtext class="ltx_mathvariant_monospace" id="A1.I4.i1.p1.1.m1.1.1.1.1.3.2.3" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.2.3a.cmml">mask</mtext></mrow></mrow><mo lspace="0em" rspace="0em" id="A1.I4.i1.p1.1.m1.1.1.1.1.2" xref="A1.I4.i1.p1.1.m1.1.1.1.1.2.cmml">​</mo><mrow id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.cmml"><mo id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.2" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.cmml"><msup id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.cmml"><mrow id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.2" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msup id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mover accent="true" id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.2" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml">X</mi><mo id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.1" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml">^</mo></mover><mi id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.3" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">k</mi></msup><mo id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.3" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mi id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.3" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.2" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.2.cmml">​</mo><msup id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3.cmml"><mover accent="true" id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3.2" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3.2.cmml"><mi id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3.2.2" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3.2.2.cmml">X</mi><mo id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3.2.1" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3.2.1.cmml">¯</mo></mover><mi id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3.3" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3.3.cmml">k</mi></msup></mrow><mo id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.3" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.I4.i1.p1.1.m1.1b"><apply id="A1.I4.i1.p1.1.m1.1.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1"><eq id="A1.I4.i1.p1.1.m1.1.1.2.cmml" xref="A1.I4.i1.p1.1.m1.1.1.2"></eq><apply id="A1.I4.i1.p1.1.m1.1.1.3.cmml" xref="A1.I4.i1.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="A1.I4.i1.p1.1.m1.1.1.3.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1.3">superscript</csymbol><apply id="A1.I4.i1.p1.1.m1.1.1.3.2.cmml" xref="A1.I4.i1.p1.1.m1.1.1.3.2"><ci id="A1.I4.i1.p1.1.m1.1.1.3.2.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1.3.2.1">¯</ci><ci id="A1.I4.i1.p1.1.m1.1.1.3.2.2.cmml" xref="A1.I4.i1.p1.1.m1.1.1.3.2.2">𝑍</ci></apply><ci id="A1.I4.i1.p1.1.m1.1.1.3.3.cmml" xref="A1.I4.i1.p1.1.m1.1.1.3.3">𝑘</ci></apply><apply id="A1.I4.i1.p1.1.m1.1.1.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1"><minus id="A1.I4.i1.p1.1.m1.1.1.1.2.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.2"></minus><apply id="A1.I4.i1.p1.1.m1.1.1.1.3.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.3"><times id="A1.I4.i1.p1.1.m1.1.1.1.3.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.3.1"></times><apply id="A1.I4.i1.p1.1.m1.1.1.1.3.2.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.3.2"><csymbol cd="ambiguous" id="A1.I4.i1.p1.1.m1.1.1.1.3.2.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.3.2">superscript</csymbol><ci id="A1.I4.i1.p1.1.m1.1.1.1.3.2.2.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.3.2.2">𝑊</ci><ci id="A1.I4.i1.p1.1.m1.1.1.1.3.2.3.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.3.2.3">𝑘</ci></apply><apply id="A1.I4.i1.p1.1.m1.1.1.1.3.3.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.3.3"><csymbol cd="ambiguous" id="A1.I4.i1.p1.1.m1.1.1.1.3.3.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.3.3">superscript</csymbol><apply id="A1.I4.i1.p1.1.m1.1.1.1.3.3.2.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.3.3.2"><ci id="A1.I4.i1.p1.1.m1.1.1.1.3.3.2.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.3.3.2.1">¯</ci><ci id="A1.I4.i1.p1.1.m1.1.1.1.3.3.2.2.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.3.3.2.2">𝑋</ci></apply><ci id="A1.I4.i1.p1.1.m1.1.1.1.3.3.3.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.3.3.3">𝑘</ci></apply></apply><apply id="A1.I4.i1.p1.1.m1.1.1.1.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1"><times id="A1.I4.i1.p1.1.m1.1.1.1.1.2.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.2"></times><apply id="A1.I4.i1.p1.1.m1.1.1.1.1.3.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3"><apply id="A1.I4.i1.p1.1.m1.1.1.1.1.3.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.1"><csymbol cd="ambiguous" id="A1.I4.i1.p1.1.m1.1.1.1.1.3.1.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.1">subscript</csymbol><ci id="A1.I4.i1.p1.1.m1.1.1.1.1.3.1.2.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.1.2">∇</ci><apply id="A1.I4.i1.p1.1.m1.1.1.1.1.3.1.3.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.1.3"><csymbol cd="ambiguous" id="A1.I4.i1.p1.1.m1.1.1.1.1.3.1.3.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.1.3">superscript</csymbol><ci id="A1.I4.i1.p1.1.m1.1.1.1.1.3.1.3.2.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.1.3.2">𝑍</ci><ci id="A1.I4.i1.p1.1.m1.1.1.1.1.3.1.3.3.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.1.3.3">𝑘</ci></apply></apply><apply id="A1.I4.i1.p1.1.m1.1.1.1.1.3.2.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.2"><ci id="A1.I4.i1.p1.1.m1.1.1.1.1.3.2.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.2.1">⋅</ci><ci id="A1.I4.i1.p1.1.m1.1.1.1.1.3.2.2.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.2.2">𝑙</ci><ci id="A1.I4.i1.p1.1.m1.1.1.1.1.3.2.3a.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.2.3"><mtext class="ltx_mathvariant_monospace" id="A1.I4.i1.p1.1.m1.1.1.1.1.3.2.3.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.3.2.3">mask</mtext></ci></apply></apply><apply id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1"><times id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.2.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.2"></times><apply id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.2.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1">superscript</csymbol><apply id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1">superscript</csymbol><apply id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2"><ci id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.1">^</ci><ci id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.2">𝑋</ci></apply><ci id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.3">𝑘</ci></apply><ci id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.3.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.1.3">𝑇</ci></apply><apply id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3">superscript</csymbol><apply id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3.2.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3.2"><ci id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3.2.1.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3.2.1">¯</ci><ci id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3.2.2.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3.2.2">𝑋</ci></apply><ci id="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3.3.cmml" xref="A1.I4.i1.p1.1.m1.1.1.1.1.1.1.1.3.3">𝑘</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I4.i1.p1.1.m1.1c">\bar{Z}^{k}=W^{k}\bar{X}^{k}-\nabla_{Z^{k}}l\cdot\texttt{mask}\left(\left(\hat{X}^{k}\right)^{T}\bar{X}^{k}\right)</annotation></semantics></math></p>
</div>
</li>
<li id="A1.I4.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A1.I4.i2.p1" class="ltx_para">
<p id="A1.I4.i2.p1.1" class="ltx_p"><math id="A1.I4.i2.p1.1.m1.1" class="ltx_Math" alttext="\bar{X}^{k+1}=\sigma(\bar{Z}^{k})" display="inline"><semantics id="A1.I4.i2.p1.1.m1.1a"><mrow id="A1.I4.i2.p1.1.m1.1.1" xref="A1.I4.i2.p1.1.m1.1.1.cmml"><msup id="A1.I4.i2.p1.1.m1.1.1.3" xref="A1.I4.i2.p1.1.m1.1.1.3.cmml"><mover accent="true" id="A1.I4.i2.p1.1.m1.1.1.3.2" xref="A1.I4.i2.p1.1.m1.1.1.3.2.cmml"><mi id="A1.I4.i2.p1.1.m1.1.1.3.2.2" xref="A1.I4.i2.p1.1.m1.1.1.3.2.2.cmml">X</mi><mo id="A1.I4.i2.p1.1.m1.1.1.3.2.1" xref="A1.I4.i2.p1.1.m1.1.1.3.2.1.cmml">¯</mo></mover><mrow id="A1.I4.i2.p1.1.m1.1.1.3.3" xref="A1.I4.i2.p1.1.m1.1.1.3.3.cmml"><mi id="A1.I4.i2.p1.1.m1.1.1.3.3.2" xref="A1.I4.i2.p1.1.m1.1.1.3.3.2.cmml">k</mi><mo id="A1.I4.i2.p1.1.m1.1.1.3.3.1" xref="A1.I4.i2.p1.1.m1.1.1.3.3.1.cmml">+</mo><mn id="A1.I4.i2.p1.1.m1.1.1.3.3.3" xref="A1.I4.i2.p1.1.m1.1.1.3.3.3.cmml">1</mn></mrow></msup><mo id="A1.I4.i2.p1.1.m1.1.1.2" xref="A1.I4.i2.p1.1.m1.1.1.2.cmml">=</mo><mrow id="A1.I4.i2.p1.1.m1.1.1.1" xref="A1.I4.i2.p1.1.m1.1.1.1.cmml"><mi id="A1.I4.i2.p1.1.m1.1.1.1.3" xref="A1.I4.i2.p1.1.m1.1.1.1.3.cmml">σ</mi><mo lspace="0em" rspace="0em" id="A1.I4.i2.p1.1.m1.1.1.1.2" xref="A1.I4.i2.p1.1.m1.1.1.1.2.cmml">​</mo><mrow id="A1.I4.i2.p1.1.m1.1.1.1.1.1" xref="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.I4.i2.p1.1.m1.1.1.1.1.1.2" xref="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.cmml">(</mo><msup id="A1.I4.i2.p1.1.m1.1.1.1.1.1.1" xref="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.cmml"><mover accent="true" id="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.2" xref="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.2.cmml"><mi id="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.2.2" xref="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.2.2.cmml">Z</mi><mo id="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.2.1" xref="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.2.1.cmml">¯</mo></mover><mi id="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.3" xref="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.3.cmml">k</mi></msup><mo stretchy="false" id="A1.I4.i2.p1.1.m1.1.1.1.1.1.3" xref="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.I4.i2.p1.1.m1.1b"><apply id="A1.I4.i2.p1.1.m1.1.1.cmml" xref="A1.I4.i2.p1.1.m1.1.1"><eq id="A1.I4.i2.p1.1.m1.1.1.2.cmml" xref="A1.I4.i2.p1.1.m1.1.1.2"></eq><apply id="A1.I4.i2.p1.1.m1.1.1.3.cmml" xref="A1.I4.i2.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="A1.I4.i2.p1.1.m1.1.1.3.1.cmml" xref="A1.I4.i2.p1.1.m1.1.1.3">superscript</csymbol><apply id="A1.I4.i2.p1.1.m1.1.1.3.2.cmml" xref="A1.I4.i2.p1.1.m1.1.1.3.2"><ci id="A1.I4.i2.p1.1.m1.1.1.3.2.1.cmml" xref="A1.I4.i2.p1.1.m1.1.1.3.2.1">¯</ci><ci id="A1.I4.i2.p1.1.m1.1.1.3.2.2.cmml" xref="A1.I4.i2.p1.1.m1.1.1.3.2.2">𝑋</ci></apply><apply id="A1.I4.i2.p1.1.m1.1.1.3.3.cmml" xref="A1.I4.i2.p1.1.m1.1.1.3.3"><plus id="A1.I4.i2.p1.1.m1.1.1.3.3.1.cmml" xref="A1.I4.i2.p1.1.m1.1.1.3.3.1"></plus><ci id="A1.I4.i2.p1.1.m1.1.1.3.3.2.cmml" xref="A1.I4.i2.p1.1.m1.1.1.3.3.2">𝑘</ci><cn type="integer" id="A1.I4.i2.p1.1.m1.1.1.3.3.3.cmml" xref="A1.I4.i2.p1.1.m1.1.1.3.3.3">1</cn></apply></apply><apply id="A1.I4.i2.p1.1.m1.1.1.1.cmml" xref="A1.I4.i2.p1.1.m1.1.1.1"><times id="A1.I4.i2.p1.1.m1.1.1.1.2.cmml" xref="A1.I4.i2.p1.1.m1.1.1.1.2"></times><ci id="A1.I4.i2.p1.1.m1.1.1.1.3.cmml" xref="A1.I4.i2.p1.1.m1.1.1.1.3">𝜎</ci><apply id="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.cmml" xref="A1.I4.i2.p1.1.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.1.cmml" xref="A1.I4.i2.p1.1.m1.1.1.1.1.1">superscript</csymbol><apply id="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.2.cmml" xref="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.2"><ci id="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.2.1.cmml" xref="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.2.1">¯</ci><ci id="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.2.2.cmml" xref="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.2.2">𝑍</ci></apply><ci id="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.3.cmml" xref="A1.I4.i2.p1.1.m1.1.1.1.1.1.1.3">𝑘</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.I4.i2.p1.1.m1.1c">\bar{X}^{k+1}=\sigma(\bar{Z}^{k})</annotation></semantics></math></p>
</div>
</li>
</ul>
<p id="A1.SS3.p5.2" class="ltx_p">By the end of the forward pass, we have computed <math id="A1.SS3.p5.2.m1.1" class="ltx_Math" alttext="\bar{X}^{K+1}" display="inline"><semantics id="A1.SS3.p5.2.m1.1a"><msup id="A1.SS3.p5.2.m1.1.1" xref="A1.SS3.p5.2.m1.1.1.cmml"><mover accent="true" id="A1.SS3.p5.2.m1.1.1.2" xref="A1.SS3.p5.2.m1.1.1.2.cmml"><mi id="A1.SS3.p5.2.m1.1.1.2.2" xref="A1.SS3.p5.2.m1.1.1.2.2.cmml">X</mi><mo id="A1.SS3.p5.2.m1.1.1.2.1" xref="A1.SS3.p5.2.m1.1.1.2.1.cmml">¯</mo></mover><mrow id="A1.SS3.p5.2.m1.1.1.3" xref="A1.SS3.p5.2.m1.1.1.3.cmml"><mi id="A1.SS3.p5.2.m1.1.1.3.2" xref="A1.SS3.p5.2.m1.1.1.3.2.cmml">K</mi><mo id="A1.SS3.p5.2.m1.1.1.3.1" xref="A1.SS3.p5.2.m1.1.1.3.1.cmml">+</mo><mn id="A1.SS3.p5.2.m1.1.1.3.3" xref="A1.SS3.p5.2.m1.1.1.3.3.cmml">1</mn></mrow></msup><annotation-xml encoding="MathML-Content" id="A1.SS3.p5.2.m1.1b"><apply id="A1.SS3.p5.2.m1.1.1.cmml" xref="A1.SS3.p5.2.m1.1.1"><csymbol cd="ambiguous" id="A1.SS3.p5.2.m1.1.1.1.cmml" xref="A1.SS3.p5.2.m1.1.1">superscript</csymbol><apply id="A1.SS3.p5.2.m1.1.1.2.cmml" xref="A1.SS3.p5.2.m1.1.1.2"><ci id="A1.SS3.p5.2.m1.1.1.2.1.cmml" xref="A1.SS3.p5.2.m1.1.1.2.1">¯</ci><ci id="A1.SS3.p5.2.m1.1.1.2.2.cmml" xref="A1.SS3.p5.2.m1.1.1.2.2">𝑋</ci></apply><apply id="A1.SS3.p5.2.m1.1.1.3.cmml" xref="A1.SS3.p5.2.m1.1.1.3"><plus id="A1.SS3.p5.2.m1.1.1.3.1.cmml" xref="A1.SS3.p5.2.m1.1.1.3.1"></plus><ci id="A1.SS3.p5.2.m1.1.1.3.2.cmml" xref="A1.SS3.p5.2.m1.1.1.3.2">𝐾</ci><cn type="integer" id="A1.SS3.p5.2.m1.1.1.3.3.cmml" xref="A1.SS3.p5.2.m1.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p5.2.m1.1c">\bar{X}^{K+1}</annotation></semantics></math>.</p>
</div>
<div id="A1.SS3.p6" class="ltx_para">
<p id="A1.SS3.p6.1" class="ltx_p">While this forward pass is non-standard, it only contains <span id="A1.SS3.p6.1.1" class="ltx_text ltx_font_typewriter">matmul</span>s, <span id="A1.SS3.p6.1.2" class="ltx_text ltx_font_typewriter">sum</span>s, <math id="A1.SS3.p6.1.m1.1" class="ltx_Math" alttext="\sigma" display="inline"><semantics id="A1.SS3.p6.1.m1.1a"><mi id="A1.SS3.p6.1.m1.1.1" xref="A1.SS3.p6.1.m1.1.1.cmml">σ</mi><annotation-xml encoding="MathML-Content" id="A1.SS3.p6.1.m1.1b"><ci id="A1.SS3.p6.1.m1.1.1.cmml" xref="A1.SS3.p6.1.m1.1.1">𝜎</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p6.1.m1.1c">\sigma</annotation></semantics></math>, and <span id="A1.SS3.p6.1.3" class="ltx_text ltx_font_typewriter">mask</span>, therefore is efficient like the standard forward pass.</p>
</div>
</section>
<section id="A1.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.4 </span>Derivation</h3>

<div id="A1.SS4.p1" class="ltx_para">
<p id="A1.SS4.p1.4" class="ltx_p">To derive the dual form, we show that:</p>
<table id="A1.Ex10" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.Ex10.m1.1" class="ltx_Math" alttext="\bar{Z}^{k}=W^{k}\bar{X}^{k}-\nabla_{Z^{k}}l\cdot\texttt{mask}\left(\left(\hat{X}^{k}\right)^{T}\bar{X}^{k}\right)" display="block"><semantics id="A1.Ex10.m1.1a"><mrow id="A1.Ex10.m1.1.1" xref="A1.Ex10.m1.1.1.cmml"><msup id="A1.Ex10.m1.1.1.3" xref="A1.Ex10.m1.1.1.3.cmml"><mover accent="true" id="A1.Ex10.m1.1.1.3.2" xref="A1.Ex10.m1.1.1.3.2.cmml"><mi id="A1.Ex10.m1.1.1.3.2.2" xref="A1.Ex10.m1.1.1.3.2.2.cmml">Z</mi><mo id="A1.Ex10.m1.1.1.3.2.1" xref="A1.Ex10.m1.1.1.3.2.1.cmml">¯</mo></mover><mi id="A1.Ex10.m1.1.1.3.3" xref="A1.Ex10.m1.1.1.3.3.cmml">k</mi></msup><mo id="A1.Ex10.m1.1.1.2" xref="A1.Ex10.m1.1.1.2.cmml">=</mo><mrow id="A1.Ex10.m1.1.1.1" xref="A1.Ex10.m1.1.1.1.cmml"><mrow id="A1.Ex10.m1.1.1.1.3" xref="A1.Ex10.m1.1.1.1.3.cmml"><msup id="A1.Ex10.m1.1.1.1.3.2" xref="A1.Ex10.m1.1.1.1.3.2.cmml"><mi id="A1.Ex10.m1.1.1.1.3.2.2" xref="A1.Ex10.m1.1.1.1.3.2.2.cmml">W</mi><mi id="A1.Ex10.m1.1.1.1.3.2.3" xref="A1.Ex10.m1.1.1.1.3.2.3.cmml">k</mi></msup><mo lspace="0em" rspace="0em" id="A1.Ex10.m1.1.1.1.3.1" xref="A1.Ex10.m1.1.1.1.3.1.cmml">​</mo><msup id="A1.Ex10.m1.1.1.1.3.3" xref="A1.Ex10.m1.1.1.1.3.3.cmml"><mover accent="true" id="A1.Ex10.m1.1.1.1.3.3.2" xref="A1.Ex10.m1.1.1.1.3.3.2.cmml"><mi id="A1.Ex10.m1.1.1.1.3.3.2.2" xref="A1.Ex10.m1.1.1.1.3.3.2.2.cmml">X</mi><mo id="A1.Ex10.m1.1.1.1.3.3.2.1" xref="A1.Ex10.m1.1.1.1.3.3.2.1.cmml">¯</mo></mover><mi id="A1.Ex10.m1.1.1.1.3.3.3" xref="A1.Ex10.m1.1.1.1.3.3.3.cmml">k</mi></msup></mrow><mo id="A1.Ex10.m1.1.1.1.2" xref="A1.Ex10.m1.1.1.1.2.cmml">−</mo><mrow id="A1.Ex10.m1.1.1.1.1" xref="A1.Ex10.m1.1.1.1.1.cmml"><mrow id="A1.Ex10.m1.1.1.1.1.3" xref="A1.Ex10.m1.1.1.1.1.3.cmml"><msub id="A1.Ex10.m1.1.1.1.1.3.1" xref="A1.Ex10.m1.1.1.1.1.3.1.cmml"><mo rspace="0.167em" id="A1.Ex10.m1.1.1.1.1.3.1.2" xref="A1.Ex10.m1.1.1.1.1.3.1.2.cmml">∇</mo><msup id="A1.Ex10.m1.1.1.1.1.3.1.3" xref="A1.Ex10.m1.1.1.1.1.3.1.3.cmml"><mi id="A1.Ex10.m1.1.1.1.1.3.1.3.2" xref="A1.Ex10.m1.1.1.1.1.3.1.3.2.cmml">Z</mi><mi id="A1.Ex10.m1.1.1.1.1.3.1.3.3" xref="A1.Ex10.m1.1.1.1.1.3.1.3.3.cmml">k</mi></msup></msub><mrow id="A1.Ex10.m1.1.1.1.1.3.2" xref="A1.Ex10.m1.1.1.1.1.3.2.cmml"><mi id="A1.Ex10.m1.1.1.1.1.3.2.2" xref="A1.Ex10.m1.1.1.1.1.3.2.2.cmml">l</mi><mo lspace="0.222em" rspace="0.222em" id="A1.Ex10.m1.1.1.1.1.3.2.1" xref="A1.Ex10.m1.1.1.1.1.3.2.1.cmml">⋅</mo><mtext class="ltx_mathvariant_monospace" id="A1.Ex10.m1.1.1.1.1.3.2.3" xref="A1.Ex10.m1.1.1.1.1.3.2.3a.cmml">mask</mtext></mrow></mrow><mo lspace="0em" rspace="0em" id="A1.Ex10.m1.1.1.1.1.2" xref="A1.Ex10.m1.1.1.1.1.2.cmml">​</mo><mrow id="A1.Ex10.m1.1.1.1.1.1.1" xref="A1.Ex10.m1.1.1.1.1.1.1.1.cmml"><mo id="A1.Ex10.m1.1.1.1.1.1.1.2" xref="A1.Ex10.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.Ex10.m1.1.1.1.1.1.1.1" xref="A1.Ex10.m1.1.1.1.1.1.1.1.cmml"><msup id="A1.Ex10.m1.1.1.1.1.1.1.1.1" xref="A1.Ex10.m1.1.1.1.1.1.1.1.1.cmml"><mrow id="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1" xref="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo id="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.2" xref="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msup id="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1" xref="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mover accent="true" id="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.2" xref="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.2.2" xref="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml">X</mi><mo id="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.2.1" xref="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml">^</mo></mover><mi id="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.3" xref="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">k</mi></msup><mo id="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.3" xref="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow><mi id="A1.Ex10.m1.1.1.1.1.1.1.1.1.3" xref="A1.Ex10.m1.1.1.1.1.1.1.1.1.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="A1.Ex10.m1.1.1.1.1.1.1.1.2" xref="A1.Ex10.m1.1.1.1.1.1.1.1.2.cmml">​</mo><msup id="A1.Ex10.m1.1.1.1.1.1.1.1.3" xref="A1.Ex10.m1.1.1.1.1.1.1.1.3.cmml"><mover accent="true" id="A1.Ex10.m1.1.1.1.1.1.1.1.3.2" xref="A1.Ex10.m1.1.1.1.1.1.1.1.3.2.cmml"><mi id="A1.Ex10.m1.1.1.1.1.1.1.1.3.2.2" xref="A1.Ex10.m1.1.1.1.1.1.1.1.3.2.2.cmml">X</mi><mo id="A1.Ex10.m1.1.1.1.1.1.1.1.3.2.1" xref="A1.Ex10.m1.1.1.1.1.1.1.1.3.2.1.cmml">¯</mo></mover><mi id="A1.Ex10.m1.1.1.1.1.1.1.1.3.3" xref="A1.Ex10.m1.1.1.1.1.1.1.1.3.3.cmml">k</mi></msup></mrow><mo id="A1.Ex10.m1.1.1.1.1.1.1.3" xref="A1.Ex10.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex10.m1.1b"><apply id="A1.Ex10.m1.1.1.cmml" xref="A1.Ex10.m1.1.1"><eq id="A1.Ex10.m1.1.1.2.cmml" xref="A1.Ex10.m1.1.1.2"></eq><apply id="A1.Ex10.m1.1.1.3.cmml" xref="A1.Ex10.m1.1.1.3"><csymbol cd="ambiguous" id="A1.Ex10.m1.1.1.3.1.cmml" xref="A1.Ex10.m1.1.1.3">superscript</csymbol><apply id="A1.Ex10.m1.1.1.3.2.cmml" xref="A1.Ex10.m1.1.1.3.2"><ci id="A1.Ex10.m1.1.1.3.2.1.cmml" xref="A1.Ex10.m1.1.1.3.2.1">¯</ci><ci id="A1.Ex10.m1.1.1.3.2.2.cmml" xref="A1.Ex10.m1.1.1.3.2.2">𝑍</ci></apply><ci id="A1.Ex10.m1.1.1.3.3.cmml" xref="A1.Ex10.m1.1.1.3.3">𝑘</ci></apply><apply id="A1.Ex10.m1.1.1.1.cmml" xref="A1.Ex10.m1.1.1.1"><minus id="A1.Ex10.m1.1.1.1.2.cmml" xref="A1.Ex10.m1.1.1.1.2"></minus><apply id="A1.Ex10.m1.1.1.1.3.cmml" xref="A1.Ex10.m1.1.1.1.3"><times id="A1.Ex10.m1.1.1.1.3.1.cmml" xref="A1.Ex10.m1.1.1.1.3.1"></times><apply id="A1.Ex10.m1.1.1.1.3.2.cmml" xref="A1.Ex10.m1.1.1.1.3.2"><csymbol cd="ambiguous" id="A1.Ex10.m1.1.1.1.3.2.1.cmml" xref="A1.Ex10.m1.1.1.1.3.2">superscript</csymbol><ci id="A1.Ex10.m1.1.1.1.3.2.2.cmml" xref="A1.Ex10.m1.1.1.1.3.2.2">𝑊</ci><ci id="A1.Ex10.m1.1.1.1.3.2.3.cmml" xref="A1.Ex10.m1.1.1.1.3.2.3">𝑘</ci></apply><apply id="A1.Ex10.m1.1.1.1.3.3.cmml" xref="A1.Ex10.m1.1.1.1.3.3"><csymbol cd="ambiguous" id="A1.Ex10.m1.1.1.1.3.3.1.cmml" xref="A1.Ex10.m1.1.1.1.3.3">superscript</csymbol><apply id="A1.Ex10.m1.1.1.1.3.3.2.cmml" xref="A1.Ex10.m1.1.1.1.3.3.2"><ci id="A1.Ex10.m1.1.1.1.3.3.2.1.cmml" xref="A1.Ex10.m1.1.1.1.3.3.2.1">¯</ci><ci id="A1.Ex10.m1.1.1.1.3.3.2.2.cmml" xref="A1.Ex10.m1.1.1.1.3.3.2.2">𝑋</ci></apply><ci id="A1.Ex10.m1.1.1.1.3.3.3.cmml" xref="A1.Ex10.m1.1.1.1.3.3.3">𝑘</ci></apply></apply><apply id="A1.Ex10.m1.1.1.1.1.cmml" xref="A1.Ex10.m1.1.1.1.1"><times id="A1.Ex10.m1.1.1.1.1.2.cmml" xref="A1.Ex10.m1.1.1.1.1.2"></times><apply id="A1.Ex10.m1.1.1.1.1.3.cmml" xref="A1.Ex10.m1.1.1.1.1.3"><apply id="A1.Ex10.m1.1.1.1.1.3.1.cmml" xref="A1.Ex10.m1.1.1.1.1.3.1"><csymbol cd="ambiguous" id="A1.Ex10.m1.1.1.1.1.3.1.1.cmml" xref="A1.Ex10.m1.1.1.1.1.3.1">subscript</csymbol><ci id="A1.Ex10.m1.1.1.1.1.3.1.2.cmml" xref="A1.Ex10.m1.1.1.1.1.3.1.2">∇</ci><apply id="A1.Ex10.m1.1.1.1.1.3.1.3.cmml" xref="A1.Ex10.m1.1.1.1.1.3.1.3"><csymbol cd="ambiguous" id="A1.Ex10.m1.1.1.1.1.3.1.3.1.cmml" xref="A1.Ex10.m1.1.1.1.1.3.1.3">superscript</csymbol><ci id="A1.Ex10.m1.1.1.1.1.3.1.3.2.cmml" xref="A1.Ex10.m1.1.1.1.1.3.1.3.2">𝑍</ci><ci id="A1.Ex10.m1.1.1.1.1.3.1.3.3.cmml" xref="A1.Ex10.m1.1.1.1.1.3.1.3.3">𝑘</ci></apply></apply><apply id="A1.Ex10.m1.1.1.1.1.3.2.cmml" xref="A1.Ex10.m1.1.1.1.1.3.2"><ci id="A1.Ex10.m1.1.1.1.1.3.2.1.cmml" xref="A1.Ex10.m1.1.1.1.1.3.2.1">⋅</ci><ci id="A1.Ex10.m1.1.1.1.1.3.2.2.cmml" xref="A1.Ex10.m1.1.1.1.1.3.2.2">𝑙</ci><ci id="A1.Ex10.m1.1.1.1.1.3.2.3a.cmml" xref="A1.Ex10.m1.1.1.1.1.3.2.3"><mtext class="ltx_mathvariant_monospace" id="A1.Ex10.m1.1.1.1.1.3.2.3.cmml" xref="A1.Ex10.m1.1.1.1.1.3.2.3">mask</mtext></ci></apply></apply><apply id="A1.Ex10.m1.1.1.1.1.1.1.1.cmml" xref="A1.Ex10.m1.1.1.1.1.1.1"><times id="A1.Ex10.m1.1.1.1.1.1.1.1.2.cmml" xref="A1.Ex10.m1.1.1.1.1.1.1.1.2"></times><apply id="A1.Ex10.m1.1.1.1.1.1.1.1.1.cmml" xref="A1.Ex10.m1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.Ex10.m1.1.1.1.1.1.1.1.1.2.cmml" xref="A1.Ex10.m1.1.1.1.1.1.1.1.1">superscript</csymbol><apply id="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1">superscript</csymbol><apply id="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.2"><ci id="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.2.1">^</ci><ci id="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.2.2">𝑋</ci></apply><ci id="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A1.Ex10.m1.1.1.1.1.1.1.1.1.1.1.1.3">𝑘</ci></apply><ci id="A1.Ex10.m1.1.1.1.1.1.1.1.1.3.cmml" xref="A1.Ex10.m1.1.1.1.1.1.1.1.1.3">𝑇</ci></apply><apply id="A1.Ex10.m1.1.1.1.1.1.1.1.3.cmml" xref="A1.Ex10.m1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A1.Ex10.m1.1.1.1.1.1.1.1.3.1.cmml" xref="A1.Ex10.m1.1.1.1.1.1.1.1.3">superscript</csymbol><apply id="A1.Ex10.m1.1.1.1.1.1.1.1.3.2.cmml" xref="A1.Ex10.m1.1.1.1.1.1.1.1.3.2"><ci id="A1.Ex10.m1.1.1.1.1.1.1.1.3.2.1.cmml" xref="A1.Ex10.m1.1.1.1.1.1.1.1.3.2.1">¯</ci><ci id="A1.Ex10.m1.1.1.1.1.1.1.1.3.2.2.cmml" xref="A1.Ex10.m1.1.1.1.1.1.1.1.3.2.2">𝑋</ci></apply><ci id="A1.Ex10.m1.1.1.1.1.1.1.1.3.3.cmml" xref="A1.Ex10.m1.1.1.1.1.1.1.1.3.3">𝑘</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex10.m1.1c">\bar{Z}^{k}=W^{k}\bar{X}^{k}-\nabla_{Z^{k}}l\cdot\texttt{mask}\left(\left(\hat{X}^{k}\right)^{T}\bar{X}^{k}\right)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="A1.SS4.p1.3" class="ltx_p">is the same as what would be computed in the primal form.
Specifically, we show that each column <math id="A1.SS4.p1.1.m1.1" class="ltx_Math" alttext="\bar{z}_{t}^{k}" display="inline"><semantics id="A1.SS4.p1.1.m1.1a"><msubsup id="A1.SS4.p1.1.m1.1.1" xref="A1.SS4.p1.1.m1.1.1.cmml"><mover accent="true" id="A1.SS4.p1.1.m1.1.1.2.2" xref="A1.SS4.p1.1.m1.1.1.2.2.cmml"><mi id="A1.SS4.p1.1.m1.1.1.2.2.2" xref="A1.SS4.p1.1.m1.1.1.2.2.2.cmml">z</mi><mo id="A1.SS4.p1.1.m1.1.1.2.2.1" xref="A1.SS4.p1.1.m1.1.1.2.2.1.cmml">¯</mo></mover><mi id="A1.SS4.p1.1.m1.1.1.2.3" xref="A1.SS4.p1.1.m1.1.1.2.3.cmml">t</mi><mi id="A1.SS4.p1.1.m1.1.1.3" xref="A1.SS4.p1.1.m1.1.1.3.cmml">k</mi></msubsup><annotation-xml encoding="MathML-Content" id="A1.SS4.p1.1.m1.1b"><apply id="A1.SS4.p1.1.m1.1.1.cmml" xref="A1.SS4.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A1.SS4.p1.1.m1.1.1.1.cmml" xref="A1.SS4.p1.1.m1.1.1">superscript</csymbol><apply id="A1.SS4.p1.1.m1.1.1.2.cmml" xref="A1.SS4.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A1.SS4.p1.1.m1.1.1.2.1.cmml" xref="A1.SS4.p1.1.m1.1.1">subscript</csymbol><apply id="A1.SS4.p1.1.m1.1.1.2.2.cmml" xref="A1.SS4.p1.1.m1.1.1.2.2"><ci id="A1.SS4.p1.1.m1.1.1.2.2.1.cmml" xref="A1.SS4.p1.1.m1.1.1.2.2.1">¯</ci><ci id="A1.SS4.p1.1.m1.1.1.2.2.2.cmml" xref="A1.SS4.p1.1.m1.1.1.2.2.2">𝑧</ci></apply><ci id="A1.SS4.p1.1.m1.1.1.2.3.cmml" xref="A1.SS4.p1.1.m1.1.1.2.3">𝑡</ci></apply><ci id="A1.SS4.p1.1.m1.1.1.3.cmml" xref="A1.SS4.p1.1.m1.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p1.1.m1.1c">\bar{z}_{t}^{k}</annotation></semantics></math> of <math id="A1.SS4.p1.2.m2.1" class="ltx_Math" alttext="\bar{Z}^{k}" display="inline"><semantics id="A1.SS4.p1.2.m2.1a"><msup id="A1.SS4.p1.2.m2.1.1" xref="A1.SS4.p1.2.m2.1.1.cmml"><mover accent="true" id="A1.SS4.p1.2.m2.1.1.2" xref="A1.SS4.p1.2.m2.1.1.2.cmml"><mi id="A1.SS4.p1.2.m2.1.1.2.2" xref="A1.SS4.p1.2.m2.1.1.2.2.cmml">Z</mi><mo id="A1.SS4.p1.2.m2.1.1.2.1" xref="A1.SS4.p1.2.m2.1.1.2.1.cmml">¯</mo></mover><mi id="A1.SS4.p1.2.m2.1.1.3" xref="A1.SS4.p1.2.m2.1.1.3.cmml">k</mi></msup><annotation-xml encoding="MathML-Content" id="A1.SS4.p1.2.m2.1b"><apply id="A1.SS4.p1.2.m2.1.1.cmml" xref="A1.SS4.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A1.SS4.p1.2.m2.1.1.1.cmml" xref="A1.SS4.p1.2.m2.1.1">superscript</csymbol><apply id="A1.SS4.p1.2.m2.1.1.2.cmml" xref="A1.SS4.p1.2.m2.1.1.2"><ci id="A1.SS4.p1.2.m2.1.1.2.1.cmml" xref="A1.SS4.p1.2.m2.1.1.2.1">¯</ci><ci id="A1.SS4.p1.2.m2.1.1.2.2.cmml" xref="A1.SS4.p1.2.m2.1.1.2.2">𝑍</ci></apply><ci id="A1.SS4.p1.2.m2.1.1.3.cmml" xref="A1.SS4.p1.2.m2.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p1.2.m2.1c">\bar{Z}^{k}</annotation></semantics></math> in the forward pass of the dual equals to <math id="A1.SS4.p1.3.m3.1" class="ltx_Math" alttext="W_{t}^{k}\,\bar{x}_{t}^{k}" display="inline"><semantics id="A1.SS4.p1.3.m3.1a"><mrow id="A1.SS4.p1.3.m3.1.1" xref="A1.SS4.p1.3.m3.1.1.cmml"><msubsup id="A1.SS4.p1.3.m3.1.1.2" xref="A1.SS4.p1.3.m3.1.1.2.cmml"><mi id="A1.SS4.p1.3.m3.1.1.2.2.2" xref="A1.SS4.p1.3.m3.1.1.2.2.2.cmml">W</mi><mi id="A1.SS4.p1.3.m3.1.1.2.2.3" xref="A1.SS4.p1.3.m3.1.1.2.2.3.cmml">t</mi><mi id="A1.SS4.p1.3.m3.1.1.2.3" xref="A1.SS4.p1.3.m3.1.1.2.3.cmml">k</mi></msubsup><mo lspace="0.170em" rspace="0em" id="A1.SS4.p1.3.m3.1.1.1" xref="A1.SS4.p1.3.m3.1.1.1.cmml">​</mo><msubsup id="A1.SS4.p1.3.m3.1.1.3" xref="A1.SS4.p1.3.m3.1.1.3.cmml"><mover accent="true" id="A1.SS4.p1.3.m3.1.1.3.2.2" xref="A1.SS4.p1.3.m3.1.1.3.2.2.cmml"><mi id="A1.SS4.p1.3.m3.1.1.3.2.2.2" xref="A1.SS4.p1.3.m3.1.1.3.2.2.2.cmml">x</mi><mo id="A1.SS4.p1.3.m3.1.1.3.2.2.1" xref="A1.SS4.p1.3.m3.1.1.3.2.2.1.cmml">¯</mo></mover><mi id="A1.SS4.p1.3.m3.1.1.3.2.3" xref="A1.SS4.p1.3.m3.1.1.3.2.3.cmml">t</mi><mi id="A1.SS4.p1.3.m3.1.1.3.3" xref="A1.SS4.p1.3.m3.1.1.3.3.cmml">k</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p1.3.m3.1b"><apply id="A1.SS4.p1.3.m3.1.1.cmml" xref="A1.SS4.p1.3.m3.1.1"><times id="A1.SS4.p1.3.m3.1.1.1.cmml" xref="A1.SS4.p1.3.m3.1.1.1"></times><apply id="A1.SS4.p1.3.m3.1.1.2.cmml" xref="A1.SS4.p1.3.m3.1.1.2"><csymbol cd="ambiguous" id="A1.SS4.p1.3.m3.1.1.2.1.cmml" xref="A1.SS4.p1.3.m3.1.1.2">superscript</csymbol><apply id="A1.SS4.p1.3.m3.1.1.2.2.cmml" xref="A1.SS4.p1.3.m3.1.1.2"><csymbol cd="ambiguous" id="A1.SS4.p1.3.m3.1.1.2.2.1.cmml" xref="A1.SS4.p1.3.m3.1.1.2">subscript</csymbol><ci id="A1.SS4.p1.3.m3.1.1.2.2.2.cmml" xref="A1.SS4.p1.3.m3.1.1.2.2.2">𝑊</ci><ci id="A1.SS4.p1.3.m3.1.1.2.2.3.cmml" xref="A1.SS4.p1.3.m3.1.1.2.2.3">𝑡</ci></apply><ci id="A1.SS4.p1.3.m3.1.1.2.3.cmml" xref="A1.SS4.p1.3.m3.1.1.2.3">𝑘</ci></apply><apply id="A1.SS4.p1.3.m3.1.1.3.cmml" xref="A1.SS4.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="A1.SS4.p1.3.m3.1.1.3.1.cmml" xref="A1.SS4.p1.3.m3.1.1.3">superscript</csymbol><apply id="A1.SS4.p1.3.m3.1.1.3.2.cmml" xref="A1.SS4.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="A1.SS4.p1.3.m3.1.1.3.2.1.cmml" xref="A1.SS4.p1.3.m3.1.1.3">subscript</csymbol><apply id="A1.SS4.p1.3.m3.1.1.3.2.2.cmml" xref="A1.SS4.p1.3.m3.1.1.3.2.2"><ci id="A1.SS4.p1.3.m3.1.1.3.2.2.1.cmml" xref="A1.SS4.p1.3.m3.1.1.3.2.2.1">¯</ci><ci id="A1.SS4.p1.3.m3.1.1.3.2.2.2.cmml" xref="A1.SS4.p1.3.m3.1.1.3.2.2.2">𝑥</ci></apply><ci id="A1.SS4.p1.3.m3.1.1.3.2.3.cmml" xref="A1.SS4.p1.3.m3.1.1.3.2.3">𝑡</ci></apply><ci id="A1.SS4.p1.3.m3.1.1.3.3.cmml" xref="A1.SS4.p1.3.m3.1.1.3.3">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p1.3.m3.1c">W_{t}^{k}\,\bar{x}_{t}^{k}</annotation></semantics></math> in the forward pass of the primal.
We invoke a simple fact.</p>
</div>
<div id="Thmfact1" class="ltx_theorem ltx_theorem_fact">
<h6 class="ltx_title ltx_runin ltx_title_theorem">
<span class="ltx_tag ltx_tag_theorem"><span id="Thmfact1.1.1.1" class="ltx_text ltx_font_bold">Fact 1</span></span><span id="Thmfact1.2.2" class="ltx_text ltx_font_bold">.</span>
</h6>
<div id="Thmfact1.p1" class="ltx_para">
<p id="Thmfact1.p1.6" class="ltx_p"><span id="Thmfact1.p1.6.6" class="ltx_text ltx_font_italic">Define matrices <math id="Thmfact1.p1.1.1.m1.3" class="ltx_Math" alttext="A=[a_{1},\dots,a_{b}]" display="inline"><semantics id="Thmfact1.p1.1.1.m1.3a"><mrow id="Thmfact1.p1.1.1.m1.3.3" xref="Thmfact1.p1.1.1.m1.3.3.cmml"><mi id="Thmfact1.p1.1.1.m1.3.3.4" xref="Thmfact1.p1.1.1.m1.3.3.4.cmml">A</mi><mo id="Thmfact1.p1.1.1.m1.3.3.3" xref="Thmfact1.p1.1.1.m1.3.3.3.cmml">=</mo><mrow id="Thmfact1.p1.1.1.m1.3.3.2.2" xref="Thmfact1.p1.1.1.m1.3.3.2.3.cmml"><mo stretchy="false" id="Thmfact1.p1.1.1.m1.3.3.2.2.3" xref="Thmfact1.p1.1.1.m1.3.3.2.3.cmml">[</mo><msub id="Thmfact1.p1.1.1.m1.2.2.1.1.1" xref="Thmfact1.p1.1.1.m1.2.2.1.1.1.cmml"><mi id="Thmfact1.p1.1.1.m1.2.2.1.1.1.2" xref="Thmfact1.p1.1.1.m1.2.2.1.1.1.2.cmml">a</mi><mn id="Thmfact1.p1.1.1.m1.2.2.1.1.1.3" xref="Thmfact1.p1.1.1.m1.2.2.1.1.1.3.cmml">1</mn></msub><mo id="Thmfact1.p1.1.1.m1.3.3.2.2.4" xref="Thmfact1.p1.1.1.m1.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="Thmfact1.p1.1.1.m1.1.1" xref="Thmfact1.p1.1.1.m1.1.1.cmml">…</mi><mo id="Thmfact1.p1.1.1.m1.3.3.2.2.5" xref="Thmfact1.p1.1.1.m1.3.3.2.3.cmml">,</mo><msub id="Thmfact1.p1.1.1.m1.3.3.2.2.2" xref="Thmfact1.p1.1.1.m1.3.3.2.2.2.cmml"><mi id="Thmfact1.p1.1.1.m1.3.3.2.2.2.2" xref="Thmfact1.p1.1.1.m1.3.3.2.2.2.2.cmml">a</mi><mi id="Thmfact1.p1.1.1.m1.3.3.2.2.2.3" xref="Thmfact1.p1.1.1.m1.3.3.2.2.2.3.cmml">b</mi></msub><mo stretchy="false" id="Thmfact1.p1.1.1.m1.3.3.2.2.6" xref="Thmfact1.p1.1.1.m1.3.3.2.3.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmfact1.p1.1.1.m1.3b"><apply id="Thmfact1.p1.1.1.m1.3.3.cmml" xref="Thmfact1.p1.1.1.m1.3.3"><eq id="Thmfact1.p1.1.1.m1.3.3.3.cmml" xref="Thmfact1.p1.1.1.m1.3.3.3"></eq><ci id="Thmfact1.p1.1.1.m1.3.3.4.cmml" xref="Thmfact1.p1.1.1.m1.3.3.4">𝐴</ci><list id="Thmfact1.p1.1.1.m1.3.3.2.3.cmml" xref="Thmfact1.p1.1.1.m1.3.3.2.2"><apply id="Thmfact1.p1.1.1.m1.2.2.1.1.1.cmml" xref="Thmfact1.p1.1.1.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="Thmfact1.p1.1.1.m1.2.2.1.1.1.1.cmml" xref="Thmfact1.p1.1.1.m1.2.2.1.1.1">subscript</csymbol><ci id="Thmfact1.p1.1.1.m1.2.2.1.1.1.2.cmml" xref="Thmfact1.p1.1.1.m1.2.2.1.1.1.2">𝑎</ci><cn type="integer" id="Thmfact1.p1.1.1.m1.2.2.1.1.1.3.cmml" xref="Thmfact1.p1.1.1.m1.2.2.1.1.1.3">1</cn></apply><ci id="Thmfact1.p1.1.1.m1.1.1.cmml" xref="Thmfact1.p1.1.1.m1.1.1">…</ci><apply id="Thmfact1.p1.1.1.m1.3.3.2.2.2.cmml" xref="Thmfact1.p1.1.1.m1.3.3.2.2.2"><csymbol cd="ambiguous" id="Thmfact1.p1.1.1.m1.3.3.2.2.2.1.cmml" xref="Thmfact1.p1.1.1.m1.3.3.2.2.2">subscript</csymbol><ci id="Thmfact1.p1.1.1.m1.3.3.2.2.2.2.cmml" xref="Thmfact1.p1.1.1.m1.3.3.2.2.2.2">𝑎</ci><ci id="Thmfact1.p1.1.1.m1.3.3.2.2.2.3.cmml" xref="Thmfact1.p1.1.1.m1.3.3.2.2.2.3">𝑏</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmfact1.p1.1.1.m1.3c">A=[a_{1},\dots,a_{b}]</annotation></semantics></math>, <math id="Thmfact1.p1.2.2.m2.3" class="ltx_Math" alttext="Q=[q_{1},\dots,q_{b}]" display="inline"><semantics id="Thmfact1.p1.2.2.m2.3a"><mrow id="Thmfact1.p1.2.2.m2.3.3" xref="Thmfact1.p1.2.2.m2.3.3.cmml"><mi id="Thmfact1.p1.2.2.m2.3.3.4" xref="Thmfact1.p1.2.2.m2.3.3.4.cmml">Q</mi><mo id="Thmfact1.p1.2.2.m2.3.3.3" xref="Thmfact1.p1.2.2.m2.3.3.3.cmml">=</mo><mrow id="Thmfact1.p1.2.2.m2.3.3.2.2" xref="Thmfact1.p1.2.2.m2.3.3.2.3.cmml"><mo stretchy="false" id="Thmfact1.p1.2.2.m2.3.3.2.2.3" xref="Thmfact1.p1.2.2.m2.3.3.2.3.cmml">[</mo><msub id="Thmfact1.p1.2.2.m2.2.2.1.1.1" xref="Thmfact1.p1.2.2.m2.2.2.1.1.1.cmml"><mi id="Thmfact1.p1.2.2.m2.2.2.1.1.1.2" xref="Thmfact1.p1.2.2.m2.2.2.1.1.1.2.cmml">q</mi><mn id="Thmfact1.p1.2.2.m2.2.2.1.1.1.3" xref="Thmfact1.p1.2.2.m2.2.2.1.1.1.3.cmml">1</mn></msub><mo id="Thmfact1.p1.2.2.m2.3.3.2.2.4" xref="Thmfact1.p1.2.2.m2.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="Thmfact1.p1.2.2.m2.1.1" xref="Thmfact1.p1.2.2.m2.1.1.cmml">…</mi><mo id="Thmfact1.p1.2.2.m2.3.3.2.2.5" xref="Thmfact1.p1.2.2.m2.3.3.2.3.cmml">,</mo><msub id="Thmfact1.p1.2.2.m2.3.3.2.2.2" xref="Thmfact1.p1.2.2.m2.3.3.2.2.2.cmml"><mi id="Thmfact1.p1.2.2.m2.3.3.2.2.2.2" xref="Thmfact1.p1.2.2.m2.3.3.2.2.2.2.cmml">q</mi><mi id="Thmfact1.p1.2.2.m2.3.3.2.2.2.3" xref="Thmfact1.p1.2.2.m2.3.3.2.2.2.3.cmml">b</mi></msub><mo stretchy="false" id="Thmfact1.p1.2.2.m2.3.3.2.2.6" xref="Thmfact1.p1.2.2.m2.3.3.2.3.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmfact1.p1.2.2.m2.3b"><apply id="Thmfact1.p1.2.2.m2.3.3.cmml" xref="Thmfact1.p1.2.2.m2.3.3"><eq id="Thmfact1.p1.2.2.m2.3.3.3.cmml" xref="Thmfact1.p1.2.2.m2.3.3.3"></eq><ci id="Thmfact1.p1.2.2.m2.3.3.4.cmml" xref="Thmfact1.p1.2.2.m2.3.3.4">𝑄</ci><list id="Thmfact1.p1.2.2.m2.3.3.2.3.cmml" xref="Thmfact1.p1.2.2.m2.3.3.2.2"><apply id="Thmfact1.p1.2.2.m2.2.2.1.1.1.cmml" xref="Thmfact1.p1.2.2.m2.2.2.1.1.1"><csymbol cd="ambiguous" id="Thmfact1.p1.2.2.m2.2.2.1.1.1.1.cmml" xref="Thmfact1.p1.2.2.m2.2.2.1.1.1">subscript</csymbol><ci id="Thmfact1.p1.2.2.m2.2.2.1.1.1.2.cmml" xref="Thmfact1.p1.2.2.m2.2.2.1.1.1.2">𝑞</ci><cn type="integer" id="Thmfact1.p1.2.2.m2.2.2.1.1.1.3.cmml" xref="Thmfact1.p1.2.2.m2.2.2.1.1.1.3">1</cn></apply><ci id="Thmfact1.p1.2.2.m2.1.1.cmml" xref="Thmfact1.p1.2.2.m2.1.1">…</ci><apply id="Thmfact1.p1.2.2.m2.3.3.2.2.2.cmml" xref="Thmfact1.p1.2.2.m2.3.3.2.2.2"><csymbol cd="ambiguous" id="Thmfact1.p1.2.2.m2.3.3.2.2.2.1.cmml" xref="Thmfact1.p1.2.2.m2.3.3.2.2.2">subscript</csymbol><ci id="Thmfact1.p1.2.2.m2.3.3.2.2.2.2.cmml" xref="Thmfact1.p1.2.2.m2.3.3.2.2.2.2">𝑞</ci><ci id="Thmfact1.p1.2.2.m2.3.3.2.2.2.3.cmml" xref="Thmfact1.p1.2.2.m2.3.3.2.2.2.3">𝑏</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmfact1.p1.2.2.m2.3c">Q=[q_{1},\dots,q_{b}]</annotation></semantics></math>, and <math id="Thmfact1.p1.3.3.m3.3" class="ltx_Math" alttext="V=[v_{1},\dots,v_{b}]" display="inline"><semantics id="Thmfact1.p1.3.3.m3.3a"><mrow id="Thmfact1.p1.3.3.m3.3.3" xref="Thmfact1.p1.3.3.m3.3.3.cmml"><mi id="Thmfact1.p1.3.3.m3.3.3.4" xref="Thmfact1.p1.3.3.m3.3.3.4.cmml">V</mi><mo id="Thmfact1.p1.3.3.m3.3.3.3" xref="Thmfact1.p1.3.3.m3.3.3.3.cmml">=</mo><mrow id="Thmfact1.p1.3.3.m3.3.3.2.2" xref="Thmfact1.p1.3.3.m3.3.3.2.3.cmml"><mo stretchy="false" id="Thmfact1.p1.3.3.m3.3.3.2.2.3" xref="Thmfact1.p1.3.3.m3.3.3.2.3.cmml">[</mo><msub id="Thmfact1.p1.3.3.m3.2.2.1.1.1" xref="Thmfact1.p1.3.3.m3.2.2.1.1.1.cmml"><mi id="Thmfact1.p1.3.3.m3.2.2.1.1.1.2" xref="Thmfact1.p1.3.3.m3.2.2.1.1.1.2.cmml">v</mi><mn id="Thmfact1.p1.3.3.m3.2.2.1.1.1.3" xref="Thmfact1.p1.3.3.m3.2.2.1.1.1.3.cmml">1</mn></msub><mo id="Thmfact1.p1.3.3.m3.3.3.2.2.4" xref="Thmfact1.p1.3.3.m3.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="Thmfact1.p1.3.3.m3.1.1" xref="Thmfact1.p1.3.3.m3.1.1.cmml">…</mi><mo id="Thmfact1.p1.3.3.m3.3.3.2.2.5" xref="Thmfact1.p1.3.3.m3.3.3.2.3.cmml">,</mo><msub id="Thmfact1.p1.3.3.m3.3.3.2.2.2" xref="Thmfact1.p1.3.3.m3.3.3.2.2.2.cmml"><mi id="Thmfact1.p1.3.3.m3.3.3.2.2.2.2" xref="Thmfact1.p1.3.3.m3.3.3.2.2.2.2.cmml">v</mi><mi id="Thmfact1.p1.3.3.m3.3.3.2.2.2.3" xref="Thmfact1.p1.3.3.m3.3.3.2.2.2.3.cmml">b</mi></msub><mo stretchy="false" id="Thmfact1.p1.3.3.m3.3.3.2.2.6" xref="Thmfact1.p1.3.3.m3.3.3.2.3.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmfact1.p1.3.3.m3.3b"><apply id="Thmfact1.p1.3.3.m3.3.3.cmml" xref="Thmfact1.p1.3.3.m3.3.3"><eq id="Thmfact1.p1.3.3.m3.3.3.3.cmml" xref="Thmfact1.p1.3.3.m3.3.3.3"></eq><ci id="Thmfact1.p1.3.3.m3.3.3.4.cmml" xref="Thmfact1.p1.3.3.m3.3.3.4">𝑉</ci><list id="Thmfact1.p1.3.3.m3.3.3.2.3.cmml" xref="Thmfact1.p1.3.3.m3.3.3.2.2"><apply id="Thmfact1.p1.3.3.m3.2.2.1.1.1.cmml" xref="Thmfact1.p1.3.3.m3.2.2.1.1.1"><csymbol cd="ambiguous" id="Thmfact1.p1.3.3.m3.2.2.1.1.1.1.cmml" xref="Thmfact1.p1.3.3.m3.2.2.1.1.1">subscript</csymbol><ci id="Thmfact1.p1.3.3.m3.2.2.1.1.1.2.cmml" xref="Thmfact1.p1.3.3.m3.2.2.1.1.1.2">𝑣</ci><cn type="integer" id="Thmfact1.p1.3.3.m3.2.2.1.1.1.3.cmml" xref="Thmfact1.p1.3.3.m3.2.2.1.1.1.3">1</cn></apply><ci id="Thmfact1.p1.3.3.m3.1.1.cmml" xref="Thmfact1.p1.3.3.m3.1.1">…</ci><apply id="Thmfact1.p1.3.3.m3.3.3.2.2.2.cmml" xref="Thmfact1.p1.3.3.m3.3.3.2.2.2"><csymbol cd="ambiguous" id="Thmfact1.p1.3.3.m3.3.3.2.2.2.1.cmml" xref="Thmfact1.p1.3.3.m3.3.3.2.2.2">subscript</csymbol><ci id="Thmfact1.p1.3.3.m3.3.3.2.2.2.2.cmml" xref="Thmfact1.p1.3.3.m3.3.3.2.2.2.2">𝑣</ci><ci id="Thmfact1.p1.3.3.m3.3.3.2.2.2.3.cmml" xref="Thmfact1.p1.3.3.m3.3.3.2.2.2.3">𝑏</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmfact1.p1.3.3.m3.3c">V=[v_{1},\dots,v_{b}]</annotation></semantics></math>.<span id="footnote17" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">17</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">17</sup><span class="ltx_tag ltx_tag_note"><span id="footnote17.1.1.1" class="ltx_text ltx_font_upright">17</span></span><span id="footnote17.5" class="ltx_text ltx_font_upright">
Our matrix </span><math id="footnote17.m1.1" class="ltx_Math" alttext="A" display="inline"><semantics id="footnote17.m1.1b"><mi id="footnote17.m1.1.1" xref="footnote17.m1.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="footnote17.m1.1c"><ci id="footnote17.m1.1.1.cmml" xref="footnote17.m1.1.1">𝐴</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote17.m1.1d">A</annotation></semantics></math><span id="footnote17.6" class="ltx_text ltx_font_upright"> would usually be denoted by </span><math id="footnote17.m2.1" class="ltx_Math" alttext="K" display="inline"><semantics id="footnote17.m2.1b"><mi id="footnote17.m2.1.1" xref="footnote17.m2.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="footnote17.m2.1c"><ci id="footnote17.m2.1.1.cmml" xref="footnote17.m2.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote17.m2.1d">K</annotation></semantics></math><span id="footnote17.7" class="ltx_text ltx_font_upright"> in another context.
We use </span><math id="footnote17.m3.1" class="ltx_Math" alttext="A" display="inline"><semantics id="footnote17.m3.1b"><mi id="footnote17.m3.1.1" xref="footnote17.m3.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="footnote17.m3.1c"><ci id="footnote17.m3.1.1.cmml" xref="footnote17.m3.1.1">𝐴</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote17.m3.1d">A</annotation></semantics></math><span id="footnote17.8" class="ltx_text ltx_font_upright"> to avoid confusion with the layer number </span><math id="footnote17.m4.1" class="ltx_Math" alttext="K" display="inline"><semantics id="footnote17.m4.1b"><mi id="footnote17.m4.1.1" xref="footnote17.m4.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="footnote17.m4.1c"><ci id="footnote17.m4.1.1.cmml" xref="footnote17.m4.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="footnote17.m4.1d">K</annotation></semantics></math><span id="footnote17.9" class="ltx_text ltx_font_upright">.
</span></span></span></span>
Define <math id="Thmfact1.p1.4.4.m4.1" class="ltx_Math" alttext="\hat{v}_{t}=\sum_{s=1}^{t}a_{s}^{T}q_{t}v_{s}" display="inline"><semantics id="Thmfact1.p1.4.4.m4.1a"><mrow id="Thmfact1.p1.4.4.m4.1.1" xref="Thmfact1.p1.4.4.m4.1.1.cmml"><msub id="Thmfact1.p1.4.4.m4.1.1.2" xref="Thmfact1.p1.4.4.m4.1.1.2.cmml"><mover accent="true" id="Thmfact1.p1.4.4.m4.1.1.2.2" xref="Thmfact1.p1.4.4.m4.1.1.2.2.cmml"><mi id="Thmfact1.p1.4.4.m4.1.1.2.2.2" xref="Thmfact1.p1.4.4.m4.1.1.2.2.2.cmml">v</mi><mo id="Thmfact1.p1.4.4.m4.1.1.2.2.1" xref="Thmfact1.p1.4.4.m4.1.1.2.2.1.cmml">^</mo></mover><mi id="Thmfact1.p1.4.4.m4.1.1.2.3" xref="Thmfact1.p1.4.4.m4.1.1.2.3.cmml">t</mi></msub><mo rspace="0.111em" id="Thmfact1.p1.4.4.m4.1.1.1" xref="Thmfact1.p1.4.4.m4.1.1.1.cmml">=</mo><mrow id="Thmfact1.p1.4.4.m4.1.1.3" xref="Thmfact1.p1.4.4.m4.1.1.3.cmml"><msubsup id="Thmfact1.p1.4.4.m4.1.1.3.1" xref="Thmfact1.p1.4.4.m4.1.1.3.1.cmml"><mo id="Thmfact1.p1.4.4.m4.1.1.3.1.2.2" xref="Thmfact1.p1.4.4.m4.1.1.3.1.2.2.cmml">∑</mo><mrow id="Thmfact1.p1.4.4.m4.1.1.3.1.2.3" xref="Thmfact1.p1.4.4.m4.1.1.3.1.2.3.cmml"><mi id="Thmfact1.p1.4.4.m4.1.1.3.1.2.3.2" xref="Thmfact1.p1.4.4.m4.1.1.3.1.2.3.2.cmml">s</mi><mo id="Thmfact1.p1.4.4.m4.1.1.3.1.2.3.1" xref="Thmfact1.p1.4.4.m4.1.1.3.1.2.3.1.cmml">=</mo><mn id="Thmfact1.p1.4.4.m4.1.1.3.1.2.3.3" xref="Thmfact1.p1.4.4.m4.1.1.3.1.2.3.3.cmml">1</mn></mrow><mi id="Thmfact1.p1.4.4.m4.1.1.3.1.3" xref="Thmfact1.p1.4.4.m4.1.1.3.1.3.cmml">t</mi></msubsup><mrow id="Thmfact1.p1.4.4.m4.1.1.3.2" xref="Thmfact1.p1.4.4.m4.1.1.3.2.cmml"><msubsup id="Thmfact1.p1.4.4.m4.1.1.3.2.2" xref="Thmfact1.p1.4.4.m4.1.1.3.2.2.cmml"><mi id="Thmfact1.p1.4.4.m4.1.1.3.2.2.2.2" xref="Thmfact1.p1.4.4.m4.1.1.3.2.2.2.2.cmml">a</mi><mi id="Thmfact1.p1.4.4.m4.1.1.3.2.2.2.3" xref="Thmfact1.p1.4.4.m4.1.1.3.2.2.2.3.cmml">s</mi><mi id="Thmfact1.p1.4.4.m4.1.1.3.2.2.3" xref="Thmfact1.p1.4.4.m4.1.1.3.2.2.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="Thmfact1.p1.4.4.m4.1.1.3.2.1" xref="Thmfact1.p1.4.4.m4.1.1.3.2.1.cmml">​</mo><msub id="Thmfact1.p1.4.4.m4.1.1.3.2.3" xref="Thmfact1.p1.4.4.m4.1.1.3.2.3.cmml"><mi id="Thmfact1.p1.4.4.m4.1.1.3.2.3.2" xref="Thmfact1.p1.4.4.m4.1.1.3.2.3.2.cmml">q</mi><mi id="Thmfact1.p1.4.4.m4.1.1.3.2.3.3" xref="Thmfact1.p1.4.4.m4.1.1.3.2.3.3.cmml">t</mi></msub><mo lspace="0em" rspace="0em" id="Thmfact1.p1.4.4.m4.1.1.3.2.1a" xref="Thmfact1.p1.4.4.m4.1.1.3.2.1.cmml">​</mo><msub id="Thmfact1.p1.4.4.m4.1.1.3.2.4" xref="Thmfact1.p1.4.4.m4.1.1.3.2.4.cmml"><mi id="Thmfact1.p1.4.4.m4.1.1.3.2.4.2" xref="Thmfact1.p1.4.4.m4.1.1.3.2.4.2.cmml">v</mi><mi id="Thmfact1.p1.4.4.m4.1.1.3.2.4.3" xref="Thmfact1.p1.4.4.m4.1.1.3.2.4.3.cmml">s</mi></msub></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmfact1.p1.4.4.m4.1b"><apply id="Thmfact1.p1.4.4.m4.1.1.cmml" xref="Thmfact1.p1.4.4.m4.1.1"><eq id="Thmfact1.p1.4.4.m4.1.1.1.cmml" xref="Thmfact1.p1.4.4.m4.1.1.1"></eq><apply id="Thmfact1.p1.4.4.m4.1.1.2.cmml" xref="Thmfact1.p1.4.4.m4.1.1.2"><csymbol cd="ambiguous" id="Thmfact1.p1.4.4.m4.1.1.2.1.cmml" xref="Thmfact1.p1.4.4.m4.1.1.2">subscript</csymbol><apply id="Thmfact1.p1.4.4.m4.1.1.2.2.cmml" xref="Thmfact1.p1.4.4.m4.1.1.2.2"><ci id="Thmfact1.p1.4.4.m4.1.1.2.2.1.cmml" xref="Thmfact1.p1.4.4.m4.1.1.2.2.1">^</ci><ci id="Thmfact1.p1.4.4.m4.1.1.2.2.2.cmml" xref="Thmfact1.p1.4.4.m4.1.1.2.2.2">𝑣</ci></apply><ci id="Thmfact1.p1.4.4.m4.1.1.2.3.cmml" xref="Thmfact1.p1.4.4.m4.1.1.2.3">𝑡</ci></apply><apply id="Thmfact1.p1.4.4.m4.1.1.3.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3"><apply id="Thmfact1.p1.4.4.m4.1.1.3.1.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.1"><csymbol cd="ambiguous" id="Thmfact1.p1.4.4.m4.1.1.3.1.1.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.1">superscript</csymbol><apply id="Thmfact1.p1.4.4.m4.1.1.3.1.2.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.1"><csymbol cd="ambiguous" id="Thmfact1.p1.4.4.m4.1.1.3.1.2.1.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.1">subscript</csymbol><sum id="Thmfact1.p1.4.4.m4.1.1.3.1.2.2.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.1.2.2"></sum><apply id="Thmfact1.p1.4.4.m4.1.1.3.1.2.3.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.1.2.3"><eq id="Thmfact1.p1.4.4.m4.1.1.3.1.2.3.1.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.1.2.3.1"></eq><ci id="Thmfact1.p1.4.4.m4.1.1.3.1.2.3.2.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.1.2.3.2">𝑠</ci><cn type="integer" id="Thmfact1.p1.4.4.m4.1.1.3.1.2.3.3.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.1.2.3.3">1</cn></apply></apply><ci id="Thmfact1.p1.4.4.m4.1.1.3.1.3.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.1.3">𝑡</ci></apply><apply id="Thmfact1.p1.4.4.m4.1.1.3.2.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.2"><times id="Thmfact1.p1.4.4.m4.1.1.3.2.1.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.2.1"></times><apply id="Thmfact1.p1.4.4.m4.1.1.3.2.2.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.2.2"><csymbol cd="ambiguous" id="Thmfact1.p1.4.4.m4.1.1.3.2.2.1.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.2.2">superscript</csymbol><apply id="Thmfact1.p1.4.4.m4.1.1.3.2.2.2.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.2.2"><csymbol cd="ambiguous" id="Thmfact1.p1.4.4.m4.1.1.3.2.2.2.1.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.2.2">subscript</csymbol><ci id="Thmfact1.p1.4.4.m4.1.1.3.2.2.2.2.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.2.2.2.2">𝑎</ci><ci id="Thmfact1.p1.4.4.m4.1.1.3.2.2.2.3.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.2.2.2.3">𝑠</ci></apply><ci id="Thmfact1.p1.4.4.m4.1.1.3.2.2.3.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.2.2.3">𝑇</ci></apply><apply id="Thmfact1.p1.4.4.m4.1.1.3.2.3.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.2.3"><csymbol cd="ambiguous" id="Thmfact1.p1.4.4.m4.1.1.3.2.3.1.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.2.3">subscript</csymbol><ci id="Thmfact1.p1.4.4.m4.1.1.3.2.3.2.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.2.3.2">𝑞</ci><ci id="Thmfact1.p1.4.4.m4.1.1.3.2.3.3.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.2.3.3">𝑡</ci></apply><apply id="Thmfact1.p1.4.4.m4.1.1.3.2.4.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.2.4"><csymbol cd="ambiguous" id="Thmfact1.p1.4.4.m4.1.1.3.2.4.1.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.2.4">subscript</csymbol><ci id="Thmfact1.p1.4.4.m4.1.1.3.2.4.2.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.2.4.2">𝑣</ci><ci id="Thmfact1.p1.4.4.m4.1.1.3.2.4.3.cmml" xref="Thmfact1.p1.4.4.m4.1.1.3.2.4.3">𝑠</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmfact1.p1.4.4.m4.1c">\hat{v}_{t}=\sum_{s=1}^{t}a_{s}^{T}q_{t}v_{s}</annotation></semantics></math>, and <math id="Thmfact1.p1.5.5.m5.3" class="ltx_Math" alttext="\hat{V}=[\hat{v}_{1},\dots,\hat{v}_{b}]" display="inline"><semantics id="Thmfact1.p1.5.5.m5.3a"><mrow id="Thmfact1.p1.5.5.m5.3.3" xref="Thmfact1.p1.5.5.m5.3.3.cmml"><mover accent="true" id="Thmfact1.p1.5.5.m5.3.3.4" xref="Thmfact1.p1.5.5.m5.3.3.4.cmml"><mi id="Thmfact1.p1.5.5.m5.3.3.4.2" xref="Thmfact1.p1.5.5.m5.3.3.4.2.cmml">V</mi><mo id="Thmfact1.p1.5.5.m5.3.3.4.1" xref="Thmfact1.p1.5.5.m5.3.3.4.1.cmml">^</mo></mover><mo id="Thmfact1.p1.5.5.m5.3.3.3" xref="Thmfact1.p1.5.5.m5.3.3.3.cmml">=</mo><mrow id="Thmfact1.p1.5.5.m5.3.3.2.2" xref="Thmfact1.p1.5.5.m5.3.3.2.3.cmml"><mo stretchy="false" id="Thmfact1.p1.5.5.m5.3.3.2.2.3" xref="Thmfact1.p1.5.5.m5.3.3.2.3.cmml">[</mo><msub id="Thmfact1.p1.5.5.m5.2.2.1.1.1" xref="Thmfact1.p1.5.5.m5.2.2.1.1.1.cmml"><mover accent="true" id="Thmfact1.p1.5.5.m5.2.2.1.1.1.2" xref="Thmfact1.p1.5.5.m5.2.2.1.1.1.2.cmml"><mi id="Thmfact1.p1.5.5.m5.2.2.1.1.1.2.2" xref="Thmfact1.p1.5.5.m5.2.2.1.1.1.2.2.cmml">v</mi><mo id="Thmfact1.p1.5.5.m5.2.2.1.1.1.2.1" xref="Thmfact1.p1.5.5.m5.2.2.1.1.1.2.1.cmml">^</mo></mover><mn id="Thmfact1.p1.5.5.m5.2.2.1.1.1.3" xref="Thmfact1.p1.5.5.m5.2.2.1.1.1.3.cmml">1</mn></msub><mo id="Thmfact1.p1.5.5.m5.3.3.2.2.4" xref="Thmfact1.p1.5.5.m5.3.3.2.3.cmml">,</mo><mi mathvariant="normal" id="Thmfact1.p1.5.5.m5.1.1" xref="Thmfact1.p1.5.5.m5.1.1.cmml">…</mi><mo id="Thmfact1.p1.5.5.m5.3.3.2.2.5" xref="Thmfact1.p1.5.5.m5.3.3.2.3.cmml">,</mo><msub id="Thmfact1.p1.5.5.m5.3.3.2.2.2" xref="Thmfact1.p1.5.5.m5.3.3.2.2.2.cmml"><mover accent="true" id="Thmfact1.p1.5.5.m5.3.3.2.2.2.2" xref="Thmfact1.p1.5.5.m5.3.3.2.2.2.2.cmml"><mi id="Thmfact1.p1.5.5.m5.3.3.2.2.2.2.2" xref="Thmfact1.p1.5.5.m5.3.3.2.2.2.2.2.cmml">v</mi><mo id="Thmfact1.p1.5.5.m5.3.3.2.2.2.2.1" xref="Thmfact1.p1.5.5.m5.3.3.2.2.2.2.1.cmml">^</mo></mover><mi id="Thmfact1.p1.5.5.m5.3.3.2.2.2.3" xref="Thmfact1.p1.5.5.m5.3.3.2.2.2.3.cmml">b</mi></msub><mo stretchy="false" id="Thmfact1.p1.5.5.m5.3.3.2.2.6" xref="Thmfact1.p1.5.5.m5.3.3.2.3.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmfact1.p1.5.5.m5.3b"><apply id="Thmfact1.p1.5.5.m5.3.3.cmml" xref="Thmfact1.p1.5.5.m5.3.3"><eq id="Thmfact1.p1.5.5.m5.3.3.3.cmml" xref="Thmfact1.p1.5.5.m5.3.3.3"></eq><apply id="Thmfact1.p1.5.5.m5.3.3.4.cmml" xref="Thmfact1.p1.5.5.m5.3.3.4"><ci id="Thmfact1.p1.5.5.m5.3.3.4.1.cmml" xref="Thmfact1.p1.5.5.m5.3.3.4.1">^</ci><ci id="Thmfact1.p1.5.5.m5.3.3.4.2.cmml" xref="Thmfact1.p1.5.5.m5.3.3.4.2">𝑉</ci></apply><list id="Thmfact1.p1.5.5.m5.3.3.2.3.cmml" xref="Thmfact1.p1.5.5.m5.3.3.2.2"><apply id="Thmfact1.p1.5.5.m5.2.2.1.1.1.cmml" xref="Thmfact1.p1.5.5.m5.2.2.1.1.1"><csymbol cd="ambiguous" id="Thmfact1.p1.5.5.m5.2.2.1.1.1.1.cmml" xref="Thmfact1.p1.5.5.m5.2.2.1.1.1">subscript</csymbol><apply id="Thmfact1.p1.5.5.m5.2.2.1.1.1.2.cmml" xref="Thmfact1.p1.5.5.m5.2.2.1.1.1.2"><ci id="Thmfact1.p1.5.5.m5.2.2.1.1.1.2.1.cmml" xref="Thmfact1.p1.5.5.m5.2.2.1.1.1.2.1">^</ci><ci id="Thmfact1.p1.5.5.m5.2.2.1.1.1.2.2.cmml" xref="Thmfact1.p1.5.5.m5.2.2.1.1.1.2.2">𝑣</ci></apply><cn type="integer" id="Thmfact1.p1.5.5.m5.2.2.1.1.1.3.cmml" xref="Thmfact1.p1.5.5.m5.2.2.1.1.1.3">1</cn></apply><ci id="Thmfact1.p1.5.5.m5.1.1.cmml" xref="Thmfact1.p1.5.5.m5.1.1">…</ci><apply id="Thmfact1.p1.5.5.m5.3.3.2.2.2.cmml" xref="Thmfact1.p1.5.5.m5.3.3.2.2.2"><csymbol cd="ambiguous" id="Thmfact1.p1.5.5.m5.3.3.2.2.2.1.cmml" xref="Thmfact1.p1.5.5.m5.3.3.2.2.2">subscript</csymbol><apply id="Thmfact1.p1.5.5.m5.3.3.2.2.2.2.cmml" xref="Thmfact1.p1.5.5.m5.3.3.2.2.2.2"><ci id="Thmfact1.p1.5.5.m5.3.3.2.2.2.2.1.cmml" xref="Thmfact1.p1.5.5.m5.3.3.2.2.2.2.1">^</ci><ci id="Thmfact1.p1.5.5.m5.3.3.2.2.2.2.2.cmml" xref="Thmfact1.p1.5.5.m5.3.3.2.2.2.2.2">𝑣</ci></apply><ci id="Thmfact1.p1.5.5.m5.3.3.2.2.2.3.cmml" xref="Thmfact1.p1.5.5.m5.3.3.2.2.2.3">𝑏</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmfact1.p1.5.5.m5.3c">\hat{V}=[\hat{v}_{1},\dots,\hat{v}_{b}]</annotation></semantics></math>, then <math id="Thmfact1.p1.6.6.m6.1" class="ltx_Math" alttext="\hat{V}=V\cdot\texttt{mask}(A^{T}Q)" display="inline"><semantics id="Thmfact1.p1.6.6.m6.1a"><mrow id="Thmfact1.p1.6.6.m6.1.1" xref="Thmfact1.p1.6.6.m6.1.1.cmml"><mover accent="true" id="Thmfact1.p1.6.6.m6.1.1.3" xref="Thmfact1.p1.6.6.m6.1.1.3.cmml"><mi id="Thmfact1.p1.6.6.m6.1.1.3.2" xref="Thmfact1.p1.6.6.m6.1.1.3.2.cmml">V</mi><mo id="Thmfact1.p1.6.6.m6.1.1.3.1" xref="Thmfact1.p1.6.6.m6.1.1.3.1.cmml">^</mo></mover><mo id="Thmfact1.p1.6.6.m6.1.1.2" xref="Thmfact1.p1.6.6.m6.1.1.2.cmml">=</mo><mrow id="Thmfact1.p1.6.6.m6.1.1.1" xref="Thmfact1.p1.6.6.m6.1.1.1.cmml"><mrow id="Thmfact1.p1.6.6.m6.1.1.1.3" xref="Thmfact1.p1.6.6.m6.1.1.1.3.cmml"><mi id="Thmfact1.p1.6.6.m6.1.1.1.3.2" xref="Thmfact1.p1.6.6.m6.1.1.1.3.2.cmml">V</mi><mo lspace="0.222em" rspace="0.222em" id="Thmfact1.p1.6.6.m6.1.1.1.3.1" xref="Thmfact1.p1.6.6.m6.1.1.1.3.1.cmml">⋅</mo><mtext class="ltx_mathvariant_monospace" id="Thmfact1.p1.6.6.m6.1.1.1.3.3" xref="Thmfact1.p1.6.6.m6.1.1.1.3.3a.cmml">mask</mtext></mrow><mo lspace="0em" rspace="0em" id="Thmfact1.p1.6.6.m6.1.1.1.2" xref="Thmfact1.p1.6.6.m6.1.1.1.2.cmml">​</mo><mrow id="Thmfact1.p1.6.6.m6.1.1.1.1.1" xref="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.cmml"><mo stretchy="false" id="Thmfact1.p1.6.6.m6.1.1.1.1.1.2" xref="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.cmml">(</mo><mrow id="Thmfact1.p1.6.6.m6.1.1.1.1.1.1" xref="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.cmml"><msup id="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.2" xref="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.2.cmml"><mi id="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.2.2" xref="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.2.2.cmml">A</mi><mi id="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.2.3" xref="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.2.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.1" xref="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.1.cmml">​</mo><mi id="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.3" xref="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.3.cmml">Q</mi></mrow><mo stretchy="false" id="Thmfact1.p1.6.6.m6.1.1.1.1.1.3" xref="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="Thmfact1.p1.6.6.m6.1b"><apply id="Thmfact1.p1.6.6.m6.1.1.cmml" xref="Thmfact1.p1.6.6.m6.1.1"><eq id="Thmfact1.p1.6.6.m6.1.1.2.cmml" xref="Thmfact1.p1.6.6.m6.1.1.2"></eq><apply id="Thmfact1.p1.6.6.m6.1.1.3.cmml" xref="Thmfact1.p1.6.6.m6.1.1.3"><ci id="Thmfact1.p1.6.6.m6.1.1.3.1.cmml" xref="Thmfact1.p1.6.6.m6.1.1.3.1">^</ci><ci id="Thmfact1.p1.6.6.m6.1.1.3.2.cmml" xref="Thmfact1.p1.6.6.m6.1.1.3.2">𝑉</ci></apply><apply id="Thmfact1.p1.6.6.m6.1.1.1.cmml" xref="Thmfact1.p1.6.6.m6.1.1.1"><times id="Thmfact1.p1.6.6.m6.1.1.1.2.cmml" xref="Thmfact1.p1.6.6.m6.1.1.1.2"></times><apply id="Thmfact1.p1.6.6.m6.1.1.1.3.cmml" xref="Thmfact1.p1.6.6.m6.1.1.1.3"><ci id="Thmfact1.p1.6.6.m6.1.1.1.3.1.cmml" xref="Thmfact1.p1.6.6.m6.1.1.1.3.1">⋅</ci><ci id="Thmfact1.p1.6.6.m6.1.1.1.3.2.cmml" xref="Thmfact1.p1.6.6.m6.1.1.1.3.2">𝑉</ci><ci id="Thmfact1.p1.6.6.m6.1.1.1.3.3a.cmml" xref="Thmfact1.p1.6.6.m6.1.1.1.3.3"><mtext class="ltx_mathvariant_monospace" id="Thmfact1.p1.6.6.m6.1.1.1.3.3.cmml" xref="Thmfact1.p1.6.6.m6.1.1.1.3.3">mask</mtext></ci></apply><apply id="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.cmml" xref="Thmfact1.p1.6.6.m6.1.1.1.1.1"><times id="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.1.cmml" xref="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.1"></times><apply id="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.2.cmml" xref="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.2.1.cmml" xref="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.2">superscript</csymbol><ci id="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.2.2.cmml" xref="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.2.2">𝐴</ci><ci id="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.2.3.cmml" xref="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.2.3">𝑇</ci></apply><ci id="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.3.cmml" xref="Thmfact1.p1.6.6.m6.1.1.1.1.1.1.3">𝑄</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="Thmfact1.p1.6.6.m6.1c">\hat{V}=V\cdot\texttt{mask}(A^{T}Q)</annotation></semantics></math>.</span></p>
</div>
</div>
<div id="A1.SS4.p2" class="ltx_para">
<p id="A1.SS4.p2.4" class="ltx_p">Now plug <math id="A1.SS4.p2.1.m1.1" class="ltx_Math" alttext="A=\hat{X}^{k}" display="inline"><semantics id="A1.SS4.p2.1.m1.1a"><mrow id="A1.SS4.p2.1.m1.1.1" xref="A1.SS4.p2.1.m1.1.1.cmml"><mi id="A1.SS4.p2.1.m1.1.1.2" xref="A1.SS4.p2.1.m1.1.1.2.cmml">A</mi><mo id="A1.SS4.p2.1.m1.1.1.1" xref="A1.SS4.p2.1.m1.1.1.1.cmml">=</mo><msup id="A1.SS4.p2.1.m1.1.1.3" xref="A1.SS4.p2.1.m1.1.1.3.cmml"><mover accent="true" id="A1.SS4.p2.1.m1.1.1.3.2" xref="A1.SS4.p2.1.m1.1.1.3.2.cmml"><mi id="A1.SS4.p2.1.m1.1.1.3.2.2" xref="A1.SS4.p2.1.m1.1.1.3.2.2.cmml">X</mi><mo id="A1.SS4.p2.1.m1.1.1.3.2.1" xref="A1.SS4.p2.1.m1.1.1.3.2.1.cmml">^</mo></mover><mi id="A1.SS4.p2.1.m1.1.1.3.3" xref="A1.SS4.p2.1.m1.1.1.3.3.cmml">k</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p2.1.m1.1b"><apply id="A1.SS4.p2.1.m1.1.1.cmml" xref="A1.SS4.p2.1.m1.1.1"><eq id="A1.SS4.p2.1.m1.1.1.1.cmml" xref="A1.SS4.p2.1.m1.1.1.1"></eq><ci id="A1.SS4.p2.1.m1.1.1.2.cmml" xref="A1.SS4.p2.1.m1.1.1.2">𝐴</ci><apply id="A1.SS4.p2.1.m1.1.1.3.cmml" xref="A1.SS4.p2.1.m1.1.1.3"><csymbol cd="ambiguous" id="A1.SS4.p2.1.m1.1.1.3.1.cmml" xref="A1.SS4.p2.1.m1.1.1.3">superscript</csymbol><apply id="A1.SS4.p2.1.m1.1.1.3.2.cmml" xref="A1.SS4.p2.1.m1.1.1.3.2"><ci id="A1.SS4.p2.1.m1.1.1.3.2.1.cmml" xref="A1.SS4.p2.1.m1.1.1.3.2.1">^</ci><ci id="A1.SS4.p2.1.m1.1.1.3.2.2.cmml" xref="A1.SS4.p2.1.m1.1.1.3.2.2">𝑋</ci></apply><ci id="A1.SS4.p2.1.m1.1.1.3.3.cmml" xref="A1.SS4.p2.1.m1.1.1.3.3">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p2.1.m1.1c">A=\hat{X}^{k}</annotation></semantics></math>, <math id="A1.SS4.p2.2.m2.1" class="ltx_Math" alttext="Q=\bar{X}^{k}" display="inline"><semantics id="A1.SS4.p2.2.m2.1a"><mrow id="A1.SS4.p2.2.m2.1.1" xref="A1.SS4.p2.2.m2.1.1.cmml"><mi id="A1.SS4.p2.2.m2.1.1.2" xref="A1.SS4.p2.2.m2.1.1.2.cmml">Q</mi><mo id="A1.SS4.p2.2.m2.1.1.1" xref="A1.SS4.p2.2.m2.1.1.1.cmml">=</mo><msup id="A1.SS4.p2.2.m2.1.1.3" xref="A1.SS4.p2.2.m2.1.1.3.cmml"><mover accent="true" id="A1.SS4.p2.2.m2.1.1.3.2" xref="A1.SS4.p2.2.m2.1.1.3.2.cmml"><mi id="A1.SS4.p2.2.m2.1.1.3.2.2" xref="A1.SS4.p2.2.m2.1.1.3.2.2.cmml">X</mi><mo id="A1.SS4.p2.2.m2.1.1.3.2.1" xref="A1.SS4.p2.2.m2.1.1.3.2.1.cmml">¯</mo></mover><mi id="A1.SS4.p2.2.m2.1.1.3.3" xref="A1.SS4.p2.2.m2.1.1.3.3.cmml">k</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p2.2.m2.1b"><apply id="A1.SS4.p2.2.m2.1.1.cmml" xref="A1.SS4.p2.2.m2.1.1"><eq id="A1.SS4.p2.2.m2.1.1.1.cmml" xref="A1.SS4.p2.2.m2.1.1.1"></eq><ci id="A1.SS4.p2.2.m2.1.1.2.cmml" xref="A1.SS4.p2.2.m2.1.1.2">𝑄</ci><apply id="A1.SS4.p2.2.m2.1.1.3.cmml" xref="A1.SS4.p2.2.m2.1.1.3"><csymbol cd="ambiguous" id="A1.SS4.p2.2.m2.1.1.3.1.cmml" xref="A1.SS4.p2.2.m2.1.1.3">superscript</csymbol><apply id="A1.SS4.p2.2.m2.1.1.3.2.cmml" xref="A1.SS4.p2.2.m2.1.1.3.2"><ci id="A1.SS4.p2.2.m2.1.1.3.2.1.cmml" xref="A1.SS4.p2.2.m2.1.1.3.2.1">¯</ci><ci id="A1.SS4.p2.2.m2.1.1.3.2.2.cmml" xref="A1.SS4.p2.2.m2.1.1.3.2.2">𝑋</ci></apply><ci id="A1.SS4.p2.2.m2.1.1.3.3.cmml" xref="A1.SS4.p2.2.m2.1.1.3.3">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p2.2.m2.1c">Q=\bar{X}^{k}</annotation></semantics></math>, <math id="A1.SS4.p2.3.m3.1" class="ltx_Math" alttext="V=\nabla_{Z^{k}}l" display="inline"><semantics id="A1.SS4.p2.3.m3.1a"><mrow id="A1.SS4.p2.3.m3.1.1" xref="A1.SS4.p2.3.m3.1.1.cmml"><mi id="A1.SS4.p2.3.m3.1.1.2" xref="A1.SS4.p2.3.m3.1.1.2.cmml">V</mi><mo id="A1.SS4.p2.3.m3.1.1.1" xref="A1.SS4.p2.3.m3.1.1.1.cmml">=</mo><mrow id="A1.SS4.p2.3.m3.1.1.3" xref="A1.SS4.p2.3.m3.1.1.3.cmml"><msub id="A1.SS4.p2.3.m3.1.1.3.1" xref="A1.SS4.p2.3.m3.1.1.3.1.cmml"><mo rspace="0.167em" id="A1.SS4.p2.3.m3.1.1.3.1.2" xref="A1.SS4.p2.3.m3.1.1.3.1.2.cmml">∇</mo><msup id="A1.SS4.p2.3.m3.1.1.3.1.3" xref="A1.SS4.p2.3.m3.1.1.3.1.3.cmml"><mi id="A1.SS4.p2.3.m3.1.1.3.1.3.2" xref="A1.SS4.p2.3.m3.1.1.3.1.3.2.cmml">Z</mi><mi id="A1.SS4.p2.3.m3.1.1.3.1.3.3" xref="A1.SS4.p2.3.m3.1.1.3.1.3.3.cmml">k</mi></msup></msub><mi id="A1.SS4.p2.3.m3.1.1.3.2" xref="A1.SS4.p2.3.m3.1.1.3.2.cmml">l</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p2.3.m3.1b"><apply id="A1.SS4.p2.3.m3.1.1.cmml" xref="A1.SS4.p2.3.m3.1.1"><eq id="A1.SS4.p2.3.m3.1.1.1.cmml" xref="A1.SS4.p2.3.m3.1.1.1"></eq><ci id="A1.SS4.p2.3.m3.1.1.2.cmml" xref="A1.SS4.p2.3.m3.1.1.2">𝑉</ci><apply id="A1.SS4.p2.3.m3.1.1.3.cmml" xref="A1.SS4.p2.3.m3.1.1.3"><apply id="A1.SS4.p2.3.m3.1.1.3.1.cmml" xref="A1.SS4.p2.3.m3.1.1.3.1"><csymbol cd="ambiguous" id="A1.SS4.p2.3.m3.1.1.3.1.1.cmml" xref="A1.SS4.p2.3.m3.1.1.3.1">subscript</csymbol><ci id="A1.SS4.p2.3.m3.1.1.3.1.2.cmml" xref="A1.SS4.p2.3.m3.1.1.3.1.2">∇</ci><apply id="A1.SS4.p2.3.m3.1.1.3.1.3.cmml" xref="A1.SS4.p2.3.m3.1.1.3.1.3"><csymbol cd="ambiguous" id="A1.SS4.p2.3.m3.1.1.3.1.3.1.cmml" xref="A1.SS4.p2.3.m3.1.1.3.1.3">superscript</csymbol><ci id="A1.SS4.p2.3.m3.1.1.3.1.3.2.cmml" xref="A1.SS4.p2.3.m3.1.1.3.1.3.2">𝑍</ci><ci id="A1.SS4.p2.3.m3.1.1.3.1.3.3.cmml" xref="A1.SS4.p2.3.m3.1.1.3.1.3.3">𝑘</ci></apply></apply><ci id="A1.SS4.p2.3.m3.1.1.3.2.cmml" xref="A1.SS4.p2.3.m3.1.1.3.2">𝑙</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p2.3.m3.1c">V=\nabla_{Z^{k}}l</annotation></semantics></math>, and <math id="A1.SS4.p2.4.m4.1" class="ltx_Math" alttext="\hat{V}=W^{k}\bar{X}^{k}-\bar{Z}^{k}" display="inline"><semantics id="A1.SS4.p2.4.m4.1a"><mrow id="A1.SS4.p2.4.m4.1.1" xref="A1.SS4.p2.4.m4.1.1.cmml"><mover accent="true" id="A1.SS4.p2.4.m4.1.1.2" xref="A1.SS4.p2.4.m4.1.1.2.cmml"><mi id="A1.SS4.p2.4.m4.1.1.2.2" xref="A1.SS4.p2.4.m4.1.1.2.2.cmml">V</mi><mo id="A1.SS4.p2.4.m4.1.1.2.1" xref="A1.SS4.p2.4.m4.1.1.2.1.cmml">^</mo></mover><mo id="A1.SS4.p2.4.m4.1.1.1" xref="A1.SS4.p2.4.m4.1.1.1.cmml">=</mo><mrow id="A1.SS4.p2.4.m4.1.1.3" xref="A1.SS4.p2.4.m4.1.1.3.cmml"><mrow id="A1.SS4.p2.4.m4.1.1.3.2" xref="A1.SS4.p2.4.m4.1.1.3.2.cmml"><msup id="A1.SS4.p2.4.m4.1.1.3.2.2" xref="A1.SS4.p2.4.m4.1.1.3.2.2.cmml"><mi id="A1.SS4.p2.4.m4.1.1.3.2.2.2" xref="A1.SS4.p2.4.m4.1.1.3.2.2.2.cmml">W</mi><mi id="A1.SS4.p2.4.m4.1.1.3.2.2.3" xref="A1.SS4.p2.4.m4.1.1.3.2.2.3.cmml">k</mi></msup><mo lspace="0em" rspace="0em" id="A1.SS4.p2.4.m4.1.1.3.2.1" xref="A1.SS4.p2.4.m4.1.1.3.2.1.cmml">​</mo><msup id="A1.SS4.p2.4.m4.1.1.3.2.3" xref="A1.SS4.p2.4.m4.1.1.3.2.3.cmml"><mover accent="true" id="A1.SS4.p2.4.m4.1.1.3.2.3.2" xref="A1.SS4.p2.4.m4.1.1.3.2.3.2.cmml"><mi id="A1.SS4.p2.4.m4.1.1.3.2.3.2.2" xref="A1.SS4.p2.4.m4.1.1.3.2.3.2.2.cmml">X</mi><mo id="A1.SS4.p2.4.m4.1.1.3.2.3.2.1" xref="A1.SS4.p2.4.m4.1.1.3.2.3.2.1.cmml">¯</mo></mover><mi id="A1.SS4.p2.4.m4.1.1.3.2.3.3" xref="A1.SS4.p2.4.m4.1.1.3.2.3.3.cmml">k</mi></msup></mrow><mo id="A1.SS4.p2.4.m4.1.1.3.1" xref="A1.SS4.p2.4.m4.1.1.3.1.cmml">−</mo><msup id="A1.SS4.p2.4.m4.1.1.3.3" xref="A1.SS4.p2.4.m4.1.1.3.3.cmml"><mover accent="true" id="A1.SS4.p2.4.m4.1.1.3.3.2" xref="A1.SS4.p2.4.m4.1.1.3.3.2.cmml"><mi id="A1.SS4.p2.4.m4.1.1.3.3.2.2" xref="A1.SS4.p2.4.m4.1.1.3.3.2.2.cmml">Z</mi><mo id="A1.SS4.p2.4.m4.1.1.3.3.2.1" xref="A1.SS4.p2.4.m4.1.1.3.3.2.1.cmml">¯</mo></mover><mi id="A1.SS4.p2.4.m4.1.1.3.3.3" xref="A1.SS4.p2.4.m4.1.1.3.3.3.cmml">k</mi></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p2.4.m4.1b"><apply id="A1.SS4.p2.4.m4.1.1.cmml" xref="A1.SS4.p2.4.m4.1.1"><eq id="A1.SS4.p2.4.m4.1.1.1.cmml" xref="A1.SS4.p2.4.m4.1.1.1"></eq><apply id="A1.SS4.p2.4.m4.1.1.2.cmml" xref="A1.SS4.p2.4.m4.1.1.2"><ci id="A1.SS4.p2.4.m4.1.1.2.1.cmml" xref="A1.SS4.p2.4.m4.1.1.2.1">^</ci><ci id="A1.SS4.p2.4.m4.1.1.2.2.cmml" xref="A1.SS4.p2.4.m4.1.1.2.2">𝑉</ci></apply><apply id="A1.SS4.p2.4.m4.1.1.3.cmml" xref="A1.SS4.p2.4.m4.1.1.3"><minus id="A1.SS4.p2.4.m4.1.1.3.1.cmml" xref="A1.SS4.p2.4.m4.1.1.3.1"></minus><apply id="A1.SS4.p2.4.m4.1.1.3.2.cmml" xref="A1.SS4.p2.4.m4.1.1.3.2"><times id="A1.SS4.p2.4.m4.1.1.3.2.1.cmml" xref="A1.SS4.p2.4.m4.1.1.3.2.1"></times><apply id="A1.SS4.p2.4.m4.1.1.3.2.2.cmml" xref="A1.SS4.p2.4.m4.1.1.3.2.2"><csymbol cd="ambiguous" id="A1.SS4.p2.4.m4.1.1.3.2.2.1.cmml" xref="A1.SS4.p2.4.m4.1.1.3.2.2">superscript</csymbol><ci id="A1.SS4.p2.4.m4.1.1.3.2.2.2.cmml" xref="A1.SS4.p2.4.m4.1.1.3.2.2.2">𝑊</ci><ci id="A1.SS4.p2.4.m4.1.1.3.2.2.3.cmml" xref="A1.SS4.p2.4.m4.1.1.3.2.2.3">𝑘</ci></apply><apply id="A1.SS4.p2.4.m4.1.1.3.2.3.cmml" xref="A1.SS4.p2.4.m4.1.1.3.2.3"><csymbol cd="ambiguous" id="A1.SS4.p2.4.m4.1.1.3.2.3.1.cmml" xref="A1.SS4.p2.4.m4.1.1.3.2.3">superscript</csymbol><apply id="A1.SS4.p2.4.m4.1.1.3.2.3.2.cmml" xref="A1.SS4.p2.4.m4.1.1.3.2.3.2"><ci id="A1.SS4.p2.4.m4.1.1.3.2.3.2.1.cmml" xref="A1.SS4.p2.4.m4.1.1.3.2.3.2.1">¯</ci><ci id="A1.SS4.p2.4.m4.1.1.3.2.3.2.2.cmml" xref="A1.SS4.p2.4.m4.1.1.3.2.3.2.2">𝑋</ci></apply><ci id="A1.SS4.p2.4.m4.1.1.3.2.3.3.cmml" xref="A1.SS4.p2.4.m4.1.1.3.2.3.3">𝑘</ci></apply></apply><apply id="A1.SS4.p2.4.m4.1.1.3.3.cmml" xref="A1.SS4.p2.4.m4.1.1.3.3"><csymbol cd="ambiguous" id="A1.SS4.p2.4.m4.1.1.3.3.1.cmml" xref="A1.SS4.p2.4.m4.1.1.3.3">superscript</csymbol><apply id="A1.SS4.p2.4.m4.1.1.3.3.2.cmml" xref="A1.SS4.p2.4.m4.1.1.3.3.2"><ci id="A1.SS4.p2.4.m4.1.1.3.3.2.1.cmml" xref="A1.SS4.p2.4.m4.1.1.3.3.2.1">¯</ci><ci id="A1.SS4.p2.4.m4.1.1.3.3.2.2.cmml" xref="A1.SS4.p2.4.m4.1.1.3.3.2.2">𝑍</ci></apply><ci id="A1.SS4.p2.4.m4.1.1.3.3.3.cmml" xref="A1.SS4.p2.4.m4.1.1.3.3.3">𝑘</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p2.4.m4.1c">\hat{V}=W^{k}\bar{X}^{k}-\bar{Z}^{k}</annotation></semantics></math> into the fact above, we have shown the desired equality.</p>
</div>
<div id="A1.SS4.p3" class="ltx_para">
<p id="A1.SS4.p3.3" class="ltx_p">Note that the <math id="A1.SS4.p3.1.m1.1" class="ltx_Math" alttext="\sigma_{k}" display="inline"><semantics id="A1.SS4.p3.1.m1.1a"><msub id="A1.SS4.p3.1.m1.1.1" xref="A1.SS4.p3.1.m1.1.1.cmml"><mi id="A1.SS4.p3.1.m1.1.1.2" xref="A1.SS4.p3.1.m1.1.1.2.cmml">σ</mi><mi id="A1.SS4.p3.1.m1.1.1.3" xref="A1.SS4.p3.1.m1.1.1.3.cmml">k</mi></msub><annotation-xml encoding="MathML-Content" id="A1.SS4.p3.1.m1.1b"><apply id="A1.SS4.p3.1.m1.1.1.cmml" xref="A1.SS4.p3.1.m1.1.1"><csymbol cd="ambiguous" id="A1.SS4.p3.1.m1.1.1.1.cmml" xref="A1.SS4.p3.1.m1.1.1">subscript</csymbol><ci id="A1.SS4.p3.1.m1.1.1.2.cmml" xref="A1.SS4.p3.1.m1.1.1.2">𝜎</ci><ci id="A1.SS4.p3.1.m1.1.1.3.cmml" xref="A1.SS4.p3.1.m1.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p3.1.m1.1c">\sigma_{k}</annotation></semantics></math> and <math id="A1.SS4.p3.2.m2.1" class="ltx_Math" alttext="\sigma^{\prime}_{k}" display="inline"><semantics id="A1.SS4.p3.2.m2.1a"><msubsup id="A1.SS4.p3.2.m2.1.1" xref="A1.SS4.p3.2.m2.1.1.cmml"><mi id="A1.SS4.p3.2.m2.1.1.2.2" xref="A1.SS4.p3.2.m2.1.1.2.2.cmml">σ</mi><mi id="A1.SS4.p3.2.m2.1.1.3" xref="A1.SS4.p3.2.m2.1.1.3.cmml">k</mi><mo id="A1.SS4.p3.2.m2.1.1.2.3" xref="A1.SS4.p3.2.m2.1.1.2.3.cmml">′</mo></msubsup><annotation-xml encoding="MathML-Content" id="A1.SS4.p3.2.m2.1b"><apply id="A1.SS4.p3.2.m2.1.1.cmml" xref="A1.SS4.p3.2.m2.1.1"><csymbol cd="ambiguous" id="A1.SS4.p3.2.m2.1.1.1.cmml" xref="A1.SS4.p3.2.m2.1.1">subscript</csymbol><apply id="A1.SS4.p3.2.m2.1.1.2.cmml" xref="A1.SS4.p3.2.m2.1.1"><csymbol cd="ambiguous" id="A1.SS4.p3.2.m2.1.1.2.1.cmml" xref="A1.SS4.p3.2.m2.1.1">superscript</csymbol><ci id="A1.SS4.p3.2.m2.1.1.2.2.cmml" xref="A1.SS4.p3.2.m2.1.1.2.2">𝜎</ci><ci id="A1.SS4.p3.2.m2.1.1.2.3.cmml" xref="A1.SS4.p3.2.m2.1.1.2.3">′</ci></apply><ci id="A1.SS4.p3.2.m2.1.1.3.cmml" xref="A1.SS4.p3.2.m2.1.1.3">𝑘</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p3.2.m2.1c">\sigma^{\prime}_{k}</annotation></semantics></math> used above can be extended to arbitrary functions that are not necessarily element-wise operations, including normalization layers.
This extension can be achieved through, for example, <span id="A1.SS4.p3.3.1" class="ltx_text ltx_font_typewriter">vjp</span> (vector-Jacobian product) in standard libraries for automatic differentiation such as JAX and PyTorch.
However, the dual form cannot accelerate operations inside <math id="A1.SS4.p3.3.m3.1" class="ltx_Math" alttext="\sigma" display="inline"><semantics id="A1.SS4.p3.3.m3.1a"><mi id="A1.SS4.p3.3.m3.1.1" xref="A1.SS4.p3.3.m3.1.1.cmml">σ</mi><annotation-xml encoding="MathML-Content" id="A1.SS4.p3.3.m3.1b"><ci id="A1.SS4.p3.3.m3.1.1.cmml" xref="A1.SS4.p3.3.m3.1.1">𝜎</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p3.3.m3.1c">\sigma</annotation></semantics></math> or its <span id="A1.SS4.p3.3.2" class="ltx_text ltx_font_typewriter">vjp</span>.</p>
<div class="ltx_pagination ltx_role_newpage"></div>
</div>
</section>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Nadaraya-Watson estimator</h2>

<section id="A2.SS0.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Derivation for the Nadaraya-Watson estimator.</h5>

<div id="A2.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="A2.SS0.SSS0.Px1.p1.4" class="ltx_p">Throughout this section, we use <math id="A2.SS0.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="\mathbf{x}" display="inline"><semantics id="A2.SS0.SSS0.Px1.p1.1.m1.1a"><mi id="A2.SS0.SSS0.Px1.p1.1.m1.1.1" xref="A2.SS0.SSS0.Px1.p1.1.m1.1.1.cmml">𝐱</mi><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px1.p1.1.m1.1b"><ci id="A2.SS0.SSS0.Px1.p1.1.m1.1.1.cmml" xref="A2.SS0.SSS0.Px1.p1.1.m1.1.1">𝐱</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px1.p1.1.m1.1c">\mathbf{x}</annotation></semantics></math> to denote the input token <math id="A2.SS0.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="x" display="inline"><semantics id="A2.SS0.SSS0.Px1.p1.2.m2.1a"><mi id="A2.SS0.SSS0.Px1.p1.2.m2.1.1" xref="A2.SS0.SSS0.Px1.p1.2.m2.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px1.p1.2.m2.1b"><ci id="A2.SS0.SSS0.Px1.p1.2.m2.1.1.cmml" xref="A2.SS0.SSS0.Px1.p1.2.m2.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px1.p1.2.m2.1c">x</annotation></semantics></math> as a random variable.
Our desired output is the corresponding output token, another random variable <math id="A2.SS0.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="\mathbf{z}" display="inline"><semantics id="A2.SS0.SSS0.Px1.p1.3.m3.1a"><mi id="A2.SS0.SSS0.Px1.p1.3.m3.1.1" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1.cmml">𝐳</mi><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px1.p1.3.m3.1b"><ci id="A2.SS0.SSS0.Px1.p1.3.m3.1.1.cmml" xref="A2.SS0.SSS0.Px1.p1.3.m3.1.1">𝐳</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px1.p1.3.m3.1c">\mathbf{z}</annotation></semantics></math>.
This is formulated as estimating the conditional expectation of <math id="A2.SS0.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="\mathbf{z}" display="inline"><semantics id="A2.SS0.SSS0.Px1.p1.4.m4.1a"><mi id="A2.SS0.SSS0.Px1.p1.4.m4.1.1" xref="A2.SS0.SSS0.Px1.p1.4.m4.1.1.cmml">𝐳</mi><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px1.p1.4.m4.1b"><ci id="A2.SS0.SSS0.Px1.p1.4.m4.1.1.cmml" xref="A2.SS0.SSS0.Px1.p1.4.m4.1.1">𝐳</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px1.p1.4.m4.1c">\mathbf{z}</annotation></semantics></math>:</p>
<table id="A2.Ex11" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A2.Ex11.m1.4" class="ltx_Math" alttext="\mathbb{E}[\mathbf{z}|\mathbf{x}=x]=\int p(z|x)~{}z~{}dz=\int\frac{p(x,z)}{p(x)}~{}z~{}dz." display="block"><semantics id="A2.Ex11.m1.4a"><mrow id="A2.Ex11.m1.4.4.1" xref="A2.Ex11.m1.4.4.1.1.cmml"><mrow id="A2.Ex11.m1.4.4.1.1" xref="A2.Ex11.m1.4.4.1.1.cmml"><mrow id="A2.Ex11.m1.4.4.1.1.1" xref="A2.Ex11.m1.4.4.1.1.1.cmml"><mi id="A2.Ex11.m1.4.4.1.1.1.3" xref="A2.Ex11.m1.4.4.1.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="A2.Ex11.m1.4.4.1.1.1.2" xref="A2.Ex11.m1.4.4.1.1.1.2.cmml">​</mo><mrow id="A2.Ex11.m1.4.4.1.1.1.1.1" xref="A2.Ex11.m1.4.4.1.1.1.1.2.cmml"><mo stretchy="false" id="A2.Ex11.m1.4.4.1.1.1.1.1.2" xref="A2.Ex11.m1.4.4.1.1.1.1.2.1.cmml">[</mo><mrow id="A2.Ex11.m1.4.4.1.1.1.1.1.1" xref="A2.Ex11.m1.4.4.1.1.1.1.1.1.cmml"><mrow id="A2.Ex11.m1.4.4.1.1.1.1.1.1.2" xref="A2.Ex11.m1.4.4.1.1.1.1.1.1.2.cmml"><mi id="A2.Ex11.m1.4.4.1.1.1.1.1.1.2.2" xref="A2.Ex11.m1.4.4.1.1.1.1.1.1.2.2.cmml">𝐳</mi><mo fence="false" id="A2.Ex11.m1.4.4.1.1.1.1.1.1.2.1" xref="A2.Ex11.m1.4.4.1.1.1.1.1.1.2.1.cmml">|</mo><mi id="A2.Ex11.m1.4.4.1.1.1.1.1.1.2.3" xref="A2.Ex11.m1.4.4.1.1.1.1.1.1.2.3.cmml">𝐱</mi></mrow><mo id="A2.Ex11.m1.4.4.1.1.1.1.1.1.1" xref="A2.Ex11.m1.4.4.1.1.1.1.1.1.1.cmml">=</mo><mi id="A2.Ex11.m1.4.4.1.1.1.1.1.1.3" xref="A2.Ex11.m1.4.4.1.1.1.1.1.1.3.cmml">x</mi></mrow><mo stretchy="false" id="A2.Ex11.m1.4.4.1.1.1.1.1.3" xref="A2.Ex11.m1.4.4.1.1.1.1.2.1.cmml">]</mo></mrow></mrow><mo rspace="0.111em" id="A2.Ex11.m1.4.4.1.1.4" xref="A2.Ex11.m1.4.4.1.1.4.cmml">=</mo><mrow id="A2.Ex11.m1.4.4.1.1.2" xref="A2.Ex11.m1.4.4.1.1.2.cmml"><mo id="A2.Ex11.m1.4.4.1.1.2.2" xref="A2.Ex11.m1.4.4.1.1.2.2.cmml">∫</mo><mrow id="A2.Ex11.m1.4.4.1.1.2.1" xref="A2.Ex11.m1.4.4.1.1.2.1.cmml"><mi id="A2.Ex11.m1.4.4.1.1.2.1.3" xref="A2.Ex11.m1.4.4.1.1.2.1.3.cmml">p</mi><mo lspace="0em" rspace="0em" id="A2.Ex11.m1.4.4.1.1.2.1.2" xref="A2.Ex11.m1.4.4.1.1.2.1.2.cmml">​</mo><mrow id="A2.Ex11.m1.4.4.1.1.2.1.1.1" xref="A2.Ex11.m1.4.4.1.1.2.1.1.1.1.cmml"><mo stretchy="false" id="A2.Ex11.m1.4.4.1.1.2.1.1.1.2" xref="A2.Ex11.m1.4.4.1.1.2.1.1.1.1.cmml">(</mo><mrow id="A2.Ex11.m1.4.4.1.1.2.1.1.1.1" xref="A2.Ex11.m1.4.4.1.1.2.1.1.1.1.cmml"><mi id="A2.Ex11.m1.4.4.1.1.2.1.1.1.1.2" xref="A2.Ex11.m1.4.4.1.1.2.1.1.1.1.2.cmml">z</mi><mo fence="false" id="A2.Ex11.m1.4.4.1.1.2.1.1.1.1.1" xref="A2.Ex11.m1.4.4.1.1.2.1.1.1.1.1.cmml">|</mo><mi id="A2.Ex11.m1.4.4.1.1.2.1.1.1.1.3" xref="A2.Ex11.m1.4.4.1.1.2.1.1.1.1.3.cmml">x</mi></mrow><mo stretchy="false" id="A2.Ex11.m1.4.4.1.1.2.1.1.1.3" xref="A2.Ex11.m1.4.4.1.1.2.1.1.1.1.cmml">)</mo></mrow><mo lspace="0.330em" rspace="0em" id="A2.Ex11.m1.4.4.1.1.2.1.2a" xref="A2.Ex11.m1.4.4.1.1.2.1.2.cmml">​</mo><mi id="A2.Ex11.m1.4.4.1.1.2.1.4" xref="A2.Ex11.m1.4.4.1.1.2.1.4.cmml">z</mi><mo lspace="0.330em" rspace="0em" id="A2.Ex11.m1.4.4.1.1.2.1.2b" xref="A2.Ex11.m1.4.4.1.1.2.1.2.cmml">​</mo><mrow id="A2.Ex11.m1.4.4.1.1.2.1.5" xref="A2.Ex11.m1.4.4.1.1.2.1.5.cmml"><mo rspace="0em" id="A2.Ex11.m1.4.4.1.1.2.1.5.1" xref="A2.Ex11.m1.4.4.1.1.2.1.5.1.cmml">𝑑</mo><mi id="A2.Ex11.m1.4.4.1.1.2.1.5.2" xref="A2.Ex11.m1.4.4.1.1.2.1.5.2.cmml">z</mi></mrow></mrow></mrow><mo rspace="0.111em" id="A2.Ex11.m1.4.4.1.1.5" xref="A2.Ex11.m1.4.4.1.1.5.cmml">=</mo><mrow id="A2.Ex11.m1.4.4.1.1.6" xref="A2.Ex11.m1.4.4.1.1.6.cmml"><mo id="A2.Ex11.m1.4.4.1.1.6.1" xref="A2.Ex11.m1.4.4.1.1.6.1.cmml">∫</mo><mrow id="A2.Ex11.m1.4.4.1.1.6.2" xref="A2.Ex11.m1.4.4.1.1.6.2.cmml"><mfrac id="A2.Ex11.m1.3.3" xref="A2.Ex11.m1.3.3.cmml"><mrow id="A2.Ex11.m1.2.2.2" xref="A2.Ex11.m1.2.2.2.cmml"><mi id="A2.Ex11.m1.2.2.2.4" xref="A2.Ex11.m1.2.2.2.4.cmml">p</mi><mo lspace="0em" rspace="0em" id="A2.Ex11.m1.2.2.2.3" xref="A2.Ex11.m1.2.2.2.3.cmml">​</mo><mrow id="A2.Ex11.m1.2.2.2.5.2" xref="A2.Ex11.m1.2.2.2.5.1.cmml"><mo stretchy="false" id="A2.Ex11.m1.2.2.2.5.2.1" xref="A2.Ex11.m1.2.2.2.5.1.cmml">(</mo><mi id="A2.Ex11.m1.1.1.1.1" xref="A2.Ex11.m1.1.1.1.1.cmml">x</mi><mo id="A2.Ex11.m1.2.2.2.5.2.2" xref="A2.Ex11.m1.2.2.2.5.1.cmml">,</mo><mi id="A2.Ex11.m1.2.2.2.2" xref="A2.Ex11.m1.2.2.2.2.cmml">z</mi><mo stretchy="false" id="A2.Ex11.m1.2.2.2.5.2.3" xref="A2.Ex11.m1.2.2.2.5.1.cmml">)</mo></mrow></mrow><mrow id="A2.Ex11.m1.3.3.3" xref="A2.Ex11.m1.3.3.3.cmml"><mi id="A2.Ex11.m1.3.3.3.3" xref="A2.Ex11.m1.3.3.3.3.cmml">p</mi><mo lspace="0em" rspace="0em" id="A2.Ex11.m1.3.3.3.2" xref="A2.Ex11.m1.3.3.3.2.cmml">​</mo><mrow id="A2.Ex11.m1.3.3.3.4.2" xref="A2.Ex11.m1.3.3.3.cmml"><mo stretchy="false" id="A2.Ex11.m1.3.3.3.4.2.1" xref="A2.Ex11.m1.3.3.3.cmml">(</mo><mi id="A2.Ex11.m1.3.3.3.1" xref="A2.Ex11.m1.3.3.3.1.cmml">x</mi><mo stretchy="false" id="A2.Ex11.m1.3.3.3.4.2.2" xref="A2.Ex11.m1.3.3.3.cmml">)</mo></mrow></mrow></mfrac><mo lspace="0.330em" rspace="0em" id="A2.Ex11.m1.4.4.1.1.6.2.1" xref="A2.Ex11.m1.4.4.1.1.6.2.1.cmml">​</mo><mi id="A2.Ex11.m1.4.4.1.1.6.2.2" xref="A2.Ex11.m1.4.4.1.1.6.2.2.cmml">z</mi><mo lspace="0.330em" rspace="0em" id="A2.Ex11.m1.4.4.1.1.6.2.1a" xref="A2.Ex11.m1.4.4.1.1.6.2.1.cmml">​</mo><mrow id="A2.Ex11.m1.4.4.1.1.6.2.3" xref="A2.Ex11.m1.4.4.1.1.6.2.3.cmml"><mo rspace="0em" id="A2.Ex11.m1.4.4.1.1.6.2.3.1" xref="A2.Ex11.m1.4.4.1.1.6.2.3.1.cmml">𝑑</mo><mi id="A2.Ex11.m1.4.4.1.1.6.2.3.2" xref="A2.Ex11.m1.4.4.1.1.6.2.3.2.cmml">z</mi></mrow></mrow></mrow></mrow><mo lspace="0em" id="A2.Ex11.m1.4.4.1.2" xref="A2.Ex11.m1.4.4.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A2.Ex11.m1.4b"><apply id="A2.Ex11.m1.4.4.1.1.cmml" xref="A2.Ex11.m1.4.4.1"><and id="A2.Ex11.m1.4.4.1.1a.cmml" xref="A2.Ex11.m1.4.4.1"></and><apply id="A2.Ex11.m1.4.4.1.1b.cmml" xref="A2.Ex11.m1.4.4.1"><eq id="A2.Ex11.m1.4.4.1.1.4.cmml" xref="A2.Ex11.m1.4.4.1.1.4"></eq><apply id="A2.Ex11.m1.4.4.1.1.1.cmml" xref="A2.Ex11.m1.4.4.1.1.1"><times id="A2.Ex11.m1.4.4.1.1.1.2.cmml" xref="A2.Ex11.m1.4.4.1.1.1.2"></times><ci id="A2.Ex11.m1.4.4.1.1.1.3.cmml" xref="A2.Ex11.m1.4.4.1.1.1.3">𝔼</ci><apply id="A2.Ex11.m1.4.4.1.1.1.1.2.cmml" xref="A2.Ex11.m1.4.4.1.1.1.1.1"><csymbol cd="latexml" id="A2.Ex11.m1.4.4.1.1.1.1.2.1.cmml" xref="A2.Ex11.m1.4.4.1.1.1.1.1.2">delimited-[]</csymbol><apply id="A2.Ex11.m1.4.4.1.1.1.1.1.1.cmml" xref="A2.Ex11.m1.4.4.1.1.1.1.1.1"><eq id="A2.Ex11.m1.4.4.1.1.1.1.1.1.1.cmml" xref="A2.Ex11.m1.4.4.1.1.1.1.1.1.1"></eq><apply id="A2.Ex11.m1.4.4.1.1.1.1.1.1.2.cmml" xref="A2.Ex11.m1.4.4.1.1.1.1.1.1.2"><csymbol cd="latexml" id="A2.Ex11.m1.4.4.1.1.1.1.1.1.2.1.cmml" xref="A2.Ex11.m1.4.4.1.1.1.1.1.1.2.1">conditional</csymbol><ci id="A2.Ex11.m1.4.4.1.1.1.1.1.1.2.2.cmml" xref="A2.Ex11.m1.4.4.1.1.1.1.1.1.2.2">𝐳</ci><ci id="A2.Ex11.m1.4.4.1.1.1.1.1.1.2.3.cmml" xref="A2.Ex11.m1.4.4.1.1.1.1.1.1.2.3">𝐱</ci></apply><ci id="A2.Ex11.m1.4.4.1.1.1.1.1.1.3.cmml" xref="A2.Ex11.m1.4.4.1.1.1.1.1.1.3">𝑥</ci></apply></apply></apply><apply id="A2.Ex11.m1.4.4.1.1.2.cmml" xref="A2.Ex11.m1.4.4.1.1.2"><int id="A2.Ex11.m1.4.4.1.1.2.2.cmml" xref="A2.Ex11.m1.4.4.1.1.2.2"></int><apply id="A2.Ex11.m1.4.4.1.1.2.1.cmml" xref="A2.Ex11.m1.4.4.1.1.2.1"><times id="A2.Ex11.m1.4.4.1.1.2.1.2.cmml" xref="A2.Ex11.m1.4.4.1.1.2.1.2"></times><ci id="A2.Ex11.m1.4.4.1.1.2.1.3.cmml" xref="A2.Ex11.m1.4.4.1.1.2.1.3">𝑝</ci><apply id="A2.Ex11.m1.4.4.1.1.2.1.1.1.1.cmml" xref="A2.Ex11.m1.4.4.1.1.2.1.1.1"><csymbol cd="latexml" id="A2.Ex11.m1.4.4.1.1.2.1.1.1.1.1.cmml" xref="A2.Ex11.m1.4.4.1.1.2.1.1.1.1.1">conditional</csymbol><ci id="A2.Ex11.m1.4.4.1.1.2.1.1.1.1.2.cmml" xref="A2.Ex11.m1.4.4.1.1.2.1.1.1.1.2">𝑧</ci><ci id="A2.Ex11.m1.4.4.1.1.2.1.1.1.1.3.cmml" xref="A2.Ex11.m1.4.4.1.1.2.1.1.1.1.3">𝑥</ci></apply><ci id="A2.Ex11.m1.4.4.1.1.2.1.4.cmml" xref="A2.Ex11.m1.4.4.1.1.2.1.4">𝑧</ci><apply id="A2.Ex11.m1.4.4.1.1.2.1.5.cmml" xref="A2.Ex11.m1.4.4.1.1.2.1.5"><csymbol cd="latexml" id="A2.Ex11.m1.4.4.1.1.2.1.5.1.cmml" xref="A2.Ex11.m1.4.4.1.1.2.1.5.1">differential-d</csymbol><ci id="A2.Ex11.m1.4.4.1.1.2.1.5.2.cmml" xref="A2.Ex11.m1.4.4.1.1.2.1.5.2">𝑧</ci></apply></apply></apply></apply><apply id="A2.Ex11.m1.4.4.1.1c.cmml" xref="A2.Ex11.m1.4.4.1"><eq id="A2.Ex11.m1.4.4.1.1.5.cmml" xref="A2.Ex11.m1.4.4.1.1.5"></eq><share href="#A2.Ex11.m1.4.4.1.1.2.cmml" id="A2.Ex11.m1.4.4.1.1d.cmml" xref="A2.Ex11.m1.4.4.1"></share><apply id="A2.Ex11.m1.4.4.1.1.6.cmml" xref="A2.Ex11.m1.4.4.1.1.6"><int id="A2.Ex11.m1.4.4.1.1.6.1.cmml" xref="A2.Ex11.m1.4.4.1.1.6.1"></int><apply id="A2.Ex11.m1.4.4.1.1.6.2.cmml" xref="A2.Ex11.m1.4.4.1.1.6.2"><times id="A2.Ex11.m1.4.4.1.1.6.2.1.cmml" xref="A2.Ex11.m1.4.4.1.1.6.2.1"></times><apply id="A2.Ex11.m1.3.3.cmml" xref="A2.Ex11.m1.3.3"><divide id="A2.Ex11.m1.3.3.4.cmml" xref="A2.Ex11.m1.3.3"></divide><apply id="A2.Ex11.m1.2.2.2.cmml" xref="A2.Ex11.m1.2.2.2"><times id="A2.Ex11.m1.2.2.2.3.cmml" xref="A2.Ex11.m1.2.2.2.3"></times><ci id="A2.Ex11.m1.2.2.2.4.cmml" xref="A2.Ex11.m1.2.2.2.4">𝑝</ci><interval closure="open" id="A2.Ex11.m1.2.2.2.5.1.cmml" xref="A2.Ex11.m1.2.2.2.5.2"><ci id="A2.Ex11.m1.1.1.1.1.cmml" xref="A2.Ex11.m1.1.1.1.1">𝑥</ci><ci id="A2.Ex11.m1.2.2.2.2.cmml" xref="A2.Ex11.m1.2.2.2.2">𝑧</ci></interval></apply><apply id="A2.Ex11.m1.3.3.3.cmml" xref="A2.Ex11.m1.3.3.3"><times id="A2.Ex11.m1.3.3.3.2.cmml" xref="A2.Ex11.m1.3.3.3.2"></times><ci id="A2.Ex11.m1.3.3.3.3.cmml" xref="A2.Ex11.m1.3.3.3.3">𝑝</ci><ci id="A2.Ex11.m1.3.3.3.1.cmml" xref="A2.Ex11.m1.3.3.3.1">𝑥</ci></apply></apply><ci id="A2.Ex11.m1.4.4.1.1.6.2.2.cmml" xref="A2.Ex11.m1.4.4.1.1.6.2.2">𝑧</ci><apply id="A2.Ex11.m1.4.4.1.1.6.2.3.cmml" xref="A2.Ex11.m1.4.4.1.1.6.2.3"><csymbol cd="latexml" id="A2.Ex11.m1.4.4.1.1.6.2.3.1.cmml" xref="A2.Ex11.m1.4.4.1.1.6.2.3.1">differential-d</csymbol><ci id="A2.Ex11.m1.4.4.1.1.6.2.3.2.cmml" xref="A2.Ex11.m1.4.4.1.1.6.2.3.2">𝑧</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.Ex11.m1.4c">\mathbb{E}[\mathbf{z}|\mathbf{x}=x]=\int p(z|x)~{}z~{}dz=\int\frac{p(x,z)}{p(x)}~{}z~{}dz.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="A2.SS0.SSS0.Px1.p1.7" class="ltx_p">Since the true probability distributions <math id="A2.SS0.SSS0.Px1.p1.5.m1.1" class="ltx_Math" alttext="p(x)" display="inline"><semantics id="A2.SS0.SSS0.Px1.p1.5.m1.1a"><mrow id="A2.SS0.SSS0.Px1.p1.5.m1.1.2" xref="A2.SS0.SSS0.Px1.p1.5.m1.1.2.cmml"><mi id="A2.SS0.SSS0.Px1.p1.5.m1.1.2.2" xref="A2.SS0.SSS0.Px1.p1.5.m1.1.2.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="A2.SS0.SSS0.Px1.p1.5.m1.1.2.1" xref="A2.SS0.SSS0.Px1.p1.5.m1.1.2.1.cmml">​</mo><mrow id="A2.SS0.SSS0.Px1.p1.5.m1.1.2.3.2" xref="A2.SS0.SSS0.Px1.p1.5.m1.1.2.cmml"><mo stretchy="false" id="A2.SS0.SSS0.Px1.p1.5.m1.1.2.3.2.1" xref="A2.SS0.SSS0.Px1.p1.5.m1.1.2.cmml">(</mo><mi id="A2.SS0.SSS0.Px1.p1.5.m1.1.1" xref="A2.SS0.SSS0.Px1.p1.5.m1.1.1.cmml">x</mi><mo stretchy="false" id="A2.SS0.SSS0.Px1.p1.5.m1.1.2.3.2.2" xref="A2.SS0.SSS0.Px1.p1.5.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px1.p1.5.m1.1b"><apply id="A2.SS0.SSS0.Px1.p1.5.m1.1.2.cmml" xref="A2.SS0.SSS0.Px1.p1.5.m1.1.2"><times id="A2.SS0.SSS0.Px1.p1.5.m1.1.2.1.cmml" xref="A2.SS0.SSS0.Px1.p1.5.m1.1.2.1"></times><ci id="A2.SS0.SSS0.Px1.p1.5.m1.1.2.2.cmml" xref="A2.SS0.SSS0.Px1.p1.5.m1.1.2.2">𝑝</ci><ci id="A2.SS0.SSS0.Px1.p1.5.m1.1.1.cmml" xref="A2.SS0.SSS0.Px1.p1.5.m1.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px1.p1.5.m1.1c">p(x)</annotation></semantics></math> and <math id="A2.SS0.SSS0.Px1.p1.6.m2.2" class="ltx_Math" alttext="p(x,z)" display="inline"><semantics id="A2.SS0.SSS0.Px1.p1.6.m2.2a"><mrow id="A2.SS0.SSS0.Px1.p1.6.m2.2.3" xref="A2.SS0.SSS0.Px1.p1.6.m2.2.3.cmml"><mi id="A2.SS0.SSS0.Px1.p1.6.m2.2.3.2" xref="A2.SS0.SSS0.Px1.p1.6.m2.2.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="A2.SS0.SSS0.Px1.p1.6.m2.2.3.1" xref="A2.SS0.SSS0.Px1.p1.6.m2.2.3.1.cmml">​</mo><mrow id="A2.SS0.SSS0.Px1.p1.6.m2.2.3.3.2" xref="A2.SS0.SSS0.Px1.p1.6.m2.2.3.3.1.cmml"><mo stretchy="false" id="A2.SS0.SSS0.Px1.p1.6.m2.2.3.3.2.1" xref="A2.SS0.SSS0.Px1.p1.6.m2.2.3.3.1.cmml">(</mo><mi id="A2.SS0.SSS0.Px1.p1.6.m2.1.1" xref="A2.SS0.SSS0.Px1.p1.6.m2.1.1.cmml">x</mi><mo id="A2.SS0.SSS0.Px1.p1.6.m2.2.3.3.2.2" xref="A2.SS0.SSS0.Px1.p1.6.m2.2.3.3.1.cmml">,</mo><mi id="A2.SS0.SSS0.Px1.p1.6.m2.2.2" xref="A2.SS0.SSS0.Px1.p1.6.m2.2.2.cmml">z</mi><mo stretchy="false" id="A2.SS0.SSS0.Px1.p1.6.m2.2.3.3.2.3" xref="A2.SS0.SSS0.Px1.p1.6.m2.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px1.p1.6.m2.2b"><apply id="A2.SS0.SSS0.Px1.p1.6.m2.2.3.cmml" xref="A2.SS0.SSS0.Px1.p1.6.m2.2.3"><times id="A2.SS0.SSS0.Px1.p1.6.m2.2.3.1.cmml" xref="A2.SS0.SSS0.Px1.p1.6.m2.2.3.1"></times><ci id="A2.SS0.SSS0.Px1.p1.6.m2.2.3.2.cmml" xref="A2.SS0.SSS0.Px1.p1.6.m2.2.3.2">𝑝</ci><interval closure="open" id="A2.SS0.SSS0.Px1.p1.6.m2.2.3.3.1.cmml" xref="A2.SS0.SSS0.Px1.p1.6.m2.2.3.3.2"><ci id="A2.SS0.SSS0.Px1.p1.6.m2.1.1.cmml" xref="A2.SS0.SSS0.Px1.p1.6.m2.1.1">𝑥</ci><ci id="A2.SS0.SSS0.Px1.p1.6.m2.2.2.cmml" xref="A2.SS0.SSS0.Px1.p1.6.m2.2.2">𝑧</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px1.p1.6.m2.2c">p(x,z)</annotation></semantics></math> are unknown, we replace them with their kernel density estimations.
Specifically, the kernel density estimation for <math id="A2.SS0.SSS0.Px1.p1.7.m3.1" class="ltx_Math" alttext="p(x)" display="inline"><semantics id="A2.SS0.SSS0.Px1.p1.7.m3.1a"><mrow id="A2.SS0.SSS0.Px1.p1.7.m3.1.2" xref="A2.SS0.SSS0.Px1.p1.7.m3.1.2.cmml"><mi id="A2.SS0.SSS0.Px1.p1.7.m3.1.2.2" xref="A2.SS0.SSS0.Px1.p1.7.m3.1.2.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="A2.SS0.SSS0.Px1.p1.7.m3.1.2.1" xref="A2.SS0.SSS0.Px1.p1.7.m3.1.2.1.cmml">​</mo><mrow id="A2.SS0.SSS0.Px1.p1.7.m3.1.2.3.2" xref="A2.SS0.SSS0.Px1.p1.7.m3.1.2.cmml"><mo stretchy="false" id="A2.SS0.SSS0.Px1.p1.7.m3.1.2.3.2.1" xref="A2.SS0.SSS0.Px1.p1.7.m3.1.2.cmml">(</mo><mi id="A2.SS0.SSS0.Px1.p1.7.m3.1.1" xref="A2.SS0.SSS0.Px1.p1.7.m3.1.1.cmml">x</mi><mo stretchy="false" id="A2.SS0.SSS0.Px1.p1.7.m3.1.2.3.2.2" xref="A2.SS0.SSS0.Px1.p1.7.m3.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px1.p1.7.m3.1b"><apply id="A2.SS0.SSS0.Px1.p1.7.m3.1.2.cmml" xref="A2.SS0.SSS0.Px1.p1.7.m3.1.2"><times id="A2.SS0.SSS0.Px1.p1.7.m3.1.2.1.cmml" xref="A2.SS0.SSS0.Px1.p1.7.m3.1.2.1"></times><ci id="A2.SS0.SSS0.Px1.p1.7.m3.1.2.2.cmml" xref="A2.SS0.SSS0.Px1.p1.7.m3.1.2.2">𝑝</ci><ci id="A2.SS0.SSS0.Px1.p1.7.m3.1.1.cmml" xref="A2.SS0.SSS0.Px1.p1.7.m3.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px1.p1.7.m3.1c">p(x)</annotation></semantics></math> is:</p>
<table id="A2.Ex12" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A2.Ex12.m1.3" class="ltx_Math" alttext="\hat{p}(x)=\frac{1}{n}\sum_{i=1}^{n}\kappa(x,x_{i})," display="block"><semantics id="A2.Ex12.m1.3a"><mrow id="A2.Ex12.m1.3.3.1" xref="A2.Ex12.m1.3.3.1.1.cmml"><mrow id="A2.Ex12.m1.3.3.1.1" xref="A2.Ex12.m1.3.3.1.1.cmml"><mrow id="A2.Ex12.m1.3.3.1.1.3" xref="A2.Ex12.m1.3.3.1.1.3.cmml"><mover accent="true" id="A2.Ex12.m1.3.3.1.1.3.2" xref="A2.Ex12.m1.3.3.1.1.3.2.cmml"><mi id="A2.Ex12.m1.3.3.1.1.3.2.2" xref="A2.Ex12.m1.3.3.1.1.3.2.2.cmml">p</mi><mo id="A2.Ex12.m1.3.3.1.1.3.2.1" xref="A2.Ex12.m1.3.3.1.1.3.2.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A2.Ex12.m1.3.3.1.1.3.1" xref="A2.Ex12.m1.3.3.1.1.3.1.cmml">​</mo><mrow id="A2.Ex12.m1.3.3.1.1.3.3.2" xref="A2.Ex12.m1.3.3.1.1.3.cmml"><mo stretchy="false" id="A2.Ex12.m1.3.3.1.1.3.3.2.1" xref="A2.Ex12.m1.3.3.1.1.3.cmml">(</mo><mi id="A2.Ex12.m1.1.1" xref="A2.Ex12.m1.1.1.cmml">x</mi><mo stretchy="false" id="A2.Ex12.m1.3.3.1.1.3.3.2.2" xref="A2.Ex12.m1.3.3.1.1.3.cmml">)</mo></mrow></mrow><mo id="A2.Ex12.m1.3.3.1.1.2" xref="A2.Ex12.m1.3.3.1.1.2.cmml">=</mo><mrow id="A2.Ex12.m1.3.3.1.1.1" xref="A2.Ex12.m1.3.3.1.1.1.cmml"><mfrac id="A2.Ex12.m1.3.3.1.1.1.3" xref="A2.Ex12.m1.3.3.1.1.1.3.cmml"><mn id="A2.Ex12.m1.3.3.1.1.1.3.2" xref="A2.Ex12.m1.3.3.1.1.1.3.2.cmml">1</mn><mi id="A2.Ex12.m1.3.3.1.1.1.3.3" xref="A2.Ex12.m1.3.3.1.1.1.3.3.cmml">n</mi></mfrac><mo lspace="0em" rspace="0em" id="A2.Ex12.m1.3.3.1.1.1.2" xref="A2.Ex12.m1.3.3.1.1.1.2.cmml">​</mo><mrow id="A2.Ex12.m1.3.3.1.1.1.1" xref="A2.Ex12.m1.3.3.1.1.1.1.cmml"><munderover id="A2.Ex12.m1.3.3.1.1.1.1.2" xref="A2.Ex12.m1.3.3.1.1.1.1.2.cmml"><mo movablelimits="false" id="A2.Ex12.m1.3.3.1.1.1.1.2.2.2" xref="A2.Ex12.m1.3.3.1.1.1.1.2.2.2.cmml">∑</mo><mrow id="A2.Ex12.m1.3.3.1.1.1.1.2.2.3" xref="A2.Ex12.m1.3.3.1.1.1.1.2.2.3.cmml"><mi id="A2.Ex12.m1.3.3.1.1.1.1.2.2.3.2" xref="A2.Ex12.m1.3.3.1.1.1.1.2.2.3.2.cmml">i</mi><mo id="A2.Ex12.m1.3.3.1.1.1.1.2.2.3.1" xref="A2.Ex12.m1.3.3.1.1.1.1.2.2.3.1.cmml">=</mo><mn id="A2.Ex12.m1.3.3.1.1.1.1.2.2.3.3" xref="A2.Ex12.m1.3.3.1.1.1.1.2.2.3.3.cmml">1</mn></mrow><mi id="A2.Ex12.m1.3.3.1.1.1.1.2.3" xref="A2.Ex12.m1.3.3.1.1.1.1.2.3.cmml">n</mi></munderover><mrow id="A2.Ex12.m1.3.3.1.1.1.1.1" xref="A2.Ex12.m1.3.3.1.1.1.1.1.cmml"><mi id="A2.Ex12.m1.3.3.1.1.1.1.1.3" xref="A2.Ex12.m1.3.3.1.1.1.1.1.3.cmml">κ</mi><mo lspace="0em" rspace="0em" id="A2.Ex12.m1.3.3.1.1.1.1.1.2" xref="A2.Ex12.m1.3.3.1.1.1.1.1.2.cmml">​</mo><mrow id="A2.Ex12.m1.3.3.1.1.1.1.1.1.1" xref="A2.Ex12.m1.3.3.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A2.Ex12.m1.3.3.1.1.1.1.1.1.1.2" xref="A2.Ex12.m1.3.3.1.1.1.1.1.1.2.cmml">(</mo><mi id="A2.Ex12.m1.2.2" xref="A2.Ex12.m1.2.2.cmml">x</mi><mo id="A2.Ex12.m1.3.3.1.1.1.1.1.1.1.3" xref="A2.Ex12.m1.3.3.1.1.1.1.1.1.2.cmml">,</mo><msub id="A2.Ex12.m1.3.3.1.1.1.1.1.1.1.1" xref="A2.Ex12.m1.3.3.1.1.1.1.1.1.1.1.cmml"><mi id="A2.Ex12.m1.3.3.1.1.1.1.1.1.1.1.2" xref="A2.Ex12.m1.3.3.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="A2.Ex12.m1.3.3.1.1.1.1.1.1.1.1.3" xref="A2.Ex12.m1.3.3.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A2.Ex12.m1.3.3.1.1.1.1.1.1.1.4" xref="A2.Ex12.m1.3.3.1.1.1.1.1.1.2.cmml">)</mo></mrow></mrow></mrow></mrow></mrow><mo id="A2.Ex12.m1.3.3.1.2" xref="A2.Ex12.m1.3.3.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="A2.Ex12.m1.3b"><apply id="A2.Ex12.m1.3.3.1.1.cmml" xref="A2.Ex12.m1.3.3.1"><eq id="A2.Ex12.m1.3.3.1.1.2.cmml" xref="A2.Ex12.m1.3.3.1.1.2"></eq><apply id="A2.Ex12.m1.3.3.1.1.3.cmml" xref="A2.Ex12.m1.3.3.1.1.3"><times id="A2.Ex12.m1.3.3.1.1.3.1.cmml" xref="A2.Ex12.m1.3.3.1.1.3.1"></times><apply id="A2.Ex12.m1.3.3.1.1.3.2.cmml" xref="A2.Ex12.m1.3.3.1.1.3.2"><ci id="A2.Ex12.m1.3.3.1.1.3.2.1.cmml" xref="A2.Ex12.m1.3.3.1.1.3.2.1">^</ci><ci id="A2.Ex12.m1.3.3.1.1.3.2.2.cmml" xref="A2.Ex12.m1.3.3.1.1.3.2.2">𝑝</ci></apply><ci id="A2.Ex12.m1.1.1.cmml" xref="A2.Ex12.m1.1.1">𝑥</ci></apply><apply id="A2.Ex12.m1.3.3.1.1.1.cmml" xref="A2.Ex12.m1.3.3.1.1.1"><times id="A2.Ex12.m1.3.3.1.1.1.2.cmml" xref="A2.Ex12.m1.3.3.1.1.1.2"></times><apply id="A2.Ex12.m1.3.3.1.1.1.3.cmml" xref="A2.Ex12.m1.3.3.1.1.1.3"><divide id="A2.Ex12.m1.3.3.1.1.1.3.1.cmml" xref="A2.Ex12.m1.3.3.1.1.1.3"></divide><cn type="integer" id="A2.Ex12.m1.3.3.1.1.1.3.2.cmml" xref="A2.Ex12.m1.3.3.1.1.1.3.2">1</cn><ci id="A2.Ex12.m1.3.3.1.1.1.3.3.cmml" xref="A2.Ex12.m1.3.3.1.1.1.3.3">𝑛</ci></apply><apply id="A2.Ex12.m1.3.3.1.1.1.1.cmml" xref="A2.Ex12.m1.3.3.1.1.1.1"><apply id="A2.Ex12.m1.3.3.1.1.1.1.2.cmml" xref="A2.Ex12.m1.3.3.1.1.1.1.2"><csymbol cd="ambiguous" id="A2.Ex12.m1.3.3.1.1.1.1.2.1.cmml" xref="A2.Ex12.m1.3.3.1.1.1.1.2">superscript</csymbol><apply id="A2.Ex12.m1.3.3.1.1.1.1.2.2.cmml" xref="A2.Ex12.m1.3.3.1.1.1.1.2"><csymbol cd="ambiguous" id="A2.Ex12.m1.3.3.1.1.1.1.2.2.1.cmml" xref="A2.Ex12.m1.3.3.1.1.1.1.2">subscript</csymbol><sum id="A2.Ex12.m1.3.3.1.1.1.1.2.2.2.cmml" xref="A2.Ex12.m1.3.3.1.1.1.1.2.2.2"></sum><apply id="A2.Ex12.m1.3.3.1.1.1.1.2.2.3.cmml" xref="A2.Ex12.m1.3.3.1.1.1.1.2.2.3"><eq id="A2.Ex12.m1.3.3.1.1.1.1.2.2.3.1.cmml" xref="A2.Ex12.m1.3.3.1.1.1.1.2.2.3.1"></eq><ci id="A2.Ex12.m1.3.3.1.1.1.1.2.2.3.2.cmml" xref="A2.Ex12.m1.3.3.1.1.1.1.2.2.3.2">𝑖</ci><cn type="integer" id="A2.Ex12.m1.3.3.1.1.1.1.2.2.3.3.cmml" xref="A2.Ex12.m1.3.3.1.1.1.1.2.2.3.3">1</cn></apply></apply><ci id="A2.Ex12.m1.3.3.1.1.1.1.2.3.cmml" xref="A2.Ex12.m1.3.3.1.1.1.1.2.3">𝑛</ci></apply><apply id="A2.Ex12.m1.3.3.1.1.1.1.1.cmml" xref="A2.Ex12.m1.3.3.1.1.1.1.1"><times id="A2.Ex12.m1.3.3.1.1.1.1.1.2.cmml" xref="A2.Ex12.m1.3.3.1.1.1.1.1.2"></times><ci id="A2.Ex12.m1.3.3.1.1.1.1.1.3.cmml" xref="A2.Ex12.m1.3.3.1.1.1.1.1.3">𝜅</ci><interval closure="open" id="A2.Ex12.m1.3.3.1.1.1.1.1.1.2.cmml" xref="A2.Ex12.m1.3.3.1.1.1.1.1.1.1"><ci id="A2.Ex12.m1.2.2.cmml" xref="A2.Ex12.m1.2.2">𝑥</ci><apply id="A2.Ex12.m1.3.3.1.1.1.1.1.1.1.1.cmml" xref="A2.Ex12.m1.3.3.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A2.Ex12.m1.3.3.1.1.1.1.1.1.1.1.1.cmml" xref="A2.Ex12.m1.3.3.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="A2.Ex12.m1.3.3.1.1.1.1.1.1.1.1.2.cmml" xref="A2.Ex12.m1.3.3.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="A2.Ex12.m1.3.3.1.1.1.1.1.1.1.1.3.cmml" xref="A2.Ex12.m1.3.3.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></interval></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.Ex12.m1.3c">\hat{p}(x)=\frac{1}{n}\sum_{i=1}^{n}\kappa(x,x_{i}),</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="A2.SS0.SSS0.Px1.p1.9" class="ltx_p">where each <math id="A2.SS0.SSS0.Px1.p1.8.m1.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="A2.SS0.SSS0.Px1.p1.8.m1.1a"><msub id="A2.SS0.SSS0.Px1.p1.8.m1.1.1" xref="A2.SS0.SSS0.Px1.p1.8.m1.1.1.cmml"><mi id="A2.SS0.SSS0.Px1.p1.8.m1.1.1.2" xref="A2.SS0.SSS0.Px1.p1.8.m1.1.1.2.cmml">x</mi><mi id="A2.SS0.SSS0.Px1.p1.8.m1.1.1.3" xref="A2.SS0.SSS0.Px1.p1.8.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px1.p1.8.m1.1b"><apply id="A2.SS0.SSS0.Px1.p1.8.m1.1.1.cmml" xref="A2.SS0.SSS0.Px1.p1.8.m1.1.1"><csymbol cd="ambiguous" id="A2.SS0.SSS0.Px1.p1.8.m1.1.1.1.cmml" xref="A2.SS0.SSS0.Px1.p1.8.m1.1.1">subscript</csymbol><ci id="A2.SS0.SSS0.Px1.p1.8.m1.1.1.2.cmml" xref="A2.SS0.SSS0.Px1.p1.8.m1.1.1.2">𝑥</ci><ci id="A2.SS0.SSS0.Px1.p1.8.m1.1.1.3.cmml" xref="A2.SS0.SSS0.Px1.p1.8.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px1.p1.8.m1.1c">x_{i}</annotation></semantics></math> is a piece of training data in general.
(Recall that for our paper, <math id="A2.SS0.SSS0.Px1.p1.9.m2.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="A2.SS0.SSS0.Px1.p1.9.m2.1a"><msub id="A2.SS0.SSS0.Px1.p1.9.m2.1.1" xref="A2.SS0.SSS0.Px1.p1.9.m2.1.1.cmml"><mi id="A2.SS0.SSS0.Px1.p1.9.m2.1.1.2" xref="A2.SS0.SSS0.Px1.p1.9.m2.1.1.2.cmml">x</mi><mi id="A2.SS0.SSS0.Px1.p1.9.m2.1.1.3" xref="A2.SS0.SSS0.Px1.p1.9.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px1.p1.9.m2.1b"><apply id="A2.SS0.SSS0.Px1.p1.9.m2.1.1.cmml" xref="A2.SS0.SSS0.Px1.p1.9.m2.1.1"><csymbol cd="ambiguous" id="A2.SS0.SSS0.Px1.p1.9.m2.1.1.1.cmml" xref="A2.SS0.SSS0.Px1.p1.9.m2.1.1">subscript</csymbol><ci id="A2.SS0.SSS0.Px1.p1.9.m2.1.1.2.cmml" xref="A2.SS0.SSS0.Px1.p1.9.m2.1.1.2">𝑥</ci><ci id="A2.SS0.SSS0.Px1.p1.9.m2.1.1.3.cmml" xref="A2.SS0.SSS0.Px1.p1.9.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px1.p1.9.m2.1c">x_{i}</annotation></semantics></math> is specifically training data for the inner loop, <span id="A2.SS0.SSS0.Px1.p1.9.1" class="ltx_text ltx_font_italic">i.e.</span> a token, which matches our notation in the main text.)</p>
</div>
<div id="A2.SS0.SSS0.Px1.p2" class="ltx_para">
<p id="A2.SS0.SSS0.Px1.p2.1" class="ltx_p">For estimating <math id="A2.SS0.SSS0.Px1.p2.1.m1.2" class="ltx_Math" alttext="p(x,y)" display="inline"><semantics id="A2.SS0.SSS0.Px1.p2.1.m1.2a"><mrow id="A2.SS0.SSS0.Px1.p2.1.m1.2.3" xref="A2.SS0.SSS0.Px1.p2.1.m1.2.3.cmml"><mi id="A2.SS0.SSS0.Px1.p2.1.m1.2.3.2" xref="A2.SS0.SSS0.Px1.p2.1.m1.2.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="A2.SS0.SSS0.Px1.p2.1.m1.2.3.1" xref="A2.SS0.SSS0.Px1.p2.1.m1.2.3.1.cmml">​</mo><mrow id="A2.SS0.SSS0.Px1.p2.1.m1.2.3.3.2" xref="A2.SS0.SSS0.Px1.p2.1.m1.2.3.3.1.cmml"><mo stretchy="false" id="A2.SS0.SSS0.Px1.p2.1.m1.2.3.3.2.1" xref="A2.SS0.SSS0.Px1.p2.1.m1.2.3.3.1.cmml">(</mo><mi id="A2.SS0.SSS0.Px1.p2.1.m1.1.1" xref="A2.SS0.SSS0.Px1.p2.1.m1.1.1.cmml">x</mi><mo id="A2.SS0.SSS0.Px1.p2.1.m1.2.3.3.2.2" xref="A2.SS0.SSS0.Px1.p2.1.m1.2.3.3.1.cmml">,</mo><mi id="A2.SS0.SSS0.Px1.p2.1.m1.2.2" xref="A2.SS0.SSS0.Px1.p2.1.m1.2.2.cmml">y</mi><mo stretchy="false" id="A2.SS0.SSS0.Px1.p2.1.m1.2.3.3.2.3" xref="A2.SS0.SSS0.Px1.p2.1.m1.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px1.p2.1.m1.2b"><apply id="A2.SS0.SSS0.Px1.p2.1.m1.2.3.cmml" xref="A2.SS0.SSS0.Px1.p2.1.m1.2.3"><times id="A2.SS0.SSS0.Px1.p2.1.m1.2.3.1.cmml" xref="A2.SS0.SSS0.Px1.p2.1.m1.2.3.1"></times><ci id="A2.SS0.SSS0.Px1.p2.1.m1.2.3.2.cmml" xref="A2.SS0.SSS0.Px1.p2.1.m1.2.3.2">𝑝</ci><interval closure="open" id="A2.SS0.SSS0.Px1.p2.1.m1.2.3.3.1.cmml" xref="A2.SS0.SSS0.Px1.p2.1.m1.2.3.3.2"><ci id="A2.SS0.SSS0.Px1.p2.1.m1.1.1.cmml" xref="A2.SS0.SSS0.Px1.p2.1.m1.1.1">𝑥</ci><ci id="A2.SS0.SSS0.Px1.p2.1.m1.2.2.cmml" xref="A2.SS0.SSS0.Px1.p2.1.m1.2.2">𝑦</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px1.p2.1.m1.2c">p(x,y)</annotation></semantics></math>, we use the product kernel:</p>
<table id="A2.Ex13" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A2.Ex13.m1.5" class="ltx_Math" alttext="\hat{p}(x,z)=\frac{1}{n}\sum_{i=1}^{n}\kappa(x,x_{i})~{}\kappa^{\prime}(z,z_{i})." display="block"><semantics id="A2.Ex13.m1.5a"><mrow id="A2.Ex13.m1.5.5.1" xref="A2.Ex13.m1.5.5.1.1.cmml"><mrow id="A2.Ex13.m1.5.5.1.1" xref="A2.Ex13.m1.5.5.1.1.cmml"><mrow id="A2.Ex13.m1.5.5.1.1.4" xref="A2.Ex13.m1.5.5.1.1.4.cmml"><mover accent="true" id="A2.Ex13.m1.5.5.1.1.4.2" xref="A2.Ex13.m1.5.5.1.1.4.2.cmml"><mi id="A2.Ex13.m1.5.5.1.1.4.2.2" xref="A2.Ex13.m1.5.5.1.1.4.2.2.cmml">p</mi><mo id="A2.Ex13.m1.5.5.1.1.4.2.1" xref="A2.Ex13.m1.5.5.1.1.4.2.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A2.Ex13.m1.5.5.1.1.4.1" xref="A2.Ex13.m1.5.5.1.1.4.1.cmml">​</mo><mrow id="A2.Ex13.m1.5.5.1.1.4.3.2" xref="A2.Ex13.m1.5.5.1.1.4.3.1.cmml"><mo stretchy="false" id="A2.Ex13.m1.5.5.1.1.4.3.2.1" xref="A2.Ex13.m1.5.5.1.1.4.3.1.cmml">(</mo><mi id="A2.Ex13.m1.1.1" xref="A2.Ex13.m1.1.1.cmml">x</mi><mo id="A2.Ex13.m1.5.5.1.1.4.3.2.2" xref="A2.Ex13.m1.5.5.1.1.4.3.1.cmml">,</mo><mi id="A2.Ex13.m1.2.2" xref="A2.Ex13.m1.2.2.cmml">z</mi><mo stretchy="false" id="A2.Ex13.m1.5.5.1.1.4.3.2.3" xref="A2.Ex13.m1.5.5.1.1.4.3.1.cmml">)</mo></mrow></mrow><mo id="A2.Ex13.m1.5.5.1.1.3" xref="A2.Ex13.m1.5.5.1.1.3.cmml">=</mo><mrow id="A2.Ex13.m1.5.5.1.1.2" xref="A2.Ex13.m1.5.5.1.1.2.cmml"><mfrac id="A2.Ex13.m1.5.5.1.1.2.4" xref="A2.Ex13.m1.5.5.1.1.2.4.cmml"><mn id="A2.Ex13.m1.5.5.1.1.2.4.2" xref="A2.Ex13.m1.5.5.1.1.2.4.2.cmml">1</mn><mi id="A2.Ex13.m1.5.5.1.1.2.4.3" xref="A2.Ex13.m1.5.5.1.1.2.4.3.cmml">n</mi></mfrac><mo lspace="0em" rspace="0em" id="A2.Ex13.m1.5.5.1.1.2.3" xref="A2.Ex13.m1.5.5.1.1.2.3.cmml">​</mo><mrow id="A2.Ex13.m1.5.5.1.1.2.2" xref="A2.Ex13.m1.5.5.1.1.2.2.cmml"><munderover id="A2.Ex13.m1.5.5.1.1.2.2.3" xref="A2.Ex13.m1.5.5.1.1.2.2.3.cmml"><mo movablelimits="false" id="A2.Ex13.m1.5.5.1.1.2.2.3.2.2" xref="A2.Ex13.m1.5.5.1.1.2.2.3.2.2.cmml">∑</mo><mrow id="A2.Ex13.m1.5.5.1.1.2.2.3.2.3" xref="A2.Ex13.m1.5.5.1.1.2.2.3.2.3.cmml"><mi id="A2.Ex13.m1.5.5.1.1.2.2.3.2.3.2" xref="A2.Ex13.m1.5.5.1.1.2.2.3.2.3.2.cmml">i</mi><mo id="A2.Ex13.m1.5.5.1.1.2.2.3.2.3.1" xref="A2.Ex13.m1.5.5.1.1.2.2.3.2.3.1.cmml">=</mo><mn id="A2.Ex13.m1.5.5.1.1.2.2.3.2.3.3" xref="A2.Ex13.m1.5.5.1.1.2.2.3.2.3.3.cmml">1</mn></mrow><mi id="A2.Ex13.m1.5.5.1.1.2.2.3.3" xref="A2.Ex13.m1.5.5.1.1.2.2.3.3.cmml">n</mi></munderover><mrow id="A2.Ex13.m1.5.5.1.1.2.2.2" xref="A2.Ex13.m1.5.5.1.1.2.2.2.cmml"><mi id="A2.Ex13.m1.5.5.1.1.2.2.2.4" xref="A2.Ex13.m1.5.5.1.1.2.2.2.4.cmml">κ</mi><mo lspace="0em" rspace="0em" id="A2.Ex13.m1.5.5.1.1.2.2.2.3" xref="A2.Ex13.m1.5.5.1.1.2.2.2.3.cmml">​</mo><mrow id="A2.Ex13.m1.5.5.1.1.1.1.1.1.1" xref="A2.Ex13.m1.5.5.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A2.Ex13.m1.5.5.1.1.1.1.1.1.1.2" xref="A2.Ex13.m1.5.5.1.1.1.1.1.1.2.cmml">(</mo><mi id="A2.Ex13.m1.3.3" xref="A2.Ex13.m1.3.3.cmml">x</mi><mo id="A2.Ex13.m1.5.5.1.1.1.1.1.1.1.3" xref="A2.Ex13.m1.5.5.1.1.1.1.1.1.2.cmml">,</mo><msub id="A2.Ex13.m1.5.5.1.1.1.1.1.1.1.1" xref="A2.Ex13.m1.5.5.1.1.1.1.1.1.1.1.cmml"><mi id="A2.Ex13.m1.5.5.1.1.1.1.1.1.1.1.2" xref="A2.Ex13.m1.5.5.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="A2.Ex13.m1.5.5.1.1.1.1.1.1.1.1.3" xref="A2.Ex13.m1.5.5.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A2.Ex13.m1.5.5.1.1.1.1.1.1.1.4" xref="A2.Ex13.m1.5.5.1.1.1.1.1.1.2.cmml">)</mo></mrow><mo lspace="0.330em" rspace="0em" id="A2.Ex13.m1.5.5.1.1.2.2.2.3a" xref="A2.Ex13.m1.5.5.1.1.2.2.2.3.cmml">​</mo><msup id="A2.Ex13.m1.5.5.1.1.2.2.2.5" xref="A2.Ex13.m1.5.5.1.1.2.2.2.5.cmml"><mi id="A2.Ex13.m1.5.5.1.1.2.2.2.5.2" xref="A2.Ex13.m1.5.5.1.1.2.2.2.5.2.cmml">κ</mi><mo id="A2.Ex13.m1.5.5.1.1.2.2.2.5.3" xref="A2.Ex13.m1.5.5.1.1.2.2.2.5.3.cmml">′</mo></msup><mo lspace="0em" rspace="0em" id="A2.Ex13.m1.5.5.1.1.2.2.2.3b" xref="A2.Ex13.m1.5.5.1.1.2.2.2.3.cmml">​</mo><mrow id="A2.Ex13.m1.5.5.1.1.2.2.2.2.1" xref="A2.Ex13.m1.5.5.1.1.2.2.2.2.2.cmml"><mo stretchy="false" id="A2.Ex13.m1.5.5.1.1.2.2.2.2.1.2" xref="A2.Ex13.m1.5.5.1.1.2.2.2.2.2.cmml">(</mo><mi id="A2.Ex13.m1.4.4" xref="A2.Ex13.m1.4.4.cmml">z</mi><mo id="A2.Ex13.m1.5.5.1.1.2.2.2.2.1.3" xref="A2.Ex13.m1.5.5.1.1.2.2.2.2.2.cmml">,</mo><msub id="A2.Ex13.m1.5.5.1.1.2.2.2.2.1.1" xref="A2.Ex13.m1.5.5.1.1.2.2.2.2.1.1.cmml"><mi id="A2.Ex13.m1.5.5.1.1.2.2.2.2.1.1.2" xref="A2.Ex13.m1.5.5.1.1.2.2.2.2.1.1.2.cmml">z</mi><mi id="A2.Ex13.m1.5.5.1.1.2.2.2.2.1.1.3" xref="A2.Ex13.m1.5.5.1.1.2.2.2.2.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A2.Ex13.m1.5.5.1.1.2.2.2.2.1.4" xref="A2.Ex13.m1.5.5.1.1.2.2.2.2.2.cmml">)</mo></mrow></mrow></mrow></mrow></mrow><mo lspace="0em" id="A2.Ex13.m1.5.5.1.2" xref="A2.Ex13.m1.5.5.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A2.Ex13.m1.5b"><apply id="A2.Ex13.m1.5.5.1.1.cmml" xref="A2.Ex13.m1.5.5.1"><eq id="A2.Ex13.m1.5.5.1.1.3.cmml" xref="A2.Ex13.m1.5.5.1.1.3"></eq><apply id="A2.Ex13.m1.5.5.1.1.4.cmml" xref="A2.Ex13.m1.5.5.1.1.4"><times id="A2.Ex13.m1.5.5.1.1.4.1.cmml" xref="A2.Ex13.m1.5.5.1.1.4.1"></times><apply id="A2.Ex13.m1.5.5.1.1.4.2.cmml" xref="A2.Ex13.m1.5.5.1.1.4.2"><ci id="A2.Ex13.m1.5.5.1.1.4.2.1.cmml" xref="A2.Ex13.m1.5.5.1.1.4.2.1">^</ci><ci id="A2.Ex13.m1.5.5.1.1.4.2.2.cmml" xref="A2.Ex13.m1.5.5.1.1.4.2.2">𝑝</ci></apply><interval closure="open" id="A2.Ex13.m1.5.5.1.1.4.3.1.cmml" xref="A2.Ex13.m1.5.5.1.1.4.3.2"><ci id="A2.Ex13.m1.1.1.cmml" xref="A2.Ex13.m1.1.1">𝑥</ci><ci id="A2.Ex13.m1.2.2.cmml" xref="A2.Ex13.m1.2.2">𝑧</ci></interval></apply><apply id="A2.Ex13.m1.5.5.1.1.2.cmml" xref="A2.Ex13.m1.5.5.1.1.2"><times id="A2.Ex13.m1.5.5.1.1.2.3.cmml" xref="A2.Ex13.m1.5.5.1.1.2.3"></times><apply id="A2.Ex13.m1.5.5.1.1.2.4.cmml" xref="A2.Ex13.m1.5.5.1.1.2.4"><divide id="A2.Ex13.m1.5.5.1.1.2.4.1.cmml" xref="A2.Ex13.m1.5.5.1.1.2.4"></divide><cn type="integer" id="A2.Ex13.m1.5.5.1.1.2.4.2.cmml" xref="A2.Ex13.m1.5.5.1.1.2.4.2">1</cn><ci id="A2.Ex13.m1.5.5.1.1.2.4.3.cmml" xref="A2.Ex13.m1.5.5.1.1.2.4.3">𝑛</ci></apply><apply id="A2.Ex13.m1.5.5.1.1.2.2.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2"><apply id="A2.Ex13.m1.5.5.1.1.2.2.3.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.3"><csymbol cd="ambiguous" id="A2.Ex13.m1.5.5.1.1.2.2.3.1.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.3">superscript</csymbol><apply id="A2.Ex13.m1.5.5.1.1.2.2.3.2.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.3"><csymbol cd="ambiguous" id="A2.Ex13.m1.5.5.1.1.2.2.3.2.1.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.3">subscript</csymbol><sum id="A2.Ex13.m1.5.5.1.1.2.2.3.2.2.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.3.2.2"></sum><apply id="A2.Ex13.m1.5.5.1.1.2.2.3.2.3.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.3.2.3"><eq id="A2.Ex13.m1.5.5.1.1.2.2.3.2.3.1.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.3.2.3.1"></eq><ci id="A2.Ex13.m1.5.5.1.1.2.2.3.2.3.2.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.3.2.3.2">𝑖</ci><cn type="integer" id="A2.Ex13.m1.5.5.1.1.2.2.3.2.3.3.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.3.2.3.3">1</cn></apply></apply><ci id="A2.Ex13.m1.5.5.1.1.2.2.3.3.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.3.3">𝑛</ci></apply><apply id="A2.Ex13.m1.5.5.1.1.2.2.2.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.2"><times id="A2.Ex13.m1.5.5.1.1.2.2.2.3.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.2.3"></times><ci id="A2.Ex13.m1.5.5.1.1.2.2.2.4.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.2.4">𝜅</ci><interval closure="open" id="A2.Ex13.m1.5.5.1.1.1.1.1.1.2.cmml" xref="A2.Ex13.m1.5.5.1.1.1.1.1.1.1"><ci id="A2.Ex13.m1.3.3.cmml" xref="A2.Ex13.m1.3.3">𝑥</ci><apply id="A2.Ex13.m1.5.5.1.1.1.1.1.1.1.1.cmml" xref="A2.Ex13.m1.5.5.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A2.Ex13.m1.5.5.1.1.1.1.1.1.1.1.1.cmml" xref="A2.Ex13.m1.5.5.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="A2.Ex13.m1.5.5.1.1.1.1.1.1.1.1.2.cmml" xref="A2.Ex13.m1.5.5.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="A2.Ex13.m1.5.5.1.1.1.1.1.1.1.1.3.cmml" xref="A2.Ex13.m1.5.5.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></interval><apply id="A2.Ex13.m1.5.5.1.1.2.2.2.5.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.2.5"><csymbol cd="ambiguous" id="A2.Ex13.m1.5.5.1.1.2.2.2.5.1.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.2.5">superscript</csymbol><ci id="A2.Ex13.m1.5.5.1.1.2.2.2.5.2.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.2.5.2">𝜅</ci><ci id="A2.Ex13.m1.5.5.1.1.2.2.2.5.3.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.2.5.3">′</ci></apply><interval closure="open" id="A2.Ex13.m1.5.5.1.1.2.2.2.2.2.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.2.2.1"><ci id="A2.Ex13.m1.4.4.cmml" xref="A2.Ex13.m1.4.4">𝑧</ci><apply id="A2.Ex13.m1.5.5.1.1.2.2.2.2.1.1.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.2.2.1.1"><csymbol cd="ambiguous" id="A2.Ex13.m1.5.5.1.1.2.2.2.2.1.1.1.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.2.2.1.1">subscript</csymbol><ci id="A2.Ex13.m1.5.5.1.1.2.2.2.2.1.1.2.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.2.2.1.1.2">𝑧</ci><ci id="A2.Ex13.m1.5.5.1.1.2.2.2.2.1.1.3.cmml" xref="A2.Ex13.m1.5.5.1.1.2.2.2.2.1.1.3">𝑖</ci></apply></interval></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.Ex13.m1.5c">\hat{p}(x,z)=\frac{1}{n}\sum_{i=1}^{n}\kappa(x,x_{i})~{}\kappa^{\prime}(z,z_{i}).</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="A2.SS0.SSS0.Px1.p2.4" class="ltx_p">At first sight, it seems absurd to factor the joint probability into two seemingly independent kernels.
But in this case, <math id="A2.SS0.SSS0.Px1.p2.2.m1.1" class="ltx_Math" alttext="\kappa^{\prime}" display="inline"><semantics id="A2.SS0.SSS0.Px1.p2.2.m1.1a"><msup id="A2.SS0.SSS0.Px1.p2.2.m1.1.1" xref="A2.SS0.SSS0.Px1.p2.2.m1.1.1.cmml"><mi id="A2.SS0.SSS0.Px1.p2.2.m1.1.1.2" xref="A2.SS0.SSS0.Px1.p2.2.m1.1.1.2.cmml">κ</mi><mo id="A2.SS0.SSS0.Px1.p2.2.m1.1.1.3" xref="A2.SS0.SSS0.Px1.p2.2.m1.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px1.p2.2.m1.1b"><apply id="A2.SS0.SSS0.Px1.p2.2.m1.1.1.cmml" xref="A2.SS0.SSS0.Px1.p2.2.m1.1.1"><csymbol cd="ambiguous" id="A2.SS0.SSS0.Px1.p2.2.m1.1.1.1.cmml" xref="A2.SS0.SSS0.Px1.p2.2.m1.1.1">superscript</csymbol><ci id="A2.SS0.SSS0.Px1.p2.2.m1.1.1.2.cmml" xref="A2.SS0.SSS0.Px1.p2.2.m1.1.1.2">𝜅</ci><ci id="A2.SS0.SSS0.Px1.p2.2.m1.1.1.3.cmml" xref="A2.SS0.SSS0.Px1.p2.2.m1.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px1.p2.2.m1.1c">\kappa^{\prime}</annotation></semantics></math> can actually be any <math id="A2.SS0.SSS0.Px1.p2.3.m2.1" class="ltx_Math" alttext="\kappa_{i}^{\prime}" display="inline"><semantics id="A2.SS0.SSS0.Px1.p2.3.m2.1a"><msubsup id="A2.SS0.SSS0.Px1.p2.3.m2.1.1" xref="A2.SS0.SSS0.Px1.p2.3.m2.1.1.cmml"><mi id="A2.SS0.SSS0.Px1.p2.3.m2.1.1.2.2" xref="A2.SS0.SSS0.Px1.p2.3.m2.1.1.2.2.cmml">κ</mi><mi id="A2.SS0.SSS0.Px1.p2.3.m2.1.1.2.3" xref="A2.SS0.SSS0.Px1.p2.3.m2.1.1.2.3.cmml">i</mi><mo id="A2.SS0.SSS0.Px1.p2.3.m2.1.1.3" xref="A2.SS0.SSS0.Px1.p2.3.m2.1.1.3.cmml">′</mo></msubsup><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px1.p2.3.m2.1b"><apply id="A2.SS0.SSS0.Px1.p2.3.m2.1.1.cmml" xref="A2.SS0.SSS0.Px1.p2.3.m2.1.1"><csymbol cd="ambiguous" id="A2.SS0.SSS0.Px1.p2.3.m2.1.1.1.cmml" xref="A2.SS0.SSS0.Px1.p2.3.m2.1.1">superscript</csymbol><apply id="A2.SS0.SSS0.Px1.p2.3.m2.1.1.2.cmml" xref="A2.SS0.SSS0.Px1.p2.3.m2.1.1"><csymbol cd="ambiguous" id="A2.SS0.SSS0.Px1.p2.3.m2.1.1.2.1.cmml" xref="A2.SS0.SSS0.Px1.p2.3.m2.1.1">subscript</csymbol><ci id="A2.SS0.SSS0.Px1.p2.3.m2.1.1.2.2.cmml" xref="A2.SS0.SSS0.Px1.p2.3.m2.1.1.2.2">𝜅</ci><ci id="A2.SS0.SSS0.Px1.p2.3.m2.1.1.2.3.cmml" xref="A2.SS0.SSS0.Px1.p2.3.m2.1.1.2.3">𝑖</ci></apply><ci id="A2.SS0.SSS0.Px1.p2.3.m2.1.1.3.cmml" xref="A2.SS0.SSS0.Px1.p2.3.m2.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px1.p2.3.m2.1c">\kappa_{i}^{\prime}</annotation></semantics></math> dependent on <math id="A2.SS0.SSS0.Px1.p2.4.m3.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="A2.SS0.SSS0.Px1.p2.4.m3.1a"><msub id="A2.SS0.SSS0.Px1.p2.4.m3.1.1" xref="A2.SS0.SSS0.Px1.p2.4.m3.1.1.cmml"><mi id="A2.SS0.SSS0.Px1.p2.4.m3.1.1.2" xref="A2.SS0.SSS0.Px1.p2.4.m3.1.1.2.cmml">x</mi><mi id="A2.SS0.SSS0.Px1.p2.4.m3.1.1.3" xref="A2.SS0.SSS0.Px1.p2.4.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px1.p2.4.m3.1b"><apply id="A2.SS0.SSS0.Px1.p2.4.m3.1.1.cmml" xref="A2.SS0.SSS0.Px1.p2.4.m3.1.1"><csymbol cd="ambiguous" id="A2.SS0.SSS0.Px1.p2.4.m3.1.1.1.cmml" xref="A2.SS0.SSS0.Px1.p2.4.m3.1.1">subscript</csymbol><ci id="A2.SS0.SSS0.Px1.p2.4.m3.1.1.2.cmml" xref="A2.SS0.SSS0.Px1.p2.4.m3.1.1.2">𝑥</ci><ci id="A2.SS0.SSS0.Px1.p2.4.m3.1.1.3.cmml" xref="A2.SS0.SSS0.Px1.p2.4.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px1.p2.4.m3.1c">x_{i}</annotation></semantics></math>, since it will be integrated out. So the two kernels do not need to be independent.</p>
</div>
<div id="A2.SS0.SSS0.Px1.p3" class="ltx_para">
<p id="A2.SS0.SSS0.Px1.p3.1" class="ltx_p">Plugging in those estimations, we obtain the Nadaraya-Watson estimator:</p>
<table id="A3.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A2.Ex14"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A2.Ex14.m1.1" class="ltx_Math" alttext="\displaystyle\hat{\mathbb{E}}[\mathbf{z}|\mathbf{x}=x]" display="inline"><semantics id="A2.Ex14.m1.1a"><mrow id="A2.Ex14.m1.1.1" xref="A2.Ex14.m1.1.1.cmml"><mover accent="true" id="A2.Ex14.m1.1.1.3" xref="A2.Ex14.m1.1.1.3.cmml"><mi id="A2.Ex14.m1.1.1.3.2" xref="A2.Ex14.m1.1.1.3.2.cmml">𝔼</mi><mo id="A2.Ex14.m1.1.1.3.1" xref="A2.Ex14.m1.1.1.3.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A2.Ex14.m1.1.1.2" xref="A2.Ex14.m1.1.1.2.cmml">​</mo><mrow id="A2.Ex14.m1.1.1.1.1" xref="A2.Ex14.m1.1.1.1.2.cmml"><mo stretchy="false" id="A2.Ex14.m1.1.1.1.1.2" xref="A2.Ex14.m1.1.1.1.2.1.cmml">[</mo><mrow id="A2.Ex14.m1.1.1.1.1.1" xref="A2.Ex14.m1.1.1.1.1.1.cmml"><mrow id="A2.Ex14.m1.1.1.1.1.1.2" xref="A2.Ex14.m1.1.1.1.1.1.2.cmml"><mi id="A2.Ex14.m1.1.1.1.1.1.2.2" xref="A2.Ex14.m1.1.1.1.1.1.2.2.cmml">𝐳</mi><mo fence="false" id="A2.Ex14.m1.1.1.1.1.1.2.1" xref="A2.Ex14.m1.1.1.1.1.1.2.1.cmml">|</mo><mi id="A2.Ex14.m1.1.1.1.1.1.2.3" xref="A2.Ex14.m1.1.1.1.1.1.2.3.cmml">𝐱</mi></mrow><mo id="A2.Ex14.m1.1.1.1.1.1.1" xref="A2.Ex14.m1.1.1.1.1.1.1.cmml">=</mo><mi id="A2.Ex14.m1.1.1.1.1.1.3" xref="A2.Ex14.m1.1.1.1.1.1.3.cmml">x</mi></mrow><mo stretchy="false" id="A2.Ex14.m1.1.1.1.1.3" xref="A2.Ex14.m1.1.1.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.Ex14.m1.1b"><apply id="A2.Ex14.m1.1.1.cmml" xref="A2.Ex14.m1.1.1"><times id="A2.Ex14.m1.1.1.2.cmml" xref="A2.Ex14.m1.1.1.2"></times><apply id="A2.Ex14.m1.1.1.3.cmml" xref="A2.Ex14.m1.1.1.3"><ci id="A2.Ex14.m1.1.1.3.1.cmml" xref="A2.Ex14.m1.1.1.3.1">^</ci><ci id="A2.Ex14.m1.1.1.3.2.cmml" xref="A2.Ex14.m1.1.1.3.2">𝔼</ci></apply><apply id="A2.Ex14.m1.1.1.1.2.cmml" xref="A2.Ex14.m1.1.1.1.1"><csymbol cd="latexml" id="A2.Ex14.m1.1.1.1.2.1.cmml" xref="A2.Ex14.m1.1.1.1.1.2">delimited-[]</csymbol><apply id="A2.Ex14.m1.1.1.1.1.1.cmml" xref="A2.Ex14.m1.1.1.1.1.1"><eq id="A2.Ex14.m1.1.1.1.1.1.1.cmml" xref="A2.Ex14.m1.1.1.1.1.1.1"></eq><apply id="A2.Ex14.m1.1.1.1.1.1.2.cmml" xref="A2.Ex14.m1.1.1.1.1.1.2"><csymbol cd="latexml" id="A2.Ex14.m1.1.1.1.1.1.2.1.cmml" xref="A2.Ex14.m1.1.1.1.1.1.2.1">conditional</csymbol><ci id="A2.Ex14.m1.1.1.1.1.1.2.2.cmml" xref="A2.Ex14.m1.1.1.1.1.1.2.2">𝐳</ci><ci id="A2.Ex14.m1.1.1.1.1.1.2.3.cmml" xref="A2.Ex14.m1.1.1.1.1.1.2.3">𝐱</ci></apply><ci id="A2.Ex14.m1.1.1.1.1.1.3.cmml" xref="A2.Ex14.m1.1.1.1.1.1.3">𝑥</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.Ex14.m1.1c">\displaystyle\hat{\mathbb{E}}[\mathbf{z}|\mathbf{x}=x]</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A2.Ex14.m2.3" class="ltx_Math" alttext="\displaystyle=\int\frac{\hat{p}(x,z)}{\hat{p}(x)}~{}z~{}dz" display="inline"><semantics id="A2.Ex14.m2.3a"><mrow id="A2.Ex14.m2.3.4" xref="A2.Ex14.m2.3.4.cmml"><mi id="A2.Ex14.m2.3.4.2" xref="A2.Ex14.m2.3.4.2.cmml"></mi><mo id="A2.Ex14.m2.3.4.1" xref="A2.Ex14.m2.3.4.1.cmml">=</mo><mstyle displaystyle="true" id="A2.Ex14.m2.3.4.3" xref="A2.Ex14.m2.3.4.3.cmml"><mrow id="A2.Ex14.m2.3.4.3a" xref="A2.Ex14.m2.3.4.3.cmml"><mo id="A2.Ex14.m2.3.4.3.1" xref="A2.Ex14.m2.3.4.3.1.cmml">∫</mo><mrow id="A2.Ex14.m2.3.4.3.2" xref="A2.Ex14.m2.3.4.3.2.cmml"><mfrac id="A2.Ex14.m2.3.3" xref="A2.Ex14.m2.3.3.cmml"><mrow id="A2.Ex14.m2.2.2.2" xref="A2.Ex14.m2.2.2.2.cmml"><mover accent="true" id="A2.Ex14.m2.2.2.2.4" xref="A2.Ex14.m2.2.2.2.4.cmml"><mi id="A2.Ex14.m2.2.2.2.4.2" xref="A2.Ex14.m2.2.2.2.4.2.cmml">p</mi><mo id="A2.Ex14.m2.2.2.2.4.1" xref="A2.Ex14.m2.2.2.2.4.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A2.Ex14.m2.2.2.2.3" xref="A2.Ex14.m2.2.2.2.3.cmml">​</mo><mrow id="A2.Ex14.m2.2.2.2.5.2" xref="A2.Ex14.m2.2.2.2.5.1.cmml"><mo stretchy="false" id="A2.Ex14.m2.2.2.2.5.2.1" xref="A2.Ex14.m2.2.2.2.5.1.cmml">(</mo><mi id="A2.Ex14.m2.1.1.1.1" xref="A2.Ex14.m2.1.1.1.1.cmml">x</mi><mo id="A2.Ex14.m2.2.2.2.5.2.2" xref="A2.Ex14.m2.2.2.2.5.1.cmml">,</mo><mi id="A2.Ex14.m2.2.2.2.2" xref="A2.Ex14.m2.2.2.2.2.cmml">z</mi><mo stretchy="false" id="A2.Ex14.m2.2.2.2.5.2.3" xref="A2.Ex14.m2.2.2.2.5.1.cmml">)</mo></mrow></mrow><mrow id="A2.Ex14.m2.3.3.3" xref="A2.Ex14.m2.3.3.3.cmml"><mover accent="true" id="A2.Ex14.m2.3.3.3.3" xref="A2.Ex14.m2.3.3.3.3.cmml"><mi id="A2.Ex14.m2.3.3.3.3.2" xref="A2.Ex14.m2.3.3.3.3.2.cmml">p</mi><mo id="A2.Ex14.m2.3.3.3.3.1" xref="A2.Ex14.m2.3.3.3.3.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A2.Ex14.m2.3.3.3.2" xref="A2.Ex14.m2.3.3.3.2.cmml">​</mo><mrow id="A2.Ex14.m2.3.3.3.4.2" xref="A2.Ex14.m2.3.3.3.cmml"><mo stretchy="false" id="A2.Ex14.m2.3.3.3.4.2.1" xref="A2.Ex14.m2.3.3.3.cmml">(</mo><mi id="A2.Ex14.m2.3.3.3.1" xref="A2.Ex14.m2.3.3.3.1.cmml">x</mi><mo stretchy="false" id="A2.Ex14.m2.3.3.3.4.2.2" xref="A2.Ex14.m2.3.3.3.cmml">)</mo></mrow></mrow></mfrac><mo lspace="0.330em" rspace="0em" id="A2.Ex14.m2.3.4.3.2.1" xref="A2.Ex14.m2.3.4.3.2.1.cmml">​</mo><mi id="A2.Ex14.m2.3.4.3.2.2" xref="A2.Ex14.m2.3.4.3.2.2.cmml">z</mi><mo lspace="0.330em" rspace="0em" id="A2.Ex14.m2.3.4.3.2.1a" xref="A2.Ex14.m2.3.4.3.2.1.cmml">​</mo><mrow id="A2.Ex14.m2.3.4.3.2.3" xref="A2.Ex14.m2.3.4.3.2.3.cmml"><mo rspace="0em" id="A2.Ex14.m2.3.4.3.2.3.1" xref="A2.Ex14.m2.3.4.3.2.3.1.cmml">𝑑</mo><mi id="A2.Ex14.m2.3.4.3.2.3.2" xref="A2.Ex14.m2.3.4.3.2.3.2.cmml">z</mi></mrow></mrow></mrow></mstyle></mrow><annotation-xml encoding="MathML-Content" id="A2.Ex14.m2.3b"><apply id="A2.Ex14.m2.3.4.cmml" xref="A2.Ex14.m2.3.4"><eq id="A2.Ex14.m2.3.4.1.cmml" xref="A2.Ex14.m2.3.4.1"></eq><csymbol cd="latexml" id="A2.Ex14.m2.3.4.2.cmml" xref="A2.Ex14.m2.3.4.2">absent</csymbol><apply id="A2.Ex14.m2.3.4.3.cmml" xref="A2.Ex14.m2.3.4.3"><int id="A2.Ex14.m2.3.4.3.1.cmml" xref="A2.Ex14.m2.3.4.3.1"></int><apply id="A2.Ex14.m2.3.4.3.2.cmml" xref="A2.Ex14.m2.3.4.3.2"><times id="A2.Ex14.m2.3.4.3.2.1.cmml" xref="A2.Ex14.m2.3.4.3.2.1"></times><apply id="A2.Ex14.m2.3.3.cmml" xref="A2.Ex14.m2.3.3"><divide id="A2.Ex14.m2.3.3.4.cmml" xref="A2.Ex14.m2.3.3"></divide><apply id="A2.Ex14.m2.2.2.2.cmml" xref="A2.Ex14.m2.2.2.2"><times id="A2.Ex14.m2.2.2.2.3.cmml" xref="A2.Ex14.m2.2.2.2.3"></times><apply id="A2.Ex14.m2.2.2.2.4.cmml" xref="A2.Ex14.m2.2.2.2.4"><ci id="A2.Ex14.m2.2.2.2.4.1.cmml" xref="A2.Ex14.m2.2.2.2.4.1">^</ci><ci id="A2.Ex14.m2.2.2.2.4.2.cmml" xref="A2.Ex14.m2.2.2.2.4.2">𝑝</ci></apply><interval closure="open" id="A2.Ex14.m2.2.2.2.5.1.cmml" xref="A2.Ex14.m2.2.2.2.5.2"><ci id="A2.Ex14.m2.1.1.1.1.cmml" xref="A2.Ex14.m2.1.1.1.1">𝑥</ci><ci id="A2.Ex14.m2.2.2.2.2.cmml" xref="A2.Ex14.m2.2.2.2.2">𝑧</ci></interval></apply><apply id="A2.Ex14.m2.3.3.3.cmml" xref="A2.Ex14.m2.3.3.3"><times id="A2.Ex14.m2.3.3.3.2.cmml" xref="A2.Ex14.m2.3.3.3.2"></times><apply id="A2.Ex14.m2.3.3.3.3.cmml" xref="A2.Ex14.m2.3.3.3.3"><ci id="A2.Ex14.m2.3.3.3.3.1.cmml" xref="A2.Ex14.m2.3.3.3.3.1">^</ci><ci id="A2.Ex14.m2.3.3.3.3.2.cmml" xref="A2.Ex14.m2.3.3.3.3.2">𝑝</ci></apply><ci id="A2.Ex14.m2.3.3.3.1.cmml" xref="A2.Ex14.m2.3.3.3.1">𝑥</ci></apply></apply><ci id="A2.Ex14.m2.3.4.3.2.2.cmml" xref="A2.Ex14.m2.3.4.3.2.2">𝑧</ci><apply id="A2.Ex14.m2.3.4.3.2.3.cmml" xref="A2.Ex14.m2.3.4.3.2.3"><csymbol cd="latexml" id="A2.Ex14.m2.3.4.3.2.3.1.cmml" xref="A2.Ex14.m2.3.4.3.2.3.1">differential-d</csymbol><ci id="A2.Ex14.m2.3.4.3.2.3.2.cmml" xref="A2.Ex14.m2.3.4.3.2.3.2">𝑧</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.Ex14.m2.3c">\displaystyle=\int\frac{\hat{p}(x,z)}{\hat{p}(x)}~{}z~{}dz</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="A2.Ex15"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A2.Ex15.m1.3" class="ltx_Math" alttext="\displaystyle=\frac{1}{\hat{p}(x)}\int\hat{p}(x,z)~{}z~{}dz" display="inline"><semantics id="A2.Ex15.m1.3a"><mrow id="A2.Ex15.m1.3.4" xref="A2.Ex15.m1.3.4.cmml"><mi id="A2.Ex15.m1.3.4.2" xref="A2.Ex15.m1.3.4.2.cmml"></mi><mo id="A2.Ex15.m1.3.4.1" xref="A2.Ex15.m1.3.4.1.cmml">=</mo><mrow id="A2.Ex15.m1.3.4.3" xref="A2.Ex15.m1.3.4.3.cmml"><mstyle displaystyle="true" id="A2.Ex15.m1.1.1" xref="A2.Ex15.m1.1.1.cmml"><mfrac id="A2.Ex15.m1.1.1a" xref="A2.Ex15.m1.1.1.cmml"><mn id="A2.Ex15.m1.1.1.3" xref="A2.Ex15.m1.1.1.3.cmml">1</mn><mrow id="A2.Ex15.m1.1.1.1" xref="A2.Ex15.m1.1.1.1.cmml"><mover accent="true" id="A2.Ex15.m1.1.1.1.3" xref="A2.Ex15.m1.1.1.1.3.cmml"><mi id="A2.Ex15.m1.1.1.1.3.2" xref="A2.Ex15.m1.1.1.1.3.2.cmml">p</mi><mo id="A2.Ex15.m1.1.1.1.3.1" xref="A2.Ex15.m1.1.1.1.3.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A2.Ex15.m1.1.1.1.2" xref="A2.Ex15.m1.1.1.1.2.cmml">​</mo><mrow id="A2.Ex15.m1.1.1.1.4.2" xref="A2.Ex15.m1.1.1.1.cmml"><mo stretchy="false" id="A2.Ex15.m1.1.1.1.4.2.1" xref="A2.Ex15.m1.1.1.1.cmml">(</mo><mi id="A2.Ex15.m1.1.1.1.1" xref="A2.Ex15.m1.1.1.1.1.cmml">x</mi><mo stretchy="false" id="A2.Ex15.m1.1.1.1.4.2.2" xref="A2.Ex15.m1.1.1.1.cmml">)</mo></mrow></mrow></mfrac></mstyle><mo lspace="0em" rspace="0em" id="A2.Ex15.m1.3.4.3.1" xref="A2.Ex15.m1.3.4.3.1.cmml">​</mo><mstyle displaystyle="true" id="A2.Ex15.m1.3.4.3.2" xref="A2.Ex15.m1.3.4.3.2.cmml"><mrow id="A2.Ex15.m1.3.4.3.2a" xref="A2.Ex15.m1.3.4.3.2.cmml"><mo id="A2.Ex15.m1.3.4.3.2.1" xref="A2.Ex15.m1.3.4.3.2.1.cmml">∫</mo><mrow id="A2.Ex15.m1.3.4.3.2.2" xref="A2.Ex15.m1.3.4.3.2.2.cmml"><mover accent="true" id="A2.Ex15.m1.3.4.3.2.2.2" xref="A2.Ex15.m1.3.4.3.2.2.2.cmml"><mi id="A2.Ex15.m1.3.4.3.2.2.2.2" xref="A2.Ex15.m1.3.4.3.2.2.2.2.cmml">p</mi><mo id="A2.Ex15.m1.3.4.3.2.2.2.1" xref="A2.Ex15.m1.3.4.3.2.2.2.1.cmml">^</mo></mover><mo lspace="0em" rspace="0em" id="A2.Ex15.m1.3.4.3.2.2.1" xref="A2.Ex15.m1.3.4.3.2.2.1.cmml">​</mo><mrow id="A2.Ex15.m1.3.4.3.2.2.3.2" xref="A2.Ex15.m1.3.4.3.2.2.3.1.cmml"><mo stretchy="false" id="A2.Ex15.m1.3.4.3.2.2.3.2.1" xref="A2.Ex15.m1.3.4.3.2.2.3.1.cmml">(</mo><mi id="A2.Ex15.m1.2.2" xref="A2.Ex15.m1.2.2.cmml">x</mi><mo id="A2.Ex15.m1.3.4.3.2.2.3.2.2" xref="A2.Ex15.m1.3.4.3.2.2.3.1.cmml">,</mo><mi id="A2.Ex15.m1.3.3" xref="A2.Ex15.m1.3.3.cmml">z</mi><mo stretchy="false" id="A2.Ex15.m1.3.4.3.2.2.3.2.3" xref="A2.Ex15.m1.3.4.3.2.2.3.1.cmml">)</mo></mrow><mo lspace="0.330em" rspace="0em" id="A2.Ex15.m1.3.4.3.2.2.1a" xref="A2.Ex15.m1.3.4.3.2.2.1.cmml">​</mo><mi id="A2.Ex15.m1.3.4.3.2.2.4" xref="A2.Ex15.m1.3.4.3.2.2.4.cmml">z</mi><mo lspace="0.330em" rspace="0em" id="A2.Ex15.m1.3.4.3.2.2.1b" xref="A2.Ex15.m1.3.4.3.2.2.1.cmml">​</mo><mrow id="A2.Ex15.m1.3.4.3.2.2.5" xref="A2.Ex15.m1.3.4.3.2.2.5.cmml"><mo rspace="0em" id="A2.Ex15.m1.3.4.3.2.2.5.1" xref="A2.Ex15.m1.3.4.3.2.2.5.1.cmml">𝑑</mo><mi id="A2.Ex15.m1.3.4.3.2.2.5.2" xref="A2.Ex15.m1.3.4.3.2.2.5.2.cmml">z</mi></mrow></mrow></mrow></mstyle></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.Ex15.m1.3b"><apply id="A2.Ex15.m1.3.4.cmml" xref="A2.Ex15.m1.3.4"><eq id="A2.Ex15.m1.3.4.1.cmml" xref="A2.Ex15.m1.3.4.1"></eq><csymbol cd="latexml" id="A2.Ex15.m1.3.4.2.cmml" xref="A2.Ex15.m1.3.4.2">absent</csymbol><apply id="A2.Ex15.m1.3.4.3.cmml" xref="A2.Ex15.m1.3.4.3"><times id="A2.Ex15.m1.3.4.3.1.cmml" xref="A2.Ex15.m1.3.4.3.1"></times><apply id="A2.Ex15.m1.1.1.cmml" xref="A2.Ex15.m1.1.1"><divide id="A2.Ex15.m1.1.1.2.cmml" xref="A2.Ex15.m1.1.1"></divide><cn type="integer" id="A2.Ex15.m1.1.1.3.cmml" xref="A2.Ex15.m1.1.1.3">1</cn><apply id="A2.Ex15.m1.1.1.1.cmml" xref="A2.Ex15.m1.1.1.1"><times id="A2.Ex15.m1.1.1.1.2.cmml" xref="A2.Ex15.m1.1.1.1.2"></times><apply id="A2.Ex15.m1.1.1.1.3.cmml" xref="A2.Ex15.m1.1.1.1.3"><ci id="A2.Ex15.m1.1.1.1.3.1.cmml" xref="A2.Ex15.m1.1.1.1.3.1">^</ci><ci id="A2.Ex15.m1.1.1.1.3.2.cmml" xref="A2.Ex15.m1.1.1.1.3.2">𝑝</ci></apply><ci id="A2.Ex15.m1.1.1.1.1.cmml" xref="A2.Ex15.m1.1.1.1.1">𝑥</ci></apply></apply><apply id="A2.Ex15.m1.3.4.3.2.cmml" xref="A2.Ex15.m1.3.4.3.2"><int id="A2.Ex15.m1.3.4.3.2.1.cmml" xref="A2.Ex15.m1.3.4.3.2.1"></int><apply id="A2.Ex15.m1.3.4.3.2.2.cmml" xref="A2.Ex15.m1.3.4.3.2.2"><times id="A2.Ex15.m1.3.4.3.2.2.1.cmml" xref="A2.Ex15.m1.3.4.3.2.2.1"></times><apply id="A2.Ex15.m1.3.4.3.2.2.2.cmml" xref="A2.Ex15.m1.3.4.3.2.2.2"><ci id="A2.Ex15.m1.3.4.3.2.2.2.1.cmml" xref="A2.Ex15.m1.3.4.3.2.2.2.1">^</ci><ci id="A2.Ex15.m1.3.4.3.2.2.2.2.cmml" xref="A2.Ex15.m1.3.4.3.2.2.2.2">𝑝</ci></apply><interval closure="open" id="A2.Ex15.m1.3.4.3.2.2.3.1.cmml" xref="A2.Ex15.m1.3.4.3.2.2.3.2"><ci id="A2.Ex15.m1.2.2.cmml" xref="A2.Ex15.m1.2.2">𝑥</ci><ci id="A2.Ex15.m1.3.3.cmml" xref="A2.Ex15.m1.3.3">𝑧</ci></interval><ci id="A2.Ex15.m1.3.4.3.2.2.4.cmml" xref="A2.Ex15.m1.3.4.3.2.2.4">𝑧</ci><apply id="A2.Ex15.m1.3.4.3.2.2.5.cmml" xref="A2.Ex15.m1.3.4.3.2.2.5"><csymbol cd="latexml" id="A2.Ex15.m1.3.4.3.2.2.5.1.cmml" xref="A2.Ex15.m1.3.4.3.2.2.5.1">differential-d</csymbol><ci id="A2.Ex15.m1.3.4.3.2.2.5.2.cmml" xref="A2.Ex15.m1.3.4.3.2.2.5.2">𝑧</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.Ex15.m1.3c">\displaystyle=\frac{1}{\hat{p}(x)}\int\hat{p}(x,z)~{}z~{}dz</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="A2.Ex16"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A2.Ex16.m1.6" class="ltx_Math" alttext="\displaystyle=\frac{1}{\sum_{i=1}^{n}\kappa(x,x_{i})}\int\sum_{i=1}^{n}\kappa(x,x_{i})~{}\kappa^{\prime}(z,z_{i})~{}z~{}dz" display="inline"><semantics id="A2.Ex16.m1.6a"><mrow id="A2.Ex16.m1.6.6" xref="A2.Ex16.m1.6.6.cmml"><mi id="A2.Ex16.m1.6.6.4" xref="A2.Ex16.m1.6.6.4.cmml"></mi><mo id="A2.Ex16.m1.6.6.3" xref="A2.Ex16.m1.6.6.3.cmml">=</mo><mrow id="A2.Ex16.m1.6.6.2" xref="A2.Ex16.m1.6.6.2.cmml"><mstyle displaystyle="true" id="A2.Ex16.m1.2.2" xref="A2.Ex16.m1.2.2.cmml"><mfrac id="A2.Ex16.m1.2.2a" xref="A2.Ex16.m1.2.2.cmml"><mn id="A2.Ex16.m1.2.2.4" xref="A2.Ex16.m1.2.2.4.cmml">1</mn><mrow id="A2.Ex16.m1.2.2.2" xref="A2.Ex16.m1.2.2.2.cmml"><msubsup id="A2.Ex16.m1.2.2.2.3" xref="A2.Ex16.m1.2.2.2.3.cmml"><mo id="A2.Ex16.m1.2.2.2.3.2.2" xref="A2.Ex16.m1.2.2.2.3.2.2.cmml">∑</mo><mrow id="A2.Ex16.m1.2.2.2.3.2.3" xref="A2.Ex16.m1.2.2.2.3.2.3.cmml"><mi id="A2.Ex16.m1.2.2.2.3.2.3.2" xref="A2.Ex16.m1.2.2.2.3.2.3.2.cmml">i</mi><mo id="A2.Ex16.m1.2.2.2.3.2.3.1" xref="A2.Ex16.m1.2.2.2.3.2.3.1.cmml">=</mo><mn id="A2.Ex16.m1.2.2.2.3.2.3.3" xref="A2.Ex16.m1.2.2.2.3.2.3.3.cmml">1</mn></mrow><mi id="A2.Ex16.m1.2.2.2.3.3" xref="A2.Ex16.m1.2.2.2.3.3.cmml">n</mi></msubsup><mrow id="A2.Ex16.m1.2.2.2.2" xref="A2.Ex16.m1.2.2.2.2.cmml"><mi id="A2.Ex16.m1.2.2.2.2.3" xref="A2.Ex16.m1.2.2.2.2.3.cmml">κ</mi><mo lspace="0em" rspace="0em" id="A2.Ex16.m1.2.2.2.2.2" xref="A2.Ex16.m1.2.2.2.2.2.cmml">​</mo><mrow id="A2.Ex16.m1.2.2.2.2.1.1" xref="A2.Ex16.m1.2.2.2.2.1.2.cmml"><mo stretchy="false" id="A2.Ex16.m1.2.2.2.2.1.1.2" xref="A2.Ex16.m1.2.2.2.2.1.2.cmml">(</mo><mi id="A2.Ex16.m1.1.1.1.1" xref="A2.Ex16.m1.1.1.1.1.cmml">x</mi><mo id="A2.Ex16.m1.2.2.2.2.1.1.3" xref="A2.Ex16.m1.2.2.2.2.1.2.cmml">,</mo><msub id="A2.Ex16.m1.2.2.2.2.1.1.1" xref="A2.Ex16.m1.2.2.2.2.1.1.1.cmml"><mi id="A2.Ex16.m1.2.2.2.2.1.1.1.2" xref="A2.Ex16.m1.2.2.2.2.1.1.1.2.cmml">x</mi><mi id="A2.Ex16.m1.2.2.2.2.1.1.1.3" xref="A2.Ex16.m1.2.2.2.2.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A2.Ex16.m1.2.2.2.2.1.1.4" xref="A2.Ex16.m1.2.2.2.2.1.2.cmml">)</mo></mrow></mrow></mrow></mfrac></mstyle><mo lspace="0em" rspace="0em" id="A2.Ex16.m1.6.6.2.3" xref="A2.Ex16.m1.6.6.2.3.cmml">​</mo><mstyle displaystyle="true" id="A2.Ex16.m1.6.6.2.2" xref="A2.Ex16.m1.6.6.2.2.cmml"><mrow id="A2.Ex16.m1.6.6.2.2a" xref="A2.Ex16.m1.6.6.2.2.cmml"><mo rspace="0em" id="A2.Ex16.m1.6.6.2.2.3" xref="A2.Ex16.m1.6.6.2.2.3.cmml">∫</mo><mrow id="A2.Ex16.m1.6.6.2.2.2" xref="A2.Ex16.m1.6.6.2.2.2.cmml"><munderover id="A2.Ex16.m1.6.6.2.2.2.3" xref="A2.Ex16.m1.6.6.2.2.2.3.cmml"><mo movablelimits="false" id="A2.Ex16.m1.6.6.2.2.2.3.2.2" xref="A2.Ex16.m1.6.6.2.2.2.3.2.2.cmml">∑</mo><mrow id="A2.Ex16.m1.6.6.2.2.2.3.2.3" xref="A2.Ex16.m1.6.6.2.2.2.3.2.3.cmml"><mi id="A2.Ex16.m1.6.6.2.2.2.3.2.3.2" xref="A2.Ex16.m1.6.6.2.2.2.3.2.3.2.cmml">i</mi><mo id="A2.Ex16.m1.6.6.2.2.2.3.2.3.1" xref="A2.Ex16.m1.6.6.2.2.2.3.2.3.1.cmml">=</mo><mn id="A2.Ex16.m1.6.6.2.2.2.3.2.3.3" xref="A2.Ex16.m1.6.6.2.2.2.3.2.3.3.cmml">1</mn></mrow><mi id="A2.Ex16.m1.6.6.2.2.2.3.3" xref="A2.Ex16.m1.6.6.2.2.2.3.3.cmml">n</mi></munderover><mrow id="A2.Ex16.m1.6.6.2.2.2.2" xref="A2.Ex16.m1.6.6.2.2.2.2.cmml"><mi id="A2.Ex16.m1.6.6.2.2.2.2.4" xref="A2.Ex16.m1.6.6.2.2.2.2.4.cmml">κ</mi><mo lspace="0em" rspace="0em" id="A2.Ex16.m1.6.6.2.2.2.2.3" xref="A2.Ex16.m1.6.6.2.2.2.2.3.cmml">​</mo><mrow id="A2.Ex16.m1.5.5.1.1.1.1.1.1" xref="A2.Ex16.m1.5.5.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A2.Ex16.m1.5.5.1.1.1.1.1.1.2" xref="A2.Ex16.m1.5.5.1.1.1.1.1.2.cmml">(</mo><mi id="A2.Ex16.m1.3.3" xref="A2.Ex16.m1.3.3.cmml">x</mi><mo id="A2.Ex16.m1.5.5.1.1.1.1.1.1.3" xref="A2.Ex16.m1.5.5.1.1.1.1.1.2.cmml">,</mo><msub id="A2.Ex16.m1.5.5.1.1.1.1.1.1.1" xref="A2.Ex16.m1.5.5.1.1.1.1.1.1.1.cmml"><mi id="A2.Ex16.m1.5.5.1.1.1.1.1.1.1.2" xref="A2.Ex16.m1.5.5.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="A2.Ex16.m1.5.5.1.1.1.1.1.1.1.3" xref="A2.Ex16.m1.5.5.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A2.Ex16.m1.5.5.1.1.1.1.1.1.4" xref="A2.Ex16.m1.5.5.1.1.1.1.1.2.cmml">)</mo></mrow><mo lspace="0.330em" rspace="0em" id="A2.Ex16.m1.6.6.2.2.2.2.3a" xref="A2.Ex16.m1.6.6.2.2.2.2.3.cmml">​</mo><msup id="A2.Ex16.m1.6.6.2.2.2.2.5" xref="A2.Ex16.m1.6.6.2.2.2.2.5.cmml"><mi id="A2.Ex16.m1.6.6.2.2.2.2.5.2" xref="A2.Ex16.m1.6.6.2.2.2.2.5.2.cmml">κ</mi><mo id="A2.Ex16.m1.6.6.2.2.2.2.5.3" xref="A2.Ex16.m1.6.6.2.2.2.2.5.3.cmml">′</mo></msup><mo lspace="0em" rspace="0em" id="A2.Ex16.m1.6.6.2.2.2.2.3b" xref="A2.Ex16.m1.6.6.2.2.2.2.3.cmml">​</mo><mrow id="A2.Ex16.m1.6.6.2.2.2.2.2.1" xref="A2.Ex16.m1.6.6.2.2.2.2.2.2.cmml"><mo stretchy="false" id="A2.Ex16.m1.6.6.2.2.2.2.2.1.2" xref="A2.Ex16.m1.6.6.2.2.2.2.2.2.cmml">(</mo><mi id="A2.Ex16.m1.4.4" xref="A2.Ex16.m1.4.4.cmml">z</mi><mo id="A2.Ex16.m1.6.6.2.2.2.2.2.1.3" xref="A2.Ex16.m1.6.6.2.2.2.2.2.2.cmml">,</mo><msub id="A2.Ex16.m1.6.6.2.2.2.2.2.1.1" xref="A2.Ex16.m1.6.6.2.2.2.2.2.1.1.cmml"><mi id="A2.Ex16.m1.6.6.2.2.2.2.2.1.1.2" xref="A2.Ex16.m1.6.6.2.2.2.2.2.1.1.2.cmml">z</mi><mi id="A2.Ex16.m1.6.6.2.2.2.2.2.1.1.3" xref="A2.Ex16.m1.6.6.2.2.2.2.2.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A2.Ex16.m1.6.6.2.2.2.2.2.1.4" xref="A2.Ex16.m1.6.6.2.2.2.2.2.2.cmml">)</mo></mrow><mo lspace="0.330em" rspace="0em" id="A2.Ex16.m1.6.6.2.2.2.2.3c" xref="A2.Ex16.m1.6.6.2.2.2.2.3.cmml">​</mo><mi id="A2.Ex16.m1.6.6.2.2.2.2.6" xref="A2.Ex16.m1.6.6.2.2.2.2.6.cmml">z</mi><mo lspace="0.330em" rspace="0em" id="A2.Ex16.m1.6.6.2.2.2.2.3d" xref="A2.Ex16.m1.6.6.2.2.2.2.3.cmml">​</mo><mi id="A2.Ex16.m1.6.6.2.2.2.2.7" xref="A2.Ex16.m1.6.6.2.2.2.2.7.cmml">d</mi><mo lspace="0em" rspace="0em" id="A2.Ex16.m1.6.6.2.2.2.2.3e" xref="A2.Ex16.m1.6.6.2.2.2.2.3.cmml">​</mo><mi id="A2.Ex16.m1.6.6.2.2.2.2.8" xref="A2.Ex16.m1.6.6.2.2.2.2.8.cmml">z</mi></mrow></mrow></mrow></mstyle></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.Ex16.m1.6b"><apply id="A2.Ex16.m1.6.6.cmml" xref="A2.Ex16.m1.6.6"><eq id="A2.Ex16.m1.6.6.3.cmml" xref="A2.Ex16.m1.6.6.3"></eq><csymbol cd="latexml" id="A2.Ex16.m1.6.6.4.cmml" xref="A2.Ex16.m1.6.6.4">absent</csymbol><apply id="A2.Ex16.m1.6.6.2.cmml" xref="A2.Ex16.m1.6.6.2"><times id="A2.Ex16.m1.6.6.2.3.cmml" xref="A2.Ex16.m1.6.6.2.3"></times><apply id="A2.Ex16.m1.2.2.cmml" xref="A2.Ex16.m1.2.2"><divide id="A2.Ex16.m1.2.2.3.cmml" xref="A2.Ex16.m1.2.2"></divide><cn type="integer" id="A2.Ex16.m1.2.2.4.cmml" xref="A2.Ex16.m1.2.2.4">1</cn><apply id="A2.Ex16.m1.2.2.2.cmml" xref="A2.Ex16.m1.2.2.2"><apply id="A2.Ex16.m1.2.2.2.3.cmml" xref="A2.Ex16.m1.2.2.2.3"><csymbol cd="ambiguous" id="A2.Ex16.m1.2.2.2.3.1.cmml" xref="A2.Ex16.m1.2.2.2.3">superscript</csymbol><apply id="A2.Ex16.m1.2.2.2.3.2.cmml" xref="A2.Ex16.m1.2.2.2.3"><csymbol cd="ambiguous" id="A2.Ex16.m1.2.2.2.3.2.1.cmml" xref="A2.Ex16.m1.2.2.2.3">subscript</csymbol><sum id="A2.Ex16.m1.2.2.2.3.2.2.cmml" xref="A2.Ex16.m1.2.2.2.3.2.2"></sum><apply id="A2.Ex16.m1.2.2.2.3.2.3.cmml" xref="A2.Ex16.m1.2.2.2.3.2.3"><eq id="A2.Ex16.m1.2.2.2.3.2.3.1.cmml" xref="A2.Ex16.m1.2.2.2.3.2.3.1"></eq><ci id="A2.Ex16.m1.2.2.2.3.2.3.2.cmml" xref="A2.Ex16.m1.2.2.2.3.2.3.2">𝑖</ci><cn type="integer" id="A2.Ex16.m1.2.2.2.3.2.3.3.cmml" xref="A2.Ex16.m1.2.2.2.3.2.3.3">1</cn></apply></apply><ci id="A2.Ex16.m1.2.2.2.3.3.cmml" xref="A2.Ex16.m1.2.2.2.3.3">𝑛</ci></apply><apply id="A2.Ex16.m1.2.2.2.2.cmml" xref="A2.Ex16.m1.2.2.2.2"><times id="A2.Ex16.m1.2.2.2.2.2.cmml" xref="A2.Ex16.m1.2.2.2.2.2"></times><ci id="A2.Ex16.m1.2.2.2.2.3.cmml" xref="A2.Ex16.m1.2.2.2.2.3">𝜅</ci><interval closure="open" id="A2.Ex16.m1.2.2.2.2.1.2.cmml" xref="A2.Ex16.m1.2.2.2.2.1.1"><ci id="A2.Ex16.m1.1.1.1.1.cmml" xref="A2.Ex16.m1.1.1.1.1">𝑥</ci><apply id="A2.Ex16.m1.2.2.2.2.1.1.1.cmml" xref="A2.Ex16.m1.2.2.2.2.1.1.1"><csymbol cd="ambiguous" id="A2.Ex16.m1.2.2.2.2.1.1.1.1.cmml" xref="A2.Ex16.m1.2.2.2.2.1.1.1">subscript</csymbol><ci id="A2.Ex16.m1.2.2.2.2.1.1.1.2.cmml" xref="A2.Ex16.m1.2.2.2.2.1.1.1.2">𝑥</ci><ci id="A2.Ex16.m1.2.2.2.2.1.1.1.3.cmml" xref="A2.Ex16.m1.2.2.2.2.1.1.1.3">𝑖</ci></apply></interval></apply></apply></apply><apply id="A2.Ex16.m1.6.6.2.2.cmml" xref="A2.Ex16.m1.6.6.2.2"><int id="A2.Ex16.m1.6.6.2.2.3.cmml" xref="A2.Ex16.m1.6.6.2.2.3"></int><apply id="A2.Ex16.m1.6.6.2.2.2.cmml" xref="A2.Ex16.m1.6.6.2.2.2"><apply id="A2.Ex16.m1.6.6.2.2.2.3.cmml" xref="A2.Ex16.m1.6.6.2.2.2.3"><csymbol cd="ambiguous" id="A2.Ex16.m1.6.6.2.2.2.3.1.cmml" xref="A2.Ex16.m1.6.6.2.2.2.3">superscript</csymbol><apply id="A2.Ex16.m1.6.6.2.2.2.3.2.cmml" xref="A2.Ex16.m1.6.6.2.2.2.3"><csymbol cd="ambiguous" id="A2.Ex16.m1.6.6.2.2.2.3.2.1.cmml" xref="A2.Ex16.m1.6.6.2.2.2.3">subscript</csymbol><sum id="A2.Ex16.m1.6.6.2.2.2.3.2.2.cmml" xref="A2.Ex16.m1.6.6.2.2.2.3.2.2"></sum><apply id="A2.Ex16.m1.6.6.2.2.2.3.2.3.cmml" xref="A2.Ex16.m1.6.6.2.2.2.3.2.3"><eq id="A2.Ex16.m1.6.6.2.2.2.3.2.3.1.cmml" xref="A2.Ex16.m1.6.6.2.2.2.3.2.3.1"></eq><ci id="A2.Ex16.m1.6.6.2.2.2.3.2.3.2.cmml" xref="A2.Ex16.m1.6.6.2.2.2.3.2.3.2">𝑖</ci><cn type="integer" id="A2.Ex16.m1.6.6.2.2.2.3.2.3.3.cmml" xref="A2.Ex16.m1.6.6.2.2.2.3.2.3.3">1</cn></apply></apply><ci id="A2.Ex16.m1.6.6.2.2.2.3.3.cmml" xref="A2.Ex16.m1.6.6.2.2.2.3.3">𝑛</ci></apply><apply id="A2.Ex16.m1.6.6.2.2.2.2.cmml" xref="A2.Ex16.m1.6.6.2.2.2.2"><times id="A2.Ex16.m1.6.6.2.2.2.2.3.cmml" xref="A2.Ex16.m1.6.6.2.2.2.2.3"></times><ci id="A2.Ex16.m1.6.6.2.2.2.2.4.cmml" xref="A2.Ex16.m1.6.6.2.2.2.2.4">𝜅</ci><interval closure="open" id="A2.Ex16.m1.5.5.1.1.1.1.1.2.cmml" xref="A2.Ex16.m1.5.5.1.1.1.1.1.1"><ci id="A2.Ex16.m1.3.3.cmml" xref="A2.Ex16.m1.3.3">𝑥</ci><apply id="A2.Ex16.m1.5.5.1.1.1.1.1.1.1.cmml" xref="A2.Ex16.m1.5.5.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A2.Ex16.m1.5.5.1.1.1.1.1.1.1.1.cmml" xref="A2.Ex16.m1.5.5.1.1.1.1.1.1.1">subscript</csymbol><ci id="A2.Ex16.m1.5.5.1.1.1.1.1.1.1.2.cmml" xref="A2.Ex16.m1.5.5.1.1.1.1.1.1.1.2">𝑥</ci><ci id="A2.Ex16.m1.5.5.1.1.1.1.1.1.1.3.cmml" xref="A2.Ex16.m1.5.5.1.1.1.1.1.1.1.3">𝑖</ci></apply></interval><apply id="A2.Ex16.m1.6.6.2.2.2.2.5.cmml" xref="A2.Ex16.m1.6.6.2.2.2.2.5"><csymbol cd="ambiguous" id="A2.Ex16.m1.6.6.2.2.2.2.5.1.cmml" xref="A2.Ex16.m1.6.6.2.2.2.2.5">superscript</csymbol><ci id="A2.Ex16.m1.6.6.2.2.2.2.5.2.cmml" xref="A2.Ex16.m1.6.6.2.2.2.2.5.2">𝜅</ci><ci id="A2.Ex16.m1.6.6.2.2.2.2.5.3.cmml" xref="A2.Ex16.m1.6.6.2.2.2.2.5.3">′</ci></apply><interval closure="open" id="A2.Ex16.m1.6.6.2.2.2.2.2.2.cmml" xref="A2.Ex16.m1.6.6.2.2.2.2.2.1"><ci id="A2.Ex16.m1.4.4.cmml" xref="A2.Ex16.m1.4.4">𝑧</ci><apply id="A2.Ex16.m1.6.6.2.2.2.2.2.1.1.cmml" xref="A2.Ex16.m1.6.6.2.2.2.2.2.1.1"><csymbol cd="ambiguous" id="A2.Ex16.m1.6.6.2.2.2.2.2.1.1.1.cmml" xref="A2.Ex16.m1.6.6.2.2.2.2.2.1.1">subscript</csymbol><ci id="A2.Ex16.m1.6.6.2.2.2.2.2.1.1.2.cmml" xref="A2.Ex16.m1.6.6.2.2.2.2.2.1.1.2">𝑧</ci><ci id="A2.Ex16.m1.6.6.2.2.2.2.2.1.1.3.cmml" xref="A2.Ex16.m1.6.6.2.2.2.2.2.1.1.3">𝑖</ci></apply></interval><ci id="A2.Ex16.m1.6.6.2.2.2.2.6.cmml" xref="A2.Ex16.m1.6.6.2.2.2.2.6">𝑧</ci><ci id="A2.Ex16.m1.6.6.2.2.2.2.7.cmml" xref="A2.Ex16.m1.6.6.2.2.2.2.7">𝑑</ci><ci id="A2.Ex16.m1.6.6.2.2.2.2.8.cmml" xref="A2.Ex16.m1.6.6.2.2.2.2.8">𝑧</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.Ex16.m1.6c">\displaystyle=\frac{1}{\sum_{i=1}^{n}\kappa(x,x_{i})}\int\sum_{i=1}^{n}\kappa(x,x_{i})~{}\kappa^{\prime}(z,z_{i})~{}z~{}dz</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="A2.Ex17"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A2.Ex17.m1.6" class="ltx_Math" alttext="\displaystyle=\frac{1}{\sum_{i=1}^{n}\kappa(x,x_{i})}\sum_{i=1}^{n}\kappa(x,x_{i})~{}\int\kappa^{\prime}(z,z_{i})~{}z~{}dz" display="inline"><semantics id="A2.Ex17.m1.6a"><mrow id="A2.Ex17.m1.6.6" xref="A2.Ex17.m1.6.6.cmml"><mi id="A2.Ex17.m1.6.6.4" xref="A2.Ex17.m1.6.6.4.cmml"></mi><mo id="A2.Ex17.m1.6.6.3" xref="A2.Ex17.m1.6.6.3.cmml">=</mo><mrow id="A2.Ex17.m1.6.6.2" xref="A2.Ex17.m1.6.6.2.cmml"><mstyle displaystyle="true" id="A2.Ex17.m1.2.2" xref="A2.Ex17.m1.2.2.cmml"><mfrac id="A2.Ex17.m1.2.2a" xref="A2.Ex17.m1.2.2.cmml"><mn id="A2.Ex17.m1.2.2.4" xref="A2.Ex17.m1.2.2.4.cmml">1</mn><mrow id="A2.Ex17.m1.2.2.2" xref="A2.Ex17.m1.2.2.2.cmml"><msubsup id="A2.Ex17.m1.2.2.2.3" xref="A2.Ex17.m1.2.2.2.3.cmml"><mo id="A2.Ex17.m1.2.2.2.3.2.2" xref="A2.Ex17.m1.2.2.2.3.2.2.cmml">∑</mo><mrow id="A2.Ex17.m1.2.2.2.3.2.3" xref="A2.Ex17.m1.2.2.2.3.2.3.cmml"><mi id="A2.Ex17.m1.2.2.2.3.2.3.2" xref="A2.Ex17.m1.2.2.2.3.2.3.2.cmml">i</mi><mo id="A2.Ex17.m1.2.2.2.3.2.3.1" xref="A2.Ex17.m1.2.2.2.3.2.3.1.cmml">=</mo><mn id="A2.Ex17.m1.2.2.2.3.2.3.3" xref="A2.Ex17.m1.2.2.2.3.2.3.3.cmml">1</mn></mrow><mi id="A2.Ex17.m1.2.2.2.3.3" xref="A2.Ex17.m1.2.2.2.3.3.cmml">n</mi></msubsup><mrow id="A2.Ex17.m1.2.2.2.2" xref="A2.Ex17.m1.2.2.2.2.cmml"><mi id="A2.Ex17.m1.2.2.2.2.3" xref="A2.Ex17.m1.2.2.2.2.3.cmml">κ</mi><mo lspace="0em" rspace="0em" id="A2.Ex17.m1.2.2.2.2.2" xref="A2.Ex17.m1.2.2.2.2.2.cmml">​</mo><mrow id="A2.Ex17.m1.2.2.2.2.1.1" xref="A2.Ex17.m1.2.2.2.2.1.2.cmml"><mo stretchy="false" id="A2.Ex17.m1.2.2.2.2.1.1.2" xref="A2.Ex17.m1.2.2.2.2.1.2.cmml">(</mo><mi id="A2.Ex17.m1.1.1.1.1" xref="A2.Ex17.m1.1.1.1.1.cmml">x</mi><mo id="A2.Ex17.m1.2.2.2.2.1.1.3" xref="A2.Ex17.m1.2.2.2.2.1.2.cmml">,</mo><msub id="A2.Ex17.m1.2.2.2.2.1.1.1" xref="A2.Ex17.m1.2.2.2.2.1.1.1.cmml"><mi id="A2.Ex17.m1.2.2.2.2.1.1.1.2" xref="A2.Ex17.m1.2.2.2.2.1.1.1.2.cmml">x</mi><mi id="A2.Ex17.m1.2.2.2.2.1.1.1.3" xref="A2.Ex17.m1.2.2.2.2.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A2.Ex17.m1.2.2.2.2.1.1.4" xref="A2.Ex17.m1.2.2.2.2.1.2.cmml">)</mo></mrow></mrow></mrow></mfrac></mstyle><mo lspace="0em" rspace="0em" id="A2.Ex17.m1.6.6.2.3" xref="A2.Ex17.m1.6.6.2.3.cmml">​</mo><mrow id="A2.Ex17.m1.6.6.2.2" xref="A2.Ex17.m1.6.6.2.2.cmml"><mstyle displaystyle="true" id="A2.Ex17.m1.6.6.2.2.3" xref="A2.Ex17.m1.6.6.2.2.3.cmml"><munderover id="A2.Ex17.m1.6.6.2.2.3a" xref="A2.Ex17.m1.6.6.2.2.3.cmml"><mo movablelimits="false" id="A2.Ex17.m1.6.6.2.2.3.2.2" xref="A2.Ex17.m1.6.6.2.2.3.2.2.cmml">∑</mo><mrow id="A2.Ex17.m1.6.6.2.2.3.2.3" xref="A2.Ex17.m1.6.6.2.2.3.2.3.cmml"><mi id="A2.Ex17.m1.6.6.2.2.3.2.3.2" xref="A2.Ex17.m1.6.6.2.2.3.2.3.2.cmml">i</mi><mo id="A2.Ex17.m1.6.6.2.2.3.2.3.1" xref="A2.Ex17.m1.6.6.2.2.3.2.3.1.cmml">=</mo><mn id="A2.Ex17.m1.6.6.2.2.3.2.3.3" xref="A2.Ex17.m1.6.6.2.2.3.2.3.3.cmml">1</mn></mrow><mi id="A2.Ex17.m1.6.6.2.2.3.3" xref="A2.Ex17.m1.6.6.2.2.3.3.cmml">n</mi></munderover></mstyle><mrow id="A2.Ex17.m1.6.6.2.2.2" xref="A2.Ex17.m1.6.6.2.2.2.cmml"><mi id="A2.Ex17.m1.6.6.2.2.2.4" xref="A2.Ex17.m1.6.6.2.2.2.4.cmml">κ</mi><mo lspace="0em" rspace="0em" id="A2.Ex17.m1.6.6.2.2.2.3" xref="A2.Ex17.m1.6.6.2.2.2.3.cmml">​</mo><mrow id="A2.Ex17.m1.5.5.1.1.1.1.1" xref="A2.Ex17.m1.5.5.1.1.1.1.2.cmml"><mo stretchy="false" id="A2.Ex17.m1.5.5.1.1.1.1.1.2" xref="A2.Ex17.m1.5.5.1.1.1.1.2.cmml">(</mo><mi id="A2.Ex17.m1.3.3" xref="A2.Ex17.m1.3.3.cmml">x</mi><mo id="A2.Ex17.m1.5.5.1.1.1.1.1.3" xref="A2.Ex17.m1.5.5.1.1.1.1.2.cmml">,</mo><msub id="A2.Ex17.m1.5.5.1.1.1.1.1.1" xref="A2.Ex17.m1.5.5.1.1.1.1.1.1.cmml"><mi id="A2.Ex17.m1.5.5.1.1.1.1.1.1.2" xref="A2.Ex17.m1.5.5.1.1.1.1.1.1.2.cmml">x</mi><mi id="A2.Ex17.m1.5.5.1.1.1.1.1.1.3" xref="A2.Ex17.m1.5.5.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A2.Ex17.m1.5.5.1.1.1.1.1.4" xref="A2.Ex17.m1.5.5.1.1.1.1.2.cmml">)</mo></mrow><mo lspace="0.330em" rspace="0em" id="A2.Ex17.m1.6.6.2.2.2.3a" xref="A2.Ex17.m1.6.6.2.2.2.3.cmml">​</mo><mstyle displaystyle="true" id="A2.Ex17.m1.6.6.2.2.2.2" xref="A2.Ex17.m1.6.6.2.2.2.2.cmml"><mrow id="A2.Ex17.m1.6.6.2.2.2.2a" xref="A2.Ex17.m1.6.6.2.2.2.2.cmml"><mo id="A2.Ex17.m1.6.6.2.2.2.2.2" xref="A2.Ex17.m1.6.6.2.2.2.2.2.cmml">∫</mo><mrow id="A2.Ex17.m1.6.6.2.2.2.2.1" xref="A2.Ex17.m1.6.6.2.2.2.2.1.cmml"><msup id="A2.Ex17.m1.6.6.2.2.2.2.1.3" xref="A2.Ex17.m1.6.6.2.2.2.2.1.3.cmml"><mi id="A2.Ex17.m1.6.6.2.2.2.2.1.3.2" xref="A2.Ex17.m1.6.6.2.2.2.2.1.3.2.cmml">κ</mi><mo id="A2.Ex17.m1.6.6.2.2.2.2.1.3.3" xref="A2.Ex17.m1.6.6.2.2.2.2.1.3.3.cmml">′</mo></msup><mo lspace="0em" rspace="0em" id="A2.Ex17.m1.6.6.2.2.2.2.1.2" xref="A2.Ex17.m1.6.6.2.2.2.2.1.2.cmml">​</mo><mrow id="A2.Ex17.m1.6.6.2.2.2.2.1.1.1" xref="A2.Ex17.m1.6.6.2.2.2.2.1.1.2.cmml"><mo stretchy="false" id="A2.Ex17.m1.6.6.2.2.2.2.1.1.1.2" xref="A2.Ex17.m1.6.6.2.2.2.2.1.1.2.cmml">(</mo><mi id="A2.Ex17.m1.4.4" xref="A2.Ex17.m1.4.4.cmml">z</mi><mo id="A2.Ex17.m1.6.6.2.2.2.2.1.1.1.3" xref="A2.Ex17.m1.6.6.2.2.2.2.1.1.2.cmml">,</mo><msub id="A2.Ex17.m1.6.6.2.2.2.2.1.1.1.1" xref="A2.Ex17.m1.6.6.2.2.2.2.1.1.1.1.cmml"><mi id="A2.Ex17.m1.6.6.2.2.2.2.1.1.1.1.2" xref="A2.Ex17.m1.6.6.2.2.2.2.1.1.1.1.2.cmml">z</mi><mi id="A2.Ex17.m1.6.6.2.2.2.2.1.1.1.1.3" xref="A2.Ex17.m1.6.6.2.2.2.2.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A2.Ex17.m1.6.6.2.2.2.2.1.1.1.4" xref="A2.Ex17.m1.6.6.2.2.2.2.1.1.2.cmml">)</mo></mrow><mo lspace="0.330em" rspace="0em" id="A2.Ex17.m1.6.6.2.2.2.2.1.2a" xref="A2.Ex17.m1.6.6.2.2.2.2.1.2.cmml">​</mo><mi id="A2.Ex17.m1.6.6.2.2.2.2.1.4" xref="A2.Ex17.m1.6.6.2.2.2.2.1.4.cmml">z</mi><mo lspace="0.330em" rspace="0em" id="A2.Ex17.m1.6.6.2.2.2.2.1.2b" xref="A2.Ex17.m1.6.6.2.2.2.2.1.2.cmml">​</mo><mrow id="A2.Ex17.m1.6.6.2.2.2.2.1.5" xref="A2.Ex17.m1.6.6.2.2.2.2.1.5.cmml"><mo rspace="0em" id="A2.Ex17.m1.6.6.2.2.2.2.1.5.1" xref="A2.Ex17.m1.6.6.2.2.2.2.1.5.1.cmml">𝑑</mo><mi id="A2.Ex17.m1.6.6.2.2.2.2.1.5.2" xref="A2.Ex17.m1.6.6.2.2.2.2.1.5.2.cmml">z</mi></mrow></mrow></mrow></mstyle></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.Ex17.m1.6b"><apply id="A2.Ex17.m1.6.6.cmml" xref="A2.Ex17.m1.6.6"><eq id="A2.Ex17.m1.6.6.3.cmml" xref="A2.Ex17.m1.6.6.3"></eq><csymbol cd="latexml" id="A2.Ex17.m1.6.6.4.cmml" xref="A2.Ex17.m1.6.6.4">absent</csymbol><apply id="A2.Ex17.m1.6.6.2.cmml" xref="A2.Ex17.m1.6.6.2"><times id="A2.Ex17.m1.6.6.2.3.cmml" xref="A2.Ex17.m1.6.6.2.3"></times><apply id="A2.Ex17.m1.2.2.cmml" xref="A2.Ex17.m1.2.2"><divide id="A2.Ex17.m1.2.2.3.cmml" xref="A2.Ex17.m1.2.2"></divide><cn type="integer" id="A2.Ex17.m1.2.2.4.cmml" xref="A2.Ex17.m1.2.2.4">1</cn><apply id="A2.Ex17.m1.2.2.2.cmml" xref="A2.Ex17.m1.2.2.2"><apply id="A2.Ex17.m1.2.2.2.3.cmml" xref="A2.Ex17.m1.2.2.2.3"><csymbol cd="ambiguous" id="A2.Ex17.m1.2.2.2.3.1.cmml" xref="A2.Ex17.m1.2.2.2.3">superscript</csymbol><apply id="A2.Ex17.m1.2.2.2.3.2.cmml" xref="A2.Ex17.m1.2.2.2.3"><csymbol cd="ambiguous" id="A2.Ex17.m1.2.2.2.3.2.1.cmml" xref="A2.Ex17.m1.2.2.2.3">subscript</csymbol><sum id="A2.Ex17.m1.2.2.2.3.2.2.cmml" xref="A2.Ex17.m1.2.2.2.3.2.2"></sum><apply id="A2.Ex17.m1.2.2.2.3.2.3.cmml" xref="A2.Ex17.m1.2.2.2.3.2.3"><eq id="A2.Ex17.m1.2.2.2.3.2.3.1.cmml" xref="A2.Ex17.m1.2.2.2.3.2.3.1"></eq><ci id="A2.Ex17.m1.2.2.2.3.2.3.2.cmml" xref="A2.Ex17.m1.2.2.2.3.2.3.2">𝑖</ci><cn type="integer" id="A2.Ex17.m1.2.2.2.3.2.3.3.cmml" xref="A2.Ex17.m1.2.2.2.3.2.3.3">1</cn></apply></apply><ci id="A2.Ex17.m1.2.2.2.3.3.cmml" xref="A2.Ex17.m1.2.2.2.3.3">𝑛</ci></apply><apply id="A2.Ex17.m1.2.2.2.2.cmml" xref="A2.Ex17.m1.2.2.2.2"><times id="A2.Ex17.m1.2.2.2.2.2.cmml" xref="A2.Ex17.m1.2.2.2.2.2"></times><ci id="A2.Ex17.m1.2.2.2.2.3.cmml" xref="A2.Ex17.m1.2.2.2.2.3">𝜅</ci><interval closure="open" id="A2.Ex17.m1.2.2.2.2.1.2.cmml" xref="A2.Ex17.m1.2.2.2.2.1.1"><ci id="A2.Ex17.m1.1.1.1.1.cmml" xref="A2.Ex17.m1.1.1.1.1">𝑥</ci><apply id="A2.Ex17.m1.2.2.2.2.1.1.1.cmml" xref="A2.Ex17.m1.2.2.2.2.1.1.1"><csymbol cd="ambiguous" id="A2.Ex17.m1.2.2.2.2.1.1.1.1.cmml" xref="A2.Ex17.m1.2.2.2.2.1.1.1">subscript</csymbol><ci id="A2.Ex17.m1.2.2.2.2.1.1.1.2.cmml" xref="A2.Ex17.m1.2.2.2.2.1.1.1.2">𝑥</ci><ci id="A2.Ex17.m1.2.2.2.2.1.1.1.3.cmml" xref="A2.Ex17.m1.2.2.2.2.1.1.1.3">𝑖</ci></apply></interval></apply></apply></apply><apply id="A2.Ex17.m1.6.6.2.2.cmml" xref="A2.Ex17.m1.6.6.2.2"><apply id="A2.Ex17.m1.6.6.2.2.3.cmml" xref="A2.Ex17.m1.6.6.2.2.3"><csymbol cd="ambiguous" id="A2.Ex17.m1.6.6.2.2.3.1.cmml" xref="A2.Ex17.m1.6.6.2.2.3">superscript</csymbol><apply id="A2.Ex17.m1.6.6.2.2.3.2.cmml" xref="A2.Ex17.m1.6.6.2.2.3"><csymbol cd="ambiguous" id="A2.Ex17.m1.6.6.2.2.3.2.1.cmml" xref="A2.Ex17.m1.6.6.2.2.3">subscript</csymbol><sum id="A2.Ex17.m1.6.6.2.2.3.2.2.cmml" xref="A2.Ex17.m1.6.6.2.2.3.2.2"></sum><apply id="A2.Ex17.m1.6.6.2.2.3.2.3.cmml" xref="A2.Ex17.m1.6.6.2.2.3.2.3"><eq id="A2.Ex17.m1.6.6.2.2.3.2.3.1.cmml" xref="A2.Ex17.m1.6.6.2.2.3.2.3.1"></eq><ci id="A2.Ex17.m1.6.6.2.2.3.2.3.2.cmml" xref="A2.Ex17.m1.6.6.2.2.3.2.3.2">𝑖</ci><cn type="integer" id="A2.Ex17.m1.6.6.2.2.3.2.3.3.cmml" xref="A2.Ex17.m1.6.6.2.2.3.2.3.3">1</cn></apply></apply><ci id="A2.Ex17.m1.6.6.2.2.3.3.cmml" xref="A2.Ex17.m1.6.6.2.2.3.3">𝑛</ci></apply><apply id="A2.Ex17.m1.6.6.2.2.2.cmml" xref="A2.Ex17.m1.6.6.2.2.2"><times id="A2.Ex17.m1.6.6.2.2.2.3.cmml" xref="A2.Ex17.m1.6.6.2.2.2.3"></times><ci id="A2.Ex17.m1.6.6.2.2.2.4.cmml" xref="A2.Ex17.m1.6.6.2.2.2.4">𝜅</ci><interval closure="open" id="A2.Ex17.m1.5.5.1.1.1.1.2.cmml" xref="A2.Ex17.m1.5.5.1.1.1.1.1"><ci id="A2.Ex17.m1.3.3.cmml" xref="A2.Ex17.m1.3.3">𝑥</ci><apply id="A2.Ex17.m1.5.5.1.1.1.1.1.1.cmml" xref="A2.Ex17.m1.5.5.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A2.Ex17.m1.5.5.1.1.1.1.1.1.1.cmml" xref="A2.Ex17.m1.5.5.1.1.1.1.1.1">subscript</csymbol><ci id="A2.Ex17.m1.5.5.1.1.1.1.1.1.2.cmml" xref="A2.Ex17.m1.5.5.1.1.1.1.1.1.2">𝑥</ci><ci id="A2.Ex17.m1.5.5.1.1.1.1.1.1.3.cmml" xref="A2.Ex17.m1.5.5.1.1.1.1.1.1.3">𝑖</ci></apply></interval><apply id="A2.Ex17.m1.6.6.2.2.2.2.cmml" xref="A2.Ex17.m1.6.6.2.2.2.2"><int id="A2.Ex17.m1.6.6.2.2.2.2.2.cmml" xref="A2.Ex17.m1.6.6.2.2.2.2.2"></int><apply id="A2.Ex17.m1.6.6.2.2.2.2.1.cmml" xref="A2.Ex17.m1.6.6.2.2.2.2.1"><times id="A2.Ex17.m1.6.6.2.2.2.2.1.2.cmml" xref="A2.Ex17.m1.6.6.2.2.2.2.1.2"></times><apply id="A2.Ex17.m1.6.6.2.2.2.2.1.3.cmml" xref="A2.Ex17.m1.6.6.2.2.2.2.1.3"><csymbol cd="ambiguous" id="A2.Ex17.m1.6.6.2.2.2.2.1.3.1.cmml" xref="A2.Ex17.m1.6.6.2.2.2.2.1.3">superscript</csymbol><ci id="A2.Ex17.m1.6.6.2.2.2.2.1.3.2.cmml" xref="A2.Ex17.m1.6.6.2.2.2.2.1.3.2">𝜅</ci><ci id="A2.Ex17.m1.6.6.2.2.2.2.1.3.3.cmml" xref="A2.Ex17.m1.6.6.2.2.2.2.1.3.3">′</ci></apply><interval closure="open" id="A2.Ex17.m1.6.6.2.2.2.2.1.1.2.cmml" xref="A2.Ex17.m1.6.6.2.2.2.2.1.1.1"><ci id="A2.Ex17.m1.4.4.cmml" xref="A2.Ex17.m1.4.4">𝑧</ci><apply id="A2.Ex17.m1.6.6.2.2.2.2.1.1.1.1.cmml" xref="A2.Ex17.m1.6.6.2.2.2.2.1.1.1.1"><csymbol cd="ambiguous" id="A2.Ex17.m1.6.6.2.2.2.2.1.1.1.1.1.cmml" xref="A2.Ex17.m1.6.6.2.2.2.2.1.1.1.1">subscript</csymbol><ci id="A2.Ex17.m1.6.6.2.2.2.2.1.1.1.1.2.cmml" xref="A2.Ex17.m1.6.6.2.2.2.2.1.1.1.1.2">𝑧</ci><ci id="A2.Ex17.m1.6.6.2.2.2.2.1.1.1.1.3.cmml" xref="A2.Ex17.m1.6.6.2.2.2.2.1.1.1.1.3">𝑖</ci></apply></interval><ci id="A2.Ex17.m1.6.6.2.2.2.2.1.4.cmml" xref="A2.Ex17.m1.6.6.2.2.2.2.1.4">𝑧</ci><apply id="A2.Ex17.m1.6.6.2.2.2.2.1.5.cmml" xref="A2.Ex17.m1.6.6.2.2.2.2.1.5"><csymbol cd="latexml" id="A2.Ex17.m1.6.6.2.2.2.2.1.5.1.cmml" xref="A2.Ex17.m1.6.6.2.2.2.2.1.5.1">differential-d</csymbol><ci id="A2.Ex17.m1.6.6.2.2.2.2.1.5.2.cmml" xref="A2.Ex17.m1.6.6.2.2.2.2.1.5.2">𝑧</ci></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.Ex17.m1.6c">\displaystyle=\frac{1}{\sum_{i=1}^{n}\kappa(x,x_{i})}\sum_{i=1}^{n}\kappa(x,x_{i})~{}\int\kappa^{\prime}(z,z_{i})~{}z~{}dz</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="A2.Ex18"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A2.Ex18.m1.4" class="ltx_Math" alttext="\displaystyle=\frac{1}{\sum_{i=1}^{n}\kappa(x,x_{i})}\sum_{i=1}^{n}\kappa(x,x_{i})~{}z_{i}." display="inline"><semantics id="A2.Ex18.m1.4a"><mrow id="A2.Ex18.m1.4.4.1" xref="A2.Ex18.m1.4.4.1.1.cmml"><mrow id="A2.Ex18.m1.4.4.1.1" xref="A2.Ex18.m1.4.4.1.1.cmml"><mi id="A2.Ex18.m1.4.4.1.1.3" xref="A2.Ex18.m1.4.4.1.1.3.cmml"></mi><mo id="A2.Ex18.m1.4.4.1.1.2" xref="A2.Ex18.m1.4.4.1.1.2.cmml">=</mo><mrow id="A2.Ex18.m1.4.4.1.1.1" xref="A2.Ex18.m1.4.4.1.1.1.cmml"><mstyle displaystyle="true" id="A2.Ex18.m1.2.2" xref="A2.Ex18.m1.2.2.cmml"><mfrac id="A2.Ex18.m1.2.2a" xref="A2.Ex18.m1.2.2.cmml"><mn id="A2.Ex18.m1.2.2.4" xref="A2.Ex18.m1.2.2.4.cmml">1</mn><mrow id="A2.Ex18.m1.2.2.2" xref="A2.Ex18.m1.2.2.2.cmml"><msubsup id="A2.Ex18.m1.2.2.2.3" xref="A2.Ex18.m1.2.2.2.3.cmml"><mo id="A2.Ex18.m1.2.2.2.3.2.2" xref="A2.Ex18.m1.2.2.2.3.2.2.cmml">∑</mo><mrow id="A2.Ex18.m1.2.2.2.3.2.3" xref="A2.Ex18.m1.2.2.2.3.2.3.cmml"><mi id="A2.Ex18.m1.2.2.2.3.2.3.2" xref="A2.Ex18.m1.2.2.2.3.2.3.2.cmml">i</mi><mo id="A2.Ex18.m1.2.2.2.3.2.3.1" xref="A2.Ex18.m1.2.2.2.3.2.3.1.cmml">=</mo><mn id="A2.Ex18.m1.2.2.2.3.2.3.3" xref="A2.Ex18.m1.2.2.2.3.2.3.3.cmml">1</mn></mrow><mi id="A2.Ex18.m1.2.2.2.3.3" xref="A2.Ex18.m1.2.2.2.3.3.cmml">n</mi></msubsup><mrow id="A2.Ex18.m1.2.2.2.2" xref="A2.Ex18.m1.2.2.2.2.cmml"><mi id="A2.Ex18.m1.2.2.2.2.3" xref="A2.Ex18.m1.2.2.2.2.3.cmml">κ</mi><mo lspace="0em" rspace="0em" id="A2.Ex18.m1.2.2.2.2.2" xref="A2.Ex18.m1.2.2.2.2.2.cmml">​</mo><mrow id="A2.Ex18.m1.2.2.2.2.1.1" xref="A2.Ex18.m1.2.2.2.2.1.2.cmml"><mo stretchy="false" id="A2.Ex18.m1.2.2.2.2.1.1.2" xref="A2.Ex18.m1.2.2.2.2.1.2.cmml">(</mo><mi id="A2.Ex18.m1.1.1.1.1" xref="A2.Ex18.m1.1.1.1.1.cmml">x</mi><mo id="A2.Ex18.m1.2.2.2.2.1.1.3" xref="A2.Ex18.m1.2.2.2.2.1.2.cmml">,</mo><msub id="A2.Ex18.m1.2.2.2.2.1.1.1" xref="A2.Ex18.m1.2.2.2.2.1.1.1.cmml"><mi id="A2.Ex18.m1.2.2.2.2.1.1.1.2" xref="A2.Ex18.m1.2.2.2.2.1.1.1.2.cmml">x</mi><mi id="A2.Ex18.m1.2.2.2.2.1.1.1.3" xref="A2.Ex18.m1.2.2.2.2.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A2.Ex18.m1.2.2.2.2.1.1.4" xref="A2.Ex18.m1.2.2.2.2.1.2.cmml">)</mo></mrow></mrow></mrow></mfrac></mstyle><mo lspace="0em" rspace="0em" id="A2.Ex18.m1.4.4.1.1.1.2" xref="A2.Ex18.m1.4.4.1.1.1.2.cmml">​</mo><mrow id="A2.Ex18.m1.4.4.1.1.1.1" xref="A2.Ex18.m1.4.4.1.1.1.1.cmml"><mstyle displaystyle="true" id="A2.Ex18.m1.4.4.1.1.1.1.2" xref="A2.Ex18.m1.4.4.1.1.1.1.2.cmml"><munderover id="A2.Ex18.m1.4.4.1.1.1.1.2a" xref="A2.Ex18.m1.4.4.1.1.1.1.2.cmml"><mo movablelimits="false" id="A2.Ex18.m1.4.4.1.1.1.1.2.2.2" xref="A2.Ex18.m1.4.4.1.1.1.1.2.2.2.cmml">∑</mo><mrow id="A2.Ex18.m1.4.4.1.1.1.1.2.2.3" xref="A2.Ex18.m1.4.4.1.1.1.1.2.2.3.cmml"><mi id="A2.Ex18.m1.4.4.1.1.1.1.2.2.3.2" xref="A2.Ex18.m1.4.4.1.1.1.1.2.2.3.2.cmml">i</mi><mo id="A2.Ex18.m1.4.4.1.1.1.1.2.2.3.1" xref="A2.Ex18.m1.4.4.1.1.1.1.2.2.3.1.cmml">=</mo><mn id="A2.Ex18.m1.4.4.1.1.1.1.2.2.3.3" xref="A2.Ex18.m1.4.4.1.1.1.1.2.2.3.3.cmml">1</mn></mrow><mi id="A2.Ex18.m1.4.4.1.1.1.1.2.3" xref="A2.Ex18.m1.4.4.1.1.1.1.2.3.cmml">n</mi></munderover></mstyle><mrow id="A2.Ex18.m1.4.4.1.1.1.1.1" xref="A2.Ex18.m1.4.4.1.1.1.1.1.cmml"><mi id="A2.Ex18.m1.4.4.1.1.1.1.1.3" xref="A2.Ex18.m1.4.4.1.1.1.1.1.3.cmml">κ</mi><mo lspace="0em" rspace="0em" id="A2.Ex18.m1.4.4.1.1.1.1.1.2" xref="A2.Ex18.m1.4.4.1.1.1.1.1.2.cmml">​</mo><mrow id="A2.Ex18.m1.4.4.1.1.1.1.1.1.1" xref="A2.Ex18.m1.4.4.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A2.Ex18.m1.4.4.1.1.1.1.1.1.1.2" xref="A2.Ex18.m1.4.4.1.1.1.1.1.1.2.cmml">(</mo><mi id="A2.Ex18.m1.3.3" xref="A2.Ex18.m1.3.3.cmml">x</mi><mo id="A2.Ex18.m1.4.4.1.1.1.1.1.1.1.3" xref="A2.Ex18.m1.4.4.1.1.1.1.1.1.2.cmml">,</mo><msub id="A2.Ex18.m1.4.4.1.1.1.1.1.1.1.1" xref="A2.Ex18.m1.4.4.1.1.1.1.1.1.1.1.cmml"><mi id="A2.Ex18.m1.4.4.1.1.1.1.1.1.1.1.2" xref="A2.Ex18.m1.4.4.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="A2.Ex18.m1.4.4.1.1.1.1.1.1.1.1.3" xref="A2.Ex18.m1.4.4.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A2.Ex18.m1.4.4.1.1.1.1.1.1.1.4" xref="A2.Ex18.m1.4.4.1.1.1.1.1.1.2.cmml">)</mo></mrow><mo lspace="0.330em" rspace="0em" id="A2.Ex18.m1.4.4.1.1.1.1.1.2a" xref="A2.Ex18.m1.4.4.1.1.1.1.1.2.cmml">​</mo><msub id="A2.Ex18.m1.4.4.1.1.1.1.1.4" xref="A2.Ex18.m1.4.4.1.1.1.1.1.4.cmml"><mi id="A2.Ex18.m1.4.4.1.1.1.1.1.4.2" xref="A2.Ex18.m1.4.4.1.1.1.1.1.4.2.cmml">z</mi><mi id="A2.Ex18.m1.4.4.1.1.1.1.1.4.3" xref="A2.Ex18.m1.4.4.1.1.1.1.1.4.3.cmml">i</mi></msub></mrow></mrow></mrow></mrow><mo lspace="0em" id="A2.Ex18.m1.4.4.1.2" xref="A2.Ex18.m1.4.4.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A2.Ex18.m1.4b"><apply id="A2.Ex18.m1.4.4.1.1.cmml" xref="A2.Ex18.m1.4.4.1"><eq id="A2.Ex18.m1.4.4.1.1.2.cmml" xref="A2.Ex18.m1.4.4.1.1.2"></eq><csymbol cd="latexml" id="A2.Ex18.m1.4.4.1.1.3.cmml" xref="A2.Ex18.m1.4.4.1.1.3">absent</csymbol><apply id="A2.Ex18.m1.4.4.1.1.1.cmml" xref="A2.Ex18.m1.4.4.1.1.1"><times id="A2.Ex18.m1.4.4.1.1.1.2.cmml" xref="A2.Ex18.m1.4.4.1.1.1.2"></times><apply id="A2.Ex18.m1.2.2.cmml" xref="A2.Ex18.m1.2.2"><divide id="A2.Ex18.m1.2.2.3.cmml" xref="A2.Ex18.m1.2.2"></divide><cn type="integer" id="A2.Ex18.m1.2.2.4.cmml" xref="A2.Ex18.m1.2.2.4">1</cn><apply id="A2.Ex18.m1.2.2.2.cmml" xref="A2.Ex18.m1.2.2.2"><apply id="A2.Ex18.m1.2.2.2.3.cmml" xref="A2.Ex18.m1.2.2.2.3"><csymbol cd="ambiguous" id="A2.Ex18.m1.2.2.2.3.1.cmml" xref="A2.Ex18.m1.2.2.2.3">superscript</csymbol><apply id="A2.Ex18.m1.2.2.2.3.2.cmml" xref="A2.Ex18.m1.2.2.2.3"><csymbol cd="ambiguous" id="A2.Ex18.m1.2.2.2.3.2.1.cmml" xref="A2.Ex18.m1.2.2.2.3">subscript</csymbol><sum id="A2.Ex18.m1.2.2.2.3.2.2.cmml" xref="A2.Ex18.m1.2.2.2.3.2.2"></sum><apply id="A2.Ex18.m1.2.2.2.3.2.3.cmml" xref="A2.Ex18.m1.2.2.2.3.2.3"><eq id="A2.Ex18.m1.2.2.2.3.2.3.1.cmml" xref="A2.Ex18.m1.2.2.2.3.2.3.1"></eq><ci id="A2.Ex18.m1.2.2.2.3.2.3.2.cmml" xref="A2.Ex18.m1.2.2.2.3.2.3.2">𝑖</ci><cn type="integer" id="A2.Ex18.m1.2.2.2.3.2.3.3.cmml" xref="A2.Ex18.m1.2.2.2.3.2.3.3">1</cn></apply></apply><ci id="A2.Ex18.m1.2.2.2.3.3.cmml" xref="A2.Ex18.m1.2.2.2.3.3">𝑛</ci></apply><apply id="A2.Ex18.m1.2.2.2.2.cmml" xref="A2.Ex18.m1.2.2.2.2"><times id="A2.Ex18.m1.2.2.2.2.2.cmml" xref="A2.Ex18.m1.2.2.2.2.2"></times><ci id="A2.Ex18.m1.2.2.2.2.3.cmml" xref="A2.Ex18.m1.2.2.2.2.3">𝜅</ci><interval closure="open" id="A2.Ex18.m1.2.2.2.2.1.2.cmml" xref="A2.Ex18.m1.2.2.2.2.1.1"><ci id="A2.Ex18.m1.1.1.1.1.cmml" xref="A2.Ex18.m1.1.1.1.1">𝑥</ci><apply id="A2.Ex18.m1.2.2.2.2.1.1.1.cmml" xref="A2.Ex18.m1.2.2.2.2.1.1.1"><csymbol cd="ambiguous" id="A2.Ex18.m1.2.2.2.2.1.1.1.1.cmml" xref="A2.Ex18.m1.2.2.2.2.1.1.1">subscript</csymbol><ci id="A2.Ex18.m1.2.2.2.2.1.1.1.2.cmml" xref="A2.Ex18.m1.2.2.2.2.1.1.1.2">𝑥</ci><ci id="A2.Ex18.m1.2.2.2.2.1.1.1.3.cmml" xref="A2.Ex18.m1.2.2.2.2.1.1.1.3">𝑖</ci></apply></interval></apply></apply></apply><apply id="A2.Ex18.m1.4.4.1.1.1.1.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1"><apply id="A2.Ex18.m1.4.4.1.1.1.1.2.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.2"><csymbol cd="ambiguous" id="A2.Ex18.m1.4.4.1.1.1.1.2.1.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.2">superscript</csymbol><apply id="A2.Ex18.m1.4.4.1.1.1.1.2.2.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.2"><csymbol cd="ambiguous" id="A2.Ex18.m1.4.4.1.1.1.1.2.2.1.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.2">subscript</csymbol><sum id="A2.Ex18.m1.4.4.1.1.1.1.2.2.2.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.2.2.2"></sum><apply id="A2.Ex18.m1.4.4.1.1.1.1.2.2.3.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.2.2.3"><eq id="A2.Ex18.m1.4.4.1.1.1.1.2.2.3.1.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.2.2.3.1"></eq><ci id="A2.Ex18.m1.4.4.1.1.1.1.2.2.3.2.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.2.2.3.2">𝑖</ci><cn type="integer" id="A2.Ex18.m1.4.4.1.1.1.1.2.2.3.3.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.2.2.3.3">1</cn></apply></apply><ci id="A2.Ex18.m1.4.4.1.1.1.1.2.3.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.2.3">𝑛</ci></apply><apply id="A2.Ex18.m1.4.4.1.1.1.1.1.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.1"><times id="A2.Ex18.m1.4.4.1.1.1.1.1.2.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.1.2"></times><ci id="A2.Ex18.m1.4.4.1.1.1.1.1.3.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.1.3">𝜅</ci><interval closure="open" id="A2.Ex18.m1.4.4.1.1.1.1.1.1.2.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.1.1.1"><ci id="A2.Ex18.m1.3.3.cmml" xref="A2.Ex18.m1.3.3">𝑥</ci><apply id="A2.Ex18.m1.4.4.1.1.1.1.1.1.1.1.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A2.Ex18.m1.4.4.1.1.1.1.1.1.1.1.1.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="A2.Ex18.m1.4.4.1.1.1.1.1.1.1.1.2.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="A2.Ex18.m1.4.4.1.1.1.1.1.1.1.1.3.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.1.1.1.1.3">𝑖</ci></apply></interval><apply id="A2.Ex18.m1.4.4.1.1.1.1.1.4.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.1.4"><csymbol cd="ambiguous" id="A2.Ex18.m1.4.4.1.1.1.1.1.4.1.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.1.4">subscript</csymbol><ci id="A2.Ex18.m1.4.4.1.1.1.1.1.4.2.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.1.4.2">𝑧</ci><ci id="A2.Ex18.m1.4.4.1.1.1.1.1.4.3.cmml" xref="A2.Ex18.m1.4.4.1.1.1.1.1.4.3">𝑖</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.Ex18.m1.4c">\displaystyle=\frac{1}{\sum_{i=1}^{n}\kappa(x,x_{i})}\sum_{i=1}^{n}\kappa(x,x_{i})~{}z_{i}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
</section>
<section id="A2.SS0.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Asymmetric kernels.</h5>

<div id="A2.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="A2.SS0.SSS0.Px2.p1.6" class="ltx_p">In modern days, people think of kernels as positive semi-definite, which might not be guaranteed for <math id="A2.SS0.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="\kappa" display="inline"><semantics id="A2.SS0.SSS0.Px2.p1.1.m1.1a"><mi id="A2.SS0.SSS0.Px2.p1.1.m1.1.1" xref="A2.SS0.SSS0.Px2.p1.1.m1.1.1.cmml">κ</mi><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px2.p1.1.m1.1b"><ci id="A2.SS0.SSS0.Px2.p1.1.m1.1.1.cmml" xref="A2.SS0.SSS0.Px2.p1.1.m1.1.1">𝜅</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px2.p1.1.m1.1c">\kappa</annotation></semantics></math> unless <math id="A2.SS0.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="\theta_{K}=\theta_{Q}" display="inline"><semantics id="A2.SS0.SSS0.Px2.p1.2.m2.1a"><mrow id="A2.SS0.SSS0.Px2.p1.2.m2.1.1" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.cmml"><msub id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.cmml"><mi id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.2" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.2.cmml">θ</mi><mi id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.3" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.3.cmml">K</mi></msub><mo id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.1" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.1.cmml">=</mo><msub id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.cmml"><mi id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.2" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.2.cmml">θ</mi><mi id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.cmml">Q</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px2.p1.2.m2.1b"><apply id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1"><eq id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.1.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.1"></eq><apply id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.1.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2">subscript</csymbol><ci id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.2.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.2">𝜃</ci><ci id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.3.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.2.3">𝐾</ci></apply><apply id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3"><csymbol cd="ambiguous" id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.1.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3">subscript</csymbol><ci id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.2.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.2">𝜃</ci><ci id="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3.cmml" xref="A2.SS0.SSS0.Px2.p1.2.m2.1.1.3.3">𝑄</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px2.p1.2.m2.1c">\theta_{K}=\theta_{Q}</annotation></semantics></math>.
However, people working on kernels decades ago, around the time when the Nadaraya-Watson estimator was popular, have been very lenient with the choice of kernels, and asymmetric kernels such as our <math id="A2.SS0.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="\kappa" display="inline"><semantics id="A2.SS0.SSS0.Px2.p1.3.m3.1a"><mi id="A2.SS0.SSS0.Px2.p1.3.m3.1.1" xref="A2.SS0.SSS0.Px2.p1.3.m3.1.1.cmml">κ</mi><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px2.p1.3.m3.1b"><ci id="A2.SS0.SSS0.Px2.p1.3.m3.1.1.cmml" xref="A2.SS0.SSS0.Px2.p1.3.m3.1.1">𝜅</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px2.p1.3.m3.1c">\kappa</annotation></semantics></math> in Equation <a href="#S2.E9" title="In Theorem 2. ‣ 2.6 Theoretical equivalences ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a> have enjoyed a long tradition:
When a kernel estimator uses <math id="A2.SS0.SSS0.Px2.p1.4.m4.1" class="ltx_Math" alttext="\theta_{K}\neq\theta_{Q}" display="inline"><semantics id="A2.SS0.SSS0.Px2.p1.4.m4.1a"><mrow id="A2.SS0.SSS0.Px2.p1.4.m4.1.1" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.cmml"><msub id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.cmml"><mi id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.cmml">θ</mi><mi id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.3" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.3.cmml">K</mi></msub><mo id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.1" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.1.cmml">≠</mo><msub id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.3" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.3.cmml"><mi id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.3.2" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.3.2.cmml">θ</mi><mi id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.cmml">Q</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px2.p1.4.m4.1b"><apply id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1"><neq id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.1.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.1"></neq><apply id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2"><csymbol cd="ambiguous" id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.1.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2">subscript</csymbol><ci id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.2">𝜃</ci><ci id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.3.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.2.3">𝐾</ci></apply><apply id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.3.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.3"><csymbol cd="ambiguous" id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.3.1.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.3">subscript</csymbol><ci id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.3.2.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.3.2">𝜃</ci><ci id="A2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3.cmml" xref="A2.SS0.SSS0.Px2.p1.4.m4.1.1.3.3">𝑄</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px2.p1.4.m4.1c">\theta_{K}\neq\theta_{Q}</annotation></semantics></math>, it is known as a balloon estimator <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite>.
Papers such as Breiman et al. <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib11" title="" class="ltx_ref">11</a>]</cite> have even used <math id="A2.SS0.SSS0.Px2.p1.5.m5.1" class="ltx_Math" alttext="\theta_{Q}" display="inline"><semantics id="A2.SS0.SSS0.Px2.p1.5.m5.1a"><msub id="A2.SS0.SSS0.Px2.p1.5.m5.1.1" xref="A2.SS0.SSS0.Px2.p1.5.m5.1.1.cmml"><mi id="A2.SS0.SSS0.Px2.p1.5.m5.1.1.2" xref="A2.SS0.SSS0.Px2.p1.5.m5.1.1.2.cmml">θ</mi><mi id="A2.SS0.SSS0.Px2.p1.5.m5.1.1.3" xref="A2.SS0.SSS0.Px2.p1.5.m5.1.1.3.cmml">Q</mi></msub><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px2.p1.5.m5.1b"><apply id="A2.SS0.SSS0.Px2.p1.5.m5.1.1.cmml" xref="A2.SS0.SSS0.Px2.p1.5.m5.1.1"><csymbol cd="ambiguous" id="A2.SS0.SSS0.Px2.p1.5.m5.1.1.1.cmml" xref="A2.SS0.SSS0.Px2.p1.5.m5.1.1">subscript</csymbol><ci id="A2.SS0.SSS0.Px2.p1.5.m5.1.1.2.cmml" xref="A2.SS0.SSS0.Px2.p1.5.m5.1.1.2">𝜃</ci><ci id="A2.SS0.SSS0.Px2.p1.5.m5.1.1.3.cmml" xref="A2.SS0.SSS0.Px2.p1.5.m5.1.1.3">𝑄</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px2.p1.5.m5.1c">\theta_{Q}</annotation></semantics></math> as a function of <math id="A2.SS0.SSS0.Px2.p1.6.m6.1" class="ltx_Math" alttext="x^{\prime}" display="inline"><semantics id="A2.SS0.SSS0.Px2.p1.6.m6.1a"><msup id="A2.SS0.SSS0.Px2.p1.6.m6.1.1" xref="A2.SS0.SSS0.Px2.p1.6.m6.1.1.cmml"><mi id="A2.SS0.SSS0.Px2.p1.6.m6.1.1.2" xref="A2.SS0.SSS0.Px2.p1.6.m6.1.1.2.cmml">x</mi><mo id="A2.SS0.SSS0.Px2.p1.6.m6.1.1.3" xref="A2.SS0.SSS0.Px2.p1.6.m6.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="A2.SS0.SSS0.Px2.p1.6.m6.1b"><apply id="A2.SS0.SSS0.Px2.p1.6.m6.1.1.cmml" xref="A2.SS0.SSS0.Px2.p1.6.m6.1.1"><csymbol cd="ambiguous" id="A2.SS0.SSS0.Px2.p1.6.m6.1.1.1.cmml" xref="A2.SS0.SSS0.Px2.p1.6.m6.1.1">superscript</csymbol><ci id="A2.SS0.SSS0.Px2.p1.6.m6.1.1.2.cmml" xref="A2.SS0.SSS0.Px2.p1.6.m6.1.1.2">𝑥</ci><ci id="A2.SS0.SSS0.Px2.p1.6.m6.1.1.3.cmml" xref="A2.SS0.SSS0.Px2.p1.6.m6.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS0.SSS0.Px2.p1.6.m6.1c">x^{\prime}</annotation></semantics></math>, known as sample-adaptive smoothing.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
<figure id="A2.F16" class="ltx_figure"><img src="/html/2407.04620/assets/x16.png" id="A2.F16.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="236" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 16: </span><span id="A2.F16.12.1" class="ltx_text ltx_font_bold">Left</span>: A residual block, the basic building block for Transformers. The sequence modeling block is instantiated into two variants: the Transformer backbone and Mamba backbone.
<span id="A2.F16.13.2" class="ltx_text ltx_font_bold">Middle</span>: TTT layer in the Transformer backbone. The LN before <math id="A2.F16.5.m1.1" class="ltx_Math" alttext="O" display="inline"><semantics id="A2.F16.5.m1.1b"><mi id="A2.F16.5.m1.1.1" xref="A2.F16.5.m1.1.1.cmml">O</mi><annotation-xml encoding="MathML-Content" id="A2.F16.5.m1.1c"><ci id="A2.F16.5.m1.1.1.cmml" xref="A2.F16.5.m1.1.1">𝑂</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.F16.5.m1.1d">O</annotation></semantics></math> comes from NormFormer <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib64" title="" class="ltx_ref">64</a>]</cite>.
<span id="A2.F16.14.3" class="ltx_text ltx_font_bold">Right</span>: TTT layer in the backbone inspired by Mamba <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib26" title="" class="ltx_ref">26</a>]</cite> and Griffin <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib18" title="" class="ltx_ref">18</a>]</cite>.
Following these two architectures, <math id="A2.F16.6.m2.1" class="ltx_Math" alttext="\sigma" display="inline"><semantics id="A2.F16.6.m2.1b"><mi id="A2.F16.6.m2.1.1" xref="A2.F16.6.m2.1.1.cmml">σ</mi><annotation-xml encoding="MathML-Content" id="A2.F16.6.m2.1c"><ci id="A2.F16.6.m2.1.1.cmml" xref="A2.F16.6.m2.1.1">𝜎</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.F16.6.m2.1d">\sigma</annotation></semantics></math> here is GELU <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib31" title="" class="ltx_ref">31</a>]</cite>.
To accommodate the extra parameters of the gate without changing the embedding dimension, we simply combine <math id="A2.F16.7.m3.1" class="ltx_Math" alttext="\theta_{K}" display="inline"><semantics id="A2.F16.7.m3.1b"><msub id="A2.F16.7.m3.1.1" xref="A2.F16.7.m3.1.1.cmml"><mi id="A2.F16.7.m3.1.1.2" xref="A2.F16.7.m3.1.1.2.cmml">θ</mi><mi id="A2.F16.7.m3.1.1.3" xref="A2.F16.7.m3.1.1.3.cmml">K</mi></msub><annotation-xml encoding="MathML-Content" id="A2.F16.7.m3.1c"><apply id="A2.F16.7.m3.1.1.cmml" xref="A2.F16.7.m3.1.1"><csymbol cd="ambiguous" id="A2.F16.7.m3.1.1.1.cmml" xref="A2.F16.7.m3.1.1">subscript</csymbol><ci id="A2.F16.7.m3.1.1.2.cmml" xref="A2.F16.7.m3.1.1.2">𝜃</ci><ci id="A2.F16.7.m3.1.1.3.cmml" xref="A2.F16.7.m3.1.1.3">𝐾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.F16.7.m3.1d">\theta_{K}</annotation></semantics></math> and <math id="A2.F16.8.m4.1" class="ltx_Math" alttext="\theta_{Q}" display="inline"><semantics id="A2.F16.8.m4.1b"><msub id="A2.F16.8.m4.1.1" xref="A2.F16.8.m4.1.1.cmml"><mi id="A2.F16.8.m4.1.1.2" xref="A2.F16.8.m4.1.1.2.cmml">θ</mi><mi id="A2.F16.8.m4.1.1.3" xref="A2.F16.8.m4.1.1.3.cmml">Q</mi></msub><annotation-xml encoding="MathML-Content" id="A2.F16.8.m4.1c"><apply id="A2.F16.8.m4.1.1.cmml" xref="A2.F16.8.m4.1.1"><csymbol cd="ambiguous" id="A2.F16.8.m4.1.1.1.cmml" xref="A2.F16.8.m4.1.1">subscript</csymbol><ci id="A2.F16.8.m4.1.1.2.cmml" xref="A2.F16.8.m4.1.1.2">𝜃</ci><ci id="A2.F16.8.m4.1.1.3.cmml" xref="A2.F16.8.m4.1.1.3">𝑄</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.F16.8.m4.1d">\theta_{Q}</annotation></semantics></math> into a single projection.
</figcaption>
</figure>
<figure id="A2.T3" class="ltx_table">
<table id="A2.T3.5" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A2.T3.5.1.1" class="ltx_tr">
<th id="A2.T3.5.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt" style="padding-top:2.5pt;padding-bottom:2.5pt;">Params.</th>
<th id="A2.T3.5.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:2.5pt;padding-bottom:2.5pt;">Blocks</th>
<th id="A2.T3.5.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:2.5pt;padding-bottom:2.5pt;">Embed. dim.</th>
<th id="A2.T3.5.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:2.5pt;padding-bottom:2.5pt;">Heads</th>
<th id="A2.T3.5.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:2.5pt;padding-bottom:2.5pt;">Train steps</th>
<th id="A2.T3.5.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:2.5pt;padding-bottom:2.5pt;">Peak LR</th>
<th id="A2.T3.5.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:2.5pt;padding-bottom:2.5pt;">Tokens</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A2.T3.5.2.1" class="ltx_tr">
<td id="A2.T3.5.2.1.1" class="ltx_td ltx_align_left ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">125M</td>
<td id="A2.T3.5.2.1.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">12</td>
<td id="A2.T3.5.2.1.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">768</td>
<td id="A2.T3.5.2.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">12</td>
<td id="A2.T3.5.2.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">4800</td>
<td id="A2.T3.5.2.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">3e-3</td>
<td id="A2.T3.5.2.1.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:2.5pt;padding-bottom:2.5pt;">2.5B</td>
</tr>
<tr id="A2.T3.5.3.2" class="ltx_tr">
<td id="A2.T3.5.3.2.1" class="ltx_td ltx_align_left" style="padding-top:2.5pt;padding-bottom:2.5pt;">350M</td>
<td id="A2.T3.5.3.2.2" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">24</td>
<td id="A2.T3.5.3.2.3" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">1024</td>
<td id="A2.T3.5.3.2.4" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">16</td>
<td id="A2.T3.5.3.2.5" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">13500</td>
<td id="A2.T3.5.3.2.6" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">1.5e-3</td>
<td id="A2.T3.5.3.2.7" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">7B</td>
</tr>
<tr id="A2.T3.5.4.3" class="ltx_tr">
<td id="A2.T3.5.4.3.1" class="ltx_td ltx_align_left" style="padding-top:2.5pt;padding-bottom:2.5pt;">760M</td>
<td id="A2.T3.5.4.3.2" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">24</td>
<td id="A2.T3.5.4.3.3" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">1536</td>
<td id="A2.T3.5.4.3.4" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">16</td>
<td id="A2.T3.5.4.3.5" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">29000</td>
<td id="A2.T3.5.4.3.6" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">1.25e-3</td>
<td id="A2.T3.5.4.3.7" class="ltx_td ltx_align_center" style="padding-top:2.5pt;padding-bottom:2.5pt;">15B</td>
</tr>
<tr id="A2.T3.5.5.4" class="ltx_tr">
<td id="A2.T3.5.5.4.1" class="ltx_td ltx_align_left ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;">1.3B</td>
<td id="A2.T3.5.5.4.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;">24</td>
<td id="A2.T3.5.5.4.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;">2048</td>
<td id="A2.T3.5.5.4.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;">32</td>
<td id="A2.T3.5.5.4.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;">50000</td>
<td id="A2.T3.5.5.4.6" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;">1e-3</td>
<td id="A2.T3.5.5.4.7" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:2.5pt;padding-bottom:2.5pt;">26B</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>Training configurations for all experiments. This table reproduces Table 12 in the <span id="A2.T3.7.1" class="ltx_text">Mamba paper.</span>
The only difference is that the learning rate they use for Mamba and Transformer is <math id="A2.T3.3.m1.1" class="ltx_math_unparsed" alttext="5\times" display="inline"><semantics id="A2.T3.3.m1.1b"><mrow id="A2.T3.3.m1.1c"><mn id="A2.T3.3.m1.1.1">5</mn><mo lspace="0.222em" id="A2.T3.3.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="A2.T3.3.m1.1d">5\times</annotation></semantics></math> the values in their Table 12, and we report the actual values (<math id="A2.T3.4.m2.1" class="ltx_math_unparsed" alttext="5\times" display="inline"><semantics id="A2.T3.4.m2.1b"><mrow id="A2.T3.4.m2.1c"><mn id="A2.T3.4.m2.1.1">5</mn><mo lspace="0.222em" id="A2.T3.4.m2.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="A2.T3.4.m2.1d">5\times</annotation></semantics></math>).
Note that this table only applies to TTT-Linear, TTT-MLP, and Transformers, as Mamba does not follow the multi-head residual block structure inherited from Transformers.
</figcaption>
</figure>
</section>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Experiment details</h2>

<section id="A3.SS0.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Architectures.</h5>

<div id="A3.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="A3.SS0.SSS0.Px1.p1.1" class="ltx_p">Our Transformer strictly follows the construction in the Mamba paper, where <em id="A3.SS0.SSS0.Px1.p1.1.1" class="ltx_emph ltx_font_italic">Transformer</em> is called <em id="A3.SS0.SSS0.Px1.p1.1.2" class="ltx_emph ltx_font_italic">Transformer++</em>.
Specifically, the Transformer architecture is based on Llama <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib73" title="" class="ltx_ref">73</a>]</cite>, with rotary positional encodings (RoPE) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib67" title="" class="ltx_ref">67</a>]</cite>, SwiGLU MLP blocks <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib63" title="" class="ltx_ref">63</a>]</cite>, and RMSNorm <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib80" title="" class="ltx_ref">80</a>]</cite> instead of LayerNorm.
Our Mamba baseline uses the public code provided by the authors.
We have verified that our baselines can reproduce the numbers reported in <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib26" title="" class="ltx_ref">26</a>]</cite>.</p>
</div>
</section>
<section id="A3.SS0.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Training configurations.</h5>

<div id="A3.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="A3.SS0.SSS0.Px2.p1.3" class="ltx_p">Our training configurations are in Table <a href="#A2.T3" title="Table 3 ‣ Asymmetric kernels. ‣ Appendix B Nadaraya-Watson estimator ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, which simply reproduces Table 12 in the Mamba paper.
As discussed in Footnote <a href="#footnote12" title="footnote 12 ‣ 3.2 Long context: Books ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12</span></a>, all models are trained with a batch size of 0.5M tokens regardless of context length.
All of our optimization hyper-parameters follow the “improved recipe" in Appendix E.2 of the Mamba paper, reproduced below:</p>
<ul id="A3.I1" class="ltx_itemize">
<li id="A3.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I1.i1.p1" class="ltx_para">
<p id="A3.I1.i1.p1.1" class="ltx_p">AdamW optimizer: <math id="A3.I1.i1.p1.1.m1.2" class="ltx_Math" alttext="\beta=(0.9,0.95)" display="inline"><semantics id="A3.I1.i1.p1.1.m1.2a"><mrow id="A3.I1.i1.p1.1.m1.2.3" xref="A3.I1.i1.p1.1.m1.2.3.cmml"><mi id="A3.I1.i1.p1.1.m1.2.3.2" xref="A3.I1.i1.p1.1.m1.2.3.2.cmml">β</mi><mo id="A3.I1.i1.p1.1.m1.2.3.1" xref="A3.I1.i1.p1.1.m1.2.3.1.cmml">=</mo><mrow id="A3.I1.i1.p1.1.m1.2.3.3.2" xref="A3.I1.i1.p1.1.m1.2.3.3.1.cmml"><mo stretchy="false" id="A3.I1.i1.p1.1.m1.2.3.3.2.1" xref="A3.I1.i1.p1.1.m1.2.3.3.1.cmml">(</mo><mn id="A3.I1.i1.p1.1.m1.1.1" xref="A3.I1.i1.p1.1.m1.1.1.cmml">0.9</mn><mo id="A3.I1.i1.p1.1.m1.2.3.3.2.2" xref="A3.I1.i1.p1.1.m1.2.3.3.1.cmml">,</mo><mn id="A3.I1.i1.p1.1.m1.2.2" xref="A3.I1.i1.p1.1.m1.2.2.cmml">0.95</mn><mo stretchy="false" id="A3.I1.i1.p1.1.m1.2.3.3.2.3" xref="A3.I1.i1.p1.1.m1.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.I1.i1.p1.1.m1.2b"><apply id="A3.I1.i1.p1.1.m1.2.3.cmml" xref="A3.I1.i1.p1.1.m1.2.3"><eq id="A3.I1.i1.p1.1.m1.2.3.1.cmml" xref="A3.I1.i1.p1.1.m1.2.3.1"></eq><ci id="A3.I1.i1.p1.1.m1.2.3.2.cmml" xref="A3.I1.i1.p1.1.m1.2.3.2">𝛽</ci><interval closure="open" id="A3.I1.i1.p1.1.m1.2.3.3.1.cmml" xref="A3.I1.i1.p1.1.m1.2.3.3.2"><cn type="float" id="A3.I1.i1.p1.1.m1.1.1.cmml" xref="A3.I1.i1.p1.1.m1.1.1">0.9</cn><cn type="float" id="A3.I1.i1.p1.1.m1.2.2.cmml" xref="A3.I1.i1.p1.1.m1.2.2">0.95</cn></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.I1.i1.p1.1.m1.2c">\beta=(0.9,0.95)</annotation></semantics></math></p>
</div>
</li>
<li id="A3.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I1.i2.p1" class="ltx_para">
<p id="A3.I1.i2.p1.1" class="ltx_p">Cosine schedule: decay to end learning rate <math id="A3.I1.i2.p1.1.m1.1" class="ltx_Math" alttext="1e-5" display="inline"><semantics id="A3.I1.i2.p1.1.m1.1a"><mrow id="A3.I1.i2.p1.1.m1.1.1" xref="A3.I1.i2.p1.1.m1.1.1.cmml"><mrow id="A3.I1.i2.p1.1.m1.1.1.2" xref="A3.I1.i2.p1.1.m1.1.1.2.cmml"><mn id="A3.I1.i2.p1.1.m1.1.1.2.2" xref="A3.I1.i2.p1.1.m1.1.1.2.2.cmml">1</mn><mo lspace="0em" rspace="0em" id="A3.I1.i2.p1.1.m1.1.1.2.1" xref="A3.I1.i2.p1.1.m1.1.1.2.1.cmml">​</mo><mi id="A3.I1.i2.p1.1.m1.1.1.2.3" xref="A3.I1.i2.p1.1.m1.1.1.2.3.cmml">e</mi></mrow><mo id="A3.I1.i2.p1.1.m1.1.1.1" xref="A3.I1.i2.p1.1.m1.1.1.1.cmml">−</mo><mn id="A3.I1.i2.p1.1.m1.1.1.3" xref="A3.I1.i2.p1.1.m1.1.1.3.cmml">5</mn></mrow><annotation-xml encoding="MathML-Content" id="A3.I1.i2.p1.1.m1.1b"><apply id="A3.I1.i2.p1.1.m1.1.1.cmml" xref="A3.I1.i2.p1.1.m1.1.1"><minus id="A3.I1.i2.p1.1.m1.1.1.1.cmml" xref="A3.I1.i2.p1.1.m1.1.1.1"></minus><apply id="A3.I1.i2.p1.1.m1.1.1.2.cmml" xref="A3.I1.i2.p1.1.m1.1.1.2"><times id="A3.I1.i2.p1.1.m1.1.1.2.1.cmml" xref="A3.I1.i2.p1.1.m1.1.1.2.1"></times><cn type="integer" id="A3.I1.i2.p1.1.m1.1.1.2.2.cmml" xref="A3.I1.i2.p1.1.m1.1.1.2.2">1</cn><ci id="A3.I1.i2.p1.1.m1.1.1.2.3.cmml" xref="A3.I1.i2.p1.1.m1.1.1.2.3">𝑒</ci></apply><cn type="integer" id="A3.I1.i2.p1.1.m1.1.1.3.cmml" xref="A3.I1.i2.p1.1.m1.1.1.3">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.I1.i2.p1.1.m1.1c">1e-5</annotation></semantics></math></p>
</div>
</li>
<li id="A3.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I1.i3.p1" class="ltx_para">
<p id="A3.I1.i3.p1.1" class="ltx_p">Linear learning rate warmup over 10% of the training steps</p>
</div>
</li>
<li id="A3.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I1.i4.p1" class="ltx_para">
<p id="A3.I1.i4.p1.1" class="ltx_p">Weight decay: 0.1</p>
</div>
</li>
<li id="A3.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I1.i5.p1" class="ltx_para">
<p id="A3.I1.i5.p1.1" class="ltx_p">Gradient clipping: 1.0</p>
</div>
</li>
<li id="A3.I1.i6" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I1.i6.p1" class="ltx_para">
<p id="A3.I1.i6.p1.1" class="ltx_p">No Dropout</p>
</div>
</li>
<li id="A3.I1.i7" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I1.i7.p1" class="ltx_para">
<p id="A3.I1.i7.p1.1" class="ltx_p">Mixed Precision</p>
</div>
</li>
</ul>
<p id="A3.SS0.SSS0.Px2.p1.2" class="ltx_p">As discussed in Footnote <a href="#footnote10" title="footnote 10 ‣ 2nd item ‣ Protocols. ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a>, all models are trained using the Llama tokenizer <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib73" title="" class="ltx_ref">73</a>]</cite>.
For experiments on the Pile, this is the only difference with the recipe in the Mamba paper, which uses two other tokenizers.
For experiments on Books, we find that the original angle of the RoPE encoding <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib67" title="" class="ltx_ref">67</a>]</cite> <math id="A3.SS0.SSS0.Px2.p1.1.m1.2" class="ltx_Math" alttext="\theta=10,000" display="inline"><semantics id="A3.SS0.SSS0.Px2.p1.1.m1.2a"><mrow id="A3.SS0.SSS0.Px2.p1.1.m1.2.3" xref="A3.SS0.SSS0.Px2.p1.1.m1.2.3.cmml"><mi id="A3.SS0.SSS0.Px2.p1.1.m1.2.3.2" xref="A3.SS0.SSS0.Px2.p1.1.m1.2.3.2.cmml">θ</mi><mo id="A3.SS0.SSS0.Px2.p1.1.m1.2.3.1" xref="A3.SS0.SSS0.Px2.p1.1.m1.2.3.1.cmml">=</mo><mrow id="A3.SS0.SSS0.Px2.p1.1.m1.2.3.3.2" xref="A3.SS0.SSS0.Px2.p1.1.m1.2.3.3.1.cmml"><mn id="A3.SS0.SSS0.Px2.p1.1.m1.1.1" xref="A3.SS0.SSS0.Px2.p1.1.m1.1.1.cmml">10</mn><mo id="A3.SS0.SSS0.Px2.p1.1.m1.2.3.3.2.1" xref="A3.SS0.SSS0.Px2.p1.1.m1.2.3.3.1.cmml">,</mo><mn id="A3.SS0.SSS0.Px2.p1.1.m1.2.2" xref="A3.SS0.SSS0.Px2.p1.1.m1.2.2.cmml">000</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.SS0.SSS0.Px2.p1.1.m1.2b"><apply id="A3.SS0.SSS0.Px2.p1.1.m1.2.3.cmml" xref="A3.SS0.SSS0.Px2.p1.1.m1.2.3"><eq id="A3.SS0.SSS0.Px2.p1.1.m1.2.3.1.cmml" xref="A3.SS0.SSS0.Px2.p1.1.m1.2.3.1"></eq><ci id="A3.SS0.SSS0.Px2.p1.1.m1.2.3.2.cmml" xref="A3.SS0.SSS0.Px2.p1.1.m1.2.3.2">𝜃</ci><list id="A3.SS0.SSS0.Px2.p1.1.m1.2.3.3.1.cmml" xref="A3.SS0.SSS0.Px2.p1.1.m1.2.3.3.2"><cn type="integer" id="A3.SS0.SSS0.Px2.p1.1.m1.1.1.cmml" xref="A3.SS0.SSS0.Px2.p1.1.m1.1.1">10</cn><cn type="integer" id="A3.SS0.SSS0.Px2.p1.1.m1.2.2.cmml" xref="A3.SS0.SSS0.Px2.p1.1.m1.2.2">000</cn></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS0.SSS0.Px2.p1.1.m1.2c">\theta=10,000</annotation></semantics></math> is sub-optimal for our Transformer baseline in long context.
Starting at context length 4k, we try <math id="A3.SS0.SSS0.Px2.p1.2.m2.2" class="ltx_Math" alttext="\theta=500,000" display="inline"><semantics id="A3.SS0.SSS0.Px2.p1.2.m2.2a"><mrow id="A3.SS0.SSS0.Px2.p1.2.m2.2.3" xref="A3.SS0.SSS0.Px2.p1.2.m2.2.3.cmml"><mi id="A3.SS0.SSS0.Px2.p1.2.m2.2.3.2" xref="A3.SS0.SSS0.Px2.p1.2.m2.2.3.2.cmml">θ</mi><mo id="A3.SS0.SSS0.Px2.p1.2.m2.2.3.1" xref="A3.SS0.SSS0.Px2.p1.2.m2.2.3.1.cmml">=</mo><mrow id="A3.SS0.SSS0.Px2.p1.2.m2.2.3.3.2" xref="A3.SS0.SSS0.Px2.p1.2.m2.2.3.3.1.cmml"><mn id="A3.SS0.SSS0.Px2.p1.2.m2.1.1" xref="A3.SS0.SSS0.Px2.p1.2.m2.1.1.cmml">500</mn><mo id="A3.SS0.SSS0.Px2.p1.2.m2.2.3.3.2.1" xref="A3.SS0.SSS0.Px2.p1.2.m2.2.3.3.1.cmml">,</mo><mn id="A3.SS0.SSS0.Px2.p1.2.m2.2.2" xref="A3.SS0.SSS0.Px2.p1.2.m2.2.2.cmml">000</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.SS0.SSS0.Px2.p1.2.m2.2b"><apply id="A3.SS0.SSS0.Px2.p1.2.m2.2.3.cmml" xref="A3.SS0.SSS0.Px2.p1.2.m2.2.3"><eq id="A3.SS0.SSS0.Px2.p1.2.m2.2.3.1.cmml" xref="A3.SS0.SSS0.Px2.p1.2.m2.2.3.1"></eq><ci id="A3.SS0.SSS0.Px2.p1.2.m2.2.3.2.cmml" xref="A3.SS0.SSS0.Px2.p1.2.m2.2.3.2">𝜃</ci><list id="A3.SS0.SSS0.Px2.p1.2.m2.2.3.3.1.cmml" xref="A3.SS0.SSS0.Px2.p1.2.m2.2.3.3.2"><cn type="integer" id="A3.SS0.SSS0.Px2.p1.2.m2.1.1.cmml" xref="A3.SS0.SSS0.Px2.p1.2.m2.1.1">500</cn><cn type="integer" id="A3.SS0.SSS0.Px2.p1.2.m2.2.2.cmml" xref="A3.SS0.SSS0.Px2.p1.2.m2.2.2">000</cn></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS0.SSS0.Px2.p1.2.m2.2c">\theta=500,000</annotation></semantics></math> following the Llama Long paper <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib78" title="" class="ltx_ref">78</a>]</cite>, and use the better perplexity for Transformer (both pretrain and finetune).</p>
</div>
</section>
<section id="A3.SS0.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Transformer finetuning.</h5>

<div id="A3.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="A3.SS0.SSS0.Px3.p1.1" class="ltx_p">Finetuning starts a new cosine schedule with the same optimization hyper-parameter as training from scratch, except the peak learning rate.
We try three peak learning rates for finetuning: 1e-5, 1e-4, and 1e-3, and select for the best perplexity.
We observe that 1e-4 works the best for the 125M models, while 1e-5 works the best for 350M and larger.
This observation is reasonable considering that the end learning rate for the Chinchilla recipe is 1e-5.</p>
</div>
</section>
<section id="A3.SS0.SSS0.Px4" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Learning rate for TTT.</h5>

<div id="A3.SS0.SSS0.Px4.p1" class="ltx_para">
<p id="A3.SS0.SSS0.Px4.p1.5" class="ltx_p">As mentioned in Subsection <a href="#S2.SS7" title="2.7 Implementation details ‣ 2 Method ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2.7</span></a>, the inner-loop base learning rate <math id="A3.SS0.SSS0.Px4.p1.1.m1.1" class="ltx_Math" alttext="\eta_{\text{base}}" display="inline"><semantics id="A3.SS0.SSS0.Px4.p1.1.m1.1a"><msub id="A3.SS0.SSS0.Px4.p1.1.m1.1.1" xref="A3.SS0.SSS0.Px4.p1.1.m1.1.1.cmml"><mi id="A3.SS0.SSS0.Px4.p1.1.m1.1.1.2" xref="A3.SS0.SSS0.Px4.p1.1.m1.1.1.2.cmml">η</mi><mtext id="A3.SS0.SSS0.Px4.p1.1.m1.1.1.3" xref="A3.SS0.SSS0.Px4.p1.1.m1.1.1.3a.cmml">base</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.SS0.SSS0.Px4.p1.1.m1.1b"><apply id="A3.SS0.SSS0.Px4.p1.1.m1.1.1.cmml" xref="A3.SS0.SSS0.Px4.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A3.SS0.SSS0.Px4.p1.1.m1.1.1.1.cmml" xref="A3.SS0.SSS0.Px4.p1.1.m1.1.1">subscript</csymbol><ci id="A3.SS0.SSS0.Px4.p1.1.m1.1.1.2.cmml" xref="A3.SS0.SSS0.Px4.p1.1.m1.1.1.2">𝜂</ci><ci id="A3.SS0.SSS0.Px4.p1.1.m1.1.1.3a.cmml" xref="A3.SS0.SSS0.Px4.p1.1.m1.1.1.3"><mtext mathsize="70%" id="A3.SS0.SSS0.Px4.p1.1.m1.1.1.3.cmml" xref="A3.SS0.SSS0.Px4.p1.1.m1.1.1.3">base</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS0.SSS0.Px4.p1.1.m1.1c">\eta_{\text{base}}</annotation></semantics></math> is set to 1 for TTT-Linear and 0.1 for TTT-MLP.
Our heuristic for setting <math id="A3.SS0.SSS0.Px4.p1.2.m2.1" class="ltx_Math" alttext="\eta_{\text{base}}" display="inline"><semantics id="A3.SS0.SSS0.Px4.p1.2.m2.1a"><msub id="A3.SS0.SSS0.Px4.p1.2.m2.1.1" xref="A3.SS0.SSS0.Px4.p1.2.m2.1.1.cmml"><mi id="A3.SS0.SSS0.Px4.p1.2.m2.1.1.2" xref="A3.SS0.SSS0.Px4.p1.2.m2.1.1.2.cmml">η</mi><mtext id="A3.SS0.SSS0.Px4.p1.2.m2.1.1.3" xref="A3.SS0.SSS0.Px4.p1.2.m2.1.1.3a.cmml">base</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.SS0.SSS0.Px4.p1.2.m2.1b"><apply id="A3.SS0.SSS0.Px4.p1.2.m2.1.1.cmml" xref="A3.SS0.SSS0.Px4.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A3.SS0.SSS0.Px4.p1.2.m2.1.1.1.cmml" xref="A3.SS0.SSS0.Px4.p1.2.m2.1.1">subscript</csymbol><ci id="A3.SS0.SSS0.Px4.p1.2.m2.1.1.2.cmml" xref="A3.SS0.SSS0.Px4.p1.2.m2.1.1.2">𝜂</ci><ci id="A3.SS0.SSS0.Px4.p1.2.m2.1.1.3a.cmml" xref="A3.SS0.SSS0.Px4.p1.2.m2.1.1.3"><mtext mathsize="70%" id="A3.SS0.SSS0.Px4.p1.2.m2.1.1.3.cmml" xref="A3.SS0.SSS0.Px4.p1.2.m2.1.1.3">base</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS0.SSS0.Px4.p1.2.m2.1c">\eta_{\text{base}}</annotation></semantics></math> is similar to how people set the outer-loop learning rate for regular training: We tried <math id="A3.SS0.SSS0.Px4.p1.3.m3.4" class="ltx_Math" alttext="\eta_{\text{base}}\in\{0.01,0.1,1,10\}" display="inline"><semantics id="A3.SS0.SSS0.Px4.p1.3.m3.4a"><mrow id="A3.SS0.SSS0.Px4.p1.3.m3.4.5" xref="A3.SS0.SSS0.Px4.p1.3.m3.4.5.cmml"><msub id="A3.SS0.SSS0.Px4.p1.3.m3.4.5.2" xref="A3.SS0.SSS0.Px4.p1.3.m3.4.5.2.cmml"><mi id="A3.SS0.SSS0.Px4.p1.3.m3.4.5.2.2" xref="A3.SS0.SSS0.Px4.p1.3.m3.4.5.2.2.cmml">η</mi><mtext id="A3.SS0.SSS0.Px4.p1.3.m3.4.5.2.3" xref="A3.SS0.SSS0.Px4.p1.3.m3.4.5.2.3a.cmml">base</mtext></msub><mo id="A3.SS0.SSS0.Px4.p1.3.m3.4.5.1" xref="A3.SS0.SSS0.Px4.p1.3.m3.4.5.1.cmml">∈</mo><mrow id="A3.SS0.SSS0.Px4.p1.3.m3.4.5.3.2" xref="A3.SS0.SSS0.Px4.p1.3.m3.4.5.3.1.cmml"><mo stretchy="false" id="A3.SS0.SSS0.Px4.p1.3.m3.4.5.3.2.1" xref="A3.SS0.SSS0.Px4.p1.3.m3.4.5.3.1.cmml">{</mo><mn id="A3.SS0.SSS0.Px4.p1.3.m3.1.1" xref="A3.SS0.SSS0.Px4.p1.3.m3.1.1.cmml">0.01</mn><mo id="A3.SS0.SSS0.Px4.p1.3.m3.4.5.3.2.2" xref="A3.SS0.SSS0.Px4.p1.3.m3.4.5.3.1.cmml">,</mo><mn id="A3.SS0.SSS0.Px4.p1.3.m3.2.2" xref="A3.SS0.SSS0.Px4.p1.3.m3.2.2.cmml">0.1</mn><mo id="A3.SS0.SSS0.Px4.p1.3.m3.4.5.3.2.3" xref="A3.SS0.SSS0.Px4.p1.3.m3.4.5.3.1.cmml">,</mo><mn id="A3.SS0.SSS0.Px4.p1.3.m3.3.3" xref="A3.SS0.SSS0.Px4.p1.3.m3.3.3.cmml">1</mn><mo id="A3.SS0.SSS0.Px4.p1.3.m3.4.5.3.2.4" xref="A3.SS0.SSS0.Px4.p1.3.m3.4.5.3.1.cmml">,</mo><mn id="A3.SS0.SSS0.Px4.p1.3.m3.4.4" xref="A3.SS0.SSS0.Px4.p1.3.m3.4.4.cmml">10</mn><mo stretchy="false" id="A3.SS0.SSS0.Px4.p1.3.m3.4.5.3.2.5" xref="A3.SS0.SSS0.Px4.p1.3.m3.4.5.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.SS0.SSS0.Px4.p1.3.m3.4b"><apply id="A3.SS0.SSS0.Px4.p1.3.m3.4.5.cmml" xref="A3.SS0.SSS0.Px4.p1.3.m3.4.5"><in id="A3.SS0.SSS0.Px4.p1.3.m3.4.5.1.cmml" xref="A3.SS0.SSS0.Px4.p1.3.m3.4.5.1"></in><apply id="A3.SS0.SSS0.Px4.p1.3.m3.4.5.2.cmml" xref="A3.SS0.SSS0.Px4.p1.3.m3.4.5.2"><csymbol cd="ambiguous" id="A3.SS0.SSS0.Px4.p1.3.m3.4.5.2.1.cmml" xref="A3.SS0.SSS0.Px4.p1.3.m3.4.5.2">subscript</csymbol><ci id="A3.SS0.SSS0.Px4.p1.3.m3.4.5.2.2.cmml" xref="A3.SS0.SSS0.Px4.p1.3.m3.4.5.2.2">𝜂</ci><ci id="A3.SS0.SSS0.Px4.p1.3.m3.4.5.2.3a.cmml" xref="A3.SS0.SSS0.Px4.p1.3.m3.4.5.2.3"><mtext mathsize="70%" id="A3.SS0.SSS0.Px4.p1.3.m3.4.5.2.3.cmml" xref="A3.SS0.SSS0.Px4.p1.3.m3.4.5.2.3">base</mtext></ci></apply><set id="A3.SS0.SSS0.Px4.p1.3.m3.4.5.3.1.cmml" xref="A3.SS0.SSS0.Px4.p1.3.m3.4.5.3.2"><cn type="float" id="A3.SS0.SSS0.Px4.p1.3.m3.1.1.cmml" xref="A3.SS0.SSS0.Px4.p1.3.m3.1.1">0.01</cn><cn type="float" id="A3.SS0.SSS0.Px4.p1.3.m3.2.2.cmml" xref="A3.SS0.SSS0.Px4.p1.3.m3.2.2">0.1</cn><cn type="integer" id="A3.SS0.SSS0.Px4.p1.3.m3.3.3.cmml" xref="A3.SS0.SSS0.Px4.p1.3.m3.3.3">1</cn><cn type="integer" id="A3.SS0.SSS0.Px4.p1.3.m3.4.4.cmml" xref="A3.SS0.SSS0.Px4.p1.3.m3.4.4">10</cn></set></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS0.SSS0.Px4.p1.3.m3.4c">\eta_{\text{base}}\in\{0.01,0.1,1,10\}</annotation></semantics></math> and used the largest value that does not cause instabilities.
For TTT-MLP, we use linear warmup for <math id="A3.SS0.SSS0.Px4.p1.4.m4.1" class="ltx_Math" alttext="\eta_{\text{base}}" display="inline"><semantics id="A3.SS0.SSS0.Px4.p1.4.m4.1a"><msub id="A3.SS0.SSS0.Px4.p1.4.m4.1.1" xref="A3.SS0.SSS0.Px4.p1.4.m4.1.1.cmml"><mi id="A3.SS0.SSS0.Px4.p1.4.m4.1.1.2" xref="A3.SS0.SSS0.Px4.p1.4.m4.1.1.2.cmml">η</mi><mtext id="A3.SS0.SSS0.Px4.p1.4.m4.1.1.3" xref="A3.SS0.SSS0.Px4.p1.4.m4.1.1.3a.cmml">base</mtext></msub><annotation-xml encoding="MathML-Content" id="A3.SS0.SSS0.Px4.p1.4.m4.1b"><apply id="A3.SS0.SSS0.Px4.p1.4.m4.1.1.cmml" xref="A3.SS0.SSS0.Px4.p1.4.m4.1.1"><csymbol cd="ambiguous" id="A3.SS0.SSS0.Px4.p1.4.m4.1.1.1.cmml" xref="A3.SS0.SSS0.Px4.p1.4.m4.1.1">subscript</csymbol><ci id="A3.SS0.SSS0.Px4.p1.4.m4.1.1.2.cmml" xref="A3.SS0.SSS0.Px4.p1.4.m4.1.1.2">𝜂</ci><ci id="A3.SS0.SSS0.Px4.p1.4.m4.1.1.3a.cmml" xref="A3.SS0.SSS0.Px4.p1.4.m4.1.1.3"><mtext mathsize="70%" id="A3.SS0.SSS0.Px4.p1.4.m4.1.1.3.cmml" xref="A3.SS0.SSS0.Px4.p1.4.m4.1.1.3">base</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS0.SSS0.Px4.p1.4.m4.1c">\eta_{\text{base}}</annotation></semantics></math> over 10% of the training steps, similar to regular training.
The number of training steps in the inner loop is <math id="A3.SS0.SSS0.Px4.p1.5.m5.1" class="ltx_Math" alttext="T/b" display="inline"><semantics id="A3.SS0.SSS0.Px4.p1.5.m5.1a"><mrow id="A3.SS0.SSS0.Px4.p1.5.m5.1.1" xref="A3.SS0.SSS0.Px4.p1.5.m5.1.1.cmml"><mi id="A3.SS0.SSS0.Px4.p1.5.m5.1.1.2" xref="A3.SS0.SSS0.Px4.p1.5.m5.1.1.2.cmml">T</mi><mo id="A3.SS0.SSS0.Px4.p1.5.m5.1.1.1" xref="A3.SS0.SSS0.Px4.p1.5.m5.1.1.1.cmml">/</mo><mi id="A3.SS0.SSS0.Px4.p1.5.m5.1.1.3" xref="A3.SS0.SSS0.Px4.p1.5.m5.1.1.3.cmml">b</mi></mrow><annotation-xml encoding="MathML-Content" id="A3.SS0.SSS0.Px4.p1.5.m5.1b"><apply id="A3.SS0.SSS0.Px4.p1.5.m5.1.1.cmml" xref="A3.SS0.SSS0.Px4.p1.5.m5.1.1"><divide id="A3.SS0.SSS0.Px4.p1.5.m5.1.1.1.cmml" xref="A3.SS0.SSS0.Px4.p1.5.m5.1.1.1"></divide><ci id="A3.SS0.SSS0.Px4.p1.5.m5.1.1.2.cmml" xref="A3.SS0.SSS0.Px4.p1.5.m5.1.1.2">𝑇</ci><ci id="A3.SS0.SSS0.Px4.p1.5.m5.1.1.3.cmml" xref="A3.SS0.SSS0.Px4.p1.5.m5.1.1.3">𝑏</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS0.SSS0.Px4.p1.5.m5.1c">T/b</annotation></semantics></math> (assume divisible).
For TTT-Linear, we tried linear warmup in the inner loop but did not observe a difference.</p>
</div>
</section>
<section id="A3.SS0.SSS0.Px5" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Experiments in Figure <a href="#S0.F2" title="Figure 2 ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> (right) and Figure <a href="#S3.F14.3" title="Figure 14 ‣ Context length as a hyper-parameter. ‣ 3.2 Long context: Books ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">14</span></a>.</h5>

<div id="A3.SS0.SSS0.Px5.p1" class="ltx_para">
<p id="A3.SS0.SSS0.Px5.p1.1" class="ltx_p">To ensure fairness to Mamba, all methods in these experiments have matched training FLOPs and are trained with the same recipe (last row of Table <a href="#A2.T3" title="Table 3 ‣ Asymmetric kernels. ‣ Appendix B Nadaraya-Watson estimator ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>) as Mamba 1.4B.
To match FLOPs with Mamba, Transformer has 19 blocks instead of 24.
For TTT-Linear and TTT-MLP, their FLOPs are already close to those of Mamba, so we change the hidden dimension of the MLP blocks from 5504 to 5808 (TTT-Linear) and 5248 (TTT-MLP).</p>
</div>
</section>
<section id="A3.SS0.SSS0.Px6" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Gradient checkpointing through time.</h5>

<div id="A3.SS0.SSS0.Px6.p1" class="ltx_para">
<p id="A3.SS0.SSS0.Px6.p1.4" class="ltx_p">By default, libraries such as JAX and PyTorch save the intermediate activations during a forward pass so they can be reused during the backward pass.
However, for a TTT layer with <math id="A3.SS0.SSS0.Px6.p1.1.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="A3.SS0.SSS0.Px6.p1.1.m1.1a"><mi id="A3.SS0.SSS0.Px6.p1.1.m1.1.1" xref="A3.SS0.SSS0.Px6.p1.1.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="A3.SS0.SSS0.Px6.p1.1.m1.1b"><ci id="A3.SS0.SSS0.Px6.p1.1.m1.1.1.cmml" xref="A3.SS0.SSS0.Px6.p1.1.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.SS0.SSS0.Px6.p1.1.m1.1c">W</annotation></semantics></math> as hidden state, this default saves <math id="A3.SS0.SSS0.Px6.p1.2.m2.3" class="ltx_Math" alttext="W_{1},\dots,W_{T}" display="inline"><semantics id="A3.SS0.SSS0.Px6.p1.2.m2.3a"><mrow id="A3.SS0.SSS0.Px6.p1.2.m2.3.3.2" xref="A3.SS0.SSS0.Px6.p1.2.m2.3.3.3.cmml"><msub id="A3.SS0.SSS0.Px6.p1.2.m2.2.2.1.1" xref="A3.SS0.SSS0.Px6.p1.2.m2.2.2.1.1.cmml"><mi id="A3.SS0.SSS0.Px6.p1.2.m2.2.2.1.1.2" xref="A3.SS0.SSS0.Px6.p1.2.m2.2.2.1.1.2.cmml">W</mi><mn id="A3.SS0.SSS0.Px6.p1.2.m2.2.2.1.1.3" xref="A3.SS0.SSS0.Px6.p1.2.m2.2.2.1.1.3.cmml">1</mn></msub><mo id="A3.SS0.SSS0.Px6.p1.2.m2.3.3.2.3" xref="A3.SS0.SSS0.Px6.p1.2.m2.3.3.3.cmml">,</mo><mi mathvariant="normal" id="A3.SS0.SSS0.Px6.p1.2.m2.1.1" xref="A3.SS0.SSS0.Px6.p1.2.m2.1.1.cmml">…</mi><mo id="A3.SS0.SSS0.Px6.p1.2.m2.3.3.2.4" xref="A3.SS0.SSS0.Px6.p1.2.m2.3.3.3.cmml">,</mo><msub id="A3.SS0.SSS0.Px6.p1.2.m2.3.3.2.2" xref="A3.SS0.SSS0.Px6.p1.2.m2.3.3.2.2.cmml"><mi id="A3.SS0.SSS0.Px6.p1.2.m2.3.3.2.2.2" xref="A3.SS0.SSS0.Px6.p1.2.m2.3.3.2.2.2.cmml">W</mi><mi id="A3.SS0.SSS0.Px6.p1.2.m2.3.3.2.2.3" xref="A3.SS0.SSS0.Px6.p1.2.m2.3.3.2.2.3.cmml">T</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="A3.SS0.SSS0.Px6.p1.2.m2.3b"><list id="A3.SS0.SSS0.Px6.p1.2.m2.3.3.3.cmml" xref="A3.SS0.SSS0.Px6.p1.2.m2.3.3.2"><apply id="A3.SS0.SSS0.Px6.p1.2.m2.2.2.1.1.cmml" xref="A3.SS0.SSS0.Px6.p1.2.m2.2.2.1.1"><csymbol cd="ambiguous" id="A3.SS0.SSS0.Px6.p1.2.m2.2.2.1.1.1.cmml" xref="A3.SS0.SSS0.Px6.p1.2.m2.2.2.1.1">subscript</csymbol><ci id="A3.SS0.SSS0.Px6.p1.2.m2.2.2.1.1.2.cmml" xref="A3.SS0.SSS0.Px6.p1.2.m2.2.2.1.1.2">𝑊</ci><cn type="integer" id="A3.SS0.SSS0.Px6.p1.2.m2.2.2.1.1.3.cmml" xref="A3.SS0.SSS0.Px6.p1.2.m2.2.2.1.1.3">1</cn></apply><ci id="A3.SS0.SSS0.Px6.p1.2.m2.1.1.cmml" xref="A3.SS0.SSS0.Px6.p1.2.m2.1.1">…</ci><apply id="A3.SS0.SSS0.Px6.p1.2.m2.3.3.2.2.cmml" xref="A3.SS0.SSS0.Px6.p1.2.m2.3.3.2.2"><csymbol cd="ambiguous" id="A3.SS0.SSS0.Px6.p1.2.m2.3.3.2.2.1.cmml" xref="A3.SS0.SSS0.Px6.p1.2.m2.3.3.2.2">subscript</csymbol><ci id="A3.SS0.SSS0.Px6.p1.2.m2.3.3.2.2.2.cmml" xref="A3.SS0.SSS0.Px6.p1.2.m2.3.3.2.2.2">𝑊</ci><ci id="A3.SS0.SSS0.Px6.p1.2.m2.3.3.2.2.3.cmml" xref="A3.SS0.SSS0.Px6.p1.2.m2.3.3.2.2.3">𝑇</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="A3.SS0.SSS0.Px6.p1.2.m2.3c">W_{1},\dots,W_{T}</annotation></semantics></math>, which uses too much memory.
With TTT mini-batch and the dual form, we still need to save (assume divisible) <math id="A3.SS0.SSS0.Px6.p1.3.m3.1" class="ltx_Math" alttext="\kappa=T/b" display="inline"><semantics id="A3.SS0.SSS0.Px6.p1.3.m3.1a"><mrow id="A3.SS0.SSS0.Px6.p1.3.m3.1.1" xref="A3.SS0.SSS0.Px6.p1.3.m3.1.1.cmml"><mi id="A3.SS0.SSS0.Px6.p1.3.m3.1.1.2" xref="A3.SS0.SSS0.Px6.p1.3.m3.1.1.2.cmml">κ</mi><mo id="A3.SS0.SSS0.Px6.p1.3.m3.1.1.1" xref="A3.SS0.SSS0.Px6.p1.3.m3.1.1.1.cmml">=</mo><mrow id="A3.SS0.SSS0.Px6.p1.3.m3.1.1.3" xref="A3.SS0.SSS0.Px6.p1.3.m3.1.1.3.cmml"><mi id="A3.SS0.SSS0.Px6.p1.3.m3.1.1.3.2" xref="A3.SS0.SSS0.Px6.p1.3.m3.1.1.3.2.cmml">T</mi><mo id="A3.SS0.SSS0.Px6.p1.3.m3.1.1.3.1" xref="A3.SS0.SSS0.Px6.p1.3.m3.1.1.3.1.cmml">/</mo><mi id="A3.SS0.SSS0.Px6.p1.3.m3.1.1.3.3" xref="A3.SS0.SSS0.Px6.p1.3.m3.1.1.3.3.cmml">b</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.SS0.SSS0.Px6.p1.3.m3.1b"><apply id="A3.SS0.SSS0.Px6.p1.3.m3.1.1.cmml" xref="A3.SS0.SSS0.Px6.p1.3.m3.1.1"><eq id="A3.SS0.SSS0.Px6.p1.3.m3.1.1.1.cmml" xref="A3.SS0.SSS0.Px6.p1.3.m3.1.1.1"></eq><ci id="A3.SS0.SSS0.Px6.p1.3.m3.1.1.2.cmml" xref="A3.SS0.SSS0.Px6.p1.3.m3.1.1.2">𝜅</ci><apply id="A3.SS0.SSS0.Px6.p1.3.m3.1.1.3.cmml" xref="A3.SS0.SSS0.Px6.p1.3.m3.1.1.3"><divide id="A3.SS0.SSS0.Px6.p1.3.m3.1.1.3.1.cmml" xref="A3.SS0.SSS0.Px6.p1.3.m3.1.1.3.1"></divide><ci id="A3.SS0.SSS0.Px6.p1.3.m3.1.1.3.2.cmml" xref="A3.SS0.SSS0.Px6.p1.3.m3.1.1.3.2">𝑇</ci><ci id="A3.SS0.SSS0.Px6.p1.3.m3.1.1.3.3.cmml" xref="A3.SS0.SSS0.Px6.p1.3.m3.1.1.3.3">𝑏</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS0.SSS0.Px6.p1.3.m3.1c">\kappa=T/b</annotation></semantics></math> <math id="A3.SS0.SSS0.Px6.p1.4.m4.1" class="ltx_Math" alttext="W" display="inline"><semantics id="A3.SS0.SSS0.Px6.p1.4.m4.1a"><mi id="A3.SS0.SSS0.Px6.p1.4.m4.1.1" xref="A3.SS0.SSS0.Px6.p1.4.m4.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="A3.SS0.SSS0.Px6.p1.4.m4.1b"><ci id="A3.SS0.SSS0.Px6.p1.4.m4.1.1.cmml" xref="A3.SS0.SSS0.Px6.p1.4.m4.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.SS0.SSS0.Px6.p1.4.m4.1c">W</annotation></semantics></math>s at the end of the mini-batches.
A standard technique to save memory in this scenario is gradient checkpointing <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib13" title="" class="ltx_ref">13</a>]</cite>, which is usually applied through layers, but we apply it through time.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
<figure id="A3.F17" class="ltx_figure"><img src="/html/2407.04620/assets/x17.png" id="A3.F17.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="461" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 17: </span>
The self-supervised TTT loss <math id="A3.F17.9.m1.1" class="ltx_Math" alttext="\ell" display="inline"><semantics id="A3.F17.9.m1.1b"><mi mathvariant="normal" id="A3.F17.9.m1.1.1" xref="A3.F17.9.m1.1.1.cmml">ℓ</mi><annotation-xml encoding="MathML-Content" id="A3.F17.9.m1.1c"><ci id="A3.F17.9.m1.1.1.cmml" xref="A3.F17.9.m1.1.1">ℓ</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.F17.9.m1.1d">\ell</annotation></semantics></math> averaged over all test sequences of the form <math id="A3.F17.10.m2.3" class="ltx_Math" alttext="x_{1},\dots,x_{T}" display="inline"><semantics id="A3.F17.10.m2.3b"><mrow id="A3.F17.10.m2.3.3.2" xref="A3.F17.10.m2.3.3.3.cmml"><msub id="A3.F17.10.m2.2.2.1.1" xref="A3.F17.10.m2.2.2.1.1.cmml"><mi id="A3.F17.10.m2.2.2.1.1.2" xref="A3.F17.10.m2.2.2.1.1.2.cmml">x</mi><mn id="A3.F17.10.m2.2.2.1.1.3" xref="A3.F17.10.m2.2.2.1.1.3.cmml">1</mn></msub><mo id="A3.F17.10.m2.3.3.2.3" xref="A3.F17.10.m2.3.3.3.cmml">,</mo><mi mathvariant="normal" id="A3.F17.10.m2.1.1" xref="A3.F17.10.m2.1.1.cmml">…</mi><mo id="A3.F17.10.m2.3.3.2.4" xref="A3.F17.10.m2.3.3.3.cmml">,</mo><msub id="A3.F17.10.m2.3.3.2.2" xref="A3.F17.10.m2.3.3.2.2.cmml"><mi id="A3.F17.10.m2.3.3.2.2.2" xref="A3.F17.10.m2.3.3.2.2.2.cmml">x</mi><mi id="A3.F17.10.m2.3.3.2.2.3" xref="A3.F17.10.m2.3.3.2.2.3.cmml">T</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="A3.F17.10.m2.3c"><list id="A3.F17.10.m2.3.3.3.cmml" xref="A3.F17.10.m2.3.3.2"><apply id="A3.F17.10.m2.2.2.1.1.cmml" xref="A3.F17.10.m2.2.2.1.1"><csymbol cd="ambiguous" id="A3.F17.10.m2.2.2.1.1.1.cmml" xref="A3.F17.10.m2.2.2.1.1">subscript</csymbol><ci id="A3.F17.10.m2.2.2.1.1.2.cmml" xref="A3.F17.10.m2.2.2.1.1.2">𝑥</ci><cn type="integer" id="A3.F17.10.m2.2.2.1.1.3.cmml" xref="A3.F17.10.m2.2.2.1.1.3">1</cn></apply><ci id="A3.F17.10.m2.1.1.cmml" xref="A3.F17.10.m2.1.1">…</ci><apply id="A3.F17.10.m2.3.3.2.2.cmml" xref="A3.F17.10.m2.3.3.2.2"><csymbol cd="ambiguous" id="A3.F17.10.m2.3.3.2.2.1.cmml" xref="A3.F17.10.m2.3.3.2.2">subscript</csymbol><ci id="A3.F17.10.m2.3.3.2.2.2.cmml" xref="A3.F17.10.m2.3.3.2.2.2">𝑥</ci><ci id="A3.F17.10.m2.3.3.2.2.3.cmml" xref="A3.F17.10.m2.3.3.2.2.3">𝑇</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="A3.F17.10.m2.3d">x_{1},\dots,x_{T}</annotation></semantics></math> where <math id="A3.F17.11.m3.1" class="ltx_Math" alttext="T=2048" display="inline"><semantics id="A3.F17.11.m3.1b"><mrow id="A3.F17.11.m3.1.1" xref="A3.F17.11.m3.1.1.cmml"><mi id="A3.F17.11.m3.1.1.2" xref="A3.F17.11.m3.1.1.2.cmml">T</mi><mo id="A3.F17.11.m3.1.1.1" xref="A3.F17.11.m3.1.1.1.cmml">=</mo><mn id="A3.F17.11.m3.1.1.3" xref="A3.F17.11.m3.1.1.3.cmml">2048</mn></mrow><annotation-xml encoding="MathML-Content" id="A3.F17.11.m3.1c"><apply id="A3.F17.11.m3.1.1.cmml" xref="A3.F17.11.m3.1.1"><eq id="A3.F17.11.m3.1.1.1.cmml" xref="A3.F17.11.m3.1.1.1"></eq><ci id="A3.F17.11.m3.1.1.2.cmml" xref="A3.F17.11.m3.1.1.2">𝑇</ci><cn type="integer" id="A3.F17.11.m3.1.1.3.cmml" xref="A3.F17.11.m3.1.1.3">2048</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.F17.11.m3.1d">T=2048</annotation></semantics></math>, for all 12 TTT layers in a network with 125M parameters train on the Pile.
The same network is also used for <math id="A3.F17.12.m4.1" class="ltx_Math" alttext="b=1" display="inline"><semantics id="A3.F17.12.m4.1b"><mrow id="A3.F17.12.m4.1.1" xref="A3.F17.12.m4.1.1.cmml"><mi id="A3.F17.12.m4.1.1.2" xref="A3.F17.12.m4.1.1.2.cmml">b</mi><mo id="A3.F17.12.m4.1.1.1" xref="A3.F17.12.m4.1.1.1.cmml">=</mo><mn id="A3.F17.12.m4.1.1.3" xref="A3.F17.12.m4.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="A3.F17.12.m4.1c"><apply id="A3.F17.12.m4.1.1.cmml" xref="A3.F17.12.m4.1.1"><eq id="A3.F17.12.m4.1.1.1.cmml" xref="A3.F17.12.m4.1.1.1"></eq><ci id="A3.F17.12.m4.1.1.2.cmml" xref="A3.F17.12.m4.1.1.2">𝑏</ci><cn type="integer" id="A3.F17.12.m4.1.1.3.cmml" xref="A3.F17.12.m4.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.F17.12.m4.1d">b=1</annotation></semantics></math> (online GD) in the left panel of Figure <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:fig:batch-size</span>.
For layers in the middle, we observe that <math id="A3.F17.13.m5.1" class="ltx_Math" alttext="\|x_{t}\|" display="inline"><semantics id="A3.F17.13.m5.1b"><mrow id="A3.F17.13.m5.1.1.1" xref="A3.F17.13.m5.1.1.2.cmml"><mo stretchy="false" id="A3.F17.13.m5.1.1.1.2" xref="A3.F17.13.m5.1.1.2.1.cmml">‖</mo><msub id="A3.F17.13.m5.1.1.1.1" xref="A3.F17.13.m5.1.1.1.1.cmml"><mi id="A3.F17.13.m5.1.1.1.1.2" xref="A3.F17.13.m5.1.1.1.1.2.cmml">x</mi><mi id="A3.F17.13.m5.1.1.1.1.3" xref="A3.F17.13.m5.1.1.1.1.3.cmml">t</mi></msub><mo stretchy="false" id="A3.F17.13.m5.1.1.1.3" xref="A3.F17.13.m5.1.1.2.1.cmml">‖</mo></mrow><annotation-xml encoding="MathML-Content" id="A3.F17.13.m5.1c"><apply id="A3.F17.13.m5.1.1.2.cmml" xref="A3.F17.13.m5.1.1.1"><csymbol cd="latexml" id="A3.F17.13.m5.1.1.2.1.cmml" xref="A3.F17.13.m5.1.1.1.2">norm</csymbol><apply id="A3.F17.13.m5.1.1.1.1.cmml" xref="A3.F17.13.m5.1.1.1.1"><csymbol cd="ambiguous" id="A3.F17.13.m5.1.1.1.1.1.cmml" xref="A3.F17.13.m5.1.1.1.1">subscript</csymbol><ci id="A3.F17.13.m5.1.1.1.1.2.cmml" xref="A3.F17.13.m5.1.1.1.1.2">𝑥</ci><ci id="A3.F17.13.m5.1.1.1.1.3.cmml" xref="A3.F17.13.m5.1.1.1.1.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.F17.13.m5.1d">\|x_{t}\|</annotation></semantics></math> rises steadily, causing all three losses to rise with it.
Even for these layers, the gap between <math id="A3.F17.14.m6.2" class="ltx_Math" alttext="\ell(W_{0};x_{t})" display="inline"><semantics id="A3.F17.14.m6.2b"><mrow id="A3.F17.14.m6.2.2" xref="A3.F17.14.m6.2.2.cmml"><mi mathvariant="normal" id="A3.F17.14.m6.2.2.4" xref="A3.F17.14.m6.2.2.4.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="A3.F17.14.m6.2.2.3" xref="A3.F17.14.m6.2.2.3.cmml">​</mo><mrow id="A3.F17.14.m6.2.2.2.2" xref="A3.F17.14.m6.2.2.2.3.cmml"><mo stretchy="false" id="A3.F17.14.m6.2.2.2.2.3" xref="A3.F17.14.m6.2.2.2.3.cmml">(</mo><msub id="A3.F17.14.m6.1.1.1.1.1" xref="A3.F17.14.m6.1.1.1.1.1.cmml"><mi id="A3.F17.14.m6.1.1.1.1.1.2" xref="A3.F17.14.m6.1.1.1.1.1.2.cmml">W</mi><mn id="A3.F17.14.m6.1.1.1.1.1.3" xref="A3.F17.14.m6.1.1.1.1.1.3.cmml">0</mn></msub><mo id="A3.F17.14.m6.2.2.2.2.4" xref="A3.F17.14.m6.2.2.2.3.cmml">;</mo><msub id="A3.F17.14.m6.2.2.2.2.2" xref="A3.F17.14.m6.2.2.2.2.2.cmml"><mi id="A3.F17.14.m6.2.2.2.2.2.2" xref="A3.F17.14.m6.2.2.2.2.2.2.cmml">x</mi><mi id="A3.F17.14.m6.2.2.2.2.2.3" xref="A3.F17.14.m6.2.2.2.2.2.3.cmml">t</mi></msub><mo stretchy="false" id="A3.F17.14.m6.2.2.2.2.5" xref="A3.F17.14.m6.2.2.2.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.F17.14.m6.2c"><apply id="A3.F17.14.m6.2.2.cmml" xref="A3.F17.14.m6.2.2"><times id="A3.F17.14.m6.2.2.3.cmml" xref="A3.F17.14.m6.2.2.3"></times><ci id="A3.F17.14.m6.2.2.4.cmml" xref="A3.F17.14.m6.2.2.4">ℓ</ci><list id="A3.F17.14.m6.2.2.2.3.cmml" xref="A3.F17.14.m6.2.2.2.2"><apply id="A3.F17.14.m6.1.1.1.1.1.cmml" xref="A3.F17.14.m6.1.1.1.1.1"><csymbol cd="ambiguous" id="A3.F17.14.m6.1.1.1.1.1.1.cmml" xref="A3.F17.14.m6.1.1.1.1.1">subscript</csymbol><ci id="A3.F17.14.m6.1.1.1.1.1.2.cmml" xref="A3.F17.14.m6.1.1.1.1.1.2">𝑊</ci><cn type="integer" id="A3.F17.14.m6.1.1.1.1.1.3.cmml" xref="A3.F17.14.m6.1.1.1.1.1.3">0</cn></apply><apply id="A3.F17.14.m6.2.2.2.2.2.cmml" xref="A3.F17.14.m6.2.2.2.2.2"><csymbol cd="ambiguous" id="A3.F17.14.m6.2.2.2.2.2.1.cmml" xref="A3.F17.14.m6.2.2.2.2.2">subscript</csymbol><ci id="A3.F17.14.m6.2.2.2.2.2.2.cmml" xref="A3.F17.14.m6.2.2.2.2.2.2">𝑥</ci><ci id="A3.F17.14.m6.2.2.2.2.2.3.cmml" xref="A3.F17.14.m6.2.2.2.2.2.3">𝑡</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.F17.14.m6.2d">\ell(W_{0};x_{t})</annotation></semantics></math> and <math id="A3.F17.15.m7.2" class="ltx_Math" alttext="\ell(W_{t};x_{t})" display="inline"><semantics id="A3.F17.15.m7.2b"><mrow id="A3.F17.15.m7.2.2" xref="A3.F17.15.m7.2.2.cmml"><mi mathvariant="normal" id="A3.F17.15.m7.2.2.4" xref="A3.F17.15.m7.2.2.4.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="A3.F17.15.m7.2.2.3" xref="A3.F17.15.m7.2.2.3.cmml">​</mo><mrow id="A3.F17.15.m7.2.2.2.2" xref="A3.F17.15.m7.2.2.2.3.cmml"><mo stretchy="false" id="A3.F17.15.m7.2.2.2.2.3" xref="A3.F17.15.m7.2.2.2.3.cmml">(</mo><msub id="A3.F17.15.m7.1.1.1.1.1" xref="A3.F17.15.m7.1.1.1.1.1.cmml"><mi id="A3.F17.15.m7.1.1.1.1.1.2" xref="A3.F17.15.m7.1.1.1.1.1.2.cmml">W</mi><mi id="A3.F17.15.m7.1.1.1.1.1.3" xref="A3.F17.15.m7.1.1.1.1.1.3.cmml">t</mi></msub><mo id="A3.F17.15.m7.2.2.2.2.4" xref="A3.F17.15.m7.2.2.2.3.cmml">;</mo><msub id="A3.F17.15.m7.2.2.2.2.2" xref="A3.F17.15.m7.2.2.2.2.2.cmml"><mi id="A3.F17.15.m7.2.2.2.2.2.2" xref="A3.F17.15.m7.2.2.2.2.2.2.cmml">x</mi><mi id="A3.F17.15.m7.2.2.2.2.2.3" xref="A3.F17.15.m7.2.2.2.2.2.3.cmml">t</mi></msub><mo stretchy="false" id="A3.F17.15.m7.2.2.2.2.5" xref="A3.F17.15.m7.2.2.2.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.F17.15.m7.2c"><apply id="A3.F17.15.m7.2.2.cmml" xref="A3.F17.15.m7.2.2"><times id="A3.F17.15.m7.2.2.3.cmml" xref="A3.F17.15.m7.2.2.3"></times><ci id="A3.F17.15.m7.2.2.4.cmml" xref="A3.F17.15.m7.2.2.4">ℓ</ci><list id="A3.F17.15.m7.2.2.2.3.cmml" xref="A3.F17.15.m7.2.2.2.2"><apply id="A3.F17.15.m7.1.1.1.1.1.cmml" xref="A3.F17.15.m7.1.1.1.1.1"><csymbol cd="ambiguous" id="A3.F17.15.m7.1.1.1.1.1.1.cmml" xref="A3.F17.15.m7.1.1.1.1.1">subscript</csymbol><ci id="A3.F17.15.m7.1.1.1.1.1.2.cmml" xref="A3.F17.15.m7.1.1.1.1.1.2">𝑊</ci><ci id="A3.F17.15.m7.1.1.1.1.1.3.cmml" xref="A3.F17.15.m7.1.1.1.1.1.3">𝑡</ci></apply><apply id="A3.F17.15.m7.2.2.2.2.2.cmml" xref="A3.F17.15.m7.2.2.2.2.2"><csymbol cd="ambiguous" id="A3.F17.15.m7.2.2.2.2.2.1.cmml" xref="A3.F17.15.m7.2.2.2.2.2">subscript</csymbol><ci id="A3.F17.15.m7.2.2.2.2.2.2.cmml" xref="A3.F17.15.m7.2.2.2.2.2.2">𝑥</ci><ci id="A3.F17.15.m7.2.2.2.2.2.3.cmml" xref="A3.F17.15.m7.2.2.2.2.2.3">𝑡</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.F17.15.m7.2d">\ell(W_{t};x_{t})</annotation></semantics></math> still increases with <math id="A3.F17.16.m8.1" class="ltx_Math" alttext="t" display="inline"><semantics id="A3.F17.16.m8.1b"><mi id="A3.F17.16.m8.1.1" xref="A3.F17.16.m8.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="A3.F17.16.m8.1c"><ci id="A3.F17.16.m8.1.1.cmml" xref="A3.F17.16.m8.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.F17.16.m8.1d">t</annotation></semantics></math> .
For visual clarity, loss values have been averaged over a sliding window of 10 timesteps.
</figcaption>
</figure>
<figure id="A3.F18" class="ltx_figure"><img src="/html/2407.04620/assets/x18.png" id="A3.F18.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="553" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 18: </span>Complete results on Books, presented by context lengths.
Figure <a href="#S3.F12" title="Figure 12 ‣ 3.2 Long context: Books ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12</span></a> in Subsection <a href="#S3.SS2" title="3.2 Long context: Books ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a> presents the subset of results for context lengths 2k and 32k.
</figcaption>
</figure>
<figure id="A3.F19" class="ltx_figure"><img src="/html/2407.04620/assets/x19.png" id="A3.F19.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="322" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 19: </span>An alternative view of our complete results on Books, presented by model sizes, with context length as the x-axis.
For all methods trained from scratch, perplexity becomes worse once the context length becomes too large. This trend is not observed with TF finetune, except for one case at the 125M scale.
The best context length increases for larger models (trained from scratch).
</figcaption>
</figure>
<figure id="A3.F20" class="ltx_figure"><img src="/html/2407.04620/assets/x20.png" id="A3.F20.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="184" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 20: </span>
Throughput for forward and generate. All the observations and ordering between methods from Figure <a href="#S3.F15" title="Figure 15 ‣ 3.3 Wall-clock time ‣ 3 Experiments ‣ Learning to (Learn at Test Time): RNNs with Expressive Hidden States" class="ltx_ref"><span class="ltx_text ltx_ref_tag">15</span></a> (for latency) still hold.
</figcaption>
</figure>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2407.04619" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2407.04620" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2407.04620">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2407.04620" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2407.04621" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Mon Aug  5 18:07:45 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2407.04620
Self-attention performs well in long context but has quadratic complexity.
Existing RNN layers have linear complexity, but their performance in long context is limited by the expressive power of their hidden state.
We propose a new class of sequence modeling layers with linear complexity and an expressive hidden state.
The key idea is to make the hidden state a machine learning model itself, and the update rule a step of self-supervised learning.
Since the hidden state is updated by training even on test sequences, our layers are calledTest-Time Training (TTT)  layers.
We consider two instantiations: TTT-Linear and TTT-MLP, whose hidden state is a linear model and a two-layer MLP respectively.
We evaluate our instantiations at the scale of 125M to 1.3B parameters, comparing with a strong Transformer and Mamba, a modern RNN.
Both TTT-Linear and TTT-MLP match or exceed the baselines.
Similar to Transformer, they can
keep reducing perplexity by conditioning on more tokens, while Mamba cannot after 16k context.
With preliminary systems optimization, TTT-Linear is already faster than Transformer at 8k context and matches Mamba in wall-clock time.
TTT-MLP still faces challenges in memory I/O, but shows larger potential in long context, pointing to a promising direction for future research.
abstract
Learning to (Learn at Test Time):RNNs with Expressive Hidden States
Abstract
2407.04620
1IntroductionIn 2020, the OpenAI scaling law paper (Kaplan et. al)  showed that LSTMs (a type of RNN)  could not scale similarly to Transformers or effectively use long context.
Now, with modern RNNs and best practices, we re-evaluate these findings in Figure2.On the left, we observe that Mamba– one of the most popular RNNs today – scales similarly to a strong Transformer, showing great progress since the LSTMs in 2020.
However, on the right, we observe the same issue with Mamba as Kaplan et al. did with LSTMs.
Tokens later in a sequence should be easier to predict on average, since they condition on more information.
This is indeed the case for Transformer, whose average perplexity at each token index decreases throughout its 32k context.
In contrast, the same metric plateaus for Mamba after 16k.This result represents an awkward reality for existing RNNs.
On one hand, the main advantage of RNNs (vs. Transformers)  is their linear (vs. quadratic)  complexity. This asymptotic advantage is only realized in practice for long context, which according to Figure3is after 8k.
On the other hand, once context is long enough, existing RNNs such as Mamba struggle to actually take advantage of the extra information being conditioned on.The difficulty with long context is inherent to the very nature of RNN layers:
Unlike self-attention, RNN layers have to compress context into a hidden state of fixed size.
As a compression heuristic, the update rule needs to discover the underlying structures and relationships among thousands or potentially millions of tokens.
In this paper, we begin with the observation that self-supervised learning can compress a massive training set into the weights of a model such as an LLM, which often exhibits deep understanding about the semantic connections among its training data – exactly what we need from a compression heuristic.TTT layers.Motivated by this observation, we design a new class of sequence modeling layers where the hidden state is a model, and the update rule is a a step of self-supervised learning.
Because the process of updating the hidden state on a test sequence is equivalent to training a model at test time, this new class of layers is calledTest-Time Training (TTT)  layers.
We introduce two simple instantiations within this class: TTT-Linear and TTT-MLP, where the hidden state is a linear model and a two-layer MLP, respectively.
TTT layers can be integrated into any network architecture and optimized end-to-end, similar to RNNs layers and self-attention.Wall-clock time.While the TTT layer is already efficient in FLOPs, we propose two practical innovations to make it efficient in wall-clock time. First, similar to the standard practice of taking gradient steps on mini-batches of sequences during regular training for better parallelism, we use mini-batches of tokens during TTT.
Second, we develop a dual form for operations inside each TTT mini-batch, to better take advantage of modern GPUs and TPUs.
The dual form is equivalent in output to the naive implementation, but trains more thanfaster. As shown in Figure3, TTT-Linear is faster than Transformer at 8k context and matches Mamba.Evaluations and open problems.While we have highlighted some results for TTT-Linear at the beginning of the paper, Section3presents more comprehensive evaluations for both TTT-Linear and TTT-MLP, and open problems exposed by our evaluations.
For example, our evaluations following the Chinchilla recipedo not cleanly fit a linear scaling trend even for the Transformer baseline.
Constrained by our academic resources, we encourage the community to join us in exploring solutions to these problems.Summary of contributions.1.We propose TTT layers, a new class of sequence modeling layers where the hidden state is a model, and the update rule is self-supervised learning.
Our perspective that the forward pass of a layer contains a training loop itself opens up a new direction for future research.2.TTT-Linear, one simple instantiation of TTT layers, outperforms Transformers and Mamba in our evaluations ranging from 125M to 1.3B parameters.3.We improve the hardware efficiency of TTT layers through mini-batch TTT and the dual form, making TTT-Linear already a practical building block for LLMs.Initial stateUpdate ruleOutput ruleCostNaive RNNSelf-attentionNaive TTTFigure 4:Top: A generic sequence modeling layer expressed as a hidden state that transitions according to an update rule.
All sequence modeling layers can be viewed as different instantiations of three components in this figure: the initial state, update rule and output rule.Bottom: Examples of sequence modeling layers and their instantiations of the three components. The naive TTT layer was shown in Figure1.
Self-attention has a hidden state growing with context, therefore growing cost per token.
Both the naive RNN and TTT layer compress the growing context into a hidden state of fixed size, therefore their cost per token stays constant.
introduction
Learning to (Learn at Test Time):RNNs with Expressive Hidden States
1Introduction
2407.04620
2MethodAll sequence modeling layers can be viewed from the perspective of storing historic context into a hidden state, as shown in Figure4.111We define a sequence modeling layer as an autoregressive mapping from one sequence to another.For example, RNN layers – such as LSTM, RWKVand Mambalayers – compress context into a state of fixed size across time.
This compression has two consequences.
On one hand, mapping an input tokento output tokenis efficient, because both the update rule and output rule take constant time per token.
On the other hand, the performance of RNN layers in long context is limited by the expressive power of its hidden state.Self-attention can also be viewed from the perspective above, except that its hidden state, commonly known as the Key-Value (KV)  cache, is a list that grows linearly with.
Its update rule simply appends the current KV tuple to this list, and the output rule scans over all tuples up toto form the attention matrix.
The hidden state explicitly stores all historic context without compression, making self-attention more expressive than RNN layers for long context.
However, scanning this linearly growing hidden state also takes linearly growing time per token.To remain both efficient and expressive in long context, we need a better compression heuristic.
Specifically, we need to compress thousands or potentially millions of tokens into a hidden state that can effectively capture their underlying structures and relationships.
This might sound like a tall order, but all of us are actually already familiar with such a heuristic.2.1TTT as updating a hidden stateThe process of parametric learning can be viewed as compressing a massive training set into the weights of a model.
Specifically, we know that models trained with self-supervision can capture the underlying structures and relationships behind their training data– exactly what we need from a compression heuristic.LLMs themselves are great examples. Trained with the self-supervised task of next-token prediction, their weights can be viewed as a compressed form of storage for existing knowledge on the internet.
By querying LLMs, we can extract knowledge from their weights.
More importantly, LLMs often exhibit a deep understanding of the semantic connections among existing knowledge to express new pieces of reasoning.Our key idea is to use self-supervised learning to compress the historic contextinto a hidden state, by making the context an unlabeled dataset and the state a model.
Concretely, the hidden stateis now equivalent to, the weights of a model, which can be a linear model, a small neural network, or anything else.
The output rule is simply:(1) Intuitively, the output token is just the prediction on, made bywith the updated weights.
The update rule is a step of gradient descent on some self-supervised loss:(2) with learning rate.222For now, consider. We will discuss more sophisticated techniques for initializingin Subsection2.7.From the compression point of view, every heuristic needs to decide which input to remember or forget. Ourremembers inputs that produce large gradients – intuitively, inputs that makelearn a lot.One choice ofis reconstructingitself.
To make the learning problem nontrivial, we first processinto a corrupted input(details in Subsection2.3) , then optimize:(3) Similar to denoising autoencoders,needs to discover the correlations between dimensions ofin order to reconstruct it from partial information.333In past experiments, we have also tried adding another model(decoder)  after(encoder) , such that the reconstruction is produced byinstead of onlyitself.
While this heftier design did slightly improve results, it made overall training less stable and added significant computational cost.
Therefore we focus on the encoder-only design.As shown in Figure5, gradient descent is able to reduce, but cannot reduce it to zero.
We discuss more sophisticated formulations of the self-supervised task in Subsection2.3.As with other RNN layers and self-attention, our algorithm that maps an input sequenceto output sequencecan be programmed into the forward pass of a sequence modeling layer, using the hidden state, update rule, and output rule above.
Even at test time, our new layer still trains a different sequence of weightsfor every input sequence. Therefore, we call it theTest-Time Training (TTT)  layer.Figure 5:The self-supervised TTT lossaveraged over all test sequences of the formwhere, for the first three TTT layers in a network with 125M parameters.
One step of gradient descent is able to reduce TTT loss fromto.
Asmoves further along the test sequence,also improves further from.
For visual clarity, loss values have been averaged over a sliding window of 10 timesteps.
See Figure17(in Appendix)  for complete results on all 12 layers.2.2Training a network with TTT layersThe forward pass of a TTT layer also has a corresponding backward pass.Our forward pass onlyconsists of standard differentiable operators except the gradient operator.
However,just maps one function to another, in this caseto, andis also composed of differentiable operators.
Conceptually, calling backward onmeans taking gradients of gradients – a well explored technique in meta-learning.TTT layers have the same interface as RNN layers and self-attention, therefore can be replaced in any larger network architecture, which usually contains many of these sequence modeling layers.
Training a network with TTT layers also works the same way as training any other language model, such as a Transformer.
The same data, recipe, and objective such as next-token prediction can be used to optimize parameters of the rest of the network.We refer to training the larger network as theouter loop, and trainingwithin each TTT layer as theinner loop.
An important difference between the two nested learning problems is that the inner-loop gradientis taken w.r.t., the parameters of, while the outer-loop gradient is taken w.r.t the parameters of the rest of the network, which we will denote by.
Throughout this paper, outer-loop parameters are always denoted bywith various subscripts.So far, the TTT layer has no outer-loop parameters, in contrast to other RNN layers and self-attention.
In Subsection2.3, we add outer-loop parameters to the TTT layer to improve its self-supervised task.
Then in Subsection2.4and2.5, we discuss two ways to improve the wall-clock time of TTT layers.⬇classTTT_Layer(nn.Module) :def__init__(self) :self.task=Task() defforward(self,in_seq) :state=Learner(self.task) out_seq=[]fortokinin_seq:state.train(tok) out_seq.append(state.predict(tok) ) returnout_seqclassTask(nn.Module) :def__init__(self) :self.theta_K=nn.Param((d1,d2) ) self.theta_V=nn.Param((d1,d2) ) self.theta_Q=nn.Param((d1,d2) ) defloss(self,f,x) :train_view=self.theta_K@xlabel_view=self.theta_V@xreturnMSE(f(train_view) ,label_view) classLearner() :def__init__(self,task) :self.task=task#Linearhere,butcanbeanymodelself.model=Linear() #onlineGDhereforsimplicityself.optim=OGD() deftrain(self,x) :#gradfunctionwrtfirstarg#ofloss,whichisself.modelgrad_fn=grad(self.task.loss) #calculateinner-loopgradgrad_in=grad_fn(self.model,x) #startingfromcurrentparams,#stepindirectionofgrad_in,self.optim.step(self.model,grad_in) defpredict(self,x) :test_view=self.task.theta_Q@xreturnself.model(test_view) Figure 6:Naive implementation of a TTT layer with a linear model and online GD in the style of PyTorch.TTT_Layercan be dropped into a larger network like other sequence modeling layers.
Training the network will optimize the parameters ofTaskinTTT_Layer, because both are subclasses ofnn.Module.
SinceLearneris not a subclass ofnn.Module,state.modelis updated manually in the inner loop for each call ofstate.train.
For simplicity, we sometimes overloadmodelasmodel.parameters.2.3Learning a self-supervised task for TTTArguably the most important part of TTT is the self-supervised task, because it determines the kind of features thatwill learn from the test sequence.
So how should we design this task?
The final goal of TTT is forto perform well on language modeling.
Instead of handcrafting a self-supervised task from human priors, we take a more end-to-end approach – directly optimizing the self-supervised task for the final goal of next-token prediction.Concretely, we learn the self-supervised task as part of the outer loop.
Starting from the naive reconstruction task in Equation3, we add some outer-loop parameters to make this task learnable.
In Subsection2.1, we did not specify the corruption that producesfrom. One design is to make it a low-rank projection, whereis a learnable matrix.444The subscripthints at a connection to self-attention, as we will establish in Subsection2.6.Following the terminology of multi-view reconstruction,is called atraining view.Moreover, perhaps not all the information inis worth remembering, so the reconstruction label can be another low-rank projectioninstead of.
Hereis called thelabel view, whereis also learnable.
In summary, our new self-supervised loss is:(4) Since bothand variouss appear together in Equation4, we emphasize again their difference in nature.
In the inner loop, onlyis optimized, therefore written as an argument of; thes are "hyper-parameters" of this loss function.
In the outer loop,are optimized alongside, andis merely a hidden state, not a parameter.
Figure6illustrates this difference with code, whereandare implemented as parameters of the TTT layer,
analogous to the Key and Value parameters of self-attention.Lastly, the training viewhas fewer dimensions than, so we can no longer use the output rule in Equation1.
The simplest solution is to create atest view, and change our output rule to:(5) This solution has an additional benefit.
The training and label views specify the information inthat is compressed intoand propagated forward through time.
The test view specifies potentially different information that is mapped to the current output tokenand propagated forward through network layers, therefore adds more flexibility to the self-supervised task.Altogether, the set of all possible choices forinduces a family of multi-view reconstruction tasks, and the outer loop can be interpreted as selecting a task from this family.
Here we have designed all views as linear projections for simplicity.
Future work might experiment with more flexible transformations, or bigger and different families of self-supervised tasks.2.4Parallelization with mini-batch TTTThe naive TTT layer developed so far is already efficient in the number of floating point operations (FLOPs) .
However, its update rulecannot be parallelized,
becausedepends onin two places: before the minus sign and inside.
Sincecontains the bulk of the computation, we focus on making this second part parallel.We approach this systems challenge through concepts in the TTT framework. There are many variants of gradient descent (GD) .
The general update rule of GD can be expressed as:(6) whereis the descent direction.
Note that once we have calculatedfor, we can then obtain all thes through acumsumby the second half of Equation6.
Our naive update rule, known asonline gradient descent,
uses.To parallelizefor, we can take all of them w.r.t..
This variant withis known asbatch gradient descent, sinceis the same as the gradient w.r.t.overas a batch.
However, in batch GD,is effectively only one gradient step away from, in contrast to online GD, whereissteps away from.
Therefore, batch GD has a smaller effective search space, which ends up hurting performance for language modeling.Figure 7:High-level computation graph of the first TTT mini-batch, where nodes are variables and edges are computations.
The blue nodes are input variables, and yellow are output.Subsection2.4: Sinceare not connected, they have no sequential dependency on each other, therefore can be computed in parallel.Subsection2.5:
We do not actually materialize the white nodes – the intermediates ands – to compute the output variables in the dual form.Our proposed solution –mini-batch gradient descent– is shown in Figure7.
Denote the TTT batch size by. We use, wheremodis the last timestep of the previous mini-batch (or 0 for the first mini-batch) , so we can parallelizegradient computations at a time.
Empirically,controls a trade-off between speed and quality, as shown in FigureLABEL:fig:batch-size.
We chosefor all experiments in this paper.In summary, there are two potential channels to propagate information fromtowhere:cumsumand the gradient operator. Thecumsumis always active, but the gradient channel is only active whenis from a previous mini-batch.
Different variants of gradient descent only affect the gradient channel,i.e., the descent direction, specifically w.r.t. whichthe gradient is taken.
However, the descent stepalways starts from, due to the autoregressive nature of the update rule, which is orthogonal to the choice of.Figure 8:Ablations on TTT mini-batch size, whereis online GD andis batch GD.
We choosefor all experiments in this paper.Left: Smallerimproves perplexity since more GD steps are taken.555In theory,can potentially be too small such that the variance between mini-batches is too high, hurting optimization.
However, we have not observed such an effect in practice.The perplexity of 11.09 atcorresponds to the final result of TTT-Linear in Figure11.Right: Forward time in dual form, with context length.
Total time (orange)  can be decomposed into time for computing thes at the end of every mini-batch (blue)  and time for(orangeblue) .666For FigureLABEL:fig:batch-size, we use a single TTT layer in TTT-Linear 1.3B, implemented in pure PyTorch.
Our fused kernel significantly improves time efficiency, but makes it difficult to cleanly decompose the time for computingvs..Time complexity for thes is, constant in, but the blue line decreases as largerallows more parallelization until hardware utilization saturates.
Time complexity foris, so the orange line first decreases with more parallelization, then increases as the extra computation forbecomes dominant.2.5Dual formThe parallelization introduced above is necessary but not sufficient for efficiency in wall-clock time.
Modern accelerators specialize in matrix-matrix multiplications, known asmatmuls.
For example, the NVIDIA A100 GPU contains highly optimized units called TensorCores that can only perform a single operation – multiplying two matrices each of size.
Without enough of thesematmuls, the TensorCores are idle, and most of the potential for the A100 is unrealized.Unfortunately, the TTT layer developed so far even with mini-batch still has very fewmatmuls.
Consider the simplest case of, where, for only the first TTT mini-batch of size.
In addition, consideras a linear model.
Copying Equation3,
our loss at timeis:As discussed in Subsection2.4, we can parallelize the computation of:for.
However, we cannot compute allof thes through a singlematmul.
Instead, we needouter products to compute them one by one.
To make matters worse, for each,is, which incurs much heavier memory footprint and I/O cost thanfor large.To solve these two problems, we make a simple observation: We do not actually need to materializeas long as we can computeat the end of the mini-batch, and the output tokens(see Figure7) .
Now we demonstrate these computations with the simplified TTT-Linear case above.
Denote, then:Socan be conveniently computed with amatmul.
To compute, we know that:(7) Denoteand the matrix.
We can derive that:wheremaskis the lower triangular mask with zeros (similar to the attention mask, but with zeros instead of infinities) , and the termcan be reused from the computation of.
Nowis also conveniently computed withmatmuls.
Pluggingback into Equation7, we obtain.We call this procedure thedual form, in contrast to theprimal formbefore this subsection, where thes ands are explicitly materialized.
As discussed, the two forms are equivalent in output.
The terminology of primal and dual follows prior work that has explored similar mathematical formulations outside of TTT.
In AppendixA, we show that the dual form still works whenis a neural network with nonlinear layers, except with more complicated notation.Time complexity of the primal form within a TTT mini-batch is.
Time complexity of the dual form isfor computingalone, then an additionalfor computing.
Compared to the primal, the dual form sacrifices theoretical complexity for hardware utilization.
In practice,is typically a few hundred andis chosen to be only 16. As a consequence, wall-clock time for computingis relatively small, as observed in the right panel of FigureLABEL:fig:batch-size.
In our JAX implementation, training with the dual form is more thanfaster than with primal.2.6Theoretical equivalencesFigure 9:Parametric learners need to define two attributes: model and optimizer (left) , and each learner uniquely induces a TTT layer (right) .
Two of the induced TTT layers: TTT-Linear and TTT-MLP, are proposed in this paper.
The TTT layer with a linear model and batch GD is equivalent to linear attention.In Subsection2.1, we mentioned thatcan be a linear model or a neural network.
In Subsection2.4, we also discussed three variants of the update rule: online GD, batch GD, and mini-batch GD.
Each of thesecombinations induces a different instantiation of the TTT layer, as illustrated in Figure9.
We now show that among these induced instantiations, the TTT layer with a linear model and batch GD is equivalent to linear attention, a widely known RNN layer.777In a nutshell, linear attentionis simply self-attention without the softmax.
Recall the definition of self-attention:.
Withoutsoftmax, this becomes, which is the simplest formulation of linear attention.
Similar to other RNN layers, it can be written in a recurrent form, whereis the hidden state.
Sincecan be computed in acumsumfor every, linear attention also has linear complexity w.r.t..Theorem 1.Consider the TTT layer withas the inner-loop model, batch gradient descent withas the update rule, and. Then, given the same input sequence, the output rule defined in Equation5produces the same output sequenceas linear attention.Proof.By definition ofin Equation4,.
By definition of batch GD in Equation6:Plugginginto the output rule in Equation5, we obtain the output token:which is the definition of linear attention.
∎In Table1, we first empirically verify the equivalence above with an improved implementation of linear attention.888The original formulation of linear attention incontains a normalizer and a feature expansion on, which can still be included in an equivalent TTT layer.
However, prior work has found that these two additions can hurt performance, which we have verified in our own experiment (first vs. second row of Table1) .
Therefore, we only construct a TTT layer equivalent to the simplest formulation of linear attention without the two additions.Then, to illustrate the contribution of each of our components (including some that will be introduced in the next subsection) , we add them row by row to the TTT layer that is equivalent to linear attention, and ultimately obtain our proposed instantiation calledTTT-Linear.
The change from batch GD to mini-batch GD contributes the most improvement by a large margin.Figure 10:RNN layers and TTT layers are both subsets ofsequence modeling layers.RNN layers have a hidden state that is fixed in size across time.
TTT layers with parametric learners are also RNN layers, since their hidden state is also fixed in size.
TTT layers with nonparametric learners can represent self-attention, as discussed in Subsection2.6.While the space of modelsoptimizers in Figure9is already large,
machine learning is much richer than optimizing the parametersof a model.
There are also nonparametric learners, such as nearest neighbors, support vector machines (SVMs) , and kernel ridge regression.
By definition, nonparametric learners do not have parameters, and instead directly uses training data.
Hence we use the notation.
We now show that for a particular nonparametric learner, the induced TTT layer is equivalent to self-attention.Theorem 2.Consider the TTT layer with the Nadaraya-Watson estimator, defined as:(8) whereis the label view discussed in Subsection2.3, and(9) is a kernel with bandwidth hyper-parametersand.
Then given the same input sequence, the output rule defined in Equation5produces the same output sequenceas self-attention.Proof.Pluggingandabove into Equation8gives us the definition of self-attention.
∎AppendixBcontains a detailed explanation of the Nadaraya-Watson estimator and kernelabove.
In contrast to Theorem1, Theorem2does not produce a different implementation from attention.For the TTT layer above, the hidden state isor a similar list of processed training data, the update rule addsto the list, and the output rule scans the list with.
In previous subsections, our hidden state has been defined as, the update rule a gradient step, and the output rule a call to.
To unify these two constructions, we define a new abstraction called a learner, which uniquely induces a TTT layer.Similar to its definition in standard machine learning packages, all learners need to implement two methods:train and predict.Now we redefine the hidden state of the induced TTT layer as the internal storage of the learner, and the update and output rules as the train and predict methods.
Under this new definition of TTT layers, both parametric learners such as that in Theorem1and nonparametric learners such as that in Theorem2can be included.
Figure10summarizes this general definition of TTT layers in the broader scope of all sequence modeling layers.This general definition has an additional benefit for parametric learners:
There can be more objects other thanin the internal storage of parametric learners, such as the optimizer state, which will also be included in the hidden state of the induced TTT layer.
This extension allows TTT layers to use more sophisticated optimizers such as Adamin future work.ConfigurationPpl.Diff.Linear attention15.91-Linear attn. improved15.230.68TTT equivalence15.230+ learnable15.27+0.04+ LN and residual in14.051.22+ mini-batch TTT12.351.70+ learnable11.990.36+ Mamba backbone11.090.90Table 1:Ablations on improving from linear attention.
All models here have 125M parameters, and are trained according to the recipe in Subsection3.1.
The last row, with perplexity 11.09, is the final result of TTT-Linear in Figure11.
Starting from the equivalence discussed in Subsection2.6, learnablehurts slightly, but the rows below cannot train stably without it.
The biggest improvement comes from mini-batch TTT (changing fromto) .
The second comes from instantiating the inner modelwith LN and residual connection.
Both of these designs would be difficult to come across without the conceptual framework of TTT.2.7Implementation detailsInstantiations of.We propose two variants of TTT layers – TTT-Linear and TTT-MLP, differing only in their instantiations of.
For TTT-Linear,, whereis square.
For TTT-MLP,has two layers similar to the MLPs in Transformers.
Specifically, the hidden dimension isthe input dimension, followed by a GELU activation.
For better stability during TTT,always contains a Layer Normalization (LN)  and residual connection. That is,wherecan beor.Learnable.The TTT initializationis shared between all sequences, even though subsequent weightsare different for each input sequence.
Instead of setting, we can learn it as part of the outer loop.
Since outer-loop parameters are always denoted bys instead ofs, we assign an alias.
In practice,adds a negligible amount of parameters comparing to the reconstruction views, because both its input and output are low dimensional.
Empirically, we observe that learningsignificantly improves training stability.Learnable.The learning rate is usually the most important hyper-parameter for gradient descent, so we experiment with learning the inner-loop learning ratein Equation6as part of the outer loop.
We makea function of the input token (therefore different across time)  for additional flexibility.
Concretely, we design, where the learnable vectoris an outer-loop parameter,is the sigmoid function, and the scalaris the base learning rate, set to 1 for TTT-Linear and 0.1 for TTT-MLP.
Alternatively,can also be interpreted as a gate for.Backbone architecture.The cleanest way to integrate any RNN layer into a larger architecture would be to directly replace self-attention in a Transformer, known in this context as a backbone.
However, existing RNNs such as Mambaand Griffinall use a different backbone from Transformers.
Most notably, their backbone contains temporal convolutions before the RNN layers, which might help collect local information across time.
After experimenting with the Mamba backbone, we find that it also improves perplexity for TTT layers, so we incorporate it into our proposed method.
See Figure16(in Appendix)  for details.
body
Learning to (Learn at Test Time):RNNs with Expressive Hidden States
2Method
2407.04620
3ExperimentsWe evaluate TTT-Linear and TTT-MLP by comparing with two baselines
– Transformer and Mamba, a modern RNN.
Our main codebase is based on EasyLM, an open-source project for training and serving LLMs in JAX.
All experiments can be reproduced using the publicly available code and datasets provided at the bottom of the first page.Datasets.Following the Mamba paper, we perform standard experiments with 2k and 8k context lengths on the Pile, a popular dataset of documents for training open-source LLMs.
However, the Pile contains few sequences of length greater than 8k.
To evaluate capabilities in long context, we also experiment with context lengths ranging from 1k to 32k inincrements, using a subset of the Pile called Books3, which has been widely used to train LLMs in long context.Backbone architecture.As discussed in Subsection2.7, Transformer and Mamba use different backbones, and TTT-Linear and TTT-MLP always use the Mamba backbone unless noted otherwise.
As an ablation study, Figure11and Figure12contain TTT layers within the Transformer backbone.
When a figure contains both the Transformer backbone and Mamba backbone, we denote them by(T) and(M) , respectively.Protocols.To ensure fairness to our baselines, we strictly follow the evaluation protocols in the Mamba paper when possible:•For each evaluation setting (e.g., dataset, context length, and method) , we experiment with four model sizes: 125M, 350M, 760M, and 1.3B parameters. For Mamba, the corresponding sizes are 130M, 370M, 790M, and 1.4B, as Mamba does not follow the Transformer configurations.•All models are trained with the Chinchilla recipe999The Chinchilla paper is another highly influential study of empirical scaling laws.
From large-scale experiments with many hyper-parameters, they observe that the compute-optimal models follow a particular training recipe.
We only follow the Chinchilla recipe used in the Mamba paper, which may be slightly different from the original recipe in.described in the Mamba paper and reproduced in our AppendixC.
Our Transformer baseline, based on the Llama architecture, also follows the baseline in the Mamba paper.
As verification, our baselines can reproduce the numbers reported in the Mamba paper in their evaluation settings.101010The only difference between our protocol and that in the Mamba paper is the tokenizer.
The Mamba paper uses two different tokenizers – GPT-2 and GPT-NeoX – for various experiments.
For consistency, we adhere to a single tokenizer throughout this paper and choose the Llama tokenizer, which is the modern state-of-the-art.•We do not experiment with hybrid architectures (e.g. Griffin) , because our baselines are not hybrid.
While hybrid architectures that use both self-attention and TTT layers may improve performance, they would reduce the clarity of our academic evaluation.3.1Short context: the PileFigure 11:Evaluations for context lengths 2k and 8k on the Pile.
Details in Subsection3.1.
TTT-Linear has comparable performance as Mamba at 2k context, and better performance at 8k.From Figure11, we make a few observations:•At 2k context, TTT-Linear (M) , Mamba, and Transformer have comparable performance, as the lines mostly overlap.
TTT-MLP (M)  performs slightly worse under large FLOP budgets.
Even though TTT-MLP has better perplexity than TTT-Linear at every model size, the extra cost in FLOPs offsets the advantage.•At 8k context, both TTT-Linear (M)  and TTT-MLP (M)  perform significantly better than Mamba, in contrast to the observation at 2k. Even TTT-MLP (T)  with the Transformer backbone performs slightly better than Mamba around 1.3B.
A robust phenomenon we observe throughout this paper is that as context length grows longer, the advantage of TTT layers over Mamba widens.•At 8k context, Transformer still has good (if not the best)  perplexity at every model size, but its line is not competitive because of the cost in FLOPs.Effect of backbone.Switching the TTT layers from Mamba backbone into Transformer backbone has two effects.
First, TTT layers with Mamba backbone perform better in our evaluations so far.
Second, with Mamba backbone, TTT-MLP at best is only comparable to TTT-Linear; but with Transformer backbone, TTT-MLP is clearly better.
We hypothesize that the temporal convolutions in the Mamba backbone help more when the sequence modeling layer has a less expressive hidden state.
The linear model is less expressive than the MLP, therefore benefits more from the convolutions.We will revisit this hypothesis in the next subsection.Lack of linear fit.The Chinchilla paper empirically observed that the compute-optimal models following their recipe fall onto a line in the log-log plot of FLOPs vs. perplexity, as is often the case for scaling law experiments.
However, we do not observe a clean linear fit in Figure11or Figure12(the analogous experiments in Books) , not even for Transformers.
This is not surprising given the differences in dataset, context length, tokenizer, and architecture.
Following the Mamba paper, we connect the points instead of fitting them with linear regression due to the large error.111111Ideally, we would have rerun all the hyper-parameters and derived a potentially new recipe for each method based on our evaluation setting, following the process in the Chinchilla paper.
If the new compute-optimal models do fall onto a line, we could then predict performance beyond the current FLOPs regime. However, this empirical study would require orders of magnitude more resources than ours.3.2Long context: BooksFigure 12:Evaluations for context lengths 2k and 32k on Books.
Details in Subsection3.2.
Our complete results for context lengths 1k, 2k, 4k, 8k, 16k, 32k, including Transformer finetuning, are in Figure18(in Appendix) .
Most observations from the Pile still hold.To evaluate capabilities in long context, we experiment with context lengths ranging from 1k to 32k inincrements, using a popular subset of the Pile called Books3.
The training recipe here is the same as for the Pile, and all experiments for the TTT layers areperformed in one training run.121212Following the Mamba paper, we always use 0.5M tokens per training batch regardless of context length. That means for context lengthwe havesequences per batch (assume divisible) .From the subset of results in Figure12, we make a few observations:•At 2k context on Books, all the observations from Pile 2k still hold, except that Mamba now performs slightly better than TTT-Linear (whereas their lines roughly overlapped for Pile 2k) .•At 32k context, both TTT-Linear (M)  and TTT-MLP (M)  perform better than Mamba, similar to the observation from Pile 8k. Even TTT-MLP (T)  with the Transformer backbone performs slightly better than Mamba at 32k context.•TTT-MLP (T)  is only slightly worse than TTT-MLP (M)  at 1.3B scale.
As discussed, it is hard to derive an empirical scaling law due to the lack of a clean linear fit. However, the strong trend for TTT-MLP (T)  suggests that the Transformer backbone might be more suitable for larger models and longer context beyond our evaluations.We only ablate the backbones for 2k and 32k due to the cost of training LLMs. For future work, we believe that given TTT layers with even more expressive hidden states, the Mamba backbone with temporal convolutions will become unnecessary.Transformer finetuning.While we have been training Transformers from scratch following the Mamba paper, in practice this approach is rarely used for long context.
The standard practice is to train a Transformer in short context, then finetune in long context.
To reflect this practice, we add another baseline,TF finetune, for context lengths 4k and above.
This baseline starts from the model trained (according to the Chinchilla recipe)  on Books 2k, then uses 20% more tokens to finetune at the designated context length, following the Llama Long paper.
See details of the TF finetune recipe in AppendixC.Our complete results for context lengths 1k, 2k, 4k, 8k, 16k, 32k, including TF finetune, are in Figure18(in Appendix) .Figure 13:Experiments on Books with context lengths ranging from 1k to 32k.
We treat context length as a hyper-parameter and connect the selected points.
Since we have Transformers trained from scratch and finetuned, we label them asTF pretrainandTF finetune.
The left panel of Figure2is a zoomed-in view between 350M and 1.3B parameters, whereTransformerisTF finetune, the stronger Transformer baseline.Context length as a hyper-parameter.While the length of the input sequence is determined by the user, the context length at which the language model processes the input is determined by the engineer as a design choice.
Therefore, context length is a hyper-parameter that can be selected just as other ones.131313To be precise, there are two hyper-parameters: the context length at which the LLM is trained, and one at which the LLM is evaluated.
Both of them can be different from the sequence length, which is determined by the user.
Transformers tend to perform poorly when the evaluation context is longer than the training context.
Therefore, we always evaluate at the training context length, making the two hyper-parameters the same.For LLMs with linear complexity, we select the argmin in perplexity, since every context length has the same FLOPs.
For Transformers, longer context costs more FLOPs, so we form a convex hull of all the points in the log-log plot and connect those on the boundary.From Figure13, we make a few observations:•The lines of TTT-Linear and TTT-MLP, the best-performing methods, almost completely overlap.
The lines of Mamba and TF finetune also mostly overlap afterFLOPs.•TF finetune performs significantly better than TF pretrain, as it benefits from long context without incurring extremely large cost in training FLOPs.
Note that the inference FLOPs of TF finetune and pretrain are equally poor, which is not reflected in this plot.•For all methods trained from scratch (including TF pretrain) , perplexity becomes worse once the context length becomes too large.
This trend is highlighted in Figure19(in Appendix) .
We leave further investigation of this trend to future work.The left panel of Figure2is a zoomed-in view of Figure13.
For clarity, we leave TF pretrain out of Figure2and only show TF finetune (labeled asTransformer)  since it is the stronger baseline.
Figure14reproduces the right panel of Figure2, now with TTT-MLP and additional discussion.Figure 14:Reproduction of the right panel of Figure2, now with TTT-MLP.
Comparing to TTT-Linear, TTT-MLP performs slightly worse at short context but better at long context.
This observation matches our expectation that the MLP as hidden state is more expressive than the linear model.
Again, all methods have matched training FLOPs as Mamba 1.4B.
For TTT-Linear and TTT-MLP, this protocol implies matched inference FLOPs.
Transformer (TF finetune)  hasthe inference FLOPs, giving it an advantage as our baseline.3.3Wall-clock timeFigure 15:Benchmark on an NVIDIA A100 GPU with 80G HBM and PCIe connections.Left: Same as Figure3, but now with TTT-MLP.
Note that our Transformer baseline is significantly faster then that in the Mamba paper, because we use vLLM, a state-of-the-art serving system, instead of the HuggingFace Transformer.Right: For generate (decode) , TTT-Linear and Mamba have almost the same latency, which is significantly smaller than that of Transformer and TTT-MLP.LLM training and inference can be decomposed into forward, backward, and generate.
Prompt processing during inference, also known as prefill, is the same operation as forward during training, except that the intermediate activations do not need to be stored for backward.
Since both forward (during training and inference)  and backward can be parallelized, we use the dual form.
Generating new tokens, also known as decode, is inherently sequential, so we use the primal form.Due to resource constraints, our experiments are written in JAX and run on TPUs.
On a v5e-256 TPU pod, the Transformer baseline takes 0.30s per iteration of training at context 2k, while TTT-Linear takes 0.27s per iteration, already 10% faster without any systems optimization.
However, Mamba (implemented in PyTorch, Triton, and CUDA)  can only run on GPUs, so for fair comparison, we also rewrite our method with preliminary systems optimization to run on GPUs.Specifically, we write a GPU kernel for forward in ThunderKittens.
Historically, RNNs have been inefficient during forward and backward due to poor use of parallelism andmatmuls.
Our goal with the forward kernel is to demonstrate the effectiveness of mini-batch TTT and the dual form for these problems.
A kernel for backward should have the same properties in efficiency as forward, but requires more complex logic for manual differentiation, therefore is left for future work.The left panel of Figure15shows the latency for batch size 16 of our forward kernel.
All models are 1.3B (1.4B for Mamba) .
Time per token grows linearly for Transformer as context length increases, but stays roughly constant for the other methods.141414We observe that forward latency of the network increases slightly for TTT-Linear, TTT-MLP, and Mamba, even though latency of each sequence modeling layer alone stays constant.
Consider the operation, whereisandis.
Its latency (normalized over)  is expected to be constant, but in practice grows slightly with.
One possible cause of this phenomenon is the GPU throttling aftergets very large.Note that our Transformer baseline is significantly faster that in the Mamba paper, because we use vLLM, a state-of-the-art serving system, instead of the HuggingFace Transformer.In addition, we write another GPU kernel for generate in Triton, and benchmark its speed in the right panel of Figure15for batch size 512.
Another popular metric for wall-clock time is throughput, which takes into account the potential benefit of being able to use a larger batch size.151515To calculate throughput for each method, we increase its batch size inincrements until GPU runs out of memory, measure the tokens per second for every batch size, and select the highest.For completeness, we report the throughput for forward and generate in Figure20(in Appendix) . All the observations and ordering between methods above still hold for throughput.
body
Learning to (Learn at Test Time):RNNs with Expressive Hidden States
3Experiments
2407.04620
4Related Work4.1Modern RNNsMamba is one of the many Structured State-Space Models. The hidden state in these models is a vector, similar to in LSTMs.
For TTT-Linear or TTT-MLP, the hidden state is a matrix or two matrices, therefore larger.
In Figure14, we find that TTT layers can take advantage of their larger hidden states to compress more information in long context, where TTT-MLP outperforms TTT-Linear, which in turn outperforms Mamba.Similar to TTT-Linear, RWKV, xLSTM, and Gated Linear Attention (GLA) also have matrix hidden states, which are inherited from linear attention.
Modern RNNs such as GLA use chunk-wise parallelism to improve hardware efficiency, so tokens inside a chunk can be processed withmatmuls instead of acumsum.
However, chunk-wise parallelism does not change the expressiveness of the model, since all temporal dependencies are still equivalent to acumsum.In contrast, mini-batch TTT allows more complex temporal dependencies across mini-batches. Each hidden statedepends on previouss within its mini-batch still through acumsum, but depends ons in previous mini-batches also through the gradient operator.
As illustrated FigureLABEL:fig:batch-size, mini-batch TTT enables a trade-off between expressiveness and hardware efficiency, since a smaller batch sizeleads to better perplexity at the cost of higher latency.
This trade-off is a unique and important feature of TTT. As shown in Table1, the intermediate batch sizesignificantly outperformswhich is fullycumsum.4.2Learning at Test TimeThe idea of learning at test time has a long history in machine learning.
One of the earliest versions of this idea is called local learning (Bottou and Vapnik) :
For each test input, train on its neighbors before making a prediction.
This procedure has been effectively applied to models ranging from SVMsto modern LLMs.Another early version of learning at test time is calledtransductive learning.
The principle of transduction, as stated by Vladimir Vapnik, is to
"… get the answer that you really need, but not a more general one."
Practical implementations of transductive learning use test data to add constraints to the margin of SVMs.
However, transductive learning usually needs multiple test instances to be empirically effective, unlike many instantiations of test-time training, which only need a test single instance (image, video, or natural language sequence)  at a time.In computer vision, the idea of learning at test time has been applied for decades to applications such as face detection, object detection, image super-resolution, and 3D reconstruction.
More recently, the same idea has also been applied to natural language processing, where it is called dynamic evaluation.
The basic approach is to directly finetune a language model on the test sequence, which often comes in the form of a prompt.Next, we discuss two relevant lines of work in detail: test-time training and fast weights.4.2.1Test-Time TrainingThe core idea ofTest-Time Training(TTT)  is that each test instance defines its own learning problem, where this test instance alone is the target of generalization.
Concretely, for each test instance, the conventional practice is to predict, using a predictorthat is optimized for all training instances on average.
TTT first formulates a learning problem defined by, then trains a modelon(often withas initialization) , and predicts.Since the test instance comes without its label, the learning problem can only be formulated with a self-supervised task. Prior work has shown that TTT with reconstruction significantly improves performance especially on outliers.
Improvements become even more pronounced when testing on video frames that arrive in a stream and TTT is autoregressive, asis trained on past frames.
The autoregressive connection makesmost relevant to our paper.Conceptually, the biggest difference between our paper and prior work is that our reconstruction task is learned in an outer loop, instead of handcrafted with human priors.
Follow-up work to TTT has explored applications such as
robot manipulationand locomotion,
among others, that often require different designs for the self-supervised task.4.2.2Fast WeightsThe general idea offast weightsis to update the parameters of a “fast” model on only the most relevant data, as opposed to the conventional practice of updating a “slow” model on all data.
This idea has existed since the 1980s.
The most relevant data can be the test instance itself, therefore TTT can be viewed as a special case of fast weights.Prior work in fast weights usually avoids forming an explicit learning problem that optimizes some objective on data.
For example, the update rule of Hebbian learning and Hopfield networkssimply adds(or some variant thereof) to the fast weights given each input.
In contrast, TTT embraces the idea of formulating an explicit learning problem, where the test instance is the target of generalization.
Our update rule is also an explicit step of optimization.The idea offast weight programmers(FWPs)  is to update the fast weights with a “slow” model.
Our inner-loop weightscan be viewed as “fast” and outer-loop weightsas “slow”.
Therefore, networks containing TTT layers can be viewed as a special case of FWPs, similar to how TTT can be viewed as a special case of fast weights.
The FWP with the Hebbian update rule above is equivalent to linear attention, therefore also to naive TTT-Linear with batch gradient descent.The definition of FWPs is very broad.
In fact, all networks with some gating mechanism, such as Transformers with SwiGLU blocks, can also be viewed as a special case of FWPs161616Consider a simple gate, whereis the input,is the output,andare learnable weight matrices,is element-wise multiplication, andis the sigmoid function.
A well known interpretation is to viewas the fast weights controlled by slow weights, then equivalently,is simply a two-layer MLP with fast weights..
Recent work has been experimenting with FWPs for language modeling: Irie et al.design “fast” networks with weights produced as output of a “slow” networks.
Clark et al.give a Transformer a final layer of fast weights, whose initialization is trained as slow weights.
Our contribution relative to existing work on FWPs, again, is formulating an explicit learning problem for the update, which enables us to borrow tools from learning such as mini-batch and LN.4.3Learning to LearnFor decades, researchers have been arguing that learning to learn, also known as meta-learning or bi-level optimization, should be a critical component of intelligence.
In prior work such as,and, the inner loop learns from an entire dataset at a time instead of a sequence, so the outer loop needs a collection of datasets or tasks.
In short, the outer loop is “one level above” regular training.
Since it is hard to collect millions of datasets, this outer loop is hard to scale.In contrast, for TTT, each sequence itself is a dataset and defines its own generalization problem. The inner loop is “one level below” regular training, so our outer loop is only another solution to the canonical problem of supervised learning,
instead of a new problem setting like generalization across datasets.
As illustrated in Table2, our outer loop is “at the same level” as regular training.
This makes our outer loop easier to scale.Inner loopOuter loopSubsectionPiece of dataTokenSequence2.1,2.2Training setSequenceDataset of sequences, e.g., BooksObjectiveReconstruction (lossNext-token predictionParameters(weights of) (rest of the network) (reconstruction views) 2.3and2.7Table 2:In summary, our paper reformulates supervised learning as learning to learn, with two nested loops.
Highlighted rows of the outer loop are the same as in the regular training.
Parameters of the outer loop become hyper-parameters of the inner loop.
Intuitively, the inner loop,i.e.TTT, is “one level below” regular training.
body
Learning to (Learn at Test Time):RNNs with Expressive Hidden States
4Related Work
2407.04620
5DiscussionWe have reformulated the canonical problem of supervised learning as learning to (learn at test time) .
Our formulation produces an alternative conceptual framework for building what is traditionally known as network architectures.
We summarize our current instantiation in Table2.The search space for effective instantiations inside this framework is huge, and our paper has only taken a baby step.
Fortunately, if our perspective holds, then heuristics from regular training can transfer to test-time training, and search can be efficient.
Next we outline some especially promising directions for future work.•Outer-loop parameterization.There are many other ways to parameterize a family of multi-view reconstruction tasks, or perhaps a more general family of self-supervised tasks. It would be a big coincidence if the first one we have tried turns out to be the best.•Systems optimization.Our systems optimization in Subsection3.3has been preliminary at best, and there are many ways to improve it.
In addition, pipeline parallelism through time might allow us to process long sequences of millions of tokens on multiple devices together.•Longer context and larger models.Constrained by our academic resources, we have not trained with millions or billions in context length, which would also require larger models according to Figure19.
The advantage of TTT layers should become more pronounced in longer context.•More ambitious instantiations of.When context length becomes longer,would also need to be larger.
For video tasks and embodied agents, whose context length can easily scale up to millions or billions,could be a convolutional neural network.•Multi-level learning to learn.Ifitself is a self-attention layer, then by Theorem2it can be interpreted as yet another inner loop nested inside the existing one. In this fashion, we can potentially build many levels of nested learning problems.Why do we study TTT? First a more basic question: Why study AI? For some of us, AI is a playground to probe about the nature of human intelligence.
Prior work often tries to model human learning with machine learning, where training is on a shuffled dataset with i.i.d. instances, and inference is on a separate test set.
However, humans do not naturally learn with i.i.d. instances or have a train-test split.
We believe that human learning has a more promising connection with TTT, our inner loop, whose data is a potentially very long sequence with strong temporal dependencies, and any piece of data can be used for both training and testing. This is why we study TTT.
body
Learning to (Learn at Test Time):RNNs with Expressive Hidden States
5Discussion
2407.04620
Author ContributionsYu Sunstarted this project with Xinhao Li in November 2022, and has been working on it full-time since June 2023.
Yu proposed the conceptual framework of the project, designed mini-batch TTT and the dual form, wrote the paper with help from others, and led the daily operations of the team.Xinhao Listarted this project with Yu Sun in November 2022, and has been working on it full-time since then.
Xinhao and Karan co-led the development of our current codebase.
Before March 2024, Xinhao was the primary contributor to our earlier codebases that shaped this project.
Xinhao made significant contributions to the project direction in discussions.Karan Dalaljoined this project full-time in June 2023.
In collaboration with Xinhao, he co-led the development of our current codebase.
Karan managed the experiments in Section3, helped write the paper, and made significant contributions to the project direction in discussions.Jiarui Xujoined this project in March 2024. He led our architectural development since he joined, and made significant contributions to the project direction in discussions.Arjun Vikramjoined this project in September 2023. He made significant contributions to our systems optimization, as well as current and earlier codebases.Genghan Zhangjoined this project in January 2024. He provided critical insights and made significant improvements to our systems optimization.Yann Duboisjoined this project in February 2024. He proposed our current instantiation of, and made significant contributions to the project direction in discussions.Xinlei Chen and Xiaolong Wanghave been supporting this project since November 2022, and the direction of test-time training for many years. Without their support in compute and organization, this project could not have survived its early stage.
They gave invaluable advice to our experiments.Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrinhave been supporting this project since May 2023. They gave invaluable advice to our experiments and presentation.
For example, Sanmi suggested us to focus on TTT-Linear,
Tatsu suggested the experiments in Figure2(left) , and Carlos outlined Section2.
body
Learning to (Learn at Test Time):RNNs with Expressive Hidden States
Author Contributions
2407.04620
AcknowledgementsPart of the compute for this project is generously supported by the Google TPU Research Cloud program.
XW is supported, in part, by the Amazon Research Award, the Cisco Faculty Award and the Qualcomm Innovation Fellowship.
SK acknowledges support by NSF 2046795 and 2205329, NIFA award 2020-67021-32799, the Alfred P. Sloan Foundation, and Google Inc.
TH is supported by a Sony Faculty Innovation Award and a gift from Panasonic.
CG acknowledges support by the Air Force Office of Scientific Research (AFOSR) , FA9550-20-1-0427, Stanford Human-Centered Artificial Intelligence (HAI)  Institute, and gifts from Google and IBM.We would like to thank Rohan Taori, Xuechen Li, Allan Zhou, Ke Chen, and Guandao Yang for many helpful discussions,
Menghao Guo for help with code release,
Xinyang Geng for help with EasyLM,
Hao Liu for help with the LWM codebase,
David Hall for help with Levanter,
Yossi Gandelsman and Yutong Bai for help at an early stage of the project,
Mert Yuksekgonul for help with figures in the paper,
Horace He and Azalia Mirhoseini for help with systems,
Sharad Vikram and Roy Frostig for answering our questions about JAX and Pallas,
Albert Gu and Tri Dao for helping us reproduce experiments in the Mamba paper,
and Kilian Weinberger and Percy Liang for advice on presentation.
Yu Sun is grateful to his PhD advisors,
Alexei A. Efros and Moritz Hardt, for their many insights from years ago that eventually became part of this paper.
conclusion
Learning to (Learn at Test Time):RNNs with Expressive Hidden States
Acknowledgements